Data Store Fundamentals

Welcome to datastore internals

This is the first lesson in Build Datastores From Scratch, so our goal is to build the mental map before we build code.

A datastore is not just a place where data lives. It is a set of promises about:

• How data is shaped
• How it is found
• How safely it is saved
• How quickly it responds
• What tradeoffs it accepts

Later lessons will build small versions of Redis, MongoDB, PostgreSQL, and Elasticsearch-style systems in Python. Today, we define the vocabulary you will use to understand every one of them.

A datastore wraps data with guarantees

At the lowest level, every datastore manages some structure: arrays, hash maps, trees, files, logs, indexes, or combinations of them.

But users do not interact with raw structures. They interact with an interface:

application request
        |
        v
  datastore API
        |
        v
storage + indexes + rules
        |
        v
      result

Here is a tiny Python example. It is not a real database, but it shows the core idea: a datastore exposes operations while hiding the internal representation.

```python
class TinyDataStore:
    def __init__(self):
        self.collections = {}

    def write(self, collection, record):
        self.collections.setdefault(collection, [])
        record_id = len(self.collections[collection])
        stored = dict(record)
        stored['id'] = record_id
        self.collections[collection].append(stored)
        return record_id

    def read(self, collection, record_id):
        return self.collections[collection][record_id]

    def scan(self, collection, predicate):
        return [row for row in self.collections.get(collection, []) if predicate(row)]

store = TinyDataStore()
store.write('trades', {'symbol': 'AAPL', 'price': 189})
store.write('trades', {'symbol': 'MSFT', 'price': 421})

print(store.read('trades', 0))
print(store.scan('trades', lambda row: row['price'] > 200))
```

A finance app needs to store account balances, transfers, and audit records. It must enforce relationships like: every transfer belongs to two valid accounts.

Which datastore style is usually the most natural starting point?

The data model shapes the system

A data model is the way a datastore wants you to organize information.

Common models include:

Model Core shape Natural question Key-value key → value What is stored under this key? Document nested JSON-like objects Which documents match these fields? Relational tables + rows + columns How do structured records relate? Search index terms → matching documents Which documents contain this text?

The same real-world fact can be represented differently depending on the model.

Example event: user 42 bought 3 shares of NVDA at 875.

• Key-value: trade:9001 -> serialized trade
• Document: { user_id, symbol, quantity, price }
• Relational: row in trades, linked to users
• Search: tokens like NVDA, bought, trade point to document 9001

True or false?

A datastore's data model strongly influences which queries are easy, fast, and natural to support.

Memory is fast; persistence survives

One of the first tradeoffs in datastore design is where data lives.

• In memory: very fast, but disappears when the process exits unless saved elsewhere.
• On disk: slower, but can survive restarts.
• Hybrid: common in real systems; memory for speed, disk for recovery.

Below is a tiny example of persistence. We write records to a JSON file, then create a new object and reload them. That reload step is the difference between temporary state and durable state.

```python
import json
import tempfile
from pathlib import Path

class JsonFileStore:
    def __init__(self, path):
        self.path = Path(path)
        self.records = []
        if self.path.exists():
            self.records = json.loads(self.path.read_text())

    def append(self, record):
        self.records.append(record)
        self.path.write_text(json.dumps(self.records))

with tempfile.TemporaryDirectory() as directory:
    path = Path(directory) / 'store.json'

    first_process = JsonFileStore(path)
    first_process.append({'symbol': 'NVDA', 'price': 875})

    second_process = JsonFileStore(path)
    print(second_process.records)
```

SQL vs NoSQL is about tradeoffs, not teams

The phrase SQL database usually refers to relational systems with tables, schemas, joins, and a query language based on SQL.

The phrase NoSQL database covers several non-relational families, including key-value, document, column-family, graph, and search-oriented systems.

A useful way to think about the difference:

• SQL systems often optimize for structured data, relationships, constraints, and transactions.
• NoSQL systems often optimize for flexible data shapes, horizontal scaling, specialized access patterns, or high throughput.

Neither category is automatically better. A good engineer asks: What are the reads, writes, correctness requirements, scale needs, and failure modes?

In a relational database, the formal definition of tables, columns, data types, and constraints is called a ____.

The course systems at a glance

We will explore four major datastore families by building small versions from scratch.

Redis-style key-value store

Think: fast lookup by key, often in memory.

MongoDB-style document store

Think: flexible JSON-like documents, queryable by fields.

PostgreSQL-style relational database

Think: tables, schemas, indexes, joins, and transactions.

Elasticsearch-style search engine

Think: text indexing, token lookup, and relevance-oriented retrieval.

Each one starts from ordinary programming primitives. The magic comes from layering data structures, APIs, persistence, indexing, and query behavior carefully.

The questions we will keep asking

As we build each datastore, keep these design questions nearby:

1. Access pattern: Do users fetch by key, filter by fields, join tables, or search text?
2. Data shape: Is the data uniform, nested, relational, or unstructured?
3. Durability: What happens after a crash or restart?
4. Performance: Which operations must be fast?
5. Correctness: What rules must never be violated?
6. Scale: Does the system need to handle more data, more users, or both?

In the next lesson, we will start with the simplest useful case: designing a small key-value store.