System Design Principles

From Data To Systems

Now that we've covered Introduction to Databases, you have the core mental model for structured data, queries, transactions, and indexes. System design asks the next question:

How do we build an entire backend system around that data so it stays useful under real traffic, failures, and change?

As an experienced coder, you already know how to implement features. A senior backend engineer also explains why a design will survive production conditions.

In this lesson, we'll focus on principles, not named architecture patterns. Those come next in Backend Architecture Patterns.

By the end, you should be able to reason about:

• Requirements before implementation
• Trade-offs instead of perfect answers
• Scalability and bottlenecks
• Reliability and failure boundaries
• Consistency as a product decision
• Design review habits used by senior engineers

Design Is Tradeoffs

A common junior mistake is searching for the "best" design. Senior engineers ask a sharper question:

Best under which constraints?

Every meaningful backend design trades one quality for another:

Goal Often Costs Lower latency More caching, complexity, stale data risk Higher reliability More redundancy, operational overhead Stronger consistency Lower availability or higher coordination cost Faster development Less flexibility later Higher scalability More moving parts and observability needs

A design principle is not a rule you blindly follow. It is a decision lens.

A practical senior-engineer framing

For each major design choice, be ready to say:

1. What problem does this solve?
2. What does it make harder?
3. What assumption would make us revisit it?

That third question is powerful. Systems age because traffic, teams, data shape, and user expectations change.

Choose The Constraint

A product team asks you to design a backend endpoint for a flash-sale checkout flow. During the sale, traffic can spike dramatically. Users care most that confirmed orders are not lost or duplicated.

Which design concern should be treated as the primary constraint?

Estimate Before Designing

System design starts with rough numbers. You do not need perfect precision. You need enough scale awareness to avoid obviously weak designs.

Back-of-the-envelope estimation helps answer questions like:

• Is this read-heavy or write-heavy?
• Can one database handle the expected writes?
• How much storage grows per day?
• Is caching likely to help?
• Which component will become expensive first?

Suppose we are designing an activity feed service. Before picking tools, estimate request volume and storage growth.

```python
# Back-of-the-envelope estimation for an activity feed service

users = 2_000_000
daily_active_ratio = 0.25
avg_feed_reads_per_active_user = 12
avg_posts_per_active_user = 0.4
avg_post_size_kb = 3

active_users = users * daily_active_ratio
reads_per_day = active_users * avg_feed_reads_per_active_user
writes_per_day = active_users * avg_posts_per_active_user

seconds_per_day = 24 * 60 * 60
read_qps = reads_per_day / seconds_per_day
write_qps = writes_per_day / seconds_per_day
storage_gb_per_day = (writes_per_day * avg_post_size_kb) / (1024 * 1024)

print(f"Active users: {active_users:,.0f}")
print(f"Read QPS: {read_qps:,.2f}")
print(f"Write QPS: {write_qps:,.2f}")
print(f"New storage per day: {storage_gb_per_day:.2f} GB")

if read_qps > write_qps * 10:
    print("Observation: This is strongly read-heavy; caching or precomputation may be useful.")
else:
    print("Observation: Reads and writes are closer; optimize both paths carefully.")
```

True or False?

A backend system must use microservices in order to be considered scalable.

Find The Bottleneck

A backend request path is only as strong as its slowest overloaded dependency.

Think of a request as a chain:

Client
  -> API layer
    -> business logic
      -> cache?
        -> database
          -> external service?

If the database saturates, adding more API servers may only produce more pressure. If an external payment provider is slow, your perfectly tuned database will not save checkout latency.

A senior design habit

For every system, identify:

• Hot path: the most frequent or latency-sensitive flow
• Critical dependency: the service/data store that must succeed
• Saturation point: what breaks first as traffic grows
• Backpressure strategy: what happens when demand exceeds capacity

Backpressure is especially important. A healthy system should degrade intentionally instead of failing chaotically.

Fill In The Blank

The maximum load a system can handle while still meeting its target latency and error-rate requirements is called its ____.

Design For Failure

Production systems fail in boring, predictable ways:

• Network calls time out
• Dependencies become slow
• Databases hit connection limits
• Queues grow faster than workers consume
• A retry policy accidentally multiplies traffic

The principle is not "avoid all failure." That is impossible.

The principle is:

Contain failure so one weak component does not collapse the whole system.

Common containment tools include:

• Timeouts: never wait forever
• Retries with limits: retry transient failures, but avoid storms
• Idempotency: make repeated requests safe when possible
• Circuit breakers: stop calling a dependency that is already failing
• Graceful degradation: return partial functionality instead of total failure

The code below shows why uncontrolled retries can amplify load.

```python
# Simple retry amplification model

incoming_requests = 1_000
failure_rate = 0.30
max_retries = 3

attempts = 0
remaining_failed_requests = incoming_requests

for attempt_number in range(max_retries + 1):
    current_attempts = remaining_failed_requests
    attempts += current_attempts
    failed = int(current_attempts * failure_rate)

    print(
        f"Attempt {attempt_number + 1}: "
        f"sent={current_attempts}, failed={failed}"
    )

    remaining_failed_requests = failed

print(f"Original incoming requests: {incoming_requests}")
print(f"Total dependency calls after retries: {attempts}")
print(f"Traffic multiplier: {attempts / incoming_requests:.2f}x")

if attempts > incoming_requests * 1.5:
    print("Warning: retries can significantly amplify pressure on a weak dependency.")
```

Consistency Is Product

In Introduction to Databases, you learned that transactions protect meaning. In system design, you decide where that protection must be strongest and where the product can tolerate delay.

Not every feature needs the same consistency guarantee.

Feature Usually Needs Bank transfer Strong correctness guarantees Checkout inventory Careful consistency around reservations/orders Social like count Often can be slightly stale Analytics dashboard Often delayed aggregation is acceptable

The key question

Ask:

If two users see different answers for a short time, is that acceptable?

If the answer is no, expect coordination costs. If the answer is yes, you may gain lower latency, better availability, or simpler scaling.

This is not just a database decision. It is a product decision expressed through backend behavior.

Review Like Senior

A strong system design review is not a tour of technologies. It is an argument that the system satisfies its most important constraints.

Use this checklist when evaluating a design:

1. Requirements: What must be true for users and the business?
2. Scale assumptions: What traffic, data size, and growth are expected?
3. Critical flows: Which requests matter most?
4. State ownership: Where does important data live?
5. Failure handling: What happens when dependencies slow down or fail?
6. Consistency needs: Where must answers be correct immediately?
7. Observability: How will we know the system is unhealthy?
8. Evolution path: What assumption would force a redesign?

The small scoring example below is not meant to replace engineering judgment. It shows how to make trade-offs explicit instead of hidden.

```python
# A lightweight trade-off scoring model for design review

criteria_weights = {
    "correctness": 5,
    "latency": 3,
    "operational_simplicity": 4,
    "scalability": 4,
    "delivery_speed": 2,
}

options = {
    "simple_monolith_with_one_db": {
        "correctness": 4,
        "latency": 3,
        "operational_simplicity": 5,
        "scalability": 3,
        "delivery_speed": 5,
    },
    "distributed_services_with_async_workflows": {
        "correctness": 4,
        "latency": 4,
        "operational_simplicity": 2,
        "scalability": 5,
        "delivery_speed": 2,
    },
}

def weighted_score(scores):
    return sum(scores[k] * criteria_weights[k] for k in criteria_weights)

for name, scores in options.items():
    print(f"{name}: {weighted_score(scores)}")

winner = max(options, key=lambda name: weighted_score(options[name]))
print(f"Highest score under current priorities: {winner}")
print("Reminder: if priorities change, the best design may change too.")
```