Snapshot Recovery and Gap Handling

Why this lesson matters

Now that we've covered Market Data Handling & Order Book Reconstruction, you can build a simple Level 2 book and detect missing sequence numbers.

Today we handle the uncomfortable part:

What should the system do after it knows the book may be wrong?

A gap is not just a logging event. It changes whether strategies may trust the market state.

incrementals arrive normally
        |
        v
gap detected -> book becomes unsafe -> snapshot recovery -> book becomes live again

This matters before moving into execution algorithms because TWAP, VWAP, POV, and market making all depend on a clean market view. A strategy should not size, price, or cancel orders from a corrupted book.

Treat recovery as a state machine

Beginners often write gap handling as one if statement. In real trading code, it is safer to model the book as being in one of a few states.

LIVE
  apply expected incrementals
  publish book to strategies

RECOVERING
  stop publishing
  buffer incoming incrementals
  wait for snapshot

LIVE again
  install snapshot
  replay valid buffered incrementals
  publish only after continuity is restored

The key behavior is fail closed: once the handler detects a gap, the book is not usable until proven safe again.

This mirrors the mindset from Pre-Trade Risk Checks: if the system cannot prove safety, it should block risky downstream behavior.

A handler expects sequence 205, but receives incremental update 208.

What is the safest immediate response?

Why buffering exists

Snapshot recovery is not instant. While the handler waits for a snapshot, more incremental messages may arrive.

If we discard those messages, we may install a snapshot and immediately be behind again.

So a common pattern is:

1. Gap detected at expected N
2. Enter RECOVERING
3. Keep incoming incrementals in a buffer
4. Receive snapshot with sequence S
5. Reset book to snapshot
6. Replay buffered updates with seq > S

Important detail: if a snapshot says it is current through sequence S, then update S is already included in the snapshot. The replay should start after S, not at S.

```cpp
#include <iostream>
#include <vector>
#include <string>

struct Incremental {
    long seq;
    std::string text;
};

int main() {
    long expectedSeq = 100;
    bool live = true;
    std::vector<Incremental> buffer;

    std::vector<Incremental> incoming = {
        {100, "bid 50 @ 10.00"},
        {101, "ask 30 @ 10.05"},
        {104, "bid 20 @ 9.99"},   // gap: 102 and 103 missing
        {105, "ask 0 @ 10.05"},
        {106, "bid 80 @ 10.01"}
    };

    for (const auto& msg : incoming) {
        if (live && msg.seq == expectedSeq) {
            std::cout << "LIVE apply " << msg.seq << ": " << msg.text << "\n";
            ++expectedSeq;
        } else if (live && msg.seq > expectedSeq) {
            std::cout << "GAP detected. expected " << expectedSeq
                      << " got " << msg.seq << "\n";
            std::cout << "Enter RECOVERING and buffer seq " << msg.seq << "\n";
            live = false;
            buffer.push_back(msg);
        } else if (!live) {
            std::cout << "RECOVERING buffer seq " << msg.seq << "\n";
            buffer.push_back(msg);
        }
    }

    long snapshotSeq = 103;
    std::cout << "\nSnapshot received through seq " << snapshotSeq << "\n";
    expectedSeq = snapshotSeq + 1;

    for (const auto& msg : buffer) {
        if (msg.seq == expectedSeq) {
            std::cout << "Replay " << msg.seq << ": " << msg.text << "\n";
            ++expectedSeq;
        }
    }

    live = true;
    std::cout << "Book live again at next expected seq " << expectedSeq << "\n";
    return 0;
}
```

True or false:

If a snapshot is labeled sequence 500, the recovery logic should replay buffered incremental updates with sequence >= 500.

Reset first, then replay

Recovery should not patch the corrupted book in place. The safer pattern is:

1. Clear the current book
2. Install the full snapshot levels
3. Set expectedSeq = snapshotSeq + 1
4. Replay buffered incrementals in exact order
5. Publish only if there is no new gap

Below is a compact reference implementation. It is intentionally beginner-friendly and uses std::map, like the previous order book lesson. The logic matters more than micro-optimizing the container today.

```cpp
#include <functional>
#include <iostream>
#include <map>
#include <vector>

struct Level {
    char side;       // 'B' or 'A'
    double price;
    int qty;
};

struct Snapshot {
    long seq;
    std::vector<Level> levels;
};

struct Update {
    long seq;
    char side;
    double price;
    int qty;         // 0 means delete
};

class Book {
    std::map<double, int, std::greater<double>> bids;
    std::map<double, int> asks;

public:
    void clear() {
        bids.clear();
        asks.clear();
    }

    void applyLevel(const Level& l) {
        if (l.side == 'B') bids[l.price] = l.qty;
        else asks[l.price] = l.qty;
    }

    void applyUpdate(const Update& u) {
        auto& side = (u.side == 'B') ? bids : asks;
        if (u.qty == 0) side.erase(u.price);
        else side[u.price] = u.qty;
    }

    void printTop() const {
        std::cout << "bid ";
        if (bids.empty()) std::cout << "EMPTY";
        else std::cout << bids.begin()->first << " x " << bids.begin()->second;

        std::cout << " | ask ";
        if (asks.empty()) std::cout << "EMPTY";
        else std::cout << asks.begin()->first << " x " << asks.begin()->second;
        std::cout << "\n";
    }
};

int main() {
    Book book;

    Snapshot snap{103, {
        {'B', 100.10, 500},
        {'B', 100.05, 900},
        {'A', 100.20, 300}
    }};

    std::vector<Update> buffered = {
        {104, 'B', 100.10, 0},
        {105, 'A', 100.18, 200},
        {106, 'B', 100.12, 100}
    };

    book.clear();
    for (const auto& level : snap.levels) {
        book.applyLevel(level);
    }

    long expectedSeq = snap.seq + 1;
    bool recovered = true;

    for (const auto& u : buffered) {
        if (u.seq == expectedSeq) {
            book.applyUpdate(u);
            ++expectedSeq;
        } else {
            std::cout << "Recovery failed at seq " << u.seq
                      << ", expected " << expectedSeq << "\n";
            recovered = false;
            break;
        }
    }

    std::cout << "Recovered? " << (recovered ? "yes" : "no") << "\n";
    book.printTop();
    return 0;
}
```

Not every snapshot is useful

A snapshot can arrive late or be older than the updates you need.

Example:

Gap detected: expected 200, received 205
Buffered: 205, 206, 207
Snapshot arrives with seq 198

This snapshot does not bridge the missing range. If we install it, we still need 199, 200, 201, 202, 203, and 204 before 205 makes sense.

Practical handlers may respond by:

• requesting another snapshot
• waiting for a newer periodic snapshot
• dropping buffered messages that are too old
• keeping the book unpublished until recovery succeeds

The rule is simple: do not declare recovery complete unless the snapshot plus buffer creates a continuous sequence path forward.

During snapshot recovery, incremental updates that arrive while the handler waits for a snapshot are usually stored in a temporary ________.

Recovery must protect downstream code

The book builder is not the final consumer. Strategies, risk checks, and the OMS may all depend on book state.

So the feed handler should expose a clear publish gate:

book_state == LIVE       -> strategy may read/publish/use top of book
book_state == RECOVERING -> strategy must not use this symbol's book

This does not mean the whole trading process must stop. A real system may mark only one symbol, one venue, or one feed as unavailable. But the affected book should not silently continue as if it were clean.

The next lesson, Core Execution Algorithms (TWAP, VWAP, POV), will assume this gate exists before execution logic decides when and how much to trade.

```cpp
#include <iostream>
#include <string>

enum class BookState {
    Live,
    Recovering
};

struct TopOfBook {
    double bid;
    double ask;
};

void strategyRead(const std::string& symbol, BookState state, const TopOfBook& top) {
    if (state != BookState::Live) {
        std::cout << symbol << ": skip strategy read; book is recovering\n";
        return;
    }

    std::cout << symbol << ": usable market state bid=" << top.bid
              << " ask=" << top.ask << "\n";
}

int main() {
    TopOfBook top{100.10, 100.20};

    strategyRead("XYZ", BookState::Live, top);
    strategyRead("XYZ", BookState::Recovering, top);

    return 0;
}
```

A snapshot arrives and buffered updates are available. Which condition must be true before the handler marks the book live again?

What you should remember

Snapshot recovery is where feed handling becomes operationally serious. The goal is not to pretend gaps never happen. The goal is to contain them.

Use this checklist when reviewing a book builder:

• Track expectedSeq per feed stream or symbol as required by the venue
• On a forward gap, mark the affected book unsafe
• Stop publishing unsafe book state to strategies
• Buffer incoming incrementals during recovery
• Install only a snapshot with a usable sequence boundary
• Replay buffered updates with seq > snapshotSeq
• Require strict continuity during replay
• Log recovery success or failure for monitoring

This completes the State, Risk, Orders section: you can now connect market state to order state with safety gates around both data and orders.

One Pager Cheat Sheet

• A sequence gap makes the book unsafe until snapshot recovery restores a trusted state.
• Model gap recovery explicitly: live books publish; recovering books buffer and stay hidden.
• If received sequence is greater than expected, stop trusting the book and recover.
• During recovery, buffer incoming incrementals and replay only those after the snapshot sequence.
• After installing snapshot S, replay buffered incrementals with sequence greater than S.
• Recovery resets the book from a full snapshot, then replays contiguous buffered updates.
• A snapshot only helps if it lets the handler rebuild a continuous sequence path.
• Recovery uses a buffer to hold incrementals that arrive while the snapshot is fetched.
• Expose book health to downstream code so strategies skip symbols under recovery.
• Only mark recovery complete after snapshot and replay restore sequence continuity.
• Gap handling is a safety system: detect, isolate, rebuild, replay, and only then republish.