Latency Instrumentation and Histograms

Why this lesson matters

Now that we've covered Low-Latency C++ for HFT, Cache-Aware Data Layout, and Networking & Kernel Bypass, we can make code faster — but only if we can prove where time goes.

A trading path has many places for latency to hide:

packet received -> parse -> update state -> signal -> risk/OMS action

A single average number is not enough. HFT-style systems care about the shape of latency:

• What is typical?
• How bad are spikes?
• Which stage creates the tail?
• Did a code change improve p99, or only the average?

Today we build the measurement habit: instrument boundaries, record durations, and summarize them with histograms and percentiles.

The next lesson, Concurrency & Lock-Free Programming, will use these ideas when we reason about thread handoff latency.

Measure edges, not vibes

A useful latency measurement starts by naming the boundaries.

For a simplified feed-to-decision path:

t0: packet/event available to our process
t1: parsed into typed fields
t2: market state updated
t3: signal decision finished
t4: action emitted internally

Then durations become simple differences:

parse latency      = t1 - t0
state latency      = t2 - t1
signal latency     = t3 - t2
action latency     = t4 - t3
end-to-end latency = t4 - t0

This is more useful than measuring only the whole loop because it tells you which stage changed.

A beginner-friendly rule

Start with coarse boundaries. Once a stage is suspicious, split it further. Do not cover every line of code with timers on day one.

A trading loop has low average latency but occasionally misses opportunities because some events take much longer than usual. Which measurement is most useful for seeing that problem?

Use a monotonic clock for intervals

For measuring elapsed time inside a process, use a clock that moves forward steadily.

In C++, the usual beginner-safe choice is:

std::chrono::steady_clock

Why not wall-clock time? Wall clocks can jump because of NTP corrections, manual changes, daylight saving rules, or system adjustments. Those are useful for human timestamps, but dangerous for tiny interval measurements.

Important caveat

Even a good clock has overhead. Calling the clock inside every tiny operation can change what you are measuring.

So the practical rule is:

• timestamp important boundaries,
• store measurements cheaply,
• print or aggregate outside the hot path when possible.

```cpp
#include <chrono>
#include <iostream>
#include <vector>

using Clock = std::chrono::steady_clock;
using ns = std::chrono::nanoseconds;

int busy_work(int x) {
    // Small deterministic workload so the compiler cannot remove everything.
    int v = x;
    for (int i = 0; i < 200; ++i) {
        v = (v * 1664525 + 1013904223) & 0x7fffffff;
    }
    return v;
}

int main() {
    constexpr int N = 1000;
    std::vector<long long> samples;
    samples.reserve(N);

    int checksum = 0;

    for (int i = 0; i < N; ++i) {
        auto start = Clock::now();
        checksum += busy_work(i);
        auto end = Clock::now();

        samples.push_back(std::chrono::duration_cast<ns>(end - start).count());
    }

    long long min_v = samples[0];
    long long max_v = samples[0];
    long long sum = 0;

    for (long long x : samples) {
        if (x < min_v) min_v = x;
        if (x > max_v) max_v = x;
        sum += x;
    }

    std::cout << "samples: " << samples.size() << "\n";
    std::cout << "min ns:  " << min_v << "\n";
    std::cout << "avg ns:  " << (sum / samples.size()) << "\n";
    std::cout << "max ns:  " << max_v << "\n";
    std::cout << "checksum: " << checksum << "\n";
}
```

Fill in the blank:

For measuring elapsed intervals in C++, prefer std::chrono::____ because it is monotonic and should not jump like wall-clock time.

A histogram shows the shape

An average compresses all samples into one number. A histogram keeps the distribution visible.

Example latency samples:

80 ns, 90 ns, 95 ns, 110 ns, 120 ns, 9000 ns

The average is pulled upward by one spike, but it still does not explain the shape. A histogram can show:

<= 100 ns       ###
<= 250 ns       ##
<= 1000 ns
>  1000 ns      #

For trading systems, this matters because one rare pause can be more damaging than many ordinary events.

Bucket design

Good buckets match the scale you care about:

• nanoseconds for tiny C++ hot-path stages,
• microseconds for network/application boundaries,
• milliseconds for operational monitoring.

Start simple. You can refine bucket edges once measurements show where the data lives.

```cpp
#include <array>
#include <iostream>
#include <string_view>

class LatencyHistogram {
public:
    void record(long long ns) {
        if (ns <= 100) ++buckets_[0];
        else if (ns <= 250) ++buckets_[1];
        else if (ns <= 500) ++buckets_[2];
        else if (ns <= 1000) ++buckets_[3];
        else if (ns <= 2500) ++buckets_[4];
        else if (ns <= 5000) ++buckets_[5];
        else ++buckets_[6];
        ++total_;
    }

    void print(std::string_view name) const {
        static constexpr std::array<std::string_view, 7> labels{
            "<= 100 ns", "<= 250 ns", "<= 500 ns", "<= 1 us",
            "<= 2.5 us", "<= 5 us", "> 5 us"
        };

        std::cout << "Histogram: " << name << "\n";
        long long cumulative = 0;
        for (size_t i = 0; i < buckets_.size(); ++i) {
            cumulative += buckets_[i];
            double pct = total_ == 0 ? 0.0 : 100.0 * cumulative / total_;
            std::cout << labels[i] << "  count=" << buckets_[i]
                      << "  cumulative=" << pct << "%\n";
        }
    }

private:
    std::array<long long, 7> buckets_{};
    long long total_ = 0;
};

int main() {
    LatencyHistogram hist;

    long long samples[] = {82, 91, 120, 180, 240, 430, 900, 1100, 3700, 9200};
    for (long long x : samples) {
        hist.record(x);
    }

    hist.print("parse stage");
}
```

True or false:

It is safe to print one log line for every market event inside the hottest latency path because printing is usually free compared with C++ computation.

Percentiles answer “how bad for most events?”

A percentile tells you the latency value below which a percentage of samples fall.

p50   = median; half of events are faster
p90   = 90% of events are faster
p99   = 99% of events are faster
p99.9 = 99.9% of events are faster

In trading, p50 can look excellent while p99.9 is unacceptable.

Example:

p50 parse latency:   90 ns
p99 parse latency:  300 ns
p99.9 latency:     8 us

That tells a different story than:

average parse latency: 140 ns

Tail latency mindset

A tail spike is not automatically a bug. It may come from packet bursts, cache misses, scheduler noise, page faults, logging, allocation, or measurement overhead. The histogram tells you where to investigate next.

Build a tiny stage profiler

Now we combine the pieces:

1. Take timestamps at stage boundaries.
2. Record each stage duration into a histogram.
3. Print summaries after the loop finishes.

This example simulates three stages from our trading path:

parse -> update state -> decide

It is not a real market data parser; earlier lessons already covered parsing and layout. Here the focus is the instrumentation pattern.

Notice the key habit: no printing inside the measured per-event path.

```cpp
#include <array>
#include <chrono>
#include <iostream>
#include <string_view>

using Clock = std::chrono::steady_clock;
using ns = std::chrono::nanoseconds;

class Histogram {
public:
    void record(long long value_ns) {
        if (value_ns <= 100) ++b_[0];
        else if (value_ns <= 250) ++b_[1];
        else if (value_ns <= 500) ++b_[2];
        else if (value_ns <= 1000) ++b_[3];
        else if (value_ns <= 2500) ++b_[4];
        else if (value_ns <= 5000) ++b_[5];
        else ++b_[6];
        ++total_;
    }

    void print(std::string_view name) const {
        static constexpr std::array<std::string_view, 7> labels{
            "<=100ns", "<=250ns", "<=500ns", "<=1us",
            "<=2.5us", "<=5us", ">5us"
        };

        std::cout << "\n" << name << "\n";
        for (size_t i = 0; i < b_.size(); ++i) {
            std::cout << "  " << labels[i] << " : " << b_[i] << "\n";
        }
    }

private:
    std::array<long long, 7> b_{};
    long long total_ = 0;
};

int fake_parse(int seq) {
    int v = seq;
    for (int i = 0; i < 80; ++i) v = (v * 31 + i) % 100000;
    return v;
}

int fake_update_state(int state, int parsed) {
    for (int i = 0; i < 120; ++i) state = (state + parsed + i) % 100000;
    return state;
}

bool fake_decide(int state) {
    for (int i = 0; i < 60; ++i) state = (state * 17 + i) % 100000;
    return state % 997 == 0;
}

int main() {
    constexpr int events = 5000;

    Histogram parse_hist;
    Histogram state_hist;
    Histogram decide_hist;
    Histogram end_to_end_hist;

    int state = 0;
    int actions = 0;

    for (int seq = 1; seq <= events; ++seq) {
        auto t0 = Clock::now();
        int parsed = fake_parse(seq);

        auto t1 = Clock::now();
        state = fake_update_state(state, parsed);

        auto t2 = Clock::now();
        if (fake_decide(state)) ++actions;

        auto t3 = Clock::now();

        parse_hist.record(std::chrono::duration_cast<ns>(t1 - t0).count());
        state_hist.record(std::chrono::duration_cast<ns>(t2 - t1).count());
        decide_hist.record(std::chrono::duration_cast<ns>(t3 - t2).count());
        end_to_end_hist.record(std::chrono::duration_cast<ns>(t3 - t0).count());
    }

    std::cout << "events processed: " << events << "\n";
    std::cout << "actions emitted:   " << actions << "\n";

    parse_hist.print("parse stage");
    state_hist.print("state stage");
    decide_hist.print("decision stage");
    end_to_end_hist.print("end-to-end");
}
```

You add instrumentation to a hot path and see latency get much worse. Which explanation is most plausible?

What to carry forward

Latency instrumentation is not about collecting every possible number. It is about collecting the numbers that help you make safe engineering decisions.

Use this checklist in the trading lab:

• Name the path: parse, state update, signal, action, network boundary.
• Timestamp boundaries: measure stage durations, not vague blocks.
• Use a monotonic clock: prefer std::chrono::steady_clock for intervals.
• Avoid hot-path logging: record compact counters or samples first.
• Keep histograms: watch distribution shape, not just averages.
• Check tails: p99 and p99.9 often reveal the real pain.
• Compare changes fairly: same machine, similar build flags, warmed-up code, similar workload.

This lesson gives you the visibility layer for the rest of the infrastructure section. When the next lesson introduces thread handoffs, you will already have the habit of asking: what did the queue or boundary do to the latency distribution?

One Pager Cheat Sheet

• Instrumentation turns “it feels fast” into stage-by-stage latency evidence.
• Place timestamps at meaningful stage boundaries so latency can be attributed to a specific part of the path.
• Tail percentiles reveal latency spikes that averages can hide.
• Use std::chrono::steady_clock for elapsed intervals, and remember that measurement itself has cost.
• steady_clock is the standard C++ clock choice for interval timing.
• A latency histogram preserves distribution shape instead of hiding spikes inside one average.
• Avoid hot-path printing; record cheaply first and report later.
• Percentiles such as p99 and p99.9 describe tail behavior that averages often hide.
• A stage profiler records per-boundary durations and reports histograms after the measured loop.
• Instrumentation can perturb the system, so measure with low overhead and interpret results carefully.
• Good latency work names boundaries, records cheap samples, summarizes distributions, and compares changes fairly.