MODULE 09 OF 12 · RELEASE 3 · OUTLINE
Observability & Agent Operations
Being able to answer "what happened?" about something that happened three weeks ago.
This module is an outline. The plain-language explanation and the lab specification below are final. The full written lesson and its runnable lab repository are still being prepared, and enrolment is not open.
An aircraft has a black box. Nobody hopes to use it. But when something goes wrong, the difference between knowing exactly what happened and guessing forever is that someone decided, in advance, to record the right things.
So what is this really about?
When a customer says "your system gave me a wrong answer last month", you either can reconstruct it or you cannot. That is decided long before the complaint, by whether each request left a trail: what came in, what was looked up, what the AI said, what it did.
QUICK CHECK
What makes a three-week-old request reconstructable?
The words you will hear
- Trace
- The full story of one request, start to finish, in order.
- Span
- One step inside that story — a single lookup or AI call, with how long it took.
- Correlation id
- A reference number stamped on everything belonging to the same request.
- Runbook
- Written instructions for a bad day, so nobody has to think it through at 3am.
By the end you will be able to
- Follow one request across every step it touched.
- Know what each request cost and where the time went.
- Be alerted about things that matter, not things that are merely busy.
How it is put together
One trace per request, with a span for every model and tool call carrying tokens, latency, and outcome. Traces carry a correlation id that appears in every log line. Cost is derived from recorded tokens rather than estimated. Alerts fire on error rate, budget burn and latency percentiles, never on raw request count.
Where you start
A system with unstructured logs and no correlation between them.
The lab
- Add a correlation id at ingress and thread it through every log line and span.
- Emit a span per model and tool call with tokens, latency and outcome.
- Derive cost from recorded tokens and attribute it per feature.
- Add alerts on error rate, latency percentile and budget burn.
- Write a runbook for the three most likely failures.
What goes wrong, on purpose
The scenario. A customer reports a wrong answer from three weeks ago and gives only an approximate time.
What should happen. The trace is locatable from the available detail and shows the inputs, retrieved context, tool calls and model outputs that produced the answer.
How you prove it works
- A test asserting every log line emitted during a request carries the correlation id.
- A test asserting recorded token counts match the provider response.
- A drill: reconstruct a specific past request using only stored telemetry.
What you walk away with
A traced request path, per-feature cost attribution, alerts on meaningful conditions, and a runbook that has been rehearsed.