MODULE 11 OF 12 · RELEASE 3 · OUTLINE
Production Reliability
Staying useful when something you depend on is slow, broken, or quietly wrong.
This module is an outline. The plain-language explanation and the lab specification below are final. The full written lesson and its runnable lab repository are still being prepared, and enrolment is not open.
A good shop has a plan for the card machine going down. Not a brilliant plan — cash only, apologies, a sign on the door. The disaster is not the card machine; it is having no plan and turning everyone away in confusion.
So what is this really about?
Something you depend on will fail. The worst kind of failure is not "down" — it is "slow", because everything piles up behind it while it pretends to be fine. Decide in advance what your system does when that happens, and rehearse it.
QUICK CHECK
Your AI provider starts taking 45 seconds per call instead of failing outright. Which is worse for your users?
The words you will hear
- Timeout
- How long you will wait before giving up. Without one, you wait forever.
- Retry
- Trying again — but a limited number of times, and only for things that are safe to repeat.
- Circuit breaker
- After enough failures, stop calling for a while. Like not queuing at a till you can see is broken.
- Degraded mode
- The cash-only sign. Reduced service, honestly explained, rather than collapse.
By the end you will be able to
- Set limits on waiting, retrying and giving up.
- Decide what reduced service looks like, before you need it.
- Prove it by deliberately breaking things.
How it is put together
Every external call has a timeout shorter than the caller’s own budget. Retries are bounded, jittered, and applied only to idempotent operations. A circuit breaker stops calling a dependency that is clearly down. Each dependency has a declared degraded mode — a cached answer, a queued task, or an honest refusal.
Where you start
A system where every dependency is assumed available and no call has a timeout.
The lab
- Add timeouts derived from the caller’s budget, not from a guessed constant.
- Add bounded jittered retries, restricted to idempotent operations.
- Add circuit breakers with a defined half-open recovery path.
- Define and implement a degraded mode per dependency.
- Inject failure — latency, errors, partial responses — and observe the result.
What goes wrong, on purpose
The scenario. The model provider becomes slow rather than unavailable: every call takes 45 seconds and eventually succeeds.
What should happen. Timeouts fire before the caller’s budget expires, the breaker opens under sustained slowness, and the degraded mode serves users rather than a queue of hanging requests.
How you prove it works
- A fault-injection suite covering latency, errors and partial responses.
- A test asserting the breaker opens and later recovers through half-open.
- A test asserting the degraded path returns something useful and honest.
What you walk away with
A system that stays responsive during a provider outage, with a rehearsed degraded mode and fault-injection tests in CI.