LESSON 01 OF 05 · 12 MIN

How Production AI Systems Actually Work

The difference between something that works while you watch and something that keeps working when you do not.

A food truck and a hospital kitchen both cook. One serves forty people on a good day and closes when it rains. The other feeds hundreds every single day, including the day the delivery is late and two staff are off sick. Nobody thinks the hospital kitchen just has better chefs.

So what is this really about?

A demo and a real system are the same difference. A demo needs to work once, while you are watching. A real system needs to keep working for things nobody predicted, while parts of it are broken. Almost everything that separates the two sits outside the clever bit — it is in what happens when something goes wrong.

A request is checked, written down, and only then handed to the slower and less reliable steps.

QUICK CHECK

Which of these is most likely to cause a real incident?

Answer: option 2. Charging someone twice is a real problem with a real apology attached. A slightly worse answer usually is not. Most incidents come from the plumbing, not the intelligence.

The words you will hear

Production: Software real people actually depend on, as opposed to a demo you are showing off.
Validation: Checking that what arrived is what you expected, before you act on it.
Durable: Written down somewhere that survives a crash — not just held in memory.
Failure domain: One part that can break by itself without dragging everything else down with it.

What you will be able to do

Describe the path a request takes through a production agent system.
Name the boundaries where a system can fail independently of the model.
Explain why model quality is rarely the reason a deployment fails.

Going deeper

A demo has one job: produce a convincing answer once, while someone is watching. A production system has a harder one — produce an acceptable answer repeatedly, for inputs nobody anticipated, while dependencies are slow or broken, without doing damage when it is wrong.

Almost everything that separates the two lives outside the model.

The path a request actually takes

Follow one request end to end and the structure becomes obvious:

Arrival. Something triggers the system — a form, a webhook, a scheduled run. This is the first place an input can be malformed, hostile, or a duplicate of one you already handled.
Validation. The input is checked against a shape you decided in advance. Anything that fails is rejected here, cheaply, before it can reach anything expensive.
Durability. If the work matters, it is recorded before anything unreliable happens. This is the difference between losing a customer enquiry and merely failing to notify someone about it.
Model call. The part everyone thinks about. It can be slow, rate-limited, or return something that does not match the shape you asked for.
Tool execution. If the model can act — send an email, write a record — this is where a wrong answer becomes a wrong action.
Response and observation. Something returns to the caller, and something is recorded so that a person can later reconstruct what happened.

Each boundary fails on its own

The useful mental shift is that these are not steps in a happy path; they are independent failure domains. The model can be perfect while the system is broken: a duplicate webhook processed twice charges someone twice, regardless of how good the reasoning was.

This is why the rest of this course spends more time on interfaces, validation and boundaries than on prompting. Prompting improves the answer. Everything else decides whether the answer arrives, arrives once, and cannot cause harm when it is wrong.

What "good" looks like

A production system can answer four questions about any request: did it arrive, was it valid, what did it do, and can it be repeated safely. If your current design cannot answer all four, that gap is the next thing to build — not a better prompt.