MODULE 05 OF 12 · RELEASE 2 · OUTLINE
Production RAG & Knowledge Systems
Letting your AI look things up in your own documents — without handing someone the wrong file.
This module is an outline. The plain-language explanation and the lab specification below are final. The full written lesson and its runnable lab repository are still being prepared, and enrolment is not open.
A good librarian does two things at once. They find the book that answers your question, and they quietly do not hand you the files you are not allowed to read. Both matter. A librarian who only does the first is a problem.
So what is this really about?
You can let an AI answer from your own documents by looking up relevant passages first and letting it read only those. The hard part is not the looking up — it is making sure the lookup respects who is asking, and that you can actually measure whether it finds the right things.
QUICK CHECK
A document the user is not allowed to read is the single best match for their question. What happens?
The words you will hear
- Retrieval
- Finding the handful of passages worth reading, out of thousands.
- Chunk
- A document cut into passage-sized pieces, because whole documents are too big to read at once.
- Embedding
- Turning a passage into numbers so "similar meaning" becomes "close together".
- RAG
- The whole pattern: look it up, then answer using what you found.
By the end you will be able to
- Build a document pipeline you can re-run safely.
- Apply permissions during the lookup, not afterwards.
- Measure whether your lookup is actually finding the right things.
How it is put together
Ingestion normalises a document, splits it with a stated strategy, embeds each chunk and stores it with its source and permission metadata. Retrieval filters by permission first, then ranks — combining vector similarity with full-text search, because each fails on cases the other handles. Generation receives only chunks the current actor may read.
Where you start
A working vector search over one unfiltered collection, with no permission model and no evaluation.
The lab
- Make ingestion idempotent on a content hash so re-running changes nothing.
- Attach source and permission metadata to every chunk.
- Apply the permission filter in the query, before ranking.
- Add full-text search alongside vector similarity and combine the rankings.
- Build a small labelled evaluation set and measure retrieval against it.
What goes wrong, on purpose
The scenario. A document the current user may not read is the best semantic match for their question.
What should happen. It is excluded at query time, never reaches the model, and cannot appear in the answer or a citation.
How you prove it works
- A cross-user retrieval test asserting no restricted chunk is ever returned.
- A regression test on the labelled set, failing if relevance drops below a baseline.
- A test asserting re-ingesting the same document creates no duplicate chunks.
What you walk away with
A retrieval pipeline with a permission filter proven by test, a relevance baseline in CI, and idempotent ingestion.