The RAG demo takes an afternoon. The RAG system a compliance officer trusts takes engineering, and the difference is almost never the model.
Retrieval is the product
When a RAG system answers wrongly, autopsy the retrieval first. In our audits, four out of five failures happen before the model ever sees the context — a distribution that matches what the wider engineering literature reports, where the ingestion and chunking layer rather than the language model accounts for the large majority of production failures.
The specific failures repeat:
- Chunks split mid-thought, so the sentence carrying the condition lands in one chunk and the rule it qualifies lands in another. The model answers from half a policy.
- Tables shredded into noise. A rate card or a limits matrix flattened into prose loses the row-column relationship that was the entire meaning.
- Stale documents outranking current ones — the 2023 policy beating the 2026 revision because nobody indexed effective dates, and semantic similarity cannot tell which is in force.
- Queries the embedding space cannot separate. Two products with near-identical descriptions and different terms retrieve each other's clauses.
None of these is fixed by a better model. Several get worse with one, because a stronger model writes a more convincing answer from the same wrong context.
Chunking is a data-modelling decision
Treating chunking as a parameter — pick a token count, add an overlap — is the most common unforced error. Chunk boundaries should follow the document's own structure: the clause, the section, the table. A policy document, a contract and a support article do not want the same strategy, and running one splitter across all three guarantees that at least two are wrong.
Metadata carries as much weight as the text. Effective dates, document type, owning department, version and jurisdiction turn retrieval from a similarity search into a filtered one — and filtering is what makes recall predictable.
Permissions are not optional
Enterprise knowledge has boundaries: HR files, deal rooms, board papers, customer records under the DPDP Act. Retrieval must respect the same access rules as the source systems, evaluated at query time against the requesting user, not hoped for at indexing time.
The failure mode here is quiet and serious. A permissive index does not throw an error; it answers helpfully, using a document the person asking was never entitled to see. This is the requirement that separates enterprise RAG architectures from tutorials, and the one most likely to stop a deployment at security review.
Build the evaluation set before the system
Collect a hundred real questions with verified answers during discovery — real ones, from the people who will use the system, not questions written by the team building it.
That set becomes the speedometer. Measure three things on every change:
- Retrieval hit-rate — did the right passage come back at all? Measured separately from the answer, because a pipeline evaluated only on final output hides where it broke.
- Answer accuracy against the verified answer.
- Citation fidelity — does the cited source actually support the sentence it is attached to? A confident answer citing an unrelated document is worse than a refusal, because it survives review.
Teams without an evaluation set are tuning blind. Every prompt change feels like an improvement, and there is no way to tell a fix from a regression.
Evaluate the stages, not just the output
RAG is a pipeline: ingestion, chunking, retrieval, reranking, generation. A failure at any stage cascades, and a single end-to-end accuracy number tells you that something is wrong without telling you which stage to open. Instrument each boundary.
Freshness is a pipeline, not a batch job
Knowledge changes daily. Quarterly re-indexing guarantees confident answers drawn from expired truth, which is the failure that damages trust fastest — because the answer is well-formed, well-cited, and wrong in a way only a domain expert notices.
Production RAG treats ingestion as data engineering: event-driven updates when a source document changes, effective-date awareness so superseded versions lose rather than compete, and deletion that actually removes the vectors rather than orphaning them.
The sequence that works
- Collect the evaluation set from real users, before building.
- Model the documents — structure, metadata, access rules — before choosing a store.
- Build retrieval and measure hit-rate on its own, with no generation in the loop.
- Add generation, then measure citation fidelity.
- Only then tune the model, which by this point is the least interesting variable.
Reverse that order and the system will demo beautifully and fail its first month in production, for reasons the team will initially attribute to the model.