RAG in Production: Chunking, Retrieval and Evaluation Choices That Matter

Retrieval-augmented generation is easy to demonstrate and hard to keep honest. A weekend prototype over fifty documents will impress a room. The same design over eight thousand documents starts returning confident answers assembled from the wrong page, and because the answers still read well, nobody notices for a month. The difference between the two is almost entirely in three choices: whether retrieval was the right approach at all, how the documents were split, and whether anyone built a way to tell if it is working.

Retrieval or fine-tuning: the question people get backwards

The two are usually presented as alternatives. They solve different problems, and the split is clean enough to decide in one meeting.

Retrieval changes what the model knows. Fine-tuning changes how the model behaves. If your problem is that the model does not have your pricing policy, your product catalogue or last quarter's board pack, that is a knowledge problem and retrieval solves it. If your problem is that the model answers in the wrong format, adopts the wrong tone, or ignores your classification taxonomy no matter how carefully you describe it, that is a behaviour problem and fine-tuning is the tool.

For most businesses the answer is retrieval, and it is retrieval for a practical reason as much as a technical one: your knowledge changes. A fine-tuned model carries a snapshot of what it was trained on, so every policy update means another training run and another evaluation cycle. A retrieval system updates when you update the document. Fine-tuning is worth its overhead when you have a stable task, a few thousand real examples of the output you want, and a format requirement that prompting has genuinely failed to hold.

Chunking is a content problem, not a parameter

Almost every tutorial splits documents into fixed windows of a few hundred tokens with some overlap. It is a reasonable default and it is wrong for most business content, because business content has structure and fixed windows destroy it. A five-hundred-token window through a policy document will regularly cut a clause away from the condition that qualifies it, and the retrieved fragment then reads as an unconditional rule.

Split on the document's own boundaries instead. Headings, clauses, table rows, question and answer pairs. Then attach the context that the fragment lost when you cut it out.

• Prepend the heading path to every chunk, so a fragment carries "Refund policy > International orders > Exceptions" with it into the index.
• Keep tables whole where you can, and store a plain sentence describing each row alongside it. Embeddings handle prose far better than pipe-separated columns.
• Store effective dates and document versions as metadata, then filter on them at query time. Superseded policies retrieving alongside current ones is a common and expensive failure.
• Index the questions people actually ask against each chunk where you have them, from your support inbox or ticket history. Query text resembles a question; source text resembles a policy. Closing that gap improves retrieval more than most model changes.

Scanned documents deserve a separate note. If a meaningful share of the corpus is images of text, the quality ceiling is set by the extraction step and no amount of retrieval tuning moves it. Measure extraction accuracy on a sample first, because that number caps everything downstream.

Retrieval: hybrid by default

Vector search matches meaning, which is why it beats keyword search on paraphrased questions. It also reliably fails on exact identifiers: part numbers, invoice references, section numbers, surnames. Those are precisely the things people search for in a business context.

Run both. Keyword search and vector search over the same corpus, combine the result lists, and rerank the top candidates with a cross-encoder. That last step is usually the largest single quality gain available, because a reranker reads the query and the passage together rather than comparing two vectors that were computed independently.

Retrieve more candidates than you pass to the model, then cut. Fetching thirty and passing the best five after reranking beats fetching five directly, and it costs very little. Passing everything you retrieved is worse than passing a few good passages: long contexts dilute attention and increase the chance the model quotes the least relevant thing you gave it.

The evaluation set is the actual deliverable

Build a set of questions with known correct answers before you launch. Between one and two hundred is enough to be useful. Draw them from real queries where you have them, and write the rest with the people who currently answer these questions by hand. Include the awkward ones: questions whose answer changed last year, questions that span two documents, and questions the corpus genuinely cannot answer.

Measure two things separately, because they fail separately. Retrieval quality asks whether the correct passage appeared in the results at all. Answer quality asks whether the model used it correctly. Conflating them wastes weeks: teams tune prompts for a fortnight when the passage was never retrieved in the first place.

The unanswerable questions matter more than any other category. A system that says it does not know is usable. A system that invents a plausible clause is worse than no system, because it moves the review burden back onto the reader and destroys the trust that made the project worth funding. Track refusal accuracy as a first-class metric and treat a drop in it as a release blocker.

What to instrument on day one

• Every query, the passages retrieved, and the answer returned. Without this you cannot debug a single complaint.
• A thumbs-down control that captures the query and the retrieved context, not just the sentiment.
• Latency at the ninety-fifth percentile, split between retrieval and generation, so you know which half to fix.
• Cost per query, tracked from the first week. It is small and easy to ignore until volume arrives.
• The share of answers where the model declined to answer. A sudden fall usually means an indexing change, not an improvement.

Citations belong in the interface from the start, linking to the source document and the specific section. They are the cheapest trust mechanism available, and they convert a complaint from "the system is wrong" into "this document is out of date", which is something you can actually fix.

If you are weighing retrieval against fine-tuning for a specific use case, or you have a prototype that works on demonstration questions and not on real ones, our AI automation and LLM application service page covers how we scope retrieval systems, including the evaluation work that has to happen before anything goes in front of staff or customers.