RAG systems, properly

What RAG is, and when not to use it

A language model knows what was in its training data, frozen at some point in the past, and nothing else. Retrieval-augmented generation fixes that by fetching relevant text at question time and putting it in the model's context, so the answer is grounded in documents you control rather than in whatever the model absorbed during pretraining.

That is the entire idea. Everything else on this page is engineering.

The appeal is real: you get current information without retraining, you can cite sources, you can revoke a document and have it stop influencing answers immediately, and you can serve tenant-specific knowledge from one model. Those properties are hard to get any other way.

I am labouring this because the most expensive RAG mistakes I have seen were not bad chunking strategies. They were teams building retrieval for a problem that was not a retrieval problem, and then spending two quarters tuning a system that could not have worked.

The anatomy of a RAG system

Two pipelines that people routinely conflate. Ingestion runs offline, on a schedule, over your documents. Retrieval and generation runs online, per request, in front of a user. They fail differently, they scale differently, and they are owned by different parts of your latency budget.

Write that down as a rule: a RAG system has two quality numbers, not one. Retrieval either put the right material in front of the model or it did not. Generation either used that material faithfully or it did not. A single end-to-end score averages the two and tells you nothing actionable.

Ingestion and chunking

Chunking is the decision that quietly caps your ceiling, because retrieval can only return chunks and a chunk that does not contain a complete thought cannot produce a complete answer. Yet it is usually the least deliberate part of the pipeline — a default of 512 tokens with 50 overlap, chosen once and never revisited.

The tension is simple. Small chunks give precise retrieval and poor context: you find the right sentence but the model cannot see the qualifier in the paragraph before it. Large chunks give rich context and diluted embeddings: a 2,000-token chunk covering four topics has an embedding that sits somewhere between all four and is strongly similar to none of them.

If you take one thing from this section: parent–child retrieval resolves the central tension rather than trading against it. You embed a 200-token chunk so the vector is sharp and specific, and you hand the model the 1,200-token section it came from so it can see the qualifiers. The cost is one extra lookup and a little more context. It is nearly always worth it.

Metadata is not optional

Every chunk should carry, at minimum: a stable source document id, the document title, the section path, a position index, a content hash, and a timestamp. Plus whatever your access control needs.

Each of those earns its place. The document id and section path let you cite precisely and let a user verify. The position index lets you fetch neighbouring chunks when an answer straddles a boundary. The content hash makes indexing idempotent, so re-running your pipeline on unchanged documents is a genuine no-op rather than a full re-embed. The timestamp lets you prefer recent material and detect staleness. And access control has to live on the chunk, because it has to be enforced during the search — see production concerns.

Contextual retrieval

A chunk pulled out of its document loses the information the document made implicit. A chunk that says "the limit was raised to 50" is useless in isolation — which limit, whose, when? The fix is to prepend a short, generated description situating the chunk in its document before you embed it: "From the 2025 Q3 enterprise pricing policy, section on rate limits: the limit was raised to 50…"

This costs one cheap model call per chunk at ingestion time — a real cost on a large corpus, which prompt caching over the document reduces substantially — and it consistently improves retrieval on corpora where chunks are heavily context-dependent. It does very little on corpora of self-contained documents. Measure on your own data before committing; the gain is corpus-dependent, not universal.

Embeddings

An embedding model maps text to a vector such that semantically similar text lands nearby. That is the whole contract, and it has two consequences people underrate.

First, "similar" means similar according to that model's training objective, which is usually general-purpose semantic similarity. It is not your notion of relevance. A model trained on web text will happily rate two paragraphs about unrelated products as highly similar because they share marketing register. On specialised corpora — legal, medical, internal jargon — general-purpose embeddings can be notably worse than they appear on public benchmarks.

Second, the embedding space is fixed at index time. Changing your embedding model means re-embedding your entire corpus. On a large index that is expensive and slow, so treat model selection as a decision with switching costs, not a parameter to fiddle with.

Choosing a model

The honest procedure is short: take a public leaderboard as a shortlist rather than an answer, then evaluate the top few on your data with your queries. Leaderboard rankings are computed on public benchmark suites; your corpus is not in them, and the ordering frequently does not transfer. Building the golden set you need for this is covered in evaluation below, and it is the single highest-value thing you can do before picking a model.

Things that actually matter in the decision:

Dimensionality. Higher dimensions generally carry more information and cost more in memory, index size and search time, roughly linearly. Many modern models support Matryoshka representation learning, where the vector is trained so that its leading prefix is itself a usable embedding — you can truncate 1536 dimensions to 512 and keep most of the retrieval quality at a third of the storage. If your model supports it, measure the truncation curve on your data; the savings are often large and nearly free.

Maximum sequence length. If your chunks exceed it, the model silently truncates and you index a vector representing only the first part of the chunk. This is a common, invisible bug: retrieval quality degrades and nothing in your pipeline reports an error. Assert that chunk token counts fit the model, in code, at ingestion.

Asymmetric search. Many models expect a prefix distinguishing a query from a document, because a short question and a long passage are differently shaped. Getting this wrong — or omitting it — measurably degrades retrieval, and it is easy to miss because the system still returns plausible-looking results. Read the model card.

Cost and latency at ingestion and at query time. These are different budgets. Ingestion is batch and can be slow; query embedding sits directly in your user-facing latency.

Quantisation

Full-precision float32 vectors are usually unnecessary. Scalar quantisation to int8 cuts memory roughly fourfold with a small recall cost; binary quantisation cuts it by roughly 32× with a larger one. The standard production pattern is to search a quantised index for speed and then rescore the top candidates with full-precision vectors, which recovers most of the lost recall at a fraction of the memory.

Whether you need this is a scale question. At 100,000 vectors, do not bother. At 100 million, memory is the dominant cost of the system and this decides your infrastructure bill.

Vector indexes and the recall triangle

Once you have vectors you need to find the nearest ones to a query vector, fast. Exact search compares the query against every vector — perfect recall, linear cost. That is fine up to a few hundred thousand vectors and untenable beyond it, so production systems use approximate nearest neighbour indexes that trade a little accuracy for a lot of speed.

The trade-off is genuinely three-way, and you cannot optimise all three:

HNSW is the workhorse and worth understanding properly, because its parameters are the ones you will actually tune. It builds a layered proximity graph: sparse long-range links at the top for coarse navigation, dense short-range links at the bottom for precision. A search enters at the top, greedily walks toward the query, drops a layer, and repeats.

Three parameters matter:

M — connections per node. Higher means better recall and a larger graph. Memory scales with it directly. Typical production values sit in the 16–64 range; past that you are usually buying very little recall for a lot of memory.

efConstruction — how hard the builder searches when inserting each node. Higher gives a better graph and a slower build. It is a one-time cost, so err high; a build that takes twice as long and permanently improves recall is a good trade.

efSearch — how hard each query searches. This is the runtime recall/latency dial and the only one you can change without rebuilding. Tune it against a measured recall target rather than guessing.

Filtered search is harder than it looks

"Find similar chunks, but only from documents this user can see" is the single most common production requirement and the one that quietly breaks ANN indexes.

The naive approach — search, then filter the results — is wrong. If a user can see 1% of the corpus, your top-100 nearest neighbours may contain zero documents they are allowed to see, and you return nothing while relevant material sits at rank 400. This is called post-filtering, and it fails silently: the system returns an empty or thin result set and the model says it does not know.

Filtering before the search is correct but expensive, because it destroys the graph structure HNSW relies on — the neighbours it wants to walk through may be excluded, and the traversal degenerates.

What production systems actually do:

Partition by the filter. If access control is coarse — per tenant, per workspace — give each partition its own index. The filter becomes routing, and search is unfiltered within the partition. This is by far the most robust answer when it fits, and it should be your first choice.

Filtered traversal. Modern index implementations evaluate the predicate during graph traversal and over-search to compensate. Effective, with a latency cost that grows as the filter gets more selective.

Over-fetch and filter. Retrieve 10× more than you need and filter after. Acceptable for weak filters, dangerous for selective ones — and it degrades gradually rather than failing loudly, which is worse.

Hybrid retrieval

Dense vector search is good at meaning and bad at exact strings. Ask for "the ACME-4471 tolerance spec" and an embedding model will cheerfully return documents about similar-sounding part numbers, because in embedding space they are similar. Ask for "how do I cancel my subscription" and lexical search will miss the document titled "Ending your plan" because they share no words.

Each approach fails exactly where the other works, which is why hybrid retrieval is not a refinement — for most real corpora it is the correct baseline.

BM25 is the lexical half worth using. It scores documents by term overlap, weighting rare terms more heavily and normalising for document length. It is decades old, has no training step, costs almost nothing, and is very hard to beat on queries containing identifiers, error codes, names, acronyms and exact quotes — which, if you look at real query logs, is a large fraction of them.

Fusing the two result lists

You now have two ranked lists with scores on incomparable scales. Cosine similarity lives in roughly [-1, 1]; BM25 is unbounded and corpus-dependent. Normalising and adding them is fragile, because the normalisation depends on the score distribution of each particular query.

Reciprocal rank fusion sidesteps this by throwing the scores away and using only the ranks:

# Reciprocal Rank Fusion. k dampens the influence of top ranks;
# 60 is the conventional default and is rarely worth tuning.
def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

A document ranked first by one retriever and absent from the other still scores well. A document ranked moderately by both scores better still. There is nothing to calibrate and nothing to drift, which is why it has become the default.

The trade-off is real though: RRF discards score magnitude, so it cannot distinguish "rank 1 with an overwhelming score" from "rank 1 by a hair." Where your scores are genuinely calibrated and comparable, a weighted score combination can beat it. Measure rather than assume; RRF is the right default, not the right answer everywhere.

Reranking

Retrieval is optimised for speed over a huge corpus, which forces a compromise: the query and the document are embedded independently. The document's vector was computed long before your query existed, so the model never got to consider them together. That is what makes the index possible, and it is also what caps its precision.

A cross-encoder removes that compromise. It takes the query and the document together as one input and outputs a relevance score, so it can attend across both. It is far more accurate and far too slow to run over a corpus — which is exactly why it belongs in a second stage, over the 50–100 candidates retrieval already narrowed to.

The pattern is: retrieve 50–100 cheaply, rerank them accurately, keep the top 5–10. Reranking typically produces a larger quality improvement than any amount of embedding-model tuning, and it is usually the highest-return single change available to a mediocre RAG system.

Using a general-purpose LLM as the reranker — asking it to score or order the candidates — is flexible and lets you express relevance criteria in words, which a trained cross-encoder cannot do. It is also slower, more expensive, and less consistent. It is a reasonable choice when relevance is genuinely bespoke; it is an expensive choice when it is not.

And reranking is skippable. If retrieval recall@10 is already very high on your evaluation set, the reranker has nothing left to fix and you are buying latency for nothing. This is the payoff for measuring the two halves separately: you can tell.

Query transformation

The user's question is frequently a poor search query. It may be too short to carry signal, phrased in vocabulary the corpus does not use, contain several questions at once, or depend on the previous turn of a conversation. Transforming it before retrieval is often the cheapest available quality win.

Conversational rewriting. "What about the enterprise tier?" is unretrievable on its own. Rewriting it against the conversation history into "What are the rate limits on the enterprise tier?" makes it retrievable. If you are building a chat interface over RAG and doing nothing else on this list, do this one — it fixes a large class of "the bot suddenly got stupid" reports.

Multi-query expansion. Generate three or four paraphrases, retrieve for each, fuse with RRF. Improves recall on ambiguous questions at the cost of several searches. Cheap if your retrieval is fast.

Decomposition. "How does our refund policy differ from our exchange policy?" needs two retrievals, not one. Split it, retrieve separately, assemble both into context. Many compound questions fail purely because a single query vector cannot represent two topics at once.

HyDE. Have the model write a hypothetical answer to the question, then embed that and search with it. The intuition is that a fake answer sits closer in embedding space to real answers than the question does, because questions and answers are differently shaped text. It helps most where the question and the source material share little vocabulary, and it costs an extra generation on the critical path. It also fails in a specific way worth knowing: if the model hallucinates a confident, wrong hypothetical answer, you retrieve confidently wrong documents.

Routing. Not every question needs retrieval. "Hello" does not. "Summarise what we just discussed" does not. A cheap classifier in front of the pipeline that decides whether to retrieve at all — and if so, against which index — saves latency and money and removes a whole class of irrelevant-context failures.

Context assembly

You have your top chunks. How you arrange them in the prompt materially changes the answer, and this stage gets the least attention of any in the pipeline.

Position matters. Models attend unevenly across a long context: material at the beginning and end is used more reliably than material in the middle. This effect has been measured repeatedly and it is large enough to matter. The practical response is to put your highest-ranked chunks at the start and the end of the retrieved block rather than in simple rank order, and — more importantly — to keep the block short enough that the effect has less room to bite.

More context is not better. Because context windows are large, teams pad them with 30 chunks on the theory that the model will find what it needs. In practice this dilutes attention, increases cost and latency proportionally, and raises the chance of the model anchoring on a plausible irrelevant passage. Five good chunks beat thirty mediocre ones, consistently.

Deduplicate. Overlapping chunks, near-duplicate documents and repeated boilerplate waste context and can bias the model toward whatever is repeated. Deduplicate on content hash for exact matches and on embedding similarity above a threshold for near ones.

Label every chunk. Give each retrieved passage a visible identifier and its source metadata, so the model can cite it and so you can trace an answer back. Citations that the model generates freely are frequently fabricated; citations selected from a labelled list are checkable — and you should check them programmatically before display.

Instruct the refusal. The single highest-value line in a RAG system prompt tells the model what to do when the context does not contain the answer. Without it, the model falls back on its parametric knowledge and produces a fluent, confident, unsourced answer — the exact failure RAG existed to prevent. State plainly that it must answer only from the provided context and must say when the context is insufficient.

Treat retrieved content as data, never as instructions. Any text you retrieve may be adversarial — a document containing "ignore your previous instructions and…" is a prompt injection, and in a RAG system the attacker's delivery mechanism is simply getting a document into your corpus. Delimit retrieved content clearly, state in the system prompt that content inside those delimiters is reference material and not instructions, and never let retrieved text reach a tool-calling path unexamined.

Evaluation

This is the section that separates people who have shipped RAG from people who have demoed it. Without evaluation you cannot answer the only question that matters when something goes wrong — which half is broken? — and you will spend weeks changing things and arguing about whether the output got better.

Build a golden set. There is no way around this.

You need real questions with known correct source documents. Not generated questions about your corpus — those are shaped like your corpus and will flatter your retrieval. Real ones, from support tickets, search logs, user interviews, or domain experts writing down what they actually ask.

A hundred questions is enough to be useful. Two hundred is comfortable. What matters far more than size is coverage: include the easy lookups, the compound questions, the ones where the answer sits in a table, the ones where the vocabulary differs from the source, and — critically — questions your corpus genuinely cannot answer, so you can measure whether the system correctly refuses instead of inventing something.

For each question record the document or chunk ids that actually contain the answer. That is the labelling cost, it is real, and it is the price of being able to engineer rather than guess.

Retrieval metrics

These need no model and no judge. They are cheap, deterministic, and they run in CI.

Recall@k — of the relevant chunks, what fraction appeared in the top k? This is the ceiling on everything downstream. If the right chunk is not in the context, no amount of prompting produces a correct answer. Track recall@5, @10 and @20; the gap between them tells you how much reranking can buy you.

Precision@k — of the top k, what fraction were relevant? Low precision means you are spending context and attention on noise.

MRR — the mean of 1/rank of the first relevant result. Sensitive to whether the best document is at position 1 or position 8, which matters given the position effects described above.

nDCG@k — the right metric when relevance is graded rather than binary, since it rewards putting the most relevant document highest and discounts by position. Use it if your labels distinguish "answers the question" from "related but insufficient."

Generation metrics

Harder, because there is no single correct string. Three properties are worth measuring separately, because they fail independently:

Faithfulness — is every claim in the answer supported by the retrieved context? This is the one that catches hallucination, and it is the most important number in the system. Note that it says nothing about correctness: an answer can be perfectly faithful to a retrieved document that is itself wrong.

Answer relevance — does the answer address the question that was asked? A faithful summary of the wrong passage scores well on faithfulness and is useless.

Correctness — does it match the known ground truth? Only measurable where you have reference answers, which is why the golden set pays for itself twice.

Programmatic checks before model-based ones

Before reaching for a judge model, write the assertions a computer can check. They are free, deterministic, and they catch a surprising share of real defects: does the answer cite at least one source; does every cited id exist in the retrieved set; does the answer stay under the length limit; does it contain the required disclaimer; does it avoid PII patterns; is it valid against the output schema.

Every check you can express as code is a check you never have to pay a model to perform or argue about.

LLM-as-judge, and its failure modes

For the genuinely subjective properties, a model scoring outputs is the practical option. It is also a measurement instrument, and an uncalibrated instrument produces confident numbers that mean nothing. The documented biases are worth stating plainly:

Position bias. When comparing two answers, judges favour one position over the other. Mitigate by evaluating both orderings and averaging.

Verbosity bias. Judges prefer longer answers, largely independent of quality. Control for length explicitly, or instruct the judge to disregard it and verify that instruction worked.

Self-preference. Judges tend to score text produced by the same model family more highly. Using a different model to judge than to generate is cheap insurance.

Scale compression. Asked for a 1–10 score, judges cluster in 6–8. Binary or three-point rubrics with explicit criteria are far more reliable than fine-grained scales.

Run it in CI, and watch it online

An evaluation suite that runs when someone remembers is not a safety net. Wire retrieval metrics and programmatic checks into CI so a prompt or chunking change that regresses recall fails the build. Reserve the slower, costlier judge-based evaluation for release candidates.

Offline evaluation still will not catch everything, because your golden set is a fixed sample of a shifting reality. Online, capture thumbs up/down, whether users clicked the citations, whether they rephrased and asked again — a rephrase immediately after an answer is one of the strongest available signals that the answer was bad. Feed the failures back into the golden set. That loop is what makes the system improve over time instead of drifting.

The failure-mode taxonomy

When a RAG system gives a bad answer, the useful question is never "why is the model wrong." It is "which stage failed." This table is the diagnostic procedure I would hand a new engineer on their first day.

Work it top to bottom. The stages are ordered so that an earlier failure makes every later measurement meaningless — there is no point debugging generation while retrieval is broken.

The last row is the one that ends careers rather than sprints. Make it a test that runs on every commit.

Production concerns

Incremental indexing

Reindexing everything on every run is affordable at prototype scale and impossible later. The property you want is idempotence: running the pipeline over unchanged documents does nothing, costs nothing, and changes nothing.

Content hashing gets you there. Hash each chunk's text; if the hash matches what is indexed, skip it entirely — no embedding call, no upsert. Use a deterministic chunk id derived from the document id and position so that updates replace rather than duplicate. Then handle the case everyone forgets: when a document shrinks from twelve chunks to eight, chunks nine through twelve are still in the index, still retrievable, and now wrong. Deleting orphans has to be part of the pipeline, not an afterthought.

Deletion is a requirement, not a feature

When a document is deleted or a user exercises a right to erasure, it has to disappear from the index, from every cache, and from any derived artefact — including the query cache that might still return its text. Build deletion in from the start; retrofitting it into a system with three layers of caching is genuinely difficult, and the deadline is usually legal rather than negotiable.

Multi-tenancy and access control

Covered in filtered search, but the principle bears repeating because the failure is catastrophic and silent: enforce access control during retrieval, never after it. A system that retrieves globally and filters in the application layer has a data leak the moment any code path forgets the filter — and it will return correct-looking results throughout, so nothing alerts you.

Separate indexes per tenant is the most robust answer where tenant count allows. It makes cross-tenant leakage structurally impossible rather than a matter of getting every query right.

Caching

Three layers, each with a different invalidation story. Query embeddings cache on the query string and effectively never expire. Retrieval results cache on the query plus the filter context and must be invalidated when the index changes. Generated answers cache on query plus retrieved chunk ids — which conveniently self-invalidates, since different retrieval produces a different key.

Prompt caching at the model layer is a separate and often larger win: a long static system prompt re-sent on every request is billed every time unless it is cached, and on a chat product that is a substantial fraction of the bill.

A cost model you can actually compute

Work it out before you scale, not after. The arithmetic is simple and the conclusions are usually surprising.

# Per query, assuming hybrid retrieval + reranking + generation.
query_embedding   = 1 call, ~20 tokens          # negligible
vector_search     = infra cost, not per-token       # amortised
rerank            = 50 candidates × cross-encoder pass
generation_input  = system prompt + 5 chunks × 400 tok = ~2,400 tok
generation_output = ~300 tok

# The two costs that actually dominate, in order:
#   1. generation input tokens  — grows linearly with chunks retrieved
#   2. reranking                — grows linearly with candidates
# Halving chunks from 10 to 5 roughly halves input cost. Measure
# whether it costs you any recall before assuming it does.

Then divide by the number of successful outcomes rather than requests. Cost per answered question is the number that governs the business; cost per token is the number that governs nothing.

I am deliberately not putting provider prices here. They change, and a page with stale prices is worse than a page with none. The formula outlives the numbers — plug in today's rates when you need a figure.

Freshness

Decide the staleness budget explicitly and instrument it. "Documents are searchable within five minutes of publication" is a commitment you can monitor and alert on. "We reindex nightly, probably" is not. Emit the age of the oldest un-indexed document as a metric; it is the one number that tells you whether ingestion is quietly falling behind.

Advanced patterns

Reach for these once the basics are measured and solid. Each solves a specific problem and adds real complexity; adopting them before you have an evaluation harness means you will not be able to tell whether they helped.

Late interaction (ColBERT-style). Instead of one vector per chunk, keep a vector per token and score by summing each query token's best match against the document. This retains far more detail than a single pooled vector and performs notably better on queries where specific terms matter. The cost is storage — an order of magnitude more vectors — and a more complex index. Worth it when single-vector retrieval is your measured bottleneck, and only then.

GraphRAG. Extract entities and relationships from the corpus into a graph, then traverse it to answer questions that require connecting facts across documents. It genuinely answers a class of question flat retrieval cannot — "what connects these two people" — at the cost of an expensive extraction pipeline and a graph that must be maintained. Justified when your questions are relational; overkill when they are lookups.

Agentic RAG. Let the model decide what to retrieve, examine what came back, and retrieve again — a loop rather than a single pass. Powerful for research-style questions where the right query is not known upfront. It also multiplies latency and cost by the number of iterations and introduces the error-compounding problem covered in agent architectures. Cap the iterations, and measure whether the second retrieval ever actually helps; on many corpora it does not.

Self-correction. Grade retrieved documents for relevance before generating, and re-query if they are poor. A cheap, contained version of the agentic idea that avoids most of its downside, and one of the better returns on this list.

Fine-tuned embeddings. Train the embedding model on your own query–document pairs. Reliably the largest retrieval improvement available on a specialised corpus, and the most operationally expensive: you need labelled pairs, a training pipeline, and a plan for re-embedding the corpus every time you retrain. Consider it when you have exhausted hybrid retrieval and reranking and have the data to do it properly.

What to build first

In order. Do not skip ahead; each step's value depends on the one before it being in place.

1. The golden set. Fifty real questions with known answer locations. Before any code. It is the instrument that makes everything after it measurable, and building it later means every earlier decision was made blind.

2. The stupidest thing that works. Recursive chunking at around 500 tokens, an off-the-shelf embedding model, a flat index, top-5, a straightforward prompt with an explicit refusal instruction. Measure recall@5 and faithfulness. This is your baseline, and on a small corpus it is sometimes also your answer.

3. Hybrid retrieval. Add BM25, fuse with RRF. Usually the largest single improvement, and it costs almost nothing.

4. Reranking. Retrieve 50, rerank, keep 5. Usually the second largest. Measure the marginal return so you know what you are paying for.

5. Parent–child chunking. Embed small, return large. Fixes the precision–context tension rather than trading it.

6. Query transformation. Conversational rewriting first if you have a chat interface, since it fixes an entire class of reported failures. Others one at a time, each with a measurement.

7. Everything in advanced. Only once you can prove the basics are not your bottleneck.