How do you architect
a production RAG system?

Retrieval-Augmented Generation fits on a napkin: embed the documents, embed the query, find the nearest neighbors, stuff them into a prompt. That description is exactly why so many RAG systems fail in production — the napkin convinces people it's a five-line problem, so they write five lines, ship them, and spend the following months firefighting.

A RAG system is not one function. It is a chain of independent judgment calls: should I even search? Did I search well? Did I find enough? Should I try again, and how? Am I confident enough to answer at all? Collapse all of those into one "retrieve top-k and generate" function and you get a system that fails silently — every wrong answer looks exactly as plausible as a right one, because nothing in the code ever had to decide anything.

"Domain-agnostic" is not a feature bolted on at the end — it's a constraint that eliminates entire categories of design before you've drawn a single box. Write these down before sketching any graph, because every later decision in this guide traces back to one of them.

Whenever you catch yourself about to write "if domain == X" anywhere in the core pipeline, that's one of these four promises telling you the logic belongs in a swappable configuration object — not in the graph itself.

Try a query like "thanks, that's all I needed" against any retrieval system. Cosine similarity never returns "nothing" — it returns a ranking, always. The top-k documents will look at least a little relevant, and the system will dutifully generate an answer grounded in context that was never relevant to begin with. That failure looks exactly like a correct answer until a human checks the sources.

Once retrieval is genuinely needed, a second blind spot appears — this time between two retrieval methods, not before them.

One more blind spot remains: hybrid retrieval is built for recall — casting a wide enough net that the right document is somewhere in the top-k. It was never built for precision — knowing which of those candidates matters most for this exact query. A bi-encoder scores query and document independently and compares vectors afterward, which is fast but structurally blind to fine-grained interaction between the two texts.

You now have reranked candidates. The next judgment call is the one most systems get wrong quietly: is this enough evidence to answer, or not?

That threshold was tuned by staring at one domain's score distribution. Move to a different domain — different embedding density, different document length, different query style — and 0.75 means something else entirely, or nothing at all.

Here is the subtlest trap in the whole design — worth feeling fully rather than skipping to the fix.

Follow that instinct to its conclusion and you hit a wall: reformulating a query after a failed retrieval takes one query in, produces one improved query out. Decomposing "compare the refund policy in the US and the EU" takes one query in, produces two independent sub-questions, each needing its own retrieval and its own evidence, recombined afterward. One node, two incompatible output shapes — and the downstream node now has to silently branch on which shape it received. That branch is hidden coupling: invisible on any diagram, discoverable only by reading the code.

The other half of this stage is the loop itself. "If insufficient, reformulate and try again" — with no limit — is an infinite loop waiting to happen: an LLM judge that's even slightly miscalibrated on an edge case can decide "insufficient" forever, for a query your corpus was never going to answer.

Here's the shape of the full pipeline so far — one reusable loop, run once for a plain query.

A naive multi-hop implementation concatenates sub-questions into one bigger retrieval query. That silently merges two independent evidence needs into one pass, and whichever sub-question dominates the combined query's semantics crowds out the other's results.

This is the one place where per-branch state is correct, not a violation of "no nested sub-state" — genuinely independent parallel work deserves genuine isolation, as long as the merge back into shared state happens through one explicit, named step.

Two more disciplines keep everything above from rotting once the graph gets big. Every LLM-calling node owns exactly one prompt template and reads only the state fields it needs — the grader never needs the raw original query if all it needs is the active query and the candidates; the reformulator never needs the final answer field, because it doesn't exist yet at that point in the run. A shared "master prompt" with conditionals is the same coupling mistake as the merged query-understanding node, one layer down.

Latency and evaluation numbers are not core logic and must never look like it. A thin wrapper around every node records timing before returning control to the graph — no node ever branches on a timer. Recall@k, MRR, and nDCG@k live entirely outside the graph, in an offline harness run against a fixed query set with known-correct answers. If a metric is ever read inside a node to decide what happens next, evaluation logic has leaked into the architecture.