Retrieval-augmented generation gets pitched as simple: embed your documents, store the vectors, retrieve the closest matches at query time, hand them to the model. As a weekend prototype, that's roughly true. As something that has to work correctly for a real user asking a real question six months from now, it's the easy 10% of the problem.
The hard 90% is retrieval quality, and it shows up in ways that never appear in a demo. A support document gets updated, but the old version is still in the index and occasionally wins the similarity search. A question spans two documents and the top-k retrieval only surfaces one of them, so the answer is confidently half-right. A chunk boundary splits a table in half, and the number the user needed is on the other side of the cut. None of these show up when you test with the five questions you wrote while building the thing.
What actually moves the needle is unglamorous: a chunking strategy that respects document structure instead of splitting on a fixed character count; a re-indexing process that runs when source documents change, not once at launch; retrieval evaluation against a real set of questions — including ones the system should refuse to answer — before anything ships; and citations back to source documents, so a wrong answer is traceable instead of a mystery.
Access control is the other piece that's easy to skip and expensive to bolt on later. If different users should see different documents, that constraint has to live in the retrieval layer, not just the UI — otherwise a cleverly-phrased question can surface content a user was never supposed to see. This is a common gap in RAG systems built quickly, and it's the kind of thing that's much cheaper to design in from the start than to retrofit after a security review flags it.
None of this means RAG is not worth building — it's one of the more reliably useful applications of LLMs precisely because it grounds the model in something checkable. It means the value is in the retrieval engineering, not the API call, and that's where the actual project time should go.