The demo trap
Retrieval-augmented generation demos beautifully. Ten documents, a handful of friendly questions, impressed stakeholders. Then the corpus grows to forty thousand documents, users ask questions the way users actually ask questions, and accuracy quietly collapses — except nobody notices, because nobody is measuring.
The uncomfortable rule we've learned across a dozen RAG deployments: if you can't state your answer accuracy as a number on a golden dataset, your accuracy is whatever your angriest user says it is.
Build the eval harness first
Before tuning anything, we build a golden set: 500–1,500 real questions with verified answers, drawn from support logs, sales calls, and subject-matter experts. Every retrieval change, chunking experiment, and prompt revision runs against it automatically. This converts RAG tuning from vibes to engineering — you can see that switching to hybrid retrieval bought 6 points and a reranker bought 4 more.
The harness also gates deploys. If a change drops golden-set accuracy, CI fails. This one practice separates systems that improve over time from systems that mysteriously got worse since March.
Where the points actually come from
In our experience the accuracy budget breaks down consistently: retrieval quality dominates (chunking strategy, hybrid search, reranking), grounding discipline comes second (citation requirements, refusal thresholds when retrieval confidence is low), and prompt engineering is a distant third. Teams over-invest in prompts because they're easy to edit, and under-invest in retrieval because it's real work. The leaderboard doesn't lie: fix retrieval first.
Working on something in this space?
We help teams turn articles like this into shipped systems. One business day to an honest answer.
Request a Quote