Retrieval quality measurement
Arcus answers that with a number instead of an impression. It replays your questions against your retrieval interface, joins what came back against what a person judged relevant, and returns eleven standard measures with the worst questions sorted to the top. It measures. It writes no answers of its own.
Write to [email protected]. Replies go out within five working days.
The question
A generation step fails loudly and gets blamed. The retrieval step in front of it fails silently and does not.
A retrieval system takes a question, searches an index and hands back a ranked list of passages. Something downstream then reads that list. It might be a language model writing an answer, it might be a search results page, it might be a person scrolling. Whatever it is, it can only work with what the list contains.
When the output is wrong, the argument in the room is almost always about the model. Change the prompt. Try a larger model. Add a grounding instruction. Sometimes that is the right argument. Often the model was never given the passage it needed, answered from whatever it did receive, and the whole debate is being held about the wrong component.
Telling those cases apart is not a matter of judgement or of reading enough transcripts. It is a measurement, and it is a measurement that has been well defined in the information retrieval literature for sixty years. The awkward part is that very few teams running a retrieval pipeline in 2026 are taking it.
| What actually happened | What the user sees | Where the fault is | What a measurement would say |
|---|---|---|---|
| The right passage was retrieved and was in the context, and the answer was still wrong | A wrong answer | The generation step | Recall at k is high for this question. Stop blaming retrieval and go and look at the prompt |
| The right passage was never retrieved at all | A confident wrong answer, or a refusal | Retrieval | Recall at k is zero for this question, and no amount of prompting will fix it |
| The right passage was retrieved at rank eighteen, and the pipeline passes eight | The same wrong answer, and a harder one to believe | Ranking, or the cut off | Recall at twenty is high, recall at eight is zero. The document was found and then thrown away |
| No passage in the index answers the question | A refusal, or an invention | The corpus | No labelled document for this question is returned under any configuration. This is a content problem wearing a software costume |
Those four rows need four different people to fix them, and three of the four are invisible if the only thing being examined is the final answer. Separating them is the entire product.
The measures
Scroll the shelf sideways. None of these are ours, and that is the point of using them.
M01
recall@k
Of the documents a person marked relevant for this question, what share appeared anywhere in the top k. A pipeline cannot use what it never fetched, so this number is the ceiling on everything downstream of it.
The ceiling on the whole pipeline
M02
precision@k
Of the k documents returned, what share were actually relevant. Low precision fills a context window with material that has to be discarded, and in a generation pipeline the thing doing the discarding is a model that will never mention it.
What the wrong ones cost
M03
nDCG@k
Rank sensitive and grade sensitive. A relevant document at position one counts for more than the same document at position nine, discounted logarithmically, then divided by the best ordering that was available for that question.
Position and grade together
M04
MRR
One divided by the rank of the first relevant document, averaged across the query set. The right number to read when the reader stops at the first useful hit and everything below it is decoration.
First useful hit only
M05
success@k
The blunt one. What share of questions returned at least one relevant document at all. It earns its place because an average hides the fact that one question in twelve comes back with nothing usable.
How often nothing usable
M06
coverage
What share of the documents in the index are ever returned by any question in the set. A document that no question reaches is paying for storage and answering nobody, and no per query measure above can see it.
The corpus, not the query
M07
redundancy@k
What share of a result set is near duplicate passages from a single source, sitting in slots a second source could have used. High recall with high redundancy is a result set that looks healthy and says one thing.
Slots spent saying it twice
M08
split@k
How often the passage that actually answers the question was cut across two chunks, so that neither half scores well enough on its own to be retrieved. A chunking fault that presents as a retrieval fault.
A chunking fault in disguise
M09
rewrite-overlap
Where the pipeline rewrites a question before retrieval, how far the result set for the rewritten question overlaps the result set for the original. Rewriting is meant to help, and this is the number that says whether it did.
Did the rewrite wander
M10
stability
Whether the same question returns the same documents after the index is rebuilt, the embedding model is swapped, or the chunker is retuned. Change that nobody measured is change your users will find first.
What moved when you rebuilt
M11
kappa
Where two people judged the same question and document pair, how often they agreed once agreement by chance is taken out. Every number above is worth exactly as much as the labels underneath it.
Measuring the ruler itself
Eleven measures. The shelf scrolls sideways.
Recall and precision at a cut off, mean reciprocal rank and discounted cumulative gain are standard information retrieval measures. The experimental method they belong to goes back to the Cranfield studies of the 1960s and has been run annually at scale since the first Text Retrieval Conference in 1992. Normalised discounted cumulative gain in the form most people use was set out by Jarvelin and Kekalainen in 2002.
A company that claimed to have invented a new measure of retrieval quality would deserve a long sceptical look, and would need to publish the definition, the derivation and a reproduction before anyone believed it. We are making no such claim. The measures are public, old and well understood. What is missing on most teams is not a new measure. It is a labelled query set, a harness that runs it, and somebody looking at the result on a Tuesday. Arcus supplies the label store, the harness and the report. The Tuesday is yours.
Every number on that shelf needs relevance judgements underneath it, and relevance judgements are made by people reading documents. We have no trick for making that cheap, and any vendor who tells you they do should be asked to show the agreement rate between their labels and yours. That is what M11 exists for.
Boundary
The category is crowded with things that sound similar, so the line is drawn in plain words. An instrument that also serves the traffic it grades is marking its own homework.
Arcus measures retrieval, and that is the whole of it. It reads result lists and computes numbers over them. There is no point at which it composes a sentence for a user to read as an answer, and that omission is what keeps the instrument independent of the thing it is grading.
| Left alone | Why the confusion happens | What does that job |
|---|---|---|
| A chatbot or an assistant | Two letters in the company name | A model provider behind whatever interface your team already chose. Nothing here produces natural language for an end user. The output is a table of numbers and a list of the questions that did worst |
| A vector database or a search index | Measuring retrieval sounds close to storing the things retrieved | The index you already run, whichever it is. This is designed to read result lists made of document identifiers, ranks and scores, and it has nowhere to put a corpus even if it wanted one |
| A model provider | Everything in this field is assumed to be selling inference | Whoever you already pay for embeddings and generation. Nothing here is trained, nothing is hosted, and no inference is served |
| An observability or performance product | Both of them watch a system that is running | Your existing tracing, logging and uptime tooling. Latency, error rate and cost per query are real numbers that matter, and none of them is retrieval quality. Confusing the two is how a slow pipeline gets described as an inaccurate one |
| A document store | Evaluating documents sounds like holding documents | Wherever your documents live now. Arcus holds identifiers, ranks, scores and labels, and never the document text |
| A public leaderboard | Benchmarks are the visible half of this field | Public benchmarks, which are useful for comparing methods and nearly useless for deciding whether your pipeline works. The measures here are computed on your questions against your corpus, which is the only comparison that decides anything on your team |
The parts
Five that carry the measurement, and one deliberately left on the far side of the boundary.
Where the questions live, alongside the human judgements about which documents answer them.
Replays a query set against your retrieval endpoint and records what came back.
The per question and aggregate table that the eleven measures are written into.
Re-runs the same query set after a change and shows what moved.
The measure that points at the instrument rather than at your pipeline.
Grading the text a language model produced at the end of the pipeline.
The run
The whole sequence, start to finish, on your questions against your corpus.
Step seven is the one teams skip. Changing the chunker and the embedding model in the same week produces a number that moved and no way to say which change moved it.
| Recorded | Never recorded | Why the line sits there |
|---|---|---|
| Document identifiers and their ranks | Document text or passage contents | Every measure on the shelf is computed from identifiers and ranks. Document text adds nothing to the arithmetic and adds a great deal to what a breach would cost |
| Retrieval scores as returned by your system | Embedding vectors | A score is a number your system already produced. A vector can often be inverted back towards the text it came from |
| A stable identifier for each question | The question text, unless you choose to include it | Real user questions are frequently personal information. The join works on identifiers, so the text is optional by construction rather than by policy |
| Timing of the run and the configuration label | Credentials, connection strings or endpoint secrets | The harness runs where your credentials already are and has no reason to carry them anywhere |
That line is published rather than left to a support answer, because a data boundary you can read before you send anything is a constraint and one you learn afterwards is an apology. Where a row of it ever changes, this page carries the change and says which line moved.
Start here
No form anywhere on this site. An address instead, with somebody at the other end of it.
Tell us the shape of your corpus and the shape of your questions, and you get back a straight read on which of the eleven measures would tell you something you do not already know, and what the labelling would cost you before it told you.
If you already measure your retrieval, we want to know what you measure and what you found. If you do not, we want to know what stopped you, because the answer is usually the labelling cost and we would rather hear that from you than assume it. And if you think the premise of this page is wrong, that is the most useful message this mailbox can receive. It is worth more than agreement.
Write to [email protected]. Replies go out within five working days. Nothing about writing to us subscribes you to anything, because there is no list to be subscribed to.