Citations are a test, not a feature
Most RAG demos show a chatbot answering from a PDF. The hard part is not the answer; it is whether the page number under the answer is real. A citation that points to page 7 when the retrieved passage came from page 4 is worse than no citation: it teaches the user to trust the wrong thing.
I built a small demo to make that failure visible, and an eval to keep it honest.
The setup
rag-citations-demo ingests a PDF, chunks it, retrieves with BM25 + embeddings, and returns an answer where every sentence carries a page number. Before the answer leaves the server, a guard checks each cited page against the set of pages the retriever actually returned. If a page was never retrieved, the citation is dropped and the sentence is marked as uncited.
The eval is deliberately small: one 5-page document, 16 chunks, 10 questions with known answer pages. For each question it records:
page_hit_at_3: did the correct page appear in the top 3 retrieved chunks?cited_page_hit: did the final answer cite the correct page?- the retrieved and cited page lists, so a failure can be inspected instead of guessed.
Current result: 9/10 on both retrieval and citation. The one miss is a question whose answer spans two pages; the retriever found one of them and the guard kept the citation, but the answer needed both. That is the kind of failure a demo hides and an eval exposes.
Why the guard matters
LLMs are good at producing plausible page numbers. They are not good at knowing which pages were retrieved. A post-generation guard is cheap and deterministic:
- collect the page ids from the retrieved chunks;
- parse cited pages from the answer;
- drop citations not in the retrieved set;
- optionally re-ask once with the invalid citations removed.
This does not make the answer correct. It makes the citation honest, which is the minimum for a document assistant a reviewer can use.
What I measure
p50 /api/ask: 4.25 ms on the demo corpus, local, no external API keys.- Citation page hit: 9/10.
- Guard rejects: counted per run; a rising count means the model is drifting, not that the guard is broken.
The demo is intentionally small so the eval can run in CI in seconds. The same structure scales: swap the corpus, keep the metrics, add questions when a real miss appears.
What this means for a client
If you are building a document assistant, ask for three things before looking at the UI:
- a retrieval metric (page hit@k),
- a citation metric (does the cited page match the retrieved page),
- a failure log you can read.
Without those, "it cites sources" is a demo, not a system.