Five retrieval decisions we made twice
Field notes from the retrieval stack behind Rayify's decision briefs, and what it cost each time the obvious design lost to the audit requirement.
Published
Most of what we know about retrieval we learned by building it wrong first. This is a set of field notes from the stack behind Rayify's decision briefs: five places where the obvious design worked, shipped, and then had to be rebuilt once we understood what the product actually required of it. The thread running through all five, which we did not see until the fourth: retrieval for a brief that someone signs is a different engineering problem from retrieval for a chat answer. Not a harder version of it. A different one. ## 1. We started with semantic search alone The default build is a dense retriever. Embed every chunk, embed the query, rank by cosine similarity, done. It works well enough on a demo corpus that you can convince yourself it is finished. Diligence queries broke it in a specific way. They are full of exact strings: CIK numbers, ticker symbols, statute citations, defined terms lifted verbatim from a contract. An embedding smears those into a neighbourhood of similar-looking tokens, which is the opposite of what you want when the analyst typed the identifier precisely because it is exact. The obvious correction is to swap in BM25. That fails in the mirror image: ask "did revenue grow?" against a filing that says "net sales increased" and lexical matching returns nothing useful. So we run both and fuse the rankings. The fusion is Reciprocal Rank Fusion (Cormack, Clarke and Buettcher, SIGIR 2009), which scores a chunk as the sum over rankers of w / (k + rank), with k set to the standard 60. The reason to fuse ranks rather than scores is that BM25 relevance and cosine similarity are not on a comparable scale, and any attempt to normalise them into one becomes a per-corpus tuning exercise that has to be redone for every new customer's document set. Ranks need no such calibration. One property we did not design and would not have thought to ask for: a chunk that appears in only one of the two rankings scores roughly half of what a chunk in both scores. That is a novelty discount, and it fell out of the arithmetic for free. ## 2. We reranked with an LLM Fusion is fast and coarse. It ranks each chunk on signals computed from that chunk in isolation, so it never models how the query and the chunk interact. That is what a reranking pass is for: take the top 50 candidates and score the pairs jointly to produce the top 10 the reasoner actually sees. The obvious way to score pairs jointly, in 2026, is to ask a model. We built that. The one design decision we got right first time was batching: the naive shape is one call per pair, which for 50 candidates is 50 round trips and turns latency into the dominant cost. One prompt scoring every candidate is a single round trip. We cap the batch at 50 because position bias in long candidate lists is real and gets worse as the list grows, a failure mode documented well beyond our own stack (see [Lost in the Middle](https://arxiv.org/abs/2307.03172), Liu et al., 2023). Then a requirement from three layers downstream killed it as the default. A brief cites its evidence by index. Re-run the same query with the same inputs, and citation 4 has to still be citation 4, or a replayed brief no longer says what the original said. An LLM reranker cannot promise that. Sampling, model updates, and provider-side changes all move the ordering, and none of them announce themselves. The replacement is a deterministic scorer: query-term coverage, term density, phrase proximity, exact-phrase presence. The features a small cross-encoder would learn, computed with arithmetic instead of a model. It is bit-stable across runs, it needs no provider and no 278MB model download, and on our labelled fixture it beats passing the fusion order straight through. The LLM reranker still exists and can be wired in deliberately. It is no longer what you get by default. The lesson we would keep: "better ranking" and "reproducible ranking" are separate objectives, and a product that sells an auditable answer has to know which one it is buying. ## 3. We cached the results before we understood the key Retrieval takes about 50 milliseconds. Reranking takes 15 to 60 seconds. An operator refining a query, adjusting filters, and re-running is paying that second number over and over for a result that barely changes, so caching the post-rerank list is close to the whole win available. That part was easy. The bugs were all in the key. Organisation ID has to be in it. A cross-tenant cache hit is not a performance bug that shows up as a slightly wrong answer; it is one customer's evidence appearing inside another customer's brief. The key derivation is the one place you can guarantee that cannot happen, so it is where the guarantee belongs. The reranking model has to be in it too, and this one is quieter. Serve a 32-billion-parameter model's ranking to a request that asked for the 8-billion one and nothing errors. You simply get a comparison between two configurations that is silently a comparison of one configuration with itself. Every filter that changes what comes back belongs in the key for the same reason. The general form: anything that changes the ranking is part of the identity of the ranking. ## 4. We filtered by date in the wrong place, twice A brief written as of 15 January must not cite a filing that landed on the 20th. That sounds like a post-processing step, and we built it as one. Wrong place. The filter belongs in the SQL WHERE clause, bound by an index, before anything expensive happens. Filtering afterwards means paying for reranking on chunks that were never eligible, and it also ruins the operator's trace. "0 chunks matched as-of 2026-01-15" tells you what happened. "Retrieved 50, reranked 50, writer rejected the brief" makes you go and find out. The second mistake was subtler and took a connector to expose. There is no single date field to filter on. SEC EDGAR stamps filing_date, meaning the day the SEC received the document. FRED stamps observation_date, meaning the period the number describes, which is not when it was published and is often months earlier. Treat those two as the same field and you will cite a revision as though it had existed before it was published, which is exactly the class of error the as-of rule exists to prevent. So the anchor field is per connector, declared by the connector. When one does not declare a date, we fall back to the timestamp recording when we retrieved the chunk. That is deliberately conservative: we saw the chunk no earlier than the world did, so the fallback can exclude evidence that was legitimately in scope, but it cannot let a document from the future through. Losing recall is recoverable. Citing the future is not. ## 5. We validated citations too late Every claim in a brief points at a chunk by a stable handle, and every cited chunk has to carry a source URL, the connector it came from, and a retrieval timestamp. Without those three, the citation cannot be checked by anyone, which makes it decoration. We originally enforced that at the end, in the schema validation on the finished document. Technically correct, and useless in practice: by then a reasoning pass had already built an argument on evidence that was never citable, and the fix was to throw the argument away and start again. Now the check happens while the evidence pack is assembled. A chunk missing any of the three is dropped before the reasoner ever sees it. The reasoner reasons over a smaller set, and everything in that set can be cited. The same layer solved a problem we had not connected to it. Two reasoning passes given the same evidence formatted differently are not comparable, and any difference in their output cannot be attributed to anything in particular. One canonical rendering of an evidence pack removes that variable. And because evidence is a pack rather than a list, it can be hashed. We store the content hash of every chunk at ingest and compare it at citation time. When they diverge, something changed underneath a brief that has already been written: an issuer re-filed, or a source was tampered with. From inside the retrieval layer those two look identical, and both have to stop an unrevised brief from going out. ## What the reversals had in common None of these were performance problems, and only the first was a relevance problem. Every other reversal came from the same place: the output is a document that someone signs their name to. That single fact makes reproducibility a hard requirement rather than a nice property, makes the cache key a confidentiality boundary, makes a date field a correctness boundary, and makes an uncitable chunk worthless no matter how relevant it is. If we were starting again, we would design the evidence contract first and the retrieval second. Every one of these five reversals was the contract arriving late and pushing the implementation into a shape it should have had from the beginning.