Retrieval is a ranking problem, not a vector problemRetrieval is a ranking problem, not a vector problem

The standard retrieval pipeline is four lines of code: embed the documents, embed the query, take the nearest neighbours, paste them into a prompt. It works impressively well on a demo corpus and disappoints reliably on a real one.

The diagnosis is almost never the embedding model. By the time a team is considering swapping it, the actual problems are usually upstream in how the corpus was split and downstream in how results were ordered and filtered.

Chunking is a modelling decision

Splitting documents every 800 characters is a decision to discard structure. A policy document has clauses. A contract has defined terms that give meaning to text pages away. An invoice has a table whose header is the only thing that explains its columns.

Fixed-size chunking cuts through all of that at arbitrary points, then asks an embedding model to recover meaning from a fragment whose context has been removed. The fix is not a larger chunk size; it is splitting along the structure the document already has, and carrying a small amount of inherited context with each piece.

• Split on real boundaries — headings, clauses, table rows — before falling back to length.
• Prepend the inherited heading path to each chunk so an isolated fragment still says what it belongs to.
• Keep a pointer from every chunk back to its exact source location, so an answer can cite a region rather than a document.
• Store the parent unit as well, and retrieve the child while passing the parent when the model needs the surrounding argument.

Nearest neighbours are not the best results

Vector similarity finds text that is semantically adjacent to the query. That is not the same as text that answers the query, and the gap between those two things is where most retrieval quality lives.

A useful pipeline retrieves generously and then ranks deliberately. Over-fetch with a hybrid of vector and lexical search, because exact terms — error codes, part numbers, defined terms — are precisely where embeddings are weakest. Then re-rank the candidates against the query with a cross-encoder, and only then truncate to what the prompt can hold.

1. 01Retrieve 50–100 candidates using both vector and keyword search.
2. 02Merge and de-duplicate, keeping the best score per source region.
3. 03Re-rank the merged set with a cross-encoder against the full query.
4. 04Apply hard filters — permissions, recency, document status — as filters, not as score adjustments.
5. 05Truncate to a context budget, keeping citation pointers intact.

Step four deserves emphasis. Permissions and document status are correctness constraints, not relevance signals. Expressing “this user may not read this document” as a score penalty means that with enough similarity it will eventually surface. Filters belong in the query, before ranking.

The missing feature is returning nothing

Nearest-neighbour search always returns its k nearest results, including when the corpus contains no answer. The pipeline then hands the model irrelevant context and a question, which is the exact set-up for a confident fabrication.

Retrieval needs an abstention threshold, set empirically against a set of questions you know the corpus cannot answer. “I do not have anything on that” is a correct answer, and it is worth scoring as a pass in evaluation rather than treating every empty response as a miss.

Measure retrieval separately from generation

The most common evaluation mistake is scoring only the final answer. When the answer is wrong, that tells you nothing about whether retrieval failed to find the right passage or generation failed to use it. Those have entirely different fixes.

Build a set of questions with the passages that should be retrieved for each, and score recall at k on retrieval alone. Then score generation separately, given correct context. Two numbers are far more actionable than one, and they stop a team from tuning prompts to compensate for a chunking problem.