Retrieval over internal docs that contradict each other
The HR wiki has three versions of the leave encashment policy. Two say unused leaves expire in March. The third, dated eight months later, says they carry forward. A support bot built on this corpus will answer correctly two times out of three and contradict itself the third.
This is the actual problem with retrieval over real corporate documents. The embeddings work. The chunking works. The search returns relevant passages. But the corpus itself is inconsistent, duplicated, and partly out of date, and the model has no way to know which document wins.
We ran into this building an internal Q&A tool for a logistics company. Their operations docs lived in Confluence, SharePoint, and a shared drive. The same SOP existed in four places with slight variations. The shipping rate sheet was a spreadsheet that someone updated in October and someone else updated differently in November. Neither version was marked superseded.
Naive retrieval averages over these conflicts. The model gets two contradictory chunks in context and either picks one arbitrarily or produces a hedge like "policies may vary." For a user asking whether a specific shipment qualifies for a discount, that answer is useless.
The first fix is metadata, and it matters more than chunking strategy. Every document needs a source, a last-modified date, and an authority level. "Authority" can be as simple as a single field: `official`, `draft`, or `legacy`. The last one is for documents that are technically still published but have been replaced by something newer. You need someone to go through the corpus and tag things. There is no embedding model that fixes a corpus nobody has curated.
When retrieval returns multiple chunks that answer the same question differently, you have two options. The cheap one is to sort by authority and recency and truncate before the context window fills. The model sees the most recent official version and ignores the rest. This works when your authority metadata is reliable.
The more robust option is to detect the conflict and surface it. After retrieval, cluster the chunks by the question they appear to answer. If two chunks from documents with different authority levels address the same question, pass both to the model with their metadata and ask it to resolve. This costs an extra LLM call per conflicting cluster, but it produces answers like "As of November 2024, the rate is ₹45/kg. An older document lists ₹42/kg, but that was superseded." The user gets the answer and the provenance.
Provenance is the real deliverable. A user who asks the bot about leave policy and gets an answer needs to verify that answer against the actual document. Every response should include a link to the source document and the specific section. Without that, the bot is a black box that occasionally hallucinates, and nobody will trust it enough to use it.
Duplicate detection is the other piece. Near-duplicate documents inflate the context window and bias the model toward whichever version has more copies in the corpus. A simple cosine similarity pass over document embeddings before indexing catches exact and near-duplicates. We hash document content and keep only the most recent version when the hash matches above a threshold. The older versions stay in storage but are excluded from the retrieval index.
None of this is glamorous. The embedding and generation parts are straightforward. The work is in the corpus: tagging, deduplicating, and building the metadata that lets retrieval prefer the right version. A client asked us whether a fine-tuned model would help. It would not. The problem is not that the model does not understand the documents. The problem is that the documents disagree with each other, and the retrieval layer has to resolve that before generation.
Budget for the curation work. It is larger than the engineering work. The logistics company's docs took three weeks to tag and deduplicate, and someone from their operations team had to make the calls on which version was authoritative. The retrieval system we built on top of that is a few hundred lines of Python and a Postgres table. The curation is the product.