The moment a RAG answer cites the right doc but the wrong section, the user stops trusting every other citation in the response. You found the PDF. You found the page. But the highlighted passage points three paragraphs off, and now the whole answer reads like a guess. I've debugged this failure mode across enough pipelines to know it's rarely a single bug. It's a stack problem: chunking boundaries that split sections mid-thought, embeddings that rank by semantic similarity instead of structural location, top-k retrieval that returns the right document but the wrong chunks inside it. GigaRAG, built for agent memory and RAG pipelines, treats section-level precision as a first-class problem, not an afterthought. This guide gives you a diagnostic framework for isolating the fault, plus concrete fixes for chunking, metadata, and reranking. It will not give you a one-line config change that solves everything. Some fixes require architectural changes. Here's what to check first.
| At a glance | Details |
|---|---|
| Core issue | Citations point to wrong passage within correct document |
| Main causes | Chunking granularity, metadata loss, reranking errors, generation drift |
| Diagnostic approach | Isolate fault in chunking, embedding, retrieval, or generation |
| Key fix | Align chunk boundaries with semantic sections and propagate metadata |
| Limitations | Semantic ambiguity and poor source structure need upstream fixes |
| Audience | RAG pipeline builders debugging citation accuracy |
In This Guide
- What Does 'Right Doc, Wrong Section' Actually Mean?
- Chunking Strategies: Fixed-Size vs Semantic-Aware
- The Four Root Causes of Wrong-Section Citations
- Answers Cite The Right Doc But The Wrong Section: A Step-by-Step Guide
- Chunking Strategies That Cause Wrong-Section Citations
- Metadata and Section Hierarchy: The Missing Layer
- Reranking and Retrieval Precision: Fixing the Top-k Problem
- Common Mistakes When Answers Cite the Right Doc but the Wrong Section
- What You Cannot Fix with Simple Config Changes
- A Diagnostic Checklist for Wrong-Section Citations
What Does 'Right Doc, Wrong Section' Actually Mean?
"Right doc, wrong section" means the retriever found the correct document, but the passage it pulled doesn't contain the answer. The citation points to a real source, yet the quoted span sits three paragraphs or two pages away from the actual evidence.
Document-level vs. section-level retrieval failure
Document-level failure means the retriever never found the right file. Section-level failure is sneakier: the document is right, but the chunk is wrong. Your system looks competent while being wrong.
Why this failure mode is uniquely damaging to user trust
Users check citations to verify answers. When the document matches but the section doesn't, the citation looks legitimate at a glance. The user trusts it, then discovers the mismatch. That's worse than an obvious miss.
A quick example: correct PDF, wrong paragraph
A user asks about cancellation terms. The retriever returns the right contract PDF, but cites the billing section. The answer says 30 days. The actual cancellation clause says 7. The document is right. The answer is wrong.
[!note] Wrong-section citations often stem from chunking that splits logical sections, causing the retriever to fetch a chunk that contains the answer but lacks the correct section label.
Chunking Strategies: Fixed-Size vs Semantic-Aware
| Factor | Fixed-Size Chunking | Semantic-Aware Chunking |
|---|---|---|
| Granularity | Uniform token/character count | Variable, based on content structure |
| Metadata propagation | Often loses section context | Preserves headings and section labels |
| Citation accuracy | Higher risk of wrong-section citations | Lower risk, citations align with logical sections |
| Implementation complexity | Low, easy to implement | Higher, requires parsing and segmentation |
| Best for | Homogeneous, well-structured text | Complex documents with clear sections |
The Four Root Causes of Wrong-Section Citations
Wrong-section citations rarely come from one bug. They come from four places in the pipeline, and each can fail alone or in combination. Here's the map.
Chunking: too large, too small, or badly split
Chunks that span multiple sections force the LLM to cite a span that mixes topics. Chunks that split a section mid-thought leave the answer stranded across two chunks. Either way, the citation lands wrong.
Embedding: semantic similarity misses section boundaries
Embeddings capture meaning, not structure. Two sections that discuss similar topics in different contexts will score close together. The retriever can't tell them apart because the vector doesn't encode where the section starts or ends.
Retrieval: top-k returns the right doc but wrong chunks
Top-k similarity pulls the k nearest chunks to the query. If the right document has many chunks, the wrong ones can crowd out the right one. The document is correct. The chunks are not.
Generation: the LLM cites the right doc but paraphrases the wrong span
The LLM receives the right chunk but paraphrases from memory or blends multiple chunks. The citation points to the right document, but the quoted text doesn't match the generated answer. This is a generation-side failure, not a retrieval one.
[!tip] For RAG pipeline builders: implement a post-retrieval validation step that cross-checks the cited section against the document's table of contents or heading hierarchy to catch mismatches early.
Answers Cite The Right Doc But The Wrong Section: A Step-by-Step Guide
- Reproduce the error with a known query and document, logging the retrieved chunk and its metadata.
- Check chunk boundaries: verify if the cited section is split across chunks or merged with adjacent content.
- Inspect metadata: confirm that section headings and page numbers are correctly attached to each chunk.
- Test retrieval: compare the embedding of the query with the embeddings of the correct and incorrect chunks to see if similarity scores are misleading.
- Evaluate reranking: disable reranking to see if it shifts the citation to the wrong section.
- Examine generation: prompt the model to output the exact source text and compare with the retrieved chunk to detect hallucinated citations.
- Apply targeted fixes: adjust chunking strategy, enrich metadata, refine reranking, or constrain generation to retrieved text.
Chunking Strategies That Cause Wrong-Section Citations
Chunking decides what the retriever can return. Get the boundaries wrong and the citation lands wrong, even when the document is right.
Fixed-size chunks: why they split sections mid-thought
Fixed-size chunking cuts text at a character or token count, ignoring structure. A 500-token chunk that starts mid-paragraph and ends three sentences into the next section mixes two topics. The embedding averages both, so the chunk matches queries for either section but cites neither cleanly. The LLM then points at a span that contains the answer only partially.
Semantic chunking: respecting section boundaries
Semantic chunking splits text where meaning shifts, usually at headings or paragraph breaks. It keeps a section's argument intact. The tradeoff: chunk sizes vary widely, and a long section may still exceed your token budget. You'll need a fallback split for oversized sections.
Sentence-window and parent-document retrieval: decoupling retrieval from citation
Sentence-window retrieval embeds small units, often single sentences, but returns the surrounding window. Parent-document retrieval embeds small chunks but returns the full parent section. Both fix the core problem: retrieval precision and citation context don't have to use the same granularity.
Chunk overlap: when it helps and when it muddies citations
Overlap keeps context across boundaries, which helps when an answer spans two chunks. But too much overlap means the same sentence appears in multiple chunks. The retriever returns near-duplicates, top-k fills with redundant spans, and the right chunk gets crowded out. Keep overlap small, 10-15% of chunk size, and test citation precision before raising it.
Metadata and Section Hierarchy: The Missing Layer
Chunk boundaries alone won't fix wrong-section citations. The retriever also needs to know where each chunk sits in the document. That's metadata's job.
Tagging chunks with section headers and heading paths
Attach the full heading path to every chunk at ingestion time. A chunk from a "Pricing" subsection under "Enterprise Plans" gets tagged with both. When the LLM cites that chunk, it can cite the section, not just a raw text span. The citation reads "Enterprise Plans > Pricing" instead of a floating paragraph.
Metadata filtering: narrowing retrieval to the right section
Store section headers as filterable fields in your vector database. When a query asks about pricing, filter candidates to chunks tagged with pricing-related headers before similarity search runs. This shrinks the candidate pool and keeps top-k from filling with chunks from the wrong section of the right document.
Propagating document hierarchy through ingestion
Parse the document tree first, then chunk. Don't chunk first and try to reconstruct structure later. Most document loaders can extract headings. Use them. If your source is a PDF without clean headings, you'll need a layout parser, and even then the structure may be unreliable.
Reranking and Retrieval Precision: Fixing the Top-k Problem
Top-k similarity search returns the chunks closest to your query embedding. That's the problem. Closest doesn't mean correct section.
Why top-k similarity fails at section granularity
Embeddings capture semantic meaning, not document position. Two chunks from different sections can score nearly identical similarity if they discuss related topics. The retriever returns both. The generator picks the first one that sounds plausible. Wrong section, right document.
Cross-encoder reranking: precision over recall
A cross-encoder reads the query and chunk together, side by side, and scores their actual relevance. It's slower than embedding similarity but far more precise. Run a bi-encoder to pull 50 candidates, then a cross-encoder to rank them. The top 5 after reranking are much more likely to sit in the right section.
Hybrid search: combining BM25 and embeddings for section-level accuracy
BM25 matches exact terms. Embeddings match meaning. Use both. A query about "cancellation policy" will match chunks containing those exact words via BM25, even if the embedding drifts toward a semantically similar but wrong section. Combine scores with reciprocal rank fusion.
Citation-aware scoring: rewarding chunks that match the query's section intent
Score chunks higher when their metadata headers align with the query's implied section. If the query asks about pricing, boost chunks tagged with pricing headers. This is metadata filtering plus reranking working together. The latency cost is real, but for citation accuracy it's usually worth it.
Common Mistakes When Answers Cite the Right Doc but the Wrong Section
Most wrong-section citations come from four habits that are easy to fix once you see them.
Mistake 1: Chunking without section awareness
Fixed-size chunks split sections mid-thought. A chunk starting at token 800 may begin halfway through a heading. Fix: chunk at heading boundaries, not arbitrary token counts.
Mistake 2: Storing chunks without structural metadata
If your chunks don't carry their section headers, the retriever can't tell where they sit. Fix: tag every chunk with its heading path at ingestion time.
Mistake 3: Relying on top-k similarity alone
Top-k returns the closest embeddings, not the right sections. Fix: add a cross-encoder reranking step before generation.
Mistake 4: Not evaluating citation accuracy separately from answer quality
Your answer may read well while citing the wrong span. Fix: measure citation precision as its own metric, not folded into answer faithfulness.
What You Cannot Fix with Simple Config Changes
Some wrong-section problems are structural, not configurational. No chunk size tweak or reranking threshold will fix them. Here's what stays hard.
Semantic ambiguity: when two sections say similar things
If your source doc has a "Pricing" section and a "Billing" section that both discuss invoices, the retriever can't reliably pick between them. Embeddings see similar text. Rerankers see similar text. The model guesses. You can reduce this with metadata, but you can't eliminate it when the source itself blurs the boundary.
Poorly structured source documents
A PDF with no headings, no table of contents, and walls of undifferentiated text gives you nothing to chunk against. You can't tag chunks with section headers that don't exist. The fix is upstream: restructure the source or accept lower citation precision.
LLM paraphrasing drift during generation
The retriever returns the right chunk. The LLM paraphrases it, then cites a nearby span that sounds related but isn't the source of the claim. This is a generation-side failure. Prompting helps. It doesn't fully solve it.
A Diagnostic Checklist for Wrong-Section Citations
When a citation lands in the wrong section, don't guess. Run the failure through these four steps in order. Each step isolates one layer of the pipeline.
Step 1: Isolate the failure—retrieval or generation?
Check what the retriever actually returned, not what the LLM cited. If the right chunk was in the top-k results but the citation points elsewhere, it's a generation problem. If the right chunk never made it into context, it's retrieval. You can't fix what you haven't located.
Step 2: Inspect chunk boundaries and metadata
Open the retrieved chunk. Does it start mid-sentence? Does it carry a section header? A chunk that spans two sections will cite whichever boundary the model notices first.
Step 3: Evaluate retrieval precision at section granularity
Run your eval set and score precision at the section level, not the document level. A system can hit 90% document precision and 40% section precision. That gap is your problem.
Step 4: Test with reranking enabled
Toggle reranking on and re-run the same queries. If section precision jumps, top-k similarity was the bottleneck. If it doesn't move, the issue is upstream in chunking or metadata.
Answers cite the right doc but the wrong section when one of these four layers fails. Isolate the layer, apply the fix, and measure section-level precision before you ship.
Frequently Asked Questions
What are the three elements required in an in-text citation?
In academic writing, in-text citations typically require the author's last name, the year of publication, and the page number (if applicable). For RAG systems, the equivalent is the document ID, the section or page reference, and the specific passage text.
How to list sources correctly?
List sources in a consistent format, such as a reference list with full bibliographic details. In RAG, ensure each chunk's metadata includes the document title, section heading, and page number so citations can be generated accurately.
What two things do you not need to cite?
Common knowledge (facts widely known and accepted) and your own original ideas or findings do not require citation. In RAG, you don't need to cite the retrieval process itself, only the source content used in the answer.
Why should you use parenthetical documentation to cite sources within your paper?
Parenthetical documentation allows readers to quickly locate the source without disrupting the flow of reading. In RAG, inline citations serve a similar purpose, enabling users to verify the answer against the exact source passage.
How can I prevent the model from hallucinating citations?
Constrain the generation to only use retrieved chunks and include explicit instructions to cite the exact chunk ID or section. Additionally, implement a post-generation check that verifies each citation against the retrieved metadata.
What chunking strategy reduces wrong-section citations?
Semantic-aware chunking that respects document structure (headings, paragraphs) and attaches section metadata to each chunk reduces wrong-section citations by ensuring chunks align with logical sections.
About GigaRAG
GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through answers cite the right doc but the wrong section or something adjacent, we publish what we have actually tested, including where it falls short.
