Almost every retrieval system eventually ingests PDFs, and almost every one produces worse results on them than on any other source. The cause is not the extraction library. It is that a PDF does not contain a document in the sense the pipeline assumes — it contains instructions for painting marks on a page, and everything else is inference.
The root problem
A PDF has no paragraphs, no reading order, no tables and no headings. It has glyphs at coordinates. Every structural feature your chunker relies on has to be reconstructed by guessing from geometry — and where the guess is wrong, the chunk is wrong.
What is actually in the file
A page's content is a sequence of drawing operators. Text is positioned explicitly:
Nothing declares that those two strings belong to the same sentence, that the sentence is a paragraph, or that the paragraph follows a heading. A generator may emit text in any order that produces the correct visual result — and some emit individual characters with explicit positioning for precise kerning.
Extraction therefore means: collect every glyph with its coordinates, group them into words by proximity, group words into lines by baseline, group lines into blocks, then guess the order of blocks. Every one of those steps is a heuristic.
💡 This is not a flaw in PDF
PDF was designed to guarantee that a page looks identical everywhere, which it does superbly. Preserving semantic structure was not a goal — that is what tagged PDF and PDF/UA later added, and the overwhelming majority of PDFs in circulation are untagged.
You are asking a format built for appearance to yield meaning. It can, imperfectly, and knowing where the imperfection lies is the difference between a working pipeline and a mysterious one.
The five failure modes
1. Multi-column interleaving
The most visible failure. A two-column page extracted by naive top-to-bottom, left-to-right ordering produces text that alternates between columns line by line:
Every sentence is destroyed and every chunk is nonsense. Worse, it fails silently — the text looks superficially plausible, embeds without error, and quietly poisons retrieval.
2. Tables become number soup
There is no table object in an untagged PDF. What you see is text positioned in a grid, with the lines drawn as separate vector graphics that carry no relationship to the text at all.
For financial, scientific or operational documents this is the most damaging failure, because the numbers are usually the point.
3. Headers and footers in every chunk
Running headers, footers, page numbers and confidentiality notices are ordinary text on the page. Extraction picks them up on every page, so they appear in every chunk.
The cost is twofold: they consume context budget, and they dilute embeddings. Every chunk now shares a common preamble, which makes chunks look more similar to each other and reduces the discrimination your retrieval depends on.
4. Hyphenation and line breaks
The same applies to ordinary line wrapping: extraction inserts a newline at the end of every visual line, so chunkers splitting on \n\n see paragraph boundaries that are not there.
5. Ligatures
Typesetting replaces certain letter pairs with single glyphs — fi, fl, ff. If the font's character mapping is incomplete, these extract as single Unicode ligature characters rather than their component letters.
The result is text that looks correct and does not match: a search for "file" fails against file, because they are different character sequences. Normalising with NFKC decomposes them and fixes it.
Three kinds of PDF
Before choosing a strategy, find out what you are dealing with. They need completely different handling.
| Type | Contains | Extraction returns | Needs |
|---|---|---|---|
| Digital | Real text objects | Text, order uncertain | Layout-aware extraction |
| Scanned | Images only | Nothing | OCR |
| Scanned + OCR layer | Images plus invisible text | Text of variable accuracy | Quality check |
⚠️ An OCR layer is not necessarily a good one
A scan with an existing text layer extracts without error and may still be full of recognition mistakes — rn read as m, 0 as O, whole words wrong on a poor scan. Nothing flags it.
Spot-check the extracted text against the visual page before indexing thousands of documents. If accuracy is poor, re-running OCR at 300 DPI with deskewing usually beats whatever the original scanner produced.
An extraction pipeline that works
Steps worth expanding
Layout analysis is what separates working extraction from the naive kind. A layout-aware tool detects column boundaries from whitespace, orders blocks accordingly, and classifies regions as body, header, caption or table. This single step eliminates the interleaving problem.
Stripping furniture is mechanical once you look for it: text that appears at a consistent vertical position on most pages, with near-identical content, is furniture.
Note the replace(/\d+/g, '#') — normalising digits means "Page 4 of 27" and "Page 5 of 27" are recognised as the same header.
Normalisation repairs the text-level damage:
The unwrap rule keeps a break only where the line ended in sentence-final punctuation or was already followed by a blank line — so real paragraph boundaries survive and visual wrapping does not.
Handling tables properly
Tables need their own path. Extract them as structured data, then serialise deliberately.
| Approach | Works when |
|---|---|
| Ruling-line detection | The table has drawn borders |
| Whitespace alignment | Columns align consistently without borders |
| Vision model | Merged cells, nested headers, irregular layouts |
Once extracted, serialise as Markdown tables — models read them reliably and they are compact. And keep each table in a single chunk with its header, or repeat the header if it must split.
✅ Add a sentence describing each table
A table's own content often lacks the words a question would use. A quarterly revenue table may never contain the word "revenue" — just region names and numbers.
Generating a one-line description at index time and prepending it makes the table findable:
When to use a vision model
Rendering each page to an image and passing it to a vision-capable model handles cases geometric extraction cannot: complex multi-column layouts, tables with merged cells, forms, diagrams with embedded text, and handwriting.
| Geometric extraction | Vision model | |
|---|---|---|
| Cost | Near zero | ~1,000+ tokens per page |
| Speed | Milliseconds | Seconds |
| Complex layouts | Poor | Good |
| Tables | Variable | Good |
| Determinism | Exact | May paraphrase or hallucinate |
🚨 Vision extraction can invent text
A geometric extractor returns what is in the file, or fails. A vision model generates a transcription — and generation can produce plausible text that was not on the page. On a poor scan it may confidently supply a number it inferred rather than read.
For contracts, financial statements and anything where exactness matters, prefer deterministic extraction and use vision only for the pages where it fails. If you must use vision on critical documents, verify the numbers against a second extraction path.
The pragmatic pattern is tiered: geometric extraction first, quality-check the result, and fall back to vision only for pages that fail the check. Most documents never reach the expensive path.
Metadata to keep
Whatever the extraction path, carry this with every chunk:
| Field | Why |
|---|---|
source | Citation, and filtering by document |
page | Lets a user verify the answer in the original |
section | Heading path — improves both embedding and citation |
type | Prose, table, code — enables type-aware handling |
extraction_method | Lets you re-process one path when you improve it |
confidence | OCR score, for down-ranking poor pages |
Page numbers earn their place immediately: an answer citing "page 14 of the service agreement" is verifiable, and one citing nothing is not.
Preparing PDFs for processing?
Split, merge, rotate and convert PDF pages entirely in your browser — nothing is uploaded, which matters for contracts and internal documents.
Open the PDF Splitter →Summary
- A PDF has no reading order — extraction reconstructs it by guessing from coordinates.
- Multi-column layouts interleave silently, producing plausible nonsense.
- Tables have no structure in the file. Extract them separately as data.
- Headers and footers land in every chunk and dilute every embedding.
- Classify the PDF first — digital, scanned, or scanned with an OCR layer.
- Convert to Markdown, not raw text. Structure survives and tokens drop.
- Normalise with NFKC and repair hyphenation before chunking.
- Use vision extraction as a fallback, not a default — it can invent text.
Frequently Asked Questions
Why does text extracted from a PDF come out jumbled?
Because a PDF stores text as marks positioned at coordinates, not as a document with reading order. Extraction reconstructs sequence by guessing from position, which works on single-column prose and fails on multi-column layouts, sidebars and anything where the generator optimised placement over sequence.
Why do tables in my PDF become unreadable after extraction?
A PDF has no table structure. What looks like a table is text positioned in a grid with lines drawn separately, so naive extraction returns cells in whatever order they appear in the content stream — usually producing a stream of numbers with no indication which column or row they belong to.
Should I use OCR on a PDF that already has text?
Not usually, but check first. A PDF can be image-only, text-only, or a scan with an OCR text layer already added. Run a text extraction and see how much you get — if a ten-page document returns almost nothing, it is image-only and needs OCR.
What is the best way to convert PDFs for RAG?
Convert to Markdown using a layout-aware tool rather than raw text extraction. Markdown preserves headings, lists and tables in a form both chunkers and models understand, and it is far more token-efficient than the alternatives. For complex layouts, vision-model extraction handles what geometric tools cannot.
Why do the same headers appear in every chunk?
Because page headers and footers are just text on the page, indistinguishable from body content to an extractor. They repeat on every page, so they land in every chunk, diluting embeddings and wasting context. Detect repeated text at consistent positions across pages and strip it before chunking.