Document processing is the most common real use for language models in business, and the most commonly over-engineered. The pitch is an agent that reads your documents and figures things out. The working system is a mostly-ordinary pipeline with a model doing one hard thing in the middle — and knowing which part is which is the difference between something you can operate and something that costs a fortune to debug.
The shape that works
Five fixed stages: ingest → classify → extract → validate → route. Every one has a known input and output, which makes the pipeline a workflow, not an agent.
The agentic part is the exception path — documents that fail validation, where the number of recovery steps genuinely varies. That is perhaps five percent of volume and all of the difficulty.
The pipeline
Nothing about stages one to five requires a model to decide what happens next. The order is fixed, the branches are enumerable, and writing them in code makes cost, latency and behaviour all predictable. Resist the urge to hand this to an agent because the components involve a model — that is the mistake this whole design exists to avoid.
Stage 1: ingest, where the accuracy is won or lost
Everything downstream operates on the text this stage produces. If it is scrambled here, no prompt fixes it — the information is already gone.
| Document kind | Approach | Watch for |
|---|---|---|
| Born-digital PDF | Text layer extraction | Column order, tables |
| Scanned PDF or image | OCR or a vision model | Skew, quality, handwriting |
| Mixed PDF | Detect per page | The trap below |
| Office documents | Native parsing | Tracked changes, comments |
| Parse and walk parts | Attachments, quoted chains |
🚨 The mixed-PDF trap
A PDF with a text layer on pages 1–3 and scanned images on pages 4–6 will extract cleanly, return plausible text, and silently omit half the document. Nothing errors. The extraction step then reports that fields are missing, and the investigation goes looking at the prompt.
Detect per page, not per document: if a page yields almost no text but has a large image, it needs OCR. A page-level character-count check catches this in three lines and saves a very confusing afternoon.
Two more ingest problems worth pre-empting. Multi-column layouts extract in the wrong reading order by default, interleaving two columns into nonsense — layout-aware extraction or a vision model is required, not optional. Tables lose their structure entirely in naive extraction, which matters because tables are usually where the numbers you want live. These are the substance of why PDFs break retrieval pipelines too, and the fixes are the same.
Stage 2: classify
A cheap, fast model with a small enum output. This is a routing step and does not need your expensive model.
Send only the first page or two. Document type is almost always determinable from the opening, and classifying on the full text multiplies the cost of the cheapest stage for no gain.
Stage 3: extract
One schema per document type. Use constrained decoding so the output shape is guaranteed rather than hoped for, and build two things into every schema.
Nullable everything. A model with no legitimate way to report absence will produce a plausible value instead. Most invented fields are the schema's fault, not the model's.
Require a source quote. This is the single highest-value design decision in the pipeline, because it converts an unverifiable claim into a checkable one. You can search the document for that string in code. If it is not there, the value was invented — and you know that automatically, for every field, without a human reading anything.
Stage 4: validate in code
The model has produced a well-shaped object. Whether it is true is a separate question, and it is answered here, deterministically.
Arithmetic checks are worth emphasising. An invoice whose line items do not sum to its total is either misextracted or genuinely wrong, and both need a human. This catch requires no model and no judgement, and it finds a meaningful share of real errors.
⚠️ Do not ask the model how confident it is
Self-reported confidence scores look useful and are not well calibrated — a model will report 0.95 on a value it invented, because the number is generated the same way the value was.
Derive confidence from things you can measure: did the source quote verify, did the arithmetic hold, did the format check pass, do two independent extraction passes agree. Those are facts about the world rather than the model's impression of itself.
Stage 5: the exception path, which is the agentic part
Now the narrow case that justifies a loop. A document has failed validation. What happens next genuinely varies: it might need re-OCR at a different resolution, extraction with a different schema, a page-order correction, or a human. You cannot enumerate the sequence in advance — which is exactly the condition that warrants an agent.
Three properties make this safe. It is capped at three attempts. Success is judged by the same deterministic validator as the main path, not by the model's opinion. And the terminal state is a human, not an apologetic message — because a document that failed three recovery attempts is genuinely unusual and someone should look at it.
The economics people get wrong
| Decision | Naive | Better |
|---|---|---|
| Classification model | Frontier | Cheapest that works |
| Vision processing | Every page | Only pages that need it |
| Text sent to extract | Whole document | Relevant pages only |
| Retries | Same model again | Change something first |
| Human review | Everything, or nothing | Flagged only |
The last row is where the return on the whole system lives. A pipeline that flags fifteen percent of documents and passes the rest cleanly has removed eighty-five percent of the manual work — and it did so because the validation is strict enough to trust. Loosening validation to reduce the flag rate does not save money; it moves the errors downstream into systems where they cost more to find.
💡 Build the boring version first
Stages one to four, one document type, no exception loop, everything that fails goes to a human. That system is quick to build, easy to measure, and immediately useful.
The failures it escalates are then your specification for the exception path — real documents, real reasons, in proportion to how often they occur. Building the loop first means guessing at that list, and the guesses are always wrong in the same direction: too clever about rare cases, insufficiently careful about common ones.
Working with PDFs before they hit your pipeline?
Split, merge, rotate and convert PDFs entirely in your browser — nothing is uploaded to a server.
Open PDF Splitter →Summary
- Ingest, classify, extract, validate, route. Fixed stages — a workflow, not an agent.
- Accuracy is won at ingest. Scrambled text cannot be fixed downstream.
- Detect scanned pages per page. Mixed PDFs silently lose half the document.
- Always include an "unknown" class so odd documents are not forced into a type.
- Nullable fields prevent invented values. Absence needs a legitimate way to be reported.
- Require a source quote per field — it makes hallucination automatically detectable.
- Validate in code: quote existence, arithmetic, formats. Never trust self-reported confidence.
- Only the exception path is agentic — capped, code-verified, escalating to a human.
Frequently Asked Questions
Should a document-processing pipeline be an agent?
Mostly not. Classification, extraction and validation are fixed stages with known inputs and outputs, which makes them a workflow. The genuinely agentic part is narrow: handling documents that fail the standard path, where the number of recovery steps varies by document.
What is the hardest part of document extraction?
Getting clean text out of the document, not the extraction itself. A PDF with multi-column layout, tables or scanned pages will produce text in the wrong reading order, and no amount of prompt engineering recovers information that was scrambled before the model saw it.
How do I stop a model inventing field values?
Constrain the output to a schema so the shape is guaranteed, require a source quote for every extracted value, and allow an explicit null. Most invented values appear when the model has no way to say the field was absent, so giving it a legitimate way to report absence removes the incentive.
How should confidence be handled in extraction?
Derive it in code rather than asking the model to score itself. Whether a quoted source string actually appears in the document, whether a value passes format and range checks, and whether two independent passes agree are all measurable. Self-reported confidence scores are not well calibrated.
Do I need a vision model for scanned documents?
For scans and image-heavy pages, yes — text extraction has nothing to extract from a picture of text. For born-digital PDFs with a proper text layer, ordinary extraction is faster, cheaper and more accurate, so detect which kind you have rather than sending everything through the expensive path.