A PDF is not a document in the way a Word file is. It is a small database of numbered objects plus an index of where each one lives, and a page is a sequence of drawing instructions. That model explains a great deal — why PDFs open instantly regardless of size, why text extraction is unreliable, and why the standard way people try to redact them does not work.
The four sections
Header (the version), body (numbered objects), cross-reference table (a byte offset for every object), and trailer (where to find the catalogue). Readers start at the end — they read startxref, jump to the index, and then to whatever object they need.
Overall layout
The reading order is backwards from how it is written. A viewer seeks to the end, reads startxref to find the index, reads the trailer to find the catalogue object, and follows references from there. Nothing before that point needs to be read at all.
This is why a 500-page PDF opens as fast as a one-page one. The viewer resolves the page tree, jumps to the byte offset of page one's content, and renders it. The other 499 pages are never touched until you scroll.
💡 The binary marker line
The second line is a comment containing bytes above 127. It exists for the same reason as PNG's signature bytes: a file transfer in text mode would alter those bytes, and their alteration is detectable. It also causes tools that sniff for binary content to classify the file correctly.
The object types
| Type | Syntax | Example |
|---|---|---|
| Boolean | — | true |
| Number | — | 42, 3.14 |
| String | ( ) or < > | (Hello), <48656C6C6F> |
| Name | / | /Type, /Page |
| Array | [ ] | [0 0 595 842] |
| Dictionary | << >> | << /Type /Page >> |
| Stream | dict + stream | Compressed page content, images, fonts |
| Null | — | null |
| Reference | n g R | 3 0 R — "object 3, generation 0" |
References are what make it a graph rather than a tree. A font used on forty pages is one object referenced forty times. So is a logo image, a colour space, or a shared piece of content — which is why a PDF with the same image on every page is not forty times larger.
Streams hold everything bulky: page drawing instructions, embedded fonts, images, and attached files. A stream's dictionary declares its length and its compression filter, most commonly /FlateDecode — the same DEFLATE algorithm used by ZIP and PNG.
Content streams
A page's appearance is a sequence of operators in postfix notation — arguments first, then the operator.
Note that the coordinate origin is the bottom-left corner and y increases upward — inherited from PostScript and the opposite of the screen convention.
⚠️ Why text extraction is unreliable
A content stream positions text; it does not describe structure. There are no paragraphs, no reading order, and no guarantee that adjacent words are adjacent in the stream. A generator can place each character individually for precise kerning, so (H) Tj (e) Tj (l) Tj is entirely legal.
Extracting text means reconstructing words and reading order from coordinates. That works well for simple layouts and fails on multi-column pages, tables, and anything where the generator optimised positioning over sequence. It is why pasted PDF text arrives with a line break at every visual line, and why column layouts interleave.
Incremental updates
This is the most consequential thing in the format for anyone handling sensitive documents.
PDF can save changes by appending — new objects, a new cross-reference table, and a new trailer are added at the end, and the original bytes are left exactly as they were.
The /Prev entry chains the tables together. A reader follows the chain and, where two versions of an object exist, uses the most recent.
There are good reasons for this design. Saving a change to a 400MB file takes milliseconds rather than rewriting everything. Digital signatures over the original bytes remain valid, because those bytes never changed — which is exactly how a signed form can accept later annotations without invalidating the signature.
🚨 Earlier versions stay in the file
If someone edits a PDF to remove a paragraph and saves incrementally, the original paragraph is still in the file. The new cross-reference table simply points elsewhere. Reading the earlier xref chain recovers the previous version.
This has produced repeated real disclosures — published documents from which recipients recovered earlier drafts, unredacted names, and superseded figures.
Why black boxes are not redaction
The single most common PDF mistake, and it follows directly from the content stream model.
Drawing a black rectangle over text adds a fill operator after the text operators. The rectangle is painted on top. The text objects are untouched — still in the stream, still extractable, still selectable by anyone who drags a cursor across the area.
The same applies to changing text colour to white, covering it with an opaque image, or placing an annotation over it. All of them are visual operations layered above data that remains present.
✅ Redaction that works
- Use a tool with a genuine redaction function, which removes the underlying text objects rather than covering them.
- Apply the redactions — marking is not applying, and the two are separate steps in most software.
- Save as a new file, not an incremental update, so no earlier version is chained.
- Verify: run
pdftotext redacted.pdf -and search the output for what should be gone. - Check metadata and attachments —
exiftoolandqpdf --show-npageswill surface what the page view does not.
The verification step is the one people skip and the one that catches errors. It takes ten seconds and is the only way to know.
Object streams and xref streams
PDF 1.5 introduced two compression improvements that make files considerably smaller and slightly harder to read by hand.
Object streams pack many small objects into one compressed stream, so a document with thousands of small dictionaries — annotations, structure elements, bookmarks — compresses them together rather than leaving each in plain text.
Cross-reference streams replace the plain-text xref table with a compressed binary one. On a large document the table itself can be hundreds of kilobytes, so this is a real saving.
--qdf produces a deliberately human-readable PDF with uncompressed streams and normalised whitespace. It is the right first step for any investigation into what a file actually contains.
Linearisation
A linearised PDF — "optimised for fast web view" — is reorganised so that the objects needed for page one appear at the front, followed by a hint table describing where subsequent pages live.
A browser can then render page one after downloading perhaps 50KB of a 20MB document, and fetch later pages on demand using HTTP range requests. Without it, the reader must download to the end to find the xref before it can display anything.
Two things are required and one is often missed: the file must be linearised, and the server must support range requests. Without ranges the browser cannot fetch arbitrary byte offsets and downloads the whole file regardless.
Inspecting a PDF
⚠️ PDFs can contain JavaScript
The format supports embedded JavaScript for form validation and interactive behaviour, and it can execute on open. It has been used as a malware vector for many years, which is why most viewers disable it by default and why enterprise policies commonly block it entirely.
PDFs can also embed arbitrary file attachments and launch actions. If you process untrusted PDFs, qpdf and gs can strip these, and re-generating the file through Ghostscript removes anything not needed to render the pages.
Working with PDFs?
Merge, split and rotate PDFs entirely in your browser — the file never leaves your device, which matters for anything confidential.
Open the PDF Merger →Summary
- A PDF is numbered objects plus an index of byte offsets. Readers start at the end.
- References make it a graph, so shared fonts and images are stored once.
- Content streams position marks on a page — there is no paragraph structure, which is why text extraction is unreliable.
- Incremental updates append rather than rewrite, so earlier versions remain inside the file.
- Black boxes do not redact. The text is still there and still extractable.
- Always verify redaction with
pdftotextbefore sending. qpdf --qdfmakes the structure readable for inspection.- Linearisation needs server range request support to deliver any benefit.
Frequently Asked Questions
What are the first bytes of a PDF file?
%PDF- followed by a version number, such as %PDF-1.7. Most files then have a comment line containing bytes above 127, which signals to transfer tools that the file is binary and must not have its line endings converted.
Why does covering text with a black rectangle not redact a PDF?
Because the rectangle is just another drawing instruction placed after the text in the content stream. The text is still present as text, still selectable, still extractable by any tool. Proper redaction removes the text objects themselves and then flattens the page.
What is a PDF incremental update?
A way of saving changes by appending new objects and a new cross-reference table to the end of the file, leaving the original bytes untouched. It makes saving fast and preserves digital signatures — and it means earlier versions of the document remain inside the file and can be recovered.
What is the cross-reference table?
An index near the end of the file listing the byte offset of every object. A reader finds it by reading the startxref value in the last few lines, which lets it jump directly to any object without scanning the whole document. This is what makes PDFs of any size open quickly.
What does linearised or fast web view mean?
A PDF reorganised so the first page's objects appear at the front of the file, with a hint table describing where everything else lives. A browser can then display page one after downloading only a small fraction of a large document rather than waiting for the whole file.