Magic Numbers: How Software Identifies Any File

Q: What is a magic number in a file?

A fixed byte sequence at a known position — usually the very start — that identifies the file's format. PDFs begin with %PDF-, PNGs with a sequence containing PNG, and ZIP archives with PK. Because it is part of the data rather than the name, it cannot be changed by renaming the file.

Q: Why is the extension not enough to identify a file?

Because it is part of the filename, not the file. Anyone can rename anything. A malicious executable named invoice.pdf is still an executable, and a WebP renamed to .jpg is still a WebP that a strict decoder will reject. Only the bytes are authoritative.

Q: How does the file command work?

It uses a database of signature rules called magic, testing byte patterns at specified offsets, following conditional rules and combining results. If nothing matches it falls back to heuristics — checking whether the content looks like valid text in some encoding, and if so which.

Q: Why does plain text have no magic number?

Because any byte sequence could legitimately be text. Identification is done by exclusion instead: if a file contains no null bytes, few control characters, and decodes cleanly as UTF-8 or another encoding, it is treated as text. That is a heuristic, and it is occasionally wrong.

Q: What is a polyglot file?

A file that is simultaneously valid in two or more formats — a working JPEG that is also a working ZIP, for instance. They exist because some formats ignore data before their signature and others ignore data after their end marker. They are a genuine security concern for anything that validates by signature alone.

A filename is a label a human chose. The bytes inside are what the file is. Nearly every binary format begins with a fixed signature — a magic number — that identifies it regardless of what it is called, and knowing how to read those signatures is the difference between guessing at a file and knowing what it is.

The principle

Extensions and MIME types are claims. Magic numbers are evidence. When they disagree, the bytes are right. Any code that decides what to do with a file based on its name is trusting whoever named it.

Why the term is "magic"

The name comes from early Unix. The a.out executable format began with a specific 16-bit value that the kernel checked before attempting to load a program — a constant with no meaning beyond "if you see this, it is one of ours". Programmers called it the magic number, and the term generalised to any fixed identifying value.

The design goal was practical. A file has no place to store a type declaration — a filesystem stores a name and bytes, not a schema — so putting the identifier in the data is the only way to make it travel with the file. It survives renaming, copying, email, archiving and transfer between operating systems that disagree about everything else.

The signature reference

Images

Format	Hex	ASCII	Offset
PNG	`89 50 4E 47 0D 0A 1A 0A`	`.PNG....`	0
JPEG	`FF D8 FF`	—	0
GIF87a	`47 49 46 38 37 61`	`GIF87a`	0
GIF89a	`47 49 46 38 39 61`	`GIF89a`	0
WebP	`52 49 46 46` … `57 45 42 50`	`RIFF`…`WEBP`	0 and 8
BMP	`42 4D`	`BM`	0
TIFF (LE)	`49 49 2A 00`	`II*.`	0
TIFF (BE)	`4D 4D 00 2A`	`MM.*`	0
ICO	`00 00 01 00`	—	0
AVIF / HEIC	`66 74 79 70`	`ftyp`	4
SVG	—	`<svg` or `<?xml`	varies

Documents and archives

Format	Hex	ASCII
PDF	`25 50 44 46 2D`	`%PDF-`
ZIP / DOCX / JAR / APK	`50 4B 03 04`	`PK..`
Empty ZIP	`50 4B 05 06`	`PK..`
RAR v4	`52 61 72 21 1A 07 00`	`Rar!...`
RAR v5	`52 61 72 21 1A 07 01 00`	`Rar!....`
7-Zip	`37 7A BC AF 27 1C`	`7z`…
GZIP	`1F 8B`	—
BZIP2	`42 5A 68`	`BZh`
XZ	`FD 37 7A 58 5A 00`	—
Zstandard	`28 B5 2F FD`	—
DOC / XLS (legacy)	`D0 CF 11 E0 A1 B1 1A E1`	—
RTF	`7B 5C 72 74 66`	`{\rtf`
SQLite	`53 51 4C 69 74 65`	`SQLite`

Media and executables

Format	Hex	Notes
MP3 (ID3)	`49 44 33`	`ID3` tag header
MP3 (raw frame)	`FF FB` / `FF F3`	No tag present
MP4 / MOV	`66 74 79 70`	`ftyp` at offset 4
WAV	`52 49 46 46` … `57 41 56 45`	`RIFF`…`WAVE`
FLAC	`66 4C 61 43`	`fLaC`
OGG	`4F 67 67 53`	`OggS`
Matroska / WebM	`1A 45 DF A3`	EBML header
Windows EXE/DLL	`4D 5A`	`MZ` — Mark Zbikowski
Linux ELF	`7F 45 4C 46`	`.ELF`
macOS Mach-O	`CF FA ED FE`	64-bit
Java class	`CA FE BA BE`	"cafe babe"
WebAssembly	`00 61 73 6D`	`.asm`
Shell script	`23 21`	`#!` — the shebang

⚠️ Not every signature is at offset zero

MP4, MOV, HEIC and AVIF all begin with a four-byte size field, so their ftyp identifier sits at offset 4. Code that only checks the first bytes misidentifies every one of them.

Container formats need two checks. WebP and WAV both start with RIFF and are distinguished by four bytes at offset 8. A validator testing only the first four bytes treats an audio file and an image as the same thing.

🚨 MZ means executable

4D 5A — the letters MZ, the initials of DOS architect Mark Zbikowski — begins every Windows executable and DLL. If you were expecting a document and the first two bytes are MZ, the file is a program whatever its name says. Delete it.

The same applies to 7F ELF on Linux and the Mach-O signatures on macOS.

How the file command works

The Unix file utility and the libmagic library behind it use a rule database with a small declarative language:

# offset  type    value       message
0         string  \x89PNG     PNG image data
>16       belong  x           \b, %d x
>20       belong  x           %d
>24       byte    x           \b, %d-bit
>25       byte    3           colormap

# The > prefix means "only test this if the parent matched".
# So it identifies PNG, then reads dimensions and colour type
# from the IHDR chunk to describe the file precisely.
                

This is why file can report far more than a format name — it walks into the structure and reads real fields.

file photo.jpg
# photo.jpg: JPEG image data, JFIF standard 1.01,
#   resolution (DPI), density 72x72, segment length 16,
#   baseline, precision 8, 1704x1080, components 3

# Just the type, for scripting
file --mime-type document
file -b --mime document

# Look at the bytes yourself
xxd -l 16 mystery
head -c 16 mystery | xxd
                

# PowerShell equivalent
Get-Content mystery.bin -AsByteStream -TotalCount 8 |
    ForEach-Object { '{0:X2}' -f $_ }
                

Why text is the hard case

Plain text has no magic number, and cannot have one — any byte sequence could legitimately be text in some encoding. Identification works by exclusion and heuristics:

Does it contain null bytes? Almost certainly binary.
What proportion of bytes are control characters outside tab, newline and carriage return? A high proportion means binary.
Does it decode cleanly as UTF-8? UTF-8's structure makes false positives unlikely.
If not, does the byte distribution resemble a known code page?
Is there a byte order mark? That is a strong hint.

BOM bytes	Encoding
`EF BB BF`	UTF-8
`FF FE`	UTF-16 little-endian
`FE FF`	UTF-16 big-endian
`FF FE 00 00`	UTF-32 little-endian

Note the ambiguity in that table: a UTF-32 LE file starts with the same two bytes as a UTF-16 LE one. You must check four bytes before concluding, and code that checks two gets it wrong.

Validating uploads

Three signals arrive with an upload and all three are supplied by the client:

Signal	Source	Trustworthy?
File extension	The filename	No
`Content-Type` header	The browser or client	No
Magic number	The file's bytes	Better, not sufficient
Successful full parse	Your decoder	Best available

// Signature check — necessary, not sufficient
const SIGNATURES = {
    'image/png':  { offset: 0, bytes: [0x89, 0x50, 0x4E, 0x47] },
    'image/jpeg': { offset: 0, bytes: [0xFF, 0xD8, 0xFF] },
    'image/gif':  { offset: 0, bytes: [0x47, 0x49, 0x46, 0x38] },
    'application/pdf': { offset: 0, bytes: [0x25, 0x50, 0x44, 0x46] },
};

function detect(buf) {
    for (const [type, sig] of Object.entries(SIGNATURES)) {
        if (sig.bytes.every((b, i) => buf[sig.offset + i] === b)) return type;
    }
    // WebP and WAV both start RIFF — check offset 8
    if (buf.toString('ascii', 0, 4) === 'RIFF') {
        const sub = buf.toString('ascii', 8, 12);
        if (sub === 'WEBP') return 'image/webp';
        if (sub === 'WAVE') return 'audio/wav';
    }
    return null;
}
                

🚨 A valid signature is not a valid file

Prepending FF D8 FF to arbitrary data makes it pass a JPEG signature check. The bytes after it can be anything at all, including a crafted payload aimed at whatever decoder eventually opens it.

Signature checking rejects the obviously wrong. It does not prove the file is well-formed. For anything you will process or serve:

Decode it fully in a sandboxed process with resource limits.
Re-encode it and store your own output — this normalises the file and discards appended data, unexpected segments and malformed structures in one step.
Never serve it from your own origin without Content-Disposition: attachment and X-Content-Type-Options: nosniff.
Generate your own filename. Never use the client's.

Polyglot files

Some files are simultaneously valid in two formats, which is possible because different formats have different tolerances about what surrounds their data.

Combination	Why it works
JPEG + ZIP	JPEG decoders stop at `FF D9`; ZIP readers seek from the end. Neither notices the other.
GIF + JavaScript	A GIF's header can be crafted so the file is also syntactically valid JS.
PDF + ZIP	PDF ignores data after `%%EOF`.
PDF + HTML	Both tolerate unexpected leading and trailing content.

The JPEG+ZIP case is the classic. It is genuinely useful — it is how self-extracting archives work — and it is genuinely a problem, because a file that passes an image validator and is later processed by an archive tool behaves as two different things.

The defence is the same as above: re-encode. Decoding a JPEG and writing a new one with your own encoder produces a file containing only what your encoder wrote. Everything appended, prepended or hidden between segments is gone, because it was never part of the decoded image.

Summary

Extensions and MIME types are claims; magic numbers are evidence.
Not every signature is at offset zero — MP4, HEIC and AVIF have ftyp at offset 4.
RIFF containers need a second check at offset 8 to distinguish WebP from WAV.
MZ, .ELF and Mach-O signatures mean executable, whatever the file is called.
Text has no signature and is identified by exclusion and heuristics.
A valid signature does not mean a valid file. Decode to be sure.
Re-encode uploads. It removes appended data, polyglots and malformed structures at once.
Send nosniff on anything user-supplied that you serve.

Frequently Asked Questions

What is a magic number in a file?