A trained model is useless where its framework cannot run, and its framework often cannot run where you need the model — a phone, a browser, an embedded board, a C# service. ONNX exists to break that coupling: describe the computation once, in a form anything can execute. It succeeded comprehensively in some domains and was quietly bypassed in others, and both outcomes are instructive.
What it is
ONNX — Open Neural Network Exchange — is a single file holding a model's computation graph and its weights, serialised with Protocol Buffers.
Because the graph is described in standard operators rather than framework code, any compliant runtime can execute it. Train once, deploy anywhere.
What is inside the file
An ONNX file is a protobuf message, and its structure follows the shape of a dataflow graph.
The key idea is that nodes reference tensors by name. There is no execution order beyond the data dependencies — a node runs when its inputs exist. That makes the file a description of what to compute rather than how to compute it, which is precisely what lets different runtimes optimise it differently.
Opsets, and where the pain lives
Operators are versioned as a set. Each ONNX release defines operator semantics, and a model declares which version it was exported against. Nearly every ONNX problem you will hit is an opset problem.
| Failure | Meaning |
|---|---|
| Unsupported operator | Runtime is older than the export |
| Operator not implemented for provider | Falls back to CPU, or fails |
| Wrong numeric results | Operator semantics changed between versions |
| Export fails outright | Source op has no ONNX equivalent |
The practical rule: export to the lowest opset that supports your model, not the newest available. A model exported at the latest opset runs only on runtimes updated to match, which is often not the phone or the embedded device you were targeting — the exact place ONNX was supposed to help.
⚠️ Dynamic control flow exports badly
ONNX export from PyTorch works largely by tracing — running the model once and recording the operations. A branch that depends on a tensor value gets recorded as whichever path that particular input took, and the condition disappears.
The result is a model that is silently wrong for inputs taking the other branch. Loops of variable length have the same problem. ONNX has If and Loop operators for this, but they need explicit handling rather than tracing. Always validate an exported model against the original on a range of inputs, not one.
Execution providers
ONNX Runtime separates the graph from the hardware that runs it through execution providers — backends registered in priority order, each claiming the subgraphs it can accelerate.
The fallback behaviour is the thing to watch. A graph can be partially accelerated, with unsupported nodes running on CPU and tensors copied back and forth at every boundary. That is functionally correct and can be slower than pure CPU execution, because the transfers dominate. If ONNX Runtime is unexpectedly slow, check how the graph was partitioned before anything else.
The browser case
This is where ONNX does something no other format really matches: real model inference in a web page, with no server.
For classification, embeddings, background removal, OCR and similar tasks this is genuinely practical, and it has the property that matters most for privacy: the data never leaves the device. It is the same argument as browser-based file tools — the work happens locally, so there is no upload to secure, retain or breach.
The constraint is size. A model has to download before it runs, so the useful range is a few megabytes to a few tens of megabytes. That rules out language models of any consequence and covers a large amount of useful vision and audio work.
Two structural limits
The 2GB protobuf ceiling. Protocol Buffers cannot address a message larger than 2GB, so larger models must use ONNX's external data format — the graph in the .onnx file, the weights in separate files alongside it.
Custom operators break portability. If your model uses an operation with no ONNX equivalent, you can register a custom operator — and you have now reintroduced the coupling ONNX existed to remove, since every runtime needs your implementation compiled in. Prefer restructuring the model to use standard operators.
Why local LLMs went elsewhere
ONNX dominates vision, audio, embeddings and classical models. It is not what people use to run a language model on a laptop, and the reasons are specific rather than a matter of fashion.
| ONNX | GGUF | |
|---|---|---|
| Designed for | Any model, any runtime | LLM inference |
| Quantisation | Basic schemes | Many, LLM-specific |
| KV cache handling | Manual | Built in |
| Loading very large files | External data files | Memory-mapped natively |
| Strength | Portability | Local LLM speed |
The pattern is generalisable: a format optimised for one workload beats a general format at that workload. ONNX's advantage is running the same model in eleven places; GGUF's is running one kind of model very well in one kind of place. Neither is a defeat for the other.
💡 When ONNX is the right choice
Use it when the deployment target differs from the training environment — mobile, browser, embedded, a non-Python service, or hardware with a vendor runtime.
Skip it when you train and serve in the same framework on the same kind of hardware. The conversion step adds a class of bugs — silent numerical differences, traced-away branches, opset mismatches — and buys you portability you were not going to use.
Security posture
ONNX is structurally safer than a pickle checkpoint, for the same reason safetensors is: the format describes data and predefined operations, not arbitrary object construction. Loading an ONNX file does not execute code the file supplies, because there is no field in which to supply any.
That is not the same as no risk. Protobuf parsers and runtime implementations have had vulnerabilities like any C++ codebase, malformed graphs can cause crashes or excessive allocation, and custom operators are, by definition, code you are choosing to run. Treat models from unknown sources with ordinary caution — but the worst case is much better than pickle's.
Working with model metadata and configs?
Format and explore JSON entirely in your browser — nothing is uploaded to a server.
Open JSON Formatter →Summary
- An ONNX file is a computation graph plus weights, in protobuf.
- Nodes reference tensors by name — the file says what to compute, not how.
- Opset mismatches cause most ONNX problems. Export to the lowest version that works.
- Traced exports silently bake in branches. Validate against the original on varied inputs.
- Execution providers can partially claim a graph, and the copies can make it slower.
- Browser inference is a genuine advantage — data never leaves the device.
- Over 2GB needs external data files, which must travel together.
- GGUF won local LLMs; ONNX still owns portable vision, audio and embeddings.
Frequently Asked Questions
What is an ONNX file?
A single file containing both a model's computation graph and its trained weights, serialised with Protocol Buffers. The graph is a list of operators and how data flows between them, which lets any compliant runtime execute the model without needing the framework it was trained in.
What is ONNX used for?
Deploying a model somewhere other than where it was trained. Train in PyTorch, then run on a phone, in a browser, inside a C# or Java service, or on specialised inference hardware — all without shipping a Python environment or the original framework.
What is an opset in ONNX?
A version number for the operator set. Each ONNX release defines operators and their semantics, and a model declares which opset version it needs. Export and runtime must agree well enough, which is why most ONNX problems appear as an unsupported operator error rather than a wrong result.
Is ONNX safer than a pickle checkpoint?
Structurally, yes. An ONNX file describes a graph of predefined operators, so loading it does not construct arbitrary objects the way unpickling does. It is not risk-free — parsers have had vulnerabilities and custom operators can introduce code — but the format has no built-in mechanism for executing arbitrary code.
Why don't people use ONNX for local LLMs?
Because GGUF was designed for that job and ONNX was not. Local LLM inference needs aggressive quantisation schemes, KV cache management and memory-mapped loading of very large files, and the llama.cpp ecosystem built all of that around GGUF. ONNX remains dominant for vision, audio, embeddings and classical models.