A quantization-deployment platform: one truth, one contract, block by default

Once the quantization framework worked, what remained was precision chosen by hand, errors that were hard to locate, and training that did not line up with deployment. This is the target architecture I led: the fixed-point model as the single source of truth, versioned contracts between training, conversion and the vehicle, and a blocking gate at every hop. It is a design draft.

QCraft Nov 2026 onward Architecture lead

6stages in one pipeline, with an identity manifest running through it
7versioned contracts linking three codebases
10gates, from the training graph to the vehicle
  • Architecture
  • Quantization framework
  • Numerical alignment
  • Mixed precision
  • Multi-chip deployment
On this page

Why redesign

By 2026 the quantization framework let model owners quantize and deploy new models themselves, without the quantization team supporting each one. Three problems remained that process alone could not fix:

  1. Precision is chosen by hand. Which layers get INT16 is written into model code, from experience, with no record of why.
  2. Error sources are hard to locate. When something mismatches, someone compares the pipeline stage by stage by hand.
  3. Training does not line up with deployment. The metric seen in training comes from fake-quant; the vehicle runs fixed-point arithmetic.

They share one root cause: the same thing is written once in training, once in conversion and once on the vehicle, and kept in sync by convention. Each piece works on its own. Together, a change in one place has consequences that are hard to predict.

Target architecture: one pipeline

target architecture of the quantization-deployment platformdesign draft · roadmap to Q2 2027TARGET ARCHITECTURE: ONE PIPELINE, ONE IDENTITY MANIFEST, THREE FEEDBACK LOOPSroad-test frames → added to golden IOlatency cost table → precision planningfirst divergence → fix the graph or the export1Float trainingmodel team trains andevaluateshands over weights,config, commit2QuantizationPTQ and QAT shareone graph buildper-layer precisionfrom a profile3Export IRfreeze thequantization stateattach golden IO andthe IO contract4Buildthe IR is rebuiltexactly oncelowered per targetcapability5Verificationhop-by-hop gates ongolden IOrelease metric onfixed-point truth6Deploymentmodel packagearchivedprovenance checkbefore the carVERSIONED CONTRACT ARTIFACTSweightsconfig + commitstate fingerprintprecision profileIR + metadatagolden IOIO contractper-chip enginesfixed-point modelparity reportlatency cost tablemodel packagegenerated paramsIdentity manifestsource · environment · quant state · graph hash · toolchain · artifact hashes · parity and perf: appended per stage, earlier parts immutableSIX DESIGN PRINCIPLESOne truthfixed-point semanticsdecide evaluation, gatesand simulationOne contractrepositories exchangeonly versioned artifactsOne kernelmodel, platform andprecision differencesare dataOne implementationIR export and rebuildare written onceBlock by defaultover the limit meansfailure, with the firstdivergence reportedEnvironment as codetoolchain versionspinned and recorded
Six stages, each with defined inputs, artifacts and ownership; an identity manifest runs through all of them; three dashed loops carry deployment facts back to training.

Six design principles, each visible in the figure:

  • One truth. Fixed-point semantics are the only basis for evaluation, gates and simulation. Fake-quant is a training proxy, and its metric is not the release metric.
  • One contract. Codebases exchange only versioned contract artifacts, never file-name conventions or verbal agreements.
  • One kernel, differences outside. There is one flow. Model differences go into adapters, chip differences into capability files, precision into a precision profile. Whatever can be data is not a branch.
  • One implementation. IR export and rebuild are written once and shared by training, conversion and simulation.
  • Block by default. Every hop has a gate. Over the limit fails and reports the first divergence; a waiver must be written into the threshold file with an expiry date.
  • Environment as code. Toolchain versions, images and configs are checked in and recorded in every artifact's identity.

Seven contracts

Contracts turn convention into files that can be validated. Each carries a schema version.

Contract Content Written by Read by
Identity manifest Source, environment, quantization state, graph hash, toolchain, artifact hashes, parity and performance Training writes the first part, conversion appends, release seals it Every stage
IR metadata Schema version, quantization scheme, rounding rule, target chip, graph hash Export Every consumer; a version mismatch is rejected
IO contract Name, order, shape, dtype, scale and meaning of every input and output, plus pre- and post-processing Model adapter Conversion, on-vehicle parameter generation
Golden IO A few frames of real deployment-domain input, with reference outputs at every hop Export Parity engine, board spot checks, on-vehicle consistency tests
Precision profile Default precision plus overrides by module name, each with evidence Precision planning Quantization kernel
Target capability Supported dtypes, rounding, unsupported operators, INT16 fallback rules Chip owner Precision legalization, build routing
Thresholds Error limits per hop and per output, and waivers with expiry dates Model owner and quantization owner Parity engine

Hop gates: one truth, two zones

one truth, two zones, a gate at every hoptarget design: today most checks record, not blockTHE FIXED-POINT MODEL IS THE TRUTH: ERROR BUDGETS ON ITS LEFT, BYTE-EXACT ONITS RIGHTTolerance zone: blocks when over budgetByte-exact zone: any difference is a defectTraining graphfake-quantIRwith metadataRebuilt modelsingle code pathQAT HBIRchip toolchainFixed-pointmodel★ the truthHBMcompiledOn-boardthe deviceG1state fingerprintunchangedG2scales identicalG3code error ≤ limitG4fixed-point budgetG5byte-exactG6byte-exact, sampledG7Simulator = modelThe simulator used for x86 replayand evaluation must match thefixed-point model byte for byte.MMetric on the truthThe fake-quant metric is only atraining proxy; the release reportlists both, and their gap isbudgeted.G8Same frame on vehicleThe same raw frame is comparedend to end, from C++pre-processing to post-processing,quantization boundaries included.↩First divergenceWhen a gate fails, a tracer walksthe graph from inputs to outputsand reports the first node over thelimit, with its training-modulename.Waivers leave a trace: a waiver names the model, the hop, the reason, the owner and an expiry date in the threshold file; it expires automatically and every activewaiver is listed in the release report.
The fixed-point model is the truth: every hop on its left gets an error budget, every hop on its right must be byte-exact.
Gate Compares Criterion
G1 training graph → IR Quantization-state fingerprint before and after export Identical
G2 IR → rebuilt model All scales and dtypes; outputs under a fixed evaluation order Scales identical
G3 rebuilt model → QAT HBIR Integer-code error per output Within the limit
G4 QAT HBIR → fixed-point model Error introduced by going fixed-point Per-output budget
G5 fixed-point model → HBM Output bytes Byte-exact
G6 HBM → board Output bytes Byte-exact, sampled at release
G7 fixed-point model → fixed-point simulator Output bytes Byte-exact
G8 simulator ↔ vehicle Final outputs for the same raw frame Consistent
M release metric Fake-quant metric vs fixed-point metric Gap within budget

Most of the comparison tools exist today. What the design adds is turning “record” into “block”, and reporting the first node over the limit automatically when a gate fails.

Precision planning loop

precision planning loopdesign draftFROM HAND-TUNED PER-LAYER PRECISION TO A PROCESS WITH INPUTS AND EVIDENCE THAT CAN BE RE-RUN1LegalizeUse the targetcapability to rejectprecisioncombinations the chipcannot run, before thegraph is built.2SensitivityTwo experiments permodule group:quantize only thisgroup; raise only thisgroup to INT16.3CostPull the on-boardlatency cost table:module → bit-width →latency, CPU fallbackor not.4SelectWithin the latencybudget, choosegreedily by metricgained permillisecond; outputthe trade-off curve.5VerifyRun PTQ or QAT withthe plan, pass thegates, check thefixed-point metric andmeasured latency.6ArchivePlan, evidence andresults go into themanifest and arereviewed; anyone canre-run it.not met: tighten or relax the candidates and reselectInputsfloat model · calibration set · small eval set · target capabilitylatency budgetOutputsone precision profile, plus the evidence for each override: why it wasraised and what it costs in latency

Per-layer precision is tuned by hand inside model code today. The design replaces that with a process that has inputs, evidence and can be re-run. It rejects the combinations the target chip cannot run, runs sensitivity experiments per module group, brings in the on-board latency cost table, and chooses greedily by metric gained per millisecond within the latency budget. The output is a precision profile with a reason for every override: why it was raised and what it costs in latency. Model code no longer holds precision settings.

Quantize once, derive for every chip

one quantization result, derived for every chipdesign draftQUANTIZATION TRAINING IS DONE ONCE AND EVERY CHIP DERIVES FROM IT; CHIP DIFFERENCES ARE DATA, IN THREEPLACESShared assetsbuilt oncefloat weightsmodel adapterprecision intentQAT trunk weightsTARGETLEGALIZED PRECISION PLANMIGRATIONBUILD AND CHECKSJ6MINT8 + INT16 allow-listL0baseline · hop gatesJ6Psame plan, different roundingL0 / L1same build codeQualcommasymmetric uint8 / uint16L1 / L2recalibrate or short fine-tuneAXERAquantizable subgraph + FP16as needederror report firstTensorRTINT8 + float-layer listL0 / L1FP32 graph checkL0direct: shared weights, exportafter legalizationL1recalibrate: activation scalesonlyL2short fine-tune: a little QAT onshared weightsL3separate QAT: the fallback,today's practice

Quantization training is done once, and every chip derives from the shared weights and precision intent. Chip differences may appear in three places only, all of them data: the capability file, the legalized precision plan, and a declarative graph-rewrite config. There are four migration levels, from direct export to a separate QAT. The gap between those two is measured first, then each chip gets its default level. Adding a new chip becomes one capability file, one build back end and one on-vehicle runtime back end.

Rules that do not rely on discipline

  • Every dependency rule maps to a check that can fail. The quantization kernel does not import concrete models; exactly one directory may import the chip vendor's quantization library; chip names appear only in chip directories and capability files; per-layer precision comes only from the precision profile.
  • Seven test tiers, slower and closer to the device as they go up. Unit tests (seconds, no GPU) → contract tests (every merge) → GPU smoke → cross-environment alignment → build and simulation regression (daily) → board spot checks → release gates.
  • Legacy only shrinks. Hand-written precision, chip branches and environment switches each have an allow-list, and CI only lets the lists get shorter.

Migration approach

Measurement and gates come first, code migration second: first measure and block, then choose, and only then release through the new pipeline. A pilot model goes first and models outside the pilot keep the current flow throughout. Every step is accepted on “behavior unchanged”.

Notes from the design phase

  • Measure before changing. Once “it feels messy” became specific numbers, the priorities sorted themselves.
  • Start from what exists. Taking stock of what each gate already has and what it lacks keeps the design from becoming a rewrite.
  • Turn conventions into files. Anything that rests on a verbal agreement will fail quietly in some toolchain upgrade.