OneModel: five perception tasks on one shared vision encoder

Merging the image backbone and neck of five perception models into one: how the latency budget works out, how the new backbone was sized, how its pretraining weights were synthesized from released models, and the end-to-end design that follows. Work in progress.

QCraft 2026 Plan and training-framework design

5 → 1image encoders (backbone + neck)
5–9 msestimated saving per frame, rate-weighted
9cameras unified at 512 × 1024
  • Multi-task learning
  • Shared backbone
  • PCA
  • Model architecture
  • End-to-end driving
On this page

The problem

On-board latency limits model capacity and frame rate. The latency model showed that the image encoder is the one large module whose cost does not grow with perception range, so it is the part to share.

Phase 1 reused the detection model's backbone directly. The other tasks lost accuracy to varying degrees, occupancy most of all. Phase 2 exists to fix “metrics fall short after merging”.

Structure

OneModel: one shared image encoder for five perception tasksin progress: joint training runs, metrics still below releaseFIVE PERCEPTION TASKS AND 9 CAMERAS SHARE ONE IMAGE BACKBONE AND NECKToday: every task encodes on its ownbackbone + neckDetection + trackingbackbone + neckVector mapbackbone + neckOccupancybackbone + neckFront-view perceptionbackbone + neckTraffic lightsOneModel: shared up to the neck, task heads read from it9 cameras512×1024sharedbackbone+ neckDetection + trackingVector mapOccupancyFront-view perceptionTraffic lights5–9 msestimated saving perframe(rate-weighted)NOT FROM SCRATCH: RELEASED MODELS SIZE THE BACKBONE AND SEED ITS WEIGHTS1Extractfeatures from the five tasks'float checkpoints, stage bystage2PCA redundancyshrink redundantdimensions, keep theinformation-dense ones3Align channelsalign each checkpoint'schannels to a common basis4Closed-form solvea linear solve yields theneck and per-taskprojections5Check errorverify reconstruction erroron held-out clipsTwo quantization-friendly fixes ride along: ReLU6 with a surrogate gradient (ReLU6 forward, a smooth surrogate's derivative backward) and a correctedreceptive-field center offset.
  • Shared up to the image neck, not into BEV space for now. The tasks use different BEV ranges and resolutions. Sharing BEV features would mean unifying both, and compute would multiply.
  • Nine cameras at 512 × 1024, with no split between a main and an auxiliary backbone. Fisheye cameras and two rear cameras stay separate for now.
  • The latency budget is rate-weighted. Taking one 10 Hz frame as the unit, a 5 Hz task counts half. Sharing the backbone alone saves about 5 ms per frame; sharing every neck as well saves about 9 ms. Whether the two front-view tasks share a neck accounts for about 4 ms of that. This saving is the budget for a larger backbone.

Sizing the backbone from released models

The width of the new backbone was not chosen by feel.

  1. Extract features stage by stage with the released float checkpoints of the five tasks.
  2. Run PCA on the features to measure how redundant each stage's channels are. Redundant dimensions can shrink; information-dense ones must stay.
  3. Align the channels of the checkpoints, then use PCA to select the dimensions to keep.
  4. Solve for the neck and the per-task projection layers in closed form, which yields a set of pretraining weights.
  5. Check the reconstruction error on held-out clips.

The resulting initial weights already carry most of what the five tasks learned, so joint training does not start from zero.

Two quantization-friendly fixes

  • ReLU6 with a surrogate gradient. From a quantization point of view the activation should be ReLU6, but swapping it in directly hurts training smoothness. The forward pass uses a plain ReLU6. The backward pass replaces its zero derivative in the saturated regions with the derivative of a smooth surrogate.
  • Receptive-field center offset. The original downsampling shifted the center of the receptive field. It is replaced with the common stride-2 plus 2 × 2 convolution, the FPN is kept from adding to the shift, and the sampling grid in the view transform gets the same correction.

How multi-task training is organized

The five tasks do not share data sources, and their labels are not on the same frames. I compared four ways to organize training:

Organization Data and gradient behavior Verdict
Rotate tasks per batch, all GPUs on the same task At each step every GPU draws data for the same task; the shared encoder and that task's head back-propagate Used first. Reuses each task's loader and sampler; tasks are mixed by planned training exposure
Same-source samples, multi-task supervision When one set of cameras has labels for several tasks, shared features run once and the losses are summed An enhancement; requires labels aligned to the same frame, coordinates and augmentation
Mixed tasks within a batch Each sample carries task-validity bits and selects its losses Later; needs changes to task enabling and cross-GPU normalization, a mask alone is not support
Fixed task per GPU Tasks occupy fixed GPU groups; shared parameters sync across groups Not the main line; communication, optimizer and resume are all more complex

One rule runs through all of it: a missing label is never a negative. A task mask only says whether supervision is valid.

Path to release

  1. Single-task restructuring. Each task first moves to the structure it will have in joint training and ships on its own.
  2. Partial merges. Tracking with occupancy; front-view perception with traffic lights (both are image-level features with no BEV transform, so they are easier to merge).
  3. Five-task merge. The backbone is trained jointly and each task ships on the frozen shared encoder. If shared parameters, normalization statistics or quantization boundaries change, a new shared version is cut and every consumer is regression-tested.
  4. Joint perception and planning training.

The main risk is at release: production tolerates very little regression. If a partial merge reaches 90 and the five-task merge only 85, the release stalls. So every step merges only when no task regresses.

One step further: an end-to-end driving model design

end-to-end driving model: design proposaldesign proposal · not yet trained · ≈140M parameters (estimate)END-TO-END DRIVING MODEL DESIGN: ONE DIFFERENTIABLE PATH FROMPIXELS TO PLAN9 cameras512×1024Kinematics1 s windowNavigationSD map + routeShared imageencoderreused as isScene tokensrays + camera ID · 672Kinematic tokens8Navigation prefix64 · cacheableStreaming transformer12 layers · d = 768Pose-aware memory8 × 48 tokens ≈ 3.2 seach tick attends to cachedmemory; keys modulated byrelative pose and age, nofeature warpingPlan head16 whole-plan modes; path as curvatureknots over arc length, speed asacceleration knots over time, rolled outdifferentiably to 8 sAuxiliary headsoccupancy + futureagents + forecastlane graphtraffic lightsdepth · train onlyworld model · train onlyTHE PATH TO THE VEHICLE IS DESIGNED IN FROM THE STARTperceptionpretrainingtemporal + futurepredictionimitationclosed-loopevaluationdistill to the vehiclemodelINT8 QAT and export
One differentiable path from pixels to plan; the shared image encoder is reused as is.

On top of the shared encoder I wrote an architecture design for an end-to-end driving model. The main choices:

  • The shared encoder is reused unchanged. Its weights, export flow and measured latency stay valid.
  • Tokens on the main path, not a BEV grid. This removes each task's private view transform and gives every loss, planning included, a path back into the encoder.
  • Vision-only input. LiDAR is used to build labels and supervise depth, never as a model input.
  • The plan head outputs control quantities. The path is curvature knots over arc length and speed is acceleration knots over time. Both are rolled out differentiably into a trajectory, so position losses train the knots directly.
  • Navigation enters as tokens built from the SD road network and the route, never from the future ego trajectory, which would leak the label.
  • Deployment is designed in from the start. A fixed-shape graph; distillation to the vehicle model, then INT8 quantization-aware training; and closed-loop metrics, not only open-loop loss, decide whether the quantized model is accepted.

Lessons

  • Share the part that does not change. A module is worth sharing when its cost does not move with the requirement.
  • Let existing models answer questions about the new one. The released checkpoints already hold the answer to how wide each stage should be.
  • Merge only without regression. A multi-task average means nothing; every task has to hold its own metric.