AR navigation perception SDK

A perception SDK for AR navigation on phones and car head units: a multi-stream parallel framework with automatic profiling. 12 features delivered in four months with CPU usage under 70%, shipped on the JMC CX743 production vehicle.

Tencent Maps 2021 – 2022 SDK design and on-device model deployment

12perception features in four months
< 70%CPU usage on the head unit
1production vehicle launch (JMC CX743)
  • Mobile / in-car SDK
  • Parallel pipelines
  • Profiling
  • On-device deployment
On this page

The problem

AR navigation overlays guidance on the live camera view, so perception has to run in real time on hardware that is also running navigation, maps and the car's own software. A phone or head-unit CPU has little headroom. A feature that is accurate but late is useless, and an SDK that starves the rest of the system will not ship.

Design

Camera frames
Parallel perception streamsroad, parking slots, …
Results to navigation
AR guidance
  • Multi-stream parallel framework. Perception tasks run as parallel streams, so a slow task does not hold up the others and new tasks can be added without rewriting the pipeline.
  • Automatic profiling built in. Every stream reports its timing and CPU cost continuously, so performance regressions show up during development, not in the car.
  • On-device models deployed and tuned for the target hardware.

Results

Road-scene perception
Parking-slot detection
AR navigation

Visual positioning SDK

My work also included the SDK for the visual positioning system used alongside AR navigation.

Visual positioning SDK architecture
Visual positioning SDK design

Lessons

  • Budget the CPU first, then the features. Real-time on shared hardware is a scheduling problem as much as a model problem.
  • Make performance visible all the time. Built-in profiling caught regressions long before integration.