TensorRT INT8 for cloud GPU models, and onnx2trt
An INT8 quantization and deployment pipeline for Tencent Maps' cloud GPU models: cosine similarity above 0.999 to float, latency −20%, GPU utilization and memory −10%, and roughly ¥3M saved in compute. The calibration tooling became the open-source onnx2trt.
> 0.999output cosine similarity, INT8 vs float
−20%inference latency
≈ ¥3Mcompute cost saved
On this page
The problem
Tencent Maps' cloud models used more GPU capacity than the budget allowed. INT8 inference was the obvious lever. The obstacle was accuracy: quantized models lost precision, and converting and testing each model by hand did not scale.
The pipeline
PyTorch model
ONNX export
Calibrationseveral algorithms
Accuracy checkcosine vs float
TensorRT INT8 engine
Serving + tests
- Export to ONNX as the common input format.
- Calibrate with more than one algorithm: ONNX Runtime's quantization calibrators and TensorRT's own calibrators, behind one interface.
- Evaluate by output similarity. Compare the quantized model's outputs with the float model's using cosine similarity. A calibration method that optimizes this similarity directly was added alongside the standard ones.
- Build and test the engine automatically: latency, GPU utilization, memory and accuracy, so every model went through the same gate.
Results
onnx2trt
I released the calibration tooling as onnx2trt, an open-source tool for ONNX → TensorRT INT8 quantization that:
- combines ONNX quantization with TensorRT's built-in calibration algorithms;
- exposes an interface for custom calibration algorithms;
- includes the cosine-similarity-driven calibration method.
Lessons
- Judge quantization by output similarity to the float model, not by the calibrator's own statistic.
- Automate the test, not just the conversion. A conversion that is not measured the same way every time cannot be compared.