What is Calibration, Really?

When we move from 32-bit floating-point numbers (FP32) to 8-bit integers (INT8), we are drastically reducing the precision of the weights in our neural network. The process of calibration is how we intelligently map the wide range of FP32 values to the limited 256 values available in INT8. A good mapping preserves the dynamic range of the activations, while a poor one can lead to 'clipping' and a severe drop in accuracy.

Choosing a Calibration Dataset

The number one rule: your calibration dataset must be representative of the data your model will see in production. It doesn't need to be large—often 500 to 1000 samples are enough—but it must capture the statistical distribution of your real-world inputs. We'll walk through a practical example of selecting a subset of the COCO dataset for an object detection model.

Using the XTorch Calibrator

XTorch simplifies this process. Instead of manually writing a TensorRT calibrator class in C++, you can simply point to your dataset folder. We'll explore the different calibration algorithms available in XTorch, like Entropy Minimization and MinMax, and explain the tradeoffs for each.

xtorch convert --model yolov8.pth --precision int8 --calibration-data ./coco_calibration_images/ --input-shape 1 3 640 640