At time of this writing, most modern ML inference workloads, especially ones in production, run on NVIDIA GPUs for better performance. NVIDIA TensorRT is one such tool that takes in a PyTorch or ONNX model, applies some model optimizations and generate TensorRT binaries to be run on compatible GPUs.

Some model optimization techniques here include:

  1. Weight and activation precision calibrations (quantization)
  2. Operation (op) / layer fusions
  3. Auto-tuning of kernels so only the best algorithms are selected to run on your specific target device (GPU)
  4. Multi stream execution etc.

Cellulose currently integrates basic TensorRT features within our dashboard. These include:

Read more about NVIDIA TensorRT here

Have questions / need help?

Please reach out to, and we’ll get back to you as soon as possible.