NVIDIA TensorRT

TensorRT is NVIDIA's family of tools for speeding up deep learning inference, including an inference compiler, runtime and model optimizer. ML engineers use it to cut latency and raise throughput when serving models trained in PyTorch or other frameworks.

It optimizes trained networks through layer and tensor fusion, kernel tuning and lower-precision calibration, then deploys them on data center GPUs, workstations, laptops and edge devices. The NVIDIA Model Optimizer adds post-training quantization and quantization-aware training with FP8, FP4, INT8, INT4 and AWQ.

The wider ecosystem includes TensorRT LLM for large language models and TensorRT for RTX on consumer PCs. The SDK is a free download with documentation, a GitHub repository and a developer guide, and NVIDIA claims up to 36 times faster inference than CPU-only setups.