TensorRT LLM is an NVIDIA library for running large language model and visual generation inference on NVIDIA GPUs. It combines custom kernels for common inference operations (attention, matrix multiplication, and mixture-of-experts layers) with runtime-level techniques such as prefill-decode disaggregation, wide expert parallelism, and speculative decoding.
The library is built on PyTorch and exposes a high-level Python LLM API that covers single-GPU, multi-GPU, and multi-node deployments with built-in parallelism strategies. It includes an OpenAI-compatible serving command, an offline inference API, and benchmarking and evaluation tools, and it connects to NVIDIA Dynamo and Triton Inference Server for wider serving setups. Several popular models are predefined in native PyTorch code that you can modify, and new architectures can be added.
Supported optimizations include a paged KV cache with reuse across requests and compression, chunked prefill, quantization, LoRA adapters, CUDA graphs, guided decoding, and multimodal and encoder-only embedding models. Visual generation of images and video with diffusion models is in beta. Model families with dedicated optimizations include DeepSeek, Llama, GPT-OSS, and Qwen.
The library collects anonymous usage telemetry by default (deployment shape, GPU details, and feature flags, not prompts, outputs, or model paths), and it can be turned off through the Python API, a CLI flag, or configuration.
Features
- Custom kernels: optimized attention, GEMM, and mixture-of-experts kernels for NVIDIA GPUs
- Python LLM API: a high-level API for single-GPU, multi-GPU, and multi-node inference
- Online serving: an OpenAI-compatible server for chat, completion, and responses endpoints
- Parallelism: tensor, pipeline, context, and wide expert parallelism
- Disaggregated serving: separates prefill and decode across workers
- Speculative decoding: draft-based and n-gram methods to raise token throughput
- KV cache management: paged attention with cache reuse across requests, compression, and sliding-window attention
- Quantization: low-precision formats with model and hardware support matrices
- LoRA serving: serves and benchmarks LoRA adapters
- Multimodal and embeddings: vision-language models and encoder-only embedding models
- Visual generation (beta): diffusion models for image and video generation
- Ecosystem integration: works with NVIDIA Dynamo and Triton Inference Server
