Triton Inference Server is NVIDIA's open-source serving software for running AI models in production. A single server loads models from a model repository and serves them from TensorRT, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and other backends, on NVIDIA GPUs, x86 and ARM CPUs, or AWS Inferentia, across cloud, data center, edge, and embedded deployments. It handles real-time, batched, ensemble, and audio/video streaming query patterns and is part of the NVIDIA AI Enterprise suite.
You place one or more models in a model repository, optionally add a model configuration to enable scheduling, batching, and instance settings, and start the server from the NGC container. Clients send requests over HTTP/REST or gRPC using the KServe v2 protocol, through the Python, C++, and Java client libraries, or through an OpenAI-compatible API for LLM deployments. For in-process use on edge devices, C and Java APIs link Triton directly into an application.
Multiple models, or multiple instances of the same model, execute concurrently on the same GPU. The dynamic batcher groups individual requests into batches, and the sequence batcher with implicit state management serves stateful models. Model pipelines chain models through Ensembles or Business Logic Scripting (BLS) in the Python backend, and decoupled models can return multiple responses or none per request. A backend API lets you write custom backends in C/C++ or Python, and repository agents run logic such as authentication, decryption, or conversion at model load time. Metrics report GPU utilization, throughput, and latency; Model Analyzer and Performance Analyzer profile and tune configurations.
Features
- Multi-framework backends: TensorRT, PyTorch, ONNX Runtime, OpenVINO, Python, RAPIDS FIL, TensorRT-LLM, and vLLM
- Hardware targets: NVIDIA GPUs, x86 and ARM CPUs, AWS Inferentia, and Jetson/JetPack
- Concurrent execution: several models or model instances run in parallel on one GPU
- Dynamic and sequence batching: request batching for throughput and stateful serving with implicit state management
- Model pipelines: Ensemble graphs and Business Logic Scripting for pre- and post-processing
- Protocols: HTTP/REST and gRPC (KServe v2), an OpenAI-compatible API, and binary tensor, shared-memory, and other protocol extensions
- Client libraries: Python tritonclient, C++, and Java, plus in-process C and Java APIs
- Custom backends: a backend API for C/C++ or Python-based backends and decoupled models
- Model management: explicit load and unload control and repository agents that run at load time
- LLM features: constrained decoding, function calling, and speculative decoding with TensorRT-LLM and vLLM
- Observability: GPU, throughput, and latency metrics, tracing, and response caching
- Deployment: NGC containers, Kubernetes and Helm examples for GCP, AWS, and FleetCommand, and multi-node scaling guides
