Triton Inference Server is inference serving software that deploys trained models from many deep learning and machine learning frameworks behind one server. It runs across cloud, data center, edge, and embedded devices, on NVIDIA GPUs, x86 and ARM CPUs, or AWS Inferentia, and handles real-time, batched, ensemble, and audio or video streaming queries.
Models live in a model repository, each with an optional model configuration that sets scheduling, batching, and instance parameters. Triton executes them through backends (execution engines such as TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL), and a Backend API lets you add custom backends in C, C++, or Python. Backend availability varies by platform.
Clients send requests over HTTP/REST or gRPC protocols based on the community KServe protocol, using Python, C++, or Java client libraries. A C API and a Java API also link Triton directly into an application for edge and other in-process use. Triton ships as Docker containers, with deployment examples for Kubernetes and Helm on GCP and AWS, and builds for Jetson. The Performance Analyzer and Model Analyzer tools profile models to tune their configuration.
Features
- Multiple frameworks: serves models from TensorRT, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and other backends
- Concurrent model execution: runs multiple models, or multiple instances of a model, at the same time
- Dynamic batching: combines individual requests into batches on the server
- Sequence batching: batches requests for stateful models, with implicit state management
- Model pipelines: chains models with ensembles or Business Logic Scripting (BLS)
- Custom backends: a Backend API for new execution engines and pre- or post-processing, written in C, C++, or Python
- Decoupled models: a model can send several responses, or none, for a single request
- Inference protocols: HTTP/REST and gRPC based on the KServe protocol, plus in-process C and Java APIs
- Repository agents: run logic such as authentication, decryption, or conversion when a model loads or unloads
- Model management: explicit loading and unloading of models while the server runs
- Metrics: GPU utilization, server throughput, server latency, and more
