Truss packages a machine-learning model's serving logic, weights, and dependencies into a deployable unit and pushes it to Baseten. You describe the deployment in a config.yaml and, where custom behaviour is needed, write the model code in Python; Truss handles containerization, dependency resolution, and GPU configuration, so no Dockerfile or Kubernetes configuration is involved.
Models from any Python serving framework are supported, including vLLM, SGLang, TensorRT-LLM, transformers, diffusers, PyTorch, and TensorFlow. truss init scaffolds a project with a config.yaml and a model/ directory, and truss push builds and deploys it. A push is a published deployment by default; --watch gives a development deployment with live reload, which engine-based TensorRT-LLM deployments do not support.
For engine-based deployments, the config alone is the specification: it names the model, selects the accelerator, and points at a Hugging Face checkpoint, and Baseten's engine builder compiles it with TensorRT-LLM. The documented example deploys Qwen 2.5 3B Instruct to an L4 GPU with FP8 quantization, an 8,192-token maximum sequence length, and two GPUs during the build phase, because quantization needs more memory at compile time than at inference.
Engine-based deployments serve an OpenAI-compatible endpoint, so code written against the OpenAI SDK works by pointing base_url at the deployment and authenticating with a Baseten API key. Truss also ships a JSON schema for config.yaml; projects created with truss init reference it automatically, giving autocompletion, hover documentation, and validation in editors that support the YAML language server.
Features
- Config-first deployment: a single config.yaml declares the model, hardware, and engine, with Python model code only when custom logic is needed
- Framework coverage: vLLM, SGLang, TensorRT-LLM, transformers, diffusers, PyTorch, and TensorFlow models all package the same way
- Managed build: containerization, dependency management, and GPU configuration are handled by the CLI and the platform
- Engine builder: trt_llm config compiles a Hugging Face checkpoint with TensorRT-LLM, including quantization type, sequence length, and tensor parallelism
- OpenAI-compatible endpoints: engine deployments accept OpenAI SDK calls by changing the base URL
- Development loop: --watch deployments support live reload for iterating without a rebuild cycle
- Deployment identity: each push returns a model ID, deployment ID, endpoint, and log URL
- Authentication options: API key paste, browser OAuth, or explicit --api-key and --browser flags
- Editor support: a published JSON schema gives config.yaml autocompletion, hover docs, and validation
- Production features: GPU selection, secrets, caching, and autoscaling on Baseten or your own infrastructure
