BentoML is a Python library for building online serving systems for AI applications and model inference. You wrap a model inference script in a class decorated with @bentoml.service, declare endpoints with @bentoml.api and standard Python type hints, and bentoml serve exposes them as an HTTP server on port 3000. The same code can compose several models into a multi-stage pipeline or an inference graph.
Packaging and deployment follow one path: bentoml build bundles code, models, and dependency configuration into a Bento, the standard deployable artifact; bentoml containerize turns a Bento into a Docker image; and bentoml deploy pushes it to BentoCloud, the hosted platform for scaling and operating BentoML services. Environments, dependencies, and model versions are declared in a config file or in the image= argument of the service decorator.
BentoML works with any ML framework, modality, or inference runtime, and reference examples cover LLMs on vLLM (Llama 3.2, Mistral, DeepSeek distillations), image generation (Stable Diffusion 3, SDXL Turbo, ControlNet, LCM LoRAs), embeddings (SentenceTransformers, ColPali), audio (ChatTTS, XTTS, WhisperX, Bark), computer vision (YOLO, ResNet), and agent stacks (function calling, LangGraph, CrewAI). Python 3.9 or later is required.
Features
- Type-hinted APIs: @bentoml.api endpoints derive request and response schemas from Python type hints
- Adaptive batching: batchable=True endpoints group concurrent requests for higher throughput
- Model parallelism and workers: multiple worker processes and GPU assignment per service
- Model composition: multi-model inference graphs and distributed services that call one another
- Bento artifact: bentoml build packages code, models, and dependencies into a reproducible unit
- Docker generation: bentoml containerize produces a container image from a Bento
- Client library: bentoml.SyncHTTPClient calls endpoints by method name from Python
- Model store: versioned model loading and management
- Observability: built-in metrics and tracing hooks
- BentoCloud deployment: bentoml deploy with concurrency-based autoscaling
- Opt-out telemetry: anonymous usage tracking disabled with --do-not-track or BENTOMLDONOT_TRACK=True