OpenLLM runs open-source or custom large language models as OpenAI-compatible API servers from one command: the server can be called by any OpenAI client library or OpenAI-compatible tool (the OpenAI Python client, LlamaIndex, and others). The same server exposes a chat UI at /chat, and a run command opens a chat session in the terminal.
Models come from model repositories: a default repository lists the current open-source models as prebuilt Bentos, and you can register additional public repositories that follow the same layout to serve your own models packaged with BentoML. OpenLLM does not store model weights itself, and gated Hugging Face models require a Hugging Face token.
Deployment beyond a local machine goes through BentoML and BentoCloud: a deploy command pushes a model to BentoCloud's managed inference infrastructure with autoscaling and observability, and the same Bento format supports Docker and Kubernetes deployments. Inference runs on vLLM as the backend.
Selected models from the default repository and their GPU requirements:
| Model | Variant | Required GPU |
|---|---|---|
| deepseek | r1-671b | 80G x16 |
| gemma3 | 3b | 12G |
| llama3.1 | 8b | 24G |
| llama3.3 | 70b | 80G x2 |
| llama4 | 17b16e | 80G x8 |
| mistral | 8b-2410 | 24G |
| mistral-large | 123b-2407 | 80G x4 |
| phi4 | 14b | 80G |
| pixtral | 12b-2409 | 80G |
| qwen2.5 | 7b | 24G |
| qwq | 32b | 80G |
Features
- OpenAI-compatible server: chat completion endpoints with an optional API key
- Chat UI: a browser chat interface at /chat on the running server
- CLI chat: a run command starts an interactive conversation in the terminal
- Model repositories: openllm model list, openllm repo update, and openllm model get manage the catalog of runnable models
- Custom models: register your own public repository of Bentos built with BentoML
- vLLM backend: inference is served by vLLM
- BentoCloud deployment: a deploy command pushes a model to managed infrastructure with autoscaling and observability
- Container deployment: the Bento packaging also targets Docker and Kubernetes
