OpenLLM runs open-source or custom large language models as OpenAI-compatible API servers from one command: openllm serve starts a server on port 3000 that any OpenAI client library or OpenAI-compatible tool (the OpenAI Python client, LlamaIndex, and others) can call by pointing at http://localhost:3000/v1. The same server exposes a chat UI at /chat, and openllm run opens a chat session in the terminal.
Models come from model repositories: a default repository lists the current open-source models as prebuilt Bentos, and you can register additional public repositories that follow the same layout (openllm repo add ) to serve your own models packaged with BentoML. OpenLLM does not store model weights itself; gated Hugging Face models require an HF_TOKEN.
Deployment beyond a local machine goes through BentoML and BentoCloud: openllm deploy pushes a model to BentoCloud's managed inference infrastructure with autoscaling and observability, and the same Bento format supports Docker and Kubernetes deployments. Inference runs on vLLM as the backend.
Selected models from the default repository and their GPU requirements:
| Model | Variant | Required GPU |
|---|---|---|
| deepseek | r1-671b | 80G x16 |
| gemma3 | 3b | 12G |
| llama3.1 | 8b | 24G |
| llama3.3 | 70b | 80G x2 |
| llama4 | 17b16e | 80G x8 |
| mistral | 8b-2410 | 24G |
| mistral-large | 123b-2407 | 80G x4 |
| phi4 | 14b | 80G |
| pixtral | 12b-2409 | 80G |
| qwen2.5 | 7b | 24G |
| qwq | 32b | 80G |
Features
- OpenAI-compatible server: openllm serve exposes chat completion endpoints on localhost:3000 with an optional API key
- Chat UI: a browser chat interface at /chat on the running server
- CLI chat: openllm run starts an interactive conversation in the terminal
- Model repositories: openllm model list, openllm repo update, and openllm model get manage the catalogue of runnable models
- Custom models: register your own public repository of Bentos built with BentoML
- vLLM backend: inference is served by vLLM
- BentoCloud deployment: openllm deploy pushes a model to managed infrastructure with autoscaling and observability
- Container deployment: the Bento packaging also targets Docker and Kubernetes
