FastChat is a platform for training, serving, and evaluating chatbots built on large language models. It is the software behind Chatbot Arena (lmarena.ai), which serves side-by-side battles between 70+ LLMs and compiles human votes into an Elo leaderboard, and it ships the training and evaluation code for the Vicuna models and the MT-Bench benchmark.
Serving uses a distributed architecture with three components: a controller that coordinates workers, model workers that host one or more models, and web servers that face users. Multiple workers can register with one controller to serve a single model at higher throughput or several models at once, and a multi-tab Gradio server includes the Arena battle UI. API-based models from OpenAI, Anthropic, Gemini, Mistral, and others are registered through a JSON endpoint file.
An OpenAI-compatible REST API lets FastChat act as a local drop-in replacement for the OpenAI API, working with the openai-python library and cURL. A command-line interface runs inference on a single GPU, across multiple GPUs with model parallelism, on CPU only, on Apple silicon via Metal, on Intel XPU, and on Ascend NPU, with 8-bit compression and CPU offloading to reduce memory. Vicuna-7B needs about 14 GB of GPU memory and Vicuna-13B about 28 GB; 8-bit compression roughly halves that.
Evaluation uses MT-Bench, a set of multi-turn open-ended questions judged by prompting an LLM such as GPT-4. Fine-tuning code is based on Stanford Alpaca with added multi-turn conversation support; Vicuna was trained on about 125K user-shared ShareGPT conversations, and LoRA and SkyPilot cloud training are also covered.
| Model | Sizes |
|---|---|
| Vicuna v1.5 | 7B, 7B-16k, 13B, 13B-16k, 33B (v1.3) |
| LongChat | 7B with 32K context |
| FastChat-T5 | 3B |
Features
- Multi-model serving: controller, model workers, and Gradio web servers that scale to several models or several replicas
- OpenAI-compatible API: REST endpoints usable from openai-python and cURL as a local replacement
- Chatbot Arena UI: side-by-side battle interface with API-based models registered from a JSON file
- Supported models: Llama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, and more
- Hardware backends: CUDA, multi-GPU model parallelism, CPU (with AVX512_BF16/AMX), Metal, Intel XPU, and Ascend NPU
- Quantization: 8-bit loading, CPU offloading, ExLlama V2, GPTQ 4-bit, and AWQ 4-bit
- MT-Bench evaluation: multi-turn question set with LLM-as-judge scoring
- Fine-tuning: Alpaca-derived training scripts with FSDP, xFormers, LoRA, and SkyPilot options
- Integrations: Hugging Face generation API and LangChain
