GuideLLM measures how language model deployments perform under realistic traffic, so teams can check service-level objectives (SLOs), tune serving configurations, and plan capacity. It sends simulated end-to-end requests to OpenAI-compatible and vLLM-native servers, generates load patterns that mirror production usage, and reports latency and token-level statistics.
Metrics include full distributions of time to first token (TTFT), inter-token latency (ITL), and end-to-end request behavior. Load profiles are synchronous, concurrent, throughput, constant rate, Poisson, and sweep, where a sweep finds the maximum sustainable rate and the safe operating range. Benchmarks are bounded by duration or request count.
Data comes from Hugging Face datasets, local JSON, CSV, or text files, or synthetic generators for text, images, and short videos, so you can benchmark text, vision-language, and audio serving without bringing a dataset. GuideLLM also replays Mooncake, WEKA, and OpenTelemetry trace files, benchmarks multi-turn conversations and tool calling, and targets completions, chat completions, embeddings, and audio transcription and translation endpoints.
Results print to the console and export as JSON, YAML, CSV, self-contained HTML reports, and static plots for dashboards and regression tracking. Built-in scenarios bundle schedules, dataset settings, and request formatting for common test patterns, and GuideLLM runs as a CLI, a Python API, or a container image for Docker, Podman, Kubernetes, and OpenShift.
| Tool | API | Full metrics | Data modalities | Profiles |
|---|---|---|---|---|
| GuideLLM | Yes | Yes | Text, image, audio, video | Synchronous, concurrent, throughput, constant, Poisson, sweep |
| inference-perf | No | No | Text | Concurrent, constant, Poisson, sweep |
| genai-bench | No | No | Text, image, embedding, rerank | Concurrent |
| llm-perf | No | No | Text | Concurrent |
| vllm/benchmarks | No | No | Text | Synchronous, throughput, constant, sweep |
Features
- SLO metrics: full distributions for time to first token, inter-token latency, and end-to-end latency
- Load profiles: synchronous, concurrent, throughput, constant, Poisson, and sweep traffic patterns
- Rate sweeps: reproducible sweeps that find maximum throughput and safe operating ranges
- Datasets: Hugging Face datasets, local files, and synthetic text, image, and video data
- Trace replay: replays Mooncake, WEKA, and OpenTelemetry trace files for realistic load
- Multi-turn and tool calling: benchmarks chat agents with multi-turn conversations and client- or server-side tool calls
- Endpoint coverage: completions, chat completions, responses, embeddings, and audio transcription and translation APIs
- Backends: OpenAI-compatible HTTP servers, the in-process vLLM Python API, and websocket audio transcription
- Reports: console summaries plus JSON, YAML, CSV, HTML, and plot outputs
- Scenarios: built-in scenario files that bundle schedules, data settings, and request formats
