LM Evaluation Harness is EleutherAI's framework for testing generative language models against a large set of evaluation tasks: over 60 standard academic benchmarks with hundreds of subtasks and variants. It uses publicly available prompts so that results stay reproducible and comparable between papers, and it is the backend for the Hugging Face Open LLM Leaderboard.
The core package is the evaluation framework, and model backends are added separately. Models can come from Hugging Face transformers (including GGUF and GPTQ-quantized weights, PEFT adapters, and steered models with steering vectors applied), vLLM, SGLang, NVIDIA NeMo, Megatron-LM, ONNX Runtime, OpenVINO, and other runtimes, or from hosted APIs and any server that mirrors the OpenAI Completions or ChatCompletions API. Models that do not return logprobs can run only generation tasks, while local models and logprob-returning APIs can run loglikelihood, perplexity, and multiple-choice tasks too.
Tasks are defined in YAML configs with Jinja2 prompt templates, and model backends, filters, metrics, and aggregations can be registered from a separate package as plugins instead of forking. A command-line interface with run, list, and validate subcommands sits alongside a Python API for programmatic use.
| Backend family | Examples | Request types |
|---|---|---|
| Local weights | Hugging Face transformers, vLLM, SGLang, NeMo, Megatron-LM, Mamba | generate, loglikelihood, rolling loglikelihood |
| ONNX and Windows | ONNX Runtime GenAI, ONNX Runtime, Windows ML | loglikelihood (plus generate on GenAI and Windows ML) |
| Hosted APIs | OpenAI Completions, OpenAI ChatCompletions, Anthropic, LiteLLM, TextSynth, Watsonx.ai | generate only for chat APIs |
| Local servers | llama.cpp server, OpenAI-compatible servers | generate, loglikelihood |
Features
- Benchmark library: over 60 academic benchmarks with hundreds of subtasks and variants, including the Open LLM Leaderboard task group
- Multi-backend models: Hugging Face transformers, vLLM, SGLang, NeMo, Megatron-LM, ONNX Runtime, OpenVINO, and AWS Neuron behind one tokenization-agnostic interface
- API and server evaluation: OpenAI, Anthropic, LiteLLM, and any OpenAI-compatible or llama.cpp server, with batched and async requests
- Multi-GPU runs: data-parallel and model-parallel evaluation through accelerate, native PyTorch tensor parallelism, and vLLM tensor and data parallelism
- Adapters and quantization: evaluates PEFT adapters such as LoRA, delta weights, GPTQ models, and GGUF files
- Config-based tasks: YAML task configs with Jinja2 prompts, configurable few-shot settings, answer extraction, and output post-processing
- Plugins: register model backends, filters, metrics, and aggregations from your own package
- Result logging: saves results and per-sample outputs, caches runs for resumption, and pushes results to the Hugging Face Hub
- Visualization: sends results to Weights and Biases or Zeno for analysis
- Predict-only mode: collects decoded generations for tasks that need post-hoc scoring outside the harness
