OpenAIEvals is a framework from OpenAI for evaluating large language models and systems built on them. It includes a registry of existing evals that test different dimensions of OpenAI models, and it lets you write custom evals for your own use cases, including private evals built from your own data without publishing that data.
Evals run against the OpenAI API (an API key is required and API usage is billed). The registry data is stored with Git LFS and can be fetched in full or per eval. Basic and model-graded evals need no evaluation code: you supply data in JSON and parameters in YAML following an eval template. For prompt chains or tool-using agents, the Completion Function Protocol wraps more complex systems so they can be evaluated the same way. Results can optionally be logged to a Snowflake database, and evals can also be configured and run in the OpenAI dashboard or with Weights & Biases.
Features
- Eval registry: existing evals covering different dimensions of OpenAI model behavior
- Custom evals: write evals for your use cases, including private evals from your own data
- Eval templates: basic and model-graded templates driven by JSON data and YAML parameters, without custom code
- Completion Function Protocol: evaluates prompt chains and tool-using agents
- Result logging: optional logging of eval results to Snowflake
- Dashboard and W&B support: run evals in the OpenAI dashboard or with Weights & Biases
- Contribution path: model-graded evals with custom YAML can be submitted to the registry
