OpenAIHumanEval is the evaluation harness for the HumanEval problem-solving dataset introduced in the paper "Evaluating Large Language Models Trained on Code". It measures how often a code model's generated completions pass hand-written unit tests, reported as pass@k.
A model's completions are written to a JSON Lines file with one task ID and completion per line, and the evaluation command runs each completion against its test suite and writes a results file that records whether every sample passed, timed out, or failed, alongside aggregate pass@1, pass@10, and pass@100 figures. Because there is no unbiased estimate of pass@k with fewer samples than k, those cases are skipped, and other k values can be requested.
The harness exists to execute untrusted model-generated code, and the execution call is deliberately disabled until the user reads the security disclaimer and enables it, with running inside a security sandbox strongly encouraged.
Features
- pass@k scoring: functional-correctness evaluation of code completions against hand-written test suites, reported as pass@1, pass@10, and pass@100
- JSON Lines interface: completions go in as one task ID plus completion per line, and a per-sample results file records passed, timed out, or failed
- Gated execution: the call that runs model-generated code is disabled by default and must be enabled deliberately after reading the sandboxing disclaimer
- Example data: sample problems and solutions ship with the harness for format checks and debugging
