Kaggle Benchmarks is a Python library for writing evaluations of AI models and agents. A task is a decorated Python function that prompts a model, checks the output with assertions, and records the exact inputs, outputs, and model interactions so a run can be reviewed or reproduced later.
Evaluations can go beyond string matching to cover code execution, tool use, multi-turn conversation, structured outputs, and image, audio, or video inputs. Tasks can also run over a whole dataset to produce aggregate metrics.
The library is designed to run inside a Kaggle notebook, where it comes preinstalled and models are provided through Kaggle's backend. Each run writes a task file and run files that Kaggle uses to build a benchmark entry and show results on a public leaderboard. Local development works through a proxy token, and the set of available models depends on the environment. A companion agent skill lets an AI coding agent write tasks for the library.
Features
- Task decorator: define an evaluation as a Python function with @kbench.task
- Multiple models: prompt and compare several LLMs from the same task code
- Structured and multimodal I/O: get dataclass or Pydantic objects back, and send image, audio, and video inputs
- Tool use: give models tools, including a built-in Python interpreter for code execution
- Assertions: built-in checks such as regex matching, plus custom assertions
- Dataset runs: evaluate over a pandas DataFrame for aggregate scores
- Leaderboards: task and run files feed public Kaggle benchmark leaderboards
- Reproducibility: captures the exact prompts, responses, and interactions of every run
