The AI Scientist

The AI Scientist runs a machine-learning study end to end with a large language model (LLM) in the loop. Starting from a template, it generates research ideas, checks them against published literature for novelty, runs the experiments, plots the results, writes a LaTeX paper, and can then produce an LLM review of that paper.

Work is organized around templates. Each one is a directory holding experiment.py for the study itself, plot.py for figures, prompt.json describing the domain, seed_ideas.json with example ideas, and a LaTeX template for the write-up. Three templates accompany the published work: NanoGPT for transformer next-token prediction, 2D Diffusion for diffusion models on low-dimensional data, and Grokking for generalization and learning speed. Community-contributed templates cover further domains such as infectious-disease modeling, image classification, quantum chemistry, earthquake prediction, and LLM probes.

A run starts with launch_scientist.py, which takes a base model, a template, and a number of ideas, and can spread those ideas across multiple GPUs with --parallel. A baseline run_0 is produced on each machine first so run-time comparisons hold on that hardware. Novelty checks and citation collection use the Semantic Scholar API, with OpenAlex as a key-free alternative and the option to skip those phases entirely.

Reviewing is a separate entry point. perform_review loads a paper as text and returns a dictionary with an overall score from 1 to 10, an accept or reject decision, and a list of weaknesses, with settings for reflections, few-shot examples, and ensembling. A batch script runs the same review across a set of ICLR submissions.

The code targets Linux with NVIDIA GPUs using CUDA and PyTorch, and needs a full LaTeX installation for the write-up. Because it executes code the model writes, the guidance is to containerize runs and restrict web access, and a community-contributed Dockerfile is included for that. Per-paper cost is put at typically under $15 with Claude Sonnet 3.5, success rates vary by template, base model, and idea complexity, and reviews are recommended with GPT-4o because other models showed positivity bias or output-format failures.