rLLM trains language agents with reinforcement learning without rewriting them. You point it at an agent you already have, and the same agent code drives both evaluation and training: a model gateway sits in front of the agent's LLM calls, captures token IDs and log probabilities by URL-routed session, and structures them into episodes (one task), trajectories (one agent run), and steps (one LLM call). A reward function scores the result and the RL algorithm updates the model weights.
Agents come from two directions. Over ten command-line harnesses are supported directly, including Claude Code, Codex, Terminus-2, mini-swe-agent, and opencode, alongside Harbor-compatible task directories. Any other program — a LangGraph graph, an OpenAI Agents SDK agent, or a plain openai.OpenAI call — is wrapped with the @rllm.rollout decorator. Rollouts run in Docker, Daytona, Modal, or locally, with snapshotting and warm pools to keep training-scale rollouts cheap.
Training backends are selected with a single flag: verl for distributed multi-GPU runs with vLLM or SGLang, tinker for a single machine, and fireworks for the Fireworks platform. Supported methods include GRPO, REINFORCE, RLOO, supervised fine-tuning, and on-policy distillation. Over 60 benchmarks are integrated across maths, code, multiple choice, question answering, search, vision-language, translation, and agentic tasks — Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500, and GPQA among them — and rllm eval pulls and runs one.
There are two entry points: a CLI (rllm model setup, rllm eval, rllm train) that needs no code, and a Python API where you decorate a rollout function and an evaluator function and hand them to an AgentTrainer. The project reports open-source results for DeepScaleR-1.5B, DeepCoder-14B, DeepSWE-32B, and FinQA-4B, and lists community projects built on it, including Tongyi DeepResearch, Terminal-Bench-RL, PettingLLMs, and SETA. It requires Python 3.11 or later.
Features
- Any harness: 10+ CLI harnesses including Claude Code, Codex, Terminus-2, mini-swe-agent, and opencode, plus Harbor-compatible task directories
- Wrap your own agent: the @rllm.rollout decorator turns existing LangGraph, OpenAI Agents SDK, or raw OpenAI-client code into a trainable rollout
- Any sandbox: Docker, Daytona, Modal, or local execution, with snapshot and warm-pool acceleration
- Swappable backends: verl for distributed multi-GPU, tinker for single-machine, and fireworks, selected by one flag
- Training methods: GRPO, REINFORCE, RLOO, supervised fine-tuning, and on-policy distillation
- 60+ benchmarks: Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500, GPQA, and more, run with rllm eval
- Model gateway: captures token IDs and log probabilities behind the scenes so eval and training share one agent implementation
- CLI or Python API: rllm model setup, rllm eval, and rllm train, or a rollout plus evaluator handed to AgentTrainer
