PrimeRL is a framework for large-scale reinforcement learning (RL) post-training of language models. RL runs fully asynchronously for high-throughput agentic training, and the same codebase covers supervised fine-tuning (SFT), RL, and evals. It scales from a single GPU to clusters of more than 1,000 GPUs and is built to train mixture-of-experts (MoE) models with over a trillion parameters.
Training uses PyTorch FSDP2 and inference uses vLLM, with FP8 inference, prefill/decode disaggregation, expert parallelism (EP), and context parallelism (CP). An orchestrator connects the inference servers and the trainer. Environments come from the verifiers library through the Environments Hub, including SWE and agentic environments, and multi-node runs deploy through Slurm.
The trainer runs PrimeRL's own model implementations, so other architectures fail at setup. Supported families are dense Llama, Qwen3 and Qwen3.5, plus these MoE families:
| Family | Example model | EP | CP |
|---|---|---|---|
| GLM-5 | zai-org/GLM-5 | yes | yes |
| Qwen3 MoE | Qwen/Qwen3-30B-A3B | yes | yes |
| Qwen3.5 MoE and VLMs | Qwen/Qwen3.5-35B-A3B | yes (MoE) | yes |
| Poolside Laguna | poolside/Laguna-XS.2 | yes | yes |
| MiniMax M2 | MiniMax/MiniMax-M2 | yes | yes |
| Nemotron H | nvidia/Nemotron-3-Nano-30B-A3B | yes | yes |
| Trinity | arcee-ai/Trinity-Mini | yes | yes |
| GLM-4, GLM-4.5, INTELLECT-3 | zai-org/GLM-4.5-Air | yes | yes |
| GPT-OSS | gpt-oss-20b | yes | yes |
Worked examples run from one consumer GPU (training a small Qwen3 model to reverse text) up to Slurm clusters of 32 to 2,048 GPUs (Qwen3-30B-A3B on math, SWE, and tool use, GLM-4.5-Air on search and terminal tasks, and GLM-5 with FP8). At least one NVIDIA GPU is required.
Features
- Asynchronous RL: off-policy training decouples inference from the trainer for high throughput
- End-to-end post-training: SFT, RL, and standalone or online evals in one framework
- Scale: FSDP2 training with EP and CP, FP8 inference, and prefill/decode disaggregation, from one GPU to 1,000+
- vLLM inference: rollouts served by vLLM, with the llm-d router and Mooncake KV offload for large deployments
- Environments: native integration with verifiers environments from the Environments Hub, including SWE and agentic tasks
- Multimodal: trains vision-language models such as Qwen3.5
- LoRA: multi-turn RL with LoRA adapters, without an SFT warmup
- Algorithms: the AIPO loss, advantage and filter plugins, and trajectory merging
- Slurm deployment: multi-node training on Slurm clusters
- Experiment tracking: logs runs to Weights & Biases
