Miles is a reinforcement learning (RL) framework for post-training large language and vision-language models, forked from slime. It pairs SGLang for high-throughput rollout with Megatron-LM for training at up to trillion-parameter scale, and also supports FSDP2 and torchtitan training backends.
Rollout and training workers are decoupled, so runs can be fully asynchronous with configurable on- and off-policy schedules. Generation runs on SGLang engines behind a router tuned for multi-turn agentic workloads, and new weights reach the engines in seconds through peer-to-peer RDMA transfer. Token-in-token-out keeps tokens identical between rollout and training, Rollout Routing Replay replays mixture-of-experts routing from rollout in the trainer, and a failed SGLang engine is recovered without restarting the run.
Recipes cover GRPO, GSPO, PPO, and REINFORCE++, plus SFT and on-policy distillation, with full-parameter or LoRA training. Models with day-0 support include DeepSeek-V4.1, Kimi-K3, Qwen3.8, GLM-5.3, MiMo-V2.6-Flash, and Nemotron. Agentic RL connects to environments such as Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers, with task sandboxes on AgentENV, Daytona, E2B, or Modal. A companion project, Miles-diffusion, applies the same approach to diffusion models.
Features
- Fully async RL: decoupled rollout and training workers with configurable on- and off-policy schedules
- Agentic rollout: SGLang generation behind a router that spreads requests across engines and health-checks them
- Fast weight updates: in-loop weight transfer in seconds, with P2P RDMA for disaggregated setups
- Low-precision training: MXFP8 and NVFP4 with a stable RL recipe, plus FP8, INT4 QAT, BF16, and FP16
- LoRA and multi-LoRA: adapters train frontier-scale models on fewer GPUs and load directly into SGLang for rollout
- Token-in-token-out: no detokenize and retokenize round-trip between rollout and training, for every model and black-box harness
- Rollout Routing Replay: replays MoE expert routing from rollout in the trainer forward pass
- Fault tolerance: recovers a failed SGLang engine and resumes the run in place
- Recipes: GRPO, GSPO, PPO, REINFORCE++, score centering, SFT, and on-policy distillation
- Hardware: NVIDIA GB300, GB200, B300, B200, H200, H100, and A100, and AMD MI355X, MI350X, MI325X, and MI300X
- Ray or Kubernetes: the same launch script runs on Ray or as a Helm release on Kubernetes
- Agentic environments: connectors for Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers, with sandboxes on AgentENV, Daytona, E2B, or Modal
