Miles is a reinforcement-learning framework for post-training large language and vision-language models at cluster scale. It was forked from slime and pairs SGLang for rollout generation with Megatron-LM for training; a PyTorch FSDP2 backend is available for training a HuggingFace implementation as-is, while the recipes, parallelism strategies, and largest models target Megatron-LM.
Rollout and training workers are decoupled, so the loop runs fully asynchronously with configurable on- and off-policy schedules and separate async rollout and eval modes. Generation sits behind a router that spreads requests across SGLang engines, preserves per-request metadata, and health-checks the fleet. Updated weights reach the engines in seconds even at trillion-parameter scale, with peer-to-peer RDMA transfer as the fast path for disaggregated deployments.
Several features address numerical and structural correctness during long runs. Token-in-token-out keeps rollout and training on the same token sequence with no detokenize and retokenize round-trip, and applies to every model and black-box harness. Rollout Routing Replay records mixture-of-experts routing during rollout and replays it in the trainer's forward pass, removing the routing mismatch between the two, with compute and communication overlapped. When an SGLang engine dies, the run recovers it in place rather than restarting.
Training precision covers MXFP8 and NVFP4 with a recipe intended to limit precision-induced divergence, alongside FP8, INT4 quantization-aware training, BF16, and FP16. LoRA and multi-LoRA adapters train frontier-scale models on fewer GPUs and load directly into SGLang for rollout. Algorithm coverage includes GRPO, GSPO, PPO, and REINFORCE++, plus supervised fine-tuning and on-policy distillation; a companion project, Miles-diffusion, adds Flow-GRPO, DiffusionNFT, and SFT on a diffusion rollout engine.
Day-0 model support is documented for DeepSeek-V4, Kimi-K3, GLM-5.2, Inkling, and Nemotron, with Kimi-K2.6 and Qwen3.5 among the other supported models. Hardware coverage spans NVIDIA GB300, GB200, B300, B200, H200, H100, and A100, and AMD MI300X, MI325, MI350, and MI355X through ROCm. Agentic training connects to environments such as Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers, with task sandboxes on AgentENV, Daytona, E2B, or Modal.
Features
- Fully async RL: decoupled rollout and training workers with configurable on- and off-policy schedules and custom async rollout and eval modes
- SGLang rollout router: requests are spread across engines with per-request metadata preserved and fleet health checks
- Fast weight updates: in-loop weight transfer in seconds at trillion-parameter scale, with P2P RDMA for disaggregated setups
- Low-precision training: MXFP8 and NVFP4 recipes plus FP8, INT4 QAT, BF16, and FP16
- LoRA and multi-LoRA: adapters train large models on fewer GPUs and load straight into SGLang for rollout
- Token-in-token-out: no detokenize and retokenize round-trip between rollout and training, for every model and black-box harness
- Rollout Routing Replay: expert routing captured during rollout is replayed in the trainer's forward pass to remove MoE routing mismatch
- Fault tolerance: a failed SGLang engine is recovered and the run resumes without a restart or pause
- Algorithm coverage: GRPO, GSPO, PPO, and REINFORCE++, plus SFT and on-policy distillation
- Agentic environments: connectors for Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers with sandboxes on AgentENV, Daytona, E2B, or Modal
- Hardware breadth: NVIDIA GB300 through A100 and AMD Instinct MI300X through MI355X via ROCm
- Diffusion training: Flow-GRPO, DiffusionNFT, and SFT on an sglang-diffusion rollout engine in Miles-diffusion
