Surogate Trainer is a training and fine-tuning engine for open language models, built as a native C++/CUDA runtime and driven from a command-line interface with a YAML config. It covers pre-training, full fine-tuning, LoRA, and reinforcement learning with GRPO and DPO, on single-GPU, multi-GPU, and GPU-plus-CPU setups.
Precision is the main axis of configuration. Pre-built recipes cover BF16 for maximum numerical accuracy, native FP8 using E4M3 for activations and weights and E5M2 for gradients with per-tensor delayed scaling, and native NVFP4 (CUTLASS FP4 E2M1) with two-level block scaling, stochastic rounding, and random Hadamard transforms on Blackwell hardware. QLoRA variants quantize online in FP8, NVFP4, or bitsandbytes formats, or load pre-quantized weights, and mixed-precision configs set separate dtypes for GEMMs, model, gradients, and LoRA.
Native CPU offloading moves weights, gradients, activations, and quants off the GPU so models can be fine-tuned at bf16 precision. Multi-node distributed data-parallel training runs through Ray, adaptive training adds automatic phase detection, multi-criteria early stopping, learning-rate management, Chinchilla token budgeting, and dynamic epoch adjustment, and mixture-of-experts training adds expert parallelism, least-loaded load balancing, and imbalance detection.
Supported architectures include Qwen3, Qwen3VL, Qwen3 MoE, Qwen3.5 and Qwen3.6, Gemma 4, Nemotron Nano, Super, and Cascade, GPT-OSS, Laguna, LFM2, and Llama 3.1 and 3.2. The engine runs on Linux x86_64 with an NVIDIA GPU from SM80 through SM121, CUDA 12.8, 12.9, or 13, NCCL, and cuDNN; a Python domain-specific language with ahead-of-time auto-differentiation is provided for adding new model architectures.
The same project also offers a hosted platform for building and running agents: an agent is designed from a model, knowledge bases, tools, skills, and guardrails, deployed as a managed cloud service on channels such as a hosted chat page, Slack, Telegram, WhatsApp, a site widget, or an API pipeline. Sessions are recorded and replayable, approvals wait in an inbox, tool access and network egress are locked when a session starts, and an agent can be published for paid access.
Features
- Training modes: pre-training, full fine-tuning, LoRA, and stacked LoRA on top of an existing adapter
- Reinforcement learning: GRPO and DPO in BF16, FP8, and NVFP4, with deterministic custom environments
- Precision recipes: BF16 baseline, FP8 hybrid with delayed scaling, and NVFP4 with two-level block scaling on SM100+
- QLoRA options: online FP8, NVFP4, or bitsandbytes quantization, or pre-quantized weight loading
- CPU offloading: weights, gradients, activations, and quants offloaded to host memory
- Distributed training: multi-threaded multi-GPU backend and multi-node DDP through Ray
- Adaptive training: phase detection, early stopping on convergence, efficiency, divergence, or plateau, and Chinchilla token budgeting
- MoE support: expert parallelism, least-loaded balancing, training metrics, and imbalance detection
- Optimizers: AdamW 8-bit and NorMuon
- Python DSL: ahead-of-time auto-differentiation for adding model architectures
- Broad GPU coverage: SM80 through SM121, with CUDA 12.8, 12.9, and 13 container images
