Surogate puts LLM training and serving in one toolkit, with two native C++/CUDA engines for NVIDIA GPUs. You configure work in Python or YAML: pretrain or fully fine-tune a model, train LoRA or QLoRA adapters, run GRPO or DPO, distill a teacher into a student, then merge, quantize, and serve the result over OpenAI- and Anthropic-compatible HTTP APIs.
The training engine compiles training graphs, uses fused kernels and low-precision recipes (BF16, hybrid FP8, and Blackwell NVFP4), and scales from one workstation to multi-GPU and multi-node clusters through Ray. CPU offload and pipeline streaming let it train models larger than one card's memory, including on PCIe systems without NVLink.
The serving engine handles continuous batching with up to 128 active sequences per model, prefix caching, speculative decoding, native GGUF loading, runtime LoRA adapters selected per request, and several models on one GPU with sleep and wake. Large mixture-of-experts models run across multiple GPUs or with experts offloaded to system RAM.
Selected published throughput comparisons:
| Workload | Hardware | Surogate | Baseline | Gain |
|---|---|---|---|---|
| Qwen3.5-0.8B serving, 1 user | 1× RTX 5090 | 802 tok/s | vLLM 346 tok/s | 2.32× |
| Qwen3.5-0.8B serving, 8 users | 1× RTX 5090 | 2,765 tok/s | llama.cpp 683 tok/s | 4.05× |
| GLM-5.3-Flash serving, 16 users | 8× RTX 5090 | 292.9 tok/s | llama.cpp 41.9 tok/s | 7.0× |
| Qwen3-0.6B LoRA training, BF16 | 1× H100 | 53,900 tok/s | Unsloth 21,300 tok/s | 2.53× |
| Qwen3-8B LoRA training | 1× RTX 5090 | 6,900 tok/s (FP4) | Unsloth 3,500 tok/s (BF16) | 1.97× |
Some rows in the full benchmark set favor vLLM, mainly prefill and time to first token on H100.
Features
- Training methods: pretraining, full fine-tuning, LoRA, QLoRA, GRPO reinforcement learning, DPO, and knowledge distillation
- Precision recipes: BF16, hybrid FP8, NVFP4, and BnB/NF4 quantization with configurable model, gradient, and adapter precision
- Multi-GPU training: data parallelism, ZeRO sharding, MoE expert parallelism, and Ray for multi-node runs
- Memory control: CPU offload for weights, gradients, optimizer state, and activations, plus recomputation for long contexts
- Compatible APIs: OpenAI Chat Completions, Completions, and Responses, and Anthropic Messages, with streaming, reasoning, and tool-call parsing
- Concurrent serving: continuous batching, chunked prefill, CUDA graphs, and prefix caching
- Speculative decoding: MTP and DFlash drafters
- Model sources: native GGUF, Hugging Face repos, and local safetensors in BF16, FP8, or NVFP4
- Runtime LoRA: load and unload PEFT adapters without restarting and pick one per request
- Multi-model hosting: several named models on one GPU with priorities and sleep to system RAM
- Operations: API-key authentication, health checks, Prometheus metrics, and request logs
