Soup is a command-line tool for fine-tuning open LLMs from a single YAML config. The config names a base model, a task, a dataset, and training settings, and Soup handles LoRA setup, quantization, batch size, and GPU detection. It trains locally with QLoRA on a single GPU, on Apple Silicon through an MLX backend, or on CPU for testing, and it also covers what comes after training: chat with the result, batch inference, evaluation, merging adapters, export, and serving.
An opt-in layer-streaming mode (in beta) keeps the frozen base model out of GPU memory and feeds it to the GPU one decoder layer at a time. Measured on an RTX 3050 Laptop with 4 GB of VRAM, Llama-3.1-8B-Instruct with NF4 trained at 119.6 tokens per second with a 3.32 GB peak, bit-exact against a normal resident run, and an H100 reproduced the same 3.32 GB footprint. The method is described in a preprint.
Training tasks span supervised fine-tuning, preference and reinforcement methods (DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, BCO), tool calling, pre-training, distillation, embedding, vision, and audio. It works with any text-generation model on the Hugging Face Hub that loads as a causal LM, and 100+ ready-made recipes cover families such as Llama, Qwen, Gemma, Mistral, Mixtral, DeepSeek, and Phi-4.
| VRAM | Max model (QLoRA 4-bit) | Example |
|---|---|---|
| 8 GB | ~7B | Llama-3.1-8B, Mistral-7B |
| 16 GB | ~14B | Phi-4-14B, Qwen2.5-14B |
| 24 GB | ~34B | CodeLlama-34B, Yi-1.5-34B |
| 48 GB | ~70B | Llama-3.3-70B |
| 80 GB+ | 70B+ (full) or MoE | Mixtral-8x22B, DeepSeek-V3 |
Features
- One config: a YAML file defines base model, task, data, LoRA, and quantization, with unknown keys rejected at load
- Layer streaming: trains an 8B model on a 4 GB laptop GPU by streaming the frozen base one decoder layer at a time (beta)
- Training methods: SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, BCO, pre-training, distillation, embedding, vision, and audio
- PEFT options: LoRA, DoRA, LoRA+, rsLoRA, VeRA, PiSSA, GaLore, and long-context methods such as YaRN and LongLoRA
- Backends: transformers, Unsloth, and MLX for Apple Silicon, plus DeepSpeed and FSDP for multi-GPU
- Data formats: Alpaca, ShareGPT, ChatML, preference pairs, vision, audio, and plaintext, auto-detected from JSONL, JSON, CSV, Parquet, or TXT
- Templates and recipes: templates for chat, code, tool calling, reasoning, and more, plus 100+ model recipes
- Evaluation: benchmarks, eval-gated training, an Elo arena, calibration, and post-train probes
- Export: GGUF for Ollama and llama.cpp, plus ONNX, TensorRT, AWQ, GPTQ, and BitNet
- Serving: an OpenAI-compatible server, an Anthropic Messages endpoint, and batch inference
- Web UI: a local dashboard for experiments, training setup, live metrics, dataset exploration, and model chat
- Governance: model registry, adapter management, signing, BOM, attestations, audit log, and air-gap mode
