LlamaFactory is a training and fine-tuning platform for open large language models. You pick a model, a dataset and a training approach in a YAML config, a command line or the LLaMA Board web UI, and it runs the job without any training code. It covers the whole post-training path from continued pre-training through supervised fine-tuning, reward modeling and preference optimization (PPO, DPO, KTO, ORPO, SimPO), and it supports multimodal models for image, video and audio understanding as well as text tasks such as multi-turn dialogue and tool use.
Training can run as 16-bit full-parameter tuning, freeze-tuning, LoRA, OFT, or 2 to 8-bit QLoRA and QOFT through AQLM, AWQ, GPTQ, LLM.int8, HQQ or EETQ quantization, on NVIDIA GPUs, AMD GPUs or Ascend NPUs. Optimizers and tricks include GaLore, BAdam, APOLLO, Adam-mini, Muon, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ, PiSSA, FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune and rsLoRA. Experiment tracking goes to LlamaBoard, TensorBoard, Wandb, MLflow or SwanLab.
Supported model families include Llama 2 to 4, Qwen2 to Qwen3.6 (including VL, Audio and Omni variants), DeepSeek (LLM, MoE, V3 and R1 distills), Gemma 1 to 3, GLM-4 and GLM-4.5, Mistral, Mixtral, Ministral and Pixtral, Phi-3 and Phi-4, Granite 3 and 4, GPT-OSS, InternLM and InternVL, Kimi-VL, LLaVA, MiniCPM, MiniMax, MiMo, Falcon, Hunyuan, ERNIE-4.5, Seed, StarCoder 2, TeleChat and Yuan 2, with many new models supported on their release day. A fine-tuned model can be served through an OpenAI-style API, a Gradio chat UI or the CLI, with vLLM or SGLang as the inference backend, and checkpoints can be exported with an Ollama modelfile.
Estimated memory per method and model size:
| Method | Bits | 7B | 14B | 30B | 70B |
|---|---|---|---|---|---|
| Full (bf16 or fp16) | 32 | 120 GB | 240 GB | 600 GB | 1200 GB |
| Full (pure_bf16) | 16 | 60 GB | 120 GB | 300 GB | 600 GB |
| Freeze/LoRA/GaLore/APOLLO/BAdam/OFT | 16 | 16 GB | 32 GB | 64 GB | 160 GB |
| QLoRA / QOFT | 8 | 10 GB | 20 GB | 40 GB | 80 GB |
| QLoRA / QOFT | 4 | 6 GB | 12 GB | 24 GB | 48 GB |
| QLoRA / QOFT | 2 | 4 GB | 8 GB | 16 GB | 24 GB |
Features
- Zero-code training: a YAML config drives the CLI, and the LLaMA Board web UI (built on Gradio) exposes the same options in a browser
- Training approaches: continued pre-training, supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO and SimPO, each available with full, freeze, LoRA, QLoRA, OFT and QOFT tuning
- Quantized tuning: 2, 3, 4, 5, 6 and 8-bit QLoRA and QOFT via AQLM, AWQ, GPTQ, LLM.int8, HQQ and EETQ
- Optimizers and algorithms: GaLore, BAdam, APOLLO, Adam-mini, Muon, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA
- Speed-ups: FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune and rsLoRA
- Multimodal tasks: image understanding, visual grounding, video recognition and audio understanding alongside multi-turn dialogue and tool use
- Model coverage: more than 100 model families including Llama, Qwen, DeepSeek, Gemma, GLM, Mistral, Phi, Granite, GPT-OSS, InternLM, LLaVA and MiniCPM, with day-0 or day-1 support for new releases
- Datasets: bundled pre-training, supervised fine-tuning and preference datasets, plus custom datasets described in a dataset info file
- Experiment tracking: LlamaBoard, TensorBoard, Wandb, MLflow and SwanLab loggers
- Inference and serving: OpenAI-style API, Gradio chat UI and CLI with a vLLM or SGLang backend, and model export with an Ollama modelfile
- Hardware backends: NVIDIA CUDA, AMD ROCm and Ascend NPU, with Docker images for each
- Model hubs: downloads from Hugging Face, ModelScope or Modelers Hub
