KTransformers is a research framework for running and fine-tuning large language models on mixed CPU and GPU hardware. It targets large mixture-of-experts (MoE) models that do not fit in GPU memory alone: frequently used experts stay on the GPU while the rest run on the CPU, so models such as DeepSeek-V3 and R1, Kimi K2, GLM, MiniMax, and Qwen3 MoE can run on consumer or workstation GPUs backed by large system memory.
It exposes two capabilities. Inference comes from kt-kernel, a set of CPU kernels for Intel AMX and AVX512 or AVX2 with INT4 and INT8 quantized weights, NUMA-aware memory management for MoE layers, and GPU-side GPTQ support, with a Python API used for serving through SGLang. Fine-tuning (SFT) integrates with LlamaFactory for LoRA and full-parameter training of very large MoE models on limited GPU memory, in BF16, FP8, and INT8.
Hardware support extends beyond NVIDIA GPUs to AMD GPUs through ROCm, Intel Arc GPUs, and Ascend NPUs, plus AMD servers with AVX512. Other additions include long-context support, multi-concurrency serving, and a three-layer prefix cache that reuses prefixes across GPU, CPU, and disk.
| Workload | Model | Hardware | Result |
|---|---|---|---|
| Inference | DeepSeek-R1-0528 (FP8) | 8×L20 GPU + Xeon Gold 6454S | 227.85 tokens/s total, 87.58 tokens/s output (8-way concurrency) |
| Fine-tuning | DeepSeek-V3 | 4x RTX 4090, ~80GB GPU memory | 3.7 it/s |
| Fine-tuning | DeepSeek-R1 | 4x RTX 4090, ~80GB GPU memory | 3.7 it/s |
| Fine-tuning | Qwen3-30B-A3B | 1x RTX 4090, ~24GB GPU memory | 8+ it/s |
Features
- CPU-GPU expert placement: hot MoE experts run on the GPU and cold experts on the CPU, with CPU-GPU expert scheduling
- CPU kernels: Intel AMX and AVX512 or AVX2 kernels for INT4 and INT8 quantized inference
- Precision options: native BF16 and FP8 per-channel precision, GPU-side GPTQ, and hybrid low-bit weights
- SGLang serving: a Python API that plugs kt-kernel into SGLang for serving
- MoE fine-tuning: LoRA and full-parameter training through LlamaFactory, with 6 to 12 times the speed of ZeRO-Offload in benchmarked MoE SFT workloads
- RL-DPO fine-tuning: preference fine-tuning through LlamaFactory
- Hardware coverage: NVIDIA GPUs, AMD GPUs through ROCm, Intel Arc GPUs, Ascend NPUs, and AVX512 AMD servers
- Prefix cache: three-layer prefix reuse across GPU, CPU, and disk
- Model support: DeepSeek V3, R1, and V4-Flash, Kimi K2 and K2.5, GLM-5, MiniMax-M2 and M3, Qwen3 MoE and Qwen3-Next, and Mixtral
