KTransformers is a research project for running and fine-tuning large language models across CPU and GPU together, aimed at mixture-of-experts models too large for the available GPU memory. It exposes two capabilities from the kt-kernel source tree: inference serving and supervised fine-tuning.
The inference side provides CPU-optimised kernels using Intel AMX and AVX512 or AVX2 for INT4 and INT8 quantised weights, NUMA-aware memory management for mixture-of-experts layers, GPTQ support on the GPU side, and heterogeneous expert placement that keeps hot experts on the GPU and cold experts on the CPU. A Python API integrates the kernels into SGLang and other serving frameworks. One reported configuration runs DeepSeek-R1-0528 in FP8 on eight L20 GPUs with a Xeon Gold 6454S at 227.85 tokens per second total throughput and 87.58 tokens per second output throughput at eight-way concurrency.
Fine-tuning runs through an integration with LLaMA-Factory covering LoRA and full-parameter training in BF16, FP8, and INT8, including block-FP8 LoRA that loads FP8 routed-expert weights straight from the checkpoint without materialising a BF16 copy. The project reports a 6-12x training speedup over ZeRO-Offload in benchmarked mixture-of-experts SFT workloads at about half the CPU memory of its previous path, with these figures:
| Model | GPU memory | Training speed | Hardware |
|---|---|---|---|
| DeepSeek-V3 | ~80 GB total | 3.7 it/s | 4x RTX 4090 |
| DeepSeek-R1 | ~80 GB total | 3.7 it/s | 4x RTX 4090 |
| Qwen3-30B-A3B | ~24 GB total | 8+ it/s | 1x RTX 4090 |
Hardware coverage extends beyond Intel CPUs and NVIDIA GPUs to AMD servers with AVX512 but no AMX, AMD GPUs through ROCm, Intel Arc GPUs, and Ascend NPUs with CPU expert offload. Model support has tracked releases from DeepSeek, GLM, Kimi, MiniMax, Qwen, and Llama, with GLM-5.3-flash adding 1M-token context and multimodal input on consumer GPUs.
Features
- CPU kernel acceleration: Intel AMX and AVX512 or AVX2 kernels for INT4 and INT8 quantised inference
- Mixture-of-experts scheduling: NUMA-aware memory management and hot-on-GPU, cold-on-CPU expert placement
- Serving integration: a Python API for SGLang and other frameworks, plus multi-concurrency serving
- Fine-tuning: LoRA and full-parameter training for very large mixture-of-experts models on limited GPU memory
- Precision options: BF16, FP8, and INT8 recipes, including block-FP8 LoRA loading FP8 expert weights directly
- Hardware coverage: Intel and AMD CPUs, NVIDIA GPUs, ROCm on AMD GPUs, Intel Arc, and Ascend NPUs
- Long context: 3-layer GPU, CPU, and disk prefix-cache reuse, and 1M-token context with GLM-5.3-flash
- Model tracking: day-zero support for releases from DeepSeek, GLM, Kimi, MiniMax, and Qwen
- Documentation set: separate inference and fine-tuning entry points, with a cookbook covering hardware checks, installation, resource planning, and troubleshooting
