Microsoftbitnet.cpp is the inference framework for 1-bit language models such as BitNet b1.58, whose weights are ternary. It provides optimised kernels for running those models on CPUs and GPUs, with NPU support listed as next, and it targets local devices rather than accelerator clusters.
Reported measurements put speedups at 1.37x to 5.07x on ARM CPUs with energy consumption down 55.4% to 70.0%, and 2.37x to 6.17x on x86 CPUs with energy down 71.9% to 82.2%, with larger models gaining more. A 100B BitNet b1.58 model runs on a single CPU at 5 to 7 tokens per second.
Three official models are published: BitNet-b1.58-2B-4T, a 2.4B-parameter ternary model trained on 4 trillion tokens with a conversational mode and a GPU inference kernel; and two 1-bit embedding models, BitNet-embedding-0.6B and BitNet-embedding-270M, reporting prefill speedups over F16 of 1.42x to 2.28x and 1.32x to 1.74x respectively at 8 threads on x86, at 2 bits per weight. Community models include bitnetb158-large and -3B, Llama3-8B-1.58-100B-tokens, and the Falcon3 and Falcon-E families, each with its own matrix of supported I2_S, TL1, and TL2 kernels per CPU architecture.
Running a model means converting a checkpoint to a quantised GGUF file and calling the inference script with a prompt, thread count, context size, temperature, and an optional chat mode; a benchmark script measures end-to-end throughput for a given model and token count. A setup script downloads a supported Hugging Face repository and applies i2_s or tl1 quantisation, and a helper converts .safetensors checkpoints. Builds need Python 3.10 or later, CMake 3.22 or later, and clang 18 or later.
The project is based on the llama.cpp framework, and its kernels build on the lookup-table methods from T-MAC, which the project points to for general low-bit models beyond ternary ones.
Features
- 1-bit inference: optimised kernels run ternary BitNet b1.58 models without dequantising to full precision
- CPU kernels: I2_S, TL1, and TL2 kernels, with support varying by model and by x86 or ARM architecture
- GPU kernel: an official GPU inference kernel extends 1-bit inference beyond CPUs
- Embedding models: two 1-bit embedding models at 0.6B and 270M parameters with I2_S conversion on x86
- Community model support: ternary conversions of Llama 3 8B and the Falcon3 and Falcon-E families run through the same setup script
- Checkpoint conversion: helpers convert .safetensors checkpoints and apply i2_s or tl1 quantisation to GGUF
- Chat mode: the inference script accepts a system prompt and runs an interactive conversation
- Benchmark script: an end-to-end benchmark reports throughput for a model, prompt length, and thread count
- Parallel kernels: configurable tiling and embedding quantisation add a further 1.15x to 2.1x over the original implementation
