bitsandbytes is a PyTorch library for k-bit quantization that cuts the memory needed to run and train large language models. It provides quantized linear layers for 8-bit and 4-bit operations and a set of 8-bit optimizers, and it is the quantization backend used by Hugging Face Transformers, Diffusers, and PEFT.
It has three main features. LLM.int8() runs inference with about half the memory by quantizing most features to 8 bits with vector-wise quantization and handling outlier features separately with 16-bit matrix multiplication. QLoRA quantizes a model to 4 bits and inserts a small set of trainable low-rank adaptation (LoRA) weights so the model can be fine-tuned in that reduced footprint. The 8-bit optimizers use block-wise quantization to keep optimizer state at a fraction of the 32-bit memory cost. Each method has its own published paper.
Hardware support on the development branch spans Linux, Windows, and macOS:
| Accelerator | LLM.int8() | QLoRA 4-bit | 8-bit optimizers |
|---|---|---|---|
| x86-64 and ARM CPU | Supported | Supported | Supported |
| NVIDIA GPU (CUDA) | Supported | Supported | Supported |
| AMD GPU (CDNA and RDNA) | Supported | Supported | Supported |
| Intel GPU (XPU) | Supported | Supported | Supported |
| Intel Gaudi (HPU) | Supported | Partial | Not supported |
| Apple Silicon Metal (MPS) | Supported, less optimized | Supported | Planned |
Features
- LLM.int8() inference: 8-bit quantization with outlier handling that roughly halves inference memory
- QLoRA 4-bit training: fine-tunes 4-bit quantized models through trainable LoRA adapter weights
- 8-bit optimizers: block-wise quantized optimizer state at a fraction of the 32-bit memory cost
- Quantized layers: drop-in 8-bit and 4-bit linear layers for PyTorch models
- Broad accelerator support: CPUs, NVIDIA, AMD, and Intel GPUs, Intel Gaudi, and Apple Silicon
- Ecosystem integration: quantization backend for Hugging Face Transformers, Diffusers, and PEFT
