MicrosoftOlive (short for ONNX Live) is a model optimization toolkit for the ONNX Runtime. Given a model and the target hardware, it selects and chains optimization techniques to produce ONNX models for inference in the cloud or on edge devices, within constraints such as accuracy and latency.
Its automatic optimizer pulls a model from Hugging Face, quantizes it (for example to int4 with GPTQ), captures the ONNX graph with its weights, and optimizes that graph. Common architectures such as Llama, Phi, Qwen, and Gemma work without extra configuration, and other architectures can be optimized by describing the model's inputs and outputs. The resulting models run on the ONNX Runtime, which embeds on-device inference into applications through Python, C/C++, C#, Java, and JavaScript bindings.
Beyond conversion and quantization, Olive covers fine-tuning, NPU targets, generating models for MultiLoRA serving, model packaging, and running workflows on remote compute such as Azure AI. It is driven from a CLI or from Python and notebooks, with examples kept in a separate recipes repository. The AI Toolkit extension for VS Code uses Olive for fine-tuning. Distributions may send usage data to Microsoft.
Features
- Automatic optimizer: one command acquires, quantizes, converts, and optimizes a Hugging Face model for a precision target
- Hardware targets: CPUs, GPUs, and NPUs, for cloud or edge inference
- Quantization: GPTQ and other quantization algorithms, with guidance on settings for small language models
- ONNX conversion: captures the model graph and stores weights in an ONNX data file
- Fine-tuning: fine-tune before optimizing, including a worked notebook for DeepSeek R1
- MultiLoRA output: generates models for serving several LoRA adapters on the ONNX Runtime
- Remote compute: runs optimization workflows on remote compute such as Azure AI
- Model packaging: packages optimized models for deployment
- Custom architectures: models outside the built-in list are supported through an input and output description
