NVIDIA Model Optimizer (ModelOpt) is a Python library of model compression techniques for faster inference: quantization, pruning, neural architecture search, distillation, speculative decoding, and sparsity. It takes Hugging Face, PyTorch, or ONNX models as input, lets you compose these techniques through Python APIs, and exports an optimized checkpoint.
Exported checkpoints target downstream inference frameworks such as SGLang, Tensorrt LLM, TensorRT, and vLLM, and a unified Hugging Face export API covers both transformers and Diffusers models. Training-based techniques run through integrations with Megatron-Bridge, Megatron-LM, and Hugging Face Accelerate. Pre-quantized checkpoints (FP8 and NVFP4 variants of models such as Llama, DeepSeek-R1, and Nemotron) are published on Hugging Face.
| Technique | Purpose |
|---|---|
| Post-training quantization | Compress model size by 2x to 4x for faster inference |
| Quantization-aware training and distillation | Recover accuracy of quantized models with a few training steps |
| Pruning | Remove weights to cut parameters and memory footprint |
| Distillation | Train a small model to behave like a larger one |
| Speculative decoding | Train draft modules that predict extra tokens during inference |
| Sparsity | Store only non-zero parameter values and their locations |
Recent work covers NVFP4 low-precision formats, an automatic mixed-precision assignment method (AutoQuantize), and Puzzletron, an algorithm for heterogeneous pruning and NAS of language and vision-language models. Agent skills for applying ModelOpt are also available as Claude Code and Codex plugins.
Features
- Quantization: post-training quantization to formats including FP8 and NVFP4 for LLMs, VLMs, diffusion models, and ONNX models
- Quantization-aware training: a few training or distillation steps to recover the accuracy lost to low-precision formats
- Pruning and NAS: removes weights and searches architectures, including the Minitron and Puzzletron methods
- Distillation: teaches smaller models to behave like larger ones, with Hugging Face and Megatron-LM examples
- Speculative decoding: trains draft modules that predict extra tokens at inference time
- Sparsity: stores only non-zero parameter values to compress models
- Model inputs: Hugging Face, PyTorch, and ONNX models
- Deployment export: checkpoints for SGLang, Tensorrt LLM, TensorRT, and vLLM
- Training integrations: Megatron-Bridge, Megatron-LM, and Hugging Face Accelerate
- Pre-quantized checkpoints: a Hugging Face collection of inference-optimized checkpoints
- Agent skills: plugins for Claude Code and Codex that apply ModelOpt workflows
