mergekit merges pre-trained language models by operating directly on their weights, combining specialized models into one without ensembling or further training. A merged model keeps the inference cost of a single model. The toolkit loads tensors lazily and works out of core, so merges run on CPU alone or with as little as 8 GB of GPU memory.
A merge is described in a YAML configuration: the method, whole models or layer slices drawn from several models, an optional base model, and parameters that can be scalars or gradients interpolated across layers and filtered by tensor name. Tokenizer settings align vocabularies across inputs (union, base, or a chosen model's vocabulary) with per-token embedding overrides, and a chat template can be set or picked automatically. An in-memory Python API merges tensors or state dictionaries directly, and supported architectures include Llama, Mistral, GPT-NeoX, and StableLM.
| Method | Core idea |
|---|---|
| Linear | Weighted average of parameters |
| SLERP, NuSLERP, Multi-SLERP | Spherical interpolation between two or more models |
| Karcher Mean | Riemannian barycenter of parameters |
| Task Arithmetic | Linear combination of task vectors relative to a base |
| TIES, DARE, DELLA, Breadcrumbs | Task arithmetic with pruning, sign consensus, or outlier removal |
| SCE | Matrix-level weighting by parameter variance |
| Model Stock | Geometric weights for interpolating many checkpoints |
| Nearswap, Arcee Fusion | Selective merging where parameters are similar or changes are salient |
| Passthrough | Copies tensors for layer stacking (frankenmerging) |
Separate tools extend the core: LoRA extraction produces low-rank adapters from fine-tuned models, a mixture-of-experts tool assembles dense models into an MoE, evolutionary methods search merge configurations, multi-stage merging chains configurations where later merges consume earlier outputs, raw PyTorch checkpoints can be merged, and tokenizer transplantation aligns one model's vocabulary to another, for example to build draft models for speculative decoding. Merged models can be published to the Hugging Face Hub with a generated model card.
Features
- Out-of-core merging: lazy tensor loading lets merges run on CPU or with 8 GB of GPU memory
- Merge methods: Linear, SLERP variants, Karcher Mean, Task Arithmetic, TIES, DARE, DELLA, Breadcrumbs, SCE, Model Stock, Nearswap, Arcee Fusion, and Passthrough
- Layer slicing: assemble models piecewise from layers of different sources (frankenmerging)
- Gradient parameters: parameter values interpolated across layers and filtered by tensor name
- Tokenizer alignment: union, base, or chosen vocabularies with per-token embedding sources and new special tokens
- LoRA extraction: PEFT-compatible low-rank approximations of fine-tuned models
- Mixture of experts: combines dense models into an MoE for direct use or further training
- Evolutionary merging: searches merge configurations automatically
- Multi-stage merges: one YAML file chains merges that feed into each other
- Tokenizer transplantation: moves a tokenizer between models, for example for speculative-decoding drafters or cross-tokenizer distillation
- Python API: merge tensors or state dictionaries already in memory, with custom merge methods
