mergekit combines the weights of pre-trained language models. Instead of ensembling several models at inference time or training a new one, it works directly in weight space, so the merged model costs the same to run as a single model. Typical uses are folding specialised checkpoints into one, transferring a capability between models without the training data, and trading off behaviours along a controlled interpolation.
It works out of core and loads tensors lazily, so a merge runs entirely on CPU or with as little as 8 GB of VRAM. Supported architectures include Llama, Mistral, GPT-NeoX, and StableLM.
A merge is described in a YAML document and executed by mergekit-yaml, which writes the merged model to an output directory. The document names a merge method, either whole models or slices of layers taken from different models, an optional base model, a data type, and parameters. Parameters can be single values or interpolated gradients across layers, can be filtered by tensor name so attention and fully connected layers differ, and resolve by precedence from a specific input slice up to a catch-all.
Tokenizer handling is explicit. A tokenizer field sets the output vocabulary — the union of all inputs, the base model's, or a specific model's — and can pin the embedding source per token, force one embedding across models, map a token onto another model's token, or zero it. For a token a model lacks, the fallback is the base model's embedding, the embedding from the only model that has it, or an average. A chat_template field selects the most common template among the inputs, a built-in template (alpaca, chatml, llama3, mistral, exaone), or a Jinja2 string.
Merge methods span weighted averaging and spherical interpolation (linear, slerp, nuslerp, multislerp, karcher), task-vector arithmetic and its sparsified variants (taskarithmetic, ties, darelinear, dareties, della, breadcrumbs, sce), geometric weight selection (modelstock), similarity-gated interpolation (nearswap, arcee_fusion), and tensor copying for layer stacking (passthrough). They differ in how many inputs they accept and whether a base model is required, optional, or unused.
Companion commands cover related weight surgery: extracting a PEFT-compatible low-rank approximation of a fine-tuned model, merging dense models into a mixture of experts, chaining several merges so later stages consume earlier outputs, merging raw PyTorch or safetensors checkpoints outside the Transformers format, transplanting a tokenizer between models for speculative decoding or cross-tokenizer distillation, and searching merge configurations with evolutionary methods.
Features
- Out-of-core merging: lazy tensor loading keeps memory low enough for CPU-only runs or 8 GB of VRAM
- Many merge methods: linear, SLERP variants, Karcher mean, task arithmetic, TIES, DARE, DELLA, Model Breadcrumbs, SCE, Model Stock, Nearswap, Arcee Fusion, and passthrough
- YAML configuration: whole models or layer slices, base model, data type, and parameters in one document
- Interpolated gradients: per-layer parameter curves and tensor-name filters for fine-grained control
- Frankenmerging: piecewise assembly of a model from layers of several inputs
- Tokenizer control: union, base, or specific vocabularies with per-token embedding sources and forced overrides
- Chat templates: automatic selection, built-in templates, or a custom Jinja2 template
- LoRA extraction: PEFT-compatible low-rank approximations of fine-tuned models
- Mixture-of-experts merging: dense checkpoints combined into an MoE for use or further training
- Multi-stage merges: several configurations in one file where later merges consume earlier outputs
- Raw PyTorch merging: the same algorithms applied to .pt or .safetensors checkpoints
- Tokenizer transplantation: vocabulary alignment for speculative decoding and distillation
