Mesh LLM

Mesh LLM pools GPUs and memory across machines and serves the result as one OpenAI-compatible API. You start one node and add more later, and the mesh decides whether a model runs locally, routes to a peer that hosts it, or splits across several machines because it is too large for one box. Meshes can be private (joined by invite token) or public, and a node can also run as an API-only client.

The project has two parts. Skippy handles model management, native inference on a patched llama.cpp runtime, split execution, and OpenAI serving. Mesh adds peer discovery, transport, routing, plugins, and a web console around the same Skippy serving code. Every node exposes the same API, and requests are routed by their model field. Large dense models load as layer stages: a coordinator plans contiguous layer ranges across nodes, and peers fetch only the GGUF fragments for their assigned stage from a layer package. Traffic between nodes, including inference requests and split-model activations, is end-to-end encrypted over QUIC, and published meshes advertise through Nostr discovery.

An experimental Mixture-of-Agents mode fans one request out to every model in the mesh in parallel, arbitrates the responses in code, and calls a reducer model only on a genuine conflict. Agent launchers connect Goose, OpenCode, Claude Code, and Pi to the mesh, and plugins declare what they provide in a manifest and expose capabilities over MCP, HTTP, inference, and mesh events.

The Skippy runtime tracks llama.cpp model family parity, with 72 reviewed family rows covering Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi, Granite, Hunyuan, EXAONE, Cohere, Falcon, RWKV, and others. Split multimodal serving is certified for Qwen2-VL, Qwen3-VL, Qwen3-VL-MoE, Hunyuan-VL, and DeepSeek-OCR. Release builds cover macOS, Linux CPU, CUDA, ROCm, and Vulkan, and Windows.