DwarfStar (ds4) is a native C inference engine that runs a small set of large open-weight models on hardware a person can own: high-memory Macs, NVIDIA DGX Spark and other CUDA machines, and AMD Strix Halo systems. It targets DeepSeek V4 and V4.1 Flash, DeepSeek V4 PRO, GLM 5.2, 5.3 and 5.3 Flash, and Qwen3.8 Flash Next, including vision variants. It is deliberately narrow rather than a general GGUF runner: it loads only the GGUF files the project itself produces, and a model may be dropped when a better one for the same machine size appears.
The engine fits these mixture-of-experts models on local machines with asymmetric 2-bit quantization of the routed experts, keeping the shared paths at higher precision. Models larger than RAM run by streaming weights from SSD, and the KV cache can be saved to disk and resumed by prompt hash, so a restart does not require a full re-prefill. Two Macs connected over RDMA can split a model with tensor parallelism, pipeline parallelism combines several machines' memory, and a multi-GPU CUDA mode serves several sessions on cards such as the L40S.
One engine has three interfaces that share model state and cache: an interactive CLI, a server with OpenAI-style chat completions and Anthropic-style messages endpoints, and a native coding agent. The agent runs inference directly, uses each model's native tool format, and saves, lists, and resumes sessions from on-disk KV snapshots. Coding agents such as Pi, OpenCode, Codex, and Claude Code connect through the server.
Further controls include adjustable thinking effort, opt-in speculative decoding (multi-token prediction for GLM and Qwen), directional steering vectors for DeepSeek V4 Flash and GLM 5.3 Flash, and a power setting that trades throughput for lower sustained GPU load. A bundled evaluation tool runs capability regression tests against a real model file. The project is beta quality and changes quickly.
| Machine | Context | Prefill t/s | Generation t/s |
|---|---|---|---|
| M5 Max, 128 GB, Q2 | 2,048 tokens | 790.2 | 39.4 |
| M5 Max, 128 GB, Q2 | 65,536 tokens | 398.5 | 27.6 |
| DGX Spark, 128 GB, Q2 | 2,048 tokens | 825.8 | 18.1 |
| DGX Spark, 128 GB, Q2 | 65,536 tokens | 823.0 | 13.8 |
Features
- Supported models: DeepSeek V4 and V4.1 Flash, DeepSeek V4 PRO, GLM 5.2 and 5.3 (including Flash), and Qwen3.8 Flash Next
- Backends: Metal on Apple Silicon, CUDA (DGX Spark and multi-GPU cards), and ROCm on Strix Halo
- Asymmetric 2-bit quantization: compresses routed experts while keeping shared paths precise
- SSD streaming: runs models larger than RAM by reading weights from a fast local SSD
- Disk KV cache: saves long prefixes to disk and resumes them by prompt hash
- Distributed inference: tensor parallelism across two Macs over RDMA, plus pipeline parallelism across machines
- Local server: chat completions and messages endpoints for API clients and coding agents
- Native coding agent: runs inference in-process with saved, resumable sessions and the model's native tool format
- Vision: image input through matching vision encoders for supported models
- Speculative decoding: opt-in multi-token prediction for GLM and Qwen
- Steering and thinking controls: directional steering vectors and an adjustable reasoning-effort level
- Capability evals: embedded regression suites run against a real model file
