local.ai is a desktop app for running and experimenting with AI models locally and offline, without a GPU. It combines a model manager, an inference streaming server, and a small note-taking interface in one native app with a Rust backend, built on the rustformers llm crate.
The model manager keeps models in a directory you choose, downloads them with a resumable, concurrent downloader, and checks file integrity with BLAKE3 and SHA256 digests. A known-good model list describes each model with recommended hardware, license, and hashes. Inference adapts to available threads and supports GGML quantization formats (q4, 5.1, 8, and f16).
Loading a model and starting the server exposes a streaming completion endpoint similar to OpenAI's. The app is designed to pair with window.ai, a browser extension that lets web apps call a model the user provides, so a web app can use a local model at no cost to the developer or the user. Notes store their inference configuration per note and are written out as plain-text MDX files. The app is distinct from the similarly named LocalAI project.
Features
- Model downloader: resumable, concurrent downloads into a directory you choose
- Integrity checks: BLAKE3 and SHA256 digests for downloaded models
- Known-good model list: model cards with recommended hardware, license, and hashes
- CPU inference: GGML quantization (q4, 5.1, 8, f16), adapting to available threads
- Streaming server: a completion endpoint similar to OpenAI's, started from the app
- window.ai pairing: acts as the local inference backend for web apps that use window.ai
- Notes with inference: per-note inference configuration, saved as plain-text MDX
