ODS (Osmantic Deployment System) installs and wires together a self-hosted AI server stack on Linux, macOS on Apple Silicon, or Windows through WSL2 and Docker Desktop. Instead of assembling local inference, a chat UI, workflows, image generation, and privacy tools by hand, one installer detects the GPU, picks a model that fits the hardware, generates credentials, and starts the services with a web UI and a control dashboard.
Local mode is the default: inference, chat history, and files stay on the machine, and ODS collects no telemetry. It goes online to download models and container images, check for releases, and run web searches for its Portal agent. Cloud and hybrid modes route requests to OpenAI, Anthropic, or Together APIs instead, for machines without a GPU.
The default Core install provides llama-server inference, the dashboard, and a chat UI (the Portal assistant on supported hosts, Open WebUI elsewhere). Voice, workflows, search, RAG, image generation, observability, and developer tools are added from the installer menu or the Extensions page, already wired to each other. Every service is an extension defined by a manifest and a Docker Compose fragment, and the dashboard, CLI, and health checks discover extensions automatically.
A bootstrap mode starts a small Qwen model as soon as the core services are up, so you can chat while the full model downloads, then swaps to the full model once its checksum is verified. Default picks by hardware:
| Hardware | Default model | Context |
|---|---|---|
| NVIDIA, 8–16 GB VRAM | Qwen3.5 9B | 64K |
| NVIDIA, 24–32 GB VRAM | Qwen3.5 27B | 64K |
| NVIDIA, 48 GB VRAM | Qwen3.6 35B-A3B | 128K |
| NVIDIA, 96 GB+ VRAM | Qwen3 Coder Next | 128K |
| AMD Strix Halo, 64–128 GB | Qwen3.6 35B-A3B | 128K |
| Apple Silicon, 8 GB | NVIDIA Nemotron3 Nano 4B | 64K |
| Apple Silicon, 48 GB+ | Qwen3.6 35B-A3B | 128K |
| CPU only, 24 GB+ RAM | Qwen3.5 9B | 64K |
Features
- Chat and inference: Open WebUI, llama-server with continuous batching, a LiteLLM gateway for local, cloud, and hybrid modes, and TEI embeddings
- Voice: Whisper speech-to-text and Kokoro text-to-speech
- Agents and automation: the Portal assistant, Hermes Agent, n8n workflows, OpenCode in the browser, and APE, an Agent Policy Engine that audits autonomous tool calls
- Knowledge and search: Qdrant for RAG, SearXNG self-hosted search, Perplexica deep research, and optional Brave Search
- Image generation: ComfyUI for node-based local image generation
- Privacy and ops: a PII-scrubbing proxy for API calls, a token usage monitor, optional Langfuse tracing, and a dashboard with GPU metrics and service health
- Hardware detection: picks a model and context size for NVIDIA, AMD Strix Halo, Apple Silicon, Intel Arc (experimental), or CPU-only machines
- Model switching: the dashboard lists models that fit the hardware, verifies downloads by checksum, imports local GGUF files, and restores the previous model if a new one fails to load
- Extensions: every service is a folder with a manifest and a Compose fragment, enabled or disabled from the dashboard or the CLI
- Stack CLI: a command-line tool for status, model management, service control, and updates
