ODS, the Osmantic Deployment System, installs and wires a local AI server onto a Windows, macOS, or Linux machine, so the pieces of a self-hosted stack arrive pre-connected rather than assembled by hand. One installer command detects the GPU, picks a model for the detected memory envelope, generates credentials, starts the services, and opens a local web UI.
The stack covers local inference, a chat interface, a control dashboard, voice input and output, agents and workflow automation, retrieval and search, image generation, and privacy and observability tooling. Local mode is the default; cloud and hybrid API modes are optional, and every port is configurable through environment variables.
Hardware detection assigns a deterministic tier and then a catalog selector reads a model library to choose an installable GGUF file, writing the choice into .env. Bootstrap mode downloads a small model first so a chat session works within minutes while the full model downloads in the background, then hot-swaps; if a new model fails to load, the previous model is restored.
Every service is an extension: a folder with a manifest and a Compose fragment that the dashboard, CLI, health checks, and Compose stack discover automatically. The ods CLI manages status, logs, restarts, mode switching between local, cloud, and hybrid, model swaps, extension enable and disable, and configuration presets.
Features
- One-command install: GPU detection, model selection, credential generation, and service startup in a single step on Linux, macOS, and Windows
- Bootstrap mode: a small model serves chat immediately while the selected model downloads, then swaps with no downtime
- Tiered model catalog: per-tier default picks and context windows for NVIDIA, AMD Strix Halo, Apple Silicon, and Intel Arc envelopes
- Extension system: manifest plus Compose fragment folders that the dashboard, CLI, and stack discover automatically
- Mode switching: ods mode local, cloud, and hybrid route inference through local hardware, hosted APIs, or both
- Voice and image: speech-to-text, text-to-speech, and node-based image generation run as part of the stack
- Retrieval and search: a vector database, an embedding service, self-hosted web search, and a research engine
- Privacy and ops: a scrubbing proxy for API calls, a token usage monitor, optional tracing, and a dashboard with GPU metrics and service health
- Rollback: a model that fails to load reverts to the previously selected one
| Tier / envelope | Default catalog pick | Context |
|---|---|---|
| NVIDIA 8 GB CPU fallback | Qwen3.5 2B (Q4KM) | 8K |
| NVIDIA 8 GB discrete VRAM | Qwen3.5 9B (Q4KM) | 32K |
| NVIDIA 24 GB discrete VRAM | Qwen3.5 27B (Q4KM) | 32K |
| NVIDIA 48 GB discrete VRAM | DeepSeek R1 Distill Llama 70B (Q4KM) | 32K |
| AMD Strix Halo 64 GB unified | Qwen3.6 35B-A3B (UD-Q4KM) | 128K |
| Apple Silicon 8 GB unified | Phi-4 Mini (Q4KM) | 128K |
| Apple Silicon 64+ GB unified | Qwen3.6 35B-A3B (UD-Q4KM) | 128K |
| Intel Arc 16 GB discrete VRAM | Phi-4 14B (Q4KM) | 16K |
Integrates with
ComfyUI
Node-graph engine for generating images, video, 3D, and audio, locally or via API
Hermes
Self-improving AI agent with a learning loop that creates and refines skills, recalls past sessions, and runs across chat platforms
Langfuse
Open-source platform for tracing, evaluating, and debugging LLM applications, self-hosted or cloud
LiteLLM
Open-source AI gateway exposing 100+ LLM providers through one OpenAI-compatible interface, as a Python SDK or self-hosted proxy
