Muna compiles AI models ahead of deployment into hardware-aware inference servers, then serves them through an OpenAI-compatible interface. Because a compiled model ships as a binary rather than a container, the project reports cold starts falling by up to 45×, with the first call landing as fast as the millionth.
The Python distribution carries both a client and a command-line interface and requires Python 3.11 or later. You create a client with an access key and reach models through the OpenAI-shaped API surface — chat completions, embeddings, transcription, and speech — passing an acceleration argument that decides where each individual request runs, from hosted GPUs to a local GPU.
A compiled model is portable across compute: the same artifact can stay on the hosted GPUs or move to your own infrastructure, including Modal, Baseten, or on-premises hardware, with one command. The catalogue of already-compiled models spans language models, audio and voice models, vision models, and embedding models.
Features
- Ahead-of-time compilation: models are compiled into inference servers targeted at the device that will run them
- OpenAI-compatible client: the official OpenAI SDK, streaming, and model aliases work against the endpoint
- Per-request placement: an acceleration argument selects where an individual inference runs
- Portable deployment: one deploy command moves a compiled model to your own GPUs or another hosting provider
- Multiple modalities: chat, embeddings, transcription, and speech behind one API
- Python client and CLI: a single distribution covering both, on Python 3.11 or later
- Prebuilt catalogue: a public repository of models already compiled for the runtime
