LiteRT-LM is Google's orchestration layer for running large language models on device with LiteRT, the runtime formerly known as TensorFlow Lite. It targets Android, iOS, the web, desktop, and IoT boards such as Raspberry Pi, and runs models on CPU, GPU, or NPU depending on the platform.
It ships language APIs for Python, Kotlin, C++, Swift, JavaScript, and Flutter. A command-line tool runs models pulled from Hugging Face, imports models, describes a model's metadata and runtime requirements before loading, and serves an OpenAI-compatible embeddings endpoint for multimodal embedding models.
Supported model families include Gemma, Llama, Phi-4, and Qwen, with vision and audio inputs and function calling for agentic workflows. LiteRT-LM runs on-device generative AI in Chrome, Chromebook Plus, and Pixel Watch, and the Google AI Edge Gallery app uses it to run models on a phone.
| Language | Status | Target |
|---|---|---|
| Python | Stable | Prototyping and scripting |
| Kotlin | Stable | Android apps and the JVM |
| Swift | Early preview | iOS and macOS |
| JavaScript | Early preview | Browsers, with WebGPU acceleration |
| Flutter | Community | Cross-platform mobile |
| C++ | Stable | Native applications |
Features
- Cross-platform: Android, iOS, web, desktop, and IoT devices such as Raspberry Pi
- Hardware acceleration: GPU and NPU backends, including on-demand KV cache growth on NPU
- Multimodality: vision and audio inputs
- Tool use: function calling APIs for agentic workflows on device
- Model families: Gemma, Llama, Phi-4, Qwen, and others, with pre-converted models on Hugging Face
- Speculative decoding: multi-token prediction drafters for Gemma 4 models
- Embeddings: EmbeddingGemma 2 text, vision, and audio embeddings with Matryoshka dimension truncation
- Model introspection: a ModelInfo API reports metadata, capabilities, and runtime requirements before loading
- CLI: runs, imports, describes, and serves models from the terminal
