AlibabaMNN is a deep learning engine for on-device inference and training, deployed inside Alibaba apps such as Taobao, Tmall, Youku, and DingTalk and on embedded devices. It converts models from TensorFlow, Caffe, ONNX, and TorchScript into its own format and runs them on mobile CPUs, GPUs, and NPUs without external dependencies.
Two runtimes build on the engine. MNN-LLM runs large language models locally on phones, PCs, and IoT devices, with support for families such as Qwen, Baichuan, Zhipu, and Llama, and MNN-Diffusion runs Stable Diffusion models locally. Sample apps include an Android and iOS multimodal chat app, a 3D avatar app that runs LLM, ASR, TTS, and animation models on-device, and a Sana-based image editor. Version 3.6.1 added a Hexagon backend for Qualcomm DSPs.
Backends cover CPU (x86 SSE4.1, AVX2, AVX512; ARMv7a, ARMv8 with FP16 and BF16), GPU (OpenCL, Vulkan, Metal, CUDA), and NPU (CoreML, HIAI, NNAPI, QNN), with FP16, BF16, and Int8 precision support varying by backend. The library is about 12 MB as an iOS static library and about 800 KB as an Android core .so, and FP16 or Int8 quantization reduces model size by 50% to 70%.
Features
- Model conversion: MNN-Converter imports TensorFlow, Caffe, ONNX, and TorchScript models (178, 52, 158, and 163 operators respectively) with graph optimization
- On-device LLM and diffusion: MNN-LLM and MNN-Diffusion runtimes for local text, multimodal, and image generation
- Multi-backend execution: CPU, OpenCL, Vulkan, Metal, CUDA, Hexagon, and NPU backends with hybrid CPU and GPU computing
- Quantization and compression: FP16 and Int8 quantization plus the MNN-Compress tool
- Dynamic models: multi-input and multi-output models, dynamic shapes, and control flow
- Training: MNN-Train builds and trains models on PC or mobile
- Companion libraries: MNN-Express for general computation with MNN operators, MNN-CV as an image-processing module, and a Python API
- Workbench: pretrained models, visual training tools, and one-click deployment to devices
