LiteRT Torch is a Python library that converts PyTorch models into the .tflite format so they can run fully on-device with LiteRT, in applications for Android, iOS, and IoT hardware. It builds on torch.export() and covers the Core ATen operators, with broad CPU coverage and initial GPU and NPU support.
The package has two parts. The PyTorch converter (beta) turns an exported model and sample inputs into a .tflite flatbuffer that runs through the LiteRT compiled model API from native C++ or Java. The Generative API (alpha) is a Torch-native library for authoring mobile-optimized transformer models, including quantization, and converting them to LiteRT-LM models for running large language models on edge devices.
The Generative API supports CPU and GPU today, with NPU support planned. A converted model can be packaged with its tokenizer into a .litertlm container for deployment.
Features
- PyTorch conversion: converts a PyTorch model and sample inputs into a .tflite flatbuffer
- On-device targets: converted models run on Android, iOS, and IoT devices through LiteRT
- Hardware coverage: broad CPU support with initial GPU and NPU support for converted models
- torch.export integration: builds on torch.export() and covers the Core ATen operators
- Generative API: authors mobile-optimized transformer models and converts them to LiteRT-LM models
- Quantization: quantizes large language models and transformer models for on-device performance
- Deployment packaging: bundles a converted model with its tokenizer into a .litertlm container