Luminal is an ahead-of-time inference compiler written in Rust. Models are recorded as static, directed acyclic computation graphs built from 15 primitive operations (unary ops such as Exp2 and Sqrt, binary ops such as Add and Mul, and reductions, gathers, scatters, and casts), then compiled and executed later. These ops cover transformers, convnets, and most popular model architectures, and the core library stays small enough to read in an afternoon.
Instead of a fixed set of one-directional rewrite rules, as in XLA, torch.compile, or TVM, the compiler searches the space of possible decisions to find optimizations, and discovers Flash Attention and similar kernels without hand-written operations. Because the whole network is one graph, the compiler also handles devices, datatypes, and autograd ahead of time. Dynamic shapes are modeled as symbolic dimensions, so the compiler can still specialize aggressively.
Luminal talks directly to accelerator APIs (CUDA and Metal) as a statically linked Rust crate, with no compatibility layers or containers. It also works as a PyTorch compiler backend through torch.compile, and its ops are tested against equivalent PyTorch implementations. The search space includes kernel libraries such as FlashInfer and cuBLASLt, and a model zoo of logical model definitions, an NN module library with transformers, and high-level ops covering much of the PyTorch API are in place.
Q8 Llama 3 8B runs at about 80% of theoretical maximum performance on an H100. A published GPT-OSS 120B throughput comparison on 8x H100 SXM:
| Engine | Tokens per second |
|---|---|
| Luminal | 36k |
| TensorRT-LLM | 28k |
| vLLM | 26k |
| PyTorch | 3k |
Alongside the open-source compiler, Luminal offers managed serverless inference endpoints that scale to zero, and licensed cloud or on-premises deployment. Roadmap items include a ROCm backend, more inference accelerator backends, thread- and warp-level dialects, a public benchmarking suite, and automatically searched model parallelism.
Features
- Primitive op set: every model reduces to 15 primitive operations, enough for transformers and convnets
- Search-based optimization: the compiler searches possible decisions instead of applying hand-tuned heuristics, discovering Flash Attention and similar optimizations
- Ahead-of-time compilation: networks are recorded as static computation graphs, compiled once, and executed later
- Symbolic dynamic shapes: arbitrary symbolic dimensions, including expressions, with shape-specific kernels compiled at runtime
- PyTorch backend: compiles PyTorch models through torch.compile with a Luminal backend
- Rust tensor API: a native Rust API for defining graphs, plus NN modules and high-level ops
- Native accelerators: CUDA and Metal backends with no intermediate abstraction layers
- Kernel library search: FlashInfer and cuBLASLt kernels included in the search space
- Low-precision dtypes: mxfp4, nvfp4, and fp8
- Compiler-managed graph: kernel fusion, devices, datatypes, and autograd optimized ahead of time
- Correctness tests: ops validated against equivalent PyTorch implementations
- Hosted inference: managed serverless endpoints, or licensed cloud and on-premises deployment
