Luminal is an inference compiler written in Rust. Instead of executing operations eagerly, it records a neural network as a static directed acyclic graph, compiles that graph ahead of time, and executes it — so devices, datatypes, and autograd are resolved at compile time and the code that runs can differ substantially from the code that was written.
Everything reduces to about 15 primitive operations: the unary Log2, Exp2, Sin, Sqrt, and Recip; the binary Add, Mul, Mod, and LessThan; and SumReduce, MaxReduce, Iota, Gather, Scatter, and Cast. This set covers transformers, convolutional networks, and most widely used architectures, and keeps the core library small enough to read in one sitting.
Optimisation is driven by search rather than hand-written rewrite rules. Luminal explores the decision space to discover optimisations such as Flash Attention on its own, in contrast to compiler stacks that lower a graph through destructive one-directional rules. Dynamic shapes are modelled natively as symbolic dimensions, including expressions such as (s, 4096) or (b, h, w + 3), which leaves the compiler full visibility into shapes while still allowing aggressive specialisation. That global view enables kernel fusion, shape-specific kernels compiled at runtime, low-precision dtypes such as mxfp4, nvfp4, and fp8, and multi-device parallelism topologies searched ahead of time.
The crate links statically and talks directly to accelerator APIs such as CUDA and Metal, with no container or virtual environment in between. It integrates with PyTorch as a compiler backend through torch.compile(model, backend=luminal_cuda), and also exposes a Rust tensor API. Documented status includes Q8 Llama 3 8B running at roughly 80% of theoretical maximum performance on an H100, several kernel libraries such as FlashInfer and cuBLASLt in the search space, a small neural-network module library with transformers, and high-level operations aiming at the most-used 80% of the PyTorch API. Correctness is checked against equivalent PyTorch implementations.
Features
- RISC-style op set: roughly 15 primitives cover transformers, convnets, and most common architectures
- Ahead-of-time compilation: the whole network is a static graph compiled before execution, with nothing left to run time
- Search-based optimisation: the compiler explores decisions rather than applying heuristics, discovering optimisations such as Flash Attention
- Symbolic shapes: arbitrary symbolic dimensions and expressions model dynamism while keeping shape information visible to the compiler
- PyTorch backend: torch.compile(model, backend=luminal_cuda) compiles existing PyTorch models
- Native accelerator access: a statically linked Rust crate calls CUDA and Metal directly, with no compatibility layer
- Kernel library integration: FlashInfer, cuBLASLt, and others participate in the search space
- Low-precision support: mxfp4, nvfp4, and fp8 dtypes are modelled by the compiler
- Diagnostics: environment flags control search, e-graph, and loop-rolling logs, canonical LLIR dumps, CUDA phase profiling, and memory debugging
