LLMCompiler orchestrates parallel function calling for LLM agents. Methods such as ReAct reason and act one function call at a time, which adds latency and cost and can hurt accuracy. LLMCompiler instead decomposes a problem into tasks, works out which can run in parallel and which depend on each other, and executes the independent ones concurrently.
The design borrows from classical compilers and has three components: a Function Calling Planner that formulates an execution plan, a Task Fetching Unit that dispatches tasks as their dependencies resolve, and an Executor that runs them in parallel. With streaming on, tasks flow from the planner to the executor as soon as they are generated. You supply the functions, their descriptions, and optional in-context examples, and the framework computes the orchestration.
It works with OpenAI GPT models (directly or through Azure endpoints), Friendli endpoints, and open-source models such as LLaMA served with vLLM. The repository reproduces the paper's evaluations on the HotpotQA, Movie Recommendation, and ParallelQA benchmarks, and can run a ReAct baseline for comparison. The paper reports latency speedups of up to 3.7x, cost savings of up to 6.7x, and accuracy gains of up to about 9% over ReAct.
LLMCompiler is also available as a pattern within LangGraph and as a LlamaIndex pack.
Features
- Parallel function calling: identifies independent tasks and executes them concurrently instead of one at a time
- Planner, fetcher, executor: a compiler-style pipeline that plans calls, dispatches them as dependencies resolve, and runs them in parallel
- Streaming plans: tasks are dispatched as soon as the planner emits them, without waiting for the full plan
- Model choice: OpenAI GPT models, Azure OpenAI and Friendli endpoints, and open-source models served with vLLM
- Custom benchmarks: a new use case needs only its functions, their descriptions, and example prompts
- ReAct baseline: runs ReAct on the same benchmarks for comparison
- Framework ports: available inside LangGraph and as a LlamaIndex pack
