PaperlayerView exact revision on arXiv

Source-linked AI summary

An LLM Compiler for Parallel Function Calling

Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, Amir Gholami

arXiv:2312.04511v3cs.CL

TL;DR

Sequential function calling can incur high latency, cost, and inaccurate behavior when multiple functions must be coordinated. LLMCompiler plans dependency-aware executions and runs independent tasks in parallel, achieving broad improvements across evaluated workloads, including up to 3.7× latency speedup, 6.7× cost savings, and approximately 9% accuracy improvement compared with ReAct.

  • Problem

    Sequential reasoning and acting for each function can cause high latency, cost, and inaccurate behavior when LLMs coordinate multiple function calls.

  • Method

    LLMCompiler uses a Function Calling Planner, Task Fetching Unit, and Executor to plan dependencies, dispatch ready tasks, and execute independent function calls in parallel.

  • Results

    Across varied function-calling patterns, LLMCompiler reports up to 3.7× latency speedup, 6.7× cost savings, and approximately 9% accuracy improvement compared with ReAct.

  • Takeaways & Limitations

    LLMCompiler provides parallel function calling across open-source and closed-source models and supports complex dependency patterns and dynamic replanning.

  • Takeaways & Limitations

    The Planner may add non-trivial overhead for queries with many tasks because fetching and execution wait for its output.

Abstract

from arXiv · show

The reasoning capabilities of the recent LLMs enable them to execute external function calls to overcome their inherent limitations, such as knowledge cutoffs, poor arithmetic skills, or lack of access to private data. This development has allowed LLMs to select and coordinate multiple functions based on the context to tackle more complex problems. However, current methods for function calling often require sequential reasoning and acting for each function which can result in high latency, cost, and sometimes inaccurate behavior. To address this, we introduce LLMCompiler, which executes functions in parallel to efficiently orchestrate multiple function calls. Drawing inspiration from the principles of classical compilers, LLMCompiler enables parallel function calling with three components: (i) a Function Calling Planner, formulating execution plans for function calling; (ii) a Task Fetching Unit, dispatching function calling tasks; and (iii) an Executor, executing these tasks in parallel. LLMCompiler automatically generates an optimized orchestration for the function calls and can be used with both open-source and closed-source models. We have benchmarked LLMCompiler on a range of tasks with different patterns of function calling. We observe consistent latency speedup of up to 3.7x, cost savings of up to 6.7x, and accuracy improvement of up to ~9% compared to ReAct. Our code is available at https://github.com/SqueezeAILab/LLMCompiler.

1. Introduction

LLMCompiler addresses the latency, cost, and accuracy inefficiencies of sequential function calling by planning dependencies and executing independent tasks in parallel. Across several workloads, it reports speedups, cost reductions, and accuracy gains compared with established baselines.

  • Sequential function calling in ReAct can increase latency and cost because each function call is followed by reasoning over its observation before the next call.
  • LLMCompiler uses a Function Calling Planner, Task Fetching Unit, and Executor to orchestrate dependent and independent function calls.The Planner identifies execution flow, the Task Fetching Unit dispatches calls in parallel, and the Executor runs associated tasks.
  • 1.80×/3.74× speedup and 3.37×/6.73× cost reduction are observed on HotpotQA and Movie Recommendation compared with ReAct.
  • Up to 2.27× speedup, 4.65× cost reduction, and 9% improved accuracy are reported on ParallelQA compared with ReAct.
  • LLMCompiler achieves a 2× speedup on Game of 24 compared with Tree-of-Thoughts and up to 101.7× speedup with 25.7% improved success rate on WebShop.

2. Related Work

Prior work improves LLM efficiency through model, systems, prompting, and plan-and-solve techniques, but important gaps remain in dependency-aware parallel function calling and broad model support. LLMCompiler is positioned as a general framework addressing these gaps.

  • Application-level optimization is important for black-box LLM services because model and inference-pipeline modifications may be restricted.
  • Skeleton-of-Thought supports parallel execution for embarrassingly parallel workloads but assumes no dependencies between tasks.
  • OpenAI’s parallel function calling generates multiple calls simultaneously but is limited to proprietary OpenAI models.
  • Prompting methods such as Decomposed Prompting, Step-Back Prompting, and Plan-and-Solve primarily target reasoning accuracy rather than function-calling efficiency.
  • LLMCompiler differs from end-to-end plan-and-solve frameworks through planning and replanning, parallel execution, and support for broader problem domains.
  • Unlike ReWOO, LLMCompiler supports parallel function calling and dynamic replanning when execution flow cannot be determined statically.

3. Methodology

LLMCompiler converts natural-language queries into dependency-aware task plans, dispatches ready tasks, and executes independent tasks concurrently. Its replanning loop adapts execution when intermediate results determine later actions.

  • The framework decomposes a query into tasks, represents their dependencies as a directed acyclic graph, and identifies which tasks can run in parallel.
  • The Function Calling Planner generates tasks, input arguments, and inter-dependencies from natural-language inputs using LLM reasoning.
  • Task Fetching Unit: The Task Fetching Unit greedily dispatches ready tasks and replaces placeholder variables with outputs from preceding tasks without requiring a dedicated LLM.
  • Executor: The Executor asynchronously runs dispatched independent tasks concurrently using user-provided tools and forwards completed results to dependent tasks.
  • Dynamic replanning: For dynamic execution patterns, intermediate results return to the Planner, which generates new tasks and dependencies until the desired result is achieved.

4. LLMCompiler Details

LLMCompiler accepts tool definitions and optional Planner examples, while streamed planning reduces blocking between planning and execution. The streaming benefit depends on tool execution characteristics.

  • Users provide tool descriptions and argument specifications, while in-context Planner examples are optional.
  • The Planner can block task fetching and execution for queries containing many tasks, creating non-trivial overhead.
  • Streaming the dependency graph lets the Executor process each task as soon as its dependencies resolve, improving latency by up to 1.3×.
  • Streaming provides greater benefit on ParallelQA because longer math-tool execution can hide Planner latency, unlike shorter search-tool execution in HotpotQA and Movie Recommendation.

5. Results

LLMCompiler is evaluated across independent, dependency-rich, dynamically replanned, and interactive function-calling tasks using proprietary GPT and open-source LLaMA-2 models. Across these settings, it generally reduces latency and cost while maintaining or improving task performance relative to sequential baselines.

  • Evaluation scope: LLMCompiler is evaluated on embarrassingly parallel, dependency-based, dynamically replanned, and interactive function-calling patterns.The experiments use HotpotQA, Movie Recommendation, ParallelQA, Game of 24, and WebShop.
  • Accuracy and cost: LLMCompiler improves accuracy over ReAct by addressing redundant function calls and premature stopping based on incomplete intermediate results.ReAct-specific prompting improves ReAct, but LLMCompiler remains on-par or better than that prompted baseline.
  • Function calling with replanning: 2× speedup is reported on Game of 24 compared to Tree-of-Thoughts through dynamic replanning based on intermediate results.With GPT-4, the detailed experiment reports a 2.89× latency improvement; with LLaMA-2, it reports 2.01× without compromising success rate.
  • Interactive decision making: 101.7× speedup and 25.7% improved success rate are reported on WebShop compared to baselines.LLMCompiler parallelizes exploration of items in the interactive environment, while broader exploration improves success rate relative to ReAct.

6. Conclusions

LLMCompiler addresses the inefficiencies of sequential function calling with compiler-inspired parallel orchestration, improving latency, cost efficiency, and accuracy across evaluated tasks.

  • LLMCompiler decomposes inputs into interdependent tasks and executes them concurrently through a Planner, Task Fetching Unit, and Executor.The framework supports open-source and closed-source models.
  • LLMCompiler achieves up to 3.7× lower latency, 6.7× greater cost efficiency, and approximately 9% higher accuracy than ReAct.
  • The paper identifies parallel function calling as a way to improve the efficiency of LLMs executing complex, large-scale tasks.

A. Accuracy Analysis: ReAct vs. LLMCompiler

The accuracy analysis attributes ReAct’s weaknesses to premature stopping and repetitive calls, while LLMCompiler’s bounded, complete searches generally produce more reliable results.

  • Premature Early Stopping of ReAct: 68.60 vs. 77.13 accuracy: ReAct’s premature stopping on Movie Recommendation underperforms LLMCompiler’s complete eight-movie searches.
  • Premature Early Stopping of ReAct: About 85% of ReAct Movie Recommendation examples stop early, whereas LLMCompiler consistently completes searches for all 8 movies.
  • Premature Early Stopping of ReAct: Fewer ReAct function calls correlate with lower Movie Recommendation accuracy, while complete LLMCompiler searches attain consistently higher accuracy.
  • Repetitive Function Calls of ReAct: Around 10% of HotpotQA examples undergo repetitive ReAct calls exceeding four, while LLMCompiler generally completes the task within 2 calls.
  • Repetitive Function Calls of ReAct: Cases with more than four ReAct calls achieve less than 10% accuracy in ReAct but around 50% with LLMCompiler.

B. Failure Case Analysis of LLMCompiler

LLMCompiler failures arise in planning, execution, and final answer generation, with executor errors constituting the largest share of failures on ParallelQA.

  • Among LLMCompiler’s 10.6% ParallelQA failures, the Planner, Executor, and final output process contribute 8%, 64%, and 28%, respectively.
  • Planner errors can assign incorrect task dependencies, although adequate tool definitions and in-context examples reduce them to three evaluation instances.
  • Executor failures include incorrect tool attributes and unit conversions, while final-output failures include drawing incorrect conclusions from collected observations.
  • Executor and final-output problems also occur in ReAct, while LLMCompiler reports slightly fewer failures in these areas through relevant tool contexts.

D. Experimental Details

Experiments compare LLMCompiler with ReAct and OpenAI parallel function calling across closed-source and open-source model settings, using controlled prompting and repeated runs for accuracy.

  • The experiments cover API-based closed-source models and open-source models served through an in-house framework.
  • The evaluations use gpt-3.5-turbo for HotpotQA and Movie Recommendation, gpt-4-turbo for ParallelQA, and gpt-4 for Game of 24.
  • Accuracy is averaged over 3 runs because OpenAI outputs retain randomness at temperature 0.
  • ReAct, OpenAI parallel function calling, and LLMCompiler use 3-, 1-, and 5-shot learning for HotpotQA, Movie Recommendation, and ParallelQA, respectively.

E.1. Parallel Speedup Modeling

LLMCompiler’s parallel execution reduces latency relative to ReAct, but planner, answering, and straggler overheads prevent ideal N-way speedups. Its latency model captures sequential planning plus the slowest parallel task, while streaming can reduce total latency.

  • Latency model: ReAct’s latency analysis accounts for sequential thought, action, and function-call observation steps.The planner latency includes thought and action generation, while tool execution is shared with LLMCompiler.
  • Latency model: LLMCompiler models parallel execution as total planning time plus the execution time of the slowest task.All parallelizable tasks are processed concurrently, so the longest task determines the execution component.
  • Latency model: Streaming the dependency graph yields T_SC ≤ T_C, potentially reducing overall latency through more efficient dependency handling.The streaming model modifies the latency calculation to account for streamed dependencies.
  • Speedup bounds: The theoretical maximum speedup γ_max equals the number of tasks N when execution latency dominates planning latency and task durations are equal.The bound assumes executor latency dominates planner latency and execution latencies remain the same.
  • Practical limits: Significant latency gains require reducing planner overhead and minimizing stragglers.In Movie Recommendation, planner and answering overheads average 1.88 and 1.62 seconds, while the slowest search averages 1.13 seconds versus 0.61 seconds for all tasks.
  • Empirical scaling: On ParallelQA, ReAct latency increases with task count, whereas LLMCompiler’s latency is less dependent on the number of parallel tasks.Figure E.5 reports end-to-end latency grouped by the maximum number of parallelizable tasks.

F. Additional Discussions about Related Works

LLMCompiler differs from related frameworks through parallel execution, replanning, and broader domain applicability. Its planner uses structured dependency-aware plans, special finish and join actions, and configurable prompts and tools.

  • Parallel Execution: LLMCompiler supports parallel function calling, unlike TPTU-OA and standalone ViperGPT, while HuggingGPT’s parallelism is restricted to HuggingFace models.TPTU-OA lacks inter-dependencies, ViperGPT requires a dedicated parallel engine, and HuggingGPT targets HuggingFace models.
  • Planning and Replanning: LLMCompiler can replan after earlier tasks when the previous plan is insufficient, unlike the compared planning-based frameworks.TPTU-SA remains iterative like ReAct, while TPTU-OA, HuggingGPT, and ViperGPT lack replanning capabilities.
  • Problem Domains: LLMCompiler targets efficient and accurate function calling across a wide range of problem domains rather than a specific application area.ViperGPT and HuggingGPT focus on vision-related settings through Python code generation or HuggingFace models.
  • Empirical comparison: The framework reports latency and accuracy benefits over ReAct and implemented TPTU baselines, though the official TPTU implementation was unavailable.The authors implemented TPTU-SA and TPTU-OA from prompts in the original paper.
  • Planner interface: The Planner generates numbered dependency-aware tasks using unique identifiers and $id references for outputs from preceding actions.Its prompt instructs the planner to maximize parallelizability and adhere to provided action types.
  • Planner interface: A hard-coded finish function stops plan generation when the plan is sufficient or when execution should precede replanning.The join action collects prior outputs and either finalizes the response or waits for plan execution.

I. ParallelQA Benchmark Generation

ParallelQA is a 113-example benchmark designed to test decomposition of interdependent search and mathematical tasks. Its construction uses diverse entities with Wikipedia-accessible attributes, while Game of 24 provides a replanning-oriented reasoning setting.

  • ParallelQA design: ParallelQA contains 113 mathematical questions requiring factual details about entities and interdependent search and arithmetic operations.Its examples combine entity lookup with calculations such as comparing hypothetical merged-state populations.
  • ParallelQA design: The benchmark selects 56 distinct entities across domains whose attributes are accessible through Wikipedia search.This design minimizes tool-execution failures so evaluation emphasizes decomposition, planning, and answer derivation.
  • Game of 24: Game of 24 requires using four numbers exactly once with basic arithmetic operations to reach 24.The task is presented as a non-trivial reasoning benchmark for language models.
  • Game of 24: Tree of Thoughts generates partial arithmetic solutions, evaluates their prospects, and continues only states labeled likely to reach 24.The process retains the top-5 states according to their values.

K. Details of WebShop Experiments

WebShop evaluates agents on instruction-following shopping tasks where selected products must satisfy multiple constraints. The experiments compare LLMCompiler with state-exploration and Monte Carlo Tree Search baselines.

  • Task and metrics: WebShop tasks require finding products that match specified attributes such as size, color, and price.Each item receives a reward reflecting how well it matches the instruction.
  • Task and metrics: WebShop reports success rate and average score as evaluation metrics.Success rate measures episodes satisfying all requirements, while average score is the mean reward across episodes.
  • Baselines: LASER solves WebShop through state exploration over pages and navigation actions.Its states include search, item, and item-detail pages.
  • Baselines: LATS uses Monte Carlo Tree Search to evaluate potential moves and balance exploration with exploitation.It adapts its strategy using successes and failures while navigating shopping tasks.
Loading 2312.04511v3…