PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowCompile: An Optimizing Compiler for Structured LLM Workflows

Junyan Li, Zhang-Wei Hong, Maohao Shen, Yang Zhang, Chuang Gan

arXiv:2605.13647v1cs.CL

TL;DR

Structured LLM workflow optimization faces a combinatorial design space, while routing-based methods usually target one runtime trade-off. FlowCompile compiles workflows before deployment by profiling sub-agents and composing their profiles through a structure-aware proxy, producing reusable configurations across trade-offs. Across diverse workflows and benchmarks, it consistently improves accuracy–latency trade-offs and reports up to 6.4x speedup.

  • Problem

    Structured workflow optimization jointly varies model choices, reasoning budgets, and workflow structures, while existing routing methods typically target a single trade-off point.

  • Method

    FlowCompile performs compile-time design-space exploration by profiling sub-agents, composing profiles with a workflow-level proxy, and producing reusable configurations.

  • Results

    FlowCompile consistently achieves lower latency at comparable or higher accuracy than baselines across four benchmarks, with up to 6.4x speedup.

  • Takeaways & Limitations

    The compiled configuration set supports deployment under diverse runtime preferences and can serve as a reusable artifact for selection or routing.

  • Takeaways & Limitations

    FlowCompile is scoped to structured workflows with predefined execution graphs and relies on an approximate proxy whose assumptions require empirical validation.

Abstract

from arXiv · show

Structured LLM workflows, where specialized LLM sub-agents execute according to a predefined graph, have become a powerful abstraction for solving complex tasks. Optimizing such workflows, i.e., selecting configurations for each sub-agent to balance accuracy and latency, is challenging due to the combinatorial design space over model choices, reasoning budgets, and workflow structures. Existing cost-aware methods largely treat workflow optimization as a routing problem, selecting a configuration at inference time for each query according to the accuracy-latency objective used during training. We argue that structured LLM workflows can also be optimized from a compilation perspective: before deployment, the system can globally explore the workflow design space and construct a reusable set of workflow-level configurations spanning diverse accuracy-latency trade-offs. Drawing inspiration from machine learning compilers, we introduce FlowCompile, a structured LLM workflow compiler that performs compile-time design space exploration to identify a high-quality, reusable trade-off set. FlowCompile decomposes a workflow into sub-agents, profiles each sub-agent under diverse configurations, and composes these measurements through a structure-aware proxy to estimate workflow-level accuracy and latency. It then identifies diverse high-quality configurations in a single compile-time pass, without retraining or online adaptation. Experiments across diverse workflows and challenging benchmarks show that FlowCompile consistently outperforms heuristically optimized workflow configurations and routing-based baselines, delivering up to 6.4x speedup. The compiled configuration set further serves as a reusable optimization artifact, enabling flexible deployment under varying runtime preferences and supporting downstream selection or routing.

1 Introduction

Structured LLM workflows create a combinatorial accuracy–latency optimization problem that existing routing methods address mainly at inference time. FlowCompile instead performs compile-time exploration to produce reusable trade-off configurations.

  • Motivation: Structured LLM workflows connect specialized sub-agents through predefined sequential, parallel, branching, or iterative execution graphs.Their explicit graphs support systematic workflow-level analysis and optimization.
  • Motivation: The design space jointly varies sub-agent models, reasoning budgets, and workflow structures, requiring multiple operating points rather than one fastest implementation.These choices trade output quality against inference cost and support diverse deployment requirements.
  • Prior approach: Existing workflow optimization methods largely use inference-time routing policies trained for a specified accuracy–latency objective.Each policy typically targets one trade-off point and must be retrained or re-optimized for different deployment requirements.
  • FlowCompile: FlowCompile performs a single compile-time search over model choices, reasoning budgets, and workflow structures to produce a reusable set of configurations.It profiles sub-agents, composes their measurements through a workflow-level proxy, and explores the resulting design space.
  • Contribution: FlowCompile introduces workflow compilation as a compiler-inspired paradigm for producing reusable accuracy–latency trade-off sets before deployment.The approach uses a structure-aware compositional proxy to lift reusable sub-agent profiles to workflow-level estimates.

2 Related Work

Prior work optimizes structured LLM workflows mainly through runtime routing, while FlowCompile explores workflow-level configurations before deployment and returns a reusable accuracy–latency set.

  • Structured LLM workflow optimization: Existing efficiency-oriented methods make inference-time decisions over models, collaboration strategies, or backend allocation.Representative routing-based methods include MaAS, MasRouter, DAAO, and related online policies.
  • Positioning: FlowCompile differs from prompt-focused compilation by optimizing workflow-level execution trade-offs rather than prompts and demonstrations.DSPy primarily targets pipeline accuracy through prompt and demonstration optimization.
  • Positioning: FlowCompile performs compile-time workflow design-space exploration and produces reusable configurations without retraining or online adaptation.This formulation spans diverse accuracy–latency trade-offs instead of learning an inference-time routing policy.
  • Machine learning compilers: Machine learning compilers decompose computational graphs into lower-level optimization units and search implementation choices under cost models.FlowCompile draws on this decomposition but targets structured LLM workflow quality–efficiency trade-offs.

3 FlowCompile

FlowCompile models structured workflows as configurable graphs, profiles sub-agents, estimates workflow performance compositionally, and searches for reusable frontier configurations.

  • Problem definition: A structured workflow is represented as W = (A, G), where sub-agents A are connected by an execution graph G.The graph specifies sequential, parallel, conditional, or iterative control flow.
  • Problem definition: Each configuration assigns sub-agent models and reasoning budgets and may select structural options, producing workflow-level accuracy and latency.The configuration is evaluated on a labeled validation set as y(c) = (Acc(c), Lat(c)).
  • Problem definition: The compiler seeks a reusable set spanning the accuracy–latency space because exhaustive validation-set evaluation is infeasible in the combinatorial design space.A five-sub-agent workflow with five models and four budgets already yields 3.2M model–budget assignments before structural choices.
  • Pipeline: FlowCompile’s pipeline comprises sub-agent profiling, workflow-level estimation, and design-space exploration.It constructs reusable profiles, composes them through lightweight estimation, and searches for configurations across deployment trade-offs.
  • Sub-agent profiling: Sub-agent profiling evaluates candidate model–budget configurations on induced datasets and records reusable empirical accuracy and latency profiles.The resulting profiles form component-level cost models reused during workflow-level exploration.
  • Workflow-level proxy: A structure-aware proxy composes sub-agent profiles according to the instantiated graph and execution model to estimate workflow accuracy and latency.The proxy need not predict absolute values exactly; it should preserve frontier structure and local ordering for reliable search.
  • Workflow-level proxy: FlowCompile uses lightweight analytical composition rules that are training-free and designed to generalize across workflow structures and deployment settings.Sequential, conjunctive-parallel, and conditional rules are combined recursively into a workflow accuracy estimator.
  • Validation: Proxy validation reports that lightweight composition over reusable profiles can identify high-quality configurations without costly end-to-end workflow execution.The proxy is an approximation, so its assumptions are empirically evaluated before relying on it for configuration search.

4 Experiments

FlowCompile validates a structure-aware proxy and evaluates compiled configurations across accuracy–latency trade-offs, preference-aware utility, transfer, and ablations. Across these evaluations, the proxy preserves useful configuration structure and the compiled set supports strong, reusable deployment choices.

  • Settings: FlowCompile evaluates four benchmarks spanning mathematical reasoning, multi-hop question answering, and code reasoning, using task accuracy, end-to-end latency, and expected utility.Latency is measured on a single H100 GPU with vLLM; profiling and evaluation use disjoint subsets.
  • Proxy Validation: The proxy matches the empirically measured HotpotQA frontier in a restricted design space and largely preserves high-quality configuration ordering in the full space.Average pairwise agreement is 0.90 for accuracy and 0.95 for latency, with average errors of 2.3 percentage points and 4.4 seconds after bias correction.
  • End-to-End Quality of Compiled Configurations: FlowCompile produces substantially better measured accuracy–latency trade-offs than baselines and achieves the highest expected utility across four benchmarks under heterogeneous preferences.It outperforms the strongest baseline by an average of +7.9 expected utility; under fixed preferences, it leads for every preference value.
  • Additional Analysis: Compiled configurations form interpretable operating regimes, with latency-priority choices favoring simpler workflows and lower-cost models or budgets, while accuracy-priority choices allocate compute to task-critical stages.This analysis complements the measured frontier and preference-aware evaluations.
  • Additional Analysis: Routing complements compilation by selecting among FlowCompile’s high-quality candidate pool according to query-conditioned difficulty, while compilation constructs the reusable trade-off set.The compiled set supports lightweight selection rather than repeated search over the full combinatorial design space.
  • Additional Analysis: Profiles transferred from MATH-500 to GSM8K preserve strong latency correlation, reasonable accuracy correlation, and competitive expected utility without GSM8K re-profiling.The transfer result suggests reduced profiling cost across related tasks sharing the same workflow.

5 Conclusion

FlowCompile reframes structured workflow optimization as compile-time exploration that produces reusable accuracy–latency trade-off configurations. It profiles sub-agents, composes workflow estimates, and selects non-dominated configurations without online adaptation.

  • Conclusion: FlowCompile treats workflow optimization as a reusable compile-time artifact rather than only runtime routing.The compiler produces configurations that can be selected under diverse deployment requirements.
  • Conclusion: The compilation algorithm profiles sub-agents under candidate model and reasoning-budget configurations before enumerating workflow configurations.It receives a workflow, validation data, model choices, reasoning budgets, and an execution model as inputs.
  • Conclusion: FlowCompile removes locally dominated sub-agent configurations and applies non-dominated sorting to proxy-estimated workflow accuracy and latency.Higher accuracy and lower latency define preference when constructing the resulting configuration set.

B.1 Dataset Splits

The experiments use disjoint profiling and held-out test data across four benchmarks, with workflow structures and sub-agent behavior modeled through a structure-aware proxy. Compilation composes cached profiles rather than repeatedly executing complete workflows.

  • Dataset Splits: A 1:4 split uses 20% of examples for profiling and 80% for held-out testing unless otherwise specified.The profile set supports profiling, configuration selection, and baseline training; reported metrics use held-out examples.
  • Dataset Splits: GSM8K and MATH-500 use disjoint benchmark splits, while HotpotQA samples 1,000 examples and LiveCodeBench uses released code-generation problems.Each benchmark is divided into disjoint profile and test sets.
  • Workflow Configuration: FlowCompile treats workflow structure as configurable, using optional branches and profiled aggregation modules across benchmark-specific workflows.The proxy composes profiled sub-agent accuracies and latencies according to each workflow structure.
  • Compilation Cost: The profiling stage can require up to D × A × M × R independent sub-agent inference calls and is parallelizable across configurations and roles.In the implementation, 32 H100 GPUs and batch size 256 yield typical profiling time of about 1 hour per benchmark.
  • Compilation Cost: Workflow-level composition evaluates candidate configurations numerically from cached profiles, with worst-case space O((1 + MR)^A).After pruning, vectorized composition completes in under one second on a CPU, and non-dominated sorting uses O(N log N).

C.2 Comparison with Exhaustive Workflow Evaluation

FlowCompile avoids infeasible exhaustive end-to-end evaluation by profiling sub-agents once and reusing cached statistics for workflow-level composition. This shifts expensive inference from a combinatorial workflow space to a smaller, parallelizable profiling problem.

  • Comparison with Exhaustive Workflow Evaluation: The full workflow spaces contain approximately 2.37B, 4.84B, 2.56M, and 1.04M configurations for GSM8K, MATH-500, LiveCodeBench, and HotpotQA.Exhaustively evaluating every configuration end-to-end is computationally infeasible in this setting.
  • Comparison with Exhaustive Workflow Evaluation: FlowCompile profiles only 325, 375, 225, and 240 sub-agent settings across the four benchmarks, with profiling taking about one hour per benchmark.On HotpotQA, exhaustive evaluation would require roughly 173,333 hours because one full-workflow evaluation takes approximately 10 minutes.
  • Comparison with Exhaustive Workflow Evaluation: Exact sub-agent-level Pareto filtering removes locally dominated choices before workflow composition under the proxy’s monotone accuracy and latency rules.This reduces each sub-agent role from roughly 65–80 settings to 6–19 settings.
  • Comparison with Exhaustive Workflow Evaluation: Profiling scales with sub-agent roles and candidate settings, whereas independent workflow assignments grow combinatorially as (MR)^A before additional structural choices.Cached profiles are reused across workflow configurations, deployment preferences, and latency budgets.
  • Comparison with Exhaustive Workflow Evaluation: FlowCompile shifts optimization from combinatorial end-to-end evaluation to linear, reusable, and parallelizable sub-agent profiling.This supports compile-time exploration even when workflow spaces contain millions or billions of configurations.

D Additional Details on Frontier Consistency under Exhaustive Evaluation

Restricted exhaustive evaluations test whether FlowCompile’s proxy preserves the empirical accuracy–latency frontier. The selected configurations match or cover high-quality operating points while most alternatives are dominated.

  • Frontier Consistency: Exhaustive evaluation compares proxy-estimated and empirically measured frontiers on restricted HotpotQA and LiveCodeBench design spaces.The experiments make end-to-end evaluation feasible by limiting model choices, reasoning budgets, and workflow structures.
  • Frontier Consistency: The proxy frontier matches the measured HotpotQA frontier and forms a high-quality empirical frontier on LiveCodeBench.Most configurations outside the proxy frontier are dominated after workflow execution.
  • Representative Operating Points: Table 5 reports accuracy-priority and latency-priority selections from FlowCompile’s compiled configuration set.The accuracy-priority selection targets the full Qwen3-14B workflow’s accuracy with lower latency, while the latency-priority selection targets larger reductions with strong performance.
  • Frontier Consistency: The selected configurations cover trade-off regimes from low latency to high accuracy, supporting the proxy’s use for compile-time search.These results provide evidence that the proxy preserves the non-dominated region sufficiently well.
  • Representative Operating Points: On HotpotQA, the accuracy-priority configuration improves F1 from 86.29 to 86.69 while reducing latency from 26.5s to 7.8s.On LiveCodeBench, it reduces latency from 329.3s to 51.5s while maintaining similar Pass@1.

F Additional Details on Preference-Aware Evaluation

FlowCompile evaluates compiled configurations across fixed and heterogeneous accuracy–latency preferences, selecting with proxy estimates while reporting measured test-set utility. Across benchmarks and most preference values, it achieves the strongest overall expected utility.

  • Evaluation protocol: FlowCompile selects compiled configurations using proxy-estimated accuracy and latency, while reported utilities use measured test-set performance.This separation avoids test-set leakage.
  • Evaluation protocol: In heterogeneous workloads, each query receives an independently sampled preference α ∼Uniform(0, 1), representing mixed latency-sensitive and accuracy-sensitive requests.Selection maximizes proxy-estimated utility for each sampled preference.
  • Evaluation protocol: Fixed-preference evaluation sweeps α from 0.05 to 0.95 in steps of 0.05, testing robustness across global accuracy–latency preferences.Figure 7 reports per-benchmark results for this setting.
  • Results: FlowCompile achieves the highest expected utility across benchmarks and most preference values, outperforming alternatives across the preference spectrum overall.Pref-Aware MaAS is competitive at isolated α values on GSM8K and MATH-500.
  • Downstream selection: Table 6 evaluates per-query routing over compiled configurations as a downstream selection mechanism.The supplied caption identifies routing over the compiled set as the table’s focus.

G Additional Details and Analysis on Per-Query Routing

FlowCompile constructs a compact global configuration set that can support lightweight query-conditioned routing. Routing improves utility and allocates more compute to harder MATH-500 questions when preferences justify the added latency.

  • Compile-time and runtime: FlowCompile produces a global accuracy–latency trade-off set that complements runtime routing by restricting selection to a compact configuration pool.The router need not search the full combinatorial design space online.
  • Utility gains: A lightweight KNN router improves expected utility from 89.2 to 91.8 on GSM8K and from 84.2 to 90.5 on MATH-500.This requires no additional workflow profiling, online optimization, or retraining.
  • Routing analysis: Difficulty labels are used only for post hoc analysis, while routing selects configurations from the compiled set without observing those labels.The analysis tests whether selected configurations correlate with problem difficulty.
  • Routing patterns: In the Latency-first regime, Simple workflows dominate across difficulty levels and mean proxy latency remains low.Latency-prioritized routing preserves inexpensive execution even for harder questions.
  • Routing patterns: In Balanced routing, Complex workflow share rises from 1.9% to 11.4% and latency from 10.6s to 15.9s between difficulty levels 1 and 5.In Accuracy-first routing, Complex share rises from 18.6% to 39.6% and latency from 20.8s to 30.3s.
  • Latency allocation: Per-query routing creates difficulty-sensitive latency, assigning easier questions cheaper configurations and harder questions higher-latency configurations.Without routing, a single global configuration yields essentially constant latency across difficulty levels.
  • Implication: The compiled set functions as a deployment menu, while KNN routing performs lightweight query-conditioned selection within it.Harder questions are more often assigned workflows with additional reasoning branches and higher latency when extra compute is likely to help.

H Additional Details on Reference-Model Ablation

A HotpotQA ablation evaluates whether FlowCompile depends on its reference model for sub-agent profiling. The resulting optimized configuration sets show largely consistent measured performance across reference models.

  • Ablation setup: FlowCompile induces sub-agent profiling data from workflow traces using a reference model, then repeats profiling, estimation, and frontier search with alternatives.The ablation compares GPT-5 with GPT-5-mini and Qwen3-1.7B on HotpotQA.
  • Results: Optimized configuration sets achieve largely consistent measured performance across reference models.Figure 10 reports similar optimized trade-off sets for different reference models.
  • Scope: The result suggests limited coupling to a particular high-capacity reference model when the reference model can generate reasonable workflow traces.This condition defines the stated scope of the robustness observation.
  • Artifact analysis: Compiled configurations are summarized using workflow-structure classes across latency-ranked regions of the compiled artifacts.The supplied table caption identifies Simple, Balanced, and Complex classes as the qualitative summary.

I Analysis of Compiled Workflow Configurations

FlowCompile’s compiled configurations form interpretable accuracy–latency operating regions, with benchmark-specific changes in structure, reasoning budget, model capacity, and repair behavior. Evaluation uses expected utility and proxy-validation metrics to compare estimated and measured configuration behavior.

  • Analysis setup: The analysis aggregates workflow structure, reasoning budget, and model choices across the compiled trade-off regions.Figures 11 and 12 provide aggregated views of these dimensions.
  • Analysis setup: Compiled configurations are grouped into Latency-priority, Balanced-priority, and Accuracy-priority regions on the accuracy–latency frontier.These regions are distinct from workflow-structure classes.
  • Overall pattern: Across benchmarks, compiled configurations show interpretable rather than arbitrary combinations of models, budgets, and workflow variants.The dominant changes across regions involve workflow structure, reasoning budget, and model capacity.
  • Math reasoning workflows: For GSM8K and MATH-500, accuracy-priority configurations increasingly favor Complex workflows over the Simple configurations common in latency-priority regions.Accuracy is obtained through selective activation of additional reasoning stages alongside larger budgets or stronger models.
  • Multi-hop QA workflows: HotpotQA keeps the same Simple workflow structure across the trade-off range, increasing model capacity and budget instead of activating additional sub-agents.The dominant choice is how much capacity to allocate within the fixed workflow.
  • Code reasoning workflows: LiveCodeBench keeps stable active sub-agents but increases execution-guided repair trials and budgets toward accuracy-priority configurations.Balanced and accuracy-priority regions differ mainly in budget and model allocation.
  • Implication: The compiled set acts as a deployment menu whose configurations spend compute on additional reasoning paths, stronger generation and formatting, or more repair attempts.The allocation depends on the workflow and target operating region.
  • Expected utility: Expected utility combines task accuracy with latency efficiency under preference α, with larger α weighting accuracy more heavily.Accuracy is F1 for QA tasks and Pass@1 for LiveCodeBench.

L Limitations

FlowCompile is scoped to structured workflows with predefined execution graphs and interfaces, and its compile-time proxy remains an approximation. Its optimization layer also does not guarantee application-level safety or reliability.

  • Scope boundary: FlowCompile is less directly applicable to open-ended agentic systems whose execution traces are dynamically constructed at inference time.Extending it to such settings would require trace abstraction or online workflow-graph construction.
  • Proxy limitation: The workflow-level proxy is an approximation whose effectiveness depends on how well independently induced sub-agent profiles capture dominant sub-agent interactions.These interactions include intermediate inputs produced by different upstream configurations.
  • Proxy limitation: The proxy’s absolute predictions for some low-quality configurations may be less accurate, despite validated frontier consistency and local order preservation on evaluated workflows.More explicit modeling of distribution shifts could improve estimation accuracy for more heterogeneous and complex workflows.
  • Deployment boundary: FlowCompile does not by itself guarantee the safety, fairness, privacy, or factual reliability of workflow outputs.High-stakes applications require domain-specific validation, monitoring, and safeguards.
Loading 2605.13647v1…