PaperlayerView exact revision on arXiv

Source-linked AI summary

Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimization

Xuehang Guo, Haoyu Wang, Shengyu Chen, Zach Chen, Wei Cheng, Qingyun Wang, Haifeng Chen

arXiv:2610.01017v1cs.AIcs.CL

TL;DR

Workflow construction and repair are challenging because agent fit and subtask success are uncertain, while fault localization and whole-workflow fixes can require supervision or costly reruns. InFlowOp uses a shared label-free cost for workflow construction and in-flow correction; across domains and backbones, it improves over single-agent and workflow baselines.

  • Problem

    Workflow constructors must choose task granularity and agent assignments under uncertainty, while improving a running workflow is costly and often relies on supervision or whole-flow fixes.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    InFlowOp uses a label-free cost balancing agent competence against execution cost to construct workflows and correct faulty subtasks locally during execution.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 7

  • Results

    Across eight domains and six backbones, InFlowOp improves over single-agent baselines by +7.15% to +11.97% under a matched turn budget and outperforms workflow baselines by up to +9.64%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Braid provides workflow-level evaluation on tasks requiring multi-agent coordination beyond single-agent competence.

    Source: §1 · p. 2

    Tasks without a checkable reference answer, including open-ended generation and long-horizon interaction with a changing stateful environment, are outside this study.

    Source: B LIMITATIONS · p. 22

Abstract

from arXiv · show

Large language models (LLMs) increasingly construct multi-agent workflows that decompose a complex task and assign specialist agents from a pool. However, building such a workflow well remains challenging: how finely to divide the task, which agent to trust with each subtask, and when to create a new specialist are all critical decisions a workflow constructor needs to settle up front. Thus, whether each subtask succeeds remains unknown until the workflow runs. Yet, improving a workflow is costly. Locating a fault usually requires a reference answer, a graded outcome, or a trained assessor, and the fix is applied to the whole workflow through re-execution, re-search, or retraining. We propose InFlowOp, which prices every decision in one label-free cost that weighs how well an agent's competence meets what a subtask demands against how much that agent takes to run. Before execution, InFlowOp bidirectionally determines the granularity of task decomposition and agent assignment following from the cost rather than from a fixed template. During execution, InFlowOp corrects a fault with the cheapest move via the same cost that serves the workflow both as it is built and as it runs. Facing the workflow-level evaluation challenge, we introduce Braid, a benchmark whose tasks require multi-agent coordination beyond single-agent capability. Across various domains and backbones, InFlowOp outperforms single agent baselines by up to $+11.97\%$, achieving $+9.64\%$ with in-flow optimization. Our project page: https://xhguo7.github.io/InFlowOp/.

1 INTRODUCTION

Multi-agent workflows offer explicit task division and specialist assignment, but constructing and improving them remains difficult because agent fit and subtask success are uncertain. InFlowOp uses one label-free cost to construct workflows and correct faults in-flow, and Braid evaluates coordination beyond single-agent capability.

1 INTRODUCTION

  • Existing workflow decisions often fix decomposition in advance, while fault diagnosis and repair have relied on supervision or whole-workflow fixes.The evaluated tasks have also largely been ones a single competent agent can handle.Source: §1 · p. 2
  • InFlowOp uses the same label-free cost during execution to identify and correct subtasks locally without losing progress.Its framework requires no reference answer, graded outcome, or trained assessor.Source: §1 · p. 2
  • Across eight domains and six backbones, InFlowOp improves over single-agent baselines by +7.15% to +11.97% under a matched turn budget and outperforms workflow baselines by up to +9.64%.The evaluation spans four workflow constructors.Source: §1 · p. 2
  • COALESCE uses cost-driven bidirectional workflow construction to choose decomposition granularity, creating an agent only when no available agent clears a competence bar.The cost weighs agent competence against execution cost.Source: §1 · p. 2
  • Braid is a workflow-level benchmark whose tasks require multi-agent coordination beyond single-agent competence.The benchmark addresses the challenge of evaluating workflows on tasks a single agent cannot solve effectively.Source: §1 · p. 2

2 PRELIMINARIES

The paper frames workflow construction as assigning specialists to subtasks and organizing their dependencies to maximize performance at minimal cost. It identifies uncertainty in agent fit and execution, alongside the expense of acting on failures, as central challenges.

2 PRELIMINARIES

  • A workflow constructor assigns specialist agents from a pool to a task, then execution measures performance against a reference answer; the objective is high performance at minimal cost.The formulation separates workflow construction from performance optimization.Source: §2 · p. 2
  • Construction must balance reliability, execution cost, and the cost of improvement when a workflow falls short.More subtasks and agents can increase latency, while full re-execution, re-search, or training makes improvement costly.Source: §2 · p. 3
  • Agent competence varies, available pools may lack a suitable specialist, and whether an assigned agent succeeds becomes known only during execution.A step that satisfies its own declared requirements can still mislead later steps.Source: §2 · p. 3
  • Re-execution or re-search discards prior progress, while post-training requires data preparation and computation that can exceed test-time re-execution.The paper identifies lower-cost workflow improvement as a core challenge.Source: §2 · p. 3
  • The agent estimator uses cards combining declared competence with an empirical profile of successes and failures accumulated through execution.Observed competence progressively complements the agent's written brief.Source: §2 · p. 3

3 INFLOWOP: THE FLOW AS BUILT, THE FLOW AS RUN

INFLOWOP uses one annotation-free cost to guide workflow construction and optimization during execution. It determines task granularity and agent assignment, then diagnoses and corrects faults locally as they arise.

3 INFLOWOP: THE FLOW AS BUILT, THE FLOW AS RUN

  • One annotation-free cost couples workflow reliability and execution latency to guide construction before execution and adaptive optimization during execution.The method minimizes this cost in two stages.Source: §3 · p. 3

3.1 BIDIRECTIONAL DECOMPOSITION-AWARE WORKFLOW CONSTRUCTION

  • A label-free cost matrix matches atom demands against agent competence and profiles, while its coverage residual flags atoms no pooled agent can be trusted with for creation at penalty γ.The matrix also includes agent latencies to guide granularity.Source: §3.1 · pp. 4–5
  • COALESCE decomposes tasks into fine atoms, then merges them where coarser subtasks lower overall cost and assigns each to a lowest-cost existing or newly created agent.Granularity emerges from cost-driven coalescing rather than a fixed choice.Source: §3.1 · pp. 4, 6

3.2 IN-FLOW DYNAMIC OPTIMIZATION

  • Execution pauses at the faulty subtask, corrects it locally, and resumes over the affected dependency closure while retaining results computed outside it.The correction cost is bounded by that closure rather than the whole workflow.Source: §3.2 · pp. 6–7
  • In-flow detection matches declared conditions against realized inputs and outputs; an unmet condition identifies which side breached the contract, while unusable output is faulty by observation.Input conditions are checked against realized inputs and output conditions against realized outputs.Source: §3.2 · p. 7
  • The cost-ordered correction ladder starts with reassignment for assignment faults and re-decomposition for decomposition faults, pricing both moves with the construction cost model.Reassignment leaves the subtask graph unchanged; re-decomposition updates it and incurs a fresh COALESCE run.Source: §3.2 · p. 7

4 BRAID: BENCHMARK REASONING OVER AGENT-INTERDEPENDENT DIVISIONS

BRAID composes tasks at different complexity levels to test coordination beyond single-agent capability. It evaluates answers with type-specific rubrics and reports mean accuracy by domain.

4 BRAID: BENCHMARK REASONING OVER AGENT-INTERDEPENDENT DIVISIONS

  • BRAID composes tasks using distributed or anchored composition at different complexity levels, and includes 19 evaluation-only arms across eight domains.The benchmark is built on 10 public datasets and applies four quality-control criteria.Source: §4 · p. 8
  • BRAID scores each question separately with answer-type-specific rubrics, averages question scores over each task’s k questions, then averages task accuracy by dataset.Rubrics cover numeric, mathematical, textual, multiple-choice, list, and coding answers.Source: §4 · p. 8
  • Figure 3 reports mean accuracy by domain for three models on the whole benchmark, with rows for domains and comparisons across two baselines and INFLOWOP.Its horizontal axis is accuracy (%); Table 1 covers the five domains evaluated for every model.Source: Figure 3 · p. 9

5 EXPERIMENTS

On BRAID, COALESCE-based construction outperformed greedy construction and the single-agent baseline, with further gains from in-flow tempering and profile scoring. Ablations examine the effects of cost estimation, agent-pool and matrix updates, profile history, and global optimization.

5.1 EXPERIMENT SETUP

  • The six-system comparison includes single-LLM and single-agent baselines, greedy and COALESCE workflows, in-flow tempering, and profile-based scoring.The evaluation covers six backbones and BRAID: 19 evaluation arms over 14 datasets from 10 public datasets across eight domains.Source: §5.1 · p. 9

5.2 WHAT THE FLOW EARNS AND WHAT EARNS THE FLOW

  • 22.21%: COALESCE exceeded greedy search by +6.72% and the single-agent baseline by +4.83%; in-flow tempering and profile scoring added +1.59% and +1.33%.These mean accuracies are for Qwen3.5-9B on BRAID.Source: Table 2 · p. 10, §5.2 · p. 10
  • Across domains, gains were largest on charts, mathematics, and finance and smallest on physics, where every method stayed at or below 17.34%.The improvement trend also held on slides, science, and code for the three backbones evaluated on the whole benchmark.Source: §5.2 · p. 10

5.3 WHAT IMPROVES THE FLOW AND WHAT WEIGHS ON IT

  • −3.27% and −3.85%: the LLM cost estimator lost these accuracy points against the rubric estimator on Qwen3.5-4B and Qwen3.5-9B, respectively.Both estimators received the same agent cards, evolving profiles, and atom demands.Source: Figure 4 · p. 10, §5.3 · p. 10
  • −1.63% and −1.41%: adding a global pass to local in-flow optimization reduced accuracy on Qwen3.5-4B and Qwen3.5-9B, respectively.The authors report that this supports keeping in-flow optimization local in the other experiments.Source: Figure 4 · p. 10, §5.3 · p. 11
  • +3.15% and +2.55%: a dynamic cost matrix outperformed a static matrix on Qwen3.5-4B and Qwen3.5-9B, respectively.The dynamic matrix incorporates newly created agents for matching against atoms.Source: §5.3 · p. 10, Figure 5 · p. 11
  • +4.49% and +3.29%: a dynamic agent pool outperformed a static pool on Qwen3.5-4B and Qwen3.5-9B, respectively.The dynamic pool creates a specialist when no available agent fits an atom well.Source: Figure 5 · p. 11, §5.3 · p. 11
  • Profiles improved scoring when they retained successful evidence; profiles containing all outcomes scored below card-only scoring across all three backbones.Within-run profile gains over the card alone were +2.54%, +1.33%, and +2.23% for Qwen3.5-4B, Qwen3.5-9B, and GPT-5.4-mini; keeping only successful outcomes across runs added +1.21%, +1.19%, and +1.30%, respectively. All-outcome profiles scored 17.34%, 21.52%, and 26.53%, below card-only scoring for those backbones.Source: Figure 6 · p. 11, §5.3 · p. 11

6 CONCLUSIONS

The conclusion presents InFlowOp as a two-stage framework for workflow construction and in-flow optimization. The related-work sections identify gaps in cost-based construction, label-free fault correction during execution, and benchmarks whose tasks themselves require coordination.

6 CONCLUSIONS

  • InFlowOp combines bidirectional decomposition-aware construction with in-flow dynamic optimization, and the paper introduces BRAID for tasks requiring multi-agent coordination beyond single-agent capability.The authors evaluate across eight domains and six backbones against single-model and workflow baselines, and ablate design choices.Source: §6 · p. 11

A.1 WORKFLOW CONSTRUCTION OVER AN AGENT POOL

  • Prior construction methods adapt task grain to templates, queries, or proposed divisions, while agent creation and reuse are not jointly priced per part before execution.The missing approach is a single price over the available pool under which agent creation participates in the same search that sets task granularity.Source: §A.1 · p. 21

A.2 OPTIMIZING A WORKFLOW AFTER EXECUTION

  • Existing workflow improvement uses training, repeated execution or search, or output revision; within-flow fault localization often requires labels, replay, or domain-checkable relations.The identified gap is locating a fault during a single execution and correcting only that part without training, reference answers, or graded outcomes.Source: §A.2 · p. 21

A.3 EVALUATING MULTI-AGENT WORKFLOWS

  • Single-model suites may not reveal the contribution of dividing work, while existing coordination benchmarks often impose division through the environment rather than task demands.The stated gap is a benchmark whose tasks require different competences across parts and depend on how those parts combine.Source: §A.3 · pp. 21–22

B LIMITATIONS

The evaluation is limited to BRAID tasks with checkable answers, and the authors identify open-ended generation and long-horizon interaction with changing environments as future extensions.

B LIMITATIONS

  • BRAID tasks are graded against a reference answer for each question, excluding settings without a checkable answer.The authors name open-ended generation and long-horizon interaction with a stateful environment as examples outside this study.Source: B LIMITATIONS · p. 22
  • The evaluation covers BRAID tasks spanning eight domains.Source: B LIMITATIONS · p. 22
  • The authors plan to extend InFlowOp to settings without checkable answers, different domains, and workflows acting on changing environments.Source: B LIMITATIONS · p. 22

C WHERE A WORKFLOW BECOMES NECESSARY: BRAID FOR WORKFLOW-LEVEL EVALUATION

BRAID targets workflow-level evaluation with tasks requiring coordination beyond a single agent’s capability. It composes tasks using two methods under quality control and evaluates workflows by mean accuracy across each task’s questions.

C WHERE A WORKFLOW BECOMES NECESSARY: BRAID FOR WORKFLOW-LEVEL EVALUATION

C.1 WHY WORKFLOW-LEVEL EVALUATION IS NECESSARY

A preliminary comparison finds that a workflow is less accurate and more costly than a single agent on original data, but performs better on BRAID under matched compute budgets. The contrast motivates evaluating workflows on tasks that require multi-agent coordination.

C.1 WHY WORKFLOW-LEVEL EVALUATION IS NECESSARY

  • −12.02% accuracy and −6.91 cost: on original data, the workflow trails the single agent while costing more.Figure 7 compares accuracy and cost; cost is measured in units of one single-agent run on the original data.Source: §C.1 · p. 22, Figure 7 · p. 23
  • On BRAID’s composed tasks, the single agent degrades significantly while the workflow performs better under a matched compute budget.Source: §C.1 · p. 22, Figure 7 · p. 23
  • Tasks solvable by one competent agent cannot effectively measure a workflow’s performance or contribution.Reliable workflow-level evaluation instead requires tasks demanding coordination beyond single-agent capability.Source: §C.1 · p. 23

C.2 BENCHMARKING WORKFLOW-LEVEL EVALUATION

BRAID builds multi-question tasks that require dividing work and routing questions to their sources. Its distributed and anchored compositions, quality controls, and broad dataset coverage define the benchmark’s evaluation setting.

C.2 BENCHMARKING WORKFLOW-LEVEL EVALUATION

  • BRAID tasks contain k questions and provided sources, requiring task division and coordinated routing from each question to its answer.Routing means locating the source that answers a question and reasoning from that source to its answer.Source: §C.2 · p. 23
  • Distributed tasks assign each question a different gold source and add k−1 distractors; anchored tasks share one gold source and add k−1 distractors.Thus distributed tasks contain 2k−1 sources, while anchored tasks contain k sources.Source: §C.2 · pp. 23–24
  • Distractors are drawn according to input type, with seeded selection favoring least-used sources to support reproducible composition without source dominance.Source: §C.2 · p. 24
  • BRAID rejects tasks that fail separation, routing, leakage, or answerability checks, and randomizes source order and uninformative file names.Source: §C.2 · p. 24
  • BRAID draws on ten public datasets across eight domains; anchored composition is available only for datasets whose sources carry at least k questions.A complexity level is retained only when it yields at least 80 questions; DocFinQA at k = 5 is dropped for insufficient size.Source: §C.2 · p. 24
  • BRAID comprises nineteen arms over fourteen BRAID datasets, adapted from ten public datasets across eight domains.Source: §C.2 · p. 24

C.3 EVALUATION METRICS

This block defines answer-type-specific evaluation and describes how InFlowOp constructs, prices, and corrects workflows using a shared label-free estimator. Its cost balances reliability, latency, and agent creation.

C.3 EVALUATION METRICS

  • Questions are graded by answer type: numeric value with optional tolerance, symbolic equivalence, normalized text or token F1, selected option, or position-wise list scoring.For short-span datasets, a gold appearing within the answer can count fully, and the best score among accepted golds is used.Source: §C.3 · pp. 25–26
  • Coding answers count only when every test passes; grading questions separately preserves partial task success rather than collapsing a task to all-or-nothing.Task accuracy averages question scores, while dataset accuracy averages task scores.Source: §C.3 · pp. 25–26

D.1 INFLOWOP OVERVIEW: FROM THE FLOW AS BUILT TO THE FLOW AS RUN

  • Workflow construction and in-flow correction reuse one estimator: it prices atoms against agents to choose task granularity, assignments, and contracts, then detects breaches and corrects subtasks locally.Assignment faults can be addressed with another ranked agent; decomposition faults trigger re-decomposition, and a correction is kept only after a clean execution.Source: §D.1 · pp. 26–27

D.2 META PRIMITIVES

  • COALESCE grows partial groupings from atomic tasks, prunes branches whose lower bound meets the incumbent, and assigns each subtask its lowest-cost covering agent.When no available agent meets the competence bar, it creates one at the stated creation penalty.Source: §D.1 · p. 27, §D.2 · p. 28

D.3 TAXONOMY OF COST

  • Workflow cost combines reliability cost, critical-path latency weighted by β, and γ per created agent; sweeping β trades finer, more reliable workflows for faster, compact ones.Because the workflow graph admits parallelism, latency is measured along its critical path rather than summed across agents.Source: §D.3 · p. 30

D.4 A STEP IS WORTH WHAT IT LEADS TO

An intermediate step’s value depends on the final flow outcome, not merely whether it meets its local condition. When local correction fails to yield a usable answer, optional global localization ranks possible upstream faults while keeping corrections local.

D.4 A STEP IS WORTH WHAT IT LEADS TO

  • A condition-compliant output can mislead the flow, while a condition-violating output can lead to the correct answer; intermediate steps are therefore judged by downstream outcomes.The example also shows why step-level ground-truth benchmarks are excluded: they can penalize behavior that produces the correct final answer.Source: §D.4 · p. 30, Figure 8 · p. 31

D.5 TEMPER LOCALLY, LOOK GLOBALLY

  • If local detection ends without a usable answer, global localization ranks subtasks by suspicion and probes them, but each correction remains local.Suspicion favors unreliable subtasks whose outputs are consumed by many others, and the feature is disabled in the main experiments.Source: §D.5 · p. 31

E IMPLEMENTATION DETAILS

The implementation measures problem-solving cost in tokens and compares methods under matched compute budgets. It reports the matched-budget results and describes the benchmark statistics, experiment configurations, and optional global-localization procedure.

E IMPLEMENTATION DETAILS

  • Problem-solving cost counts input and output tokens across construction, execution, and in-flow optimization; single-agent compute is matched to the workflow task by task.This prevents any task from giving a workflow more compute than its baseline receives.Source: E IMPLEMENTATION DETAILS · p. 31
  • Global localization probes highest-suspicion untried subtasks after the local pass fails to produce a usable answer, then applies local tempering and resumes affected work.The procedure is bounded by the global-probe budget.Source: E IMPLEMENTATION DETAILS · p. 32
  • At matched turn budget, InFlowOp gains +7.15%, +7.75%, +10.68%, +11.97%, and +7.38% over single agents on five backbones.Best-of-N spends 4× the turns and reaches 15.83%, 20.63%, 21.72%, 21.25%, and 43.82%, remaining below InFlowOp on every backbone.Source: Figure 9 · p. 32, E IMPLEMENTATION DETAILS · p. 32
  • Table 3 summarizes BRAID’s 19 evaluation-only arms across eight domains, while Table 4 lists core experiment configurations and the settings used for ablations.BRAID includes questions at complexity levels k = 3 and k = 5 where datasets provide enough questions.Source: E IMPLEMENTATION DETAILS · p. 31, Table 3 · p. 33, Table 4 · p. 34

Related papers

Loading 2610.01017v1…