PaperlayerView exact revision on arXiv

Source-linked AI summary

Locating and Closing Information Gaps in Multi-Agent Systems

Yaxin Gong, Chongming Gao, Gangyi Zhang, Chenxiao Fan, Leyang Shen, Jiakai Wang, Dong Wang, Yang Liu, Haoyan Liu, Xinting Hu

arXiv:2610.40186v1cs.AI

TL;DR

Fixed multi-agent workflows can leave needed evidence out of a receiver’s context, while adding a link can also displace a correct answer. SPLICE trials a query-specific link and uses workflow changes to select between the candidate and original answers; it improves scores across five benchmarks relative to the static skeleton.

  • Problem

    Fixed workflows can prevent evidence gathered by one agent from reaching another that needs it, while a new link can also propagate an incorrect conclusion.

    Source: §1 · p. 1

  • Method

    SPLICE identifies a possible query-specific information gap, adds one directed edge, re-executes affected downstream nodes, and selects using final answers and workflow changes.

    Source: §4 · p. 3, §2.2 · p. 3

  • Results

    Across five benchmarks, SPLICE raises Web tool-use task score by 8.0% relative to the static skeleton.

    Source: §6 · p. 8

  • Takeaways & Limitations

    SPLICE composes with two published systems and a generic chain, and transfers zero-shot to four backbones.

    Source: §6 · p. 8

    The present work operates with a single gap and one real trial per query.

    Source: §6 · p. 8

Abstract

from arXiv · show

Multi-step Web tasks often require several LLM agents to gather and combine evidence. The communication graph of a fixed workflow restricts which agents can read each other's outputs: evidence retrieved by one agent may never reach the agent that needs it, leaving an information gap and an incomplete or incorrect answer. Adding a connection can deliver the missing evidence, but can also lead the receiver to adopt an incorrect upstream conclusion that displaces a correct answer; the final output alone does not separate the two. We propose Selective Probing and Link Insertion for Closing information gaps in multi-agent Execution (SPLICE), which locates information gaps per query and decides what to adopt after adding an edge. After the original workflow runs, SPLICE uses the query and node outputs to locate a possible gap, adds one directed edge from a source to a receiver to deliver the relevant evidence, and re-executes the receiver and affected downstream nodes, producing a candidate answer. The adoption decision reads both the final answers of the two runs and the process changes the edit caused inside the workflow: how far the receiver moved, where its new content came from, and whether the change reached the final answer. A selector trained on these process features chooses between adopting the candidate answer and retaining the original. On five benchmarks, SPLICE raises the task score on Web tool use by 8.0% and the mean score on four reasoning and planning benchmarks by 5.7% relative to the static skeleton, at 1.25 to 1.44$\times$ the skeleton's cost, and transfers zero-shot to four unseen backbones.

1 Introduction

Fixed workflows can leave evidence gathered by one agent out of the context of another agent that needs it. SPLICE locates query-specific gaps, tests a directed link, and uses changes inside the workflow to choose whether to keep the resulting answer.

1 Introduction

  • A fixed workflow can leave relevant evidence available elsewhere in the graph but absent from the context of the agent that needs it.The paper calls this an information gap; its cross-border example is a transit-visa detail unavailable to the visa agent.Source: §1 · p. 1
  • Adding a link can deliver missing evidence, but it can also make a receiver adopt an incorrect upstream conclusion and displace a correct answer.Final outputs alone do not distinguish these outcomes.Source: §1 · p. 1
  • After the original run, SPLICE uses the query and node outputs to locate a gap, adds one directed edge, and re-executes affected downstream nodes.The trial produces a candidate answer; SPLICE can retain the original instead, and the agents themselves remain unchanged.Source: §1 · p. 2
  • SPLICE diagnoses which questions have room for improvement and scores candidate edges by whether they would change the final output.It sends evidence held by the source but missing from the receiver’s context, rather than relaying the upstream answer.Source: §1 · p. 2
  • The selector uses receiver movement, the source of new content, and whether the change reaches the final answer, alongside both runs’ final answers.These process changes inform whether to adopt the candidate or retain the original.Source: §1 · p. 2
  • SPLICE improves Web tool use by 8.0% and the mean on four reasoning benchmarks by 5.7% relative to the static skeleton.It composes with two published systems and a generic chain, and transfers zero-shot to four backbones.Source: §1 · p. 2

2 Related Work

Prior work has explored both multi-agent communication topologies and ways to choose among candidate outputs. This paper combines per-query edge addition with a post-trial adoption decision informed by changes inside the workflow.

2 Related Work

  • The review organizes prior work around communication topology and candidate choice after modification.These are the two lines of work the paper identifies as directly relevant.Source: §2 · p. 2

2.1 Multi-Agent Communication Topology

  • Topology research spans fixed role-based workflows, graph optimization, and per-query graph edits, but the surveyed methods use the modified output without reconsidering it after the workflow runs.The paper contrasts that pattern with adding an edge per query and deciding after a trial whether to adopt its candidate answer.Source: §2.1 · p. 2

2.2 Selecting Among Candidate Outputs

  • Candidate-selection methods include voting, reflection, debate, verifiers, and evidence-action selection, while some compare candidates alongside the original answer.TrajSelector is excluded from the paper’s comparison because the API backbones do not expose its required latent representations.Source: §2.2 · p. 2
  • SPLICE’s candidate comes from a per-query edge addition and trial, and its selector uses workflow changes rather than only final answers.The original answer remains the default when the candidate is not adopted.Source: §2.2 · p. 3

3 Preliminaries

The section formalizes workflows as directed graphs, then defines which missing links may be tried and how a trial changes execution. It also distinguishes selective, retractable policies from commits that cannot reject a trial’s answer.

3.1 Multi-Agent Workflows as Graphs

  • A workflow is a directed acyclic graph of LLM agents and readable-output edges, executed in topological order to produce a final answer and trace.Each node conditions its answer on the query, its subtask, and outputs from its in-neighbours.Source: §3.1 · p. 3
  • Because intermediate products travel only along edges and agents forward their own answers, evidence may be rewritten at each hop or never reach a disconnected receiver.The query is visible to every node; the fixed design-time graph governs visibility of inter-node products.Source: §3.1 · p. 3

3.2 Communication Rewiring

  • Candidate edges are missing forward links from an upstream source to a receiver that do not create a cycle.Sources may reach the receiver indirectly or lie on a parallel branch; the candidate set is fixed by the skeleton, while useful edges vary by query.Source: §3.2 · p. 3
  • An information gap occurs when a source holds evidence the receiver needs but cannot read or recover from the query alone.The definition refers to the reference answer and is unobservable at deployment, where the method uses a ranking proxy.Source: §3.2 · p. 3
  • A trial adds a candidate edge and message, then re-executes the receiver and its descendants to produce a candidate output and score.The score change classifies a modification as beneficial, neutral, or harmful.Source: §3.2 · p. 3
  • A rewiring policy chooses whether to trial, which edges and messages to try, and whether to adopt a trial or retain the original answer.Selectivity means trial choices vary by query; retractability means the adoption decision can retain the original.Source: §3.2 · p. 3

4 Method

SPLICE intervenes after a workflow run, selecting a query and edge for a single trial and generating evidence for the receiver. It then uses changes inside the workflow to choose whether to adopt the candidate answer or retain the original.

4 Method

  • SPLICE runs after workflow execution, trialling one candidate edge for each selected query before choosing whether to adopt the candidate answer or retain the original.At most a budget fraction of queries are trialled; decisions before the trial rely only on the base trace.Source: §4 · p. 3

4.1 Diagnosis and Targeted Generation

  • Query-level diagnosis prioritizes questions with the largest predicted room for improvement, then edge-level diagnosis selects the edge most likely to change the final output.The edge estimate uses base-trace features, including receiver distance to the terminal node and agreement between the terminal answer and final output.Source: §4.1 · pp. 3–4
  • Targeted generation organizes the source’s content around entities and numbers in the receiver’s subtask that its parent outputs do not cover.The gap items tell the generator what to foreground instead of relaying the source answer verbatim.Source: §4.1 · p. 4

4.2 Process-Conditioned Selection

  • The selector uses paired base and trial traces to measure how far the receiver moved, where added content came from, and whether the change survived downstream.These process features distinguish unused messages, revisions not attributable to the message, and changes absorbed before reaching the final answer.Source: §4.2 · p. 4
  • A three-class language-model selector combines process evidence with the query, workflow, predicted base score, and both answers to choose whether to adopt or retain.Deployment weights potential gains by remaining headroom and potential losses by the score already in hand, so better original answers require stronger evidence for adoption.Source: §4.2 · p. 4

5 Experiments

SPLICE is evaluated as a query-specific repair to fixed multi-agent workflows, with experiments testing its overall performance, components, gap localisation, and transfer across settings. Across the reported workloads, targeted diagnosis and rollback improve scores while avoiding some harmful edits.

5.1 Experimental Setup

  • ToolBench replays archived API responses, while the four reasoning and planning benchmarks use reference-based or executable evaluation; question-level splits, listed baselines, and paired bootstrap intervals define comparisons.Workflow nodes use gpt-4o-mini, the DPO-trained selector uses Qwen2.5-7B-Instruct, and the study reports budget fractions 0.3, 0.5, and 1.0.Source: §5.1 · p. 5, Table 2 · p. 5

5.2 End-to-End Performance (RQ1)

  • At b=0.5, SPLICE gains +0.0509 over the static skeleton on ToolBench Micro and +0.0377 on the weighted mean of four benchmarks, at 1.44× and 1.25× the respective costs.It also exceeds Nexa + rollback by 0.0243 on ToolBench and 0.0213 on the four benchmarks.Source: §5.2 · pp. 5–6

5.3 Component Contribution (RQ2)

  • Predicted and actual base scores track closely, concentrating the b=0.5 trial budget on the two lowest quintiles; edge ranking yields a 0.61 fix ratio versus 0.46 for random edges.Removing query-level diagnosis raises cost without raising score, and the selected edge performs better because its value depends on the receiver’s post-trial response.Source: §5.3 · p. 6, Table 3 · p. 6
  • At matched 1.44× budget, the complete message scores 0.6856 versus 0.6665 with no edge or message; removing attribution features costs 0.0076 relative to full.The key factual statement carries roughly seven tenths of the 0.0159 gap between filler and the complete message; process evidence also helps beyond raw trace text and direct gain forecasting.Source: §5.3 · pp. 6–7

5.4 Gap Localisation (RQ3)

  • The edge selector’s gap-hit precision@1 is 0.67, above random edges (0.41), terminal distance (0.49), and content coverage (0.55); adopted repairs satisfy all four audit conditions at 0.74 and 0.71 for two judges.The audit reports substantial agreement (Cohen’s κ=0.77).Source: §5.4 · p. 7, Table 5 · p. 7, Table 6 · p. 7
  • On ToolBench, SPLICE raises fact support from 0.71 to 0.83 and task completion from 0.63 to 0.74, while lowering unsupported claims from 0.18 to 0.09.In the offline probe pool, 613 of 28,560 edges satisfy all four gap conditions, split between parallel-branch and distant-ancestor sources.Source: §5.4 · p. 7, Table 7 · p. 7

5.5 Generalisation and Boundaries (RQ4)

  • Across six host-by-benchmark cells, SPLICE improves on each host’s own output; on 2Wiki-Chain, it gains +0.0233 despite a net-negative candidate pool where Always loses 0.0067.This supports SPLICE as a layer on existing workflows rather than a replacement for their topology.Source: §5.5 · p. 8, Table 9 · p. 8
  • The selector transfers without retraining to four unseen backbones, with gains of +0.028 to +0.051; gains are positive across nine topologies, while b=0.5 is the knee for both workloads.Across the nine topology and scale settings, gains range from +0.0154 to +0.0383 and do not rise monotonically with candidate-edge count.Source: §5.5 · p. 8, Table 10 · p. 8, Figure 3 · p. 8

6 Conclusion and Future Work

SPLICE repairs query-specific information gaps and selects between the original and candidate answers using changes inside the workflow. Across five benchmarks, it improves scores over static workflows, while the current method is limited to one gap and one real trial per query.

6 Conclusion and Future Work

  • 8.0% on Web tool use and 5.7% mean score on four reasoning and planning benchmarks over the static skeleton, at 1.25 to 1.44× its cost.SPLICE also composes with two published systems and a generic chain, and transfers zero-shot to four unseen backbones.Source: §6 · p. 8
  • SPLICE locates a gap per query, adds one directed edge, re-executes affected downstream nodes, and chooses between answers using workflow changes.The selection uses both final answers and the process trace left by the edit.Source: §6 · p. 8
  • The present work operates with a single gap and one real trial per query, with multi-step repair and cheaper adoption criteria proposed as future directions.The paper also identifies extending beyond read-only Web to writing and transactional APIs as future work.Source: §6 · p. 8

A.1 Data and splits

The appendix specifies question-level dataset splits and evaluation procedures, along with the workflow structures and edge-selection setup used across the benchmarks.

A.1 Data and splits

  • ToolBench’s archive lacked a split field, so the authors added a fixed-seed 70/15/15 split and evaluated validation and test together as 230 held-out requests.Other benchmarks selected validation quantities within the ToolBench training split.Source: §A.1 · p. 10
  • PlanCraft scores a plan only when its steps execute against the benchmark recipe library and the target enters inventory; infeasible tasks require reporting impossibility.Unparsable crafting sequences score 0.Source: §A.1 · p. 10
  • ToolBench uses 765 solvable requests and four-node workflows; candidate edges are forward edges missing from each skeleton.Tool-enabled nodes call tools before generating outputs from returned API content; Table 11 lists skeletons as adjacency lists.Source: §A.1 · p. 10, Table 11 · p. 10
  • The four reasoning benchmarks allow three candidate edges per question, except PlanCraft with two; deployment selects one edge before executing a candidate.The offline probe pool executes two edges per question across 14,280 questions, totaling 28,560 executed edges.Source: §A.1 · p. 10

A.2 Decisive-edge filter

The decisive-edge filter restricts selector training and evaluation to edits that can change the final answer, while leaving end-to-end test scores unchanged.

A.2 Decisive-edge filter

  • 20,106 of 28,560 probed edges leave the final answer byte-identical to baseline, so only the remaining 8,454 decisive edges form the selector’s training and evaluation unit.The filter covers the offline probe pool across train, validation, and test; learned components are fit on training data alone.Source: §A.2 · p. 10
  • The filter does not affect end-to-end test scores; adopt, fix, and harm counts are restricted to output-changing questions in the probe pool.Those counts are absolute counts over the 5,143 questions whose output can change.Source: §A.2 · p. 10
  • Table 12 reports per-question token counts and cost multiples relative to the static skeleton; Table 13 presents the process-evidence block.Source: Table 12 · p. 10, Table 13 · p. 11

A.3 Selector implementation

The adoption selector is a trained 7B model that uses the original and candidate answers plus process evidence to choose whether to adopt the candidate.

A.3 Selector implementation

  • The 7B adoption selector is trained with DPO on preference pairs derived from reference answers, using question, original answer, candidate answer, and trial process evidence.It scores beneficial, neutral, and harmful outcomes, while deployment chooses between adopting the candidate and retaining the original.Source: §A.3 · p. 11
  • Training uses the training split, validation selects checkpoints, and inference uses predicted headroom rather than a reference answer or gold input.Both stages train a LoRA adapter with frozen backbone weights.Source: §A.3 · p. 11
  • Inference runs locally rather than through an external API and takes roughly 0.3 seconds per question on a single GPU.Its cost is included in the reported cost column.Source: §A.3 · p. 11

A.4 Process-evidence fields

Process evidence records how a receiver’s content changes after a trial, using message and context provenance. Ablations remove or shuffle this evidence block while preserving its format and length.

A.4 Process-evidence fields

  • The evidence block groups receiver additions A, deletions D, delivered-message words M, and pre-existing-context words P.Content words are normalized tokens longer than three characters; numbers are extracted separately.Source: §A.4 · p. 11
  • Every quantity is computed from archived traces without reference answers, and the block is rendered identically at training and inference.Source: §A.4 · p. 11
  • Ablations either remove the evidence block or shuffle query-to-block pairings while retaining the block’s format and length statistics.Source: §A.4 · p. 11

A.5 Baselines

The appendix specifies baseline comparisons and evaluation controls, then illustrates information-gap closure and a rejected trial through archived cases. It also describes paired competitor comparisons and ToolBench claim-support evaluation.

A.5 Baselines

  • Baselines share workflow skeletons, questions, and splits; comparison methods range from edge expansion and reruns to learned graph selection, rollback, verification, and voting.Source: §A.5 · p. 11

A.6 Evaluation protocol

  • Disjoint question-level splits separate training, validation, and one-time testing; ToolBench checkpoint selection occurs within training because it lacks a separate validation split.Source: §A.6 · p. 11

A.7 Paired differences against every competitor

  • Table 14 reports paired SPLICE–competitor differences on both workloads with 95% paired-bootstrap confidence intervals computed over questions.Source: §A.7 · p. 11, Table 14 · p. 12

A.8 Verifiable instances of a closed gap

  • A PlanCraft edge carries the recipe’s actual ingredients to inventory analysis, while a ToolBench edge delivers a service-generated API payload omitted from the synthesizer’s inputs.Source: §A.8 · p. 12

A.9 A trial the selector rejected

  • Despite a polished tutorial, SPLICE rejects a trial where failure details vanish, 76% of added content is untraceable, and Δ = −0.0342.The selector retains the original answer.Source: §A.9 · p. 12

A.10 Evidence-support protocol

  • Two blinded judges label claim support against replayed API bodies, while task completion is scored separately; human–judge agreement is 0.90 on a random subsample.Source: §A.10 · p. 12

Related papers

Loading 2610.40186v1…