PaperlayerView exact revision on arXiv

Source-linked AI summary

CollabFlow: Recursive Self-Improvement of Agent Collaboration

Xiao Huang, Mingda Zhang, Junming Zhang, Qiang Huang, Hanwen Zhang, Yue Dai, Zijia Wang, Xiaoying Tang

arXiv:2609.38662v3cs.MAcs.AI

TL;DR

Existing multi-agent collaboration leaves its self-improvement loop open: teams and routes may be predefined, messages exchanged verbatim, and reward maximization may concentrate learning on a few teams. CollabFlow learns complete-Agent teams and evidence-gated communication, then uses CTB to retrain the director from outcomes while keeping positive-reward teams in play. Across twelve datasets, it outperforms baselines and maintains gains across rounds.

  • Problem

    Existing collaboration leaves the feedback loop open through pre-defined roles and routes, verbatim exchange, and reward-maximizing policies that concentrate on a few teams.

    Source: §1 · pp. 1–2

  • Method

    CollabFlow trains a director to construct complete-Agent teams and select evidence-conditioned communication protocols, then applies CTB to retrain it from outcomes while crediting each team once across construction orders.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    On twelve datasets, CollabFlow leads every baseline in and out of distribution, beats fixed paradigms and ablated variants, lifts six frozen executors, and uses the fewest tokens per item.

    Source: §6 · p. 9

  • Takeaways & Limitations

    CollabFlow keeps several good teams in play while its self-generated target moves less between rounds as records accumulate.

    Source: §1 · p. 2, §6 · p. 9

    Single-writer permissions and ordered commits prevent conflicts between internal writers, but do not guarantee correct plans, fresh observations, safe action semantics, or consistency with external participants.

    Source: I LIMITATIONS · p. 36

Abstract

from arXiv · show

Recursive self-improvement (RSI) lets a system improve from its own outcomes; in LLM-based multi-agent systems, Agents refine one another within a task, and outcomes improve how they collaborate across tasks. However, existing multi-agent collaboration leaves this loop open: collaboration is pre-defined at the operator level, topology-only learning keeps verbatim exchange that propagates errors, and reward maximization on a system's own outcomes concentrates on a few teams. To address these challenges, we propose CollabFlow, an RSI system of Learned Agent Collaboration: a trainable Collab-Director constructs teams of complete Agents, a frozen executor runs them, and each round's outcomes retrain the director. Within each round, the edges of a collaboration graph carry protocols of Evidence-Conditioned Communication: a receiver adopts a differing answer only when the sender's evidence is stronger by a margin, so the director learns who communicates and how. Across rounds, we further propose Collaborative Trajectory Balance (CTB), a flow-based objective that credits each team once across its construction orders and targets a reward-proportional distribution over teams, so several good teams stay in play. We also bound how far this self-generated target moves between rounds, which shrinks as records accumulate. On twelve datasets, CollabFlow outperforms all baselines and keeps improving across rounds. Code is available at https://anonymous.4open.science/r/CollabFlow-631E.

1 INTRODUCTION

LLM multi-agent systems can improve through within-task communication and across-task learning from outcomes, but existing collaboration leaves that feedback loop open. CollabFlow learns complete-Agent teams, evidence-gated communication, and cross-round updates, and outperforms baselines across twelve datasets while maintaining diverse teams.

1 INTRODUCTION

  • The paper frames RSI as learning from a system’s own outcomes, with multi-agent systems suited to the setting because Agents already refine one another’s answers.The outcomes of those exchanges can improve how teams collaborate across tasks.Source: §1 · p. 1
  • Existing systems leave the collaboration loop open through pre-defined operator-level roles and routes, verbatim message exchange, and reward-driven concentration on a few teams.Topology learning chooses who communicates but not how messages are used, while construction orders split credit for the same team.Source: §1 · pp. 1–2
  • CollabFlow trains a Collab-Director to build teams of complete Agents, runs them with a frozen executor, and retrains the director from each round’s outcomes.Within a round, the director also selects graph topology and Evidence-Conditioned Communication protocols; receivers adopt a differing answer only when sender evidence is stronger by a margin.Source: §1 · p. 2
  • CTB credits each team once across its construction orders and targets a reward-proportional team distribution, keeping every positive-reward team in play.The bound on movement of the self-generated target shrinks as sampled-team records accumulate.Source: §1 · p. 2
  • Across twelve datasets, CollabFlow outperforms direct-inference, fine-tuning, workflow-search, and agent-RL baselines in and out of distribution, while using the fewest tokens per item.Most gains persist across evolution steps, and the team distribution stays diverse and stable.Source: §1 · p. 2

2 RELATED WORK

Prior orchestration work adapts operator sequences, Agent organization, or communication patterns, while self-evolving systems refine topologies or latent messages. CollabFlow instead makes complete-Agent team communication the object of recursive improvement.

Multi-Agent Collaboration and Self-Evolution

  • Prior multi-agent orchestration fixes roles and routes, selects Agents sequentially, predicts sparse topologies, or designs exchanged information, but messages are still exchanged verbatim by convention.Self-evolving systems also adapt topologies through trace auditing or refine latent messages over recursion rounds.Source: §2 · p. 3

Recursive Self-Improvement and RL for Agents

  • CollabFlow jointly selects team topology and communication protocols, retrains these choices on its outcomes, and scores the canonical complete-Agent team rather than each construction order..Source: §2 · p. 3
  • RSI and Agent RL improve prompts, harnesses, skills, weights, or workflows using rewards, but reward maximization can collapse to one mode and narrow what later rounds learn from.These objectives use trajectory rewards to sequence Agents, edit workflow graphs, or evolve skills.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries formalize teams as protocol-labeled collaboration graphs and construction trajectories that map valid builds to canonical teams. They define round-level updates from outcomes and a team-distribution objective under fixed rewards.

3 PRELIMINARIES

  • A collaboration graph selects complete Agents, labels directed edges with communication protocols, and specifies an output rule.Agent profiles include roles, tools, and inherited permissions; stateful teams select exactly one authorized Executor.Source: §3 · p. 3
  • A collaboration trajectory builds a team through sufficient states and actions, ending in a canonical team terminal or an abort.Accepted stops for the same canonical team enter the same absorbing node, so valid trajectories decode to that team.Source: §3 · p. 3
  • Across rounds, the director, backward policy, normalizer, and reliability snapshot update from the batch of outcomes, with each unique outcome counted once.Teams built in one round retrain the director and define the next round’s reward through their success counts.Source: §3 · p. 3
  • For fixed positive team rewards, the objective seeks a team distribution close to a reward-weighted target, with divergence such as total variation.The context and reward are fixed per round.Source: §3 · p. 3

4 METHODOLOGY: COLLABFLOW

CollabFlow nests within-round evidence-conditioned Agent communication inside an across-round loop that retrains a team-building director from evaluated outcomes. Its method fixes the executor, validates team construction, and uses CTB to balance team rewards across canonical teams and construction histories.

4.1 LEARNED AGENT COLLABORATION: WHAT EACH ROUND CHANGES

  • The trainable Collab-Director builds teams through checked actions, while a frozen executor runs them; canonicalization merges construction histories that differ only in serialization order.The executor and Agents remain unchanged, so changes in output distribution come from the teams the director builds.Source: §4.1 · p. 4

4.2 MESSAGE-LEVEL IMPROVEMENT: EVIDENCE-CONDITIONED COMMUNICATION

  • On each directed edge, the director selects a protocol, and a receiver adopts a differing answer only when the sender’s evidence exceeds its own by a margin.The method fixes κ = 1: a two-level evidence gap adopts, while a one-level gap triggers revision; stated confidence does not enter the comparison.Source: §4.2 · p. 5

4.3 TEAM-LEVEL IMPROVEMENT: COLLABORATIVE TRAJECTORY BALANCE

  • CTB scores each complete canonical team once across its construction orders and uses the loop’s own records to set team rewards.The reward combines answer score with a Jeffreys-smoothed posterior success rate and a structural prior; each unique outcome is counted once after the batch.Source: §4.3 · pp. 5–6
  • CTB trains on a length-normalized trajectory-balance residual with a KL penalty that damps director updates while leaving balanced directors as fixed points.The self-generated targets of consecutive rounds have a total-variation bound that shrinks as sampled teams’ records accumulate.Source: §4.3 · p. 6
  • At exact valid-path balance, team probabilities are reward-proportional, so every positive-reward team retains probability instead of the distribution collapsing to one team.With β = 1 and rewards 0.6, 0.4, and 0.2, the probabilities are 1/2, 1/3, and 1/6; reward maximization places all mass on the first team.Source: §4.3 · p. 6

5 EXPERIMENTS

Across twelve IID and OOD benchmarks, CollabFlow leads the baselines, and its team-level objective and recursive training retain diverse, persistent gains. The experiments also compare paradigms, components, frozen executors, and training dynamics.

In distribution (RQ1)

  • 2.27 EM and 4.37 accuracy points above the strongest baseline on average, CollabFlow leads every IID dataset and metric, with average gains significant at p < 10−2.SFT and GRPO add at most 5.5 accuracy points to the backbone, whereas learned teams add 33.3.Source: Table 1 · p. 7, §5 · p. 7

Out of distribution (RQ2)

  • 3.51 EM and 4.22 F1 are CollabFlow’s OOD margins; it leads every OOD row but HumanEval, where it ties AFlow.Fine-tuning moves the backbone by under one accuracy point.Source: §5 · p. 7

Across executors (RQ3)

  • CollabFlow raises every IID and OOD average across six other frozen executors; weaker executors gain more, narrowing the IID spread from 23.3 to 9.3 points.The in-distribution association between executor weakness and gain is Pearson r = −0.96.Source: §5 · p. 7, Figure 4 · p. 8

Paradigms and components (RQ4)

  • Every paradigm and ablated variant falls below the full model on all twelve datasets, with larger losses out of distribution.Removing director training costs 13.7 IID and 19.5 OOD points, while removing communication costs most; verbatim exchange, forced revision, and all-ONEWAY edges cost 5.5–9.8 IID and 14.3–15.6 OOD points.Source: Table 2 · p. 8, §5 · p. 9

Team-level objective (RQ5)

  • CTB beats GRPO, PPO, and CollabFlow without CTB on every team-level measure, has the smallest team-distribution shift, and finds 26 distinct successful paths against at most 15.Its reward-proportional target keeps every positive-reward team in play; training accuracy passes 0.80 within 25 steps.Source: §5 · p. 9, Figure 5 · p. 9

Recursive self-improvement (RQ6)

  • Success rises from 34.4% to 75.0% over the twenty evolution steps with the largest one-step gain; 15 steps remain ahead five steps later and account for 83.6% of 55 edits.CollabFlow also uses the fewest tokens per item with the lowest spread.Source: §5 · p. 9, Figure 5 · p. 9

6 CONCLUSION

CollabFlow combines evidence-conditioned communication within rounds with CTB-based director retraining across rounds. Across twelve datasets, it leads the baselines, lifts six other frozen executors, retains most gains, and uses the fewest tokens per item.

6 CONCLUSION

  • CollabFlow makes agent collaboration recursively self-improving through evidence-conditioned communication and CTB retraining on its own records.Its target distribution moves between rounds by a bound that shrinks as records accumulate, keeping several good teams in play.Source: §6 · p. 9
  • Across twelve datasets, CollabFlow leads every baseline in and out of distribution, beats fixed paradigms and ablated variants, lifts six other frozen executors, retains most step gains, and uses the fewest tokens per item.The conclusion points to Appendices I and J for limitations and future work.Source: §6 · p. 9

AI USE STATEMENT

The authors used generative AI tools for editing, typesetting, layout, and figure/table placement, and manually reviewed the assisted materials.

AI USE STATEMENT

  • Generative AI tools assisted with clarity edits, LATEX typesetting, page layout, and placement and sizing of existing figures and tables.The authors state the tools were not used to generate or modify experimental data or scientific-figure content, formulate hypotheses or mathematical claims, or design or implement the methodology.Source: AI USE STATEMENT · p. 10
  • The authors manually reviewed and verified all AI-assisted text, figures, numbers, citations, and formatting against implementation and experimental records.They take responsibility for the final content and AI-assisted artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The authors state that the work follows ICLR ethics principles and reports results without fabrication, falsification, or intentional misrepresentation.

ETHICS STATEMENT

  • The authors report that every reported number is produced by released code and that they use public benchmarks and models under their intended purposes and licenses.CollabFlow Agents use permitted tools; a single Executor holds write permission on stateful tasks, and medical-question results do not authorize clinical use.Source: ETHICS STATEMENT · p. 10
  • The study involved no human participants or collection of personal information.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

This section documents CollabFlow’s reproducibility materials and formalizes its contracts for team construction, evidence-conditioned communication, execution, and evidence handling. It specifies conditions for construction validity and graph-level CTB reward-proportional sampling, and describes reward snapshots and bounds on how the self-generated target moves as records accumulate.

REPRODUCIBILITY STATEMENT

  • The code release includes training and evaluation code, trained Collab-Director weights, and train/test splits for all twelve datasets.The appendix also identifies where to find the training procedure, evaluation protocols, metrics, proofs, and per-benchmark results.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 COLLAB-DIRECTOR INPUT AND OUTPUT CONTRACT

  • The director receives task, compatible profiles, partial graph, permitted protocols, and remaining budget, but not reference answers or hidden evaluator labels.Execution records distinguish director construction decisions from child-agent messages and tool or environment evidence; reproducibility logs include task, resource, communication, and cost details.Source: §A.1 · p. 13, §A.5 · p. 15

A.2 AGENT PROFILES

  • Agent profiles and executor parameters stay fixed during training, while the director selects profiles, skills, and communication; outcome-driven evolution can add, modify, or delete library skills.Binding a retrieved strategy to an Agent grants no tool permissions and does not override write permission.Source: §A.2 · p. 14

A.4 COMMUNICATION AND OUTPUT SEMANTICS

  • Execution records actual messages and responses, while revisions use tool-free calls and retain only evidence valid for the returned candidate.Stateful tasks restrict communication to structured Advisor-to-Executor handoffs; output modes include selecting one result, integrating records, verifying candidates, or formatting submissions.Source: §A.4 · pp. 14–15

B.1 OBJECTS, CONDITIONING, AND THE TWO SOURCES OF RANDOMNESS

  • Conditional on valid completion, answers follow the graph distribution and frozen execution kernel; unconditionally, the answer law also includes construction-abort probability.Construction and execution use separate budget reservations, so identical terminal graphs receive the same execution budget regardless of construction order.Source: §B.1 · p. 16

B.2 CHECKED CONSTRUCTION: SOUNDNESS AND COVERAGE

  • Every accepted completed trajectory yields a valid canonical team, and each feasible target team has a legal construction trajectory under the stated conditions.The construction graph is finite and acyclic with bounded records and a finite decision cap.Source: §B.2 · p. 17

B.3 ACTION LIKELIHOODS AND LEGAL-SUPPORT NORMALIZATION

  • The director scores legal action strings and samples from their normalized action kernel; grammar masking defines a potentially different sampler whose own likelihood must be used.Action-string log probability is the sum, not the mean, of token log probabilities.Source: §B.3 · p. 18

B.4 CANDIDATE VERSIONS, EVIDENCE APPLICABILITY, AND DUPLICATION

  • Evidence records remain applicable and unique after copying, keeping, or revising candidates, provided versions are filtered and immutable records are deduplicated.A new candidate without applicable incoming evidence inherits an empty evidence set and score zero.Source: §B.4 · p. 19
  • The gate scores candidates by their maximum applicable evidence category: 2 for a successful executable check, 1 for retrieval support absent category-2 evidence, and 0 otherwise.Repeated copies do not increase the maximum; a changed artifact is a new version subject to the applicability contract.Source: §B.4 · p. 19

B.5 ONE-EDGE REVISION IDENTITY

  • On incorrect-receiver cases, the flip rate is δC + P(J | C, c)cJ; increasing the margin reduces adoption and increases revision, trading false adoption against true adoption.The identity requires no independence between candidates and is evaluated under the actual pair-selection and evidence-generation law.Source: §B.5 · p. 20

B.6 RUNTIME SCHEDULING, FINITE EXECUTION, AND A LINEAR BASELINE

  • With bounded calls, quotas, and scheduling, execution terminates in finite time and uses at most |E|rmax gate evaluations per one-pass schedule or Rcomm|E|rmax per cyclic schedule.A fixed asynchronous serial order makes runs reproducible because updates from incoming peers can depend on their order.Source: §B.6 · pp. 20–21
  • The DeGroot baseline with primitive stochastic W converges to consensus π⊤h(0), fixed by initial opinions and W rather than evidence about correctness.The gate replaces this fixed weighting with an evidence comparison.Source: §B.6 · p. 21

B.7 SINGLE-WRITER SERIALIZATION OF INTERNAL MUTATIONS

  • Under enforced exclusive Executor permission and serial completion, internal writes are totally ordered and serializable in the sense of Papadimitriou (1979).The claim is restricted to team-controlled components; state composition additionally excludes external interleaving mutations or conditions on them.Source: §B.7 · pp. 21–22

B.8 NORMALIZED REVERSE PATHS AND GRAPH-LEVEL CTB

  • Under T1–T4, graph-level CTB gives outcome probability PF(z) = wz/W, with Z = W; at zero residual, the reverse kernel governs probability across a team’s construction orders.For explicit abort, the valid-graph expression holds after conditioning on valid completion, while the unconditional normalizer also includes abort weight.Source: §B.8 · p. 23

B.9 TERMINAL ALIASES, ABORT, AND THE MEANING OF VALID-ONLY REPLAY

  • With duplicated terminal aliases, graph probability is proportional to the sum of their weights; valid-only replay yields the conditional reward-proportional graph law when valid-path balance holds.Graph-level temperature β controls preference for high-reward teams while retaining alternatives; duplicated construction paths earn no extra weight.Source: §B.9 · pp. 23–24

B.10 APPROXIMATE BALANCE AND PERTURBATION BOUNDS

  • For a uniform residual bound |ρ| ≤ η, total variation from the target is at most tanh(η/2), and each KL divergence is at most 2η.The bounds also apply to valid-conditional graph laws when the residual bound covers valid paths and raw valid-only likelihoods are used.Source: §B.10 · p. 24

B.11 REWARD SNAPSHOTS, POSITIVE SUPPORT, AND MOVING COUNTERS

  • Rewards combine a positive floor, a Jeffreys-posterior success-rate factor, and a relative-description-length prior; records count once and each batch uses pre-batch counts.The method uses λ = 1 with code lengths in nats; the structural-prior ablation sets λ = 0.Source: §B.11 · p. 25
  • The positive reward floor keeps log rewards finite and gives failed outcomes a small positive target mass.Source: §B.11 · p. 25
  • With fixed score table, structural code, graph support, and β, the target changes by a tilt τG∈[e^-βd,e^βd], where d shrinks as sampled teams accumulate records.Teams with no new unique outcomes or zero answer scores contribute nothing to d; the bound holds for any realized counts, regardless of team-selection policy.Source: §B.11 · p. 26

B.12 THE PRACTICAL TRAINING LOSS

This appendix specifies how CTB is trained and interpreted, how reward-weighted occupancy tracks teams and transitions, and what assumptions and accounting bound the procedures. It also details the training records, action probabilities, diagnostics, and structural construction costs.

B.12 THE PRACTICAL TRAINING LOSS

  • With a moving reference policy, every balanced director remains a fixed point of the regularized loss; the update’s KL term damps movement away from that point.For fresh execution rewards, the ideal graph weights are exp(βmG), where mG is the conditional mean log reward; the residual-gradient variance shrinks as the mean-square residual shrinks.Source: §B.12 · pp. 26–27

B.13 OCCUPANCY FLOWS, CONDITIONAL BRIDGES, AND PATH DIAGNOSTICS

  • Reward-weighted occupancy Hc(a) sums the weights of valid teams containing communication decision a, independent of how each team’s construction orders are allocated.Hc(a)/W is the target probability of returning a valid team containing the edge; Hc(a)/WV conditions on valid completion.Source: §B.13 · p. 28

B.14 PROOF DEPENDENCIES

  • The proof dependency chain runs from checked construction and normalized path laws to fixed terminal identities and reward snapshots, which link balance or residual bounds to the target distribution.The reward-snapshot corollary bounds target movement between rounds; communication and single-writer results are runtime contracts alongside this chain.Source: §B.14 · p. 29

C.1 TRAINING PROCEDURE

  • Each round freezes the resource context, pre-batch counters, and reward-code version, collects valid construction paths and observed rewards, updates the director, and then updates counters once per unique outcome.Inference freezes the director and executor; execution failures of valid graphs remain valid graph samples with observed rewards.Source: §C.1 · p. 29, §C.2 · p. 30

C.2 RELIABILITY STATISTICS AND REWARD BOOKKEEPING

  • Rewards combine answer score with a Jeffreys-smoothed success rate and a structural prior; records are keyed by graph and task cluster, and unique outcomes update counters after the batch.All rollouts in an update use pre-batch counts, and records bind task, graph, resource and runtime versions, execution reservation, factual score, and snapshot identifier.Source: §C.2 · p. 30

C.3 ACTION LIKELIHOODS, MASKS, AND BACKWARD TERMS

  • Action likelihoods use normalized probabilities over legal actions, while the learned backward kernel assigns normalized probabilities to each team’s legal construction orders.The backward kernel unbuilds a team by removing a component that leaves a legal prefix; the uniform ablation gives each order equal probability.Source: §C.3 · pp. 30–31

C.4 TRAJECTORY CONTRASTS AND DIAGNOSTIC EVIDENCE

  • Trajectory diagnostics store forward-minus-backward log probabilities, while optional same-task contrasts compare runs by factual score and report their shared actions, divergence, evidence, costs, and outcomes.Cached probabilities can be reused, but new backward evaluations and other computation remain chargeable.Source: §C.4 · p. 31

C.5 STRUCTURAL COMPLEXITY AND COST ACCOUNTING

  • The conservative construction bound is T(A + Pn^2 + nK + Co + 1) candidate inspections, but it covers construction only, not end-to-end execution and evaluation costs.Total cost must additionally account for Agent execution, revisions, tool calls, and evaluator work.Source: §C.5 · p. 31

D.1 BENCHMARK INVENTORY AND SPLITS

This material documents the benchmark and evaluation setup, including task adapters, held-out evaluation, baselines, and structural controls. It specifies what is compared and how evaluation examples and configurations are handled.

D.1 BENCHMARK INVENTORY AND SPLITS

  • Table D.1 inventories twelve benchmarks with their metrics and evaluation-item counts; IID evaluation examples are kept out of training.Train and test identifiers are disjoint, including derived prompts and duplicated source instances.Source: §D.1 · p. 32, Table D.1 · p. 32

D.2 TASK ADAPTERS AND FACTUAL SCORING

  • Task adapters define evidence access, answer extraction, and correctness matching for QA, reasoning, code, embodied, and conversational tasks.Code adapters run submissions against permitted test harnesses; evaluator-only labels and hidden tests must not enter Agent prompts.Source: §D.2 · p. 32

D.3 FROZEN HELD-OUT EVALUATION

  • OOD evaluation freezes the director, Agent population, and training-derived configuration, and uses no OOD instance for training or tuning.The six OOD comparisons include within-family changes and stronger domain or interface shifts.Source: §D.3 · p. 32

E.1 BASELINE INVENTORY

  • The main comparison includes direct inference, SFT, GRPO, AFlow, FlowSteer, an evolving orchestrator, SkillFlow, and SkillRL.Where interfaces permit, comparisons match backbone, access, call limits, and interaction horizon; unmatched settings are reported explicitly.Source: §E.1 · pp. 32–33, Table E.1 · p. 33

E.2 FIXED TEAMS AND RANDOM LEGAL GRAPHS

  • Structural controls include single-Agent, parallel-Agent, fixed-chain, stateful Executor–Advisor, and randomly sampled legal-graph teams.Fixed topologies are selected using training or validation data, not an oracle choosing per-test-instance topologies; independent outputs require a specified aggregation mechanism.Source: §E.2 · p. 33

E.3 COMPONENT ABLATIONS

This appendix specifies the component ablation rows, evaluation metrics, resource reporting, implementation configuration, and checks. It describes reporting conventions rather than supplying ablation outcomes in these passages.

E.3 COMPONENT ABLATIONS

F EVALUATION METRICS

  • Evaluation metrics follow each benchmark’s declared evaluator, with deviations such as answer normalization, partial credit, and simulator-specific success rules recorded.The reporting conventions distinguish binary task-success indicators from continuous QA F1.Source: F EVALUATION METRICS · p. 34, §F.1 · p. 34

F.1 TASK PERFORMANCE AND MAIN-TABLE AVERAGES

  • Suite scores average EM and F1 separately across two QA benchmarks, and average primary scores across four other benchmarks; reported deltas are percentage-point differences from direct Qwen3.5-9B.QA EM and F1 remain separate from other score columns.Source: §F.1 · p. 34

F.2 RESOURCES, FAILURES, AND VARIABILITY

  • Resource reporting counts model tokens, calls, tool calls, latency, failures, and caching, with evaluator and simulator computation separately recorded.Run summaries use suite scores computed within each independent run and report arithmetic means and sample standard deviations across runs.Source: §F.2 · p. 34

G.1 CONFIGURATION

  • Qwen3.5-9B serves as both Collab-Director and frozen executor, and methods share one Agent population, tool set, and per-task budget.The released code contains the reported runs’ configurations, prompts, seeds, and checkpoints.Source: §G.1 · p. 34

G.2 IMPLEMENTATION CHECKS

  • Implementation checks enable conditional communication updates and enforce revision constraints and single-writer checks; terminal-distribution implementation is specified to satisfy T1–T3.Stateful execution checks test both Advisor attempts at mutation and overlapping internal state-changing tool effects.Source: §G.2 · pp. 34–35

H ADDITIONAL ANALYSES AND COMPLETE REPORTING TABLES

  • The additional-analyses appendix identifies per-benchmark score and token-cost tables as supporting material for Figures 4 and 5f.The supplied passages do not give the underlying table values.Source: H ADDITIONAL ANALYSES AND COMPLETE… · p. 35

H.1 PER-BENCHMARK SCORES WITH OTHER EXECUTORS

The appendix reports per-benchmark scores with other frozen executors and token costs per item. The supplied passages identify the tables’ scope but do not provide their numerical values.

H.1 PER-BENCHMARK SCORES WITH OTHER EXECUTORS

  • Table H.1 lists per-benchmark CollabFlow scores and gains over direct inference with the same executor; DeepSeek V4 appears only for the six IID benchmarks.Averages cover each part’s six benchmarks.Source: §H.1 · p. 35, Table H.1 · p. 35

H.2 TOKEN COST PER ITEM

  • Table H.2 compares tokens used per item by CollabFlow and four comparison systems, counting input and output tokens from every model call in an episode.Source: §H.2 · p. 35

I LIMITATIONS

CollabFlow is bounded by a fixed set of Agents, protocols, output modes, and resource caps; its conclusions apply within that admissible space. The paper also identifies limits on evidence-based updates, evaluation claims, and real-world use.

I LIMITATIONS

  • The representation fixes the Agent population, protocols, output modes, and resource caps, so construction coverage does not imply a capable Agent exists for every task.Additional organization choices can increase sampling cost without improving task performance.Source: I LIMITATIONS · p. 36
  • Single-writer controls prevent conflicts between internal writers under enforced permissions and ordered commits, but do not guarantee correct plans, fresh observations, safe actions, or consistency with external participants.Diagnostic traces may associate improvement with a message even when extra computation or other changes also contribute.Source: I LIMITATIONS · p. 36
  • Evidence-conditioned updates can favor an incorrect candidate when authentic evidence is incomplete or misleading.Reliability estimates from sparse or nonstationary executions need not be calibrated, and finite training does not ensure the exact terminal distribution proved under fixed conditions.Source: I LIMITATIONS · p. 36
  • Runtime likelihood surrogates, incomplete budget accounting, and unimplemented adapters or extensions limit which claims a particular evaluated system can support.Source: I LIMITATIONS · p. 36

J FUTURE WORK

  • Future work could test controlled message interventions that separate peer information from additional reasoning while retaining resource and permission accounting.Source: J FUTURE WORK · p. 36
  • The authors identify open-ended recursion—learning the update rule or parts of the evaluator jointly with the team—as a direction beyond the fixed setup.Source: I LIMITATIONS · p. 35, J FUTURE WORK · p. 36

K APPLICABILITY AND BROADER IMPACT

  • The framework targets tasks with specifiable candidate Agents, permissible communication, execution interfaces, and factual evaluators; benchmark success, including on medical reasoning, does not authorize unsupervised clinical decisions or unsafe real-world execution.Consequential applications require domain-specific validation, limited permissions, isolated execution, and appropriate human approval.Source: K APPLICABILITY AND BROADER IMPACT · p. 36

Related papers

Loading 2609.38662v3…