PaperlayerView exact revision on arXiv

Source-linked AI summary

CoEvoke: Evoking Multi-Agent Collaboration for Trajectory Collection

Lin Wu, Shu Yao, Yuheng Wang, Jingru Fan, Huatao Li, Yihan Li, Liu Yixuan, Cheng Liu, Jian Chang, Zeyu Zhao, Aohua Zhang, Huiying Zhao, Shuyuan Zhu, Chen Qian

arXiv:2610.14021v1cs.LGcs.AIcs.MAcs.SE

TL;DR

Task-solving rollouts may not expose the diagnosis, feedback, and revision choices needed to learn collaboration. CoEvoke uses controlled perturbations and restricted guidance to collect reconstructable trajectories. Training improves control-role performance with frozen solvers under unperturbed evaluation.

  • Problem

    Task-solving rollouts may omit the diagnosis, targeted feedback, and revision choices that collaborative agents need to learn.

    Source: §1 · p. 1

  • Method

    CoEvoke perturbs candidates, observations, or model assignments and provides restricted reference-based guidance to collect reconstructable multi-agent trajectories.

    Source: §1 · p. 2

  • Results

    6.73 percentage points on ACPBench-Hard and 7.60–9.40 points on SWE-bench Verified: CoEvoke trajectory training improves control-role performance with frozen solvers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Gains with frozen solvers on unperturbed inputs without private-reference guidance support collection design as a means to improve learned checking, revision, and acceptance.

    Source: §7 · p. 10

    CoEvoke is validated on small teams, leaving its effectiveness in larger systems with more complex collaboration untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Training multi-agent collaboration requires trajectories of checking, feedback, and repair, yet task-solving rollouts may end in immediate approval or repeat unsuccessful attempts without useful correction. We introduce CoEvoke, a cross-domain framework that evokes these interactions during trajectory collection. Controlled perturbations create recoverable difficulties, while restricted reference-based guidance directs investigation. The resulting dataset contains 4,567 reconstructable trajectories over 4,130 software-engineering and reasoning tasks, linking public interactions to evidence, decisions, and task-native outcomes. Separate experiments over 2,287 same-task pairs show feedback increases of 47.72-100 percentage points across seven perturbation groups; guidance has smaller, model-dependent effects. With identical training tasks, decision-record counts, and updates, perturbation-enriched judge training improves ACPBench-Hard success from 35.38% to 42.12% (+6.73 points). SWE verifier training gains 7.60-9.40 points over interface-matched untrained references. Workflow and post-training evaluations preserve task-solving topology with unperturbed inputs and no private-reference guidance. These results support designing collection conditions to obtain corrective supervision that improves collaborative decisions with frozen solving agents.

1 INTRODUCTION

CoEvoke addresses the gap between task-solving rollouts and trajectories that expose checking, feedback, and repair. It uses controlled perturbations and restricted guidance to collect reconstructable collaboration records, with training gains under unperturbed evaluation.

1 INTRODUCTION

  • Task-solving rollouts may omit error diagnosis, targeted feedback, and revision choices, limiting supervision for learning useful collaboration.Immediately correct answers can bypass repair, while unsuccessful teams may repeat attempts without productive direction.Source: §1 · p. 1
  • CoEvoke perturbs candidates, observations, or model assignments and supplies restricted reference-based guidance to evoke checking and direct investigation.These controls retain task objectives and repair capacity.Source: §1 · pp. 1–2
  • Evaluation spans software engineering, mathematics, logical entailment, and constraint solving, with human-calibrated audits assessing trajectory quality.The framework also tests whether collection exposes useful correction and improves learned decisions in downstream workflows.Source: §1 · p. 2
  • The dataset contains 4,567 trajectories across 4,130 tasks, linking public role and tool events, feedback, revisions, and evaluator outcomes for decision-record extraction.A trajectory can yield several decision records, whose inputs exclude later repairs and evaluator outcomes.Source: §1 · p. 2
  • 6.73 percentage points on ACPBench-Hard and 7.60–9.40 points on SWE-bench Verified: training with CoEvoke trajectories improves control-role performance with frozen solvers.Evaluation uses unperturbed inputs without private-reference guidance.Source: §1 · p. 2

2 RELATED WORK

Prior work studies coordinated agents, synthetic trajectory generation, and feedback or verification. CoEvoke focuses on how collection conditions evoke checking and repair within these workflows.

2.1 FROM REASONING TO COORDINATED AGENTS

  • CoEvoke extends work on role-based cooperation, communication, debate, and orchestration by studying how collection conditions evoke checking and repair.Prior approaches address role-playing, task decomposition, candidate comparison, and agent sequencing.Source: §2.1 · p. 3

2.2 SYNTHETIC DATA AND AGENT TRAJECTORY GENERATION

  • Trajectory-generation work builds execution guidance, verifiable blueprints, repository tasks, or fine-tuning collections; CoEvoke intervenes on existing tasks to train control roles with frozen solvers.AgentRefine synthesizes environments for corrective supervision, while MALT trains role-specific models through generator–verifier–refiner trees and propagated rewards (Fu et al., 2025; Motwani et al., 2025).Source: §2.2 · p. 3

2.3 FEEDBACK, VERIFICATION, AND PROCESS EVALUATION

  • Feedback and verifier research motivates measuring multi-turn reliability and coordination; CoEvoke combines task-native outcome evaluation with human-calibrated process audits.Agent approval and task success remain distinct evidence.Source: §2.3 · p. 3

3 COEVOKE: EVOKING COLLABORATION DURING COLLECTION

CoEvoke preserves task-solving workflows while changing collection conditions to evoke checking, feedback, and repair. It turns retained public events into decision-specific supervision, excluding private references and later outcomes from controller inputs.

3.1 TASKS AND WORKFLOWS

  • Task workflows specify public context, optional private references, a task-native evaluator, and permitted roles and communication routes.Only a designated collection node may read private references; downstream workflows retain task-solving roles and routing without perturbations or private-reference guidance.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 PERTURB-AND-GUIDE

  • Perturb creates recoverable discrepancies, while Guide supplies restricted direction for investigation; the operations can be combined at designated roles.Runtime constraints restrict recipients and guidance structure, but do not guarantee semantic nondisclosure.Source: §3.2 · pp. 4–5
  • Perturbation changes a designated initial interaction while preserving evidence and repair capacity; examples include candidate corruption and partial observation.Candidate corruption is retained only when deterministic task-specific checks show the original candidate is correct and the perturbed candidate is incorrect.Source: §3.2 · p. 4

3.3 FROM RECONSTRUCTABLE TRAJECTORIES TO DECISION SUPERVISION

  • The framework preserves retained public role/tool events and their temporal links, including feedback, revisions, selections, and tool responses.Reconstruction covers exported interactions, not every teacher-visible input; immediate SWE test-pass summaries can inform collection-time review but are excluded from public trajectories and training contexts.Source: §3.3 · p. 5
  • Decision supervision pairs each retained public prefix with a target action, excluding private references, later revisions, and evaluator outcomes from the input.Targets cover judge selection or repair, critic feedback, and verifier acceptance or rejection.Source: §3.3 · p. 5

4 DATA GENERATED BY COEVOKE

The dataset links reconstructable trajectories to their source tasks, events, revisions, and evaluated artifacts. Audits report generally high process grades and no observed protected-content disclosure, with quality estimates limited to the probability-audit design.

4.1 SCALE AND COMPOSITION

  • 4,567 reconstructable trajectories cover 4,130 tasks, with source tasks, role events, revisions, and evaluated artifacts linked.Inclusion requires complete public role outputs and interaction links, regardless of task success; experimental runs are excluded from the dataset.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 HUMAN AND MODEL AUDITS

  • The audit assesses seven dimensions of process quality alongside task-native outcomes and separately grades disclosure from L0 to L2, with LX for insufficient context.Experts contributed 240 ratings over 160 trajectories to calibrate model assessment and evidence-based adjudication.Source: §4.2 · pp. 5–6
  • Grade A accounts for 93.1% of the weighted probability sample, 2.0% is labeled L1, and no L2 disclosure is detected.These weighted estimates describe the fixed probability-audit design; zero observed L2 cases does not guarantee zero population risk.Source: §4.2 · p. 6

5 EXPERIMENTS

The experiments test whether collection interventions evoke corrective interaction, whether workflows remain effective without those interventions, and whether the resulting trajectories improve control-role training. Perturbations substantially increase feedback, and matched judge and verifier training improves downstream task success.

5.1 EVALUATION DESIGN

  • Evaluation separates collection interventions from downstream tests, which retain task-solving topology and use unperturbed inputs without private-reference guidance.The perturbation study uses 2,287 same-task pairs; downstream training updates only control roles, and process measures distinguish feedback, revision, approved revision, and selection as the final source.Source: §5.1 · pp. 6–7

5.2 DO PERTURBATIONS EVOKE CORRECTIVE INTERACTION?

  • 99.42 points: input perturbation raises feedback from 0.33% to 99.75%; weak-to-strong generation raises it from 5.14% to 70.46%, a 65.32-point increase.Both conditions solve all 1,197 input-perturbation tasks, but 732 perturbed runs (61.15%) yield a selected correct revision, defined by a changed revised final answer and a PASS outcome.Source: Figure 3 · p. 7, §5.2 · p. 7

5.3 DOES RESTRICTED GUIDANCE IMPROVE TASK SOLVING?

  • 3.25 points: reference-based O2 raises success from 87.00% to 90.25% across 400 SWE runs; all five models use fewer net provider tokens, with an 8.55% aggregate reduction.Reasoning gains range from 0.85 to 3.75 points, while SWE token reductions range from 3.91% to 17.95%; provider-token totals do not represent a common tokenizer or monetary cost.Source: §5.1 · p. 7, Table 3 · p. 8, §5.3 · p. 8

5.5 DOES TRAINING ON COEVOKE TRAJECTORIES IMPROVE TASK SOLVING?

  • 6.73 points: Perturb-SFT reaches 42.12% versus Natural-SFT’s 35.38% on ACPBench-Hard; SWE verifier training gains 7.60–9.40 points over interface-matched untrained references.The ACPBench-Hard comparison uses 884 shared tasks, 891 decision records, and 379 updates; matching records and updates does not imply equal token exposure, and the result uses one training seed.Source: §5.5 · pp. 8–9, Figure 4 · p. 9

5.4 ARE THE WORKFLOWS EFFECTIVE WITHOUT PERTURB-AND-GUIDE?

  • On unperturbed inputs without private-reference guidance, CoEvoke beats budget-matched single-agent controls in eight of nine external model–benchmark combinations.All three models gain on SWE-bench Verified and ACPBench-Hard; on MathArena, Qwen and Gemma gain 6.25 and 3.47 points while Ministral loses 2.78 points. Internal Qwen results vary by task, with Zebra rising from 52% to 60% while DeepMath and ProofWriter each fall one point.Source: §5.4 · p. 8, Figure 4 · p. 9

6 DISCUSSION AND LIMITATIONS

The matched reasoning experiment supports joint corrective supervision and token savings, but not a superadditive interaction. The evaluation and training claims remain bounded by small-team validation and SFT, while verifier inputs vary dynamically across runs.

6 DISCUSSION AND LIMITATIONS

  • The matched 180-task 2 × 2 experiment supports joint corrective supervision and token savings, without establishing a superadditive interaction.It covers candidate corruption with one checker.Source: §6 · p. 9
  • CoEvoke is validated on small multi-agent teams; effectiveness in larger, more complex systems remains untested, and post-SFT RL and broader capabilities remain open questions.Training is restricted to SFT to isolate the dataset’s contribution.Source: §6 · p. 9
  • Fixed verifier interfaces ensure consistent evidence-selection rules, not identical dynamic inputs or classification of a fixed final-patch set.Accept/reject decisions affect subsequent execution, so candidates and evidence can differ across runs.Source: §6 · p. 9

7 CONCLUSION

CoEvoke makes corrective collaboration an objective of trajectory collection. Its reported gains with frozen solvers on unperturbed inputs and without private-reference guidance support collection design as a means to improve learned decisions.

7 CONCLUSION

  • CoEvoke makes corrective collaboration an objective of trajectory collection, using recoverable perturbations and restricted guidance to produce reconstructable trajectories with public evidence and outcome links.Full trajectories support decision-specific supervision without requiring every controller to receive the entire interaction.Source: §7 · p. 10
  • Gains with frozen solvers on unperturbed inputs and without private-reference guidance support collection design as a means to improve learned checking, revision, and acceptance.Source: §7 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement describes decision-supervision inputs and evaluation conditions, and identifies upstream material licensing and AI assistance disclosures.

REPRODUCIBILITY STATEMENT

  • Each retained public context is paired with its target action, excluding private references, subsequent revisions, and evaluator outcomes from the input.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Workflow and post-training evaluations retain task-solving topology while using unperturbed inputs and no private-reference guidance.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Upstream task text, code, and patches retain their respective licenses; generative AI assisted manuscript writing, translation, and polishing.Its use in trajectory generation and auditing is described in Sections 3 and 4.Source: REPRODUCIBILITY STATEMENT · p. 10

Related papers

Loading 2610.14021v1…