PaperlayerView exact revision on arXiv

Source-linked AI summary

CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems

Yifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, Bryan Wang

arXiv:2509.24088v2cs.MA

TL;DR

Multi-agent errors can cascade, while their earliest decisive steps are difficult to localize across long, complex trajectories. CORRECT reuses compact schemata distilled from prior failures; it improves step-level localization accuracy by up to 20 points over prior work while remaining training-free and lightweight.

  • Problem

    Errors can cascade across agents, making it challenging to pinpoint the precise agent and step that first triggered failure.

    Source: §1 · p. 1

  • Method

    CORRECT distills recurrent failures into compact schemata and retrieves and instantiates relevant schemata to diagnose new trajectories at runtime.

    Source: §1 · p. 2

  • Results

    Up to 20 points higher step-level localization accuracy than prior work, with CORRECT remaining training-free and lightweight.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CORRECT-Error provides a reusable, extensible benchmark of over 2,000 step-annotated trajectories validated for alignment with expert-labeled errors.

    Source: §1 · p. 2

    Decisive error labels can be subjective, and this ambiguity may destabilize future training, evaluation, or online adaptation that relies on them.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multi-agent systems (MAS) are increasingly capable of tackling complex real-world tasks, yet their reliance on inter-agent coordination, tool use, and long-horizon reasoning makes error recognition particularly challenging. Minor errors can propagate across agents, escalating into task failures while producing long, intertwined execution trajectories that impose significant costs for both human developers and automated systems to debug and analyze. Our key insight is that, despite surface differences in failure trajectories (e.g., logs), MAS errors often recur with similar structural patterns. This paper presents CORRECT, the first lightweight, training-free framework that leverages an online cache of distilled error schemata to recognize and transfer knowledge of failure structures across new requests. This cache-based reuse allows LLMs to perform targeted error localization at inference time, avoiding the need for expensive retraining while adapting to dynamic MAS deployments in subseconds. To support rigorous study in this domain, we also introduce CORRECT-Error, a large-scale dataset of over 2,000 annotated trajectories collected through a novel error-injection pipeline guided by real-world distributions, and further validated through human evaluation to ensure alignment with natural failure patterns. Experiments across seven diverse MAS applications show that CORRECT improves step-level error localization up to 19.8% over existing advances while at near-zero overhead, substantially narrowing the gap between automated and human-level error recognition.

1. Introduction

Multi-agent failures can cascade across agents, making it difficult to identify the first step that caused failure. CORRECT reuses compact error schemata to localize errors, while CORRECT-Error provides a large-scale annotated benchmark.

1. Introduction

  • Errors can propagate across agents and cause task failure, so identifying the decisive agent and step matters for reliable deployment and operational management.The introduction identifies safe upgrades, targeted restarts, and service monitoring as examples of operational uses.Source: §1 · p. 1
  • Failure structures recur across requests, but inserting prior trajectories as in-context examples can falter because logs are long, noisy, and may underperform zero-shot baselines.On WHO&WHEN, over 80% of failure trajectories had a counterpart with at least 0.8 semantic similarity in error logs.Source: §1 · p. 2
  • CORRECT distills prior failures into reusable schemata and retrieves relevant ones to diagnose new trajectories in dynamic online environments.The schemata encode failure signatures, triggering contexts, and propagation patterns.Source: §1 · p. 2
  • Up to 20 points higher step-level localization accuracy than prior work, with training-free and lightweight inference-time knowledge transfer.CORRECT distills recurrent failures into compact schemata and retrieves them for new requests.Source: §1 · p. 2
  • CORRECT-Error contains over 2,000 step-annotated MAS trajectories generated through real-world-guided error injection and validated against expert-labeled errors.The authors describe it as a reusable, extensible benchmark for the community.Source: §1 · p. 2

2. Background and Motivation

Decisive error recognition seeks the earliest agent step whose correction turns a failed MAS trajectory into a success. Existing approaches face accuracy, annotation-cost, and reuse challenges, even though failure traces often share semantic similarity.

2.1. Decisive Error in Multi-Agent Systems

  • A decisive error is a step in a failed trajectory whose correction would change the outcome to success; the earliest such step is the target for localization.Fine-grained attribution can guide targeted changes to roles, orchestration, or individual agents without overhauling the whole system.Source: §2.1 · p. 2

2.2. Motivations of CORRECT

  • On WHO&WHEN, Qwen-2.5-7B achieved only 3.5% step-level accuracy with LLM-as-a-judge methods as trajectories became more complex.The compared variants are all-at-once, step-by-step, and binary search.Source: §2.2 · p. 2
  • More than 80% of WHO&WHEN failed requests had cosine similarity above 0.8, yet in-context retrieval degraded accuracy amid long trajectories and low signal-to-noise ratio.The passage reports that 17% of trajectories exceed 32K tokens; Figure 1 analyzes failure traces on WHO&WHEN.Source: Figure 1 · p. 3, §2.2 · p. 3
  • Annotating fewer than 200 MAS trajectories took over 30 expert hours, while variation across applications and requests makes generalization difficult.The section also notes that fine-tuning approaches depend on large, high-quality labeled datasets and face efficiency and expense challenges.Source: §2.2 · pp. 2–3

3. CORRECT: Condensed Error Recognition via Knowledge Transfer

CORRECT distills recurring multi-agent failures into reusable error schemata and uses them to guide training-free diagnosis. Its workflow extracts compact schemas offline, retrieves and instantiates relevant schemas online, and adapts the cache as new errors appear.

3. CORRECT: Condensed Error Recognition via Knowledge Transfer

  • CORRECT represents failures with schemas that capture error signatures, triggering context, and detection heuristics.The framework’s overview describes offline extraction, online schema-guided recognition, and dynamic schema management.Source: §3.1 · p. 3, §3 · p. 3

3.1. Error Schema Extraction

  • Clustered extraction groups semantically similar failure trajectories and generates one representative schema per cluster.The cluster count increases until marginal cohesion gains saturate; a relatively small cache can reach an accuracy plateau.Source: §3.1 · pp. 3–4

3.2. Schema-Guided Error Recognition Online

  • At diagnosis time, CORRECT retrieves top-k relevant schemas and prompts an LLM to instantiate them against the target trajectory.The condensed schemas direct attention to salient failure patterns without processing full historical trajectories.Source: §3.2 · p. 4
  • CORRECT expands the cache for confirmed novel errors and distills frequently accessed schemas by replaying candidates on prior trajectories.Expansion favors trajectories dissimilar to cached schemas, while distillation retains the candidate with the highest replay accuracy.Source: §3.2 · p. 4

4. CORRECT-Error: A Large-Scale Error Detection Benchmark

CORRECT-Error is a benchmark of synthetic multi-agent failure trajectories generated from a small set of human-annotated seeds. Human labelers often mistook synthetic errors for genuine ones, supporting the pipeline’s alignment with natural failure patterns.

4.1. Bootstrap Error Synthesis Pipeline

  • The bootstrap pipeline uses human-annotated errors as seeds to generate scalable synthetic data while preserving structural and semantic integrity.It collects successful trajectories and annotated failures, matches them by semantic similarity, then injects an adapted error into the target trajectory.Source: §4.1 · p. 5

4. CORRECT-Error: A Large-Scale Error Detection Benchmark

  • CORRECT-Error contains over 2,000 trajectories across seven datasets and provides 12.3× more data than WHO&WHEN.The benchmark spans tasks including multi-hop QA, planning, mathematical reasoning, and scientific problem-solving.Source: §4 · p. 4, §4.2 · p. 5

4.2. Human-Alignment Analysis

  • Four independent expert labelers completed over 120 hours of evaluation, assessing mixed synthetic and human-annotated trajectories without knowing their origin.The near-50% source-identification results indicate that labelers struggled to distinguish the two sources.Source: §4.2 · p. 5
  • 47.1% of synthetic trajectories were misclassified as human-labeled, while 42.3% of genuine trajectories were correctly identified.In addition, 94.4% of synthetic trajectories were judged genuine by at least two labelers, and 52.9% received unanimous consensus.Source: Figure 3 · p. 5, §4.2 · p. 5
  • The authors conclude that the injection pipeline captures nuanced real-world failure patterns and supports automated, scalable benchmark generation.They characterize the data as having precise ground-truth labels and realistic error distributions.Source: §4.2 · p. 5

5. Experiments

Experiments compare CORRECT with existing error-recognition approaches on WHO&WHEN and CORRECT-ERROR, then test performance across tasks and under transfer, online adaptation, and ablation settings. CORRECT improves error localization across benchmarks while retaining low overhead.

5.1. End-to-End Performance

  • On WHO&WHEN human-crafted data, CORRECT raises Qwen-2.5-7B Acc@0 from 3.5% to 12.1% and GPT-5 from 8.6% to 17.2%.On algorithm-generated data, Gemini-2.5-Flash improves from 31.8% to 38.9% (+7.1 points), while GPT-5 has the largest gain (+19.8 points).Source: Table 1 · p. 6, §5.1 · pp. 6–7
  • Across seven CORRECT-ERROR datasets, Qwen-2.5-7b gains average accuracy of 20.1%, 27.6%, and 28.7% at Acc@1, Acc@3, and Acc@5, respectively.The comparison spans multiple datasets and reports error-recognition accuracy.Source: Table 2 · p. 7, §5.1 · p. 7
  • Transferred schemas outperform baselines across model families, and stronger schema generators improve detection; online accuracy remains stable when only 10–20% of new schemata enter the cache.Using GPT-5 rather than Qwen-72B as schema generator raises Gemini-2.5-Flash accuracy from 8.6% to 10.3% and Qwen-2.5-7B from 10.3% to 12.2%.Source: Figure 4 · p. 7, §5.1 · p. 7, Table 3 · p. 7

5.2. Ablation Studies

  • With 10% of the schema library, CORRECT reaches 69.1% Acc@1 on MMLU pro and 35.4% on HotpotQA; gains plateau beyond 50%, while one retrieved schema already raises Acc@1 to 12.1%.Acc@1 rises to 13.8% with five schemas and 15.5% with ten, with diminishing gains.Source: Figure 5 · p. 8, §5.2 · p. 8
  • With five schemas, CORRECT reaches over 71.5% of oracle performance, while LLMs detect injected exact-step errors at only 21% on flipped and 17–18% on non-flipped trajectories.The oracle uses each trajectory’s own ground-truth schema.Source: Figure 6 · p. 8, §5.2 · p. 8
  • Under 3.5% latency overhead accompanies API usage rising from $0.86 to $0.89, and tested similarity and hotness settings leave performance stable.The reported strong balance uses δ=0.7 and refines the top 20% most-accessed schemata.Source: §5.2 · p. 8
  • Targeted restarts from CORRECT’s localized steps raise end-to-end success from 10% to 20%; with all five retrieved schemata randomized, accuracy remains above baseline for Qwen-7B and GPT-5.Under fully random retrieval, Qwen-7B accuracy is 6.9% versus a 3.5% baseline, and GPT-5 is 12.1% versus 8.6%.Source: §5.2 · pp. 8–9
  • CORRECT transfers across datasets, task formulations, and agentic environments, improving category recall without step-level annotations and tolerant accuracy on GAIA, ALFWorld, and WebShop.On those three AgentErrorBench tasks, GPT-5-nano accuracy improves from 36% to 46%, 22% to 35%, and 16% to 24%, respectively.Source: §5.2 · p. 9

6. Limitations

The paper identifies three limitations: decisive-error labels can be subjective, schema integration with skill systems is unresolved, and semantic similarity may not be the best retrieval signal.

6. Limitations

  • Decisive-error labels can be subjective because multi-agent failures involve interacting agents and cascading effects, potentially destabilizing future training, evaluation, or online adaptation.The authors leave more structured retrieval mechanisms, including a dedicated retrieval agent, for future work.Source: §6 · p. 9
  • How to integrate reusable error schemata with skill systems in multi-agent frameworks remains an open direction.Source: §6 · p. 9
  • CORRECT mainly retrieves schemata through semantic similarity, and whether that is the best signal remains unclear despite its robustness to retrieval noise and low costs.Source: §6 · p. 9

7. Conclusion

The conclusion presents CORRECT as a schema-guided framework for identifying decisive errors in new multi-agent runs and introduces CORRECT-ERROR as a benchmark. It reports improved error-recognition accuracy and frames the approach as a practical path toward reliable, interpretable, and scalable deployment.

7. Conclusion

  • CORRECT distills recurrent multi-agent failures into compact, reusable schemata for accurate, lightweight, training-free identification of decisive errors in new runs.Source: §7 · p. 9
  • The paper releases CORRECT-ERROR, a large-scale, high-fidelity benchmark capturing realistic error patterns.Source: §7 · p. 9
  • CORRECT significantly improves error-recognition accuracy and offers a practical path toward reliable, interpretable, and scalable multi-agent system deployment.Source: §7 · p. 9

Impact Statement

The paper aims to improve multi-agent system reliability and interpretability through precise, efficient error recognition. It argues that better debuggability in real-world deployments can support safer, more robust, and more maintainable AI systems.

Impact Statement

  • The work aims to advance multi-agent system reliability and interpretability by enabling precise and efficient error recognition.Source: Impact Statement · p. 9
  • The paper connects improved debuggability in real-world settings, including software engineering, scientific discovery, and automated decision-making, with safer, more robust, and more maintainable AI systems.Source: Impact Statement · p. 9
  • Identifying decisive errors and their propagation paths is intended to help practitioners prevent cascading failures, reduce operational costs, and support responsible system upgrades and monitoring.Source: Impact Statement · p. 9

A.1. Impact Statement

The impact statement describes a method for improving error recognition in large language model-based multi-agent systems without additional training. It says more accurate and efficient diagnosis can reduce debugging costs and support safer deployment when used responsibly.

A.1. Impact Statement

  • The method aims to improve error recognition in large language model-based multi-agent systems to enhance reliability, interpretability, and maintainability.Source: §A.1 · p. 12
  • More accurate and efficient diagnosis of failure sources without additional training can reduce debugging costs and support safer deployment of complex AI systems.Source: §A.1 · p. 12
  • The paper states that it foresees no significant negative societal impacts when the work is used responsibly.Source: §A.1 · p. 12

A.2. MAS failure trajectories are long and complex

About 17.2% of MAS trajectories exceed the Qwen models’ maximum content limit.

A.2. MAS failure trajectories are long and complex

  • 17.2% of MAS trajectories exceed the Qwen models’ maximum content limit.The paper notes that MAS failure traces are often complex and long.Source: §A.2 · p. 12

A.3. Specifics of experimental settings

The appendix reports CORRECT’s evaluation setup and additional evidence on synthetic-data validation, schema reuse, robustness, transfer, and failure cases. It also describes how CORRECT differs from static taxonomies and where multi-error trajectories can challenge its predictions.

A.3. Specifics of experimental settings

  • Evaluation excludes the correct answer from prompts to simulate deployment without known ground truth; schema generation uses GPT-5 and BAAI-BPE-M3 embeddings for similarity mapping.The pipeline hosts open-source models with vLLM and accesses GPT-series models through the OpenAI API.Source: §A.3 · p. 12

A.4. More details of the CORRECT-Error

  • 94.4% of synthetic trajectories fooled at least two labelers, while 52.9% were unanimously mistaken for genuine errors.CORRECT-Error trajectories were generated with a Magentic-One implementation using selector-group workflow control in AutoGen.Source: §A.4 · p. 13

A.7. Comparison to Human-Curated Taxonomies

  • CORRECT distills fine-grained, step-level schemata from observed trajectories and updates them asynchronously, unlike static taxonomies that cannot capture emerging failure modes under distribution drift.The appendix notes that human-curated taxonomies require expertise and can quickly become outdated as models, tools, and task distributions shift.Source: §A.7 · p. 14

A.8. Human Verification, Automated Validation, and Scalability

  • Only 58 schemata are sufficient to improve HotpotQA detection accuracy from 14.7 to 35.4, and user feedback can serve as weak supervision without additional annotation costs.When human input is unavailable, CORRECT can use an LLM-as-a-judge with confidence filtering to validate successful trajectories before adding schemata.Source: §A.8 · p. 14

A.10. Robustness to Context-Length Truncation

  • Using 500, 1000, or 2000 characters per turn yields nearly identical performance despite raw MAS logs exceeding 60k–70k tokens.The trajectories are embedded with BGE-M3, whose context window is 8192; the authors say long logs do not hinder schema matching in practice.Source: §A.10 · p. 14, Table 4 · p. 17

A.11. Failure Cases: Multi-Error Trajectories

  • In multi-error trajectories, CORRECT can predict a later error when its structural signal is stronger than that of the earliest decisive error.In the example, both Step 4 and Step 12 meet the heuristic, but Step 12 is a more overt instance of incorrect search behavior.Source: §A.11 · p. 17

A.13. Transferability to Alternative Domains and Supervision Formats

  • CORRECT improves detection across evaluated AgentErrorBench environments, with the largest gains on ALFWorld, and also improves recall on both error-category datasets.The error-category comparison is not a direct step-level comparison because that dataset lacks step-level annotations.Source: §A.13 · pp. 17–18, Table 8 · p. 18

A.14. Robustness to Retrieval Noise

  • Even with multiple irrelevant schemata in the retrieval pool, CORRECT consistently improves over the baseline.The evaluation holds k=5 schemata fixed while increasing the number of injected hard negatives.Source: §A.14 · p. 18, Table 9 · p. 18

Related papers

Loading 2509.24088v2…