Aegis: Automated Error Generation and Attribution for Multi-Agent Systems

1 INTRODUCTION

• A single agent’s error can cascade through interactions into a distant final error, making root-cause analysis and systematic debugging difficult.Existing attribution resources are small and rely on costly expert annotation, constraining the creation of larger datasets.Source: §1 · pp. 1–2
• Aegis programmatically injects context-aware interventions into successful executions and validates which agents and error modes caused the resulting failures.Its reproducible labels turn the human-annotation bottleneck into an engineering problem that can be scaled.Source: §1 · p. 2
• Aegis produces 9,533 annotated error trajectories across six MAS frameworks and six benchmarks spanning varied architectures and task domains.The pipeline collects successful baselines, generates faulty variants, and retains runs that fail as intended with reliable attribution labels.Source: §1 · p. 2
• Aegis data supports supervised fine-tuning, reinforcement learning, and contrastive learning, and training improves attribution on Aegis-Bench and out-of-distribution Who&When.The paradigms use input-target pairs, hierarchical attribution-aware reward, and positive-negative trajectory pairs, respectively.Source: §1 · p. 2
• The paper contributes a reproducible generation pipeline, a diverse annotated dataset and evaluation protocol, empirical validation, and open-source code, data, and models.The authors position these resources as a foundation for future research on reliable and debuggable MAS.Source: §1 · p. 2

2 RELATED WORKS

• Prior MAS research studies communicative, role-based, debate, dynamic, graph-based, scaling, and tool-augmented systems across complex tasks.These advances also expose systems to fragility, motivating attention to reliability (Huang et al., 2024; Zheng et al., 2025).Source: §2 · p. 2
• Automated synthesis work generates tasks, benchmarks, challenger-solver loops, curricula, prompts, and verifiable examples across LLM domains.These efforts position automatic synthesis as a route toward scalable, self-improving intelligence, which Aegis extends to MAS error attribution.Source: §2 · p. 3
• MAS error-attribution research includes taxonomies and benchmarks, while related work examines cascading risks, social vulnerabilities, and defenses against adversarial attacks.Distributed-systems research also supplies established approaches to tracing, root-cause analysis, and topology-aware anomaly detection.Source: §2 · p. 3

3 PROBLEM FORMULATION

• The framework’s goal is fine-grained attribution of MAS failures to responsible agents and their error modes.The formulation focuses on identifying both who is responsible and the nature of the errors.Source: §3 · p. 3

3.1 MAS TRAJECTORIES AND FAILURES

• A MAS comprises a finite set of agents operating in a state space and joint action space, with each agent having a local action space.A scheduling policy selects the active agent at each discrete time step.Source: §3.1 · p. 3
• A trajectory is the sequence of system states and actions, and its final outcome is classified by a binary function in which 1 indicates failure.The system evolves through a transition function after the scheduled agent takes an action.Source: §3.1 · p. 3

3.2 FINE-GRAINED ERROR ATTRIBUTION

• For a failed trajectory, the ground-truth label is a structured set of true agent and error-mode pairs.Each pair identifies a faulty agent and the non-empty set of error modes committed by that agent.Source: §3.2 · p. 3
• The error taxonomy contains M modes, and the attribution target associates each faulty agent with its applicable subset of those modes.The notation represents faulty agents as a subset of all agents and their error modes as subsets of the taxonomy.Source: §3.2 · p. 3
• The diagnostic model maps a failed trajectory to a predicted attribution map that approximates its ground-truth label.The task is to learn fθ so that its prediction for a trajectory τ approximates G(τ).Source: §3.2 · p. 3

4.1 DATA SOURCE

• The dataset covers six MAS frameworks with varied topologies and coordination patterns, including configurable networks, dynamic graphs, debate, hierarchical roles, orchestrator-executor, and tool-using ReAct systems.The frameworks include MacNet, DyLAN, Debate, AgentVerse, Magnetic-One, and SmolAgents.Source: §4.1 · pp. 3–4
• Figure 1 depicts data construction on the left, where successful trajectories receive controlled, context-aware errors, and three learning methods on the right.The dataset structure supports the error-attribution task.Source: Figure 1 · p. 4
• Six benchmarks cover mathematical reasoning, code generation, scientific problem solving, multidisciplinary knowledge, and general assistant capabilities.The tasks are MATH, GSM8K, HumanEval, SciBench, MMLU-Pro, and GAIA.Source: §4.1 · p. 4

4.2 DATA COLLECTION

• Data collection first creates deterministic successful baseline trajectories, then uses them as references for failure injection.The baseline is generated under deterministic agent settings, such as temperature 0 and a fixed random seed.Source: §4.2 · p. 4
• An adaptive manipulator injects task-relevant errors using prompt injection or response corruption, producing distinct faulty counterparts from correct trajectories.Examples include introducing an infinite loop in coding or a plausible but incorrect calculation in math.Source: §4.2 · p. 4

4.3 IMPLEMENTATION DETAILS

• Generated trajectories are retained only when intervention induces failure, with attribution labels known by construction from the injected errors.Non-invasive wrappers intercept agent behavior; the implementation uses a 14-mode taxonomy and reports a final dataset of 9,533 trajectories.Source: §4.2 · p. 5, §4.3 · p. 5

5 METHODOLOGY

• Ground-truth attribution maps support learning trajectory-to-diagnosis mappings, while agent and error labels provide multi-level signals for reinforcement learning and correct-to-faulty pairs support contrastive learning.These properties make the dataset usable across three distinct learning paradigms.Source: §5 · p. 5

5.1 SUPERVISED FINE-TUNING

• SFT trains a sequence-to-sequence model to map serialized trajectory logs to JSON identifying faulty agents and their error modes.The prompt includes a role, taxonomy definitions, and the full conversation log; training maximizes the target output's conditional probability given the prompt.Source: §5.1 · p. 5

5.2 REINFORCEMENT LEARNING

• RL parses predictions and ground truth into agent-error pairs, assigning negative reward to malformed outputs and penalizing duplicate or excessive predictions.Partial credits are non-repeatable, and the reward is normalized by the maximum attainable score before GRPO optimization.Source: §5.2 · p. 6

5.3 CONTRASTIVE LEARNING

• DCL represents each trajectory as turns, uses attention to weight salient turns, and aligns turn representations with separate agent and error-mode prototype banks.The resulting agent and error-mode distributions are combined into a joint probability.Source: §5.3 · p. 6
• DCL combines multi-label classification, supervised contrastive learning, and hierarchical consistency regularization to train attribution representations.The contrastive loss draws salient error turns toward positive counterparts and away from negatives; the consistency term penalizes agent-error pair probabilities exceeding either constituent probability.Source: §5.3 · p. 6

Datasets

• Aegis-Bench contains 100 trajectories sampled from each of six benchmarks; the remainder is split 80%/20% for training and validation using a fixed random seed.Who&When is also used as an out-of-distribution test set.Source: §6 · p. 6
• Who&When's free-text mistake reasons are mapped by Gemini-2.5-Flash to one of 14 predefined error modes for consistent evaluation labels.Source: §6 · p. 7

Metrics

• At Pair, Agent, and Error levels, the evaluation reports Micro-F1 and Macro-F1; Macro-F1 gives equal weight to classes, including infrequent error modes.Table 1 reports these scores as percentages on Aegis-Bench and Who&When.Source: §6 · p. 7, Table 1 · p. 7

Training Details

• SFT and RL fine-tuning focuses on 7B–14B models; Qwen3-8B-Reasoning receives GRPO only, while contrastive models use all-MiniLM-L6-V2 for two epochs.SFT and RL use four A800 GPUs; contrastive models use a separate four-GPU 4090 cluster.Source: §6 · p. 7

Comparison Models

• Open-source and proprietary comparison models are evaluated zero-shot on both benchmarks using the same instruction as the SFT models.The comparison set includes Qwen2.5-72B-Instruct, GPT-4.1, o3, Gemini-2.5-Pro, and Claude-Sonnet-4.Source: §6 · p. 7

• 26.51 average score gives Aegis-SFT the strongest overall result, nearly doubling base Qwen2.5-14B-Instruct’s 13.99.Aegis-GRPO scores 18.41, and o3 is the strongest general-purpose LLM at 20.24.Source: §7 · p. 8
• Micro-F1 exceeds Macro-F1 across models, and Agent-level attribution is easier than Error-level attribution.The authors relate the first gap to long-tail error modes and the second to the deeper semantic understanding needed to diagnose why failures occur.Source: §7 · p. 8
• Aegis-SFT gains broadly across task domains but less on GAIA, while Dylan and MacNet are harder architectures where fine-tuning helps most.On smaller Magentic-One and SmolAgents systems, SFT slightly underperforms its base model, whereas GRPO remains stable.Source: §7 · p. 8, Figure 2 · p. 8
• GRPO reward improves steadily, with 14B models reaching higher reward levels than 7B models.Qwen3-8B-Thinking initially suffers output truncation from verbose reasoning but quickly learns to produce parsable answers within token limits.Source: §7 · pp. 8–9, Figure 3 · p. 9
• CoT prompting hurts base open-source models but boosts them after GRPO training; GPT-4o-mini and Gemini-2.5-Flash benefit without that training.The authors attribute base-model failures often to verbose reasoning disrupting the required JSON format.Source: §7 · p. 9, Figure 3 · p. 9
• OOD Who&When performance peaks after two SFT epochs and then declines, although in-domain Aegis-Bench performance keeps improving.The authors select two epochs to balance in-domain performance with OOD generalization, attributing later overfitting to synthetic-data style patterns.Source: §7 · p. 9, Figure 3 · p. 9
• Trajectory-level embeddings are compact and separated, whereas individual-turn embeddings are diffuse and overlapping.The authors attribute the trajectory representation to attention-based aggregation of sparse signals from high-variance turns.Source: §7 · p. 9, Figure 4 · p. 9
• Removing compositional consistency causes near-collapse in Pair-level accuracy, while semantic prototypes and the compositional design also prove critical.The reported trends hold on the OOD Who&When benchmark; DCL and its ablations are averaged over three seeds.Source: §7 · p. 9

8 CONCLUSION

• Aegis programmatically injects controlled faults into successful executions to produce more than 9,500 annotated failures, addressing MAS error-attribution data scarcity.The experiments report specialized state-of-the-art attribution performance and strong generalization to the human-annotated Who&When benchmark.Source: §8 · p. 10
• A predefined error taxonomy and controlled injection plans limit the approach; the authors identify more complex, emergent failures as future research.Source: §8 · p. 10

A CASE STUDIES

• Two Who&When case studies illustrate error propagation and show that even strong proprietary models can struggle to identify root-cause failures.Source: A CASE STUDIES · p. 15

A.1 CASE 1: FINANCIAL PLANNING TASK

• Aegis identifies Tickets_Pricing_Expert as the root cause, while Gemini-2.5-Pro blames the verifier and GPT-4o-mini labels the faulty trajectory passed.The pricing error corrupts the downstream budget calculation.Source: Figure 5 · p. 15, §A.1 · p. 15

A.2 CASE 2: GEOGRAPHIC REASONING TASK

• All four models misattribute the geographic case’s subtle distance-conversion error, where 1 mile is incorrectly treated as 20 blocks.The assumption inflates calculated distances and leads the system to conclude that no supermarket meets the two-block requirement.Source: §A.2 · p. 16

B DATASET DETAILS

• Building Aegis requires judgments about task-result correctness to filter valid data.Source: B DATASET DETAILS · p. 16

B.1 TASK EVALUATION

• A standardized evaluation framework supports heterogeneous datasets through dataset-specific evaluators and answer-extraction and comparison logic.It covers six benchmark categories, including mathematical reasoning, code generation, scientific computing, multiple choice, and general tasks.Source: §B.1 · pp. 16–17

B.2 DATASET COMPOSITION

• Aegis contains 9,533 trajectories across six task domains and six frameworks, with 14 error modes; injection success is 76.5% in LLM-Debate and 34.1% in Dylan.The injection strategy also targets an average of 3.6 agents in MacNet versus 1.9 in LLM-Debate.Source: §B.2 · p. 17

B.3 DATASET QUALITY VALIDATION

• Programmatic labels achieve κ = 0.81 agreement with human annotators, compared with κ = 0.85 human-human agreement; the cited MAST benchmarks are κ = 0.77 and κ = 0.88, respectively.Source: §B.3 · p. 18
• Aegis-Bench trajectories are heavily interspersed with MAST and WhoWhen embeddings, with a real-versus-synthetic Silhouette score of 0.11.The reported centroid distance is 0.521 for Aegis versus a real benchmark and 0.490 between the two real benchmarks.Source: §B.3 · pp. 18–19, Figure 7 · p. 19, Table 4 · p. 19
• The authors conclude that Aegis-Bench is not distributionally misaligned and that its variation falls within the observed range between real-world datasets.Source: §B.3 · p. 19

C.1 SUPERVISED FINE-TUNING

• SFT minimizes cross-entropy on response tokens only, so input prompt tokens do not contribute to the loss.The loss mask selects response tokens for training.Source: §C.1 · p. 19

C.2 REINFORCEMENT LEARNING

• GRPO estimates each response’s advantage relative to responses from the same prompt, then uses PPO-style optimization with importance sampling and clipping.Group normalization uses reward mean and standard deviation, with a small constant for numerical stability; the approach is intended to reduce sensitivity to reward-scale variation and miscalibration.Source: §C.2 · p. 20

C.3 DISENTANGLED CONTRASTIVE LEARNING

• DCL treats failed trajectories as bags, selects salient evidence turns, and contrasts them against corresponding clean turns and agent/error prototypes.It also uses other prototypes and hard negatives, then combines agent and error predictions into a pair attribution constrained by hierarchical consistency.Source: §C.3 · pp. 20–21
• DCL combines multi-label classification across agent, error-mode, and agent-error levels with supervised contrastive learning and a hierarchical consistency regularizer.The hierarchical loss penalizes pair probabilities that exceed the probabilities of their agent or error components.Source: §C.3 · pp. 21–22

D.1 TRAINING DETAILS

• LLM training uses SFT followed by GRPO, while DCL is trained from scratch with a shared MiniLM encoder and contrastive loss on top-K evidence turns.The LLM stages use 4 GPUs; DCL uses 4 GPUs, runs for 2 epochs of 500 steps, and tunes key hyperparameters on validation data.Source: §D.1 · pp. 22–23, §C.3 · p. 22, Table 5 · p. 22

D.2 RESULTS DETAILS

• Precision and recall rankings agree with the F1 conclusions, and three-seed DCL results are reported as confirming ablation-study stability.Tables 8 and 9 break down micro/macro precision and recall; Table 10 reports mean and standard deviation over three runs.Source: §D.2 · p. 23, Table 8 · p. 24, Table 9 · p. 25
• Agent MF1 correlates weakly (r=0.166), while Agent µF1 correlates more strongly (r=0.714); pair-level correlation is low (r=-0.106).The authors attribute the weaker Agent MF1 correlation to differing long-tail roles and the pair-level correlation to scores clustered near zero.Source: §D.2 · p. 23
• Aegis-Bench and Who&When scores show strong, statistically significant positive correlations for Agent µF1 and Error MF1/µF1.The authors interpret these correlations as evidence that training on Aegis teaches skills for diagnosing real-world failures.Source: §D.2 · p. 23, Table 11 · p. 26

E.1 ADAPTIVE MANIPULATOR PROMPT TEMPLATES

• The prompt-injection template uses agent context and a specified error pattern to modify input so the target task fails, while keeping the injection subtle.It requires outputting only the modified input.Source: §E.1 · p. 27
• The response-corruption template uses agent context to turn a correct response into an incorrect, misleading, or harmful but plausible one.It requires outputting only the corrupted response without revealing the modification.Source: §E.1 · p. 28

E.2 ERROR MODES INSTRUCTION

• The induced-error prompts cover task and role specification deviation, redundant loops, ignoring history, removing stopping criteria, and restarting handled work.Source: §E.2 · pp. 28–29
• Further prompts target removing conversation history or termination conditions, repeating handled tasks, making ambiguous requests, and deviating from the main goal.Source: §E.2 · p. 29
• Other modes induce dangerous assumptions, goal drift, withholding critical information, and rejecting other agents’ suggestions or corrections.Source: §E.2 · p. 29
• The prompts also create stubborn non-cooperation, inconsistent reasoning and actions, premature termination, and omission of verification steps.Source: §E.2 · p. 30
• The final modes sabotage verification by removing checks or making the verification process flawed so that failures are approved as successes.Source: §E.2 · p. 30

E.3 EVALUATION PROMPTS

• The same evaluation prompts are used for AEGIS-Bench and who&when because the benchmarks share a unified data format.The passage presents this as a way to ensure fair assessment.Source: §E.3 · p. 30
• Standard and chain-of-thought evaluation prompts ask a MAS quality-assurance analyst to identify and categorize agent errors in conversation logs.The chain-of-thought prompt specifies turn-by-turn analysis against the error definitions before compiling the findings.Source: §E.3 · pp. 30–32
• The document requires exact error codes, presents error definitions with examples, and groups functional mistakes into FM-1 task execution, FM-2 communication, and FM-3 quality and verification categories.Source: §E.3 · pp. 30–31
• The evaluation output must be a single JSON object listing faulty agents and their error types, or an empty faulty_agents list when no errors are found.The prompt requires exact error codes and provides examples for multiple errors and no errors.Source: §E.3 · p. 32
• The evaluation prompt asks reviewers to summarize each agent, inspect task, role, communication, and quality failures, then identify faulty agents and assign error codes.It requires exact agent-name matching and a JSON conclusion, including an empty faulty_agents list when no errors are found.Source: §E.3 · pp. 33–34
• A separate answer-checking prompt prioritizes semantic alignment over exact wording, accepts incomplete phrasing that captures the main idea, and requires only “Correct” or “Incorrect.”Source: §E.3 · pp. 34–35
• The Who&When classifier maps a mistake reason to exactly one of 14 FM error codes, choosing the closest match when no category clearly fits.Source: §E.3 · p. 35