ARIA: Runtime Inspection and Amendment for Multi-Agent Systems

• Inter-agent dependencies can propagate local mistakes into execution-level failures, while runtime repair must identify failure from an incomplete trajectory and amend the continuation.Prior work supports retrospective diagnosis and repair, but runtime intervention requires acting before a faulty state compounds.Source: §1 · pp. 1–2
• ARIA builds a dataset of 3,953 execution-verified failure–repair pairs from over 24,000 trajectories spanning four MAS frameworks, six benchmarks, and eight host models.Pairs are retained only when the repaired execution passes benchmark evaluation while its unmodified counterpart fails; source selection balances failure frequency with repairability.Source: §1 · p. 2
• ARIA trains the model through supervised learning, preference learning, and policy optimization using execution feedback on its proposals.The stages teach repair generation from verified examples, favor successful repairs and useful diagnoses, and refine effectiveness, transfer, and intervention decisions.Source: §1 · p. 2
• A 7B ARIA model achieves state-of-the-art executed repair accuracy in runtime and post-hoc settings, outperforming zero-shot frontier models.Source: §1 · p. 2
• ARIA is a MAS-agnostic framework that detects and repairs failures from partial execution trajectories without access to host-system internals.Its compact model combines a runtime error-detection gate with a repair-patching capability.Source: §1 · p. 2

2.1 DATASET CREATION

• Source configurations are selected per fixed framework–benchmark pair to maximize verified repair yield, balancing failure availability with conditional repairability rather than ranking host models globally.The method estimates source success and conditional repair success under a fixed protocol; invalid infrastructure runs and unresolved repair attempts are handled separately.Source: §2.1 · pp. 3–4
• Repair attributions map localized failure mechanisms to candidate interventions, and supervision is retained only when patched executions succeed while unpatched controls fail.The taxonomy has six groups and 26 categories; each attribution guides where to cut the trajectory and how to replace its continuation.Source: §2.1 · p. 4

2.2 POST-TRAINING FOR GENERALIZABLE REPAIR

• Post-training has three stages: structured repair supervision, outcome-anchored preference learning, and execution-grounded policy optimization.The first two stages use corpus-derived supervision; the third evaluates current-policy samples to address distribution shift and align training with recovery and intervention costs.Source: §2.2 · p. 4
• Stage 1 jointly learns structured diagnoses, intervention cuts, and replacement continuations, plus an inspection gate that detects validated failures and supports post-hoc and runtime repair.Feedback dropout trains with and without feedback; successful repaired prefixes serve as negative examples for the gate.Source: §2.2 · pp. 4–5
• Stage 2 prefers verified repairs over failed continuations and favors diagnoses that improve repair prediction for related incidents, while treating diagnosis uplift as a transfer proxy rather than executed success.Selected diagnoses are internalized with verified repairs before Stage 3.Source: §2.2 · p. 5
• Stage 3 re-executes proposed repairs to optimize recovery and intervention costs, credits diagnoses for within- and cross-configuration utility, and separately trains the gate to recover failures without harming successful runs.Matched-seed task outcomes compare gate-controlled intervention with an untreated baseline; alternating repair and gate optimization produces the evaluated model.Source: §2.2 · p. 5

3.1 EXPERIMENTAL SETUP

• The experiments assess post-hoc and runtime repair across four multi-agent frameworks and six benchmarks, measuring success rate, harmful interventions, and execution budget.The frameworks are MetaGPT, OpenManus, Magentic-One, and AG2; benchmarks span mathematics, knowledge, code and repository repair, and tool use.Source: §3.1 · p. 6, §3.2 · p. 6

3.2 REPAIR EFFECTIVENESS AND EXECUTION COST (RQ1)

• ARIA achieves the highest post-hoc success rate with both backbones, leads on every benchmark, and matches zero-shot and localization baselines in execution cost.DoVer requires more replay to validate interventions through re-execution.Source: §3.2 · pp. 6–7, Table 1 · p. 7
• Under a single-execution budget, ARIA delivers the strongest aggregate runtime success–harm trade-off with both backbones and generally leads across frameworks.Magentic-One with the Llama backbone is the sole per-framework exception; moving to the stronger Qwen backbone widens ARIA’s aggregate advantage.Source: Table 2 · p. 7, §3.2 · pp. 7–8, Figure 3 · p. 8

3.3 GENERALIZATION AND SCALING (RQ2)

• With Magentic-One trajectories withheld, unseen-system success closely tracks in-domain performance across scales, and larger models improve recovery and selectivity; at 7B, ARIA surpasses the strongest baseline on both metrics.The similarly sized Llama control closely follows Qwen2.5-7B. Held-out success rises with scale for both coordination structures; AG2 stays closer to its in-domain curve, while Magentic-One has the larger transfer gap.Source: Table 3 · p. 8, §3.3 · p. 8, Figure 4 · p. 9

3.4 ABLATION OF THE POST-TRAINING STAGES (RQ3)

• Stage 2 chiefly reduces harmful interventions, Stage 3 yields the larger recovery gain, and the full pipeline outperforms partial variants in-domain and on the unseen system.Every richer variant converts more failures to successes than successes to failures in both settings; the stages contribute complementary, transferable behavior.Source: Table 4 · p. 9, §3.4 · p. 9, Figure 4 · p. 9

4 CONCLUSION

• ARIA achieves strong post-hoc and runtime repair performance, transfers to unseen host models and systems, and improves with larger repair backbones.The authors report that interventions control harmful actions and execution cost, and that ablations confirm contributions from all three training stages.Source: §4 · p. 9
• From over 24,000 trajectories, ARIA builds 3,953 execution-verified repair pairs and trains with structured supervision, preference learning, and execution-grounded policy optimization.Each pair specifies what failed, where to intervene, and how execution should continue; the three-stage training moves costly repair search offline.Source: §4 · p. 9

AI USE STATEMENT

• Generative AI was used only for language editing and polishing; the authors say they manually reviewed and verified assisted text and take responsibility for the final content.They state that AI was not used for ideation, methodology, code, data collection, experiments, result analysis, or scientific claims.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The study uses established benchmarks and machine-generated trajectories, without human participants, sensitive data, or high-stakes deployment, but automated interventions may still cause unintended behavior.The authors caution that ARIA is not a guarantee of safety and recommend task-specific evaluation and appropriate human oversight before real-world deployment.Source: ETHICS STATEMENT · p. 10

A RELATED WORK

• Failure-taxonomy work maps multi-agent risks, but labels assigned to completed runs do not specify which edit would recover a failed task.Hammond et al. (2025) distinguish miscoordination, conflict, and collusion; Cemri et al. (2025) develop MAST and report taxonomy-guided system-design revisions; Jia et al. (2026) perturb executions with MAS-FIRE and find that robustness depends on more than foundation-model strength.Source: A RELATED WORK · p. 16
• Failure-diagnosis methods localize responsible agents or steps in terminated trajectories, but provide explanations rather than runtime inspection decisions or corrections.Zhang et al. (2025) report 53.5% accuracy for the responsible agent and 14.2% for the decisive step; Barke et al. (2026) and Li et al. (2026) also target diagnosis.Source: A RELATED WORK · p. 16
• Some repair methods prevent error propagation or unsafe actions, while CausalFlow and DoVer validate task-oriented repairs but rely on different settings and infrastructure.Xie et al. (2026) and Liu et al. (2026) focus on prevention; CausalFlow repairs single-agent traces at deployment time, while DoVer requires MAS checkpoint and replay infrastructure.Source: A RELATED WORK · p. 16
• ARIA instead trains one structured predictor on verified repairs across systems, emitting a diagnosis, cut point, and replacement continuation in one pass, with a prefix-level runtime gate and no host-system checkpoint or replay hook.The passage contrasts this learned predictor with per-incident test-time correction search.Source: A RELATED WORK · p. 16

B.1 EXECUTION CONFIGURATION AND BENCHMARK EVALUATION

• Each experiment configuration fixes the MAS, benchmark split and task selection, host and repair models, API protocol, decoding settings, execution limits, and evaluator.Each sample also records the source model’s identity and generation settings.Source: §B.1 · p. 17
• The source model that generated a failure can differ from the endpoint used for repair generation and paired reruns.Samples record both model identities and their generation settings.Source: §B.1 · p. 17
• The four native frameworks use separate dependency environments, while benchmark loading and evaluation use the default environment.Native revisions of OpenManus, MetaGPT, Magentic-One, and AG2 are pinned and recorded with each experiment; Magentic-One also requires Playwright Chromium.Source: §B.1 · p. 17
• Repository-local adapters normalize native messages, tool activity, and workspace artifacts without modifying the framework checkouts.Source: §B.1 · p. 17
• Native termination alone does not establish benchmark success; missing official reports, container failures, and unresolved evaluator outputs retain an unknown correctness label.Source: Table 5 · p. 17, §B.1 · p. 17
• The ten-task integration matrix uses OlympiadBench rows 1215–1224, the first ten answer-bearing English, text-only mathematics examples, and does not define the paper’s full task pools.The initial proof-only rows lack final-answer targets.Source: §B.1 · p. 17
• GAIA attachments require an authorized Hugging Face account.Source: §B.1 · p. 17

B.2 SOURCE ADMISSION AND INCIDENT CONSTRUCTION

• A cognitive repair incident requires native completion and an explicit incorrect evaluator verdict; transport, runtime, tool-backend, and evaluator failures are recorded separately without cognitive labels.Source cells fix the framework, benchmark, and host-model configuration; records are deduplicated by sample identity using the latest result.Source: §B.2 · p. 17
• Incidents expose the public task, constraints, and normalized observable events, while excluding reference answers, hidden tests, gold patches, and private evaluation details from model inputs.Native traces are preferred, with messages and available workspace artifacts as fallbacks; the default budget is 64 events, retaining the first four and last 60 when longer. Localization cannot certify an omitted event as the earliest error, and references remain only in host-side audit records.Source: §B.2 · p. 17
• Successful source runs provide non-intervention supervision, with the maintained repair configurations setting the failure-relative success quota rate to ρ = 0.25.For F > 0 failures and ρ > 0, the default quota is max(1, ⌊ρF⌋), capped by available successes; otherwise it is zero. Eligible successes are taken in source order unless overridden, without repair generation or paired reruns.Source: §B.2 · p. 17

B.3 LOCALIZATION, TRAJECTORY CUTS, AND REPEATED REPAIR TRIALS

• Each repair trial predicts localization, attribution, and a patch: it identifies bad events and evidence, assigns repair codes and diagnosis, then specifies a retained cut and replacement events.The patch must copy the accepted attribution’s diagnosis exactly and cannot reproduce the discarded suffix unchanged; schema and grounding checks validate event references and order, not causal correctness.Source: §B.3 · pp. 17–18
• A cut retains the trajectory through event e_k and replaces its suffix; it must precede the earliest localized bad event, and patches allow at most eight replacement events and 12,000 serialized characters.A null cut retains an empty prefix; replacement identifiers cannot collide with retained identifiers, and models cannot fabricate tool-result events.Source: §B.3 · p. 18
• Configured independent trials each restart from the same incident, while bounded retries fix invalid stage outputs within a trial; maintained configurations use n = 1.Eighteen configurations allow two structured-output retries per stage and six allow eight; increasing n preserves invalid generations and failed repairs, and deterministic decoding does not ensure distinct outputs.Source: §B.3 · p. 18
• A repair is verified only when the patched execution passes and a fresh shared unpatched control fails; the first verified success is canonical, while every verified trial may supply training data.Paired runs use separate workspaces and the benchmark evaluator; reconstructed continuation contexts validate recovery under that protocol, not checkpoint-exact causal replay.Source: §B.3 · p. 18

B.4 TRAINING PROJECTION AND DATA PARTITIONING

• A shared projection procedure derives examples from detailed repair records, deduplicating identifiers, rejecting conflicts, checking answer blindness, and matching DPO pairs to structured-repair SFT rows.Manifests keep source-task, repair-trial, verified-repair, and training-row counts separate because one incident can yield multiple prefixes and verified patches.Source: §B.4 · p. 18
• Gate targets turn positive at or after the selected first bad index, while success prefixes and prefixes inside the verified replacement receive zero.If no valid first-bad index exists, only the terminal failure prefix is positive; gate inputs omit evaluator feedback. Valid localization and attribution labels can train even when the resulting patch fails, but structured-repair positives and preferences require a verified passing patch paired with a failing control.Source: §B.4 · p. 18
• Preference training chooses a verified patch and rejects an executed failed trial when available; otherwise, it encodes the observed failed suffix after the chosen cut.The fallback retains at most three events, shares the chosen diagnosis and cut, and is not a newly executed negative; rejection labels remain audit metadata, not model-visible cues.Source: §B.4 · pp. 18–19
• Task identifiers—not individual prefixes or trials—are assigned to seeded 60%/20%/20% train, validation, and test partitions, and all derived rows inherit the task assignment.Per-cell proportions are approximate, strata with fewer than three tasks go to training, and audits check held-out overlap by task ID and normalized task text. For unseen-system evaluation, the held-out axis is excluded from training, calibration, and checkpoint selection.Source: §B.4 · p. 19
• RQ1 uses complete task-level partitions, while RQ2 and RQ3 use separate fixed subsets for repeated backbone-scaling and ablation runs.Unseen-system tests exclude all held-out-framework trajectories from training, calibration, validation, and checkpoint selection: RQ2 holds out Magentic-One or AG2, and RQ3 holds out Magentic-One. Subset caps and task pools are fixed by study manifests, so rates across different pools are not directly comparable.Source: §B.4 · p. 19

B.5 POST-TRAINING PROCEDURE

• The five sequential phases are structured SFT, repair DPO, diagnosis transfer, repair GRPO, and gate GRPO.The reference trainer uses AdamW and LoRA; merged-weight export remains LoRA rather than full-parameter fine-tuning.Source: §B.5 · p. 19
• Input reduction preserves event identities and JSON structure, and targets are not truncated into incomplete JSON.Oversized or ungrounded records follow the configured admission policy and are recorded in the report.Source: §B.5 · p. 19

B.5.1 STAGE 1: STRUCTURED SUPERVISION

• Structured SFT predicts gate probability, localization, attribution, and structured repair, while masking prompt tokens from the causal-language-model loss.Source: §B.5.1 · p. 19
• In cascade training, attribution uses recorded localization and repair uses recorded attribution; deployment uses model predictions, allowing errors to propagate between stages.Feedback dropout removes evaluator feedback at a configured probability rather than paraphrasing hidden evaluator information.Source: §B.5.1 · p. 19

B.5.2 STAGE 2: REPAIR PREFERENCES AND DIAGNOSIS TRANSFER

• Repair DPO compares chosen and rejected records under a shared input using frozen stagestart reference log probabilities.Diagnosis-transfer mining builds training-only incident families from sparse token-count representations of feedback and repair edits, using connected components of a thresholded mutual-nearest-neighbor graph based on mean cosine similarity; families require at least three members.Source: §B.5.2 · p. 19
• Diagnosis candidates include source, policy-generated, and eligible sibling diagnoses; the selected diagnosis maximizes calibration-sibling repair-likelihood uplift subject to thresholds and positive uplift on a reserved training sibling.The lowest-scoring distinct candidate supplies the preference rejection; the reserved sibling is training data, not validation or test data.Source: §B.5.2 · p. 19
• Diagnosis DPO is followed by SFT on the complete verified repair record with the selected diagnosis substituted; this transfer signal measures likelihood uplift, not newly executed recovery.Source: §B.5.2 · p. 19

B.5.3 STAGE 3: EXECUTION-GROUNDED UPDATES

• Valid repair patches receive backend-scored outcomes, with native execution distinguished from evaluator-only or recorded outcomes; diagnosis can use sibling rewards, and low-variation groups are discarded.Diagnosis and localization advantages are standardized across branches, while patch advantages are standardized within a branch; missing scoreable sibling evidence falls back to the source reward.Source: §B.5.3 · p. 20
• The trainer uses a reference-policy penalty and token-level GRPO, but its single inner update yields a probability ratio of one rather than multiple clipped PPO epochs.The default reference-policy penalty coefficient is 0.02.Source: §B.5.3 · p. 20
• Gate optimization samples historical trajectory prefixes against paired same-host controls; scalar mode freezes repair parameters, whereas contextual mode updates shared LoRA parameters.Historical-prefix rollouts differ from live interception starting at task initialization, and the selected gate mode is part of checkpoint identity.Source: §B.5.3 · p. 20

B.6 INTEGRATION WITH NATIVE MAS RUNTIMES

• At the native model-response boundary, ARIA inspects only the observable prefix, and the host validates and decides whether a proposed replacement can be enacted.The current integrations allow at most one pending plain-text substitution per run; unsupported proposals remain shadow records, and malformed predictions are recorded as errors rather than negative gate decisions.Source: §B.6 · p. 20
• Baseline comparisons use repository-local adaptations over shared interfaces, with matched task and host settings within panels and method-specific prompts and extra calls retained in audit and cost records.Post-hoc methods receive failed logs, public feedback, and constraints; live Spark to Fire and SafeAgent receive prefixes without feedback and share the pending-text substitution limit.Source: §B.6 · pp. 20–21

B.7 SUCCESS, HARM, AND EXECUTION-COST ACCOUNTING

• Runtime harm is task-normalized across the fixed evaluation pool, while strict headline success and harm rates are withheld until every planned case resolves.With binary grading, harm means a passing control becomes a failure after intervention; the live reporter separately reports a harmful-substitution lower bound, unresolved cases, and decision errors.Source: §B.7 · p. 21
• Execution budget counts recorded full-MAS executions per task, with runtime including initial execution and post-hoc cost excluding the already acquired failed source.ARIA calls, tokens, and latency are reported separately; post-hoc candidate selection stops at evaluator-confirmed success, while runtime selection uses public-output agreement rather than hidden verdicts.Source: §B.7 · p. 21

B.8 ARTIFACTS AND RESUME SEMANTICS

• Reproducibility artifacts retain configuration, provenance, outcomes, validation and reporting records; repair resume restarts incomplete samples, while sample-level guarantees do not ensure exact optimizer or RNG restoration.Native batch resume skips completed samples and retries failed ones in clean workspaces.Source: §B.8 · p. 21

C REPAIR ATTRIBUTION TAXONOMY

• The taxonomy comprises six groups and 26 categories, with each tag linked to a candidate patch operator; names are shortened for presentation.Colors indicate group membership.Source: Table 6 · p. 22
• Each attribution identifies one primary localized mechanism, while secondary codes can record contributing factors.The attribution proposes what to change.Source: C REPAIR ATTRIBUTION TAXONOMY · p. 22
• RA-3.3 suggests replacing a finite-search claim of proof with a deductive argument, whereas RA-6.3 calls for revising acceptance after an ignored failed check.Paired execution determines whether the resulting patch provides positive repair supervision.Source: C REPAIR ATTRIBUTION TAXONOMY · p. 22

D REPAIR UTILITY: PROOF AND COUNTING UNIT

• E[FiRi] = (1−a(c))r(c) for a valid source execution, yielding the expected repair utility identity.Fi indicates source failure and Ri indicates successful repair under the fixed repair protocol.Source: D REPAIR UTILITY: PROOF AND COUNTING… · p. 22
• Summing over N source executions gives E[Σ FiRi] = NS(c).The passage defines S(c) = (1−a(c))r(c).Source: D REPAIR UTILITY: PROOF AND COUNTING… · p. 22
• Repair success is counted per incident: at least one verified passing patch paired with a failing control makes an incident successful.The utility measures incidents yielding usable repairs per valid source execution, not successful trial-level training rows.Source: D REPAIR UTILITY: PROOF AND COUNTING… · p. 22
• Comparisons of source cells must hold trial count, endpoint, and execution limits fixed.The counting unit must remain fixed when estimating r(c).Source: D REPAIR UTILITY: PROOF AND COUNTING… · p. 22
• Unresolved outcomes are reported separately and never counted as verified repairs.Source: D REPAIR UTILITY: PROOF AND COUNTING… · p. 22

E RELATED-WORK CAPABILITY COMPARISON

• Table 7 compares capabilities each method explicitly reports; its check marks indicate reported capabilities, not performance comparisons.Verified repair is defined as re-executing a corrective modification under a task evaluator.Source: Table 7 · p. 23