Source-linked AI summary
CatchBench: When Can an Agent Failure Be Caught?
Yue Zhao
TL;DR
CatchBench asks when agent failures can be caught under PRE, LIVE, and POST evidence, addressing the gap left by benchmarks that fix one state or vary telemetry. It evaluates seven task contracts through a common interface, finding limited resolved ordering and exposing label-construction shortcuts and inadmissible injected substrates. The paper therefore treats benchmark scores as interpretable only alongside the process that produced their labels and the tests applied to that process.
Problem
Prior benchmarks do not score agent auditing across all three evidence states under one task-method interface, although the available record determines which audits are possible.
Method
CatchBench holds the auditor’s question and data interface fixed while evaluating seven task contracts across PRE, LIVE, and POST with board-specific labels, metrics, and entrants.
Results
47 of 118 pre-declared contrasts separate, while one position-only rule reaches perfect detection on a configuration source by exploiting its label-construction process.
Takeaways & Limitations
Benchmark results should be read with their label-generation process and admissibility tests because corpus construction can determine scores and injected substrates may lack evidential validity.
Takeaways & Limitations
Gold-derived boards remain mechanism diagnostics because the injected substrates fail the admissibility requirements for evidential use.
Abstract
from arXiv · showhide
When can an agent failure be caught? An audit is usually limited by the record rather than by the method. CatchBench therefore puts one auditor's question to three information states: the declared configuration before a run (PRE), a growing prefix of its trace (LIVE), and the finished trace (POST). Prior benchmarks fix one of these states or vary the telemetry; to our knowledge none scores all three under one task-method interface. Each state admits different questions, so seven task contracts carry their own labels and metrics rather than one leaderboard. Four are evidential; three are Gold-derived mechanism diagnostics. The release scores 72 entrants, from rule scanners and structural models to eleven LLM judges across nine model families (GPT, Claude, Gemini, Gemma, Llama, Qwen, DeepSeek, Mistral, Nova), over 1187 declared configurations and 1162 recorded runs. Most of the arena does not order: 47 of 118 pre-declared contrasts separate, and the rest are published unresolved rather than ranked. The two sharpest results cut against our own data. One rule ignores every name and permission; it flags each capability declared after the first. On one of six configuration sources it reaches a perfect F1, so a score there measures how the corpus was built rather than how well a method reasons. Our admissibility bar then rejected one injected substrate and withheld evidential status from the other. A benchmark number is therefore not interpretable until the process behind its labels is published and tested for the shortcut it may leave. We report both, and regenerate every ordering from released predictions with no model call.
1 INTRODUCTION
CatchBench frames agent auditing around three information states and holds the auditor’s question and data interface fixed while evidence changes. Its results emphasize unresolved comparisons and label-construction shortcuts that can make benchmark scores uninterpretable.
- Benchmark scope: 72 entrants are evaluated across nine scored boards using 1187 declared configurations and 1162 recorded runs.Entrants include rule scanners, structural models, supervised references, anomaly detectors, and LLM judges from nine families.
- Benchmark motivation: CatchBench scores auditors across PRE, LIVE, and POST information states under one task-method interface.The arena uses separate contracts because each state supports different audit questions.
- Results: 47 of 118 pre-declared contrasts separate, while the remaining orderings are published unresolved.Comparisons use pre-declared, Holm-corrected families rather than forcing a complete ranking.
- Results: A position-only rule achieves perfect detection on one configuration source by exploiting how capabilities were labeled.It flags all 510 excess capabilities there without using names or permissions, showing that the score reflects corpus construction.
- Evidence validity: Neither injected Gold substrate clears the admissibility bar, so the three Gold boards are mechanism diagnostics rather than evidential benchmarks.The file-level baseline detects all planted steps and no clean controls, exposing a construction shortcut.
2 RELATED WORK
CatchBench builds on fixed-data, many-method benchmark contracts while distinguishing its reliability-framed audit tasks from neighboring agent-auditing, observability, attribution, and graph-security benchmarks.
- Fixed-data benchmarks: ADBench and BOND provide the fixed-data, many-method, per-type, and runtime comparison template that CatchBench inherits.Neither benchmark audits agents directly; their contribution is the comparison structure.
- Agent-auditing neighbors: R-Judge and Agent Security Bench establish agent-auditing precedents with different evaluated records, threats, and scoring contracts.R-Judge uses human-annotated interaction records, while Agent Security Bench executes attacks and defenses across tool-using scenarios.
- Adjacent systems: AgentTelemetry, Who&When Pro, GAMMAF, and AgenTracer differ in observability, post-attribution, graph-security, or counterfactual-injection design.CatchBench’s distinction is a reliability-framed task contract, not the claim that agent-security or agent-auditing benchmarks do not exist.
3 THE BENCHMARK
The benchmark separates PRE, LIVE, and POST because each state permits different audit questions, then gives every contract its own labels, metrics, entrants, floors, and statistical tests. Corpus size and admissibility limits constrain which comparisons can be resolved.
- 3.1 THE THREE INFORMATION STATES: PRE permits static over-privilege checks, LIVE supports predictive early warning, and POST supports forensic localization, detection, and attribution.A method built for one state cannot read another state’s evidence, so the states remain separate tracks.
- 3.2 AUDIT SCENARIOS AND THEIR METRICS: Seven task contracts become nine scored blocks because localization has natural-failure and injected-fault instantiations, with each contract fixing its task, labels, metric, and baselines.Every board also prints its trivial floor, entrant field, and declared separations.
- 3.2 AUDIT SCENARIOS AND THEIR METRICS: Localization uses Top-k and MRR, detection and cause attribution use ROC-AUC, streaming adds time to detection, and online detection fixes a false-positive budget.The metric follows the audit question rather than convention.
- 3.3 THE SCORING CONTRACT: Every method receives the same task object on each board, enabling paired comparisons using McNemar, DeLong, bootstrap, or exact randomization tests.Sixteen comparison families are declared before scoring.
- 3.3 THE SCORING CONTRACT: Only gaps near 0.20 on the 126-run Top-1 column and near 0.06 on the 376-run detection board separate reliably.Unresolved comparisons are reported as point-estimate orderings rather than claimed results.
- 3.3 THE SCORING CONTRACT: Table 2 reports nine blocks with board-specific floors, entrant fields, LIVE-prefix averages, and 47 resolved contrasts out of 118.Gold boards are marked as mechanism diagnostics rather than artifact-controlled evidence.
4 THE DATA
CatchBench combines natural trace failures, declared configurations, and injected fault mechanisms, while preserving each label process and testing injected substrates before assigning evidential weight.
- Data families: Who&When supplies natural localization failures, SWE-Gym and tau-bench supply POST and LIVE runs, and six sources supply PRE configurations.SWE-Gym and tau-bench also support the two Gold injection substrates.
- Natural trace data: Who&When contributes 126 Algorithm-Generated failures, excluding 58 Hand-Crafted runs from a distinct population.Structural methods require a validated conversion before Who&When Pro can join the board.
- PRE data: The PRE corpus contains 1187 configurations with four explicit label processes, including LLM judging, roster relabeling, declared-minus-used labels, and controlled injection.Unused in one observed run is only a proxy for unneeded access.
- Gold data: Gold injects known dependency faults into real runs because labeled dependency-state failures are scarce at scale.The injection site supplies detector-independent localization and attribution targets.
- Admissibility: The admissibility bar requires realism, no trivial construction baseline, comparable distributions, detector-independent labels, and human validation.Both Gold substrates fail at least one required test and therefore carry no evidential weight.
- Admissibility: Gold v2 mutates an argument value and rebuilds dependency edges, removing the file-level edited-edge marker but retaining unresolved validation limits.Its stale-state sample lacks the validation needed for evidential use.
5 RESULTS
The results are organized as separate board questions, ordered by the agent lifecycle from pre-run auditing through post-run analysis.
- Five questions are evaluated in lifecycle order, from before a run to after it.
- The results section treats each board as answering its own audit question.
- The lifecycle framing covers what can be audited before execution, during execution, and after execution.
1. Does a static harness audit beat flagging every declared capability? (Section 5.1, PRE)
PRE results show that static methods sometimes beat source-specific flag-all floors, but gains depend strongly on label process and can reflect corpus construction rather than reasoning.
- 0.695 F1: the held-out judge beats the combined scanner’s 0.648 on 1182 commonly judged configurations.
- 0.990 and 0.972 F1 on two sources contrast with 0.362 to 0.744 on judge-labeled sources, showing strong dependence on label process.
- 0.053 F1: the combined scanner improves over flag all, reaching 0.654 against 0.601.
- Four of six configuration sources separate from their floors, while sweagent and mcp remain unresolved.
- PRE contributes a labeled dataset and shared arena, not a new detector.
6 DISCUSSION AND LIMITATIONS
CatchBench argues that audit methods are comparable only relative to the evidence state and task board, while unresolved contrasts and data limitations constrain broad rankings.
- Read together, the boards indicate which evidence makes methods comparable rather than selecting one universally best method.
- Leaders differ across boards because each information state determines what evidence and audit are admissible.
- GPT-5.5 leads POST localization, while size-normalized dependency features gain over run size on SWE-Gym POST detection.
- Three open problems remain: dropped grounding, online stale-state detection, and the tau-bench domain gap.
- Gold-derived boards remain mechanism diagnostics, and some label coverage and admissibility decisions limit evidential scope.
7 CONCLUSION
CatchBench provides a shared, state-specific arena for agent auditing while separating evidential results from mechanism diagnostics and reporting unresolved comparisons.
- Seven task contracts span PRE, LIVE, and POST, with four evidential boards and three Gold-derived mechanism diagnostics.
- The file-level Gold substrate was rejected for artifact leakage, while tau-bench stale-state data lack determined corpus adequacy.
- 71 of 118 registered contrasts remain unresolved and are published as such rather than forced into a ranking.
ARTIFACT AVAILABILITY
The artifact includes CatchBench code, committed PRE records, cached LLM-judge predictions, and diagnostic scripts, while raw traces and task descriptions remain upstream.
- The released artifact includes code, committed PRE derived records, cached LLM-judge predictions, and diagnostic scripts.
- Raw Who&When, SWE-Gym, and tau-bench traces are not redistributed.
- Raw PRE task and role descriptions remain available from their cited upstream sources.
REPRODUCIBILITY STATEMENT
CatchBench makes its scored board reproducible without model-service access by reading committed prediction caches and checking pinned trace-corpus commits.
- The scored board reads committed prediction caches and makes no model-service calls.A third party can reproduce every number without an API key.
- All three trace corpora are pinned by commit, with the pins checked before scoring.
- Appendix G documents pinned inputs, additional requirements for regenerating cached predictions, and continuous-integration checks.
ETHICS AND DATA STATEMENT
CatchBench documents data provenance, redaction, benchmark boundaries, and comparison procedures while retaining auditable references and limiting claims about privacy and neighboring systems.
- Data handling: All 1187 released PRE rows retain source and label-process information while excluding task or role specification prose.The records preserve auditability without redistributing contact-bearing descriptions.
- Data handling: The PII scan found zero occurrences and zero distinct values across seven checked identifier patterns, but does not guarantee that no indirect identifier survives.
- Data handling: The corpus retains source repositories, commits, paths, capability names, labels, and scanner features so readers can check upstream material.Researchers are asked to respect source licences and avoid contacting, profiling, or ranking template authors.
- Benchmark scope: CatchBench-Gold plants known faults in real traces but assigns them mechanism-diagnostic rather than evidential status when substrates fail admissibility tests.
- Benchmark scope: CatchBench differs from neighboring benchmarks by holding data and scored tasks fixed while comparing methods across PRE, LIVE, and POST states.
- Evaluation boundaries: The comparison-family table is generated from prespecified statistical-test definitions, with family membership based on the claim rather than observed p-values.
B COMPARISON FAMILIES AND ORDERING TESTS
CatchBench declares comparison families and correction procedures in advance, reports unresolved contrasts explicitly, and emphasizes domain- and protocol-specific results rather than a single ranking.
- Statistical tests: Holm-adjusted tests cover 16 prespecified families, with exact or paired procedures selected for binary outcomes, ROC-AUC, and Gold attribution.
- Ordering policy: 47 of 118 declared contrasts separate after correction, while 71 remain unresolved.Unresolved means failure to reject, not evidence of equivalence.
- Ordering policy: Figure 5 shows family sizes split into separates and unresolved verdicts, without plotting or ranking point estimates.
- Protocol contrasts: The all-at-once protocol is the default, while binary search is the only within-model protocol contrast separating after correction and lowers Qwen3-32B from 0.349 to 0.127.
- LIVE results: At the first SWE-Gym prefix, auditable versus size separates by +0.101 ROC-AUC, whereas later auditable-versus-ECOD contrasts remain unresolved.
- LIVE results: Tau-bench auditable-versus-ECOD contrasts separate at all four prefixes, with differences from +0.064 to +0.107.
- LIVE results: SWE-Gym reaches early-warning ROC-AUC 0.742 for auditable features versus 0.629 for size at 25%, while tau-bench remains below 0.70.
C.3 ADDITIONAL POST LOCALIZATION RESULTS
Post-localization results favor full-trace LLM judges, while structural methods provide a weaker but useful reference and broader applicability. Across detection and transfer analyses, several apparent gains remain unresolved or narrow under paired tests, seed controls, or source qualification.
- Post localization: Eight LLM judges remain statistically unresolved from 0.333 to 0.452 Top-1 across 126 runs.The panel spans 0.127 to 0.452, resolving only at its extremes.
- Structural reference: 0.211 Top-1 and 0.614 Top-3 are achieved by exec-rank (sup.), versus 0.159 and 0.516 for the position prior.The Top-3 gap separates, while the Top-1 gap does not at this corpus size.
- Failure detection: On SWE-Gym, auditable (size+deps) reaches 0.804 ROC-AUC against 0.663 for size (flat), the strongest corrected detection effect reported.The corresponding tau-bench contrast is 0.665 against 0.619 and remains unresolved.
- Transfer diagnostics: 0.652 ± 0.169 within-size ROC-AUC is obtained by GAAN on 262 matched positive-negative comparisons, but this is not a beyond-size board advantage.The interval across initializations is [0.571, 0.734], and the evaluated population changes after matching node-count strata.
- Online detection: At realized false-positive rates of 6.1% and 11%, the causal span z-score catches only 6% and 11% of stale reads online.Across five injection seeds, its span z-score remains 0.054 ± 0.012 and 0.124 ± 0.012.
- Post localization: GPT-5.5 scores 0.452 Top-1 against 0.211 for the highest structural method on Who&When post localization.Eight judges form an unresolved 0.333–0.452 band across 126 runs.
D.1 PRE SCANNER AND SOURCE DIAGNOSTICS
PRE scanner results expose how label processes and corpus construction constrain interpretation. CatchBench therefore reports source-specific diagnostics, including a positional shortcut that achieves perfect F1 on one source.
- The permissions rule matches task verbs to grant levels, while functionality compares non-generic task and capability subjects.
- The held-out judge reaches 0.990 and 0.972 on roster-relabeled and injected synthetic labels but only 0.362 to 0.744 on judge-labeled sources.
- 0.990 against 0.750 on injecagent and 0.972 against 0.763 on synthetic are reported as construction effects rather than reasoning results.
- 1.000 F1 on the roster-relabeled injecagent source came from flagging the capability tail by position alone.The rule made 510 true positives, with no false positives or false negatives, because the first declared capability was always the minimum.
- SWE-Gym is heavily balanced toward resolved rows: 188 resolved and 188 unresolved runs are selected from a source with an 8.1% pass rate.The resulting board is denser in successes than the source population and should not be read as a sample of it.
- Both Gold injection substrates break a file-level predecessor invariant, allowing a construction-only baseline to rank every injected target first while flagging no clean controls.This failure of artifact control means the injected boards serve as mechanism diagnostics rather than evidential benchmark boards.
F.1 GOLD BOARD MECHANISM DIAGNOSTICS
Gold boards diagnose fault mechanisms but do not carry evidential benchmark status because their substrates admit construction artifacts. After eligibility matching, stale-state remains localizable while dropped grounding does not separate from the floor.
- Gold boards rank run steps against injected targets, but their three boards are reported as mechanism diagnostics rather than artifact-controlled evidence.
- 0.653 ± 0.028 Top-1 across five seeds shows stale-state localization above its 0.028 random floor.Within the eligibility-matched pool, max-span reaches 0.795 ± 0.020 against a 0.350 floor.
- 0.005 ± 0.000 for span detectors and 0.029 ± 0.008 for degree leave dropped-grounding localization near its 0.036 floor.After matching, position reaches 0.321, but it does not separate from the corresponding floor.
- Eligibility matching controls target selection but not degree: has-dep scores 0.350 against a 0.350 floor for stale-state and 0.075 against 0.277 for dropped grounding.
- PyGOD’s nominal overall lead remains unresolved across run resampling, illustrating why aggregate Gold rankings hide fault-specific differences.Its initialization-axis lead over deterministic max-span is +0.031 with a 95% interval of [0.012, 0.049].
- Dropped grounding passes current fixed-margin readings but remains inadmissible until all six controls demonstrate power on Top-1 and run-level AUC.
G ARTIFACT AND REPRODUCIBILITY DETAIL
CatchBench’s reproducibility system ties scoring and figures to committed artifacts while exposing what remains rerunnable only with external services or unavailable prose. It also documents a corrected PyGOD harness defect that previously produced degenerate rankings.
- The scored board reads committed prediction caches and calls no model service, while corpus revisions are pinned by commit and checked before scoring.
- Figures and adjacent tables parse the same released board file, and CI checks that committed images still match the board data.
- Third parties can rerun the complete scored board without an API key, but regenerating POST predictions requires a gateway or compatible endpoint.Rebuilding the PRE held-out-judge cache additionally requires prose-retaining PRE data kept outside the repository.
- Tables 2, 4, and 12 are CI-checked from committed records, whereas Table 22 is printed by a script and can drift when the paper changes.
- Commit pins are not archival guarantees, and the exact auditable revision and resolved dependency environment remain unlocked.
- 182 of 188 Gold runs previously received constant PyGOD scores because batched loaders caused only the first batch-sized nodes to be scored.Joining chunks into a plain Data object reduced the degeneracy to 0 of 188, and corrected numbers are reported throughout.