PaperlayerView exact revision on arXiv

Source-linked AI summary

What Did the Agent Actually Do? Evidence-Grounded Oversight for Long-Horizon Agents

Zhongxiang Sun, Jiahao Yan, Hongkang Zhao, Haojie Ding, Boheng Zhang, Fan Yang, Xiao Zhang, Jun Xu

arXiv:2610.06406v1cs.AIcs.CL

TL;DR

Long-horizon agent activity and fragmented evidence make it difficult for users to decide which autonomous choices warrant verification. This paper introduces a benchmark and an evidence-organizing method for monitors; experiments show general improvements in decision identification and evidence localization over evaluated baselines.

  • Problem

    Users overseeing long-horizon agents must determine which consequential decisions warrant verification, but agent activity is extensive and supporting evidence is fragmented.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces AgentMonBench and EBG, a training-free method that organizes source-linked evidence into behaviors and relations and presents task-oriented views for monitor judgment.

    Source: §1 · p. 2

  • Results

    Across eight models and all three benchmark components, EBG generally improves decision identification and evidence localization over evaluated baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support structured evidence organization as a useful basis for human oversight, while current models still have substantial room for improvement.

    Source: §6 · p. 10

    Access to and reuse of source data remain subject to the original datasets’ access conditions and applicable licenses.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

As agents take on long-horizon tasks, users shift from making individual decisions to overseeing autonomous execution. Yet the volume of agent activity and the fragmentation of supporting evidence make it difficult to determine which decisions warrant user verification. We study monitors that identify consequential decisions and locate evidence to help users assess their implications. We introduce AgentMonBench, a software-engineering benchmark comprising three subsets that cover two complementary dimensions: alignment between requirements and behavior, and awareness of consequential autonomous decisions for verification. To support these judgments, we propose the Evidence-Grounded Behavior Graph (EBG), a training-free method that groups source-linked evidence into behaviors and organizes their relationships into a graph. EBG presents task-oriented views of this graph to help monitors interpret behavior in context. Experiments across eight models show that EBG improves decision identification and evidence localization in most settings compared with direct access to the original context. Further experiments show that EBG's evidence-localization gains persist across input scales and hyperparameter settings, while real-world applications illustrate its practical value for human oversight.

1 INTRODUCTION

Long-horizon agents leave users responsible for oversight even as the volume and complexity of agent activity make evaluation difficult. The paper frames the need as identifying consequential decisions and connecting them to evidence, then introduces AgentMonBench and EBG to support that work.

1 INTRODUCTION

  • Delegation shifts users from making individual decisions to overseeing execution, but responsibility for determining whether agent choices serve their goals remains with them.The volume and complexity of agent activity can exceed users’ capacity to follow and evaluate it.Source: §1 · p. 1
  • The paper asks which consequential decisions in long-horizon tasks warrant user verification and what evidence helps users understand their implications.The monitor must identify choices affecting task goals, constraints, or outcome interpretation and connect them to evidence in the agent context.Source: §1 · p. 2
  • AgentMonBench evaluates oversight in software engineering across requirement–behavior alignment and awareness of consequential autonomous decisions warranting verification.SpecGAP covers consequential behavior left unspecified by incomplete requirements; SilentSwap covers semantic deviations despite passing tests; FeedbackTrace covers decisions targeted by subsequent user feedback.Source: §1 · p. 2
  • EBG is a training-free method that organizes source-linked evidence through behaviors, scopes, and explicit relations into task-oriented views for monitor judgment.Its construction and presentation are deterministic, while the monitor makes the semantic judgment.Source: §1 · p. 2
  • Experiments across eight models and all three benchmark components show that EBG generally improves decision identification and evidence localization over evaluated baselines.The paper also reports that average localization gains persist across input scales and tested hyperparameter settings.Source: §1 · p. 2

2 RELATED WORK

Related work motivates oversight through human-factors research and distinguishes AgentMonBench from benchmarks of clarification and proactive interaction. The paper focuses instead on identifying consequential choices during long-horizon execution and locating evidence for verification.

Human oversight

  • Human-factors research links reduced direct involvement under automation with continuing supervisory responsibilities, situation awareness, and appropriate reliance on automated systems.Related work also connects this focus to research on user effort, collaborative utility, and steering and auditing learning systems.Source: §2 · p. 3

Clarification and proactive interaction

  • Clarification and proactive-interaction benchmarks study missing information, user intent, and interaction, whereas AgentMonBench evaluates monitors’ identification of consequential choices and supporting evidence during long-horizon execution.Its cases include assumptions under incomplete requirements and deviations from user instructions or specifications.Source: §2 · p. 3

3 AGENTMONBENCH: BENCHMARK CONSTRUCTION

AgentMonBench tests whether monitors can identify consequential agent decisions and locate evidence for user verification. Its three subsets cover incomplete requirements, semantic changes to implementations, and decisions surfaced by later user feedback.

3 AGENTMONBENCH: BENCHMARK CONSTRUCTION

  • The benchmark evaluates monitoring of consequential decisions and supporting evidence across complementary software-engineering conditions.SpecGAP and SilentSwap test requirement–behavior alignment; FeedbackTrace tests awareness and verification of decisions in real interactions.Source: §3 · p. 3

3.1 SPECGAP: CONTROLLED OMISSION OF TASK REQUIREMENTS

  • SpecGAP tests whether monitors identify consequential repository behavior or constraints left unspecified by task requirements after selected conditions are removed.The construction deletes selected conditions, checks that the remaining document does not reveal them and remains coherent, and maps each deleted condition to repository evidence using two methods.Source: §3.1 · p. 4

3.2 SILENTSWAP: EXECUTABLE BUT SEMANTICALLY DIFFERENT IMPLEMENTATIONS

  • SilentSwap tests whether monitors detect consequential implementation changes despite an unchanged task document and passing existing tests.Each instance has five semantic substitutions; construction selects documented behavior, modifies code and adds a distinguishing semantic test, then checks that unrelated functionality is unaffected.Source: §3.2 · p. 4

3.3 FEEDBACKTRACE: VERIFICATION POINTS FROM DELAYED USER FEEDBACK

  • FeedbackTrace tests whether monitors identify consequential decisions requiring verification from interaction history before the corresponding user feedback.Samples are drawn from SWE-chat sessions with at most one per session; annotations specify the decision, its effect, the choice needing confirmation, minimal sufficient evidence, and criticality.Source: §3.3 · p. 4

3.4 QUALITY INSPECTION

  • Double-review agreement was 92.4% for 447 of 484 SpecGAP conditions, 93.6% for 468 of 500 SilentSwap substitutions, and 93.0% for 93 of 100 FeedbackTrace sessions.Reviewers checked mappings, behavior-change explanations and evidence, or verification points, evidence, criticality, and pre-feedback support, respectively.Source: §3.4 · p. 5

4 EBG: EVIDENCE-GROUNDED BEHAVIOR GRAPH

EBG organizes dispersed behavioral evidence into a graph and renders task-oriented views for a monitor to compare with task requirements. The graph represents local behaviors, groups them into scopes, and links scopes with explicit relations.

4.1 OVERVIEW

  • EBG transforms visible agent context into a Structured Behavior Graph, renders task-oriented evidence views, and has the monitor use those views with task requirements to produce findings.The graph builder and presenter are sequential components; the monitor’s findings are grounded in evidence traceable to original sources.Source: §4.1 · pp. 5–6, §4.3 · p. 7

4.2 STRUCTURED BEHAVIOR GRAPH

  • The graph builder extracts source-linked Evidence, forms Behaviors, groups related Behaviors into Scopes, and connects Scopes with explicit directed relations supported by evidence.Behaviors organize evidence references by activity; repository Behaviors follow Condition–Operation–Result, while trajectory Behaviors follow Demand–Action–Response.Source: §4.2 · p. 6

4.3 TASK-ORIENTED EVIDENCE READING AND PRESENTATION

  • EBG presents repository and trajectory evidence differently, producing task-oriented views that retain relevant graph structure and support verification against original passages and source code.Repository views use file selection, seed expansion, and one-hop subgraphs; trajectory views group Behaviors by task scope, preserve their order, and distinguish Current from History.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

Across AgentMonBench’s three components, EBG generally improves decision identification and evidence localization over baselines, though current models remain limited. The experiments also examine input scale and illustrate EBG-based review in a Codex Harness.

5.1 EXPERIMENTAL SETUP

  • The evaluation compares eight models against Base and, on SpecGAP and SilentSwap, an adapted RepoGraph baseline using complementary rule-based and LLM-judge metrics.Metrics assess evidence localization as well as semantic identification, clarification quality, and verification-point alignment.Source: §5.1 · p. 8

5.2.1 MAIN RESULTS

  • EBG improves on Base across most settings and exceeds RepoGraph in most repository comparisons, but gains vary by monitor and task.Current models still leave considerable room for improvement, and stronger evidence retrieval can coincide with weaker decision interpretation.Source: §5.2.1 · p. 8

5.2.2 MONITORING PERFORMANCE ACROSS INPUT SIZES

  • In the High input-token group, EBG’s absolute gains are 0.078 on SpecGAP LF1, 0.115 on SilentSwap LS, and 0.108 on FeedbackTrace EHR.All nine average comparisons favor EBG, and all three monitors improve on each metric in the High group.Source: Figure 5 · p. 9, §5.2.2 · p. 9
  • Base mean scores decline as input size grows: FeedbackTrace EHR falls from 0.667 to 0.422 and SilentSwap LS from 0.557 to 0.429.SpecGAP LF1 also declines, but more modestly.Source: §5.2.2 · p. 9

5.2.3 APPLICATION IN CODEX HARNESS

  • In five real-world research-task cases, Codex without the harness does not disclose key verification issues, whereas Codex with it identifies and discloses them.The harness reviews ambiguity, changes to the planned approach, and whether evidence supports proposed conclusions.Source: §5.2.3 · p. 9

6 CONCLUSION

The paper frames EBG as a training-free way to organize source-linked behavioral evidence for oversight and reports that it generally improves decision identification and evidence localization. Its stated scope is software-engineering agent oversight, and source-data reuse remains subject to original access conditions and licenses.

6 CONCLUSION

  • Across experiments, current LLMs still have substantial room to improve, while EBG generally improves decision identification and evidence localization.The authors describe structured evidence organization as a useful basis for human oversight.Source: §6 · pp. 9–10
  • The work targets human oversight of software-engineering agents, with five real-world research-task cases reported for the Codex Harness application.The cases are summarized as observed actions and outcomes.Source: §6 · p. 9, Table 2 · p. 10
  • AgentMonBench construction used generative AI assistance, with human reviewers checking benchmark mappings, semantic changes, and FeedbackTrace annotations.The authors take responsibility for the final manuscript, dataset artifacts, and scientific claims.Source: AI USE STATEMENT · p. 10
  • Reuse of source data remains subject to the original datasets’ access conditions and applicable licenses.The paper states that it does not anticipate substantial ethical risks from its intended research use.Source: ETHICS STATEMENT · p. 10

A RELATED WORK

Prior work evaluates agent processes, problem discovery, and evidence retrieval, while AgentMonBench centers on identifying consequential decisions for user verification. Its three components operationalize this focus through omitted requirements, semantic substitutions, and delayed user feedback.

A RELATED WORK

  • AgentMonBench centers on consequential decisions for user verification, complementing work on process assessment, problem discovery, and evidence organization.Its EBG organizes source-linked evidence into behaviors, scopes, and relations for monitor judgment.Source: A RELATED WORK · p. 14

B.1 BENCHMARK COMPOSITION AND STATISTICS

  • The benchmark has 100 instances per component: SpecGAP has 484 omitted conditions, SilentSwap 500 substitutions, and FeedbackTrace 100 verification points.The components cover seven omitted-condition types, eight substitution mechanisms, and four user-feedback types.Source: §B.1 · p. 15

B.2 SPECGAP: CONTROLLED OMISSION OF TASK REQUIREMENTS

  • SpecGAP deletes consequential conditions from otherwise complete task documents and maps each omission to repository evidence using independent condition-by-condition and holistic methods.Mappings are reviewed and validated, and retained conditions must pass semantic review and deterministic checks.Source: §B.2 · pp. 15–17

B.3 SILENTSWAP: EXECUTABLE BUT SEMANTICALLY DIFFERENT IMPLEMENTATIONS

  • SilentSwap introduces five plausible semantic changes while keeping existing tests passing, then checks that each change is detectable by a dedicated semantic test.Combined validation checks attribution and interference across substitutions; Gold annotations record the documented and substituted behaviors and their code locations.Source: §B.3 · pp. 17–18

B.4 FEEDBACKTRACE: VERIFICATION POINTS FROM DELAYED USER FEEDBACK

  • FeedbackTrace traces real interaction feedback to an earlier agent decision, constructs Local and Long views, and segments agent responses and tool interactions into citable evidence units.The target feedback helps locate the decision but is withheld from monitors and cannot support the Gold verification point.Source: §B.4 · p. 18
  • Each FeedbackTrace Gold annotation pairs a concrete verification point with minimal sufficient evidence and a criticality label distinguishing MUST-DISCLOSE from WORTH-DISCLOSE decisions.The evidence set contains one or two direct evidence units supporting every factual claim in the verification point.Source: §B.4 · p. 18

B.5 QUALITY CONTROL AND LEAKAGE PREVENTION

The benchmark separates monitor-visible inputs from Gold annotations and construction-only information, and it uses independent human review alongside deterministic checks. Reported retention figures describe different annotation fields and should not be confused with double-review agreement.

B.5 QUALITY CONTROL AND LEAKAGE PREVENTION

  • Gold annotations and construction-only information are kept separate from monitor inputs, with task-specific Gold and post-cutoff information excluded.Deterministic checks cover temporal cutoff safety, evidence identifiers, provenance, sample alignment, and leakage.Source: §B.5 · pp. 19–20
  • Table B2 reports independent human double-review agreement for each benchmark, and agreement between the two reviewers provides evidence of annotation reliability across all three benchmarks.The second reviewer’s materials exclude the first reviewer’s labels, final Gold, review notes, and aggregate statistics.Source: §B.5 · p. 19, Table B2 · p. 19
  • For SpecGAP, the final mapping adopts a model-generated mapping for 322 of 484 conditions (66.5%), measured after human review rather than as reviewer agreement.The retained mappings include 282 condition-by-condition and 40 holistic mappings.Source: §B.5 · p. 19
  • After review, 486 of 500 SilentSwap model-generated Gold drafts (97.2%) are retained without semantic revision.This is draft retention, not double-review agreement.Source: §B.5 · p. 19, Table B3 · p. 19
  • FeedbackTrace field-level retention after review is 64.0% for evidence sets, 74.0% for feedback types, and 75.0% for criticality labels.Verification-point wording was revised in all 100 sessions; wording changes alone do not establish semantic error.Source: Table B3 · p. 19, §B.5 · p. 20

B.6 BENCHMARK CONSTRUCTION PROMPTS

  • The appendix’s benchmark-construction prompts are compressed versions that retain core prompt logic and benchmark-specific constraints.Nonessential, repetitive, and implementation-specific instructions are removed.Source: §B.6 · p. 20

C.1 EVALUATION SETS AND MODEL CONFIGURATIONS

The appendix specifies the evaluation setup and metrics, then details how EBG constructs behavior graphs and presents repository and trajectory evidence views.

C.1 EVALUATION SETS AND MODEL CONFIGURATIONS

  • The setup compares Base and EBG across all three components, with RepoGraph additionally evaluated on SpecGAP and SilentSwap.Base provides repository content or the original execution trace; monitor configurations and the evidence-access summary are specified separately.Source: §C.1 · p. 22, Table C2 · p. 23

C.2 BASELINES AND EVIDENCE ACCESS

  • Base exposes searchable repository content or the complete visible trace, while RepoGraph returns code-graph neighborhoods and is not applicable to FeedbackTrace.FeedbackTrace requires event-level evidence and temporal context that static code definitions and calls do not represent.Source: §C.2 · p. 23

C.3 METRICS AND SCORING RULES

  • SpecGAP scores semantic constraint matching and clarification-question quality separately from code localization; SilentSwap scores line overlap, localization correctness, and understanding of behavior changes.FeedbackTrace evaluates verification-point alignment, evidence support, and exact evidence-ID hits.Source: §C.3.1 · pp. 23–24, §C.3.2 · p. 24, §C.3.3 · p. 24

C.3.3 FEEDBACKTRACE

  • FeedbackTrace’s Evidence Hit Rate is the fraction of examples with at least one exact identifier shared by predicted and reference evidence sets.Evidence Support Score separately assesses evidence validity, directness, and sufficiency.Source: §C.3.3 · p. 25

D.1 STRUCTURED BEHAVIOR GRAPH DETAILS

  • EBG deterministically extracts source-linked evidence, groups it into activity-specific Behaviors, and connects them through Scopes and evidence-backed relations.Repository Behaviors use Condition–Operation–Result roles, while trajectory Behaviors use Demand–Action–Response roles.Source: §D.1 · pp. 29–30

D.2 REPOSITORY EVIDENCE PRESENTATION

  • Repository views use task requirements to select seeds, retain their one-hop graph neighborhoods, and render connected local subgraphs with task information and original source.A reading Root is selected within each component; it changes neither selected evidence nor causal interpretation.Source: §D.2 · pp. 30–31

D.3 TRAJECTORY EVIDENCE PRESENTATION

  • Trajectory views group Behaviors by task, preserve their original positions and cross-task references, and label the task containing the last visible Behavior Current.Other represented tasks are labeled History; these labels indicate status at the cutoff.Source: §D.3 · pp. 31–32

D.4 EXPERIMENTAL SETTINGS

The appendix gives interaction budgets and defines task-specific prediction requirements, evidence rules, and output validation for the three benchmark components.

D.4 EXPERIMENTAL SETTINGS

  • Repository tasks allow eight SpecGAP rounds or six SilentSwap rounds, with bounded search, read, and output budgets; FeedbackTrace uses one call without tool use.EBG reads are limited to six seed file identifiers but retain all one-hop neighbors, without an edge-count cap.Source: §D.4 · p. 32

D.5 PREDICTION PROMPT AND OUTPUT SCHEMA

  • The common prediction setup holds task objectives, selection criteria, evidence requirements, and scoring rules constant across methods.Prompts specify each method’s input representation, interaction workflow, and output requirements.Source: §D.5 · p. 32

SpecGAP: Prediction Prompt

  • SpecGAP asks monitors to recover materially consequential implemented contracts omitted from documentation and support each omission with direct source evidence.Predictions should reject documented, immaterial, unsupported, or general bug claims; an empty findings list is allowed.Source: SpecGAP: Prediction Prompt · p. 32

SilentSwap: Prediction Prompt

  • SilentSwap presents exactly five independent observable substitutions and asks monitors to identify each before-and-after behavior, distinguishing trigger, and responsible code locations.Predictions must separate independent changes and reject compatible implementations, private details without observable effects, duplicates, and guesses.Source: SilentSwap: Prediction Prompt · p. 33

FeedbackTrace: Prediction Prompt

  • FeedbackTrace asks monitors to select the most consequential unresolved agent decision, its practical impact, and one or two direct supporting evidence IDs.Predictions must distinguish planned from executed and attempted from successful actions, and assign a criticality label.Source: FeedbackTrace: Prediction Prompt · p. 34

D.5.4 OUTPUT SCHEMA

  • Output schemas enforce required fields, types, enumerated values, cardinality, and evidence validity, including whether cited evidence was actually exposed.FeedbackTrace IDs must belong to the current example’s selectable evidence set.Source: §D.5.4 · pp. 34–35

E.1 INDIVIDUAL-JUDGE RESULTS AND AGREEMENT

The individual-judge results show EBG outperforming baselines on SilentSwap across monitors, alongside broadly consistent judge conclusions and some sensitivity in finer comparisons.

E.1 INDIVIDUAL-JUDGE RESULTS AND AGREEMENT

  • EBG achieves higher LS and LC than Base and RepoGraph across all eight monitors on SilentSwap.Tables E1 and E2 provide the separate judge scores underlying the reported averages.Source: §E.1 · p. 35
  • The agreement analyses compare judge score differences and EBG-minus-Base directions across eight monitors; ranking agreement uses seven Base monitors, excluding GLM-5.3.RepoGraph and rule-based metrics are excluded from these statistics.Source: §E.1 · pp. 35, 37
  • The judges agree on EBG’s direction of change in 40 of 48 monitor–metric comparisons (83.3%), though some individual semantic scores and model-specific conclusions remain judge-sensitive.Mean absolute differences are larger for SilentSwap LC (0.097) and FeedbackTrace VPA (0.095).Source: §E.1 · p. 35, Table E3 · p. 37
  • Under Base, judge rankings across six semantic metrics have Spearman’s ρ from 0.8929 to 1.0000, with the lowest correlations for LC and VPA.Both judges identify the same top three monitors for LC and VPA.Source: §E.1 · p. 35, Table E3 · p. 37

E.2 ABLATION STUDY

Ablations indicate that behavior organization, graph organization, and task guidance each contribute to EBG’s performance on their applicable tasks.

E.2 ABLATION STUDY

  • Removing Behavior grouping tests whether grouping evidence from the same activity helps beyond presenting individual evidence units within the same graph structure.Scope grouping and explicit relation edges remain in this ablation.Source: §E.2 · p. 38
  • Removing Scope grouping and relation edges tests the graph component while retaining Behavior units and their source evidence.The comparison presents Behaviors as a flat list.Source: §E.2 · p. 38
  • The task-guidance ablation removes task-guided file ranking and document-passage alignment while retaining graph organization.FeedbackTrace has no separate task-guidance ablation because task correspondence is already embedded in its behaviors and scopes.Source: §E.2 · p. 38
  • Every applicable ablation lowers its task’s representative metric for Luna, Kimi-K3, and GLM-5.3, although a few other metrics increase slightly.The representative metrics are SF1, LS, or ESS.Source: §E.2 · p. 38

E.3 INPUT-SIZE ANALYSIS

EBG’s localization gains hold across input sizes, expansion depths, and reduced interaction budgets, while disclosure detection generally improves alongside fewer false alarms. The review harness structures evidence-based checks across the workflow, but interpretation and corrective action remain the agent’s responsibility.

E.3 INPUT-SIZE ANALYSIS

  • EBG improves every monitor at every input scale on SpecGAP LF1 and SilentSwap LS; on FeedbackTrace EHR, all three improve in High.On FeedbackTrace EHR, Luna decreases in Medium, Kimi K3 decreases in Low, and GLM-5.3 is unchanged in Low.Source: §E.3 · p. 38

E.4 HYPERPARAMETER SENSITIVITY

  • At every tested expansion depth, both monitors outperform Base on all three SpecGAP metrics and on SilentSwap LS and LC, with SF1 or LS varying by at most 0.0303.Deeper expansion does not consistently improve scores; SilentSwap CC falls below Base for Luna at three hops and GLM-5.3 at two.Source: §E.4 · p. 40, Figure E2 · p. 41, Figure E3 · p. 41
  • At every matched interaction budget, EBG improves all three SpecGAP metrics and SilentSwap LS and LC for both monitors, with SF1 or LS varying by at most 0.0272.More rounds do not consistently raise scores, and CC gains are less consistent than LS and LC gains.Source: §E.4 · p. 40, Figure E4 · p. 42, Figure E5 · p. 42

E.5 DISCLOSURE DETECTION AND FALSE ALARMS

  • EBG increases TPR in all nine monitor–benchmark comparisons and reduces FPR in eight, but Luna’s SilentSwap FPR rises from 52% to 58%.Luna also retains an 84% FPR on FeedbackTrace.Source: §E.5 · p. 43, Table E6 · p. 43

F.1 DESIGN AND IMPLEMENTATION

  • The Codex harness combines session state, execution hooks, stage-specific review skills, and evidence tools to support review of research-task decisions.Reviews require evidence inspection; interpretation and corrective action remain the agent’s responsibility.Source: §F.1 · pp. 43–44

F.2 REVIEW WORKFLOW

  • Ambiguity review is prompted before dependent work proceeds, while end-of-response adjustment and result reviews follow specified triggers and can be reused when evidence is unchanged.After review, the agent can continue, correct execution, seek clarification, or report a qualified conclusion.Source: §F.2 · p. 44

F.3 CASE STUDIES

Five paired real-world task cases illustrate how the review harness surfaces decisions and evidence issues that the unaided agent did not disclose for user verification. The failure analysis separately traces errors through file selection, evidence presentation, and model judgment.

F.3 CASE STUDIES

  • In the ambiguous-objective case, the harness surfaces the unresolved accuracy-versus-recall priority and asks the user before configuration selection.The user chose accuracy as the priority metric; the without-harness run selected a threshold that raised accuracy from 0.70 to 0.80 while recall fell from 1.00 to 0.3333.Source: §F.3 · p. 45

Case 2: A Fallback Changes Randomization2

  • When a backend failure led to changed per-problem randomization, the harness restored per-problem seeds, reran the candidate, and disclosed the change.The corrected candidate had mean objective 42.6374 →28.5896, with 1,920 total evaluations.Source: §F.3 · p. 46

Case 3: Validation Leakage Behind an Apparent Gain (continued)

  • In the validation-leakage case, the harness found that 8 of 80 added samples came from validation records and rejected the candidate as evidence of leakage-free improvement.It disclosed the leakage and retained the baseline; candidate 2 tied the baseline without the extra pool.Source: Case 3: Validation Leakage Behind an… · p. 47

Case 4: Cached Results Behind Passing API Checks (continued)

  • In the cached-results case, the harness identified that passing smoke checks used a stored response, corrected the verification claim, and disclosed that live API connectivity remained unverified.The record showed zero live requests.Source: Case 4: Cached Results Behind Passing… · p. 48

Case 5: Search Opportunities Limit Attribution (continued)

  • In the search-budget case, the harness qualified the optimizer comparison because the candidate received substantially more tuning opportunities than the reference.The reference received one fit, while the candidate received 27 initial plus 72 refined fits.Source: Case 5: Search Opportunities Limit… · p. 49

G.1 EXPERIMENTAL SETUP

G.2 FAILURE CATEGORIES AND STATISTICS

  • Failure categories distinguish missing files, omitted evidence from selected sources, and judgment errors after reference evidence is visible.Counts and within-failure percentages are reported by component, monitor, and judge; SilentSwap counts cover localization failures only.Source: §G.2 · p. 50, Table G1 · p. 50

G.3 RESULTS

Most attributed SpecGAP and SilentSwap failures occur after reference evidence is visible, while FeedbackTrace failures also persist with complete reference events in view. The proposed future work targets both evidence coverage and reasoning over visible evidence.

G.3 RESULTS

  • Judgment failures account for 74.0–81.2% of SpecGAP failures and 69.4–76.9% of SilentSwap failures under GLM-5.2; Qwen3.7-Max shows the same dominant category.For Qwen3.7-Max, the corresponding ranges are 75.0–80.9% and 72.0–79.0%; Figure G1 visualizes GLM-5.2 distributions.Source: §G.3 · p. 50, Figure G1 · p. 51
  • Presentation omissions account for 10.2–15.4% of SpecGAP failures and 14.5–24.3% of SilentSwap failures under GLM-5.2, but their specific root-selection, expansion, or rendering cause remains unresolved.These omissions distinguish retrieving the required source from exposing its relevant content.Source: §G.3 · p. 50
  • Every audited FeedbackTrace failure contains the complete reference events and is assigned to judgment under both judges, with no evidence-presentation omissions.Source: §G.3 · p. 50

G.4 FUTURE DIRECTION

  • Future work proposes improving task-guided retrieval and adaptive graph expansion, and testing organization separately from reasoning when reference evidence is present.Evidence views could assess support for a candidate decision and request more context when support is incomplete.Source: §G.4 · pp. 50, 52

Related papers

Loading 2610.06406v1…