Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair

• Silent failures pass available syntactic and functional checks despite retaining or introducing a vulnerability, so a confirmed failure alone does not locate its origin.Different origins may call for different remedies, from prompt refinement to an additional verification checkpoint.Source: §1 · p. 1
• SAGE localized an origin in 93 of 95 confirmed silent failures; most involved an unaddressed security requirement or an inadequate defence choice.The cases came from 3,684 repair traces produced by six agent frameworks and six base models on SecurityEval and CVEfixes.Source: §1 · p. 2
• When the agent introduced vulnerable code, the origin preceded the introducing write in 14 cases; repeated scoring and an independent judge model reproduced the same origin type in most cases.For inherited vulnerable code left untouched, no introducing write exists; the trace may instead show an initial plan omitting the relevant security requirement.Source: §1 · pp. 1, 3
• Repeated scoring and an independent judge reproduced origin type more consistently than exact turn, with exact-turn agreement lowest for traces retaining only the final file.Agreement with human annotators on code provenance and the introducing write was moderate to substantial (κ between 0.58 and 0.74) and varied across task subsets.Source: §1 · p. 3
• SAGE combines quotation-anchored assessment of security reasoning with reconstructed artefact history to identify the earliest trace-attributable divergence from security intent.Its evaluation uses repeated scoring, cross-judge comparison, task-disjoint subsets, and human annotation.Source: §1 · pp. 2–3

• Security studies and benchmarks motivate evaluation beyond final test outcomes, but do not explain where security failure first becomes attributable in an agentic repair trajectory [30] [32].Test-passing patches may still be incorrect [41].Source: §2 · p. 3
• Prior studies of silent failures classify cases by category, agent role, and workflow stage, but do not provide systematic turn-level origin localization [6].Silent failures can remain hidden behind patches that pass available syntactic and functional checks.Source: §2 · p. 3
• Existing attribution methods largely target observable task failures, labelled steps, injected faults, repeated trajectories, or anomalous steps [10] [46] [48].Their evidence conditions do not link a step to a confirmed security deficiency in a functionally passing patch.Source: §2 · pp. 3–4
• Because LLM-judge agreement varies and verdicts can shift with prompt formulation or answer position, SAGE does not treat one judge's assessment as ground truth.Its checks include prefix-only, quotation-anchored scoring and multiple reliability assessments.Source: §2 · p. 4
• SAGE localizes a confirmed failure in a single functionally passing trace by combining the failure anchor, prefix-only reasoning assessment, and reconstructed artefact history.It separates origin from code provenance and the introducing write, and evaluates reliability through repeated scoring, cross-judge comparison, task-disjoint resampling, and human annotation.Source: §2 · p. 4

• Phase I constructs the corpus of confirmed silent failures.Source: §3.1 · p. 4
• Phase II applies SAGE to localize the silent failures' origins.Source: §3.1 · p. 4
• Phase III assesses localization consistency and agreement of artefact-derived outputs with human annotation.Source: §3.1 · p. 4
• Table 1 presents the selected agent frameworks.Source: Table 1 · p. 5

3.2.1 Task Sampling.

• 150 tasks were sampled from 1,236 eligible SecurityEval and CVEfixes tasks, then run across 36 framework–model configurations.The sample comprised 75 tasks per dataset, stratified by CWE family.Source: §3.2.1 · p. 5, §3.2.2 · p. 5

3.2.3 Experimental Setup.

• 3,684 of 5,400 runs produced valid traces for verification, and 331 were flagged as silent-failure candidates.Runs without valid modifications, including crashes, timeouts, and incomplete generations, were excluded.Source: §3.2.4 · p. 5

3.2.4 Execution and Verification.

• Screening checked syntax, functionality, static security findings, and exploitability-oriented patterns; only manually confirmed candidates entered the corpus.SecurityEval had no test suites, so its functional checks covered imports and preservation of input-file functions and classes.Source: §3.2.4 · pp. 5–6

3.2.5 Manual Confirmation.

• Manual review retained true-positive and partial cases, excluded false positives, and yielded 95 confirmed cases from 19 tasks.Two annotators agreed on 54 of 60 candidates (90.00%; κ=0.74).Source: §3.2.5 · p. 6

3.2.6 Trace Normalization.

• Framework-specific records were normalized into canonical turns that combine an agent message with associated tool calls, results, and artefact changes.A turn may also include handoffs, planning artefacts, review comments, intermediate code, and auxiliary artefacts.Source: §3.2.6 · p. 7

3.3 Phase II: SAGE Origin Localization

• SAGE assigns both an introducing-write position from artefact history and an origin turn from artefact history plus per-turn scores.The origin may precede the write, and inherited vulnerabilities need no introducing write.Source: §3.3 · p. 7, §3.3.2 · p. 8

3.3.1 SAGE Turn-Level Scoring.

• The judge scores four dimensions from each trace prefix, requiring verbatim quotations for non-zero scores and withholding later turns and confirmation labels.Applicable planning, implementation, review, and verification turns with no expressed security reasoning receive a zero; administrative events are excluded.Source: §3.3.1 · pp. 7–8

3.3.2 SAGE Origin Localization.

• Code provenance distinguishes agent-introduced deficiencies, which appear in later snapshots, from inherited deficiencies already present in the input file.An introduced deficiency includes a new weakness, insufficient mitigation, or removal or weakening of an existing defence.Source: §3.3.2 · p. 8
• The origin rule selects the earliest linked turn with an inadequate defence, an unaddressed security requirement, or—if neither applies—a defect in the code change.The defect-in-change candidate applies only to agent-introduced vulnerabilities when threat recognition or defensive intent is expressed at the write.Source: §3.3.2 · p. 8
• An outcome is undetermined when no candidate turn links to the confirmed deficiency or required scores are unavailable; automated checks verify provenance and write-history alignment.Fragment matching locates code, while manual confirmation establishes the vulnerability.Source: §3.3.2 · p. 9

3.4 Phase III: SAGE Evaluation

• Repeated scoring and cross-judge comparison assess judge-dependent outputs, while human annotation and task-disjoint resampling assess artefact-derived outputs and sensitivity to task composition.The evaluation uses repeated runs, an independent second judge, and task-level resampling analyses.Source: §3.4 · p. 9, §3.4.1 · pp. 9–10

3.4.1 Reliability and Robustness.

• A second annotator labels a stratified subset of 60 cases without seeing SAGE scores or localization outputs; human comparisons cover provenance and introducing-write labels.The first annotator labels all 95 cases and may be familiar with the cases and rule design.Source: §3.4.2 · p. 10

3.4.3 Localization Coverage and Boundary Conditions.

• Coverage analyses record trace evidence, turn and write counts, reconstruction conditions, localization outcomes, provenance, and supporting evidence.Boundary conditions are examined across failure categories, frameworks, base models, reconstruction conditions, and artefact availability.Source: §3.4.3 · p. 10
• Reliability is assessed through joint localization coverage and agreement on origin type and exact or within-one-turn localization; human comparisons address provenance and write labels.Krippendorff’s α assesses origin type, turn, and ordinal dimension-score agreement; human comparisons use percent agreement and Cohen’s κ.Source: §3.4.4 · p. 10

4.1 Trace Evidence and Localized Origins (RQ1)

• 93 of 95 confirmed cases received an origin assignment, while two remained undetermined.SAGE assigned origins in 97.89% of cases.Source: §4.1 · p. 10, Table 5 · p. 11
• 51 origins were unaddressed requirements, 37 inadequate defences, and five code-change defects; most unaddressed-requirement and inadequate-defence origins involved inherited vulnerabilities.Inherited vulnerabilities accounted for 41/51 unaddressed-requirement origins and 31/37 inadequate-defence origins.Source: §4.1 · p. 11, Figure 4 · p. 11
• 14 of 19 agent-introduced vulnerabilities had an origin before the introducing write; the other five coincided with it in single-message Aider traces.SAGE also assigned origins in 74/76 cases without an identifiable introducing write.Source: §4.1 · pp. 11–12, Figure 4 · p. 11

4.2 Reliability of SAGE Localization (RQ2)

• Across repeated scoring, origin-type agreement was 55/57, versus 50/57 for exact turns; across judges, it was 46/50 versus 39/50.Human agreement was higher for code provenance than introducing-write labels, and varied across task subsets.Source: §4.2 · pp. 13–14, Table 6 · p. 13

4.3 Variation and Boundary Conditions (RQ3)

• Absent-defence cases had higher agreement with human references than insufficient-defence cases, but repeated origin-turn stability showed a different pattern.For absent defences, provenance and introducing-write agreement was 93.75% (60/64); consistently localized insufficient-defence cases retained the same origin turn in 16/16 cases.Source: §4.3 · pp. 14–15, Figure 6 · p. 15
• Agreement varied across agent frameworks and base models, and repeatability could coexist with lower human agreement.GPT-5.5 traces had 62.50% human agreement on both artefact labels while repeated scoring assigned the same turn in 10/10 consistently localized cases.Source: §4.3 · pp. 15–16, Table 7 · p. 15
• Final-only traces had the lowest agreement across all three evaluations, including 25.00% exact-turn agreement across judges, compared with 90.91% for single-write traces.The final-only groups were small; the authors note that missing intermediate artefact states limits evidence for precise origin turns.Source: Figure 6 · p. 15, §4.3 · p. 16

5.1 Implications

• 73/95 vulnerabilities were inherited; SAGE localized origins from reasoning in 72, while 14/19 agent-introduced cases diverged before the write.Code-level analysis would have identified an origin in only five agent-introduced cases.Source: §5.1 · p. 16
• Origin-type agreement exceeded exact-turn agreement under repeated scoring (96.49% vs 87.72%) and between judges (92.00% vs 78.00%), supporting separate reporting.Small score differences can shift the selected turn without changing the origin type; one-turn tolerance recovered only part of the disagreement.Source: §5.1 · pp. 16–17
• Human agreement assesses artefact-derived provenance and writes, whereas repeatability assesses judge-dependent origin types and turns; neither establishes that a reasoning-level origin is correct.For example, repeated turns were stable in some groups despite lower agreement with an annotator.Source: §5.1 · p. 17
• Final-only traces had the lowest agreement in all three comparisons, although their small group and lower agreement for per-write traces also qualify the snapshot-preservation implication.The authors suggest recording a snapshot after each write and preserving individual write boundaries.Source: §5.1 · p. 17
• The observed missed-detection pattern is corpus-specific, and whether review checks reduce silent failures remains untested without controlled intervention studies.At least one missed detection opportunity was flagged in 59 of 95 cases.Source: §5.1 · p. 18

5.2 Threats to Validity

• The origin turn is the earliest observable, trace-attributable divergence under SAGE, not proof that the turn caused the vulnerability or that intervention there would prevent it.Incomplete records may conceal earlier divergence, and a zero score means no relevant reasoning was recorded, not that the agent had none.Source: §5.2 · p. 18
• Judge scoring may vary or share biases; the two judges jointly localized 50/60 cases with 78.00% exact-turn agreement, and shared reconstruction errors also remain possible.The measures used to address these risks reduce but do not remove the threat.Source: §5.2 · p. 18
• The evidence covers 95 confirmed Python failures in two security-focused benchmarks, and may not generalize to other languages, task types, or deployment settings.Candidate screening excludes vulnerable patches not flagged by either screening approach; estimates for groups under ten cases are particularly uncertain.Source: §5.2 · p. 18

6 Conclusion

• SAGE localizes the origin of confirmed silent failures, distinguishing the first observable divergence from the final patch’s evidence that a repair is insecure.For inherited vulnerabilities, the origin can be localized even when no code change marks it.Source: §6 · pp. 18–19
• Origin types are reproduced more consistently than exact turns, and judge consistency and agreement with human annotators measure different outputs.The conclusion recommends reporting both kinds of agreement separately.Source: §6 · p. 19
• Agreement was lowest for final-file-only traces, motivating preservation of intermediate code states; future work will test other languages, task types, and safeguards through controlled studies.The paper does not establish that safeguards placed at identified origins prevent silent failures.Source: §6 · p. 19