PaperlayerView exact revision on arXiv

Source-linked AI summary

Confident and Wrong: Silent Semantic Failures in Coding Agents

Aman Mehta

arXiv:2603.25764v3cs.SEcs.AI

TL;DR

Coding-agent evaluations can mistake action for correctness, especially when repeated runs produce confident but wrong patches. The paper measures this gap across repeated SWE-bench trials, finding that submit rate can reverse model rankings and that silent semantic failure dominates some models. It therefore argues for test-verified, uncertainty-aware evaluation that also measures abstention.

  • Problem

    Submit rate is often used as a cheap proxy for coding-agent ability, although it may not reflect whether patches actually fix bugs.

  • Method

    The study compares submit rate with test-verified resolve rate across four models and repeated SWE-bench trials, while measuring reliability, interventions, and abstention.

  • Results

    Submit rate and resolve rate disagree on model ranking: GPT-5 submits 100% and resolves 44%, while Gemini submits 70% and resolves 50%.

  • Takeaways & Limitations

    Evaluation should report test-verified correctness over repeated runs with uncertainty, alongside reliability and specificity rather than submit rate alone.

  • Takeaways & Limitations

    The evidence is limited to 50 SWE-bench Verified Python tasks, one scaffold, and a specificity probe using 50 trials per model on a 10-task subset.

Abstract

from arXiv · show

As coding agents move into production workflows, teams need to know not only whether an agent completes a task, but whether its action can be trusted. We show that completion and trustworthiness diverge sharply and systematically. Across 1,750 trajectories on 50 SWE-bench Verified tasks, we compare four frontier models over repeated runs and separate submit rate from test-verified resolve rate. GPT-5 submits a patch on 100% of runs but resolves only 44%; Llama 4 submits on 99% but resolves 18%; and Gemini, despite submitting least often at 70%, resolves more tasks than GPT-5 (50% versus 44%). These gaps are not random: they concentrate in one dangerous failure mode we call silent semantic failure. Qualitatively, on a buggy task the agent submits a plausible-looking patch on all five runs, yet none pass, the same misinterpretation repeated rather than random error. Quantitatively, it dominates failure, covering 80% of Llama 4's failing runs and 68% of GPT-5's, and it is invisible: the outcomes are confidently and consistently wrong, so completion-based and consistency-based monitoring both look healthy exactly when the agent should not be trusted. Lightweight pre-edit prompts do not close the gap. A second probe isolates the instinct to act: given an already-fixed bug, where the right move is to abstain, most models still edit the correct code. This action bias, acting when no action is warranted, is exactly what completion metrics reward. The throughline is measurement: submit rate captures action, but trust requires validity. So evaluation must catch up: score agents by test-verified correctness over repeated runs, report its uncertainty, and reward those that know when not to act.

1 Introduction

The paper argues that submit rate measures action rather than correctness, creating a dangerous gap when agents repeatedly produce plausible but test-failing patches. Across repeated evaluations, silent semantic failure and action bias show why validity, consistency, and abstention must be measured together.

  • Measurement problem: 100% submit rate but 44% resolve rate shows that completion can look flawless while every patch fails on 19 of 50 tasks.GPT-5 submits a confident-looking patch on every run for those tasks, but none pass.
  • Action bias: Action bias appears both as confident wrong patches when bugs exist and as edits to already-correct code when abstention is warranted.The specificity probe treats unnecessary edits as the mirror image of silent semantic failure.
  • Validity: Submit rate and resolve rate disagree on model ranking: Gemini submits 70% but resolves 50%, surpassing GPT-5’s 44%.Bootstrap leaderboards reverse the GPT-5/Gemini ranking depending on the indicator.
  • Reliability: Repeated consistency can amplify a wrong interpretation: pacing, procedure, and patch consistency are near-orthogonal, with correlations no greater than 0.14.The hierarchy holds across all five repositories, but consistency does not establish validity.
  • Interventions: Two pre-edit guards fail to improve overall resolve rate, indicating that lightweight prompting does not repair interpretation failures.Trajectory signals partially recover validity without running tests, but the guard interventions do not raise resolve rate.

2 A Measurement Framework for Coding Agents

The framework distinguishes a hidden capability from the indicators used to approximate it, treating test-verified resolution as the criterion for whether submitted patches actually fix issues. Because repeated trajectories are noisy, it estimates performance across runs with uncertainty.

  • Construct measurement: Coding-agent capability is a hidden quantity θ, while submit rate is an indicator and test-verified resolve rate is the criterion.Construct validity asks whether the indicator tracks the criterion; here, it does not.
  • Reliability versus validity: Reliable repeated scores can still be invalid when an agent returns the same confident, wrong patch on every run.This reliable-but-invalid cell defines silent semantic failure.
  • Repeated measurement: Resolve probability is estimated from k repeated trials with bootstrap uncertainty because each trajectory is a dependent, noisy draw.The coefficient of variation measures step-count reliability, while resolve rate measures validity.

3 Related Work

Related work increasingly treats benchmarks as measurement instruments and emphasizes construct validity, instance-level reporting, and deployment-relevant metrics. Existing agent benchmarks commonly aggregate single-run results, leaving reliability underexamined, while prior failure research covers adjacent agent behaviors.

  • Measurement and benchmark validity: Prior measurement work imports construct-validity theory into machine learning and advocates instance-level, multi-metric, deployment-relevant evaluation.This paper operationalizes those ideas for coding agents by naming a concrete validity failure.
  • Agent benchmarks: SWE-bench and related benchmarks evaluate agents on real issues or interactive tasks, but most report single-run aggregates.Single-run reporting leaves reliability unexamined.
  • Consistency and safety: Prior studies document repetition variance, self-consistency, error categorization, recovery, tool misuse, and planning limits across agent systems.The paper focuses on when these failures become silently, reliably wrong.

4 Experimental Setup

The study evaluates four models on 50 SWE-bench Verified tasks spanning five Python repositories, using repeated trials and guard interventions. It separates submission from test-verified resolution and examines the resulting outcome patterns.

  • Tasks: 50 tasks span five mature Python repositories covering web, symbolic-math, machine-learning, visualization, and astronomy bugs.The design selects 10 tasks per repository for domain and bug-type diversity.
  • Models: Four hosted frontier models are compared with identical tools, prompts, and task descriptions.The models are Claude 4.5 Sonnet, GPT-5, Gemini 3.1 Pro, and Llama 4 Maverick.
  • Protocol: Each model-task pair receives 5 independent trials at temperature 0.5 with a 250-step limit, producing 1,000 baseline trajectories.The study adds 750 guard trajectories.
  • Metrics: Submit rate counts non-empty patches, resolve rate counts patches passing the full test suite, and CV measures step-count consistency.Lower CV indicates greater consistency.
  • Interventions: Two pre-edit guards test whether interpretation prompting or writing a minimal failing test can improve validity.The interpretation guard runs on Claude and GPT-5; the test-anchored guard runs on GPT-5.

5 Results

Repeated-run evaluation shows that reliability and validity diverge: models can behave consistently while submit rate misranks capability and misses silent semantic failures. Process signals recover part of the missing validity signal, while lightweight pre-edit guards do not improve resolve rate overall.

  • Reliability: Claude has the lowest step-count CV at 17.7%, followed by GPT-5 at 30.3%, Gemini at 44.9%, and Llama 4 at 58.1%.The ordering holds across all five repositories, establishing a stable reliability hierarchy.
  • Reliability: Step-count CV correlates weakly with command divergence (r=0.14) and essentially not with patch divergence (r≈0), so reliability has distinct pacing, procedure, and product facets.The three facets are near-orthogonal across 200 model-task cells.
  • Validity: GPT-5 submits 100% but resolves 44%, while Gemini submits 70% but resolves 50%, reversing their ranking under submit rate versus test-verified correctness.Bootstrap leaderboards reverse the GPT-5/Gemini order depending on the indicator.
  • Failure taxonomy: Silent semantic failure is defined by 5/5 submission and 0/5 resolution, and over two-thirds of Llama 4’s and GPT-5’s failures are invisible to submission monitoring.SSFR is 80% for Llama 4 and 68% for GPT-5.
  • Failure mechanism: Silent-failure patches are far more similar across repeated runs than unrelated patches, indicating convergent misinterpretation rather than random error.The same wrong fix is reconstructed repeatedly, while silent failures also cluster on shared interpretation-trap tasks.
  • Interventions: A 3-gram loop detector and trajectory length partially recover validity without tests, but pre-edit guards leave overall resolve rates essentially unchanged.The test-anchored guard gains on isolated bugs and loses on complex suites, while interpretation failures resist lightweight prompting.

6 Specificity: do agents know when to abstain?

The specificity probe tests whether agents abstain when an issue is already fixed. Most models still edit correct code, showing that submission-based evaluation rewards action even when no change is warranted.

  • Specificity: On 60–90% of runs, all four models edit correct code instead of abstaining on already-fixed bugs.Only GPT-5 abstains cleanly with any frequency, at 40%, yet it still over-acts on the majority of runs.
  • Specificity: Abstention is a distinct specificity capability that submission-based evaluation punishes.The probe labels non-empty patches as false actions and empty patches as clean abstentions.

7 Discussion: a measurement protocol

The paper proposes evaluating coding agents with test-verified correctness, repeated-run uncertainty, reliability measures, process signals, and abstention tests rather than submit rate alone. The central warning is that consistency without validity can create false confidence.

  • Measure specificity with already-fixed bugs, because agents that cannot abstain are miscalibrated even when their resolve rate is high.
  • Score agents by test-verified resolve rate rather than submit rate, because high consistency without validity can manufacture false confidence.
  • Report uncertainty from repeated trials, using at least k ≥3 runs with bootstrap confidence intervals.Best-of-3 raises Claude from 65→74% and GPT-5 from 44→58%.
  • Add test-free process measures, including 3-gram loop detection and short-trajectory flags, to recover part of the missing validity signal.The 3-gram detector achieves 100% recall with 0% false positives in the reported setting.

8 Limitations

The study’s conclusions are bounded by its benchmark, scaffold, language, scale, and abstention-probe coverage. The authors call for replication across languages and scaffolds and broader specificity evaluation.

  • The evidence comes from SWE-bench Verified Python tasks using one minimal mini-SWE-agent scaffold, so compiled languages and richer scaffolds remain open questions.
  • The benchmark includes 50 tasks, a modest scale mitigated by paired tests and bootstrap confidence intervals.
  • The specificity probe covers 50 runs per model on a 10-task subset rather than the full benchmark.
  • Future work should replicate across languages and scaffolds, test structural interventions, and scale abstention into a full specificity benchmark paired with resolve rate.

9 Conclusion

The conclusion is that submit rate measures action rather than correctness, with errors concentrated in confidently repeated silent semantic failures. Case studies and reliability analyses show why test-verified validity must anchor evaluation.

  • Submit rate measures whether an agent acts, not whether it is right, and can reverse the GPT-5/Gemini resolve ranking.Silent semantic failures are reliable, confident, test-failing patches where agents repeat the same wrong fix.
  • On astropy-13236, Claude and GPT-5 submitted on all five runs yet resolved none, illustrating reliable-but-invalid behavior.Claude spent 30–50 steps on a warning interpretation, while GPT-5 made the same error in 7 steps.
  • On matplotlib-25311, Gemini submitted 0/5 after repeated read-loops without editing, a costly but honest failure caught by loop detection.
  • Reliability facets are near-orthogonal, so step-count consistency does not capture patch-level consistency.The reported correlation is r ≤0.14.
  • Silent-semantic-failure patches are more self-similar across runs than variable-failure patches, indicating repeated convergence on the same wrong fix.

E Statistical Details

Bootstrap results show that submit rate and resolve rate produce different model rankings, while pairwise reliability comparisons reveal broad differences across models.

  • −12.6% is the Claude-versus-GPT-5 pairwise CV difference, significant at p < 0.001 under Bonferroni α = 0.0083.Claude also differs significantly from Gemini and Llama 4, while GPT-5 versus Gemini is borderline and Gemini versus Llama 4 is nonsignificant after correction.
  • 1.000 [1.000, 1.000] is GPT-5’s submit rate, ranking first in 98.3% of bootstrap samples.Claude submits at 0.969 [0.936, 0.996], Gemini at 0.697 [0.600, 0.796], and Llama 4 at 0.988 [0.972, 1.000].
  • 0.651 [0.600, 0.700] is Claude’s resolve rate, ranking first in every bootstrap sample.Gemini reaches 0.498 [0.420, 0.560], GPT-5 0.444 [0.360, 0.520], and Llama 4 0.180 [0.100, 0.260].

F Cost-Benefit of Multi-Run Evaluation

The section describes multi-run evaluation strategies and bootstrap resampling as ways to contextualize submit-rate leaderboards, with cost and ranking stability reported separately.

  • Table 6 compares multi-run submit-rate strategies using mean actual API cost per task.It includes hard-majority voting and Best-of-k, where Best-of-k is defined as an oracle upper bound.
  • Table 7 contextualizes submit-rate leaderboards through 1,000 bootstrap resamples.It reports the fraction of samples in which each model achieved each rank.
  • Table 7 also reports single-run reliability as the fraction of runs agreeing with the five-run majority vote.
Loading 2603.25764v3…