PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

Priyam Sahoo, Gaurav Mittal, Xiaomin Li, Shengjie Ma, Benjamin Steenhoek, Pingping Lin, Yu Hu

arXiv:2605.12925v3cs.SEcs.AI

TL;DR

Binary pass/fail evaluation does not distinguish principled solutions from chaotic trial-and-error processes. AgentLens assesses SWE-agent trajectories against task-level references and labels their actions by intent. Across eight backends, process-quality rankings differ from pass-rate rankings, and 10.7% of passing trajectories on PTA-eligible tasks are Lucky Passes.

  • Problem

    Binary pass/fail evaluation does not distinguish principled SWE-agent solutions from trial-and-error processes.

    Source: §1 · p. 1

  • Method

    AgentLens assesses trajectories against task-level Prefix Tree Acceptor references built from passing solutions and labels actions by context-sensitive intent stages.

    Source: §1 · p. 2

  • Results

    Across eight backends, process-quality rankings differ from pass-rate rankings; 10.7% of passing trajectories on 47 PTA-eligible tasks are Lucky Passes.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Process-aware assessment distinguishes correct outcomes reached through coherent, low-waste processes from weak successful trajectories reached through brittle behavior.

    Source: §7 · p. 10

    The evaluation focuses on post-hoc OpenHands-style traces on SWE-bench Verified tasks and uses fixed score weights for deterministic model comparisons.

    Source: §A.1 · p. 15

Abstract

from arXiv · show

Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and define AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We plan to release the project repository soon, including AgentLens-Bench artifacts, the AgentLens SDK, and the analysis tooling.

1 Introduction

Binary pass/fail evaluation hides meaningful differences in how SWE agents reach successful outcomes. AgentLens introduces process-level assessment to identify those differences and characterize solution quality.

Motivation

  • Pass-only evaluation treats direct solutions and trial-and-error successes alike, obscuring demonstration quality, a useful model-comparison axis as pass rates converge, and potential deployment risks.Trajectory datasets filtered by pass rate may therefore select brittle or wasteful successful demonstrations alongside direct, coherent solutions.Source: §1 · pp. 1–2

Lucky Pass findings

  • Among 1,136 passing trajectories, 10.7% are Lucky, 69.1% Solid, and 20.2% Ideal; Lucky rates span 0.5%–23.2%, and quality-score rankings disagree with pass-rate rankings across all eight configurations.Some models move by as many as five rank positions when ranked by quality score instead of pass rate.Source: §1 · pp. 1–2, Figure 1 · p. 2

AgentLens framework

  • AgentLens builds task-level Prefix Tree Acceptors by merging passing solutions and labels actions as Exploration, Implementation, Verification, or Orchestration using trajectory history rather than tool identity alone.The resulting references represent known-good strategies while accommodating valid alternative paths and flagging redundant retries, irrelevant exploration, and divergence.Source: §1 · p. 2

Dataset and validation

  • On 2,614 trajectories from 60 SWE-bench Verified tasks, 47 had enough passing trajectories for task-level PTAs, forming AGENTLENS-Bench with 1,815 process-annotated trajectories; seven annotators yielded Fleiss’ κ = 0.933.Source: §1 · p. 3

PTA-based scoring

  • The composite quality score combines structural alignment with coverage, coherence, and temporal-profile signals, and significantly separates passing from failing trajectories on a pilot validation set (p = 0.0017).Source: §1 · p. 3

Planned tooling

  • The planned project repository will provide an SDK and web interface for process-aware analysis, with a deterministic pipeline that requires neither LLM calls nor external API access.The pipeline supports ATIF logs and OpenHands traces, while other agent logs can be handled with a lightweight adapter.Source: §1 · pp. 3–4

2 Related Work

Prior work evaluates final correctness, characterizes agent processes, or provides SWE-agent trajectories, but the passages identify no dataset combining the process-level annotations AgentLens-Bench provides. AgentLens distinguishes itself with process scores, context-sensitive intent labels, multi-solution PTA references, and structured inefficiency attribution.

Outcome-based SWE-agent benchmarks

  • SWE-bench and adjacent coding benchmarks evaluate final correctness, while ABC documents measurement errors such as insufficient test coverage; AgentLens instead assesses the process producing outcomes.SWE-bench established binary pass/fail evaluation; ABC reports measurement errors (Zhu et al., 2025).Source: §2 · p. 4

Process-level trajectory evaluation

  • Graphectory is the closest prior work, encoding execution traces as graphs; AgentLens adds deterministic, decomposable SWE-process scores and structured inefficiency attribution with divergence localization.Other studies characterize SWE-agent trajectories through thought-action-result patterns, length, variance, or patch quality (Bouzenia and Pradel, 2025; Majgaonkar et al., 2025).Source: §2 · p. 4

Trajectory datasets for SWE agents

  • Existing SWE-agent datasets organize trajectories primarily by outcome, whereas AgentLens-Bench combines per-trajectory quality scores, reference graphs, divergence localization, and waste annotations.Examples include SWE-Gym, R2E-Gym (Jain et al., 2025), SWE-smith (Yang et al., 2025), and OpenHands logs (Wang et al., 2024b).Source: §2 · p. 4

3 How AGENTLENS Works

AGENTLENS turns raw logs into labeled states, builds a task-specific reference from passing solutions, scores each candidate against that reference, and returns a structured quality report.

3.1 From raw logs to labeled states

  • Each log step becomes a state with its tool, target file, affected line range, content hash, position, and context-sensitive intent label across four cognitive phases.The labeler tracks whether implementation has occurred and which files were edited, so reading a file can be exploration before a patch and verification afterward.Source: §3.1 · pp. 4–5

3.2 Building a PTA reference

  • A task reference merges at least two passing trajectories into a Prefix Tree Acceptor: shared prefixes merge, while successful strategy divergences remain separate branches.Equivalent actions can be matched despite surface differences, such as grep and rg calls expressing the same search intent.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Scoring a candidate trajectory

  • Four signals—structural alignment, set coverage, trajectory coherence, and temporal profile similarity—form the 0–100 score, weighted 20%, 15%, 30%, and 35%, respectively.Weights were selected by grid search on a disjoint pilot calibration set and held fixed; coherence and temporal similarity together receive 65% of the total weight.Source: §3.3 · pp. 5–6

3.4 Reports, tiers, and waste signals

  • Reports include the composite score, per-stage coverage, divergence localization, and five localized waste categories; passing trajectories are classified as Ideal, Solid, or Lucky.Passing scores of at least 70 are Ideal, scores from 47 to below 70 are Solid, and scores below 47 are Lucky; waste categories are regression loops, blind retries, redundant steps, unnecessary exploration, and cyclic patterns.Source: §3.4 · p. 6

4 Experimental Setup

The experiments assess AgentLens on OpenHands trajectories from SWE-bench Verified, using a task-filtered benchmark subset and frozen signal weights calibrated on separate pilot data. The setup also compares three alignment baselines and reports discrimination, threshold, and significance metrics.

Compute environment

  • AgentLens scoring and analyses run locally on 11 CPU cores and 18GB memory without GPU workers; trajectory generation uses separate external model API calls.Source: §4 · p. 6

Dataset

  • AgentLens-Bench contains 1,815 trajectories across 47 tasks and eight model backends, selected from 2,614 OpenHands trajectories across 60 SWE-bench Verified tasks.The benchmark subset has 1,136 passing and 679 failing trajectories; the full corpus has 1,389 passing, 1,217 failing, and 8 unrecorded outcomes. OpenHands (Wang et al., 2024b) is evaluated on SWE-bench Verified (Chowdhury et al., 2024).Source: §4 · p. 6

Calibration and holdout

  • Weights were calibrated on a separate 278-trajectory pilot set, frozen for scaled evaluation, and each scored trajectory was excluded from its task-specific reference PTA.Grid search optimized AUROC with step 0.05 and unit-sum constraint, yielding w = (0.20, 0.15, 0.30, 0.35), pilot AUROC = 0.755, and pilot F1 = 0.791; no pilot data entered scaled-set discrimination.Source: §4 · p. 6

Baselines

  • AgentLens is compared with individual-trajectory matching, TF-IDF alignment in the BERTScore style (Zhang et al., 2020), and text-embedding-3-large dense alignment.Source: §4 · p. 6

Metrics

  • Micro-averaged AUROC is the primary discrimination metric, thresholds use Youden’s J, and significance testing reports Kolmogorov–Smirnov p-values.Source: §4 · p. 6

5 Results

Among successful trajectories, AgentLens distinguishes Ideal, Solid, and Lucky processes, showing that passing patches are not equally good demonstrations. Its combined score also separates pass/fail outcomes while exposing process-quality mismatches that binary evaluation misses.

5.1 Passing trajectories are not behaviorally homogeneous

  • Among 1,136 passing trajectories, 229 are Ideal (20.2%), 785 Solid (69.1%), and 122 Lucky (10.7%).Some non-Ideal trajectories are efficient-but-atypical: coherent, temporally organized paths with low structural overlap that the full score distinguishes from Lucky Passes.Source: §5.1 · p. 7

5.2 Lucky Passes reveal recurring weak-success mechanisms

  • C2 Brute-Force Convergence and C3 Incomplete Implementation together account for 68.0% of Lucky Passes.The five mutually exclusive categories are assigned using signals including trajectory length, verification coverage, waste patterns, implementation coverage, and coherence.Source: §5.2 · p. 7, Table 1 · p. 7
  • Failing trajectories are 58% more likely to inspect outside the known-good solution space (F/P = 1.58), and cyclic patterns are 32% more prevalent (F/P = 1.32).Lucky trajectories with blind retries waste 11.4 steps per instance, versus 2.7 in Ideal trajectories, a 4.2× increase; waste is measured relative to task-level PTA paths.Source: §5.2 · pp. 7–8
  • All eight models’ QS and PR rankings disagree; Lucky rates range from 0.5% to 23.2%, while Opus 4.6’s 77.3% pass rate includes an 18.7% Lucky rate.GPT-4o rises from 8th by pass rate to 3rd by quality score, but this shift is conditional on the tasks it solves and may reflect task selection.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 The combined score separates passing and failing trajectories

  • On the scaled 47-task set, the combined score separates passing and failing trajectories at AUROC = 0.766, accuracy = 72.0%, F1 = 0.723, and KS p = 0.0017.No individual signal reaches p < 0.05; AgentLens outperforms TF-IDF and dense embedding alignment by 0.065–0.094 AUROC, while individual trajectory matching scores AUROC = 0.805.Source: §5.3 · p. 8, Table 3 · p. 9
  • Among failing trajectories, 54.9% are Partial-fail and 45.1% Off-track, indicating that roughly half follow a reasonable strategy but make a localized error.Partial-fail trajectories remain close enough to known-good processes to exceed the failure-tier threshold; Off-track trajectories diverge earlier or more completely.Source: §5.3 · p. 9

5.4 Intent-stage labels are reliable

  • Intent-stage labels reached Fleiss’ κ = 0.933 and 96.0% raw agreement; against consensus on 192 of 200 states, the heuristic achieved 93.8% accuracy and macro-F1 = 0.933.Disagreements concentrated on post-implementation read_file calls, the boundary case targeted by the context-sensitive labeler.Source: §5.4 · p. 9

6 Robustness and Ablations

Controlled ablations on a disjoint 278-trajectory pilot set test the score’s signals, PTA merge count, and merge order. The results support fusing structural and behavioral signals, using k = 5 for scaled evaluation, and treating reference-set choice as the main source of PTA variation.

Signal contribution

  • Removing temporal profile divergence, trajectory coherence, set coverage, and structural alignment lowers AUROC by 0.037, 0.031, 0.024, and 0.016, respectively; ±0.05 weight changes reduce it by at most 0.006.The findings support fusing structural and behavioral signals rather than treating them as substitutes.Source: §6 · p. 9

Merge-count sensitivity

  • On the pilot set, k = 2 achieves AUROC = 0.749 with full task coverage, while k = 5 achieves AUROC = 0.777 with broader coverage; k ≥6 reflects survivorship bias, so k = 5 is used.Smaller merge counts can penalize valid strategies absent from the reference, while larger counts can become overly permissive; at k ≥6, only a small subset of easier-to-score task resamples remains.Source: §6 · p. 9

Merge-order robustness

  • Trajectory selection explains 64.1% of PTA-score variance and merge ordering 35.9%; 8 of 10 trajectory combinations are order-invariant, making reference-set choice the larger source of variation.The remaining order effects arise when an ambiguous exploration prefix can be merged or branched, and are bounded.Source: §6 · pp. 9–10

7 Conclusion · Appendix · A Limitations, Future Directions, and Positioning

AGENTLENS evaluates SWE-agent trajectories by process as well as outcome, revealing that some successful runs rely on weak, recurring behaviors. Its process-aware scoring also changes how model backends compare by quality rather than pass rate.

7 Conclusion

  • AGENTLENS converts raw trajectories into intent-labeled state sequences and merges passing trajectories into task-level PTA references.This process-aware framework evaluates more than binary pass/fail outcomes.Source: §7 · p. 10
  • AGENTLENS distinguishes direct solutions, valid alternative paths, weak successes reached through brittle behavior, and failed trajectories close to known-good processes.It scores whether an agent reached a correct patch and whether its process was coherent and low-waste.Source: §7 · p. 10
  • 10.7% of passing trajectories on the 47 PTA-eligible tasks are Lucky Passes, and Brute-Force Convergence and Incomplete Implementation account for 68.0% of them.The evaluation covers 2,614 OpenHands trajectories from 60 SWE-bench Verified tasks.Source: §7 · p. 10
  • Across all eight evaluated model backends, quality-score rankings differ from pass-rate rankings, while Lucky rates range from 0.5% to 23.2%.These results show that process-aware scoring changes model comparison.Source: §7 · p. 10
  • The evaluation used 2,614 OpenHands trajectories from 60 SWE-bench Verified tasks, with 47 PTA-eligible tasks forming AGENTLENS-Bench.The Lucky Pass rate is reported for passing trajectories on those 47 tasks.Source: §7 · p. 10
  • The authors plan to release AGENTLENS-Bench, the AGENTLENS SDK, and a web interface soon to support process-aware analysis in coding-agent evaluation.The stated goal is to make process-aware analysis a standard layer in coding-agent evaluation.Source: §7 · p. 10

A.1 Limitations … A.4 Extended Related Work and Positioning

AgentLens is a post-hoc process-analysis layer whose current evaluation focuses on OpenHands-style traces, while its proposed uses extend to training, longitudinal tracking, and other agent domains. Its process scores are intended to complement—not replace—functional correctness, security review, or human judgment.

A.1 Limitations

  • AgentLens analyzes OpenHands-style SWE-bench Verified traces; applying it to other agent scaffolds mainly requires trace-format adapters or ATIF conversion.Its core scoring pipeline need not change for those extensions.Source: §A.1 · p. 15
  • The paper fixes score weights for deterministic model comparisons, but future deployments may tune them to domain-specific priorities such as verification discipline, exploration cost, or repair efficiency.Source: §A.1 · p. 15

A.2 Future Directions

  • AgentLens scores could serve as dense reinforcement-learning rewards for coherent, low-waste processes and help organize training curricula by process quality, divergence type, or recoverability.The paper cites Wei et al. (2025) in proposing process-quality reward signals.Source: §A.2 · p. 15
  • Longitudinal tracking could reveal whether model-version pass-rate gains reflect cleaner reasoning, broader exploration, or more retries, and trajectory analysis could extend to web navigation and computer-use agents.The passage cites Zhou et al. (2023) and Xie et al. (2024) for those adjacent domains.Source: §A.2 · p. 15

A.3 Broader Impacts

  • By exposing brittle successes, recoverable failures, and wasteful processes, AgentLens can support safer deployment decisions, cleaner training demonstrations, and diagnosis of weaknesses hidden by pass/fail benchmarks.Source: §A.3 · p. 15
  • AgentLens scores are complementary diagnostics, not substitutes for functional correctness, security review, or human judgment; high-scoring trajectories may still contain incorrect or insecure patches.The paper positions the framework as an additional evaluation layer, not an automatic code-deployment approval mechanism.Source: §A.3 · p. 15

A.4 Extended Related Work and Positioning

  • The extended related-work section compares AgentLens-Bench with related SWE-agent trajectory collections and AgentLens with related evaluation frameworks.Source: §A.4 · p. 15

B Additional Method Details … C.6 Score Distributions

Appendix B provides reproducibility-level details on tool mapping, state equivalence, intent labeling, and scoring; Appendix C adds behavioral, waste, validation, and score-distribution analyses. Together, these details explain how process quality is measured and where the evaluation signals are informative.

B Additional Method Details; B.1 Tool Registry; B.2 State Equivalence Engine

  • The tool registry supplies default intent hints, while a confidence-weighted equivalence engine matches cross-agent states using content hashes, AST scopes, and overlapping file regions.Registry hints are defaults that the context-sensitive labeler may override; the engine addresses differences in tool names, read regions, and terminal arguments.Source: §B.1 · p. 16, Table 6 · p. 16, §B.2 · pp. 16–17

B.3 Intent-stage Labeling Flow

  • The seven-rule intent labeler uses context, edited-file state, and command semantics to distinguish stages; naive terminal-tool labeling collapses 70–80% of PTA states into one category.File-inspection commands remain Exploration, while other context-sensitive rules consider prior source edits, test-file targets, and prior modifications.Source: §B.3 · p. 17, Figure 4 · p. 18

B.4 Scoring Details

  • 0.20 · Φstruct + 0.15 · Φcov + 0.30 · (100 · Φcoh) + 0.35 · (100 · Φtemp) combines structural alignment, set coverage, trajectory coherence, and temporal profile divergence.Behavioral signals contribute 65% of the combined score, a result of grid-search optimization rather than a design choice.Source: §B.4 · pp. 17–19

B.5 Scoring Signal Examples

  • Figure 5 contrasts principled and chaotic examples across coherence, structural alignment, set coverage, and temporal profile divergence, illustrating complementary scoring signals.The examples show that a candidate can cover many PTA states yet have low structural alignment when it visits them in a scrambled order.Source: §B.5 · p. 19, Figure 5 · p. 20

C Additional Experimental Results; C.1 Behavioral Profiles

  • Roughly 53% of passing trajectories are efficient-but-atypical: their coherence and temporal alignment are high despite low structural F1, so the combined score classifies them as Solid, not Lucky.This illustrates why combining signals avoids treating principled but unconventional paths as chaotic.Source: §C.1 · pp. 20–21, Table 7 · p. 21

C.2 Pass/Fail Waste Breakdown; C.3 Ideal-versus-Lucky Waste

  • Failing trajectories show higher prevalence of unnecessary exploration (F/P = 1.58) and cyclic patterns (F/P = 1.32), while Lucky passes have lower blind-retry prevalence but greater waste when retries occur.Cyclic instances cost 7.8 steps in failing runs versus 4.6 in passing runs; waste detections exclude patterns already present in the merged PTA.Source: §C.2 · p. 21, Table 8 · p. 21, §C.3 · p. 21

C.4 Failure-mode Gallery; C.5 Heuristic Labeler Validation

  • The failure-mode gallery illustrates six annotated timelines, and heuristic validation reports 93.8% accuracy and macro-F1 = 0.933 against annotator consensus.Agreement is strongest for Orchestration (κ = 1.000), while the difficult terminal-command E-versus-V boundary reaches κ = 0.939; Implementation’s κ = 0.713 reflects only 8 sampled states.Source: Figure 6 · p. 22, §C.5 · p. 22

C.6 Score Distributions

  • At the fixed 46.4 Youden-J threshold, passing scores concentrate at 50–75 while failing scores peak around 30–40; the distributions differ (KS p = 0.0017).The overlap around 35–55 includes Solid passing trajectories and Partial-fail trajectories, often involving a localized implementation step.Source: §C.6 · pp. 22–23, Figure 7 · p. 23

C.7 Baseline Results … D.4 Model-specific Lucky Pass Signatures

AgentLens complements outcome-based evaluation with interpretable PTA diagnostics and finds substantial variation in the quality and mechanisms of passing trajectories across models. Its extended analysis profiles five Lucky Pass categories and shows that category patterns are associated with model identity.

C.7 Baseline Results

  • Individual matching achieves the highest AUROC, but the PTA provides structured diagnostics by localizing an out-of-branch edit to step 8 and flagging wasted exploration.The branch-aware output supports trajectory curation, model comparison, and waste reports.Source: §C.7 · pp. 23–24, Table 12 · p. 23

C.8 Model-comparison Visualizations

  • A 46× Lucky-rate gap separates Opus 4.5 (0.5%) from GPT-4.1 (23.2%), while pass rate and quality score show substantial rank disagreement across eight models.GPT-4o is lower-left by pass rate but upper-left by quality; Opus 4.6 is upper-right by pass rate but mid-right by quality.Source: §C.8 · p. 24, Figure 9 · p. 25, Figure 8 · p. 25

D Extended Lucky Pass Analysis

  • Appendix D extends the Lucky Pass analysis with signal profiling, category distributions, waste and model cross-tabs, task-level patterns, case studies, and verification-gap analysis.Categories are assigned automatically by a deterministic, reproducible pipeline without manual labels.Source: D Extended Lucky Pass Analysis · p. 24

D.1 Signal Landscape and Taxonomy Design

  • Early divergence (99.2%) and missing verification (94.3%) are near-universal among 122 Lucky trajectories, so the taxonomy differentiates categories using length, waste, completeness, and coherence.A priority-ordered decision tree assigns each trajectory to one mutually exclusive category using structural signals.Source: §D.1 · p. 24

D.2 Category Distribution

  • Of 122 Lucky Passes, C2 brute-force convergence (42, 34.4%) and C3 incomplete implementation (41, 33.6%) lead; C1, C4, and C5 account for 19, 5, and 15.C1 fixes are short and unverified; C2 retries blindly or cycles; C3 partial fixes pass despite limited test coverage; all five C4 cases come from GPT-4.1; C5 reflects valid alternative strategies and is least concerning.Source: Figure 10 · p. 26, §D.2 · p. 26

D.3 Waste as a Category Differentiator

  • C1 and C5 both have near-zero waste for different reasons—shortness versus coherent alternative strategies—while C2 has nearly half its trajectory wasted.Table 13 reports waste metrics by category.Source: §D.3 · p. 26, Table 13 · p. 27

D.4 Model-specific Lucky Pass Signatures

  • Category and model are strongly associated (χ2(28) = 102.47, p < 0.0001; Cramér’s V = 0.458): Opus 4.6 dominates C1, GPT-4.1 C2 and C4, and Codex models C3.Opus 4.6 accounts for 89.5% (17/19) of C1; GPT-4.1 accounts for 18/42 C2 cases and all C4 cases, while GPT-5.2-Codex and GPT-5.3-Codex cluster in C3. Sonnet 4.5 and Opus 4.5 produce only 2 and 1 Lucky Passes, respectively.Source: §D.4 · p. 27, Table 14 · p. 27

D.5 Task-level Concentration … E.2 Merge-count Sensitivity

Across the merged sections, Lucky Passes cluster by task but vary by model-task interaction, while case studies show how distinct behaviors can pass binary tests. Ablations show that behavioral and structural signals all matter, and that PTA merge count trades solution coverage against scoring constraints.

D.5 Task-level Concentration

  • 30 of 47 PTA-eligible tasks produce Lucky Passes, and the top 10 account for 77 of 122 (63.1%); Lucky Pass type varies with the model-task interaction.The task psf__requests-1724 yields C1 from Opus 4.6, C2 from GPT-4.1, and C3 from GPT-5.3-Codex; task-category concentration is significant, χ2(116) = 248.45, p < 0.0001, Cramér’s V = 0.714.Source: §D.5 · pp. 27–28, Table 15 · p. 27

D.6 Case Study: One Task, Five Behavioral Profiles

  • Five passing trajectories on psf__requests-1724 score 22–88, though binary evaluation treats them alike; AgentLens distinguishes an Ideal repair, a Solid repair, and three Lucky mechanisms.GPT-4o’s Ideal trajectory finds the relevant method.upper() call, fixes it, adds a regression test, runs relevant tests, and reviews the diff; the Lucky trajectories stop after an unverified edit, implement incompletely, or spend 34 states reading and grepping without progress.Source: §D.6 · p. 28, Table 16 · p. 28

D.7.1 C2: GPT-5.2-Codex on matplotlib__matplotlib-22719; D.7 Additional Lucky Pass Cases

  • The matplotlib C2 trajectory reaches a correct edit after searching the wrong directory, but 12 of 18 steps are wasted and only 2 of 19 alignment steps match ground truth.It costs 235,346 tokens versus a 9-step ground-truth solution; the additional cases report trajectory length, token cost, stage sequence, and ground-truth comparison.Source: §D.7.1 · p. 28, §D.7 · p. 28

D.7.2 C3: GPT-5.3-Codex on psf__requests-1724; D.7.3 C4: GPT-4.1 on django__django-11066

  • The requests C3 trajectory covers 0% of ground-truth implementation steps because it writes a reproducer instead of editing source; the Django C4 trajectory spends 57 steps exploring after verification.C4 uses 2.62M tokens for a one-line fix, and all five C4 instances come from GPT-4.1, which continues exploring after tests pass.Source: §D.7.2 · pp. 28–29, §D.7.3 · p. 29

D.7.4 C5: Opus 4.6 on pylint-dev__pylint-4970; D.7.5 C5: GPT-5.2-Codex on sphinx-doc__sphinx-10323

  • The pylint C5 agent uses an alternative implementation with early-return guards, while the Sphinx C5 trajectory completes an E→I→V→O lifecycle despite low ground-truth coverage.For pylint, 14.3% coverage reflects skipped verification and orchestration more than missed implementation. For Sphinx, 11.3% coverage reflects a structurally different approach, supporting multi-reference ground truths.Source: §D.7.4 · p. 29, §D.7.5 · pp. 29–30

D.8 Verification Gap

  • 94.3% of 122 Lucky Passes have missing verification; verification ranges from zero by definition for C1 to 87% for C5, marking C1 as most concerning and C5 as least.The authors identify verification as the most actionable factor; interventions differ by pattern, including verification training for C1, termination heuristics for C2, and better test suites for C3.Source: §D.8 · p. 30

E.1 Signal Contribution; E Ablation Details

  • Removing temporal profile or trajectory coherence drops AUROC by −0.037 or −0.031, versus −0.024 for set coverage and −0.016 for structural alignment; all four signals contribute.Shifting any single weight by ±0.05 reduces combined AUROC by at most 0.006. Ablations use a disjoint 278-trajectory pilot set, with weights and thresholds frozen before scaled experiments.Source: E Ablation Details · p. 30, §E.1 · p. 31, Table 18 · p. 31

E.2 Merge-count Sensitivity

  • k = 5 yields AUROC 0.777 versus 0.749 for k = 2, while covering a broader solution space; its 31 resamples reflect PTA-size scoring limits, not methodological failure.At k ≥6, rising AUROC reflects survivorship bias because only easier-to-score resamples remain. The study uses k = 5 for scaled experiments.Source: Table 19 · p. 31, §E.2 · p. 31

E.3 Merge-order Robustness

Because incremental PTA construction could make scores depend on arbitrary merge order, this study tests merge-order robustness on one pilot task. It finds trajectory selection is the larger source of PTA variation than merge ordering.

E.3 Merge-order Robustness

  • The robustness test addresses whether incremental PTA construction makes quality scores depend on the arbitrary order in which passing trajectories are merged.The concern is that the same set of passing trajectories could yield different scores under different processing orders.Source: §E.3 · p. 32
  • Eight of ten trajectory combinations had zero within-combination variance, while observed ordering variation arose when ambiguous exploration prefixes affected whether states were merged or branched.Even in the combinations with nonzero ordering variance, AUROC ranges were bounded and the effects were smaller than those of trajectory selection.Source: §E.3 · p. 32
  • 64.1% of total variance came from trajectory selection versus 35.9% from merge ordering across 60 scoring runs.The study evaluated 10 random trajectory combinations, each under all 6 merge permutations, for k = 4 on astropy__astropy-12907.Source: Table 20 · p. 32, §E.3 · p. 32

F Token Cost and Statistical Tests · F.1 Statistical Tests for Lucky Pass Taxonomy

Lucky Pass categories carry sharply different token costs despite identical binary outcomes, and the taxonomy is associated with both model choice and task characteristics. Verification coverage also varies by category.

F Token Cost and Statistical Tests

  • C1 averages 34K tokens per trajectory; C2 costs 25× more, while C4 costs 40× more and reaches 2.62M tokens for a one-line fix.C3 and C5 are described as moderately expensive.Source: F Token Cost and Statistical Tests · p. 32
  • C4 costs 40× more than C1 on average for the same binary evaluation outcome.The reported token expenditure varies by a factor of 40 across Lucky Pass categories.Source: Table 21 · p. 32, F Token Cost and Statistical Tests · p. 32
  • Across 122 Lucky Passes, approximately 58.3M tokens were spent; C1-level efficiency would require 4.1M, leaving 54M excess tokens, or 93% of the aggregate.The excess is attributed to C2 and C4 patterns.Source: F Token Cost and Statistical Tests · pp. 32–33
  • At current API pricing, 122 Lucky Passes cost roughly 14× the C1-efficiency alternative, while pass/fail scores conceal token-efficiency differences that affect deployment economics.C2 trajectories cost 25× more per trajectory than C1 trajectories despite receiving identical binary scores.Source: F Token Cost and Statistical Tests · p. 33

F.1 Statistical Tests for Lucky Pass Taxonomy

  • Both taxonomy associations are highly significant with large to very large effect sizes: model choice predicts Lucky Pass type, and task characteristics predict the dominant category.The reported association tests concern model choice and task characteristics.Source: §F.1 · p. 33, Table 22 · p. 33
  • Verification coverage is zero for C1 by definition, 40–60% for C2 and C3, and 87% for C5.The per-category coverage supports the causal structure described in Section D.8.Source: §F.1 · p. 33

Related papers

Loading 2605.12925v3…