PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

Priyam Sahoo, Gaurav Mittal, Xiaomin Li, Shengjie Ma, Benjamin Steenhoek, Pingping Lin, Yu Hu

arXiv:2605.12925v3cs.SEcs.AI

TL;DR

Binary pass/fail evaluation can treat principled and chaotic SWE-agent solutions as equivalent, so AgentLens assesses trajectory quality using process-aware references and intent labels. It finds that 10.7% of passing trajectories are Lucky Passes, while process-quality rankings differ from pass-rate rankings across models.

  • Problem

    Binary test outcomes do not distinguish principled SWE-agent solutions from chaotic trial-and-error processes.

  • Method

    AgentLens labels trajectory intent stages and scores trajectories against task-level PTA references merged from multiple passing solutions.

  • Results

    10.7% of passing trajectories are Lucky Passes, and process-quality rankings differ from pass-rate rankings across all eight model backends.

  • Takeaways & Limitations

    Process quality provides a distinct basis for comparing SWE agents and filtering successful trajectories beyond pass rate.

  • Takeaways & Limitations

    The evaluation focuses on OpenHands-style traces on SWE-bench Verified, and its fixed score weights may require tuning for domain-specific priorities.

Abstract

from arXiv · show

Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and define AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We plan to release the project repository soon, including AgentLens-Bench artifacts, the AgentLens SDK, and the analysis tooling.

1 Introduction

The paper shows that binary pass/fail evaluation conflates principled SWE-agent solutions with weak processes that happen to pass. AGENTLENS addresses this gap with process-aware trajectory scoring, task-level PTA references, and the AGENTLENS-Bench dataset.

  • Motivation and findings: 10.7% of 1,136 passing trajectories exhibit Lucky Pass behavior, including regression cycles, blind retries, missing verification, or temporally disordered stages.Across models, Lucky rates range from 0.5% to 23.2%, while only 20.2% of passing trajectories are Ideal and 69.1% are Solid.
  • Evaluation implications: Pass-rate rankings disagree with process-quality rankings on all eight configurations, showing that successful trajectories are not behaviorally homogeneous.AGENTLENS separates passing trajectories into Lucky, Solid, and Ideal tiers, while the combined score significantly separates passing from failing trajectories on a pilot validation set (p = 0.0017).
  • Method: AGENTLENS uses context-sensitive intent labeling and merges multiple passing trajectories into task-level Prefix Tree Acceptor references of known-good strategies.The labeler distinguishes exploration, implementation, verification, and orchestration from trajectory history rather than tool identity alone.
  • Dataset: 1,815 trajectories from 47 PTA-eligible SWE-bench Verified tasks form AGENTLENS-Bench, with quality scores, waste annotations, divergence metadata, tier labels, and task-level references.The broader evaluation covers 2,614 trajectories from 60 tasks; 47 tasks have enough passing trajectories to construct PTAs.
  • Lucky Pass taxonomy: Lucky Passes decompose into five behavioral categories with significant model associations (χ2(28) = 102.47, p < 0.0001).The taxonomy identifies recurring weak-process mechanisms among trajectories that nevertheless reach correct patches.

2 Related Work

Prior SWE-agent evaluation largely relies on binary task outcomes, while newer work studies process-level trajectory behavior and trajectory datasets. AgentLens-Bench addresses a stated dataset gap by combining per-trajectory quality scores with reference graphs, divergence localization, and waste annotations.

  • Outcome-based SWE-agent benchmarks: SWE-bench established binary pass/fail as the standard for coding-agent evaluation, with later benchmarks refining this outcome signal.Refinements include human validation, decontamination, live issue streams, multi-language coverage, and realistic task pricing.
  • Process-level trajectory evaluation: Graphectory is identified as the closest prior process-level work, encoding execution traces as graphs and computing process-centric metrics independently of task success.Other studies analyze thought-action-result patterns, trajectory length and variance, patch quality, or broader agent behavior.
  • Trajectory datasets for SWE agents: Existing SWE-agent trajectory datasets primarily filter or organize traces by outcome rather than providing integrated process-quality annotations.AgentLens-Bench fills the stated gap by combining per-trajectory quality scores, ground-truth reference graphs, divergence localization, and waste annotations.

3 How AGENTLENS Works

AGENTLENS parses trajectories into context-sensitive intent states, compares candidates with a task-specific PTA built from multiple passing solutions, and produces a structured process-quality report. Its assessment combines structural, coverage, coherence, and temporal signals to quantify alignment and inefficiency.

  • Pipeline: AGENTLENS evaluates each candidate in four stages: log parsing, task-specific reference-graph construction, candidate scoring, and structured quality reporting.The pipeline converts raw trajectories into labeled states, builds references from passing solutions, scores candidates, and returns reports.
  • State Parsing: Each log step records tool, target file, affected lines, content hash, position, and one of four intent stages: Exploration, Implementation, Verification, or Orchestration.Intent labels describe reading and searching, source edits, tests and checks, or bookkeeping and reasoning-related actions.
  • State Parsing: The context-sensitive rule-based labeler uses implementation history and edited files because identical tools can represent exploration or verification in different contexts.For example, reading a test file may be exploratory before patching but verifying afterward.
  • Reference Construction: A PTA merges shared prefixes across k ≥2 passing trajectories while preserving divergent successful strategies as branches, so each root-to-terminal path represents a known-good solution.Equivalent actions can be merged despite surface variation, including different search tools or overlapping file regions.
  • Scoring and Reporting: AGENTLENS combines structural alignment, set coverage, trajectory coherence, and temporal signals into a 0–100 quality score, then reports stage coverage, divergence points, and five waste categories.The waste categories are regression loops, blind retries, redundant steps, unnecessary exploration, and cyclic patterns.

4 Experimental Setup

AgentLens is evaluated on OpenHands trajectories from SWE-bench Verified, with task-specific PTA construction and calibrated, frozen signal weights. The study compares trajectory-alignment baselines and uses micro-averaged AUROC, Youden’s J thresholds, and Kolmogorov–Smirnov significance testing.

  • Compute environment: AgentLens scoring, PTA construction, stratification, and waste analysis run CPU-only on 11 CPU cores and 18GB memory, while trajectory generation uses external model APIs.Post-hoc analysis is separate from trajectory generation and does not require GPU workers.
  • Dataset: 2,614 trajectories span 60 SWE-bench Verified tasks and eight model backends, including 1,389 passes, 1,217 failures, and 8 unrecorded outcomes.The trajectories were generated by the OpenHands coding agent.
  • Calibration and holdout: 278 pilot trajectories across 10 tasks calibrated frozen signal weights w = (0.20, 0.15, 0.30, 0.35), achieving pilot AUROC = 0.755 and pilot F1 = 0.791.Calibration used grid-search weight optimization with step 0.05, a unit-sum constraint, and AUROC maximization; pilot trajectories were excluded from scaled evaluation.

5 Results

AGENTLENS shows that passing SWE-agent trajectories differ substantially in process quality, identifying Lucky Passes and distinguishing efficient atypical solutions from weak successes. Its composite score also separates passing from failing trajectories and changes model comparisons beyond binary pass rates.

  • 5.1 Passing-Trajectory Quality: 10.7% of 1,136 passing trajectories are Lucky, versus 20.2% Ideal and 69.1% Solid under fixed tier thresholds.Binary evaluation labels all passing trajectories identically, while AGENTLENS separates their process quality.
  • 5.1 Passing-Trajectory Quality: Low structural overlap can indicate an efficient-but-atypical valid solution rather than a Lucky Pass when coherence and temporal organization remain strong.AGENTLENS uses the full score to distinguish unconventional valid paths from weak processes that merely have low PTA overlap.
  • 5.2 Lucky Passes: C2 Brute-Force Convergence and C3 together account for 68.0% of Lucky Passes, which are categorized into five mutually exclusive mechanisms.The taxonomy uses trajectory length, verification coverage, waste patterns, implementation coverage, and coherence.
  • 5.2 Lucky Passes: 11.4 steps are wasted per Lucky trajectory with blind retries, compared with 2.7 in Ideal trajectories, a 4.2× increase.This pattern reflects repeated similar actions instead of systematic debugging.
  • 5.3 Outcome Validation: The combined score achieves AUROC = 0.766, accuracy = 72.0%, F1 = 0.723, and KS p = 0.0017 on the scaled 47-task set.Only the combined score significantly separates passing and failing trajectories; no individual signal reaches p < 0.05.
  • 5.3 Outcome Validation: 54.9% of failing trajectories are Partial-fail and 45.1% are Off-track, suggesting roughly half of failures are structurally recoverable.Partial-fail trajectories stay near known-good processes but make localized errors, whereas Off-track trajectories diverge earlier or more completely.

6 Robustness and Ablations

Controlled ablations on a disjoint 278-trajectory pilot set support fusing structural and behavioral signals, while showing that PTA merge count and reference-set choice affect robustness. Removing temporal profile divergence causes the largest AUROC drop, and trajectory selection explains more PTA variance than merge ordering.

  • Signal contribution: Removing any signal lowers AUROC, with the largest drops from temporal profile divergence (−0.037) and trajectory coherence (−0.031).Set coverage drops AUROC by −0.024 and structural alignment by −0.016, supporting fusion of structural and behavioral signals.
  • Merge-count sensitivity: PTA merge count controls a precision–coverage trade-off: small k is compact but restrictive, whereas large k covers more strategies but becomes more permissive.The ablation evaluates fixed merge counts on the disjoint pilot set; k = 2 achieves AUROC = 0.749.
  • Merge-order robustness: 64.1% of PTA score variance comes from trajectory selection, compared with 35.9% from merge ordering.The merge-order test evaluates 10 trajectory combinations and all 6 merge permutations for each combination on one task.
  • Merge-order robustness: 8 of 10 trajectory combinations are fully order-invariant, and the remaining effects are bounded and smaller than those caused by selecting different reference trajectories.Order effects arise when ambiguous exploration prefixes can be merged or branched, making reference-set choice the main source of PTA variation.

7 Conclusion · Appendix · A Limitations, Future Directions, and Positioning

AGENTLENS evaluates SWE-agent trajectories beyond binary pass/fail outcomes by labeling intents and merging passing runs into task-level PTA references. On AGENTLENS-Bench, it identifies 10.7% of passing trajectories as Lucky Passes, with Brute-Force Convergence and Incomplete Implementation comprising 68.0% of them.

  • 7 Conclusion: AGENTLENS converts raw SWE-agent trajectories into intent-labeled state sequences and task-level PTA references.The framework merges passing trajectories for the same task into PTA references.
  • 7 Conclusion: AGENTLENS assesses whether agents reached correct patches through coherent, low-waste processes.This process-aware view distinguishes direct solutions, valid alternative paths, and weak successes.
  • 7 Conclusion: 2,614 OpenHands trajectories from 60 SWE-bench Verified tasks were evaluated.The AGENTLENS-Bench subset comprises 47 PTA-eligible tasks.
  • 7 Conclusion: 10.7% of passing trajectories on the 47 PTA-eligible tasks are Lucky Passes.These are correct outcomes reached through weak processes.
  • 7 Conclusion: 68.0% of Lucky Passes are accounted for by Brute-Force Convergence and Incomplete Implementation.The weak successes follow recurring patterns rather than isolated accidents.
  • 7 Conclusion: AGENTLENS-Bench is defined over the 47 PTA-eligible tasks used to analyze successful-run variation.The benchmark is the task subset on which the Lucky Pass result is reported.
  • Appendix: Appendix F.1 contains the statistical tests for the Lucky Pass taxonomy.The supplied appendix passage is a table-of-contents entry for this analysis.

A.1 Limitations … A.4 Extended Related Work and Positioning

AgentLens is positioned as a post-hoc, process-aware diagnostic framework for OpenHands-style SWE-agent trajectories, with extensions planned for training and broader evaluation. It complements functional correctness, security review, and human judgment rather than replacing them.

  • A.1 Limitations: AgentLens evaluates OpenHands-style coding-agent traces on SWE-bench Verified tasks using task-level PTA references and reproducible scoring.Its controlled scope supports process-quality analysis, while broader scaffolds mainly require trace-format adapters or ATIF conversion.
  • A.1 Limitations: Extending AgentLens to other agent scaffolds primarily requires trace-format adapters or ATIF conversion, not changes to the core scoring pipeline.
  • A.2 Future Directions: Quality scores can provide dense reinforcement-learning rewards that encourage correct patches alongside coherent, low-waste solution processes.The paper cites Wei et al. (2025) in this direction.
  • A.2 Future Directions: PTA references can organize training curricula by process quality, divergence type, or recoverability rather than final outcome alone.
  • A.3 Broader Impacts: AgentLens exposes brittle successes, recoverable failures, and wasteful processes to support safer deployment decisions, cleaner demonstrations, and diagnosis of hidden weaknesses.It may also improve benchmark incentives by rewarding coherent solutions over excessive retries or accidental success.
  • A.3 Broader Impacts: Process scores are complementary diagnostics, not replacements for functional correctness, security review, or human judgment.Incomplete tests or references can allow a high-scoring trajectory to contain an incorrect or insecure patch, so AgentLens is not an automatic deployment-approval mechanism.
  • A.4 Extended Related Work and Positioning: Table 4 compares AgentLens-Bench with related SWE-agent trajectory collections, while Table 5 compares AgentLens with related evaluation frameworks.

B Additional Method Details … B.5 Scoring Signal Examples

Appendix B supplies reproducibility-level details for AgentLens, covering tool mapping, cross-agent state equivalence, context-sensitive intent labeling, four scoring signals, inefficiency analysis, and worked examples. It specifies how trajectory quality is computed and illustrates how principled and chaotic behaviors receive different scores.

  • B Additional Method Details: Appendix B organizes reproducibility details around the AgentLens pipeline, including tool registration, state equivalence, intent labeling, scoring definitions, and worked signal examples.The appendix also describes planned practitioner-facing web interface material.
  • B.1 Tool Registry: Tool-registry mappings provide default intent-stage hints and comparison strategies, which the context-sensitive labeler may override using trajectory history, edited-file state, and command semantics.The abbreviated registry used in experiments is shown in Table 6.
  • B.2 State Equivalence Engine: The equivalence engine merges heterogeneous agent states through an ordered confidence-weighted cascade spanning exact hashes, AST scopes, overlapping line ranges, and semantically grouped terminal commands.An optional LLM fallback was disabled for reproducibility, and the first matching rule determines the decision and confidence.
  • B.3 Intent-stage Labeling Flow: The seven-rule intent labeler combines fixed tool-based stages with context-sensitive rules that disambiguate terminal commands using edits, test-file targets, and prior modifications.Naively assigning one stage to all terminal commands causes 70–80% of PTA states to collapse into one category.
  • B.4 Scoring Details: AgentLens scores trajectories using structural alignment, coherence, temporal profile divergence, and set coverage, while separately localizing five categories of behavioral waste.Waste categories are regression loops, blind retries, redundant steps, unnecessary exploration, and cyclic patterns.
  • B.4 Scoring Details: The combined score assigns weights 0.20, 0.15, 0.30, and 0.35 to structural alignment, set coverage, coherence, and temporal profile, respectively.Behavioral signals receive 65% of the weight as an empirical outcome of AUROC-maximizing grid search, not a prior design choice.
  • B.5 Scoring Signal Examples: Worked examples show principled versus chaotic trajectories scoring Φcoh = 1.00 versus 0.51, structural F1 = 0.86 versus 0.27, and Φcov = 0.80 versus 0.20.The temporal example reports principled-segment JSD values of 0.02, 0.03, and 0.01, yielding Φtemp = 0.92; the chaotic trajectory reverses expected phase timing.

C Additional Experimental Results … D.4 Model-specific Lucky Pass Signatures

The appendix supplies supporting analyses for AgentLens, showing that combined process signals distinguish behavioral profiles, validate the labeler and score, and provide interpretable diagnostics beyond outcome evaluation. Extended Lucky Pass analysis further identifies five categories and model-specific signatures, including verification skipping, brute-force persistence, excessive exploration, and incomplete implementations.

  • C.1 Behavioral Profiles: Four complementary signals explain why combined scoring is more informative than any single measure, while efficient-but-atypical trajectories comprise roughly 53% of passing trajectories and are classified as Solid.These trajectories combine high coherence and temporal alignment with low structural F1, which structural alignment alone would misclassify.
  • C.2 Pass/Fail Waste Breakdown: Passing trajectories can contain more regression loops, blind retries, and redundant steps than failing ones because thorough Ideal trajectories are longer; blind-retry severity is the key Lucky-pass discriminator.Waste detections exclude patterns already present in the merged PTA, and Lucky trajectories waste substantially more steps when retries occur.
  • C.4 Failure-mode Gallery: Representative failure-mode timelines illustrate regression loops, blind-retry clusters, temporal disorder, intent confusion, unnecessary exploration, and cyclic behavior, while quantitative claims use the full 1,815-trajectory set.The appendix also reports an Ideal-versus-Lucky waste comparison identifying blind-retry severity as a fingerprint.
  • C.5 Heuristic Labeler Validation: The labeler reaches Fleiss’ κ = 0.933 and 96.0% raw agreement overall, with Orchestration at κ = 1.000, the E-versus-V boundary at κ = 0.939, and Implementation at κ = 0.713.Against annotator consensus, the deterministic heuristic achieves 93.8% accuracy and macro-F1 = 0.933; eight disagreements center on post-implementation read_file calls.
  • C.6 Score Distributions: 46.4 is the fixed Youden-J threshold: passing scores concentrate in 50–75 with a tail to 95, failing scores peak around 30–40, and the distributions differ at p = 0.0017.The overlap around 35–55 contains both Solid passing and Partial-fail trajectories.
  • C.7 Baseline Results: Individual matching achieves the highest AUROC, but AGENTLENS’s PTA provides structured, branch-aware diagnostics by localizing divergence and identifying wasted exploration.On astropy__astropy-13236, the PTA localizes a divergence to step 8 and detects an edit outside both valid solution branches.
  • C.8 Model-comparison Visualizations: The Lucky-rate range is 46×, from Opus 4.5 (0.5%) to GPT-4.1 (23.2%), while pass rate and mean quality score show substantial rank disagreement across eight model configurations.GPT-4o ranks lower by pass rate but higher by quality, whereas Opus 4.6 ranks high by pass rate but mid-range by quality.

D.5 Task-level Concentration … E.2 Merge-count Sensitivity

Lucky Passes concentrate in a small set of tasks, arise through distinct model-task failure modes, and expose why binary pass rates miss important process differences. The ablations support four scoring signals and select k = 5 merged trajectories as a balance between solution coverage and scoring precision.

  • D.5 Task-level Concentration: 63.1% of Lucky Passes come from the top 10 tasks, with psf__requests-1724 alone producing 16; task concentration is strong (Cramér’s V = 0.714).30 of 47 PTA-eligible tasks produce at least one Lucky Pass; χ2(116) = 248.45, p < 0.0001.
  • D.6 Case Study: One Task, Five Behavioral Profiles: On psf__requests-1724, quality scores range from 22 to 88, separating one Ideal, one Solid, and three Lucky trajectories that binary evaluation treats equivalently.GPT-4o follows targeted implementation, regression testing, relevant tests, and diff review; the Lucky trajectories fail through unverified editing, incomplete implementation, or 34 states of repetitive exploration.
  • D.7 Additional Lucky Pass Cases; D.7.1 C2: GPT-5.2-Codex on matplotlib__matplotlib-22719: 12 wasted steps and 67% waste severity characterize the matplotlib C2 trajectory, which reaches a correct edit after a wrong-directory detour and redundant reads.The 18-state trajectory costs 235,346 tokens, versus 9 ground-truth states, with only 2 of 19 alignment steps matching.
  • D.7.2 C3: GPT-5.3-Codex on psf__requests-1724: 0% of ground-truth implementation steps are covered by the psf__requests-1724 C3 trajectory, which writes a reproducer instead of editing source code.Despite only 6 states and 64,314 tokens, it has no verification or orchestration and is classified as incomplete_implementation at high severity.
  • D.7.3 C4: GPT-4.1 on django__django-11066: 2.62M tokens and 57 post-verification exploration steps characterize the Django C4 trajectory, while all five C4 instances come from GPT-4.1.The 100-state trajectory makes a one-line fix, costs 77× the C1 average, and has only 4 alignment steps matching.
  • D.7.4 C5: Opus 4.6 on pylint-dev__pylint-4970; D.7.5 C5: GPT-5.2-Codex on sphinx-doc__sphinx-10323: C5 includes both a genuinely alternative fix and a complete E→I→V→O lifecycle, showing that low ground-truth coverage can reflect structural divergence rather than poor solution quality.The pylint trajectory has 14.3% coverage with two implementation-aligned steps, while the Sphinx trajectory has quality score 37 and 11.3% coverage despite reproducing, fixing, and re-verifying the bug.
  • D.8 Verification Gap: 94.3% of 122 Lucky Passes have missing verification, and the C1-to-C5 verification gradient identifies different interventions for Opus 4.6, GPT-4.1, Codex variants, and alternative approaches.Lucky Passes combine agent shortcomings, test-suite gaps, and task characteristics; verification is the most directly actionable factor.
  • E Ablation Details; E.1 Signal Contribution; E.2 Merge-count Sensitivity: −0.037 and −0.031 are the largest AUROC drops from removing behavioral signals, while k = 5 achieves AUROC 0.777 versus 0.749 for k = 2.All four signals contribute meaningfully; k = 5 broadens solution-space coverage, while reduced task coverage from 41 to 31 resamples reflects PTA-size scoring limits.

E.3 Merge-order Robustness

The study tests whether the incremental order used to merge passing trajectories changes PTA structure and downstream quality scores. In a pilot task, merge ordering explained less variance than trajectory selection, and most trajectory combinations were order-invariant.

  • Motivation: Merge-order dependence could make PTA-based scoring fragile because identical passing trajectories might yield different quality scores under different processing orders.The concern arises from constructing the PTA incrementally by merging trajectories one at a time.
  • Experimental setup: 60 total scoring runs evaluated 10 random trajectory combinations, each tested across all 6 merge permutations, on astropy__astropy-12907 with k = 4.The experiment used one pilot task and reported per-combination results in Table 20.
  • Results: 64.1% of total variance came from trajectory selection, versus 35.9% from merge ordering.This variance decomposition quantifies the relative contribution of selecting trajectories and choosing their merge order.
  • Results: Eight of ten trajectory combinations produced zero within-combination variance and were fully order-invariant.Only three combinations had nonzero ordering variance in the reported results.

F Token Cost and Statistical Tests · F.1 Statistical Tests for Lucky Pass Taxonomy

Lucky Pass categories differ sharply in token cost despite identical binary outcomes: C4 averages 40× C1, while C2 costs 25× C1. Across 122 Lucky Passes, excess expenditure totals 54M tokens, and statistical tests show model choice and task characteristics predict the category produced.

  • F Token Cost and Statistical Tests: 40× separates average C4 from C1 token cost for the same binary evaluation outcome.C1 averages 34K tokens per trajectory, while C4 includes a 2.62M-token one-line-fix trajectory.
  • F Token Cost and Statistical Tests: 25× makes C2 more expensive than C1 because repeated retries consume many additional steps.C2 is the Brute-Force Convergence category, whereas C1 is Minimal & Unverified.
  • F Token Cost and Statistical Tests: 54M tokens, or 93% of aggregate expenditure, represent excess cost beyond solving all 122 Lucky Passes at C1 efficiency.Aggregate expenditure is approximately 58.3M tokens versus 4.1M tokens at 34K per trajectory.
  • F Token Cost and Statistical Tests: 14× is the approximate API-cost premium for 122 Lucky Passes versus solving them all at C1 efficiency.The excess is attributable to C2 and C4 patterns, which are more expensive than C1.
  • F Token Cost and Statistical Tests: Binary pass/fail scores hide token efficiency, even though C2 costs 25× more per trajectory than C1 and changes deployment economics.Both categories receive identical binary scores despite their different computational costs.
  • F.1 Statistical Tests for Lucky Pass Taxonomy: Highly significant associations with large to very large effect sizes show that model choice predicts Lucky Pass category, not merely its occurrence.The statistical tests also identify task characteristics as predictors of which category dominates.
  • F.1 Statistical Tests for Lucky Pass Taxonomy: Task characteristics strongly predict which Lucky Pass category dominates, complementing the model-choice association.The passage characterizes the corresponding effect sizes as large to very large.
  • F.1 Statistical Tests for Lucky Pass Taxonomy: 87% verification coverage is reported for C5, while C1 has zero by definition and C2 and C3 range from 40–60%.These per-category verification rates support the causal structure described in Section D.8.
Loading 2605.12925v3…