PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Test-Time Compute for Agentic Coding

Joongwon Kim, Wannan Yang, Kelvin Niu, Hongming Zhang, Yun Zhu, Eryk Helenowski, Ruan Silva, Zhengxing Chen, Srinivasan Iyer, Manzil Zaheer, Daniel Fried, Hannaneh Hajishirzi, Sanjeev Arora, Gabriel Synnaeve, Ruslan Salakhutdinov, Anirudh Goyal

arXiv:2604.16529v1cs.SEcs.AIcs.CLcs.LG

TL;DR

Long-horizon coding agents produce noisy trajectories that existing test-time scaling methods cannot easily compare or reuse. The paper introduces structured rollout summaries with RTV and PDR, consistently improving frontier agents across two coding benchmarks, including Claude-4.5-Opus’s 70.9% to 77.6% gain on SWE-Bench Verified.

  • Problem

    Long-horizon coding trajectories are noisy and difficult for standard test-time scaling methods to compare or reuse directly.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    The framework converts rollouts into compact structured summaries used by RTV for parallel aggregation and agentic PDR for sequential refinement.

    Source: §1 · p. 2, §2.2 · p. 3, §5 · p. 15

  • Results

    Across frontier LLMs and SWE-Bench Verified and Terminal-Bench v2.0, the framework consistently improves over single-attempt baselines, including Claude-4.5-Opus from 70.9% to 77.6% on SWE-Bench Verified.

    Source: §1 · p. 2, §5 · p. 15

  • Takeaways & Limitations

    For long-horizon agents, test-time scaling works when prior experience is represented so it supports both selection and reuse.

    Source: §5 · p. 15

    The framework uses fresh rollouts in newly initialized environments rather than retaining persistent workspace artifacts across attempts.

    Source: §5 · p. 15

Abstract

from arXiv · show

Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.

1 Introduction

Agentic coding makes test-time scaling difficult because long, noisy rollout trajectories cannot be directly compared or reused as compact outputs can. The proposed solution represents prior experience with structured summaries, enabling parallel selection and sequential refinement through RTV and agentic PDR.

  • Long-horizon coding produces noisy, verbose trajectories of environment interactions that are difficult to compare or reuse, so standard test-time scaling methods do not transfer cleanly.Unlike bounded completions, attempts include file operations, code edits, logs, commands, and responses to intermediate failures.Source: §1 · p. 2
  • Compact structured summaries capture hypotheses, decisions, progress, and failure modes while omitting low-value trace details, making representation the interface for selecting and reusing prior attempts.The paper identifies prior-agent-experience representation as a central bottleneck in inference-time scaling for long-horizon coding.Source: §1 · p. 2
  • RTV recursively selects strong attempts from rollout populations without ground-truth outcomes, while agentic PDR conditions new rollouts on summaries distilled from prior attempts.Together, these methods support parallel aggregation and sequential refinement within one representation-centric framework.Source: §1 · p. 2
  • Structured summaries outperform raw trajectories for parallel comparison, recursive small-group selection beats flatter large-group selection, and refinement improves when using multiple high-quality prior summaries.The quality of refinement context is strongly related to subsequent rollout performance.Source: §1 · p. 2

2 Methodology

The methodology scales agentic coding by representing multi-step rollout trajectories as compact summaries, then selecting and reusing those representations through parallel and sequential inference-time scaling. Its unified pipeline combines Recursive Tournament Voting with Parallel-Distill-Refine to select, refine, and reselect candidate rollouts.

2.1 Problem Formulation

  • Each rollout is an interleaved trajectory of model thoughts and bash commands, with terminal observations appended to the accumulated context after every action.Observations include expected outputs and unexpected error messages, enabling multi-step interaction with an external bash environment.Source: §2.1 · p. 3
  • Rollout-level scaling uses parallel independent executions with selection and sequential fresh executions conditioned on information extracted from earlier attempts.Rollouts are the primary scaling unit because token-level scaling is difficult to interpret consistently across steps.Source: §2.1 · p. 3

2.2 Rollout Summaries as Reusable Representations

  • Compact summaries provide the common interface for comparing parallel rollouts and reusing prior experience sequentially without replaying full interaction histories.They convert noisy trajectories containing useful signal and low-signal detail into bounded, reusable representations.Source: §2.2 · p. 3

2.3 Parallel Selection via Recursive Tournament Voting

  • RTV recursively compares compact rollout summaries in small groups, using repeated voting rounds to reduce N candidates until one remains without ground-truth outcomes.It executes N rollouts, summarizes each, selects one per group, and repeats the tournament-like reduction.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

2.4 Sequential reuse via Parallel-Distill-Refine

  • PDR summarizes prior rollouts, samples K summaries as refinement context, and runs each next-iteration rollout in a freshly initialized environment conditioned on that experience.The final-iteration performance is estimated by averaging the scores of its N rollouts rather than executing only one rollout.Source: §2.4 · pp. 4–5

2.5 A unified pipeline: selection, then reuse, then selection

  • The unified PDR + RTV pipeline executes N rollouts, selects K summaries with RTV, conditions N fresh rollouts on them, and applies final RTV to return one rollout.Using K < N narrows the search space, while K > 1 preserves diversity for cross-pollinating solution approaches.Source: §2.5 · p. 5
  • Structured summaries outperform full rollout traces for parallel aggregation across Gemini-3-Flash and Claude-4.5-Sonnet on SWE-Bench Verified and Terminal-Bench v2.0.Summaries retain useful diagnoses, partial fixes, and failure modes while discarding repeated terminal output and dead-end exploration.Source: §2.2 · p. 3, Figure 4 · p. 6

3 Results and Analyses

Across two agentic coding benchmarks, PDR+RTV consistently improves frontier-model performance by combining structured-summary selection with sequential refinement. Analyses show that compact representations, recursive small-group voting, and stronger multi-rollout refinement contexts are central to these gains.

3.1 Experimental Setup

  • Across SWE-Bench Verified and Terminal-Bench v2.0, experiments use five frontier models, N = 16 rollouts, T = 2, K = 4, G = 2, and V = 8.SWE-Bench uses its full test set, while Terminal-Bench evaluates 88 of 89 available tasks.Source: §3.1 · p. 6

3.2.1 Parallel Aggregation Ablations

  • Compact structured summaries consistently outperform full rollout traces for selecting long-horizon agentic rollouts because they preserve decisive information while discarding low-signal detail.The advantage is especially decisive in later tournament rounds, when surviving candidates require more nuanced comparison.Source: §3.2.1 · p. 6
  • Pairwise recursive comparisons with vote aggregation outperform flat many-candidate comparisons, with G = 2 strongest and diminishing returns beginning around V = 8.These findings motivate the default RTV configuration G = 2 and V = 8.Source: §3.2.1 · p. 7, Figure 5 · p. 7

3.2.2 Sequential Refinement Ablations

  • Sequential refinement improves when conditioning on multiple prior rollouts, and improves further when RTV selects those rollouts rather than sampling them randomly.For Gemini-3.1-Pro, pass@1 rises from 72.69% to 73.75% with one prior rollout, 76.94% with random-K, and 79.25% with select-K.Source: §3.2.2 · p. 8, Table 1 · p. 8
  • Iteration-1 success rates increase monotonically with refinement-context quality, reaching 97–99% for contexts with 4/4 successful rollouts.RTV produces more 4/4 passing contexts than random selection: 62 versus 52.8 tasks for Claude-4.5-Sonnet and 71 versus 58.5 for Gemini-3.1-Pro.Source: §3.2.2 · p. 9, Table 2 · p. 9

3.3 Main Results

  • 8.39 points: on SWE-Bench Verified, PDR+RTV raises GPT-5-0825 from 61.41% to 69.80%, while all five evaluated models improve over Iter 0.The gains combine improved rollout quality after Select-K refinement with an additional final RTV selection boost.Source: §3.3 · p. 9, Table 3 · p. 9
  • 16.20 points: PDR+RTV raises Claude-4.5-Sonnet from 40.62% to 56.82% on Terminal-Bench v2.0, with final RTV adding selection gains after refinement.Across all five models, final performance improves over Iter 0 on Terminal-Bench v2.0, with larger absolute gains than on SWE-Bench Verified.Source: §3.3 · p. 9, Table 3 · p. 9
  • Refinement substantially shortens agent trajectories while improving pass@1; Claude-4.5-Opus drops from 41.23 to 14.31 steps on SWE-Bench Verified and from 24.43 to 12.14 on Terminal-Bench v2.0.Failing rollouts consume more steps than passing rollouts within each iteration.Source: §3.3 · p. 10, Table 4 · p. 10

4 Analysis

The analysis shows that sequential refinement improves rollout populations when selected context is strong, while recursive tournament voting continues to exploit diversity after refinement. It also identifies reduced parallel gains and judge-selection accuracy as important limitations.

4.1 Sequential Refinement Dynamics

  • Across both benchmarks, iteration-1 pass-count transition matrices shift upward, indicating that sequential refinement improves pass rates for many tasks.Mass above the diagonal denotes improvements, while mass below denotes regressions.Source: §4.1 · p. 10, Figure 8 · p. 11
  • 0.1%, 33.4%, 55.5%, 85.4%, and 99.2% are Claude-4.5-Opus’s iteration-1 pass@1 scores when 0, 1, 2, 3, or 4 selected rollouts pass.The same increasing pattern holds across all four models and both benchmarks, showing that refinement-context quality predicts next-iteration performance.Source: Table 5 · p. 11, §4.1 · p. 11
  • 209/500 to 350/500 (+141) tasks reach 16/16 passing rollouts for Claude-4.5-Opus on SWE-Bench Verified, while 0/16 tasks rise 73/500 to 94/500 (+21).The resulting distribution is bimodal, but successful-task increases outweigh failing-task increases, improving average pass@1.Source: Figure 9 · p. 12, §4.1 · p. 12

4.2 Parallel Aggregation Dynamics

  • RTV’s average pass@1 generally rises across rounds, whereas pass@N decreases or stays constant as tournament selection can remove successful or failing candidates unevenly.Iteration-1 pass@N starts lower than iteration 0 but converges to a higher final pass@1 because its remaining tasks are more favorable for RTV.Source: §4.2 · pp. 13–14
  • Parallel aggregation remains valuable after refinement because iteration-1 rollout populations retain useful intra-task diversity for recursive selection.The second RTV instance still improves performance, although less than the first because refinement reduces mixed pass/fail task outcomes.Source: Figure 11 · p. 13, Figure 12 · p. 13, §4.2 · p. 14
  • 76.16% to 76.60% (+0.44%) is Gemini-3.1-Pro’s iteration-1-to-final pass@1 improvement, illustrating limited gains when judge selection accuracy is low.Table 6 comparisons are not controlled head-to-head evaluations because judges operate on different rollout pools and summaries; judge training or specialization could improve RTV.Source: §4.2 · pp. 14–15, Table 6 · p. 14

5 Conclusion

The paper argues that inference-time scaling for long-horizon agentic coding depends on compact representations of prior experience that enable reliable comparison, selection, and reuse. Its framework combines structured rollout summaries with parallel RTV and sequential PDR, improving frontier-agent performance while motivating persistent artifact reuse as future work.

Core contribution

  • Structured rollout summaries address the central bottleneck of representing prior experience for reliable comparison and reuse in long-horizon agentic coding.Each long interactive trajectory is converted into a compact, structured summary rather than reused as an unbounded trace.Source: §5 · p. 15
  • RTV selects successful rollouts through recursive small-group comparisons, while agentic PDR conditions fresh rollouts on summaries distilled from prior attempts.Combining both operators balances exploitation through refinement with exploration through multiple parallel rollouts.Source: §5 · p. 15

Empirical findings

  • The framework consistently improves single-attempt baselines across frontier LLMs and SWE-Bench Verified and Terminal-Bench v2.0.The reported gains span Claude-4.5-Opus, Gemini-3.1-Pro, Claude-4.5-Sonnet, Gemini-3-Flash, and GPT-5-0825.Source: §5 · p. 15
  • Analyses attribute the gains to representation and reuse: summaries outperform raw trajectories, higher-quality contexts improve later rollouts, final RTV exploits residual diversity, and PDR uses about 50% fewer steps while increasing success rates.These findings indicate that the benefit is not explained solely by allocating more compute.Source: §5 · p. 15

Future work

  • Future work would replace summary-only reuse with persistent workspace artifacts such as notes, patches, tests, scripts, and tools, while addressing stale or low-value information.The current setting provides summaries to fresh rollouts in newly initialized environments; persistent artifact maintenance remains an open question.Source: §5 · p. 15

Limitations

  • Gemini-3.1-Pro’s abnormally lower success rate as a judge may be related to API failure rates during final RTV experiments caused by infrastructure issues.This limitation concerns the final RTV experiments and reflects reported infrastructure failures.Source: §5 · p. 15

A Rollout statistics

Rollout statistics from the main experiments compare five models across SWE-Bench Verified and Terminal-Bench v2.0, measuring pass@1, pass@16, and task-level rollout consistency. Gemini-3.1-Pro performs best across both benchmarks, while model rankings vary by benchmark and GPT-5-0825 performs worst.

Setup

Metrics

  • Each iteration measures average pass@1, pass@16 with N=16, and the number of tasks containing mixed passing and failing rollouts.For SWE-Bench Verified, the experiments use the bash-only mini-SWE-agent harness; for Terminal-Bench v2.0, they use the Terminus 1 harness.Source: A Rollout statistics · p. 18

Results

  • Gemini-3.1-Pro performs best across both benchmarks, closely followed by Claude-4.5-Opus, while GPT-5-0825 performs worst.The comparisons are reported in the main-experiment rollout-statistics tables for SWE-Bench Verified and Terminal-Bench v2.0.Source: Table 7 · p. 18, Table 8 · p. 18, A Rollout statistics · pp. 18–19
  • Gemini-3-Flash outperforms Claude-4.5-Sonnet on SWE-Bench Verified.For SWE-Bench Verified, the evaluation uses the bash-only mini-SWE-agent harness.Source: A Rollout statistics · p. 18

B Model comparisons

The study compares five frontier models using pairwise success rates across SWE-Bench Verified and Terminal-Bench v2.0, finding Gemini-3.1-Pro most competitive overall, followed closely by Claude-4.5-Opus.

B Model comparisons

  • Gemini-3-Flash slightly outperforms Claude-4.5-Sonnet on SWE-Bench Verified but underperforms it on Terminal-Bench v2.0.GPT-5-0825 is the least competitive among the compared models.Source: B Model comparisons · p. 19
  • Gemini-3-Flash defeats Gemini-3.1-Pro in only 10/500 SWE-Bench Verified tasks and 1/88 Terminal-Bench v2.0 tasks.Smaller models generally struggle more to outperform larger models in the same family; the Claude family shows a similar but less pronounced pattern.Source: B Model comparisons · p. 19
  • Gemini-3.1-Pro has the highest overall pairwise win rates across both benchmarks, with Claude-4.5-Opus close behind.Pairwise comparisons count tasks where one model succeeds in at least one of N=16 iteration-0 rollouts while the other fails all rollouts.Source: Figure 13 · p. 19, B Model comparisons · p. 19

C Sequential refinement dynamics

This section presents additional results from the sequential refinement analyses of the PDR+RTV main experiments, following the analyses in Section 4.1.

C Sequential refinement dynamics

C.1 Context quality analysis

Select-K (RTV) refinement produces more contexts containing only passing rollouts than random-K refinement, improving average pass@1 for subsequent rollouts. A task-level analysis further stratifies refinement contexts by passing-rollout count and examines iteration-1 pass-rate distributions across two models.

Context quality analysis

  • Select-K (RTV) refinement yields more all-passing contexts and higher average pass@1 than random-K refinement.Table 2 compares the two refinement strategies after stratifying tasks by the number of passing rollouts in their context windows.Source: §C.1 · p. 19
  • For 100 randomly sampled SWE-Bench Verified tasks, the analysis stratifies contexts by passing-rollout count and compares bucketed iteration-1 pass-rate distributions for Claude-4.5-Sonnet and Gemini-3.1-Pro.Figure 14 measures the impact of the average performance of selected K rollouts in the PDR refinement context on iteration-1 rollout pass rates.Source: §C.1 · pp. 19–20, Figure 14 · p. 20

D New solution discoveries

PDR+RTV enables frontier coding models to discover solutions after all 16 initial rollouts fail, with new discoveries concentrated more heavily in Terminal-Bench v2.0 than SWE-Bench Verified. These discoveries also reveal substantial cross-model complementarity, while some tasks remain uniquely solvable after refinement.

New solution discoveries

Cross-model complementarity

  • All Terminal-Bench v2.0 tasks self-improved by Claude-4.5-Opus were solved by Gemini-3.1-Pro initially, while all but one self-improved by Gemini-3.1-Pro were solved by Claude-4.5-Opus or Claude-4.5-Sonnet.This indicates that iteration-0 successes from other models often provide useful complementary evidence for refinement.Source: D New solution discoveries · p. 20, Table 12 · p. 21
  • Claude-4.5-Sonnet, Gemini-3-Flash, and GPT-5-0825 self-improve on mcmc-sampling-stan, which Gemini-3.1-Pro alone had solved initially.Gemini-3.1-Pro also solves large-scale-text-editing in iteration 1 although no model solves it initially.Source: D New solution discoveries · p. 21

E PDR + RTV qualitative examples

This section presents qualitative examples of PDR+RTV rollouts using structured summaries to combine approaches, reconcile findings, and avoid prior dead ends across SWE-Bench Verified and Terminal-Bench v2.0. It also examines a SQLite-recovery rollout, detailing its reasoning and implementation steps without establishing task completion.

  • PDR+RTV agents synthesize prior rollout summaries, reconcile disagreements, and reuse successful patterns to avoid repeating extensive exploration or dead ends.The examples show agents consolidating consensus, selecting among divergent approaches, and directly applying working fixes or scripts.Source: E PDR + RTV qualitative examples · pp. 21, 22, 23, 25, 26
  • Claude-4.5-Opus solves django_django-13033 by combining four analyses, applying both identified substitutions, reusing a working sed command, and preinstalling dependencies after prior failures.The summaries identify the root cause at line 730, the pieces[-1] fix, and the dependencies asgiref, pytz, and sqlparse.Source: E PDR + RTV qualitative examples · p. 22, Figure 15 · p. 23
  • Gemini-3.1-Pro’s sympy rollout consolidates consensus on guarding _split_gcd, considers divergent alternatives, reproduces a prior TypeError, and adopts the necessary two-file fix.The rollout uses prior findings about the empty-tuple IndexError and sqrt_biquadratic_denest TypeError to justify its final strategy.Source: E PDR + RTV qualitative examples · p. 23, Figure 16 · p. 24
  • The Terminal-Bench examples select correct interpretations from conflicting attempts: separate SPARQL criteria in sparql-university and SQLite serial-type parsing with a direct B-tree recovery script.The agents use successful prior insights to avoid incorrect combined filters and dead-end recovery approaches.Source: E PDR + RTV qualitative examples · pp. 25–26, Figure 17 · p. 26

Rollout implementation

  • The rollout constructs a Python script to parse SQLite B-tree leaf pages, decode varints and serial types, recover matching rows, and write /app/recover.json.The implementation reads the corrupted database, parses records, filters words beginning with “testword,” and saves word-value pairs.Source: E PDR + RTV qualitative examples · p. 28

Validation

  • The agent runs the recovery script, inspects the output file, and checks row count, required fields, and JSON validity.These commands provide explicit post-execution format checks, but the excerpt does not report their resulting values.Source: E PDR + RTV qualitative examples · p. 28

Outcome status

  • The excerpt is a rollout executed by Claude-4.5-Sonnet on the sqlite-db-truncate task, and its state is marked is_task_complete: false.Thus, the example documents an in-progress trajectory rather than a confirmed successful completion.Source: E PDR + RTV qualitative examples · p. 28, Figure 18 · p. 29

F Example rollout trajectories

This section presents example rollout trajectories for SWE-Bench Verified and Terminal-Bench v2.0, covering both initial and refined traces. Because full trajectories are too long to fit, only subsets are shown.

F.1 Initial rollout trajectories

The section presents initial rollout trajectories for SWE-Bench Verified and Terminal-Bench v2.0, illustrating how an agent investigates, reproduces, diagnoses, and fixes a coding issue. In the SWE-Bench example, the agent resolves a SymPy block-multiplication failure by preserving ZeroMatrix structure.

Initial rollout examples

  • The rollouts include examples from both SWE-Bench Verified and Terminal-Bench v2.0, with Figures 19 and 20 showing representative trajectories.Figure 19 shows a subset of a successful Gemini-3.1-Pro rollout on SWE-Bench Verified.Source: §F.1 · p. 29, Figure 19 · p. 34

SWE-Bench trajectory

  • The SWE-Bench issue concerns repeated multiplication of a BlockMatrix containing ZeroMatrix blocks, which fails with an AttributeError involving a Zero object lacking cols.Single multiplication succeeds, whereas repeated multiplication triggers the failure.Source: §F.1 · pp. 29–30
  • After installing missing mpmath dependencies, the agent reproduces successful triple multiplication and confirms the resulting block has type ZeroMatrix.The initial reproduction failed because mpmath was unavailable; rerunning after installation produced Matrix([ [a**3, 0], [ 0, 0]]) and ZeroMatrix output.Source: §F.1 · pp. 31–33
  • The agent identifies the root cause in MatAdd postprocessing: an artificial scalar zero converts ZeroMatrix into Zero, and removing that scalar preserves matrix identity structure.The change returns the matrix addition directly when no non-matrix terms exist, preventing the erroneous scalar element.Source: §F.1 · p. 33

Terminal-Bench trajectory

  • A second initial trajectory addresses creating a self-signed TLS certificate with OpenSSL for an internal development server.The supplied passage introduces the Terminal-Bench task requirements but does not provide the subsequent trajectory outcome.Source: §F.1 · p. 34

1. Create a directory at ‘/app/ssl/‘ to store all files

The task creates a self-signed SSL certificate setup under `/app/ssl/`, including a protected RSA key, certificate, combined PEM, and verification record. A Python checker validates the files and certificate details, with the completed rollout showing all required files present and correct key permissions.

  • A 2048-bit RSA key and a 365-day self-signed certificate are generated for Organization Name “DevOps Team” and Common Name “dev-internal.company.local”.The key is saved as `/app/ssl/server.key`, and the certificate as `/app/ssl/server.crt`.Source: §1 · pp. 34–35
  • The certificate and private key are concatenated into `/app/ssl/server.pem`, while `verification.txt` records the subject, validity dates, and SHA-256 fingerprint.OpenSSL commands create the combined PEM and extract the requested certificate metadata into the verification file.Source: §1 · pp. 35–36
  • The Python checker verifies certificate and key existence, extracts the Common Name and expiration date in YYYY-MM-DD format, loads the certificate with ssl, and prints a success message when checks pass.The script is created at `/app/check_cert.py` and uses both `ssl` and `subprocess` as specified.Source: §1 · pp. 36–37
  • The completed setup contains server.crt, server.key, server.pem, and verification.txt, with server.key protected by permissions 600.The directory listing reports all four files; server.key is shown as `-rw-------` while the other files are readable.Source: §1 · pp. 34, 35, 37

F.2 Refined rollout trajectories

This section presents refined rollout trajectories in which later attempts overcome systematic failures in initial rollouts on Terminal-Bench v2.0, including gpt2-codegolf and large-scale-text-editing tasks. One example shows recovery from Vim macro output failures through silent substitutions, small-scale testing, byte-for-byte verification, and subsequent full-scale execution.

  • Claude-4.5-Opus succeeded on gpt2-codegolf after all iteration-0 rollouts failed.This example is shown in Figure 21.Source: §F.2 · p. 38, Figure 21 · p. 42
  • Gemini-3.1-Pro succeeded on large-scale-text-editing after all iteration-0 rollouts failed across models.This example is identified as Figure 22 in the supplied passage.Source: §F.2 · p. 38
  • In the gpt2-codegolf trajectory, refinement addressed incorrect checkpoint-weight interpretation and produced the expected output, passing the test.The final program was 3467 bytes, compiled successfully, and generated “WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.”Source: §F.2 · pp. 38, 41

Transformation

  • The refinement transforms each row from three comma-separated fields with variable spaces into reversed, semicolon-separated uppercase fields ending in ;OK.The expected format is illustrated as GAMMA1;BETA1;ALPHA1;OK for an input such as alpha1, BeTa1, GaMmA1.Source: §F.2 · p. 45

Refinement

  • The agent replaced noisy substitution macros with :silent commands to prevent terminal flooding during repeated Vim execution.The earlier approach produced excessive output because substitution echoes were not redirected by verbosefile; the refined approach tested silent substitutions on 10 lines.Source: §F.2 · pp. 46–47

Constraints

  • The final execution is constrained to three non-empty macros, fewer than 200 total keystrokes, allowed Vim commands only, and headless invocation.The rollout records 38 total macro characters and requires vim -Nu NONE -n -Es /app/input.csv -S /app/apply_macros.vim.Source: §F.2 · p. 48

Validation

  • A successful refinement rollout achieved a byte-for-byte match between the transformed input and expected output on a test slice.The agent reports that cmp returned exit code 0 with no output, indicating exact agreement.Source: §F.2 · p. 48

Trajectory context

  • The rollout is identified as a successful refinement trajectory executed by Gemini-3.1-Pro on the large-scale-text-editing Terminal-Bench v2.0 task.The figure presents a subset of the successful trajectory rather than an aggregate benchmark result.Source: Figure 22 · p. 49

G Example summaries

This section illustrates structured rollout summaries across SWE-Bench Verified and Terminal-Bench v2.0, covering code-fix reasoning and artifact-generation tasks. The examples capture requirements, diagnoses, changes, evidence, verification outcomes, task-specific constraints, and distinctions between confirmed and unverified properties.

Example coverage

  • The cited passage identifies a Gemini-3.1-Pro structured summary for SWE-Bench Verified (sympy__sympy_17630) and a Claude-4.5-Opus structured summary for Terminal-Bench v2.0 (openssl_selfsigned_cert).The SWE-Bench summary is associated with rollout trace Figure 19, while the Terminal-Bench summary is associated with rollout trace Figure 20 and presented as Figure 24.Source: G Example summaries · p. 49

SWE-Bench Verified example

  • The SWE-Bench summary identifies a BlockMatrix multiplication failure caused by scalar Zero objects replacing ZeroMatrix objects.Repeated multiplication such as block_collapse(b * b * b) raised an AttributeError because zeros were converted to scalar Zero objects.Source: G Example summaries · pp. 49–50
  • The rollout traced the failure to an artificial scalar 0 inserted before MatAdd canonicalization when nonmatrices was empty.The summary links this behavior to get_postprocessor and explains why MatAdd(0, ZeroMatrix, ZeroMatrix) canonicalized to scalar 0.Source: G Example summaries · p. 50
  • The fix modified _postprocessor in matexpr.py to call mat_class(*matrices) when nonmatrices is empty, avoiding the artificial scalar identity.The change was confined to a single file and added an if not nonmatrices condition.Source: G Example summaries · pp. 50–51
  • After patching, block multiplication produced ZeroMatrix values and the reproduction passed; the expressions suite reported 190 passed and 4 expected to fail.The verification record also confirms block_collapse(b * b * b), repeated block multiplication, and b._blockadd(b) worked without errors, with no attempted checks left unconfirmed.Source: G Example summaries · pp. 52, 54, 55

SymPy code-fix example

  • The SymPy summary identifies the BlockMatrix failure, attributes it to an artificial scalar identity in the postprocessor, and records the corrective change.The fix avoided passing the artificial scalar identity, with evidence from the git diff and related implementation steps.Source: G Example summaries · pp. 56–57
  • 190 expressions tests passed with 4 expected failures, while the reproduction produced ZeroMatrix blocks of the required type.The summary also records that the reproduction script passed and the final patch was inspected.Source: G Example summaries · p. 56

TLS certificate example

  • The TLS example specifies five required artifacts, including a 2048-bit RSA key, a 365-day self-signed certificate, a combined PEM, verification data, and a Python checker.Constraints include key permissions 600, organization DevOps Team, and common name dev-internal.company.local.Source: G Example summaries · p. 57
  • TLS verification for dev-internal.company.local found the DevOps Team subject, validity from Mar 5 2026 through Mar 5 2027, server.key permissions 600, and a successful checker run.Source: G Example summaries · p. 59

Test coverage

  • The summary distinguishes normal execution from untested failure handling: missing or invalid certificates were not tested, and no error outcomes were recorded.The tested scenario was normal execution of check_cert.py with a valid certificate.Source: G Example summaries · pp. 62, 64

Completion summary

  • The agent reports successful completion of the SSL setup, including required files, permissions, certificate identity, verification output, and a successful Python check.The completion claim states that all tasks were completed successfully, including a 2048-bit RSA key, 365-day certificate, combined PEM, verification.txt, and check_cert.py.Source: G Example summaries · p. 63

Evidence inventory

  • The evidence inventory confirms all SSL files existed, verification.txt contained certificate details, and check_cert.py printed the expected identity, expiration date, and success message.The inventory records server.crt, server.key, server.pem, and verification.txt, while execution output shows Common Name dev-internal.company.local and expiration date 2027-03-05.Source: G Example summaries · p. 63

Unverified aspects

  • Exact 2048-bit key size, 365-day validity, and correct key-certificate ordering in server.pem remain unverified because content or exact durations were not inspected.File sizes and approximately one-year date separation provide indirect evidence, but the summary does not explicitly verify these properties.Source: G Example summaries · p. 64

Example summary

  • Figure 24 presents this structured summary as generated by Claude-4.5-Opus for the Terminal-Bench v2.0 openssl_selfsigned_cert task.The figure corresponds to the example summary described in this section.Source: Figure 24 · p. 65

H Example groupwise comparisons

Figures 25 and 26 illustrate groupwise comparisons in which candidate solutions are assessed across requirements, correctness, verification, completeness, and execution evidence. In both examples, the comparisons identify stronger validation and solution quality while exposing limitations in weaker candidates.

SWE-Bench Verified

  • Solution 1 correctly fixes Django’s root cause by repairing through-model dependency generation, whereas Solution 2 only masks the symptom and risks silent migration failures.Solution 1 is targeted to autodetector.py; Solution 2 is overly broad and leaves the through model unresolved.Source: H Example groupwise comparisons · p. 65, Figure 25 · p. 66

Terminal-Bench v2.0

  • Solution 1 passed verification on both videos, producing frame outputs within expected ranges, whereas Solution 2 was tested only on the example video and was possibly incomplete.Solution 1 reported takeoff 51 and landing 62 for the example video, and takeoff 219 and landing 233 for the test video; Solution 2 reported takeoff 53 and landing 60 only for the example video.Source: H Example groupwise comparisons · p. 68
  • Both video-processing interpretations use reasonable computer-vision approaches, but Solution 1 is better supported because it was tested across multiple videos and against expected frame ranges.Solution 2 uses similar background subtraction and contour detection but leaves accuracy, generalization, and error handling unverified.Source: H Example groupwise comparisons · p. 69
  • Solution 1 wins the video-processing comparison because it is explicitly complete, fully confirmed, and more thoroughly validated, while both solutions tie on compliance and reproducibility.The comparison cites valid pytest verification, complete outputs, and full execution evidence as the decisive advantages.Source: H Example groupwise comparisons · p. 69, Figure 26 · p. 70

Related papers

Loading 2604.16529v1…