PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Test-Time Compute for Agentic Coding

Joongwon Kim, Wannan Yang, Kelvin Niu, Hongming Zhang, Yun Zhu, Eryk Helenowski, Ruan Silva, Zhengxing Chen, Srinivasan Iyer, Manzil Zaheer, Daniel Fried, Hannaneh Hajishirzi, Sanjeev Arora, Gabriel Synnaeve, Ruslan Salakhutdinov, Anirudh Goyal

arXiv:2604.16529v1cs.SEcs.AIcs.CLcs.LG

TL;DR

Long-horizon coding agents produce trajectories that existing test-time scaling methods cannot directly compare or reuse, creating a representation bottleneck. The paper introduces compact structured rollout summaries shared by RTV for parallel selection and agentic PDR for sequential refinement, consistently improving frontier agents across two coding benchmarks, including Claude-4.5-Opus’s gains from 70.9% to 77.6% on SWE-Bench Verified and 46.9% to 59.1% on Terminal-Bench v2.0.

  • Problem

    Long-horizon coding trajectories are noisy and difficult to compare or reuse directly, so standard test-time scaling methods do not transfer cleanly to agentic coding.

  • Method

    The framework converts rollouts into compact structured summaries used by RTV for parallel aggregation and agentic PDR for sequential refinement.

  • Results

    The framework consistently improves frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0; Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified.

  • Takeaways & Limitations

    Test-time scaling for long-horizon agents works when prior experience is transformed into representations that support selection and reuse.

  • Takeaways & Limitations

    The current pipeline uses freshly initialized environments; retaining persistent workspace artifacts across attempts remains an open question.

Abstract

from arXiv · show

Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.

1 Introduction

Agentic coding makes test-time scaling difficult because long, noisy trajectories cannot be directly compared or reused. The paper addresses this by representing rollouts as compact summaries that support parallel selection and sequential refinement, yielding gains across frontier agents and coding benchmarks.

  • Motivation: Long-horizon coding trajectories contain useful progress and failure signals but are too noisy and verbose for direct comparison or reuse.Standard test-time scaling therefore does not transfer cleanly from short, bounded outputs to agentic coding.
  • Approach: Each rollout is converted into a compact structured summary that preserves hypotheses, decisions, progress, and failure modes while omitting low-value trace details.These summaries provide a reusable interface for later computation without replaying full interaction histories.
  • Approach: Recursive Tournament Voting selects among parallel rollouts through recursive small-group comparisons, while agentic PDR conditions new rollouts on summaries distilled from prior attempts.The two operators address parallel aggregation and sequential refinement respectively.
  • Findings: Structured summaries outperform raw trajectories for parallel aggregation, and recursive small-group selection outperforms flatter large-group selection.Sequential refinement also benefits from conditioning on multiple prior summaries rather than a single prior attempt.
  • Findings: The unified recipe consistently improves frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0.Claude-4.5-Opus rises from 70.9% to 77.6% on SWE-Bench Verified and from 46.9% to 59.1% on Terminal-Bench v2.0.
  • Motivation: The paper identifies representation of prior agent experience as a central bottleneck in inference-time scaling for long-horizon agentic coding.The claim is supported through targeted preliminary experiments.
  • Conclusion: The results frame long-horizon inference-time scaling as a problem of representation, selection, and reuse.This view summarizes the paper’s proposed interface and inference operators.

2 Methodology

The methodology scales agentic coding by representing long rollout trajectories as compact summaries, then applying parallel selection and sequential reuse. RTV recursively selects among summarized rollouts, while PDR conditions fresh rollouts on selected prior summaries before a final RTV selection.

  • 2.1 Problem Formulation: Agentic coding rollouts interleave actions and terminal observations across multiple steps, making rollout count the primary test-time scaling unit.Each action includes the agent’s thought and bash commands, while observations include expected outputs and unexpected errors.
  • 2.2 Rollout Summaries as Reusable Representations: Compact structured summaries preserve useful rollout signal while removing low-signal details, providing bounded objects for comparison and reusable context for future rollouts.The summaries capture information from prior attempts without replaying their full interaction histories.
  • 2.3 Parallel Selection via Recursive Tournament Voting: RTV executes N parallel rollouts, summarizes each one, and recursively compares summaries in small groups until a single rollout remains.The procedure aims to select the highest-quality rollout without access to ground-truth outcomes, test cases, or test samples.
  • 2.4 Sequential reuse via Parallel-Distill-Refine: PDR samples K summaries from a previous iteration and uses their concatenation as refinement context for fresh rollouts in newly initialized environments.This sequential scaling dimension reuses information extracted from earlier attempts while generating new rollouts.
  • 2.5 Unified PDR + RTV Pipeline: The unified pipeline summarizes N iteration-0 rollouts, uses RTV to select K summaries, runs N conditioned iteration-1 rollouts, and applies final RTV to return the top-1 rollout.The design combines exploitation through selecting higher-quality prior rollouts with exploration through multiple refinement-context summaries and fresh rollouts.
  • 2.5 Unified PDR + RTV Pipeline: Structured summaries improve parallel aggregation over direct rollout traces on SWE-Bench Verified and Terminal-Bench v2.0 for Gemini-3-Flash and Claude-4.5-Sonnet.The comparison evaluates summaries as representations against full rollout traces.

3 Results and Analyses

Experiments evaluate the framework’s ablations, sequential and parallel dynamics, and main benchmark performance across frontier models. Results consistently favor compact summaries, recursive small-group selection, and selected multi-rollout refinement contexts.

  • Experimental Setup: The evaluation covers SWE-Bench Verified and Terminal-Bench v2.0 across five frontier models using the framework’s main experimental configuration.The main setup uses N = 16, T = 2, K = 4, G = 2, and V = 8; Terminal-Bench evaluation includes 88 of 89 available tasks.
  • Parallel Aggregation Ablations: Compact structured summaries outperform full rollout trajectories for selecting among long-horizon agentic rollouts.Summaries preserve decisive information while discarding low-signal detail that makes full trajectories difficult to compare reliably.
  • Parallel Aggregation Ablations: Pairwise recursive comparisons yield the strongest performance, while increasing vote aggregation improves local decisions with diminishing returns around V = 8.The findings motivate the default RTV configuration G = 2 and V = 8.
  • Parallel Aggregation Ablations: RTV improves over the average initial rollout across all five models and both benchmarks, with larger gains on Terminal-Bench v2.0.For Claude-4.5-Sonnet, performance rises from 67.4% to 73.6% on SWE-Bench Verified and from 40.6% to 54.6% on Terminal-Bench v2.0.
  • Sequential Refinement Ablations: Sequential refinement improves when conditioning on multiple prior rollouts, and improves further when RTV selects those rollouts rather than sampling them randomly.With K = 4, Gemini-3.1-Pro reaches 79.25% and Claude-4.5-Sonnet reaches 78.06% during iteration 1 under select-K refinement.
  • Sequential Refinement Ablations: Higher-quality refinement contexts produce stronger next-iteration rollouts, with success increasing from near-zero for 0/4 successful contexts to 97–99% for 4/4 contexts.RTV also produces more 4/4 passing contexts than random selection: 62 versus 52.8 tasks for Claude-4.5-Sonnet and 71 versus 58.5 for Gemini-3.1-Pro.
  • Main Results: The unified method consistently improves single-attempt baselines, with gains across both benchmarks and all evaluated frontier models.Claude-4.5-Opus improves from 70.94% to 77.60% on SWE-Bench Verified and from 46.95% to 59.09% on Terminal-Bench v2.0.
  • Main Results: The refinement context can improve search efficiency, while passing trajectories are consistently shorter than failing trajectories within each iteration.The reported analysis connects prior context with more direct navigation toward solutions and fewer exploratory or erroneous steps.

4 Analysis

The analysis examines how sequential refinement and parallel aggregation reshape rollout populations and affect selection quality. It finds that refinement context quality predicts later success, while recursive aggregation remains useful after refinement but with diminishing gains and judge-dependent limitations.

  • Sequential Refinement Dynamics: Iteration-1 pass@1 rises sharply as more selected iteration-0 rollouts are passing.For Claude-4.5-Opus, scores increase from 0.1% with 0 of 4 passing context rollouts to 99.2% with 4 of 4 passing rollouts.
  • Sequential Refinement Dynamics: Across both benchmarks, the number of tasks with 16/16 passing rollouts increases from iteration 0 to iteration 1.The analysis attributes this shift to high-quality rollouts selected into the refinement context.
  • Sequential Refinement Dynamics: Low-success rollout populations can also produce more tasks with 0/16 passing rollouts after refinement.RTV may retain one or fewer successful candidates among its top four, limiting the quality of the refinement context.
  • Parallel Aggregation Dynamics: Parallel aggregation remains valuable after refinement because refined rollout populations retain useful intra-task diversity.The second RTV instance still improves performance, although its gains are smaller because refinement reduces the number of mixed pass/fail outcomes.
  • Parallel Aggregation Dynamics: Judge selection accuracies are not controlled head-to-head comparisons across models.Each judge evaluates different rollout pools and summaries, so observed differences also reflect pool difficulty, diversity, and summary informativeness.
  • Parallel Aggregation Dynamics: Gemini-3.1-Pro’s lowest judging accuracy corresponds to only a +0.44% improvement from average iteration-1 pass@1 to final pass@1.The reported values are 76.16% average iteration-1 pass@1 and 76.60% final pass@1.

5 Conclusion

The paper argues that effective test-time scaling for long-horizon agentic coding depends on representing prior experience in bounded forms that later computation can compare and reuse. Compact rollout summaries support parallel selection and sequential refinement, yielding consistent gains while leaving persistent workspace artifacts as an open direction.

  • The method treats prior-experience representation as the central bottleneck in test-time scaling for long-horizon agentic coding.
  • Compact structured summaries expose rollout hypotheses, decisions, progress, and failure modes for Recursive Tournament Voting and Parallel-Distill-Refine.
  • The framework combines exploitation through refinement of high-quality rollouts with exploration through multiple parallel rollouts.
  • The unified RTV-PDR pipeline consistently improves single-attempt baselines across frontier language models and SWE-Bench Verified and Terminal-Bench v2.0.
  • Persistent workspace artifacts remain a future direction beyond reusing textual descriptions of prior experience.
  • Gemini-3.1-Pro experienced many API failures during final RTV experiments, which may relate to its abnormally lower success rate as a judge.

A Rollout statistics

This appendix reports rollout statistics for the paper’s main experiments on SWE-Bench Verified and Terminal-Bench v2.0, using the mini-SWE-agent and Terminus 1 harnesses. It measures pass rates, mixed rollout outcomes, and model performance across both benchmarks.

  • The analysis measures average pass@1, pass@16 with N=16, and tasks containing mixed passing and failing rollouts for each iteration.
  • SWE-Bench Verified uses the bash-only mini-SWE-agent harness, while Terminal-Bench v2.0 uses the Terminus 1 harness.
  • Rollout statistics cover SWE-Bench Verified and Terminal-Bench v2.0 in Tables 7 and 8.
  • Gemini-3.1-Pro performs best across both benchmarks, closely followed by Claude-4.5-Opus.
  • Gemini-3-Flash outperforms Claude-4.5-Sonnet on SWE-Bench Verified, whereas Claude-4.5-Sonnet outperforms Gemini-3-Flash on Terminal-Bench v2.0.
  • GPT-5-0825 performs worst among the evaluated models, which the passage attributes to its earlier release date.

B Model comparisons

The model-comparison analysis evaluates pairwise success differences using 16 iteration-0 rollouts per model. Gemini-3.1-Pro is strongest overall, while relative model rankings vary by benchmark and smaller same-family models rarely outperform larger counterparts.

  • Pairwise comparisons count tasks where one model succeeds in at least one of N=16 rollouts while another fails all rollouts.
  • Gemini-3.1-Pro is overall most competitive across both benchmarks, closely followed by Claude-4.5-Opus.
  • Gemini-3-Flash has slightly higher win rates than Claude-4.5-Sonnet on SWE-Bench Verified but lower win rates on Terminal-Bench v2.0.
  • Gemini-3-Flash beats Gemini-3.1-Pro on only 10/500 SWE-Bench Verified tasks and 1/88 Terminal-Bench v2.0 tasks.
  • GPT-5-0825 is the least competitive model in the comparison.

C Sequential refinement dynamics

This appendix provides additional results for the sequential refinement analyses used in the PDR+RTV main experiments. The supplied passage identifies these results as an extension of the analysis in Section 4.1.

  • The appendix provides additional results for the sequential refinement analyses in the PDR+RTV main experiments.
  • The sequential refinement analyses are described as corresponding to Section 4.1.
  • The supplied appendix passage does not state a numerical finding from these additional analyses.

C.1 Context quality analysis

The analysis compares refinement contexts selected by RTV with randomly selected contexts and examines how context quality relates to next-iteration performance. Select-K refinement produces more all-passing contexts and improves average pass@1, while Figure 14 shows task-level pass-rate distributions by context quality.

  • Context quality: Select-K refinement yields more refinement contexts containing only passing rollouts than random-K refinement.The comparison stratifies tasks by the number of passing rollouts in each refinement context.
  • Context quality: Select-K refinement contributes to a higher average pass@1 score than random-K refinement.
  • Task-level distributions: Figure 14 visualizes task-level pass-rate distributions for 100 SWE-Bench Verified tasks using Claude-4.5-Sonnet and Gemini-3.1-Pro.Tasks are grouped by the number of passing rollouts in the refinement context and measured by average iteration-1 pass rates.

D New solution discoveries

The PDR+RTV refinement step discovers solutions on tasks that models fail across all initial parallel rollouts. These discoveries occur across both benchmarks and reveal complementary strengths across models.

  • New solution discovery: PDR+RTV finds solutions on tasks where models fail across all N=16 iteration-0 rollouts but succeed in at least one iteration-1 rollout.Tables 9 and 10 list these tasks for SWE-Bench Verified and Terminal-Bench v2.0.
  • Cross-benchmark comparison: 13 Terminal-Bench v2.0 tasks show 0→1 improvement capabilities across all models, compared with 5 SWE-Bench Verified tasks.SWE-Bench Verified contains almost six times as many tasks as Terminal-Bench v2.0.
  • Cross-model overlap: All Terminal-Bench v2.0 tasks self-improved by Claude-4.5-Opus were solved by Gemini-3.1-Pro in at least one iteration-0 rollout.
  • Cross-model overlap: All but one Terminal-Bench v2.0 task self-improved by Gemini-3.1-Pro were solved by Claude-4.5-Opus or Claude-4.5-Sonnet initially.
  • New solution discovery: Gemini-3.1-Pro solves large-scale-text-editing during iteration 1 with PDR+RTV despite none of the models solving it initially.The task is an example of refinement producing a successful rollout after initial failure.

E PDR + RTV qualitative examples

Qualitative rollout excerpts show agents using multiple prior summaries to consolidate repeated findings, avoid duplicated exploration, and resolve disagreements before executing targeted fixes or solutions.

  • Example coverage: The examples cover Django, SymPy, SPARQL, and SQLite tasks across SWE-Bench Verified and Terminal-Bench v2.0.
  • SWE-Bench Verified: Claude-4.5-Opus uses four prior analyses to identify the Django bug’s root cause and directly apply the shared pieces[-1] fix.It reuses a working sed command and installs three dependencies together before testing.
  • SWE-Bench Verified: Gemini-3.1-Pro consolidates four prior rollouts’ SymPy diagnosis and applies the agreed fix at the root of _split_gcd.The rollout reproduces the IndexError before modifying the function.
  • Terminal-Bench v2.0: Claude-4.5-Opus synthesizes four prior SPARQL attempts, separates the EU and enrollment conditions, and selects the approach used by successful attempts.The prior summaries expose both expected test outputs and the need for separate FILTER EXISTS clauses.
  • Terminal-Bench v2.0: Claude-4.5-Sonnet combines four summaries of SQLite recovery findings and resolves a datatype conflict using the interpretation from successful attempts.All four attempts recovered the corrupted database’s rows, used Python binary parsing, and identified the truncated entry.

F Example rollout trajectories

The paper provides abbreviated rollout trajectories for both benchmarks, including initial and refined traces. The examples are partial because full trajectories are too long to include.

  • Trajectory coverage: Example rollout trajectories are provided for SWE-Bench Verified and Terminal-Bench v2.0.
  • Trajectory coverage: The paper includes both initial rollout traces and refined rollout traces, but only a subset of each full trajectory.Full trajectories are omitted because they are too long to fit.

F.1 Initial rollout trajectories

The initial rollout examples show agents diagnosing issues through iterative terminal interactions, with early failures, environment errors, and eventual progress toward reproductions or fixes.

  • Initial trajectories are presented as examples from SWE-Bench Verified and Terminal-Bench v2.0.
  • SymPy’s initial rollout reproduces a BlockMatrix multiplication failure after repeated multiplication, raising an AttributeError on a scalar Zero object.
  • The agent identifies that resulting zeros have type Zero rather than ZeroMatrix, while lacking enough SymPy-internal knowledge to explain the cause initially.
  • The rollout encounters a missing mpmath dependency before installation enables further reproduction attempts.
  • After installing mpmath, the reproducer successfully evaluates double BlockMatrix multiplication, while triple multiplication remains the failing operation.

1. Create a directory at ‘/app/ssl/‘ to store all files

The rollout creates an SSL workspace and builds the requested key, certificate, combined PEM, verification file, and certificate-checking script with specified metadata and permissions.

  • A 2048-bit RSA private key is generated at /app/ssl/server.key and assigned permissions 600.
  • The self-signed certificate is configured for 365 days with Organization Name DevOps Team and Common Name dev-internal.company.local.
  • The private key and certificate are concatenated into /app/ssl/server.pem.
  • verification.txt records the certificate subject, validity dates, and SHA-256 fingerprint, while check_cert.py verifies files and reports certificate details.
  • The agent creates /app/ssl and uses it as the workspace for the required SSL files.

F.2 Refined rollout trajectories

Refined trajectories show agents using prior failures and task-specific evidence to improve subsequent attempts, including a successful constrained GPT-2 implementation after earlier failures.

  • Refinement examples include tasks that succeeded after all iteration-0 rollouts had failed.
  • Earlier GPT-2 attempts produced repeated nonsense instead of the expected license text, motivating closer examination of the checkpoint format.
  • The agent inspects checkpoint dimensions, weight layout, and expected tensor sizes before implementing the dependency-free C program.
  • The resulting gpt2.c program is 3467 bytes, compiles with gcc -O3 -lm, and passes the provided test.

G Example summaries

Structured summaries condense rollout findings, actions, code changes, and verification outcomes into examples spanning SymPy and Terminal-Bench tasks.

  • The examples include structured summaries for SWE-Bench Verified and Terminal-Bench v2.0, linked to representative rollout traces.
  • The SymPy summary identifies scalar-zero insertion during MatAdd construction as the cause of the BlockMatrix failure.
  • The proposed fix changes matexpr.py so empty nonmatrix arguments no longer introduce an artificial scalar identity.
  • The summary records a single-file modification and reports no unresolved issues or unverified claims.
  • Verification reports successful triple multiplication, ZeroMatrix outputs, and 190 passed tests with 4 expected failures.
  • The SSL example records generated certificate details, including DevOps Team, the internal common name, and one-year validity.

H Example groupwise comparisons

The section contrasts two candidate solutions by correctness, scope, verification, and execution evidence. Solution 1 is consistently favored because it addresses the root cause and is more thoroughly validated.

  • Code Change Quality Assessment: Solution 1 fixes the root cause by correcting dependency generation for through models, whereas Solution 2 suppresses the resulting error.The underlying issue is that the target model is used instead of the through model when resolving dependencies; suppressing the error can leave migration state incomplete.
  • Code Change Quality Assessment: Solution 1 is targeted to through-model dependency logic, while Solution 2 makes broader changes across schema backends and migration operations.
  • Code Change Quality Assessment: Solution 1 is explicitly complete, with expected outputs on both example and test videos; Solution 2 was tested only on the example video.Solution 1 reports takeoff and landing results within expected ranges for both videos, while Solution 2 does not report test-video validation.
  • Overall Assessment: The final comparison ranks Solution 1 higher on verification validity, output completeness, and execution evidence, while both solutions are compliant and self-contained.
  • Verification Validity Assessment: Solution 1’s pytest verification validated frame detection on both videos, whereas Solution 2 used manual execution without running the available test suite.The pytest run returned 5 passed in 1.66s and included validation of takeoff frame 219 and landing frame 233 on the test video.
Loading 2604.16529v1…