PaperlayerView exact revision on arXiv

Source-linked AI summary

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

Tan Yu, Alexander Bukharin, Khushi Bhardwaj, Jennifer Williams, Zirui Liu, Jonathan Lingjie Li, Soumye Singhal, Joseph Jennings, Sanjeev Satheesh, Yash Jain, Ashish Vaswani, Venkat Krishna Srinivasan, Matthew Papakipos, Hyunwoo Kim, Jian Zhang, Oleksii Kuchaiev, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jonathan Cohen, Jiantao Jiao

arXiv:2610.10478v1cs.AIcs.SE

TL;DR

Choosing a base checkpoint for agentic post-training is costly, while existing evaluations provide limited evidence of its eventual agentic coding performance. This paper probes base models at verifier-certified decisive steps in successful agent trajectories, and all three probes closely rank checkpoints by post-trained SWE-bench Verified performance.

  • Problem

    Existing end-to-end benchmarks are difficult for base checkpoints to operate, while non-agentic benchmarks do not consistently predict post-trained agentic performance.

    Source: §1 · p. 1

  • Method

    The paper replays successful agent trajectories, identifies the first verifier-passing code change, and probes base models there with action-likelihood, multiple-choice, and sampled-continuation screens.

    Source: §1 · p. 2

  • Results

    All three probes rank ten base checkpoints in close agreement with post-trained SWE-bench Verified performance; Decisive-Action BPB reaches ρ=0.964 on held-out DeepSWE trajectories.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Successful trajectories and task verifiers can turn existing or future agentic coding benchmarks into base-model evaluations.

    Source: §5 · p. 11

Abstract

from arXiv · show

How can we predict which base checkpoint is worth an expensive round of agentic post-training? End-to-end pass@$K$ tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding: many base checkpoints cannot reliably produce the well-formed tool invocation required to complete a task end-to-end. Single-shot or short-horizon tasks avoid these tool-calling failures by collapsing a multi-step interaction into a fixed prompt and a single patch, but they sidestep the core capability we care about: maintaining coherent state over many tool-using steps as the repository evolves. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Replaying each trajectory and rerunning tests after every code-changing step identifies the decisive step: the first step whose cumulative patch flips the repository from failing to passing, certifying that the recorded action solves the task given the prior context. Motivated by a coverage principle for agentic traces, we build three screens at this step that do not require a base checkpoint to drive the harness from a cold start: (i) Decisive-Action BPB (bits per byte) measures the probability mass on the certified action, (ii) Patch MCQ tests the checkpoint's choice between that action and alternatives rejected by the same verifier, and (iii) prefix-conditioned pass@$K$ evaluates support for functionally-correct generations and credits any continuation that the tests accept. Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained SWE-bench Verified pass@$1$. As our methods need only a benchmark's successful trajectories and its verifier, they can be applied to turn future agentic coding benchmarks into base-model evaluations.

1. Introduction

The paper evaluates base-checkpoint agentic coding potential before costly post-training by probing verifier-certified decisive actions from successful trajectories; three probes closely track post-trained SWE-bench Verified performance across ten checkpoints.

1. Introduction

  • The evaluation challenge matters because realizing agentic capabilities requires substantial compute, engineering, and evaluation after checkpoint selection.Source: §1 · p. 1
  • Existing evaluations are either too agentic for base checkpoints to run reliably or too limited to predict sustained agentic performance consistently.Source: §1 · p. 1
  • At that decisive step, BPB scores golden-action likelihood, Patch MCQ tests discrimination from rejected alternatives, and prefix-conditioned pass@K tests sampled continuations with the task verifier.Source: §1 · pp. 2–3
  • The framework replays successful agent trajectories and tests after code-changing steps, defining the first verifier-passing step’s action as the golden action and rejected alternatives as distractors.Source: §1 · p. 2
  • All three probes rank ten base checkpoints in close agreement with post-trained SWE-bench Verified performance; Decisive-Action BPB reaches ρ=0.964 on held-out DeepSWE trajectories.Source: §1 · p. 3

2. Related Work

Prior evaluations either run agentic tasks that base checkpoints handle poorly or measure bounded coding tasks that omit agentic behavior; related proxy methods differ in how they define correct choices and assess likelihood.

2. Related Work

  • End-to-end agentic benchmarks are poorly suited to base checkpoints, while non-agentic code benchmarks can be run but do not test agentic behavior.Source: §2 · p. 3
  • Prior base-checkpoint proxies use answer discrimination, but Patch MCQ grounds its golden action and distractors in the task verifier rather than human or model-based judgments.Source: §2 · pp. 3–4
  • The coverage principle connects post-training potential to probability mass on high-quality responses already present in base models.Source: §2 · p. 4
  • Bits per byte normalizes likelihood for sequence length and supports comparisons across checkpoints with different tokenizers.Source: §2 · p. 4

3. Trajectory-Derived Protocols

The protocol converts successful agentic coding trajectories into verifier-grounded base-model probes by identifying the first code-changing step that passes the task tests, then testing action likelihood, choice, and sampled continuation at that point.

3.1. decisive-step discovery for agentic trajectories

  • The decisive step is the earliest code-changing step whose cumulative patch passes the task verifier; its evaluation context contains only earlier trajectory steps.Source: §3.1 · p. 4
  • Restricting BPB to the verifier-derived decisive step raises rank correlation with SWE-bench Verified from 0.915 to 0.964 across ten checkpoints and corrects two checkpoint-pair orderings.Source: Figure 2 · p. 5

3.2. Decisive-Action BPB

  • Decisive-Action BPB measures the golden action’s negative log-likelihood in bits per byte, making scores comparable across checkpoints with different tokenizers.Source: §3.2 · p. 5

3.3. Patch MCQ

  • Patch MCQ presents the verifier-passing golden action alongside alternatives that fail the same verifier, then compares their answer-letter likelihoods jointly in a shared context.Source: §3.3 · pp. 5–6

3.4. Prefix-conditioned pass@K

  • Prefix-conditioned pass@K samples continuations from the trajectory prefix before the decisive action and credits any continuation that passes the task verifier.Source: §3.4 · p. 6

4. Experiments

The experiments compare trajectory-derived probes with post-training rankings across checkpoints, trajectory sources, benchmarks, a controlled SFT recipe, and probe ablations. The probes show close rank agreement on SWE-bench Verified and Multilingual, with results also testing baseline benchmarks and evaluation choices.

4.1. Methodology

  • The study correlates three probe scores across ten base checkpoints with downstream post-trained pass@1 on SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.1.Probe trajectories come from SWE-bench Verified, SWE-Pro, and DeepSWE; Spearman ρ measures checkpoint ordering, while Pearson r measures linear agreement.Source: §4 · p. 7, §4.1 · p. 7

4.2. Correlations with SWE-Verified

  • All three probes track Verified post-training performance; DeepSWE versus Verified trajectories yield ρ=0.964 versus 0.915 for BPB and 0.951 versus 0.906 for prefix-conditioned pass@K, while Patch MCQ yields 0.867 versus 0.903.The matched-trajectory comparisons do not uniformly improve agreement: only Patch MCQ gains from using Verified trajectories.Source: §4.2 · p. 7, Figure 5 · p. 8, Figure 3 · p. 8
  • Bounded benchmarks do not recover post-trained rankings: MBPP has ρ=0.091, HumanEval ρ=−0.394, and RepoBench XFirst ρ=0.830, below the probes’ ρ=0.867–0.964.Source: §4.2 · p. 7

4.3. Correlations with SWE-Multilingual

  • On SWE-bench Multilingual, the probes reach ρ=0.976 for Decisive-Action BPB, 0.927 for Patch MCQ, and 0.945 for prefix-conditioned pass@K.Pearson r is lower, at 0.82–0.85; the passage attributes the gap between correlation statistics to Gemma 4 26B’s 17.3% Multilingual versus 52.0% Verified performance.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

4.4. SFT Experiments

  • Under an identical 1,000-step, 16.8B-token SFT recipe, probes computed on SWE-Pro trajectories order three checkpoints consistently with their post-SFT SWE-bench Verified performance.The authors describe this as a continuing correlation trend, not a precise estimate, because only three checkpoints were tested.Source: §4.4 · p. 9, Table 1 · p. 9

4.5. Ablation Studies

  • Masking tool-call formatting improves BPB agreement across trajectory corpora, and decisive-step BPB correlates much better than scoring all turns.Source: §4.5 · pp. 9–10, Figure 7 · p. 10
  • Jointly scoring all Patch MCQ options yields stronger agreement than independent scoring, with reasoning adding a smaller further gain on both question sets.With reasoning, agreement reaches ρ=0.867 on DeepSWE questions and ρ=0.903 on SWE-Verified questions.Source: §4.5 · p. 10, Table 2 · p. 10
  • For prefix-conditioned pass@K, K=16 captures nearly all attainable agreement: DeepSWE reaches ρ=0.952, while Verified prefixes reach ρ=0.988 at K=8 and 16.At K=32, Verified-prefix agreement falls to 0.906 as pass rates approach saturation, up to 83.7%.Source: §4.5 · p. 10, Table 3 · p. 10

5. Conclusion

The paper introduces a way to evaluate base checkpoints’ agentic coding potential before expensive post-training by probing successful trajectories at verifier-identified decisive steps. Across ten base checkpoints, its three probes rank models in close agreement with post-trained SWE-bench Verified performance.

5. Conclusion

  • The framework replays successful agent trajectories and uses the task verifier to identify the first step that resolves the task.This supplies context the base checkpoint cannot yet reliably produce and avoids requiring it to operate a tool-use harness across a full trajectory.Source: §5 · p. 11
  • Three complementary screens assess decisive-action likelihood, choice among verifier-rejected alternatives, and support for passing continuations.The probes are Decisive-Action BPB, Patch MCQ, and Prefix-Conditioned pass@K.Source: §5 · p. 11
  • Across ten base checkpoints, all three probes rank models in close agreement with post-trained SWE-bench Verified performance.The framework uses successful trajectories and the task verifier to extend agentic coding benchmarks into base-checkpoint evaluations before expensive post-training.Source: §5 · p. 11

A. Independent Audit of SWE -MCQ Construction

The audit finds substantial correctness and uniqueness defects in APTBench’s SWE multiple-choice questions. The section also evaluates APTBench scores against post-trained SWE-bench Verified performance on the same ten checkpoints.

A. Independent Audit of SWE -MCQ Construction

  • 727 of 1,682 APTBench questions (43.2%) were classified as invalid because they lacked a uniquely correct designated answer and incorrect distractors.The audit covered Fix Patch, Plan, Test Patch, and Handle Error questions.Source: A. Independent Audit of SWE -MCQ… · p. 16, Table 4 · p. 16
  • The audit found both incorrect designated answers and non-unique choices, including identical options where only one was marked correct.Handle Error MCQ #67’s designated answer did not address the question, while Plan MCQ #66 contained identical choices A and C.Source: A. Independent Audit of SWE -MCQ… · p. 16
  • These defects weaken MCQ accuracy as a measure of distinguishing correct from incorrect answers and motivate verifier-validated Patch MCQ choices.Patch MCQ requires the golden action to pass and each distractor to fail the same verifier from the same environment state.Source: A. Independent Audit of SWE -MCQ… · p. 16

B. APTBench Reproduction

  • APTBench was evaluated on the same ten base checkpoints, comparing its scores with the SWE-bench Verified performance of their corresponding post-trained releases.The section reports Pearson and Spearman correlations for this comparison.Source: B. APTBench Reproduction · p. 16, Table 5 · p. 17

C. Experimental Setup Details

The experimental setup matches base checkpoints to post-trained downstream targets and evaluates three probes across benchmark cohorts. It also specifies the rollout settings and treats correlation estimates as descriptive rather than calibrated guarantees.

C. Experimental Setup Details

  • The evaluation cohort pairs base checkpoints with post-trained releases whose reported SWE-bench Verified and Terminal-Bench 2.1 pass@1 percentages serve as downstream targets.The target-release identity matters because some model families released multiple candidates.Source: Table 6 · p. 17
  • On DeepSWE, decisive-step BPB increases Spearman correlation with post-trained SWE-bench Verified from 0.915 to 0.964.This is the reported all-step versus decisive-step comparison on the matched ten-checkpoint cohort.Source: C. Experimental Setup Details · pp. 17–18
  • Patch MCQ uses two four-option sets with chance accuracy 0.250: 756 DeepSWE questions and 576 SWE-bench Verified questions.The sets cover 392 and 249 instances respectively, across the same ten checkpoints and matched none preambles.Source: C. Experimental Setup Details · p. 18
  • Prefix-conditioned pass@K samples 8, 16, or 32 rollouts on 120 DeepSWE instances and 98 SWE-bench Verified prefixes.After the decisive-step fork, the candidate controls the remaining episode; the SWE-bench Verified arm uses the same benchmark for prefixes and downstream validation, and scores are not comparable across agent formats.Source: C. Experimental Setup Details · p. 18
  • Reported correlations are descriptive point estimates without bootstrap intervals, so the authors treat them as indicative rather than calibrated predictive guarantees.The primary downstream correlations use ten models; other comparisons use their stated completed coverage.Source: C. Experimental Setup Details · p. 18

D. Evaluation Infrastructure

The evaluation infrastructure replays recorded agent trajectories and runs the benchmark verifier to find decisive steps, then branches into static or sampled scoring. Its design limits verifier executions and addresses replay integrity and endpoint-throughput risks.

D. Evaluation Infrastructure

  • All three probes share a pipeline, with verifier executions treated as the scarce resource and scoring paths diverging only at the final stage.The pipeline’s first five stages run once per trajectory corpus; only stage six is checkpoint-specific.Source: D. Evaluation Infrastructure · p. 18, Figure 8 · p. 19
  • A recording proxy captures wire requests and responses, enabling agent-agnostic replay across four agent families while retaining their original step boundaries.Each instance runs in its own task-image container; resets preserve ignored build artifacts, and prompts exclude benchmark hints as a leakage control.Source: D. Evaluation Infrastructure · pp. 18–19
  • Replay records verifier outcomes separately from infrastructure errors, and a command mismatch raises rather than falling through to live execution.This prevents a failed verifier run from being mistaken for a rejected patch and avoids replaying a different command sequence than the recorded context describes.Source: D. Evaluation Infrastructure · p. 19
  • Bisection reduces verifier executions per trajectory from |C_i| to O(log |C_i|), assuming that once a cumulative patch passes, later prefixes also pass.A linear-scan mode exists for corpora that violate this monotonicity assumption; both modes first verify the maximum prefix as a gate, and reported decisive-step metadata used bisection.Source: D. Evaluation Infrastructure · pp. 19–20
  • Endpoint saturation can turn rate-limited rollouts into apparently low scores, while context-limit drops remove some of the longest, most-explored prefixes.The authors cap runs below measured concurrency ceilings, purge empty predictions from degraded windows, and compare on the completed intersection.Source: D. Evaluation Infrastructure · p. 20

E. Bounded Code Benchmarks as Baselines

Bounded coding benchmarks vary widely in their agreement with post-trained agentic performance, and even the strongest baseline trails the trajectory-derived probes. Their correlations are descriptive, and the comparison is not controlled.

E. Bounded Code Benchmarks as Baselines

  • RepoBench XFirst is the strongest bounded baseline by rank agreement (ρ=0.830); CRUXEval-O is genuinely informative (r=0.814, ρ=0.758), while MBPP (ρ=0.091) and HumanEval (ρ=−0.394) show little or negative rank signal.Across ten checkpoints, MBPP spans 67.2–86.0 versus 38.8–80.6 for the downstream target, consistent with range compression.Source: E. Bounded Code Benchmarks as Baselines · p. 21
  • Across ten checkpoints, MBPP spans 67.2–86.0 and RepoBench XFirst 73.2–82.8, while the downstream target spans 38.8–80.6.A practitioner without a trajectory corpus may still find CRUXEval-O and RepoBench XFirst informative.Source: E. Bounded Code Benchmarks as Baselines · p. 21
  • The comparison is descriptive, not controlled: baselines are published-style scores, probes use selected trajectories, and downstream scores come from different labs’ post-training recipes and harnesses.The evidence supports large variation in agreement within one cohort, not a basis for choosing a bounded benchmark in advance.Source: Table 7 · p. 20, E. Bounded Code Benchmarks as Baselines · p. 21

F. End-to-End Evaluation of Base Checkpoints

End-to-end evaluation ties most base checkpoints at the floor, while the probes transfer to Terminal-Bench with prefix-conditioned pass@K showing the strongest agreement. The implementation uses a universal trajectory rendering and scores semantic assistant content for BPB.

F. End-to-End Evaluation of Base Checkpoints

  • Five of six base checkpoints resolve no SWE-bench Verified task; Qwen-3.5-35B-A3B-Base scores 20.2% and ranks first despite its post-trained family ranking fourth.The exception may reflect familiarity with agent-formatted interaction rather than coding ability.Source: Table 8 · p. 21, F. End-to-End Evaluation of Base… · p. 21

G. Correlations with Terminal-Bench 2.1

  • On Terminal-Bench 2.1, prefix-conditioned pass@K has ρ=0.930 agreement, versus ρ=0.833 for Decisive-Action BPB and ρ=0.796 for Patch MCQ.The probes are derived from repository-issue trajectories, whereas Terminal-Bench scores command-line workflows.Source: Figure 9 · p. 22, G. Correlations with Terminal-Bench 2.1 · p. 22

H.1. Universal Chat Template and Formatting Masks

  • The universal renderer keeps messages in chronological order, uses ordinary-text role labels, and serializes tool calls consistently without model-specific control tokens.For Decisive-Action BPB, only assistant-generated semantic content is scored; fixed labels and tool-call wrapper syntax are excluded.Source: §H.1 · pp. 22–23

I. Patch MCQ Detail

Patch MCQ conclusions depend on how answer choices are read: the joint, rotated answer-letter read outperforms independent option scoring and can alter checkpoint rankings. Rotation gains vary, and reasoning protocols do not uniformly dominate.

I. Patch MCQ Detail

  • The joint read averages answer-letter probabilities across four cyclic option rotations, then selects the option with the largest aggregate.Complete rotations are required so the position prior cancels; questions with any failed read are left unscored.Source: §I · pp. 24–25
  • Using one rotation instead of four costs 0.039 accuracy on average; the gain ranges from −0.005 to +0.081 and reorders the top two checkpoints.DeepSeek V4 Pro leads on one read (0.4555 versus 0.4271) but trails on the aggregate (0.5026 versus 0.5080).Source: Table 12 · p. 25, §I · p. 26
  • For post-trained SWE-bench Verified pass@1, joint-read ρ is 0.867 on DeepSWE and 0.903 on SWE-Verified, versus argmin-BPB’s 0.665 and 0.578.The reads also rank checkpoints differently, with Spearman ρ=0.543 on DeepSWE and 0.322 on SWE-Verified.Source: Table 13 · p. 25, §I · p. 26
  • Reasoning raises mean accuracy by +0.025 on DeepSWE and +0.008 on SWE-Verified, but the protocols remain distinct and neither uniformly dominates.Without run-to-run spread, small per-checkpoint differences should not be treated as measured effects.Source: §I · pp. 26–27

J. Prefix-Conditioned pass@𝐾Detail

Prefix-conditioned pass@K forks from the decisive step and lets the checkpoint continue the task, with the harness bridged to accept base-model completions. Its benchmark rankings disagree on some checkpoint pairs, and the held-out correction is a consistency check rather than independent evidence.

J. Prefix-Conditioned pass@K Detail

  • A bridge renders each agent request as plain text, samples from the base checkpoint’s completion endpoint, and repackages its output as the native tool call.The prompt orders transcript, steer, primer, and prefill; the primer begins the tool-call turn.Source: J. Prefix-Conditioned pass@𝐾Detail · p. 27, Figure 11 · p. 28
  • Prefilling the action envelope raises parseable forwards from 27% to 50% and repository-editing forwards from 10% to 27% on eight DeepSWE prefixes.The envelope must match the agent’s wire format: a JSON-array opening for function calls or a fenced command-block opening in text mode.Source: J. Prefix-Conditioned pass@𝐾Detail · pp. 27–28
  • The base checkpoint samples each action directly without an intervening plan; a two-step plan reduced parseable forwards from 56% to 38%.A 1,024-token cap raised parse rate from 68% to 80% compared with the 512-token cap.Source: J. Prefix-Conditioned pass@𝐾Detail · p. 28
  • Caching both filesystem images and observation logs reproduced byte-identical patches and cut prefix wall time by 96% on ten prefixes.Prefixes with background services use the uncached fallback because committing a container drops running processes.Source: J. Prefix-Conditioned pass@𝐾Detail · pp. 28–29
  • Each rollout is scored in a separate harness run to avoid prediction-file collisions, while generation and scoring are pipelined after byte-identical validation.Source: J. Prefix-Conditioned pass@𝐾Detail · p. 29
  • The rollout replays steps before the decisive step, masks that action, and gives the checkpoint control for the remaining episode, capped at three generated steps.An instance counts as resolved if at least one continuation passes the verifier.Source: J. Prefix-Conditioned pass@𝐾Detail · p. 27, Table 14 · p. 29
  • On held-out DeepSWE prefixes, pass@K ranks DeepSeek V4 Flash above Nemotron Ultra 550B at every k, matching BPB’s correction, but this is not independent evidence.At k=32, the values are 40.00% versus 26.67%; the probe forks at the decisive step.Source: J. Prefix-Conditioned pass@𝐾Detail · p. 29, Table 14 · p. 29
  • The SWE-bench Verified prefix arm disagrees on three pairs, all involving Nemotron Ultra, which it ranks above DeepSeek V4 Flash, Kimi K2, and Hy3 Preview despite lower downstream scores.Source: J. Prefix-Conditioned pass@𝐾Detail · p. 30

K. Additional SFT Correlation Results

This section compares probe correlations across SWE trajectory sources against post-SFT SWE-bench Verified scores for three models, alongside public-score correlations for three and ten models. The three-model correlations are descriptive because one pairwise inversion changes Spearman correlation substantially.

K. Additional SFT Correlation Results

Related papers

Loading 2610.10478v1…