Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

1 Introduction

• During pretraining, no established approach evaluates base models on end-to-end agentic tasks, limiting observation of changes across checkpoints and training stages.Source: §1 · p. 1
• Standard agent scaffolds assume instruction-following models, while fine-tuning checkpoints adds cost and changes the model being evaluated.Pretraining-side alternatives largely use static or likelihood-based proxies with limited visibility into environmental interaction.Source: §1 · pp. 1–2
• End-to-end interaction complements scalar proxy scores by exposing the execution process behind task success, including exploration, testing, and recovery.Source: §1 · p. 2
• ASAP preserves the base model’s completion interface and adapts agent scaffolds with structured trajectories, short reference prefixes, and minimal recovery.It evaluates without weight updates or conversion to a chat-style agent.Source: §1 · p. 2
• ASAP distinguishes base models and pretraining checkpoints across software-engineering and deep-research settings, including where conventional benchmark changes are marginal.Its execution trajectories support behavioral analysis of exploration, editing, testing, recovery, and task completion, as well as comparison with post-trained models.Source: §1 · p. 2

2 Related Work

• Prior approaches use static trajectory-derived tasks, fixed benchmark items, coding or state-tracking signals, and trajectory likelihood as lightweight proxies.These evaluate fixed examples or reference trajectories rather than behavior generated by the evaluated model.Source: §2 · pp. 2–3
• ASAP adapts existing agent scaffolds so pretrained base models execute original tasks end-to-end through their native completion interface.The pipeline uses a short reference-trajectory prefix, model continuation, and minimal recovery for premature empty submissions or repeated actions.Source: §2 · p. 3, Figure 1 · p. 3
• End-to-end execution also enables trajectory-level analysis across pretraining checkpoints and comparison of behavior before and after post-training.Source: §2 · p. 3

3 Methodology

• ASAP adapts existing agent scaffolds for native text completion without a chat template or system prompt, using structured trajectories, reference prefixes, and lightweight recovery.Patches leave the underlying task environment and scaffold logic largely unchanged; Pass@k measures solvability under repeated sampling, while Intersection@n measures instances solved in all n runs.Source: §3 · p. 3, §3.4 · p. 5

3.1 Markdown and XML Shell-Log Format

• Markdown+XML combines free-form thinking with XML-delimited actions and observations, and yields more stable trajectories than tested pure JSON, XML, or Markdown formats.A completion generates thinking and an action; the harness executes it, appends an observation, and continues until submission, the step limit, or a repeated-action loop.Source: §3.1 · pp. 3–4

3.2 Reference Trajectory Prefix Bootstrapping

• A short prefix from an instruction-tuned model’s reference trajectory helps the base model enter and navigate the interaction pattern without revealing the solution.The remaining trajectory is generated autonomously, and each instance uses the same prefix across evaluated models and checkpoints.Source: §3.2 · p. 4

3.3 Lightweight Recovery from Structural Failures

• For premature empty submissions or repeated actions, the harness gives one hint and recovery opportunity, then terminates if the failure recurs.It never inserts task-specific actions or edits on the model’s behalf; unrecovered trajectories are simply terminated.Source: §3.3 · pp. 4–5

3.4 Implementation & Evaluation

• Lightweight patches adapt SWE-agent, OpenHands, Terminus, and BrowseComp to base-model continuation while retaining their environments and largely unchanged scaffold logic.The evaluation uses Pass@k for attainable solvability and Intersection@n for consistency across all n runs.Source: §3.4 · p. 5

4.1 Experimental Setup

• The evaluation compares six open-source base models across SWE-bench Verified, Terminal-Bench 2.1, and BrowseComp-Plus under identical ASAP configurations.Each model is evaluated with eight independent rollouts.Source: §4.1 · p. 5, §4 · p. 5, §4.2 · p. 5

4.2 Main Results

• Pass@1 ranges from 7.12% to 37.70% with SWE-agent and from 4.00% to 39.23% with OpenHands on SWE-bench Verified; Terminal-Bench 2.1 spans 12.50%–26.69% and BrowseComp-Plus 14.64%–29.44%.All evaluated models achieve non-trivial success, and Qwen3.5-35B-A3B-Base and MiMo-V2.5-Base perform strongly across heterogeneous settings.Source: §4.2 · p. 6
• SWE-bench Verified Pass@1 rankings under SWE-agent and OpenHands correlate at Spearman ρ ≈0.89, but exact rankings and performance vary across scaffolds and domains.For example, Qwen3.5-9B-Base moves from 21.85% with SWE-agent to 32.95% with OpenHands, while Qwen3-30B-A3B-Base decreases.Source: §4.2 · p. 6
• MiMo-V2.5-Base reaches 37.70% Pass@1 and solves 82 instances in all eight SWE-agent runs, while Qwen3-30B-A3B-Base rises from 7.38% Pass@1 to 24.20% Pass@8 but solves none in all eight.Repeated sampling distinguishes occasional solvability from a stable core of reliably solved instances.Source: §4.2 · p. 6

5.1.1 Cross-stage performance on ASAP and general benchmarks

• SWE-bench Verified Pass@1 rises from 10.15% at stage2 end to 27.30% at stage3 end, then to 30.65% at stage5.The largest transition is stage2 to stage3; stage3-mid already reaches 22.95%, and stage4 reaches 29.95%.Source: §5.1.1 · pp. 7–8, Figure 2 · p. 8
• Within stages 1, 3, and 4, ASAP Pass@1 gains 2.23, 4.35, and 1.10 points, while conventional scores change by only +0.25, +0.09, and −0.13.Stage2 differs: conventional scores improve +2.31 points versus +0.73pp in Pass@1, with QA and instruction-following data but no agent trajectories.Source: §5.1.1 · p. 8

5.1.2 Behavioral decomposition

• From stage1 to stage2, parsable turns rise from 88.81% to 99.41% and valid tool calls from 79.78% to 97.06%, while only 10.15% of stage2 trajectories resolve tasks.This pattern indicates stronger interface reliability than agentic ability; autonomous editing and active testing improve relatively slowly.Source: §5.1.2 · p. 9
• From stage2 to stage3, autonomous edits reach 42.86%, active testing 65.64%, applicable patches 63.88%, and task resolution 27.30%.Parsability and valid tool calls remain high, while repetitive command loops and abnormal termination become less common.Source: §5.1.2 · p. 9, Figure 3 · p. 9
• Stages 4 and 5 show modest changes across capability dimensions, and trajectory-level analysis decomposes aggregate gains into behavioral changes unavailable to accuracy-only benchmarks.The authors describe this analysis as useful for diagnosing deficiencies and guiding pretraining-data design.Source: §5.1.2 · p. 9

5.2 Behavioral Differences Before and After Post-Training

• Across two model families, post-trained models more reliably revise after failed tests, avoid empty patches, submit final solutions, and align modified files with gold-patch scope.The comparison is descriptive rather than a scaffold-controlled estimate; Qwen-Base mainly struggles with recovery and completion, while MiMo-Base more often remains in exploration.Source: §5.2 · p. 10

• ASAP evaluates pretrained base models on end-to-end agentic tasks without weight updates or conversion to a chat-style interface, using execution trajectories to diagnose behavioral change.The framework adapts agent scaffolds through structured continuation, short trajectory prefixes, and minimal recovery mechanisms.Source: §6 · p. 10

• Table 4 lists the key configuration parameters used across ASAP evaluations.Source: §A.1 · p. 14, Table 4 · p. 14

• The example begins with a free-form reasoning segment about locating a relevant file and understanding a bug.Source: §A.2 · p. 14
• A bash action searches for Python files and filters for paths containing “models.”Source: §A.2 · p. 14
• The observation returns ./src/models.py and ./tests/test_models.py as matching paths.Source: §A.2 · p. 14

A.3 Full evaluation results

• Table 5 presents the section’s Pass@k results across agentic benchmarks.Source: §A.3 · p. 14, Table 5 · p. 15
• The table reports results for SWE-bench Verified with SWE-agent and OpenHands, Terminal-Bench 2.1 with Terminus-2, and BrowseComp-Plus with its native scaffold.Source: Table 5 · p. 15
• The table reports percentages, and #Inter counts instances resolved in all eight runs.Source: Table 5 · p. 15

A.4.1 Input Format

• Markdown+XML outperforms pure XML on Pass@1, rising from 27.00% to 33.00% for Qwen3.5-35B-A3B and from 38.00% to 42.00% for MiMo-V2.5-Base.Pure JSON performs particularly poorly, while Markdown remains below formats that explicitly delimit interaction fields with XML tags.Source: §A.4.1 · p. 16, Table 6 · p. 16

A.4.2 Prefix Length and Source Robustness

• Increasing prefix length consistently improves both models’ performance, while MiMo-V2.5-Base remains ahead of Qwen3.5-35B-A3B across prefix lengths and both Pass@1 and Pass@3.The default is K = 3; longer prefixes were limited because preliminary experiments found some exposed solution-bearing information, including parts of the gold patch.Source: §A.4.2 · pp. 16–17, Table 7 · p. 17
• Across two independently constructed prefix sets, the maximum absolute performance difference is 3 percentage points, and MiMo-V2.5-Base remains ahead of Qwen3.5-35B-A3B.The results indicate that the observed model separation is not specific to the default reference prefixes.Source: §A.4.2 · p. 17, Table 8 · p. 18

A.5 Consistency with Lightweight SFT

• Base and post-SFT model orderings match across base Pass@1, base Pass@8, post-SFT Pass@1, and post-SFT Pass@3, with Spearman ρ = 1.0 for all four pairwise comparisons.The authors treat this as an aggregate consistency check, not evidence that ASAP fully predicts post-trained performance.Source: §A.5 · pp. 17–18, Table 9 · p. 18

A.6 External Network Access Audit

• Across 24,000 audited trajectories, 0.6% attempt network access and 0.07% are flagged as potentially seeking a gold solution; the maximum corresponding rates for Qwen3-30B-A3B-Base are 2.15% and 0.23%.Under the pessimistic assumption that every gold-flagged trajectory succeeds solely through external lookup, its direct contribution to Pass@1 is at most 0.23 percentage points for any audited model.Source: §A.6 · pp. 18–19, Table 10 · p. 18

A.7 Failure Pattern Analysis

• Only 43.6% of Qwen’s non-empty submissions are correct, compared with 63.3% for MiMo; Qwen more often submits imprecise patches, while MiMo more often ends without a patch after exploration.Qwen’s failures include wrong-abstraction-level edits and regressions; MiMo’s selective editing is paired with difficulty transitioning from exploration to implementation.Source: §A.7 · p. 19, Table 11 · p. 19
• The failure profiles suggest targeting Qwen with repair-precision data and MiMo with trajectories that move from bounded exploration to focused editing, validation, and timely submission.The authors propose evaluating candidate continuations from the same repository state with the task grader to form preference pairs.Source: §A.7 · pp. 19–20