Source-linked AI summary
CodeMonkeys: Scaling Test-Time Compute for Software Engineering
Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, Azalia Mirhoseini
TL;DR
Selecting a final edit from many generated candidates remains a challenge for scaling test-time compute on real-world software issues. CodeMonkeys combines iterative edit-and-test trajectories with test-based and model-based selection; it achieves 57.4% on SWE-bench Verified for approximately $2,300 and also improves results when selecting over an ensemble of existing submissions.
Problem
Generating multiple candidate edits does not by itself provide an actionable way to improve issue solve rates; selection must distinguish correct from incorrect edits.
Source: §1 · p. 2Method
CodeMonkeys generates edits with tests through feedback-driven multi-turn trajectories, samples many candidate pairs, then combines test-based filtering with model selection.
Source: §1 · p. 2, §2.3 · p. 9Results
57.4% on SWE-bench Verified for approximately $2,300; selection over an ensemble of existing submissions scores 66.2%, above its best member's 62.8%.
Source: §1 · p. 3Takeaways & Limitations
The selection method can combine candidate edits from heterogeneous sources and outperform the best-performing member of the tested ensemble.
Source: §1 · p. 3The authors report that each stage of the system still has room for improvement.
Source: §3 · p. 10
Abstract
from arXiv · showhide
Scaling test-time compute is a promising axis for improving LLM capabilities. However, test-time compute can be scaled in a variety of ways, and effectively combining different approaches remains an active area of research. Here, we explore this problem in the context of solving real-world GitHub issues from the SWE-bench dataset. Our system, named CodeMonkeys, allows models to iteratively edit a codebase by jointly generating and running a testing script alongside their draft edit. We sample many of these multi-turn trajectories for every issue to generate a collection of candidate edits. This approach lets us scale "serial" test-time compute by increasing the number of iterations per trajectory and "parallel" test-time compute by increasing the number of trajectories per problem. With parallel scaling, we can amortize up-front costs across multiple downstream samples, allowing us to identify relevant codebase context using the simple method of letting an LLM read every file. In order to select between candidate edits, we combine voting using model-generated tests with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys resolves 57.4% of issues from SWE-bench Verified using a budget of approximately 2300 USD. Our selection method can also be used to combine candidates from different sources. Selecting over an ensemble of edits from existing top SWE-bench Verified submissions obtains a score of 66.2% and outperforms the best member of the ensemble on its own. We fully release our code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.
1 Introduction
CodeMonkeys addresses the challenge of turning many candidate edits into reliable SWE-bench solutions by combining serial and parallel test-time compute with candidate selection. It reports 57.4% on SWE-bench Verified for approximately $2,300 and shows that selection can improve results from an ensemble of existing submissions.
1 Introduction
- The paper asks how to design a system around test-time compute scaling, since generating candidates alone leaves the problem of selecting correct edits.Earlier coverage results did not provide an actionable recipe for improving issue solve rates.Source: §1 · p. 2
- The system scales serial compute through iterative edit-and-test revision from execution feedback, and parallel compute by sampling many edit-and-test pairs per issue.This combination achieves 69.8% coverage on SWE-bench Verified.Source: §1 · pp. 2–3
- Parallel sampling lets the system amortize codebase-context identification across downstream samples, with a model scanning every file as the chosen method.The scan contributes less than 15% of total costs when run once per issue.Source: §1 · p. 3
- 57.4% on SWE-bench Verified was achieved with the combined selection method at an inference cost of approximately $2,300.The approach combines voting with model-generated tests and model-based selection.Source: §1 · p. 3
- CodeMonkeys combines test-based voting to filter candidates with a model trajectory that selects among the remaining edits.The authors report that adding serial compute to selection by having models write and run distinguishing tests can improve selection.Source: §1 · p. 3
- 66.2% on SWE-bench Verified from selecting over the submission ensemble exceeded its best member's 62.8%.The ensemble comprised top leaderboard submissions and had 80.8% candidate coverage.Source: §1 · p. 3
2 Designing a SWE-bench Solver that Scales Test-Time Compute
The section frames SWE-bench Verified solving as context identification, candidate generation, and selection, then builds a solver that scales inference through iterative edits and multiple candidate trajectories.
2.1 Identifying Relevant Codebase Context
- Scanning and ranking relevant files yields 92.6% recall at a 128k-token limit, with 74,570 average context tokens and 50.5x compression.The system shares one codebase-wide scan across downstream edits to amortize its cost.Source: Figure 3 · p. 5, §2.1 · p. 5
2.2 Generating Candidate Codebase Edits with Corresponding Tests
- Testing scripts are drafted before editing, then revised alongside patches using pre- and post-edit execution feedback; selection writes tests to distinguish the top three candidates.The tests can verify that a script fails before an edit and passes afterward.Source: Figure 4 · p. 6, §2.2 · p. 7, §2.3 · p. 9
- Ten edit-test trajectories per instance, each capped at eight iterations, reach 69.8% coverage; beyond early iterations, similar-cost allocations yield similar coverage.Serial and parallel compute are not fully interchangeable: more candidates can make selection harder, while more iterations can leave prematurely approved, stuck trajectories unchanged.Source: Figure 5 · p. 7, §2.2 · p. 8
2.3 Selecting Between Candidate Edits
- The selection pipeline compares candidates using generated-test voting, model selection, top-3 filtering, and a multi-turn state machine that can write new distinguishing tests.Test-based voting narrows the candidate pool before the state machine chooses a final edit.Source: §2.3 · p. 9
- 57.4% SWE-bench Verified score with top-3 filtering plus the selection state machine, outperforming random selection and closing about half the random-to-oracle gap.The selection state machine is the most expensive selection method tested but contributes less than 10% of total system costs.Source: Table 2 · p. 8, Figure 6 · p. 8, §2.3 · p. 9
2.3.1 Barrel of Monkeys: Selecting over an Ensemble of Samples from Existing SWE-bench Submissions
- 66.2% ensemble score exceeds the best individual member, Blackbox AI Agent, at 62.8%.The Barrel of Monkeys ensemble combines CodeMonkeys edits with submissions from four leading SWE-bench Verified entries.Source: §2.3.1 · pp. 9–10
3 Limitations and Future Work
The authors identify remaining opportunities to improve context retrieval, candidate generation, and selection, and note that other test-time-compute approaches are outside this work’s scope.
Context
- The file-level filter-and-rank procedure omits relevant files for 7.4% of SWE-bench Verified instances.The retrieval assumption that base models know the repositories may not hold for new or private repositories poorly represented in training data, likely requiring retrieval-pipeline changes for high recall.Source: §3 · p. 10
Generating Edits and Tests
- Generation uses only issue-reproduction test feedback and positive sampling temperature for diversity, leaving other feedback and diversity strategies unexplored.The authors suggest regression tests could help prevent new bugs, while informing attempts of other state machines could encourage greater diversity and reduce redundant generations.Source: §3 · p. 10
Selection
- Selection recovers roughly half the random-to-oracle score gap on CodeMonkeys and an even smaller fraction on Barrel of Monkeys.The authors identify repository test suites as an unused selection signal; Agentless and the Gemini team include these tests in their selection procedures.Source: §3 · p. 10
Other Approaches
- The work omits ensembling different models and dedicated reasoning models trained to explore candidate solutions through extended chain of thought.The authors frame these as additional test-time-compute approaches beyond those studied here.Source: §3 · pp. 10–11
4 Related Work
Related work spans SWE-bench systems with varied workflows and tools, alongside approaches that scale inference through serial deliberation, parallel sampling, and selection. CodeMonkeys shares a structured issue-resolution pipeline with prior systems but uses distinct implementations.
AI for Software Engineering
- SWE-bench evaluates real-world GitHub issues from popular Python repositories, and reported resolution rates rose from below 5% initially to above 60% in current approaches.The passage attributes progress to stronger models and improved frameworks for equipping models with tools and guiding issue resolution.Source: §4 · p. 11
- Issue-resolution frameworks range from flexible tool use to structured state machines or ordered steps; Agentless also divides work into context identification, candidate editing, and selection.CodeMonkeys differs from Agentless in implementations including LLM-powered context scanning and multi-turn feedback loops for editing and selection.Source: §4 · p. 11
- Prior SWE-bench systems scale serial compute with reflection or tool feedback, and parallel compute with candidate edits selected using tests or models.SWE-search and CodeStory combine serial and parallel scaling through tree search over intermediate states with model-based value estimation.Source: §4 · p. 11
Scaling Test-Time Compute
- Test-time scaling has used tree search for games and mathematical proof, while longer deliberation and training models to deliberate have improved reasoning capabilities.The passage cites reinforcement learning as one way to optimize deliberation.Source: §4 · p. 11
- Parallel sampling can raise coverage, with execution-based testing aiding coding-sample selection; other methods use reward models, prompting-based verifiers, sequential attempts, or multiple model roles.PlanSearch converts some parallel samples into sequential attempts to improve diversity, while Archon, Mixture-of-Agents, and MALT study combinations of model components.Source: §4 · p. 12
A DeepSeek-V3 Results
On a 100-instance random subset of SWE-bench Verified, DeepSeek-V3 was evaluated as a cheaper alternative to Claude Sonnet 3.5. Claude scored higher, while the authors note the potential value of cheaper-model candidates when selection can identify correct samples.
DeepSeek-V3 Results
- 45.74% was Claude Sonnet 3.5’s score on the 100-instance subset, 6.02% higher than DeepSeek-V3’s best score.The DeepSeek API was over an order of magnitude cheaper than Claude’s; the authors note potential benefits from generating many cheaper-model candidates if selection can identify correct samples.Source: A DeepSeek-V3 Results · p. 17
B Experimental Details
The release includes the code and trajectories for CodeMonkeys, along with commands to reproduce the paper’s figures and tables.
Experimental Details
- The release provides CodeMonkeys code for SWE-bench Verified, figure-and-table generation commands, and complete problem-solving trajectories.The release also includes the code and trajectories at the paper’s project page.Source: B Experimental Details · p. 17
B.1 State Machine Details
The testing, editing, and selection state machines share an iterative structure: models receive execution feedback and either revise their work or approve it. Table 3 details the inputs and tasks for each machine.
B.1 State Machine Details
- Each state machine begins with an initial prompt and task before entering its feedback loop.Table 3 details the inputs and initial task for all three state machines.Source: §B.1 · p. 18, Table 3 · p. 18
- At each iteration, the model receives execution feedback from the previous iteration and may revise its work or approve it to terminate.Source: §B.1 · p. 18
B.2 Hyperparameters
Experimental details for each part are provided in Table 4.
B.2 Hyperparameters
- Table 4 summarizes CodeMonkeys hyperparameters.Source: §B.2 · p. 18
C Local Compute for Relevance
The relevance stage runs Qwen-2.5-Coder-32B-Instruct locally and estimates compute throughput and cost from hardware, model, and token-processing assumptions.
C Local Compute for Relevance
- The relevance stage runs Qwen-2.5-Coder-32B-Instruct locally in bf16 across 8xL40S GPUs.Each L40S has a bf16 compute throughput of 362.05 TFLOPS.Source: C Local Compute for Relevance · p. 18
- For a dense transformer without parameter sharing, FLOPs per token are estimated as 2 * num parameters, with hardware utilization assumed to be 20%.The paper uses this estimate to calculate per-node throughput.Source: C Local Compute for Relevance · p. 19
- The relevance stage processes 1.32084 · 109 tokens across 500 SWE-bench Verified instances, and its compute cost is estimated using RunPod’s $8.24-per-hour price for an 8xL40S node.The cost calculation uses the node’s estimated throughput and processing time.Source: C Local Compute for Relevance · p. 19