PaperlayerView exact revision on arXiv

Source-linked AI summary

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Elad Sharony, Ido Greenberg, Jialin Song, Shie Mannor, Gal Chechik, Eli Meirom

arXiv:2610.15894v1cs.LGcs.AIcs.SE

TL;DR

Code-repair search must choose which proposed edit receives a costly generation and test attempt, while pass rates alone can obscure failure details and repair distance. CodeTriage ranks edits using failure context and proposed-edit text, predicting solve-distance distributions before generating revised programs. It finds fixes with fewer revisions and output tokens than the strongest baselines on those metrics, and transfers to unseen repair LLMs without retraining.

  • Problem

    Existing repair search policies use pass rates and search history, although equal pass rates can conceal different failures and distances from a fix.

    Source: §1 · p. 1

  • Method

    CodeTriage uses a 0.6B priority model to read test output, failure diagnosis, and proposed edits, predicting solve-distance distributions to rank edits before generation and testing.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On complete repair searches, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric, and transfers without retraining to eleven unseen repair LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Separating repair proposal from semantic prioritization offers a practical way to allocate test-time computation without proposer-specific tuning.

    Source: §8 · p. 9

    Experiments demonstrate generalization on held-out Python tasks and transfer across 11 unseen repair LLMs; other languages and repository-scale repair would extend the demonstrated scope.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce CodeTriage, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.

1 INTRODUCTION

Because each repair attempt incurs program generation and testing, efficient repair depends on choosing which proposed edit receives that costly attempt. CodeTriage uses failure context and edit text to rank repairs across programs, reducing search cost on held-out tasks.

1 INTRODUCTION

  • Code-repair attempts require generating and testing revisions, so under a fixed budget success depends on which proposed repair receives the next attempt.Prior search policies use pass rates and refinement counts, but equal pass rates can conceal different failure causes and repair distances.Source: §1 · p. 1
  • CodeTriage uses a 0.6B-parameter priority model to rank proposed edits before revised programs are generated and tested.It reads test output, failure diagnosis, and proposed-edit text to estimate the likelihood and number of repair steps to a fix; the repair LLM still proposes and implements edits.Source: §1 · p. 2
  • On complete repair searches, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric.Without retraining, it also achieves the highest success rate averaged across budgets for eleven unseen repair LLMs.Source: §1 · p. 2
  • The paper releases a 12.2-million-node repair-tree corpus that records alternative repair paths and their outcomes for research on code-repair search.The corpus supports studying repair choices beyond the policy introduced here.Source: §1 · p. 2

2 PROBLEM SETTING

The paper models code repair as a search over proposed edits, where each expansion generates and tests a revised program. A policy selects among the unexpanded edits using only information already observed, with the goal of finding a solution in as few expansions as possible.

2 PROBLEM SETTING

  • Each repair-tree node is a program tested against the full suite, and expanding an edit generates and tests a child program at the cost of one edge expansion.The proposer supplies k edits describing what to change and why, before their revised programs are generated.Source: §2 · p. 2
  • The frontier contains every proposed edit not yet tried, including alternatives from previously explored programs; their programs and pass rates remain unknown.Figure 1 distinguishes the observed program state and proposed edits from these unexpanded frontier edges.Source: Figure 1 · p. 2, §2 · p. 3
  • A search policy assigns priorities to frontier edits, and each expansion reveals the child’s program and test results while adding its proposed edits to the frontier.The objective is to minimize expansions until the first solution is found.Source: §2 · p. 3
  • The policy may use observed programs and feedback, proposed-edit text, and search history, but not outcomes of unexpanded edits; CodeTriage does not change the proposer.Exhaustive-tree evaluation replays policies on the same precomputed candidates to isolate expansion order, complemented by online search with eleven unseen proposers.Source: §2 · p. 3

3 CODETRIAGE: READING THE FRONTIER AS SURVIVAL DISTRIBUTIONS

CodeTriage predicts how many expansions an edit’s branch needs to reach a fix, accounting for incomplete observations at the depth limit. It turns the predicted distribution into a depth-normalized priority and expands the highest-scoring frontier edit.

3 CODETRIAGE: READING THE FRONTIER AS SURVIVAL DISTRIBUTIONS

  • Solve distance counts the edge expansion plus the shortest successful path from its child; a solving child has distance 1.When the child is at depth d in a tree of maximum depth D, the largest observable solve distance is b = D − d + 1.Source: §3 · p. 3
  • The model predicts probabilities over solve distances and a tail probability beyond D; if a branch reaches the depth limit without a fix, training treats its distance as censored beyond the remaining depth b.The censored likelihood sums probability mass beyond b rather than treating the branch as permanently unsolvable.Source: §3 · p. 3
  • The priority averages cumulative probabilities of finding a fix within each available number of steps, normalized by remaining depth b.This favors likely short paths, ignores distances beyond the remaining depth, and leads deterministic search to expand the frontier edge with the largest score.Source: §3 · pp. 3–4
  • A 0.6B-parameter decoder-only model reads parent feedback, diagnosis, and proposed-edit text, then combines its last-token state with four parent-level numeric features to predict solve-distance probabilities.The features are pass rate, normalized depth, failed-test fraction, and normalized program length; an MLP head emits the distribution.Source: §3 · pp. 3–4, Figure 2 · p. 4

4 EXPERIMENTAL DETAILS

The experiments build exhaustive repair trees from Python problems, then compare search policies on held-out tasks and test whether a policy trained with one proposer transfers to unseen repair LLMs.

Dataset

  • The main and auxiliary corpora contain 12.2 million executed nodes in exhaustive repair trees recording proposed edits, resulting programs, and test results.Trees are retained when they contain at least one reachable fix; depth-limited branches supply lower bounds on solve distance.Source: §4 · p. 4

Data splitting

  • All trees derived from a programming problem stay in the same train, validation, or test split, so test problems are unseen during training and validation.Ablations and diagnostics use a separate validation split of 4,996 trees.Source: §4 · pp. 4–5

Baselines and model

  • Policies search the same proposed edits, isolating how they allocate expansions; baselines include random, BFS, DFS, pass-rate, REx, AB-MCTS, and BG-MCTS.REx and BG-MCTS are tuned on validation, while the stronger AB-MCTS variant is reported for each metric; CodeTriage uses a LoRA-trained Qwen3-0.6B-Base priority model.Source: §4 · p. 5

Evaluation protocol

  • Offline replay uses 5,001 exhaustive trees from 982 held-out tasks, while transfer tests the unchanged policy with eleven unseen repair LLMs.For offline replay, each tree contains a fix within the maximum repair depth; transfer retains starting programs regardless of solvability and weights proposers equally.Source: §4 · p. 5

5 RESULTS

Across repair-tree evaluation, transfer, ablations, and scaling experiments, CodeTriage reduces search costs, transfers across unseen proposers, and benefits from edit text and comparisons across the search frontier.

5.1 FEWER EXPANSIONS AND OUTPUT TOKENS

  • 21% fewer expansions than AB-MCTS (11.54 versus 14.59) and 12% fewer output tokens than REx on the same test roots.CodeTriage also uses 30% fewer expansions than random search; Figure 3 reports mean expansions and output tokens through the first solution.Source: §5.1 · p. 5, Figure 3 · p. 6

5.2 TRANSFER TO UNSEEN REPAIR LLMS

  • Across eleven unseen proposers, CodeTriage improves mean AUSC(36) from 81.60 to 82.50, with higher scores than the best baseline for all eleven.Gains are statistically significant for nine proposers; Figure 4 shows success across budgets and AUSC gains over each proposer’s strongest baseline.Source: Figure 4 · p. 6, §5.2 · p. 7

5.3 ABLATIONS

  • Removing parent-state text raises cost 16%, removing proposed-edit text raises it 22%, and removing both raises it 45% versus the full model.On trees from three additional proposers, either text input cuts cost by 14–21% relative to the matched no-text model, and using both is best for every proposer.Source: §5.3 · p. 7
  • Global ranking reduces mean search cost from 13.32 to 12.20 expansions, while CodeTriage achieves the lowest tested cost at 11.79 expansions.Using only the probability of a solution within remaining depth raises cost 4%, and using only the next-expansion solution probability raises it 15%; removing depth normalization raises cost 8%, while per-depth ranks raise it 18%.Source: §5.3 · p. 7

5.4 SEARCH PERFORMANCE ACROSS TRAINING-DATA AND MODEL SIZES

  • Increasing training data from 1% to 100% reduces E[T] by 19%, from 14.67 to 11.90 expansions, and 75% of the data is within 1% of the full-data result.Training on all four proposer corpora reduces mean cost 3.5%, from 8.29 to 8.00 expansions; the deep Qwen3-Coder-30B subset matches the full corpus at 8.291 versus 8.290.Source: §5.4 · pp. 7–8, Figure 5 · p. 8
  • The deployed 0.6B model uses 7% more expansions than the 14B model (11.87 versus 11.10) despite having more than 20× fewer parameters.Across five model families, Qwen3.5-2B uses 5% fewer expansions than the deployed model (11.31), while SmolLM2-360M uses 7% more (12.70).Source: Figure 5 · p. 8, §5.4 · p. 8

6 RELATED WORK

Prior work guides code-repair search through scalar rewards, search statistics, or learned evaluations; CodeTriage instead uses repair content and solve-distance predictions to prioritize proposed edits.

Adaptive search over stochastic program candidates

  • REx uses Thompson sampling with pass-rate-informed Beta posteriors, while AB-MCTS and BG-MCTS adapt search using reward posteriors or remaining budget [Miyamoto et al., 2026].These expansion rules use scalar evaluations and search-state statistics.Source: §6 · p. 8

Learned search priorities

  • Learned search includes distance-to-go heuristics, successor action values, and process or outcome scoring [Uesato et al., 2022; Cobbe et al., 2021].Tree-of-Thoughts, RAP, and LATS use model-based evaluations [Zhou et al., 2023], while DREAM scores plans by predicted success.Source: §6 · p. 8
  • CodeTriage learns solve-distance distributions from depth-censored branches to prioritize proposed edits across the full frontier.In depth-limited repair trees, solved branches reveal solve distance and branches without an observed fix provide lower bounds, supplying supervision from both.Source: §6 · p. 8

Code repair and execution feedback

  • CodeTree guides code-generation search with execution feedback and LLM critiques, while SWE-Search combines MCTS with an LLM value agent.Unlike approaches that improve subsequent attempts through feedback, CodeTriage keeps the proposer fixed and learns which proposed edits to expand.Source: §6 · pp. 8–9

7 LIMITATIONS AND FUTURE WORK

The demonstrated scope is held-out Python repair with transfer across eleven repair LLMs; cross-branch information sharing and candidate generation conditioned on prior decisions are outside the work’s scope.

Task and language scope

  • Experiments demonstrate generalization on held-out Python tasks and transfer across eleven repair LLMs; other programming languages and repository-scale repair remain extensions.The selection interface uses execution feedback and proposed-edit descriptions.Source: §7 · p. 9

Information sharing and longer horizons

  • Sharing information across branches and generating additional candidates in response to prior decisions fall outside the work’s scope.The value of deeper training trajectories motivates learning over longer repair horizons.Source: §7 · p. 9

8 CONCLUSION

CodeTriage prioritizes proposed repairs across programs and depths, reducing search costs while transferring to unseen repair LLMs. The paper also releases a large repair-tree corpus and reports how repair difficulty varies across proposers.

8 CONCLUSION

  • CodeTriage requires 21% fewer expansions and 12% fewer output tokens than the strongest baseline, and its frozen policy achieves the highest average success across budgets for eleven unseen proposers.Ablations support both semantic inputs and global comparison across parents and depths.Source: §8 · p. 9

A.1 FOUR-PROPOSER CORPUS

  • The released corpus contains 12.2 million repair-tree nodes; the four-proposer corpus covers 12,187,444 nodes over the same 45,424 root programs.All four corpora use the same task-level split.Source: §8 · p. 9, §A.1 · p. 13

8 CONCLUSION

  • Separating repair proposal from semantic prioritization offers a practical way to allocate test-time computation without proposer-specific tuning.Descriptions of unexecuted actions serve as a control signal for search.Source: §8 · p. 9

A DATASET AND SPLITS

  • Problems, rather than trees, are assigned to the 70/15/15 train, validation, and test splits, keeping prompt styles for one problem together.The split contains 4,537/986/982 canonical tasks.Source: A DATASET AND SPLITS · p. 13

A.1 FOUR-PROPOSER CORPUS

  • Across proposers, solve-distance correlations range from 0.52 to 0.68: difficult programs tend to remain difficult, but proposer choice changes substantial parts of the ordering.The two Qwen3-Coder models correlate at 0.570.Source: §A.1 · p. 13

B TRAINING-PROPOSER TRANSFER ANALYSIS

Training-source effects grow with repair difficulty, while deeper and multi-proposer training can improve search across evaluation proposers. The section also details the deployed model and adaptations used for baseline comparisons.

B TRAINING-PROPOSER TRANSFER ANALYSIS

  • Across off-diagonal training–evaluation pairs, the mean transfer penalty rises from 0.11 to 3.33 expansions as root solve distance increases from 1 to 4.The transfer table compares mean expansions to first fix, with lower values better.Source: B TRAINING-PROPOSER TRANSFER ANALYSIS · p. 14, Table 3 · p. 14
  • On distance-4 roots from other corpora, Qwen3-Coder-30B training has a mean transfer penalty of −0.01 expansions, versus 3.72–5.53 for other training corpora.The table defines penalties relative to training on the evaluation corpus; negative values indicate improvement.Source: B TRAINING-PROPOSER TRANSFER ANALYSIS · p. 14, Table 4 · p. 14
  • Deep rather than shallow supervision saves 0.38–2.91 expansions on distance-4 roots across all four evaluation proposers, with similar costs on immediately solved roots.The deep stratum includes roots with solve distance 3 or 4, or no fix within the tree.Source: B TRAINING-PROPOSER TRANSFER ANALYSIS · p. 14, Table 5 · p. 15
  • Training on all four corpora lowers mean validation search cost from 8.29 to 8.00 expansions and yields the lowest cost on three evaluation proposers.On the fourth proposer, it is within 0.01 expansions of that proposer’s corpus-trained model.Source: B TRAINING-PROPOSER TRANSFER ANALYSIS · p. 15, Table 6 · p. 15

C TRAINING AND ARCHITECTURE DETAILS

  • The deployed priority model uses a frozen Qwen3-0.6B-Base backbone with LoRA adapters and combines parent feedback, diagnosis, edit text, and four parent-level numeric features.The four numeric features are constant across siblings, while edit text is the only sibling-varying language-model input.Source: C TRAINING AND ARCHITECTURE DETAILS · p. 15

D BASELINE ADAPTATION AND TUNING

  • For exhaustive-tree baseline adaptation, exhausted REx and AB-MCTS arms are removed, while BG-MCTS’s completion bonus is zero because revealed nodes are complete programs.The policies are tuned on validation only.Source: D BASELINE ADAPTATION AND TUNING · p. 16

E COMPLETE EXHAUSTIVE-TREE RESULTS

The exhaustive-tree results compare search policies on shared held-out trees, and the ablations examine search cost and the contribution of repair-text inputs.

E COMPLETE EXHAUSTIVE-TREE RESULTS

F MECHANISM ABLATIONS

  • Removing repair text or replacing distributional priorities increases mean expansions to the first solution relative to each block’s full-model reference.The ablation summary defines lower E[T] as better and reports matched-seed means with standard errors.Source: Table 9 · p. 17

F.1 INPUTS

  • Against the no-text policy, the full combination of parent-state and edit text saves 5.35±0.04 expansions; removing parent-state text raises cost by 1.84, and removing edit text by 2.59.Parent-state text combines execution feedback and diagnosis.Source: §F.1 · p. 17, Table 10 · p. 17

F.2 INPUT ABLATIONS UNDER PROPOSER SHIFT

The ablations test which inputs and ranking choices improve CodeTriage’s search, including under proposer shift. They also compare training objectives, a cheaper text baseline, and scaling configurations.

F.2 INPUT ABLATIONS UNDER PROPOSER SHIFT

  • Under proposer shift, parent-state text saves 1.33–2.19 expansions and edit text saves 1.38–1.95; the full model is strongest for every proposer.Each row uses the same deterministic 2,000-root subsample for every variant.Source: §F.2 · p. 18, Table 11 · p. 18

F.3 OBJECTIVES

  • Distance-MSE, Soft-BCE, sibling ranking, and global ranking use different targets or comparisons to train edit priorities.Global ranking uses depth-normalized priorities as targets; sibling ranking compares children of one solvable parent, while global ranking compares frontier edits.Source: §F.3 · p. 18, §F.4 · p. 18

F.4 GLOBAL-FRONTIER RANKING

  • Global comparisons reduce mean search cost from 13.32 to 12.20 expansions, while CodeTriage reaches 11.79 expansions.The eight-edge variant reaches 12.17 expansions; the remaining paired difference between global ranking and CodeTriage is not statistically resolved with three seeds.Source: §F.4 · pp. 18–19, Table 13 · p. 19

F.5 CHEAP TEXT BASELINE

  • The learned priority model costs 16% less than hashing TF–IDF plus logistic regression on identical inputs, measured as 2.18 expansions.The text baseline recovers substantial value from text.Source: §F.5 · p. 19, Table 14 · p. 19

G PRIORITY-RULE ABLATIONS

  • No tested priority rule improves on A/b; removing its depth normalizer raises search cost by 0.92 expansions, while replacing score magnitudes with within-depth ranks raises it by 2.16.Score magnitudes help compare edits across depths, beyond ordering edits at the same depth.Source: G PRIORITY-RULE ABLATIONS · p. 19, Table 15 · p. 20

H SCALING TABLES

  • The supplied scaling passages identify training-data and backbone scaling analyses, but provide no outcome values or comparisons.The backbone sweep applies the same distributional objective and priority across the listed model families.Source: H SCALING TABLES · p. 20

I CROSS-PROPOSER EVALUATION PROTOCOL AND RESULTS

The paper evaluates a frozen priority model trained on Qwen3-Coder-30B repairs across eleven unseen proposers, alongside calibration and inference-cost analyses. CodeTriage improves aggregate search performance and illustrates how it can find a repair through a branch with lower initial test pass rates.

I CROSS-PROPOSER EVALUATION PROTOCOL AND RESULTS

J CALIBRATION

  • Including censored edges reduces conditional timing ECE from 0.069±0.010 to 0.027±0.006 and unconditional solve-within-depth ECE from 0.172±0.006 to 0.068±0.004.All three matched seeds improve on both measures; Figure 7 distinguishes the conditional and unconditional calibration quantities.Source: J CALIBRATION · p. 23, Figure 7 · p. 23

K PRIORITY-MODEL INFERENCE COST

  • Scoring a complete search costs about 0.51 proposer expansions, while CodeTriage saves 3.06 expansions against AB-MCTS-β.The estimate puts the avoided generation at roughly six times the scoring cost, before counting saved test executions; shared parent-state encoding increases throughput 2.96×, from 123 to 363 edges/s on one H100.Source: K PRIORITY-MODEL INFERENCE COST · p. 24, Table 18 · p. 24

L QUALITATIVE REPAIR EXAMPLE

  • The held-out Qwen3-Coder-30B example finds a reference-consistent repair through a branch with lower initial pass rates than its siblings.The figure compares CodeTriage with the best adaptive baseline’s mean over three inference seeds.Source: L QUALITATIVE REPAIR EXAMPLE · p. 25, Figure 8 · p. 26

L.1 RECOVERING THE TIME AND CAPACITY CONSTRAINTS

  • CodeTriage finds the first test-passing program in 14 expansions, versus 40.7 for AB-MCTS-N; the successful branch starts from a sibling passing only 3/7 tests.The two depth-1 siblings passing 5/7 tests have no passing descendant within D = 4; CodeTriage returns to the 3/7 branch and follows p3 → p6 → p13 → p14.Source: §L.1 · pp. 25–26

Related papers

Loading 2610.15894v1…