PaperlayerView exact revision on arXiv

Source-linked AI summary

Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents

Kang Chen, Junjie Nian, Yixin Cao, Yugang Jiang

arXiv:2608.22191v1cs.AIcs.SE

TL;DR

Repository-level agents must coordinate correlated sibling actions and independently sampled attempts even though software patches lack canonical answer forms. Risa uses native MoE routing with decision-matched comparison sets for exploration, commitment, and final-patch selection without an external judge or selection-time test execution. It raises the gpt-oss macro-average from 44.9% to 48.2%, matches text consensus, and transfers to Qwen3.6 on the full benchmark.

  • Problem

    Repository repair involves nested coordination over correlated same-prefix actions and independently developed non-canonical patches.

  • Method

    Risa uses routing-derived behavioral roles, history-based novelty, guarded peer convergence, and decision-token agreement across separately sampled attempts.

  • Results

    Risa improves over Uniform across six gpt-oss conditions, lifting the macro-average from 44.9% to 48.2%, and transfers to Qwen3.6 while matching Text.

  • Takeaways & Limitations

    Routing acts as a practical coordination signal for nested software-agent decisions across MoE families.

  • Takeaways & Limitations

    Risa assumes accessible sparse-MoE routing and repeated trajectories, so its current instantiation is naturally suited to white-box MoE agents.

Abstract

from arXiv · show

Software-engineering agents solve repository-level tasks through long, stochastic tool-use trajectories, and repeated attempts often find fixes missed by one run. Test-time scaling is difficult because patches lack canonical answer forms, while sibling actions from a shared prefix are correlated. We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution. Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control. We therefore introduce Risa (Routing-Informed Steering and Arbitration): within trajectories, routing encourages diverse exploration and controlled convergence during patch commitment; across separately sampled trajectories, agreement at informative patch positions selects a final candidate. We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings. Risa's routing arbitration raises the macro-average resolved rate from 44.9% under uniform sampling to 48.2% on the gpt-oss family, matching text consensus without answer-string matching, and it transfers to Qwen3.6, where it improves on uniform choice and matches text consensus on the full 500-task benchmark.

1 Introduction

The introduction frames repository repair as two nested coordination problems and proposes Risa, which matches routing comparisons to each decision. Across gpt-oss conditions, Risa improves resolved rate over uniform sampling and transfers to another MoE family.

  • Repository repair requires coordinating correlated same-prefix actions and independently developed non-canonical patches.These nested choices make ordinary best-of-N selection insufficiently matched to candidate dependence.
  • Native MoE routing traces provide a shared internal coordinate for comparing textually different candidates and behavioral roles.The traces record selected experts and weights for every token and layer, supporting relational comparisons across actions.
  • Risa uses role-conditioned exploration, controlled patch commitment, and decision-token agreement to steer trajectories and select final patches without external judging.Its comparison sets differ by decision: recent history for exploration, peer convergence for writing, and separately sampled attempts for terminal selection.
  • 48.2% macro-average resolved rate lifts gpt-oss performance from 44.9% under Uniform, while Text reaches 48.0%.Across six model–effort conditions, Risa improves over Uniform by 2.3–5.7 points.
  • Risa is evaluated across two model scales, three reasoning-effort settings, and a second MoE family, improving over Uniform in every condition.The evaluation operates directly on routing traces rather than an external selection mechanism.

2 Related Work

Related work has used external feedback, verifiers, semantic or likelihood comparisons, and internal-state signals for test-time scaling and software repair. This paper instead applies native MoE routing to heterogeneous long-horizon trajectories and non-canonical patches.

  • Prior work studies lexical and semantic routing specialization, but not how routing meaning changes across nested choices in long tool-use trajectories.
  • Internal-state and sparse-neuron methods rank candidate quality or support label-free best-of-N selection, whereas this work maps routing across action, trajectory, and attempt scales.
  • Self-consistency, likelihood-based methods, language-model judges, and confidence-guided scaling address repeated free-form outputs through text or external selection signals.
  • Software-repair systems use repository tools, trajectory reuse, summaries, verifiers, entropy, or tests; Risa complements them with routing comparisons over histories, siblings, and independent attempts.

3 What Routing Reveals Across Agent Decisions

The paper turns sparse MoE router traces into comparable routing fingerprints and tests what they reveal about agent behavior, progress, patch decisions, and candidate selection. It finds that behavioral role, decision-token localization, and decision-matched comparison sets make routing informative across nested agent choices.

  • 3.1 From Router Rows to Comparable Fingerprints: Routing fingerprints integrate token-level expert weights into normalized layer-by-expert profiles, compared with weighted Jaccard across candidate spans.The profiles remove span-length effects and represent where the MoE allocated computation.
  • 3.2 Insight 1: Routing Strongly Encodes Behavioral Role: Routing separates inspection, search, execution, and patch-writing actions, achieving 0.940 three-way holdout accuracy versus a 0.746 majority-class baseline.Write-versus-rest recall is 0.93 at precision 1.00, supporting a high-precision exploratory-versus-commitment gate.
  • 3.3 Insight 2: Routing Tracks Computational Revisitation: Resolved rate falls from 33.8% in the most-different repetition quintile to 5.4% in the most-similar, linking routing-space revisitation with stagnation.Each executed action is compared with its three closest fingerprints among up to the previous 64 actions.
  • 3.4 Insight 3: Decision Tokens Preserve Fine-Grained Routing Information: On long outputs, context-identity AUC is 0.93 for decision-token fingerprints, compared with 0.88 for whole-span and 0.98 for opening-window fingerprints.Whole-span fingerprints mix increasingly many expert pathways as occupancy rises, while trailing windows reach 0.58 AUC.
  • 3.4 Insight 3: Decision Tokens Preserve Fine-Grained Routing Information: Decision tokens concentrate a median 72% of total token surprisal in the least-probable quarter, providing a compact window for routing variation.Mean cross-attempt agreement is 0.65 at decision tokens and 0.56 elsewhere; decision-token routing reaches AUC 0.69 on shorter candidates.
  • 3.5 Insight 4: Comparison Sets Determine Agreement Meaning: Across separately sampled attempts, decision-token agreement ranks resolving above non-resolving attempts with AUC 0.66, whereas same-prefix siblings require history-relative comparison.Risa therefore favors disagreement with recent computation for exploration and agreement across trajectories for commitment.

4 Method: Risa

Risa uses routing traces to coordinate exploration, patch writing, and final patch selection with comparison rules matched to each decision. It steers sibling actions within attempts and arbitrates among separately sampled final diffs without new candidate execution.

  • Within-attempt steering: Role centroids classify candidates as inspect/execute, test, or write, then collapse these roles into exploratory versus write behavior.The centroids are fit per model configuration from parsed tool-call labels and frozen before evaluation.
  • Within-attempt steering: At each step, Risa samples 16 sibling generations, fingerprints only their proposed tool-invocation spans, and compares them with the last 64 selected actions.The history stores selected and executed action fingerprints, allowing routing-based comparison of proposed actions rather than preceding reasoning.
  • Within-attempt steering: Exploratory candidates maximize disagreement with recent executed history, while a cohort of at least two write candidates activates guarded peer-convergence scoring.The commitment score combines local support and cohort centrality with entropy and peak-mass dispersion guards.
  • Cross-attempt arbitration: After four attempts, Risa re-encodes each nonempty final diff and selects the patch with the highest mean decision-token agreement across attempts.This terminal arbitration adds one teacher-forced prefill per available patch but no candidate generation or execution.
  • Evaluation setup: Table 2 compares terminal selectors on common Risa-steered K=4 pools, conditioning rates on tasks with at least two graded patches.The table reports 496–498 eligible tasks per gpt-oss condition and 498 for the Qwen3.6 condition.

5 Experiments

On SWE-bench Verified, routing-based arbitration improves over uniform selection across gpt-oss settings and transfers to Qwen3.6 while matching text consensus. The fixed-pool design separates terminal arbitration from candidate generation, and decision-token readouts provide the strongest terminal signal.

  • Main results: Risa gains 2.3–5.7 points over Uniform across six gpt-oss conditions, raising the macro-average from 44.9% to 48.2%.Risa-H reaches 48.3% and Text reaches 48.0% on the same macro-average.
  • Cross-family transfer: Risa resolves 45.2% of eligible Qwen3.6-35B-A3B tasks versus 41.7% for Uniform and 45.0% for Text on the 498-task condition.Both Risa and Text significantly outperform Uniform, while Risa matches Text by exact McNemar testing.
  • Robustness and controls: Routing improves over Uniform in every reported condition while candidate availability varies from 79.1% to 99.5% Yield.All terminal selectors receive the same Risa-steered attempt pools, isolating arbitration effects.
  • Comparison with text consensus: Risa and Risa-H slightly exceed Text on the gpt-oss macro-average and match Text on the full Qwen condition.The gpt-oss macro-averages are 48.2% and 48.3% for Risa and Risa-H versus 48.0% for Text; Qwen results are 45.2% versus 45.0%.
  • Terminal readout: Decision-token routing ranks mixed-outcome attempts better than whole-span and matched controls, and the localized rule gains in every reported condition.The same rule transfers to Qwen as a significant 3.5-point gain over Uniform.
  • Steering diagnostics: On a hard 80-instance set, steering raises submittable-patch yield from 79% to 94%.On a fixed 200-task 20b subset, the full pipeline reaches 50.5% versus 45.4% for unguided generation with Uniform selection.

6 Conclusion

Risa treats sparse-MoE routing as a reference-dependent computational coordinate for coordinating nested software-agent decisions. Its routing arbitration improves over uniform selection across gpt-oss conditions and transfers to Qwen3.6 while matching text consensus.

  • Conclusion: Risa uses routing novelty for exploration, guarded peer support for writing, and decision-token agreement for final patch selection.These rules coordinate decisions within trajectories and across independently developed patches.
  • Conclusion: Across six gpt-oss conditions, routing arbitration gains 2.3–5.7 points over Uniform and reaches a 48.2% macro-average versus 48.0% for Text.On the full Qwen benchmark, it reaches 45.2% versus 41.7% for Uniform and 45.0% for Text.
  • Conclusion: The results establish routing as a practical coordination signal that transfers across MoE families.The transfer claim is supported by the reported gpt-oss and Qwen benchmark results.

7 Limitations

Risa’s current instantiation is scoped to white-box sparse-MoE agents with accessible routing and repeated trajectories. Extending the principle to dense, closed, or other-domain systems requires alternative readouts or role definitions.

  • Scope boundary: Risa assumes accessible sparse-MoE routing and repeated trajectories, making the current method naturally suited to white-box MoE agents.This is the paper’s stated scope boundary for the present instantiation.
  • Extension boundary: Dense or closed models require alternative readouts, while other domains require role definitions suited to their action spaces.The paper identifies combining routing with semantic or execution-based evidence as a future direction for rare outlier repairs.

A Implementation Details

The implementation uses sparse-MoE routing traces to classify actions, steer candidate selection, and arbitrate among final patches under fixed evaluation protocols.

  • Sampling protocol: Each step samples 16 sibling actions from one prefix, while each task produces four independently sampled attempts.Siblings share an immediate prefix; attempts may follow different repository paths.
  • Routing representations: Routing fingerprints accumulate layer–expert gate mass over action spans, and cosine similarity to fitted centroids predicts exploratory, test, or write roles.Inspect and execute actions are collapsed into exploration, while centroids are fit separately for each architecture.
  • Within-attempt control: Exploration selects candidates least similar to recent executed actions, using up to 64 history fingerprints and a sibling-medoid fallback.The closest three historical matches determine each candidate’s average similarity.
  • Write control: Write selection uses fixed standardized routing statistics, including entropy, peer similarity, centrality, and concentration guards.The coefficients were fixed before the main campaign from a separate pilot and were not optimized per task or model.
  • Terminal arbitration: Terminal arbitration re-encodes each available final patch once and compares routing agreement across attempts without re-executing patches.The selector adds one prefill and one discarded token per available patch.

C Terminal Selection-Rule Ablations

The ablations separate routing-gated candidate generation from terminal arbitration and compare routing signals with parsed roles, surface consensus, and uncontrolled sampling.

  • Ablation design: The fixed-pool ablation gives every selector the same Risa-steered attempts, isolating terminal arbitration from candidate generation and step budget.Every rule makes a realized 1-of-K choice on the same candidate pools.
  • Role-gate comparison: The routing gate agrees with an exact parsed-role gate on 96.8% of 130,464 candidate actions.99.8% of candidates contain a parseable invocation, making the two gates near-interchangeable in labeling accuracy.
  • Write-step selection: Routing-gated write cohorts resolve 50.5% of mixed-outcome step branches versus 49.2% for same-step uniform choice.The +1.3-point difference has a step-bootstrap interval of [−2.5, +5.5].
  • Score components: Mean peer similarity is the clearest standalone write-score signal, while entropy and concentration terms act as stability guards.All write-score coefficients remain fixed across tasks, models, and effort settings.
  • Pipeline composition: End to end, steered Risa reaches 50.5% versus 45.4% for uncontrolled Uniform at 20b on the fixed 200-task subset.At 120b, steered Risa-H reaches 55.3% versus 53.5% for uncontrolled Uniform.

G.1 Full-Benchmark Evaluation

On the full SWE-bench Verified benchmark, routing arbitration improves over Uniform and matches text consensus across the Qwen3.6 cross-family evaluation.

  • Evaluation scope: The cross-family evaluation uses Qwen3.6-35B-A3B on 498 eligible tasks with four attempts and 16 candidates per step.The result is reported separately from the six-condition gpt-oss grid.
  • Full-benchmark result: Risa improves over Uniform by 3.5 points on the full benchmark, with a 95% CI of [+1.9, +5.0] and p<0.001.Text improves over Uniform by 3.3 points with a 95% CI of [+1.7, +4.8].
  • Routing versus text consensus: Risa matches Text through 10 routing-only wins and 9 text-only wins, with exact McNemar p=1.000.Risa-H attains the highest deployable resolved rate at 45.6%.
  • Complete-attempt subset: Among 476 tasks with four graded attempts, Risa resolves 46.6% versus 43.0% for Uniform and 46.4% for Text.Risa-H reaches 47.1%, while Oracle reaches 52.1%.
  • Patch availability: The reported patch pool contains 1,964 completed task-attempt cells out of 2,000 scheduled, with a 73.5% patch yield.Grading these final patches gives a 42.1% attempt-level resolved rate.
  • Parameterization and hardware: Evaluation parameters were fixed using compute constraints, diagnostic data, and a separate pilot rather than resolved-rate tuning.The experiments used one shared server configuration.

I.1 Cost of the Method

Risa adds substantial inference cost through batched sibling sampling, while token-level routing readouts identify informative patch positions and remain stable across broad parameter ranges.

  • Compute cost: An n=16 steered step costs approximately twice a single-sample step, despite a 98.9% prefix-cache hit rate.A medium-effort steered rollout generates about 90K tokens, or roughly 360K per task across four attempts.
  • Selection overhead: Terminal selection adds one short prefill and one discarded token per available patch without generating or executing another patch.Representative costs include 22 GPU-h for 20b low effort and 222 GPU-h for 20b high effort.
  • Evaluation metric: The primary metric is official SWE-bench Verified resolved rate, determined by fail-to-pass and pass-to-pass tests inside the instance container.Selectors are compared as realized 1-of-K decisions rather than absolute score calibration.
  • Readout localization: Decision-token routing readouts achieve .657 AUC for ranking resolved above unresolved patches, versus .639 for whole-span readouts.The decision-token window is the lowest-probability 25% of tokens.
  • Long-span robustness: On long spans, identity AUC is .93 for decision-token readouts and .88 for whole-span readouts, compared with .58 for trailing-64 windows.Opening-64 readouts retain the highest reported identity AUC at .98.
  • Choice concentration: The lowest-probability quarter carries a median 72% of total token surprisal across 658 final-step generations.This concentration motivates using low-probability decision positions for routing comparison.
  • Sensitivity: Routing-window fractions from 10–50% remain stable, while the selected 25% fraction attains the top offline AUC and realized macro-average.Threshold-sweep pairwise AUC remains within .536–.580.

J.5 Novelty Across Steps and Trajectories

Routing-history similarity is nearly uninformative for choosing among same-step siblings but strongly separates resolved from unresolved trajectories. Risa uses this distinction to diversify candidate generation and increase the supply of submittable patches.

  • Within-step routing-history quintiles produce near-chance discrimination, supporting history disagreement as a computation-diversity prior.The deployed score has pairwise AUC 0.49, while the whole-history mean has AUC 0.52.
  • Risa matches routing comparisons to the decision: it promotes exploration among same-prefix actions while using trajectory-level routing signals for later control.This operationalizes the contrast between weak within-step outcome separation and strong across-trajectory progress separation.
  • Across 1,021 rollouts, resolve rates fall from 33.8% in the most-different quintile to 5.4% in the most-similar quintile.The deployed score yields AUC 0.656; the whole-history mean yields AUC 0.611.
  • 94% of attempts produced submittable patches with routing-guided steering versus 79% when the controller was disabled on the 80-task diagnostic pool.Patchless attempts fell from 21% to 6%, increasing the candidate pool available for terminal selection.
Loading 2608.22191v1…