PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-RM: Execution-free Feedback For Software Engineering Agents

KaShun Shum, Binyuan Hui, Jiawei Chen, Lei Zhang, X. W., Jiaxi Yang, Yuzhen Huang, Junyang Lin, Junxian He

arXiv:2512.21919v1cs.CL

TL;DR

Execution-based feedback for SWE agents is sparse and depends on reliable unit tests, while TTS alone may not predict reward-model usefulness in RL. The paper evaluates discrimination and calibration alongside TTS, uses controlled studies to train SWE-RM, and reports stronger TTS and RL performance.

  • Problem

    Execution-based feedback is sparse and requires reliable unit-test coverage, while execution-free reward models remain underexplored for realistic SWE agents and TTS alone may not predict RL behavior.

  • Method

    The paper evaluates reward models with TTS, AUC, and calibration, then studies data scale, policy mixtures, source composition, and context length to develop SWE-RM.

  • Results

    SWE-RM improves Qwen3-Coder-Flash from 51.6% to 62.0% and Qwen3-Coder-Max from 67.0% to 74.6% on SWE-Bench Verified TTS.

  • Takeaways & Limitations

    Execution-free, well-calibrated reward modeling supports strong SWE-agent performance across TTS and RL.

  • Takeaways & Limitations

    TTS alone cannot explain downstream RL success, motivating broader evaluation with discrimination and calibration.

Abstract

from arXiv · show

Execution-based feedback like unit testing is widely used in the development of coding agents through test-time scaling (TTS) and reinforcement learning (RL). This paradigm requires scalable and reliable collection of unit test cases to provide accurate feedback, and the resulting feedback is often sparse and cannot effectively distinguish between trajectories that are both successful or both unsuccessful. In contrast, execution-free feedback from reward models can provide more fine-grained signals without depending on unit test cases. Despite this potential, execution-free feedback for realistic software engineering (SWE) agents remains underexplored. Aiming to develop versatile reward models that are effective across TTS and RL, however, we observe that two verifiers with nearly identical TTS performance can nevertheless yield very different results in RL. Intuitively, TTS primarily reflects the model's ability to select the best trajectory, but this ability does not necessarily generalize to RL. To address this limitation, we identify two additional aspects that are crucial for RL training: classification accuracy and calibration. We then conduct comprehensive controlled experiments to investigate how to train a robust reward model that performs well across these metrics. In particular, we analyze the impact of various factors such as training data scale, policy mixtures, and data source composition. Guided by these investigations, we introduce SWE-RM, an accurate and robust reward model adopting a mixture-of-experts architecture with 30B total parameters and 3B activated during inference. SWE-RM substantially improves SWE agents on both TTS and RL performance. For example, it increases the accuracy of Qwen3-Coder-Flash from 51.6% to 62.0%, and Qwen3-Coder-Max from 67.0% to 74.6% on SWE-Bench Verified using TTS, achieving new state-of-the-art performance among open-source models.

1 INTRODUCTION

Software-engineering agents use execution-based and execution-free feedback for TTS and RL, but unit-test feedback is sparse and depends on reliable test coverage. The paper proposes evaluating reward models beyond TTS and develops SWE-RM for strong performance across TTS and RL.

  • Execution-based verifiers rely on concrete outcomes such as unit-test results, whereas execution-free verifiers provide continuous model-based scores without sandbox environments.
  • Sparse pass/fail feedback and incomplete or unreliable unit-test coverage limit the ability to distinguish trajectories and support effective RL or TTS.
  • TTS alone can misrepresent verifier quality because nearly identical TTS performance can lead to drastically different RL behavior.
  • AUC and calibration complement TTS by measuring trajectory discrimination and whether confidence scores align with empirical correctness.
  • 51.6% to 62.0% and 67.0% to 74.6% are the reported SWE-Bench Verified TTS accuracy gains for Qwen3-Coder-Flash and Qwen3-Coder-Max using SWE-RM.

2 RELATED WORK

Prior SWE verifier research distinguishes execution-based feedback from model-based scoring, but execution-free work has focused mainly on TTS. This paper argues that verifier quality also requires broader evaluation dimensions.

  • Execution-based SWE verifiers rely on unit tests, while execution-free verifiers use model-based scoring.
  • Existing execution-free verifier research has primarily emphasized test-time scaling.
  • The paper extends verifier-quality analysis beyond TTS to include classification-related quality dimensions.

3 WHAT DEFINES A VERSATILE REWARD MODEL FOR SWE?

The paper finds that TTS does not fully characterize reward models for RL: discrimination and calibration also matter. It develops a broader evaluation using TTS, AUC, and calibration to explain and address this gap.

  • 3.1 INITIAL FINDINGS: LIMITATIONS OF RELYING SOLELY ON TTS: Similar TTS performance can conceal sharply different RL outcomes: verifier A supports smooth improvement, whereas verifier B becomes unstable and eventually collapses training.
  • 3.1 INITIAL FINDINGS: LIMITATIONS OF RELYING SOLELY ON TTS: TTS measures whether the best trajectory ranks first, but overlooks discrimination across trajectories and confidence alignment with correctness.
  • 3.2 MOVING BEYOND TTS: THE NEED FOR MORE VERSATILE EVALUATION ON VERIFIER QUALITY: AUC evaluates separation of resolved and unresolved trajectories across the distribution, while ECE measures calibration between predicted confidence and empirical correctness.
  • 3.2 MOVING BEYOND TTS: THE NEED FOR MORE VERSATILE EVALUATION ON VERIFIER QUALITY: +4.7% versus +4.5% TTS improvements coexist with a 0.095 AUC gap between verifier A and verifier B.
  • 3.2 MOVING BEYOND TTS: THE NEED FOR MORE VERSATILE EVALUATION ON VERIFIER QUALITY: Verifier B exhibits widespread over- and under-confidence, showing that calibration and TTS capture distinct verifier properties.
  • 3.2 MOVING BEYOND TTS: THE NEED FOR MORE VERSATILE EVALUATION ON VERIFIER QUALITY: The paper treats the early examples as motivation, then begins supervised reward-model training and large-scale ablations to develop a versatile verifier.

4 HOW TO TRAIN A VERSATILE REWARD MODEL FOR SWE?

The paper systematically studies training data, composition, policy, source, and context length to build versatile execution-free SWE reward models. These investigations motivate a practical recipe balancing verifier accuracy, calibration, test-time scaling, and scalability.

  • 4.1 TRAINING METHODS: The reward model is trained as a generative YES/NO classifier and evaluated using AUC, ECE, RM@K, and test-time scaling on SWE-bench Verified.The setup uses Qwen3-30B-A3B and evaluates each instance with 32 independent runs.
  • 4.1 TRAINING METHODS: More training data improves verifier discrimination and calibration, although test-time scaling gains diminish beyond 25k examples.A 500-example model has ECE 0.481, seven times higher than a 100k-example model.
  • 4.2 DATA SCALING AND RATIO EFFECT: A 2:1 positive-to-negative ratio generally achieves the best overall AUC, calibration, and test-time scaling across both model scales.The ratio also uses a smaller negative-data pool while retaining all available positive data, so it becomes the default configuration.
  • 4.3 CONTEXT LENGTH CONSTRAINT: Extending context to 128k scores over 99% of trajectories, while longer supported contexts improve test-time scaling by enabling more trajectories to be evaluated.The execution-free verifier scales to 256k context length for complex and long trajectories.
  • 4.4 POLICY AND SOURCE ABLATION: Mixing policy data provides a better overall balance across AUC, ECE, and test-time scaling than relying on a single policy source.On-policy data can be stronger on individual metrics, such as test-time scaling on Qwen3-Coder-Max, but Mix-Policy performs better overall.
  • 4.4 POLICY AND SOURCE ABLATION: A mixed source setup centered on SWE-rebench and supplemented with SWE-smith and SWE-Gym balances data quality, calibration, and scaling effects.SWE-rebench performs best under single-source evaluation for AUC and RM@32, while additional sources improve calibration and data scaling.

5 SWE-RM: A VERSATILE REWARD MODEL FOR TTS AND RL

SWE-RM is evaluated as an execution-free verifier across test-time scaling and reinforcement learning, with comparisons spanning verifier quality, feedback types, and multiple SWE benchmarks. Hybrid feedback achieves the strongest reported RL results by combining continuous execution-free signals with verifiable execution-based rewards.

  • 5.1 A NEW STATE-OF-THE-ART IN TTS: SWE-RM consistently outperforms all baselines across AUC, ECE, and RM@32, achieving the best discrimination, calibration, and test-time scaling performance.The gains extend beyond Qwen3-Coder models to OpenHands-LM-32B, supporting verifier generalization.
  • 5.2.2 EXECUTION-FREE FEEDBACK BENEFITS RL TRAINING: Execution-based RL feedback is limited by unchecked test quality, long-horizon execution, and sparse 0/1 rewards, motivating finer-grained execution-free signals.Execution-free feedback progresses faster initially, but unverified inaccuracies can weaken later convergence.
  • 5.2.1 RL SETUP: The RL study compares hybrid, execution-free-only, execution-based-only, and poorly calibrated execution-free feedback under a common SWE-bench Verified setup.Evaluation uses one greedily decoded trajectory and patch per instance for final pass@1.
  • 5.2.2 EXECUTION-FREE FEEDBACK BENEFITS RL TRAINING: 54.8% versus 51.8% pass@1: hybrid feedback improves RL performance by about 3 absolute points over execution-based feedback.Figure 7 also shows faster, smoother training improvements with hybrid feedback.
  • 5.2.2 EXECUTION-FREE FEEDBACK BENEFITS RL TRAINING: Hybrid feedback consistently achieves better RL performance across SWE-Bench Live, SWE-Bench Multilingual, Multi-SWE-Bench Mini, and Terminal Bench.Execution-free-only feedback is comparable to execution-based-only feedback on these broader tasks.

6 CONCLUSION

The paper argues that verifier quality for SWE agents cannot be judged by test-time scaling alone. It develops SWE-RM around discrimination and calibration, reporting strong open-source TTS and RL performance.

  • 6 CONCLUSION: Test-time scaling alone is insufficient to measure verifier quality for SWE agents; RL additionally requires strong discrimination and reliable calibration.The paper identifies AUC and low ECE as the relevant complementary qualities.
  • 6 CONCLUSION: SWE-RM is a 30B-parameter mixture-of-experts verifier with 3B activated parameters and up to 256k context.It is trained using ablations over data scale, positive/negative ratios, policy mixtures, and source composition.
  • 6 CONCLUSION: +3 absolute pass@1: SWE-RM produces faster, more stable RL training than execution-based feedback counterparts.It also achieves state-of-the-art open-source TTS gains on SWE-Bench Verified.
  • 6 CONCLUSION: Execution-based and execution-free verifiers form the paper’s two main feedback categories for SWE tasks.Execution-based verifiers rely on unit tests, whereas execution-free verifiers use model-based scoring.

B.2 AGENTIC REINFORCEMENT LEARNING FEEDBACK IN SWE TASKS

Agentic SWE reinforcement learning faces sparse or misleading execution-based feedback because unit tests may be unreliable and cannot distinguish trajectories with the same test outcome. The paper frames execution-free feedback as a finer-grained alternative and relates verifier metrics to RL updates.

  • B.2 AGENTIC REINFORCEMENT LEARNING FEEDBACK IN SWE TASKS: Execution-based feedback can be sparse or misleading because it cannot distinguish trajectories that share either a passing or failing test outcome.This limits its usefulness as an optimization signal in agentic reinforcement learning.
  • B.2 AGENTIC REINFORCEMENT LEARNING FEEDBACK IN SWE TASKS: Execution-free feedback provides more fine-grained rewards and improves training efficiency and performance in SWE agent reinforcement learning.The paper presents this as the first integration of execution-free feedback into SWE agentic RL.
  • C THEORETICAL LINK BETWEEN TTS, AUC, AND ECE AND RL DYNAMICS: TTS, AUC, and ECE correspond to three distinct reward-model failure modes when scores are used as RL optimization signals.The analysis represents trajectories with RM scores r(τ) and binary correctness labels c(τ).
  • C THEORETICAL LINK BETWEEN TTS, AUC, AND ECE AND RL DYNAMICS: The gap between RM-scored updates and updates using true correctness determines RL stability and correctness.The paper uses simple policy gradients as an illustrative analysis that extends in nature to GRPO and GSPO.

C.1 TTS: EXTREME-TOP ERRORS AND THEIR IMPACT ON RL

The theoretical analysis separates extreme-top ranking errors from global pairwise mis-rankings and connects both to harmful RL updates. Incorrect top-ranked trajectories can reinforce undesirable behavior, while AUC controls the fraction of reversed positive-negative updates.

  • C.1 TTS: EXTREME-TOP ERRORS AND THEIR IMPACT ON RL: TTS@k measures whether the highest-scored trajectory among k samples is correct, so 1−TTS@k is the probability that an incorrect trajectory receives the largest reward.The metric focuses on the extreme top of the ranking rather than all sampled pairs.
  • C.1 TTS: EXTREME-TOP ERRORS AND THEIR IMPACT ON RL: An incorrect top-ranked trajectory receives the strongest reward signal for an unresolved behavior, causing the policy to increase that behavior’s sampling probability over iterations.This is the RL consequence of an extreme-top TTS error.
  • C.1 TTS: EXTREME-TOP ERRORS AND THEIR IMPACT ON RL: AUC measures the probability that the reward model ranks a positive trajectory above a negative one, and each mis-ranking induces an update opposite to the ideal direction.The reversed-update fraction is 1−AUC across positive-negative trajectory pairs.
  • C.1 TTS: EXTREME-TOP ERRORS AND THEIR IMPACT ON RL: The expected fraction of bad reversed updates grows in proportion to 1−AUC because gradient contributions aggregate linearly.Unlike TTS, AUC captures global ranking correctness affecting every sampled trajectory.

C.3 ECE: CALIBRATION ERROR AND SYSTEMATIC BIAS IN RL UPDATES

Calibration determines whether reward-model scores correspond to true success frequencies, affecting bias and variance in RL updates. The section formalizes calibration error as a source of systematic distortion and unstable training.

  • Calibration means that trajectories assigned confidence α succeed at frequency α, ensuring predicted scores match observed success rates.For example, a score of 0.7 should correspond to a 70% success rate.
  • Perfect calibration makes the reward-model update unbiased in expectation because E[c | r] = r.Under this condition, the reward-model-induced gradient matches the ideal expected gradient without systematic bias.
  • High calibration error introduces systematic distortion into reward-model scores and RL updates.The bias function compares true success frequency with predicted probability at each score, while ECE approximates the expected magnitude of this bias.
  • Miscalibration adds variance to policy-gradient estimates by coupling calibration error with ∇θ log πθ(τ).Because ECE approximates expected absolute calibration bias, higher ECE typically corresponds to a larger noise contribution and less stable RL training.

D.1 DETAILED TRAINING SETUP

The reward model is trained and evaluated using large, heterogeneous SWE-agent trajectories, long-context classification, and comparisons against execution-free and execution-based verifiers. Evaluation uses SWE-bench Verified with repeated trajectory sampling and AUC, ECE, and selection metrics.

  • Over 400k multi-turn trajectories are collected from OpenHands and SWE-Agent across four SWE data sources.Trajectories run for up to 100 iterations and use different policy models and data sources.
  • Resolved and unresolved labels are assigned from fail2pass execution results, with cleaning that removes cases lacking any successful trajectory.The filtering targets instances affected by over-strict tests, unfair tests, or under-specified descriptions.
  • The study compares SWE-RM with execution-free and execution-based baselines, including SWE-Gym, DeepSWE, and Agentless verifiers.Pass@k measures whether any sampled trajectory succeeds, while RM@K measures the success rate of selected trajectories.
  • Evaluation on 500 human-verified SWE-bench Verified tasks uses 32 independent runs per instance, producing 16k trajectories.Accuracy is measured with AUC and calibration with ECE, using ten confidence bins.
  • The verifier classifies full multi-turn trajectories by predicting a single <YES> or <NO> token, trained with next-token prediction loss.At inference, the special-token log probability supplies the reward-model score; training supports a 256k context window.

D.4 ADDITIONAL ANALYSIS ON CONTEXT CONSTRAINT

Increasing the verifier context window from 32k to 256k substantially improves scoring coverage for long SWE trajectories. The larger window costs more memory but has similar inference latency because only one output token is generated.

  • A 256k-token context window reduces no-score cases by enabling full-trajectory scoring for longer inputs.It accommodates longer trajectories, multi-file code, and richer histories that would otherwise be truncated or excluded.
  • The 256k context increases deployment memory usage to roughly twice that of a 32k model.The authors state that deployment remains practical with two A100 GPUs.
  • Inference latency is more or less unchanged across context lengths because one-token output computation can be parallelized.The principal trade-off identified for the longer context is higher memory usage rather than latency.

E RL TRAINING DETAILS

RL training uses a Qwen3-30B-A3B model warmed up with supervised fine-tuning and agentic trajectories from SWE datasets. The policy interacts with tools over multiple turns, while optimization masks tool responses and applies group-based advantage estimation.

  • RL starts from a Qwen3-30B-A3B model with supervised fine-tuning warm-up on in-house agentic trajectories.The model uses a mixture-of-experts architecture following current coding-agent practice.
  • Training uses SWE-Gym and SWE-rebench data, 16 rollouts per problem, up to 100 iterations, and a 128k context length.The context length is chosen to cover most problem cases more cost-effectively than 256k.
  • Each agentic trajectory alternates policy actions and tool responses across T steps, forming τ = {a1, o1, a2, o2, . . . , aT, oT}.Tool responses are masked during optimization, and the training objective uses group-based advantage estimation.
  • Optimization incorporates standard techniques including Clip High and NO KL loss.
Loading 2512.21919v1…