PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents

Yiran Wu, Andrew Zhao, Yurui Chang, Qingyun Wu, Xuan Feng

arXiv:2610.12598v1cs.AIcs.SE

TL;DR

Binary test rewards do not distinguish debugging quality among trajectories with the same outcome. BSR uses the current policy to compare same-task, same-outcome trajectories and refine rewards. It improves results across the evaluated Qwen3 configurations on SWE-bench Verified and SWE-bench-Live Lite, supporting comparative judgments as complementary supervision.

  • Problem

    Binary verifier rewards fail to distinguish trajectories with substantially different debugging quality when they share an outcome.

    Source: §1 · p. 2

  • Method

    BSR has the current policy compare observable debugging behavior within same-task, same-verifier-outcome groups and converts preferences into bounded auxiliary rewards.

    Source: §1 · p. 2

  • Results

    BSR improves over verifier-only RL on SWE-bench Verified and SWE-bench-Live Lite across the evaluated Qwen3-14B and Qwen3-32B configurations.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    The policy’s comparative judgments can complement binary verification by letting otherwise reward-equivalent trajectories contribute to learning.

    Source: §1 · p. 3, §6 · p. 10

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) provides reliable supervision for software engineering agents through executable tests, but binary rewards fail to distinguish trajectories with substantially different debugging quality. We investigate whether the current policy can judge its own trajectories and use these judgments to improve task performance without a separately trained reward model. To this end, we introduce Behavioral Self-Rewarding (BSR), a reward-refinement method that reuses the policy to compare observable debugging behavior among trajectories from the same task with the same verifier outcome. BSR aggregates pairwise preferences into task-local scores and converts them into bounded auxiliary rewards, introducing reward contrast while preserving the verifier-defined pass/fail ordering. This provides additional supervision from already-collected trajectories without requiring new environment interactions. Experiments on SWE-bench Verified and SWE-bench-Live show consistent improvements across the evaluated Qwen3-14B and Qwen3-32B configurations, supporting using the policy's own comparative judgments as a complementary learning signal for training software engineering agents.

1 Introduction

BSR addresses the lack of learning signal among same-outcome software-engineering trajectories by using the current policy to compare them. The introduction reports improvements across evaluated benchmarks and models, with no separate trajectory reward model.

1 Introduction

  • Repository-level software engineering gives agents long-horizon, behaviorally complex interactions, while final patches can often be checked with test suites.The setting requires reasoning under partial information as agents interact with changing repositories.Source: §1 · pp. 1–2
  • Binary test outcomes erase differences in debugging quality, leaving same-outcome rollout groups with no outcome-derived relative advantage despite costly SWE interactions.The current policy's comparisons offer a complementary source of supervision from repository-level interaction experience.Source: §1 · pp. 2–3
  • BSR compares same-task trajectories within shared verifier-outcome groups, aggregates pairwise behavioral preferences, and turns rankings into bounded auxiliary rewards that preserve pass/fail ordering.The comparisons focus on observable debugging actions and outcomes, not writing style or confidence.Source: §1 · p. 2
  • BSR improves average accuracy across the evaluated configurations on three benchmarks, with consistent Pass@3 gains on the two SWE benchmarks.The introduction reports this pattern for Qwen3-14B and Qwen3-32B.Source: §1 · p. 2

2 Preliminaries

The preliminaries model software-engineering tasks as multi-step repository interactions evaluated by executable binary verifiers. Group-relative methods derive learning advantages from reward variation, which identical-outcome groups lack.

2 Preliminaries

  • Each task pairs an initial repository and issue description with an executable verifier; the policy gathers evidence, edits code, and returns a patch across sampled trajectories.The verifier gives reward 1 when the final patch passes and 0 otherwise.Source: §2 · p. 3
  • GRPO normalizes rewards within each sampled group, while LOOP adapts group-relative optimization to interactive agents with a leave-one-out baseline.Both use the resulting advantages in clipped policy-gradient objectives without a learned value model.Source: §2 · p. 3
  • All-pass or all-fail groups have identical binary rewards and therefore zero advantages, even when their trajectories differ in debugging quality.This motivates refining rewards within outcome groups while retaining verifier-defined task success.Source: §2 · p. 3

3 Method

Behavioral Self-Rewarding (BSR) reuses the current policy to compare task-local trajectories with the same verifier outcome, then turns those preferences into bounded auxiliary rewards. Its pairwise behavioral rubric and reward map add within-outcome contrast while preserving the verifier’s pass/fail ordering.

3.1 Pairwise Behavioral Ranking

  • The current policy compares trajectories from the same task and verifier-outcome bucket, with separate criteria for progress among failures and execution quality among successes.For training comparisons, the judge receives a reference patch, while the acting policy does not access it during training or evaluation.Source: §3.1 · pp. 4–5, §3 · p. 4
  • Pairwise judgments identify a preferred trajectory and coarse preference strength, then aggregate into task-local scores without requiring calibrated quality ratings across tasks.The model sees compact trajectory representations retaining actions and outcomes, with deterministic trajectory statistics as supporting evidence.Source: §3 · p. 4, §3.1 · p. 5, §3.2 · p. 5

3.2 From Pairwise Comparisons to Rewards

  • The reward map standardizes fitted scores within each outcome bucket, suppresses near-zero deviations with a soft deadzone, and centers and bounds active rewards.Buckets with fewer than two trajectories receive zero auxiliary reward because no pairwise comparison is possible.Source: §3.2 · pp. 5–6
  • BSR adds bounded auxiliary rewards derived from within-outcome comparisons to verifier rewards, preserving higher rewards for every passing trajectory than every failing trajectory.This contrast can restore group-relative learning signal in all-fail groups and refine distinctions within shared outcomes in mixed groups.Source: §3.2 · p. 6

4 Experiments

Across the evaluated Qwen3 models and SWE benchmarks, BSR improves average accuracy and Pass@3 over verifier-only RL. The experiments also examine behavior changes and test which reward-refinement choices account for the gains.

4 Experiments

  • Training uses filtered R2E-Gym rollouts capped at 40 agent steps, while evaluation allows 50 steps and reports average accuracy across three runs and Pass@3.The main BSR setting refines only the failure bucket.Source: §4 · p. 6

4.1 Main Results

  • BSR improves both average accuracy and Pass@3 in every evaluated setting on SWE-bench Verified and SWE-bench-Live; Qwen3-14B gains from 32.1% to 34.7% accuracy on Verified and 6.33% to 8.11% on Live.The cited examples use the 400-step budget on SWE-bench Verified and SWE-bench-Live, respectively.Source: §4.1 · pp. 6–7, Table 1 · p. 7
  • BSR more often views a related file before the first edit (0.748 versus 0.733), converts searches into edits more often (0.494 versus 0.465), and performs more post-edit testing (0.311 versus 0.301).The aggregate pattern is consistent with stronger reference-oriented investigation, search follow-through, and verification.Source: Table 2 · p. 7, §4.1 · p. 8

4.3 Ablation: Verifier-Outcome Buckets

  • Failure-only refinement leads the tested bucket choices at 33.2% average accuracy and 46.6% Pass@3; refining both buckets reaches 33.1% and 44.6%, while success-only reaches 32.0% and 44.0%.Extending refinement to successful trajectories adds comparisons without improving either reported metric over failure-only refinement.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 Additional Study: Random-reward control

  • Amplitude-matched random rewards achieve 30.9% average accuracy and 42.8% Pass@3, below verifier-only RL at 31.0% and 45.0%; BSR reaches 33.2% and 46.6%.The control ignores pairwise trajectory preferences, indicating that reward amplitude alone does not reproduce BSR's gains.Source: Table 5 · p. 8, §4.4 · pp. 8–9

4.2 Ablation: Deadzone and Softness

  • The deadzone-plus-softness mapping reaches 33.2% average accuracy and 46.6% Pass@3, outperforming both the mapping without either component and the deadzone-only variant on both metrics.The deadzone-only variant reaches 32.6% average accuracy but 44.0% Pass@3; removing both yields 32.0% and 45.6%.Source: Table 3 · p. 8, §4.2 · p. 8

5 Related Work

Prior work supplies verifier-based and learned evaluative signals for agentic RL and trajectory preferences. BSR differs by using the acting policy to compare debugging behavior across complete, same-task, same-outcome trajectories for on-policy learning.

Verifiable rewards for agentic RL.

  • Hybrid-reward methods combine binary verification with learned or rubric-based signals, while BSR obtains its additional signal from the current policy’s task-local trajectory preferences.HERO uses outcome-conditioned reward-model scores, whereas PAPO integrates rubric-based process feedback; BSR uses bounded reward refinement for on-policy RL.Source: §5 · p. 9

Software engineering agents, environments, and training.

  • Unlike dedicated SWE trajectory reward models, BSR uses the acting policy’s within-task, within-outcome comparisons without a separate reward-model training stage.The experiments test the procedure in a fixed SWE-agent setup.Source: §5 · p. 9

Process supervision and trajectory preferences.

  • Unlike response- or reasoning-step preference methods, BSR judges observable debugging behavior across complete tool-interaction trajectories and converts preferences into bounded rewards for on-policy RL.Self-Rewarding Language Models use response evaluations for iterative DPO (Yuan et al., 2024); Process-based Self-Rewarding judges mathematical reasoning steps (Zhang et al., 2025).Source: §5 · p. 9

6 Conclusion

BSR refines verifier rewards by comparing same-task trajectories with the same outcome and turning policy judgments into bounded auxiliary rewards. Experiments show improvements over verifier-only RL across the evaluated models and benchmarks.

6 Conclusion

  • Improvements over verifier-only RL on SWE-bench Verified and SWE-bench-Live Lite span evaluated Qwen3-14B and Qwen3-32B configurations.The results support policy-generated comparative judgments as a complement to binary verification for learning from repository-level interaction experience.Source: §6 · pp. 9–10
  • BSR converts same-task, same-outcome trajectory preferences into bounded auxiliary rewards while preserving the verifier-defined pass/fail ordering.It reuses the policy as a post-rollout judge and does not require a separate reward model.Source: §6 · p. 9
  • The method supplies additional supervision from already-collected trajectories without requiring new environment interactions.The policy’s comparative judgments can help distinguish trajectories that would otherwise receive equivalent binary rewards.Source: §6 · pp. 9–10

A Additional Method Details

The appendix details how BSR schedules same-outcome trajectory comparisons, aggregates preferences into task-local scores, and prepares interaction traces for judgment.

A.2 Weighted Bradley–Terry Score Fitting

  • Within each outcome bucket, weighted Bradley–Terry scores model preference probability as the sigmoid of score differences, fitted with preference-strength-weighted likelihood and L2 regularization.The fitted scores are task-local summaries of the policy’s comparative judgments.Source: §A.2 · pp. 13–14

A.1 Pairwise Comparison Scheduling.

  • For buckets with fewer than five trajectories, scheduling is exhaustive; larger buckets use Swiss-style rounds capped at ⌊n/2⌋ comparisons per round, with comparisons judging progress or cleaner behavior and randomized pair order.The Swiss schedule uses T = min(⌈log2 n⌉ + 2, n − 1) rounds.Source: §A.1 · p. 13, Table 6 · p. 14

A.2 Weighted Bradley–Terry Score Fitting

  • The score-fitting procedure uses diagonal second-order preconditioning and recenters scores each iteration because downstream rewards depend only on relative values.Source: §A.2 · p. 14

A.3 Trajectory Summarization

  • Trajectory summaries retain ordered tool actions and observations while removing internal thinking; deterministic reductions preserve errors and key outputs, with LLM summarization reserved for still-long observations.Actions are reparsed into compact function-style calls, and long numbered file views retain their first and last portions with an omission marker.Source: §A.3 · pp. 14–15

A.4 Other Potential Instantiations of Behavioral Reward

The appendix describes group ranking and rubric grading as alternatives to pairwise ranking, but reports no clear improvement from either in preliminary experiments. It contrasts their efficiency and assessment styles with the trade-offs of pairwise comparisons.

A.4 Other Potential Instantiations of Behavioral Reward

  • Preliminary group-ranking and rubric-grading experiments showed no clear improvement, so neither alternative was tuned or investigated further.Source: §A.4 · p. 15
  • Group ranking jointly scores trajectories in one verifier-outcome bucket on a coarse 1–5 scale, offering a global view and fewer direct comparisons but sensitivity to group composition or listwise presentation.Source: §A.4 · p. 15
  • Rubric grading independently assesses progress criteria and red flags, producing weighted-penalty rewards without trajectory comparisons, but depends on rubric completeness and calibration.Source: §A.4 · p. 15
  • Pairwise ranking offers local relative supervision but costs judgment compute; both ranking approaches lack a task-independent quality scale, motivating future combinations with absolute rubric assessment.Source: §A.4 · p. 15

B.1 Detailed Setup

The appendix details the training and evaluation setup, reports training observations that informed rollout settings, and describes the infrastructure and added training cost of BSR.

B.1 Detailed Setup

  • Training used 4,578 filtered R2E-Gym tasks; evaluation covered 500 SWE-bench Verified tasks and 300 SWE-bench-Live Lite tasks, with three runs per benchmark.Evaluation used temperature 1 and a maximum of 50 agent steps; results report mean and standard deviation of accuracy, plus Pass@3.Source: §B.1 · p. 16
  • Reducing the training rollout limit from 50 to 40 steps improved performance and reduced training time, while lowering sampling temperature or adding KL regularization reduced performance.More than 95% of successful trajectories finished in fewer than 40 steps; evaluation retained a 50-step maximum.Source: §B.1 · p. 16
  • The training setup used a two-node Kubernetes cluster, with 128 CPU cores and approximately 500 GB of memory per node, for up to 512 concurrent environments.The authors pre-pulled container images and used small per-pod resource requests; listed operational considerations include network access, pod limits, image retention, stable connections, and cleanup.Source: §B.1 · pp. 16–17
  • Asynchronous environment pre-creation and starting each rollout as soon as its own environment was ready reduced waiting on startup and slow environments.The appendix also includes a Terminal-Bench 2.1 performance table covering 89 tasks and three evaluation trials.Source: §B.1 · p. 17, Table 8 · p. 17
  • BSR required 19.86% more wall-clock training time than verifier-only RL in the measured Qwen3-14B configuration, and the comparison was not fully compute-matched.BSR at 270 steps already exceeded the 400-step baseline in average accuracy; the authors identify selective scheduling, batching, and overlap as ways to reduce overhead.Source: §B.1 · p. 17

B.2 Additional Evaluation on Terminal-Bench 2.1

On Terminal-Bench 2.1, BSR improved both average accuracy and Pass@3 across all evaluated model sizes and training budgets.

B.2 Additional Evaluation on Terminal-Bench 2.1

  • BSR consistently improved average accuracy and Pass@3 across all evaluated model sizes and training budgets on Terminal-Bench 2.1.The evaluation covered 89 tasks with three independent trials; the passage reports no specific metric values.Source: §B.2 · p. 17

B.3 Additional Results

The appendix provides additional training curves for the evaluated models and plots results from verifier-outcome-bucket and reward-mapping ablations.

B.3 Additional Results

  • Figure 3 plots mean training-set verifier scores for BSR and the baseline, with curves smoothed using an EMA span of 31.The caption identifies the plotted score as the execution verifier score.Source: §B.3 · p. 17, Figure 3 · p. 18
  • Figure 4 presents the full 400-step SWE-bench Verified evaluation curves for Qwen3-14B, and Figure 5 shows detailed training curves for Qwen3-32B.Both evaluations used temperature 1, a 50-step maximum, and three runs.Source: Figure 4 · p. 18, Figure 5 · p. 19, §B.3 · p. 20
  • Figures 6 and 7 show SWE-bench Verified ablations of verifier-outcome buckets and reward mapping, respectively.The bucket variants apply BSR to failed trajectories, successful trajectories, or both; the reward-mapping variants remove the deadzone and/or softness.Source: Figure 6 · p. 19, Figure 7 · p. 19, §B.3 · p. 20
  • Table 9 summarizes appendix trajectory statistics, with preferred directions indicated for process metrics and workload statistics described as context-dependent.The caption cautions that basic workload statistics should be interpreted in context.Source: Table 9 · p. 20

B.4 Trajectory Behavior Statistics

The appendix reports that BSR is associated with stronger investigation and verification behaviors, alongside fewer steps and faster trajectories. Its self-judge also maintains meaningful agreement with an external judge during training.

B.4 Trajectory Behavior Statistics

  • The authors treat reference-relevant files viewed before editing, search-to-edit conversion, post-edit test rate, and duplicate command rate as primary indicators of debugging behavior.Workload measures such as steps and search counts provide supporting context because lower or higher values can have multiple interpretations.Source: §B.4 · p. 20
  • BSR more often viewed a reference-relevant file before the first edit (0.748 vs. 0.733), had higher search-to-edit conversion (0.494 vs. 0.465), and performed more post-edit testing (0.311 vs. 0.301).It slightly reduced duplicate commands (0.208 vs. 0.209); workload statistics also showed fewer steps and searches, slightly more reading, more tests, fewer errors, and faster trajectories.Source: §B.4 · pp. 20–21
  • The observed aggregate patterns are consistent with stronger reference-oriented investigation, search follow-through, and verification, with workload statistics providing complementary evidence about debugging effort.This connects the process metrics to the accompanying workload trends without treating any single workload measure as unambiguous.Source: §B.4 · p. 21

B.5 Judge Quality Investigation

  • The self-judge maintained meaningful agreement with Kimi K2.5 across training, with macro F1 generally around 0.6–0.7, despite some decline at later checkpoints.At each evaluation point, 200 sampled comparisons were evaluated every 10 training steps; downstream BSR models continued to improve.Source: Figure 8 · p. 21, §B.5 · p. 21

Related papers

Loading 2610.12598v1…