PaperlayerView exact revision on arXiv

Source-linked AI summary

BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation

Yingxiang Yang, Weihang Xiao, Zhunxuan Wang, Joshua Flashner, Niresh Agarwal

arXiv:2610.09804v1cs.LG

TL;DR

GRPO gives every token in a rollout the same sequence-level advantage, motivating a more efficient way to use process supervision without a value network. BoT-GRPO aggregates external token-level rewards with inverse-length weighting to compute per-token advantages. It converges faster on React code generation and AIME, with higher final React compile and win rates than the compared GRPO methods.

  • Problem

    GRPO assigns the same sequence-level advantage to every token, while process supervision can add costly rollouts, model-training pipelines, or critics.

    Source: §1 · pp. 1–2

  • Method

    BoT-GRPO is a critic-free GRPO extension that pools token-level rewards, weights them by inverse source-sequence length, and computes per-token advantages from weighted group statistics.

    Source: §1 · p. 2

  • Results

    BoT-GRPO converges faster on React and AIME and achieves higher final React compile and win rates; on React, it reaches >80% compile rate by step 40, up to 1.9× faster than GRPO.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The method’s local rewards serve as bounded, stable auxiliary signals that densify outcome feedback, rather than as exact per-token correctness labels.

    Source: §3 · p. 4, §3.1 · p. 6

    The study covers only two domains, and each configuration uses a single seed unless noted, making reported point estimates and speedups upper bounds.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimization (GRPO) every token in a rollout receives the same advantage. We ask how to make process supervision efficient: accelerating convergence and improving final quality without the cost of value networks. We propose Bag-of-Tokens Group Relative Policy Optimization (BoT-GRPO), which extends GRPO to token-level reward models through a length-invariant "bag of tokens" aggregation: it collects all token-level rewards across rollouts, weights each by the inverse of its source sequence length, and computes per-token advantages relative to weighted group statistics. BoT-GRPO is critic-free, and is a drop-in replacement wherever GRPO is used when token-level reward is available. On React front-end code generation, BoT-GRPO reaches $80\%$ compile rate up to $1.9\times$ faster than GRPO and converges faster than modern GRPO variants (GSPO, DAPO, PURE) while reaching higher final compile and VLM-judged win rates. On a second task, AIME mathematical reasoning, BoT-GRPO delivers absolute Pass@$k$ gains up to $8.1\%$ over GRPO in half the steps. For both tasks we compare the algorithm's performance on reasoning vs. non-reasoning base-model families (Qwen2.5-3B, SmolLM3-3B, Phi-4-mini-reasoning). Our experiments also yield a practical recipe for the reward model itself: reward stability matters more than richness: clean, bounded, stable fine-grained signals consistently accelerate learning where noisier alternatives stall.

1 Introduction

GRPO assigns every token in a rollout the same sequence-level advantage, while efficient process supervision needs a way to use token-level rewards without adding costly training machinery. BoT-GRPO supplies a critic-free, length-invariant aggregation, and the paper reports faster convergence and higher final quality across its evaluated tasks.

1 Introduction

  • GRPO gives every token the same sequence-level advantage, so useful and harmful tokens receive identical credit despite their different contributions.The paper identifies this coarse credit assignment as a source of wasted optimization and additional rollouts, GPU-hours, and wall-clock time.Source: §1 · p. 1
  • Earlier process-supervision approaches incur costs through continuation rollouts, separate model-training pipelines, critics, or curated expert trajectories.Even implicit-PRM methods that cheaply produce token-level rewards change the reward source rather than how GRPO converts token rewards into advantages.Source: §1 · p. 2
  • BoT-GRPO uses inverse source-sequence-length weights to aggregate token rewards into per-token advantages while keeping each rollout equally represented in group statistics.It is critic-free, accepts external token-level rewards, and adds only bookkeeping overhead.Source: §1 · p. 2
  • BoT-GRPO reaches target quality in fewer steps and achieves higher final compile and win rates than GRPO and GSPO, DAPO, and PURE, with speedups across three base-model families.The introduction also reports similar faster convergence and higher stable performance on AIME.Source: §1 · p. 2
  • The paper identifies a practical reward recipe and compares BoT-GRPO across reasoning and non-reasoning base models.Its listed contributions also include reward-signal ablations for code generation and supporting AIME results.Source: §1 · p. 2

2 Related Work

Prior work addresses the cost of process rewards, token-level credit assignment, and GRPO stability through different mechanisms. BoT-GRPO instead focuses on fairly aggregating external token-level rewards within GRPO, retaining its critic-free character.

Efficient RL for Reasoning

  • BoT-GRPO targets training-time efficiency by densifying credit assignment and correcting response-length bias, with the aim of reaching target quality in fewer rollouts.This contrasts with the cited inference-time work on curbing overthinking or directly controlling reasoning length.Source: §2 · p. 2

Process Rewards and Their Cost

  • Process rewards can outperform outcome-only supervision, but existing approaches incur costs from human step labels, continuation rollouts, or separate verifier-model training.Implicit process rewards reduce signal-generation cost, while other work develops richer step-evaluation signals.Source: §2 · pp. 2–3

Token-Level Reward Methods

  • Token-level credit methods include attention-based reward redistribution, token-wise reward models, and learned token-level Q-functions, while BoT-GRPO retains GRPO’s critic-free simplicity.The cited Q-RM work reports 12× faster convergence on GSM8K.Source: §2 · p. 3

GRPO Variants

  • GRPO variants target stability or credit assignment through entropy weighting, eligibility traces, multi-turn grouping, or changes to the optimization objective.The paper compares BoT-GRPO with DAPO, GSPO, and PURE as representative modern variants.Source: §2 · p. 3
  • BoT-GRPO differs by accepting external token-level rewards from any source and applying length-invariant aggregation, which the paper describes as orthogonal to reward-source choices.The related-work discussion says it can incorporate entropy-weighted, eligibility-trace, sequence-level, or turn-level signals.Source: §2 · p. 3

3 Method

BoT-GRPO turns token-level rewards into group-relative token advantages by pooling tokens across rollouts and weighting each token by the inverse length of its source sequence. The method treats local signals as bounded auxiliary feedback alongside global outcomes, with task-specific reward constructions illustrating its reward-agnostic interface.

3 Method

  • Standard GRPO normalizes sequence-level rewards within a group but assigns identical advantage signals to every token in a rollout.For terminal-reward tasks, the method section says the unbiased Monte Carlo estimate is the same terminal reward at every position.Source: §3 · pp. 3–4
  • BoT-GRPO pools tokens from a prompt’s rollouts, weights each by 1/Lk, computes weighted group statistics, and calculates per-token advantages against those statistics.The inverse-length weighting gives each rollout equal contribution to group statistics regardless of its length, with overhead linear in the total number of tokens.Source: §3 · p. 4
  • BoT-GRPO combines global outcomes with local token signals, treating the latter as bounded auxiliary feedback rather than policy-invariant reward shaping.The authors state that their reward construction does not meet the structure required for strict policy-invariance guarantees.Source: §3 · p. 4
  • When the global reward dominates bounded local rewards, substantially suboptimal policies under the original objective remain suboptimal under the shaped objective.The authors characterize this as a weak safeguard against catastrophic policy alteration, not an optimality guarantee.Source: §3 · p. 4
  • Local rewards are intended to densify sparse outcome feedback, not certify token correctness; the method’s explanation relies on global-reward dominance and error propagation after an incorrect step.The authors say gains do not require exact local signals, but do require signals that are bounded, stable, and positively correlated with success.Source: §3 · pp. 4–5

3.1 Practical Reward Construction

  • The aggregation algorithm is reward-agnostic and drop-in, while the particular reward construction is task-specific and can be replaced with another token-level source.The paper illustrates the interface with React code-generation and mathematical-reasoning reward designs.Source: §3.1 · pp. 5–6
  • For React code generation, the reward recipe combines global compile, lint, and code-quality signals with localized line penalties from diagnostics and judge feedback.Compiler and ESLint signals are deterministic or semi-deterministic; local line penalties are softened across nearby lines by a spatial decay function.Source: §3.1 · pp. 5–6
  • For mathematical reasoning, the practical reward combines global outcome, step-level evaluation, and format compliance, with step scores assigned as correct, incorrect, or uncertain.The default error-propagation strategy sets subsequent step scores to zero after a step is marked incorrect.Source: §3.1 · p. 6

4 Experiments

The experiments compare BoT-GRPO with GRPO variants on React code generation and test transfer to AIME across base models. They also examine which reward signals are needed for React training.

4.1 Setup

  • The evaluation covers held-out React prompts and AIME problems, uses distinct training and evaluation judges for the React quality metric, and compares methods with shared reward information.Experiments use 3B–4B models; the authors note this limits claims at larger scale.Source: §4.1 · pp. 6–7

4.2 BoT-GRPO vs. RL Baselines (React)

  • BoT-GRPO crosses every React compile-rate threshold first and wins against all four baselines throughout training, with the strongest VLM win rates against GRPO and PURE.Its win rates are 62–90% against GRPO, 67–78% against PURE, 56–78% against GSPO, and 58–63% against DAPO.Source: Figure 1 · p. 8, Figure 2 · p. 8

4.3 Cross-Base-Model Behavior (React)

  • Across Qwen2.5-3B, SmolLM3-3B, and Phi-4-mini-reasoning, BoT-GRPO reaches higher stable React compile rates, with the largest difference on Qwen2.5-3B.Qwen2.5-3B exceeds 90% compilation, while the other two models reach 60–70%.Source: §4.3 · pp. 8–9, Figure 3 · p. 9

4.4 Minimal Reward Recipe: What Signals Are Necessary? (React)

  • Compiler and ESLint signals alone yield about 74% compile rate but stagnate on visual quality; an LLM code judge is necessary, while a VLM judge is optional.The VLM can further improve perceptual defects, but must be introduced carefully to avoid destabilizing convergence.Source: §4.4 · p. 9, Figure 4 · p. 9

4.5 AIME Corroboration

  • On AIME, BoT-GRPO gains 3.0 points on Pass@1, 8.1 on Pass@2, and 5.5 on Pass@4 over GRPO while converging in roughly half the training steps.BoT-GRPO leads GRPO throughout training on all three tested base models.Source: §4.5 · pp. 9–10, Figure 5 · p. 10, Figure 6 · p. 10

5 Conclusion

BoT-GRPO is a critic-free, length-invariant extension of GRPO that speeds convergence on React and AIME. The reported comparisons are promising, but their scope and experimental setup constrain the claims.

5 Conclusion

  • On React, BoT-GRPO converges up to 1.9× faster than GRPO and faster than GSPO, DAPO, and PURE, with higher final compile and VLM-judged win rates; the advantage holds on AIME.The method adds no value network or verifier and has O(∑k Lk) overhead.Source: §5 · p. 10
  • The AIME evaluation uses public 1983–2024 problems that may overlap pretraining, limiting absolute claims while leaving the relative comparison intact.The AIME step-evaluation reward also requires an LLM judge, adding cost, latency, and noise.Source: §5 · p. 10
  • Each configuration uses a single seed unless noted, so reported point estimates and speedups are upper bounds.The authors identify multiple seeds, larger-scale and longer-rollout models, and broader process-reward baselines as next steps.Source: §5 · p. 10

A AIME Reward-Design Ablations

These AIME ablations use Qwen2.5-3B and examine local reward strategy, local reward weight, and step-evaluation window size. The main-text defaults are error propagation, local weight 0.5, and window 3.

A AIME Reward-Design Ablations

  • All ablation runs use Qwen2.5-3B unless noted; the main-text defaults are error propagation, ws=0.5, and window 3.The appendix varies each default choice.Source: A AIME Reward-Design Ablations · p. 14

Local Reward Strategy

  • Error propagation substantially outperforms context-aware and independent local reward strategies, which fail to learn effectively and remain near baseline.Independent scoring misses cascading errors, while context-aware scoring adds variance through dependence on the judge’s prior outputs.Source: A AIME Reward-Design Ablations · p. 14, Figure 7 · p. 15

Local Reward Weight

Step Evaluation Window

B React VLM Reward Stability

Deterministic line-level rewards accelerate convergence reliably, while local VLM rewards are noisier despite being semantically richer. Down-weighting local VLM contributions and separating screenshot scoring from code attribution improves compile-rate stability.

B React VLM Reward Stability

  • Deterministic line-level rewards accelerate convergence reliably, whereas line-level VLM rewards are noisier but semantically richer.Source: B React VLM Reward Stability · p. 15
  • 0.236 peak-to-final compile rate drop with full-weight 1-turn local VLM fell to 0.108 at w=0.2 and 0.020 with 2-turn scoring, versus 0.050 for global-only VLM.Lower drops indicate greater stability.Source: Figure 10 · p. 16, B React VLM Reward Stability · p. 16
  • The 1-turn judge sees both screenshot and source code, while 2-turn scoring locks the screenshot-only score before attributing issues to code lines.The 2-turn protocol reduces the opportunity for code content to bias the visual score.Source: B React VLM Reward Stability · p. 16

C Reward Hacking Under Compiler+Linter-Only Reward

Compiler-and-linter-only rewards can favor syntactically acceptable React output that renders a blank page. The example shows why compile rate alone does not measure whether generated code produces visible output.

C Reward Hacking Under Compiler+Linter-Only Reward

  • Compiler-and-linter-only reward reached ∼74% validation compile rate yet produced apps that rendered a blank page.For the illustrated batch, compile success was 1.0 even though React rendered nothing.Source: C Reward Hacking Under… · p. 16
  • The representative rollout passes static checks but contains unterminated className strings and repeated meaningless tokens, leaving its app blank.The policy emits a syntactically plausible JSX skeleton with a container, heading, and labelled inputs.Source: C Reward Hacking Under… · p. 16
  • Compile rate alone is inadequate because syntactic rewards cannot detect whether a program produces visible output; adding the LLM code judge removes this degenerate solution.Source: C Reward Hacking Under… · p. 17

D Judge Prompts

The appendix documents the model-based judges and how their global scores and line attributions contribute to rewards and win-rate evaluation. It specifies distinct protocols for code assessment, screenshot scoring, and pairwise comparison.

D Judge Prompts

  • Judge prompts for the React training and pairwise VLM judges are reproduced for reproducibility, including global-only and line-level variants; all run on Amazon Bedrock.Source: D Judge Prompts · p. 17
  • Line-level judges identify up to three negatively impactful lines; their impacts are divided by 10, combined with compiler and ESLint penalties, then spatially decayed into token penalties.Malformed or missing scores yield zero, and malformed attribution entries add no local penalty.Source: D Judge Prompts · p. 17

D.1 LLM Code Judge

  • The LLM code judge scores source for validity, task accomplishment, and aesthetic intent, with a non-React sentinel; its line-attribution version adds up to three offending lines without another judge call.Source: §D.1 · p. 18

D.2 VLM Screenshot Judge

  • The global-only VLM judge sees the rendered screenshot and prompt, not source code, so its score captures perceptual quality independently of the code judge.Source: §D.2 · p. 19
  • In the 1-turn VLM protocol, screenshot and numbered source arrive in the same call, so code is not withheld during screenshot scoring.Source: §D.2 · p. 19
  • In 2-turn scoring, Turn 1 fixes the screenshot-based score before Turn 2 receives source for line attribution, preventing code from changing the visual score.The protocol adds one attribution call before retries.Source: §D.2 · p. 20

D.3 Pairwise VLM Judge

  • Pairwise VLM evaluation uses four ABBA presentation rounds to reduce position bias; majority vote selects the winner, with a 2–2 split worth 0.5 to each model.The judge is instructed to pick the better app based on completeness, layout, visual polish, and usability.Source: §D.3 · p. 21

Related papers

Loading 2610.09804v1…