Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning

1. Introduction

• Existing RL approaches commonly rely on sparse, verifiable rewards from single-domain tasks, while open-ended multi-domain reasoning lacks standard answers for generalizable reward design.The paper identifies this as a challenge for training efficiency.Source: §1 · p. 1
• Purely online RL explores a narrow policy space because it uses limited on-policy samples and short-horizon updates, restricting access to diverse reasoning trajectories and higher-quality solutions.The paper frames this as a separate limitation from reward sparsity.Source: §1 · p. 2
• The paper introduces RGR-GRPO, which adds dense rubric-based rewards and offline rollout refinement to GRPO for multi-domain reasoning.The framework is motivated by preliminary findings that rubrics provide informative rewards and rubric-guided self-refinement improves responses across training stages.Source: §1 · p. 2
• Question-specific rubrics combine weighted factual criteria for answer accuracy with process criteria for reasoning soundness, and an LLM judge aggregates criterion scores into scalar rewards.The paper presents this reward design as a way to extend verifiable feedback beyond mathematics and code.Source: §1 · p. 2
• For off-policy guidance, RGR-GRPO uses rubric failures in high-reward rollouts as targeted prompts for self-refinement into improved reasoning trajectories.The authors describe this as expanding exploration beyond the on-policy solution space.Source: §1 · p. 2
• RGR-GRPO outperforms outcome-based reward methods on the 7B model, with average gains of 7.0% in mathematics, 5.4% in physics, 8.4% in chemistry, and 6.6% in general reasoning.It also maintains smoother entropy decline and outperforms standard RL across SciBench pass@k settings.Source: §1 · p. 2

2.1. Preliminary

• GRPO compares rewards among responses sampled for the same question, normalizes them into relative advantages, and uses those advantages to update the policy without a separate value function.The update constrains policy shifts through clipping and KL regularization.Source: §2.1 · pp. 2–3

2.2. Rubric-based Fine-Grained Rewards

• Rubric criteria separately assess factual correctness of intermediate and final answers and the logical soundness of reasoning steps.This separates factual verification from process evaluation.Source: §2.2 · p. 3
• RGR-GRPO builds rubrics from each question and a reference answer, with criteria carrying descriptive specifications and adaptive weights.An expert LLM generates and filters reference answers before generating the question-conditioned rubrics.Source: §2.2 · p. 3
• Binary criterion scores are aggregated with criterion weights into a scalar reward, with process rewards considered only when factual verification fails.A response satisfying every factual criterion receives full reward even if its reasoning deviates from the predefined rubrics; the design is intended to balance reward reliability and density while reducing reward hacking and intra-group variance.Source: §2.2 · pp. 3–4

2.3. Rubric-Guided Offline Exploration

• Exploration assessment checks whether the best on-policy response satisfies every rubric criterion; if it does, further off-policy exploration is skipped, otherwise refinement follows.The authors say this avoids unnecessary off-policy updates and reduces entropy-explosion risk.Source: §2.3 · p. 4
• When the best response fails criteria, rubric-based self-refinement conditions the policy on that response and its unsatisfied criteria to generate a refined rollout.The failed criteria are concatenated in order in the refinement prompt.Source: §2.3 · p. 4
• Mix-Policy GRPO combines refined off-policy rollouts with initial on-policy responses for a joint policy update.The off-policy refinement term is shaped to upweight low-probability tokens in refined trajectories, encouraging learning from successful out-of-distribution behavior while limiting failed refinements.Source: §2.3 · pp. 4–5
• The framework is presented as balancing on-policy stability with off-policy exploration flexibility.Its overview depicts rubric construction, exploration assessment, refinement when needed, and reshaped sampling probabilities for policy updates.Source: Figure 2 · p. 3, §2.3 · p. 5

• The study uses about 10k WebInstruct-Verify samples and evaluates mathematics, physics, chemistry, and general reasoning against on-policy reward and off-policy strategy baselines.On-policy rewards include binary outcome verification, Likert comparison, and rubric aggregation; off-policy baselines include LUFFY and Critique-GRPO.Source: §3 · p. 5
• Ground-truth-based critiques produce more stable and stronger performance than critiques from a strong model under this setting.Source: §3 · p. 5
• Training runs for 400 steps with batch size 96, learning rate 1×10−6, and eight rollouts per prompt, including one off-policy rollout for RGR-GRPO and off-policy baselines.The setup removes GRPO probability-ratio clipping and KL regularization by setting β = 0.Source: §3 · p. 5

4.1. Main Results

• On the 7B model, RGR-GRPO outperforms the base model by an average of 25.1%, the official instruct model by 6.7%, and second-best on-policy Rubric-GRPO by 3.7%.RGR-GRPO maintains stable hybrid-policy training and consistently delivers the best results.Source: §4.1 · p. 5

4.2. Out-of-distribution Performance

• With medical and computer science data comprising less than 1% of training, RGR-GRPO achieves the best OOD performance on MedMCQA and CS-Bench.All evaluated methods improve after RL training.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3. Analysis of Pass@k Curves Across Subjects

• The base model surpasses on-policy R1-GRPO at k=64 on Sci-Math, while the performance gap narrows as k grows.At Pass@1, RL-trained models already substantially outperform the base model.Source: §4.3 · p. 6
• RGR-GRPO achieves superior Pass@k performance across physics, chemistry, and math, with improvements persisting more steadily as k increases.The supplied figure covers Qwen2.5-7B on Sci-Bench.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4. Policy Exploration in GRPO with Rubric Guidance

• RGR-GRPO entropy falls early, then fluctuates between 0.2 and 0.4, while on-policy Rubric-GRPO rapidly collapses in entropy and converges quickly.RGR-GRPO continues improving through off-policy rubric guidance as on-policy gains saturate.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5. Ablation Study

• The ablation study reports that each examined rubric category and off-policy shaping and exploration-assessment configuration contributes to overall performance.The supplied table caption identifies the results as average-performance ablations.Source: §4.5 · p. 7, Table 2 · p. 7

in Reward Enrichment and Self-Refinement

• The section isolates rubrics’ roles in reward provision and self-refinement within dynamic RL training.The authors present these analyses as motivation for RGR-GRPO.Source: in Reward Enrichment and Self-Refinement · p. 7

5.1. Rubric-Based Dense and Effective Rewards

• Outcome rewards yield lower average reward because their sparse 0–1 feedback leaves many sampled groups without informative signals.The comparison uses online training with three reward mechanisms.Source: §5.1 · p. 7, Figure 7 · p. 8
• Likert rewards are dense but can be noisy and overly dependent on judge preference, leading to unstable optimization and mediocre MMLU-Pro performance.The cited comparison attributes the outcome to judge-model preference.Source: §5.1 · pp. 7–8
• Rubric rewards use explicit criteria and binary judgments per aspect to produce dense, interpretable feedback and the best MMLU-Pro outcomes.The authors report steady performance improvement under this reward.Source: §5.1 · p. 8, Figure 7 · p. 8

5.2. Rubric-Guided Self-Refinement at Test Time

• The self-refinement evaluation compares no rubric guidance, factual criteria alone, and both factual and process criteria across Outcome-GRPO checkpoints.It evaluates reasoning accuracy using 4K additional multi-domain WebInstruct-Verify samples and Math-Verify.Source: §5.2 · p. 8
• Rubric-guided self-refinement consistently improves generations, and its gains remain stable as training progresses.The figure reports improvements across different RL training stages.Source: §5.2 · p. 8, Figure 8 · p. 8
• The authors report potential for rubrics to improve exploration when on-policy learning signals become saturated.They also describe rubrics as enhancing reasoning robustness during policy rollouts.Source: §5.2 · p. 8

Rubric-based Approaches

• Prior rubric-based methods structure evaluation criterion by criterion, compute fine-grained RL rewards, or use rubric-guided rollout signals.The paper restricts its rubrics to factual and process dimensions to address risks from unconstrained design, including reward hacking and conflicting objectives.Source: §6 · p. 8

Off-policy Guided Exploration

• Off-policy guidance can expand exploration, but unconstrained rollouts in complex reasoning may cause entropy explosion or semantic drift.RGR-GRPO combines exploration assessment, rubric-constrained guidance, and adaptive length penalties to preserve stability and diversity.Source: §6 · p. 8

7. Conclusion

• Across 14 multi-domain benchmarks, RGR-GRPO consistently outperforms existing RL baselines.It also maintains stable entropy fluctuations during training and achieves superior pass@k performance.Source: §7 · p. 9

A.1. Datasets

• WebInstruct-Verify provides verified ground-truth answers across multiple disciplines, enabling comparison with outcome-based reward baselines.The authors randomly sampled subsets of the dataset and filtered out simple questions using Likert scores.Source: §A.1 · p. 12
• About 10k high-quality, multi-domain training examples were retained after filtering generated responses and reference answers.Initial responses came from Qwen2.5-7B-Base; prompts with excessively long O3-generated reference answers were removed.Source: §A.1 · p. 12
• Factual and Process rubrics were derived from each prompt and its reference answer.The training data were concentrated mainly in mathematics, physics, and chemistry; Computer Science and Health were smaller shares but performed well in their respective evaluations.Source: §A.1 · p. 12

A.2. Training

• Table 3 summarizes the distribution of rubric types across training and validation examples.The caption identifies the scope but does not state the counts.Source: Table 3 · p. 12
• GRPO training used Qwen2.5-Base with Verl 2 and vLLM 3 as the rollout generator.Training ran on 8 H100 GPUs, while the LLM-as-Judge reward service ran across machines with 4 L40S GPUs each.Source: §A.2 · p. 12

A.3. Evaluation

• The evaluation suite covers general scientific, mathematical, physics, chemistry, and out-of-distribution reasoning tasks.The listed benchmarks include MMLU and GPQA, Sci-Math and MATH, PIQA and Sci-Physics, ChemBench and Sci-Chemistry, plus CS-Bench and MedMCQA.Source: §A.3 · p. 12
• A unified evaluation protocol supports consistent comparison across diverse scientific domains, with temperature 0.0 and maximum response length 8192.The authors follow the Language Model Open Science Evaluation framework for standardized, reproducible benchmarking.Source: §A.3 · pp. 12–13

B. RGR-GRPO Algorithm

• RGR-GRPO first assesses exploration needs, then refines the best rollout based on failed rubrics, and finally merges on- and off-policy rollouts for policy updates.The algorithm input includes a pretrained policy, reward model, rubric-based reward function, question set, and self-refinement template.Source: B. RGR-GRPO Algorithm · p. 13, Algorithm 1 RGR-GRPO: Reward and… · p. 14

Algorithm 1 RGR-GRPO: Reward and Guidance through Rubrics

• When on-policy exploration suffices, the algorithm performs an on-policy GRPO update and proceeds to the next iteration.The flow then identifies failed rubric items and generates an off-policy refined response conditioned on those failures.Source: Algorithm 1 RGR-GRPO: Reward and… · p. 14
• The final Mix-Policy GRPO step shapes off-policy probability ratios and optimizes an objective combining on- and off-policy terms.The algorithm separately labels the off-policy and on-policy objectives.Source: Algorithm 1 RGR-GRPO: Reward and… · p. 14

C. Necessity of Exploration Assessment

• EA is necessary for training stability because it avoids unnecessary high-variance updates that can cause distributional shift and entropy explosion.The theorem characterizes EA as an adaptive variance controller.Source: C. Necessity of Exploration Assessment · p. 15, §3 · p. 16

1. Definitions: Gradient Estimators and Distributions

• The on-policy distribution samples initial rollouts from the old policy, while the refinement distribution conditions on the best response and failed rubric criteria.When failed criteria make refinement necessary, the refinement distribution differs from the on-policy distribution.Source: §1 · p. 15

2. Variance Analysis of Gradient Estimators

• The mixed-policy estimator combines G−1 on-policy samples with one refined off-policy sample, whose importance-sampling mismatch raises its variance above the on-policy estimator.The mismatch arises because the refined response is sampled from πrefine while the importance-sampling denominator is based on πon.Source: §1 · p. 15, §2 · p. 15

3. The Role of Exploration Assessment as an Adaptive Controller

• When on-policy exploration finds a perfect solution, EA uses the low-variance update; when it does not, EA uses the higher-variance mixed update to learn from an out-of-distribution trajectory.The latter is a deliberate stability-for-exploration trade-off that expands the exploration boundary.Source: §3 · p. 16

D. Analysis of Off-Policy Training Stability

• LUFFY and Critique-GRPO collapsed around 100+ steps, with uncontrolled spikes in entropy, response length, and KL divergence; RGR-GRPO’s curves were nearly identical to stable on-policy Rubric-GRPO.The authors attribute the reproduced baselines’ instability primarily to excessive distributional shift from off-policy guidance.Source: D. Analysis of Off-Policy Training… · pp. 16–17

E. Prompt Templates

• The experiments use Qwen3-32B as the judge for Likert- and rubric-based rewards, a prompt-policy model for refined responses, and O3 to generate question-specific rubrics.Refined responses are conditioned on failed rubrics.Source: E. Prompt Templates · p. 17, §E.2 · p. 18, §E.3 · p. 19

E.1. LLM-Judge Prompts

• The Likert judge checks a single criterion as satisfied or not, while the reference-based judge rates overall response quality on a 1-to-10 scale.The prompt templates ask each judge to return a JSON rating.Source: §E.1 · pp. 17–18

E.2. Self-Refinement with Failed Rubrics

• The refinement prompt instructs the model to revise its previous response to satisfy all rubric items and put its final answer in a box.It requests clear, concise, step-by-step reasoning.Source: §E.2 · p. 18

E.3. Rubric Generation

• Rubrics are generated from the question, reference answer, and ground truth, with factual criteria covering verifiable final facts or answers and process criteria covering key intermediate steps, calculations, theorems, or reasoning needed to derive the final answer.Each rubric item has exactly two keys, description and weight; weight is an integer from 1 to 5, with 5 the most important.Source: §E.3 · p. 19