PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning

Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Kongcheng Zhang, Jiale Zhao, Jingwen Yang, Yihe Zhou, Jianwei Lv, Tongya Zheng, Hengtong Lu, Wei Chen, Yan Xie, Mingli Song

arXiv:2508.16949v7cs.LGcs.AI

TL;DR

RL improvement depends on high-quality samples, yet exploration is bounded by LLM capabilities, and open-ended tasks lack single verifiable answers. RuscaRL combines rubric guidance during rollout generation with rubric-based rewards for training; experiments report competitive performance and expanded reasoning boundaries.

  • Problem

    Open-ended tasks require multidimensional evaluation without a single verifiable ground-truth answer, while RL exploration for high-quality samples remains bounded by LLM capabilities.

    Source: §1 · pp. 1–2

  • Method

    RuscaRL uses checklist-style rubrics as differentiated, gradually decayed rollout guidance and as criteria for verifiable rewards during RL training.

    Source: §1 · p. 2

  • Results

    RuscaRL consistently outperforms strong baselines, competes with leading models, and produces novel responses that the initial model could barely generate, expanding the reasoning boundary.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that broader investment is needed in open, diverse, and domain-rich rubric datasets.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities. Despite the encouraging results, a fundamental dilemma persists as RL improvement relies on learning from high-quality samples, yet the exploration for such samples remains bounded by the inherent limitations of LLMs. This, in effect, creates an undesirable cycle in which what cannot be explored cannot be learned. In this work, we propose Rubric-Scaffolded Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework designed to break the exploration bottleneck for open-ended reasoning. Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit scaffolding for exploration during rollout generation, where different rubrics are provided as external guidance within task instructions to steer diverse high-quality responses. This guidance is gradually decayed over time, encouraging the model to internalize the underlying reasoning patterns; (2) verifiable rewards for exploitation during model training, where we can obtain robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL on open-ended reasoning tasks. Extensive experiments demonstrate the superiority of the proposed RuscaRL across various benchmarks, effectively expanding reasoning boundaries under the Best-of-N evaluation. Our code is available at https://github.com/IANNXANG/RuscaRL.

1. Introduction

Open-ended tasks lack a single verifiable answer, while RL exploration remains bounded by models’ existing capabilities. RuscaRL uses checklist-style rubrics to guide exploration and provide training rewards, and achieves competitive results, including parity with leading models on HealthBench-500.

1. Introduction

  • Open-ended tasks such as medical consultation and creative writing require multidimensional evaluation and lack a single verifiable ground-truth answer.Rubric-based evaluation decomposes responses into checklist criteria and aggregates LLM-as-a-Judge scores into scalar rewards for RLVR.Source: §1 · pp. 1–2
  • RuscaRL enables Qwen3-30B-A3B-Instruct-2507 to perform on par with OpenAI-o3 on HealthBench-500.Figure 1 compares different LLMs on HealthBench-500.Source: Figure 1 · p. 2, §1 · p. 2
  • RL needs high-quality samples, but exploration is bounded by LLM capabilities, while policy entropy can collapse toward limited reasoning trajectories.Prior methods often favor high-reward responses already supported by base models rather than expanding reasoning boundaries.Source: §1 · p. 2
  • RuscaRL uses checklist-style rubrics as external rollout guidance, varying scaffolding within groups and gradually removing it to encourage internalization.Rubric criteria also support binary grader judgments aggregated into reward signals for RL on general tasks.Source: §1 · p. 2
  • The framework combines rubric scaffolding for exploration with rubric-based verifiable rewards for exploitation during RL training.The authors describe instructional scaffolding as a way to improve rollout diversity and quality for exploration.Source: §1 · p. 2

2. Related Works

Prior work applies RL with verifiable rewards to math and coding, and rubric-based rewards to general tasks, but exploration remains constrained. RuscaRL contrasts its structured, decaying rubric guidance with approaches that stay within the initial policy distribution or offer coarse signals.

2. Related Works

  • RL with verifiable rewards has improved reasoning in math and coding, but remains difficult to extend to general tasks where correctness is hard to verify.Prompt engineering and supervised fine-tuning also face limits in scalability and cross-domain generalization.Source: §2 · p. 3
  • Rubric-based RL uses LLM graders to provide fine-grained feedback on tasks without ground truth, with reported applications in agentic search, instruction following, and medical consultation.Source: §2 · p. 3
  • Existing exploration approaches are described as either exploring within the initial policy distribution or providing coarse guidance, whereas RuscaRL uses verifiable rubric criteria and scaffolding decay.Source: §2 · p. 3

3. Preliminary

The paper adopts GRPO for RL training with rubric-based rewards. Its group-based sampling and advantage estimation use response rewards relative to the sampled group.

3. Preliminary

  • For each instruction, GRPO samples a group of G responses from the old policy and optimizes the current policy with a group-based objective.The instruction is sampled from the training-data distribution.Source: §3 · p. 3
  • Unlike PPO, GRPO eliminates the need for a value model by using group-based advantage estimation.Source: §3 · p. 3
  • The advantage is normalized using the mean and standard deviation of the G sampled rewards within each group.Each response’s reward is denoted ri.Source: §3 · p. 3

4. Methodology

RuscaRL uses rubrics both to guide rollout exploration and to provide criterion-based rewards for training. Its methodology combines varied, gradually withdrawn guidance with rubric-scored rewards for open-ended responses.

4. Methodology

  • RuscaRL combines rubric-guided rollout generation, with intra-group differentiation and inter-step decay, and rubric-based rewards for model training.The rubric criteria are provided as external guidance during rollout generation and used to assess responses during training.Source: §4 · p. 3, §4.2 · p. 5, §4.3 · p. 5

4.1. Rubric-based Evaluation System

  • A rubric is a set of verifiable criteria with point values; an LLM grader judges each criterion as satisfied or not, and the results are aggregated into a score.The system forms a criterion score vector by multiplying each binary judgment by its criterion’s points, and computes the total possible score from positive-point criteria for normalization.Source: §4.1 · p. 4

4.2. Rubric-based Scaffolding for RL Exploration

  • Intra-group differentiation assigns samples within each group different rubric-scaffolding levels, giving some stronger guidance and others weaker guidance to encourage response diversity.The group-level scaffolding ratios decrease linearly across samples.Source: §4.2 · p. 5
  • Inter-step scaffolding decay gradually reduces rubric guidance over training, transitioning from initial support toward independent reasoning.The decay uses a sigmoid function over training progress, with a midpoint and steepness parameter.Source: §4.2 · p. 5
  • The integrated scaffolding ratio combines inter-step decay with each sample’s group-level ratio, jointly varying guidance within groups and across training.The rubric subset size is determined by the scaffolding ratio and the size of the full rubric.Source: §4.2 · p. 5

4.3. Rubric-based Reward for RL Exploitation

  • Rubric criterion scores are summed and normalized by the total possible positive-point score to produce the reward used for RL training.The resulting scores are generally in [0, 1], though occasional negative scores are possible; training maximizes expected reward using a policy-gradient framework.Source: §4.3 · p. 5

5. Experiments

Across medical, writing, and instruction-following tasks, RuscaRL outperforms rubric-based and other baseline methods in most comparisons, while its analysis and ablations examine exploration behavior and scaffolding choices. Experiments cover multiple model series and benchmark domains, with analyses focused on Qwen2.5-7B-Instruct and HealthBench.

5. Experiments

  • Experiments span medical, writing, instruction-following, and STEM benchmarks, using Qwen and Llama model families trained with GRPO.The evaluated benchmarks include HealthBench-500, LLMEval-Med, MedQA, MedMCQA, WritingBench, Creative Writing v3, IFEVAL, and IFBench.Source: §5 · p. 5, §5.1 · pp. 5–6

5.2. Overall Performance

  • RuscaRL achieves the best performance on most tasks, including 56.4 vs. 52.0 and 36.8 accuracy on HealthBench-500 with Qwen2.5-7B-Instruct.The comparison favors RuscaRL over Rubric-based RL and MeRF in the direct RL setting; in SFT-then-RL, both methods improve over SFT, with RuscaRL gaining more on most tasks.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3. Analysis

  • At N=2048, RuscaRL surpasses the initial model and Rubric-based RL, while its steeper Best-of-N curve reaches equal performance with fewer samples.The analysis uses Best-of-N to reflect reasoning boundaries at large N and sampling efficiency at small N.Source: §5.3 · p. 7, Figure 3 · p. 8
  • RuscaRL generates more novel responses and maintains higher training-time diversity, while Rubric-based RL shows faster diversity collapse, especially on semantic distance.RuscaRL’s diversity rises rapidly early in training, then remains relatively high with a gradual decline.Source: §5.3 · pp. 7–8, Figure 4 · p. 8
  • RuscaRL’s policy entropy first rises and then declines, while validation accuracy remains best throughout training; Rubric-based RL instead plateaus as entropy continuously declines.Figure 3 also displays the sigmoid scaffolding-decay schedule.Source: §5.3 · p. 7, Figure 3 · p. 8

5.4. Ablation Studies

  • Linear intra-group differentiation outperforms both no-scaffolding and fully scaffolded binary settings, with gains attributed to varied scaffolding levels that improve diversity.The strategy works synergistically with multi-sampling algorithms such as GRPO.Source: §5.4 · p. 8
  • The sigmoid decay function performs best among tested schedules, whereas constant full scaffolding performs much worse.The schedule provides early guidance for exploration, then gradually reduces external guidance so the model can internalize useful reasoning patterns; linear and power decay perform worse.Source: §5.4 · pp. 8–9, Figure 5 · p. 9
  • With Qwen2.5-7B-Instruct on HealthBench, the best sigmoid parameters are α = 125 and t0 = 0.2, and performance remains strong across a wide hyperparameter range.Extreme settings degrade performance: overly fast or slow removal can cause instability, insufficient early exploration support, or overfitting to scaffolding.Source: §5.4 · p. 9, Figure 5 · p. 9

6. Conclusion and Discussion

RuscaRL applies instructional scaffolding theory to open-ended LLM reasoning, combining gradually decaying rubric guidance with rubric-based rewards. Its conclusion reports consistent gains over strong baselines and novel responses beyond the initial model, while the pseudocode summarizes the training loop.

6. Conclusion and Discussion

  • RuscaRL combines gradually decaying rubric guidance with rubric-based rewards to support effective RL training on general tasks.The conclusion describes rubrics as external guidance that encourages internalization and as references for robust, fine-grained evaluation.Source: §6 · p. 9
  • RuscaRL consistently outperforms strong baseline methods and achieves competitive results compared with leading models.The conclusion also reports that its fine-tuned model produces responses the initial model could barely generate, expanding the reasoning boundary.Source: §6 · p. 9
  • The authors identify broader investment in open, diverse, domain-rich rubric datasets as an urgent need and list rubric feedback, multimodal tasks, and agent systems as future work.Source: §6 · p. 9

A. Algorithm Pseudocode

  • The training procedure includes intra-group scaffolding differentiation, inter-step scaffolding decay, and rubric-based reward computation.The algorithm pseudocode shows these components in the rollout, evaluation, and update loop.Source: A. Algorithm Pseudocode · p. 15

B.1. Detailed Training Settings

The section specifies the models, training data, configurations, and evaluations used, then reports results across medical, STEM, training-strategy, and SFT comparisons. It also describes additional metrics for novelty and response diversity.

B.1. Detailed Training Settings

  • Training spans Qwen2.5, Qwen3, and Llama-3 models, with domain-specific datasets for medical, writing, instruction-following, and STEM tasks.The medical data use 4,500 HealthBench samples excluding HealthBench-500; other-domain rubrics are generated with GPT-4.1, and writing combines two LongWriter datasets.Source: §B.1 · p. 15
  • Most models use Temperature=0.7, Top-P=0.8, and Top-K=20; Qwen2.5-7B-Instruct instead uses Temperature=1.0 with Top-P and Top-K disabled after rapid policy entropy collapse.The t0 decay parameter varies by model, from 0.1 to 0.3.Source: §B.1 · p. 15

B.2. Detailed Evaluation Settings

  • Evaluations average three runs; medical and writing benchmarks use GPT-4.1 and Claude-Sonnet-4 judges, respectively, while IFEVAL and IFBench report prompt-level strict accuracy.Generation settings use Temperature=0.7, Top-P=0.8, and Top-K=20, with maximum lengths of 4,096 tokens for non-writing and 16,000 for writing tasks.Source: §B.2 · p. 16

C.1. Performance Across Different Models

  • RuscaRL improves performance across model series and scales, and consistently outperforms rubric-based RL on HealthBench-500 and LLMEval-Med for two Qwen3 instruction-tuned models.The latter comparison indicates that rubric scaffolding remains effective for stronger initial models; Tables 4 and 5 identify the respective comparison scopes.Source: §C.1 · p. 16, Table 5 · p. 17, Table 4 · p. 17

C.2. Performance in the STEM Domain

  • Qwen2.5-7B-Instruct shows consistent performance improvements across all evaluated STEM benchmarks.The benchmarks include GPQA-Diamond, MMLU, MMLU-Pro, MATH-500, AMC 2023, AIME 2024, and AIME 2025; Figure 6 presents STEM benchmark results.Source: Figure 6 · p. 16, §C.2 · p. 17

C.3. Mixed Training Analysis

  • Domain-specific training performs best across most benchmarks; health-only training works well medically but improves non-medical tasks less, while mixed training offers moderate gains across task categories.Health-only and domain-specific training use the same medical data and therefore have identical medical benchmark performance.Source: §C.3 · p. 17, Table 6 · p. 18

C.4. Supervised Fine-tuning vs. RuscaRL

  • GPT-4.1 demonstration SFT improves weaker Qwen2.5-7B-Instruct results on HealthBench-500 (+14.9) and WritingBench (+17.6), but degrades stronger Qwen3-30B-A3B-Instruct results on those benchmarks (-3.1 and -11.7).RuscaRL improves performance across both model scales, whereas the SFT outcomes vary with initial model capability.Source: §C.4 · p. 18, Table 7 · p. 18

C.5. Additional Metrics Analysis

  • Novelty is measured with sequence-likelihood importance ratios, while diversity uses 1-Self-BLEU and semantic distance across 16 responses per instruction.Semantic distance is the average cosine distance between answer embeddings; lower Self-BLEU indicates higher diversity.Source: §C.5 · pp. 18–19

C.5.1. NOVELTY ANALYSIS

The analysis finds higher novelty and a different diversity trajectory for RuscaRL than for rubric-based or conventional RL. RuscaRL produces high-importance responses and maintains diversity after early gains, while conventional RL diversity collapses faster.

C.5.1. NOVELTY ANALYSIS

  • The mean importance ratio reaches 5424.62 for RuscaRL, with 321 samples above 2, 11 above 10, and 7 above 100.The thresholds correspond to responses considered difficult, very difficult, and nearly impossible for the original model to generate.Source: §C.5.1 · p. 19, Table 9 · p. 19
  • RuscaRL’s top sample reaches ρseq = 2,638,481.94, versus 35.66 for rubric-based RL; its high-importance samples often have meaningful score improvements.The reported score differences include 0.54, 0.89, 0.67, and 0.86.Source: §C.5.1 · p. 19

C.5.2. DIVERSITY ANALYSIS

  • RuscaRL rapidly increases diversity early in training, then maintains a relatively stable high level with gradual decline, while conventional RL diversity collapses faster, especially in semantic distance.Source: §C.5.2 · pp. 19–20

C.6. Importance Sampling Analysis

The analysis weighs importance-ratio choices for scaffolded training data and a no-scaffold target policy. The theoretically unbiased correction has higher variance, while a practical alternative performs best across most benchmarks.

C.6. Importance Sampling Analysis

  • For a no-scaffold target policy trained on scaffolded data, the theoretically correct importance ratio provides an unbiased estimator but can have high variance from state mismatch.Source: §C.6 · p. 20
  • The ratio comparing no-scaffold policies acts as a proximal update rather than true importance sampling correction, and achieves the best performance across most benchmarks.The analysis links this approach to internalizing scaffolding knowledge while maintaining training stability.Source: §C.6 · p. 20
  • The ratio matching scaffold conditioning in numerator and denominator performs worse than the proximal alternative because it does not encourage scaffold-free reasoning patterns.Source: §C.6 · p. 20

C.7. Training Runtime

The appendix reports training-runtime optimizations, grader-selection results, and robustness tests under rubric noise. RuscaRL outperforms rubric-based RL under mild rubric perturbations, while both methods degrade under severe corruption.

C.7. Training Runtime

  • The reported synchronous configuration takes roughly 115 seconds per step, with rubric-based rewards adding roughly twofold training cost.This configuration uses 64 instructions, eight rollouts per instruction, and an average of 11.5 criteria per rubric; the authors characterize the cost as shared by rubric-based methods and worthwhile given open-ended-task gains.Source: §C.7 · pp. 20–21
  • Qwen3-30B-A3B reduces Reward-stage time from 60 to 18 seconds per step, with only slight degradation in HealthBench-500 performance.The authors also state that asynchronous rollout-reward overlap reduces latency, and that lightweight graders and pipeline optimizations can substantially reduce Reward-stage latency.Source: §C.7 · p. 21, Table 11 · p. 21

C.8. Grader Selection and Human Agreement

  • Qwen3-32B achieves Cohen’s Kappa of 0.76 and F1 of 0.90 against human judgments on 940 expert-annotated criteria.Stronger graders generally improve human agreement, but gains saturate at larger scales; stronger graders also generally improve downstream performance, with marginal gains beyond Qwen3-32B.Source: §C.8 · p. 21

C.9. Robustness to Noisy Rubrics

  • Under mild rubric noise, RuscaRL outperforms rubric-based RL on four medical benchmarks, while both methods degrade substantially under Inverse and Negated corruptions.The comparison covers Ambiguous, Contradictory, and 50% removed variants; Table 13 reports results over three runs.Source: §C.9 · p. 22, Table 13 · p. 22

D. Prompt Templates

Appendix D presents templates for rubric evaluation, rubric-guided scaffolding, and rubric data generation. The templates specify how graders assess criteria and how criteria guide responses or are constructed.

D.1. Grader Prompt Template

  • The grader prompt evaluates the last assistant response against one rubric item and returns an explanation plus a boolean indicating whether the criterion is met.For multi-part criteria, all parts must be met; examples introduced by “such as,” “for example,” or “including” need not all appear.Source: §D.1 · p. 23
  • The grader prompt requires false if any required criterion is unmet, while allowing examples after phrases such as “including” to be illustrative rather than exhaustive.This distinction is part of the prompt’s rules for handling multi-sentence or multi-criterion rubric items.Source: §D.1 · p. 23
  • For criteria with negative points, the grader still marks whether undesirable behavior occurred, so a good response can correctly receive false.The smoking example illustrates that accurate advice does not meet a criterion penalizing failure to provide accurate information.Source: §D.1 · pp. 23–24

D.2. Scaffolding Prompt Template

  • The scaffolding prompt inserts selected rubric criteria as explicit guidance, separating points to include from points to avoid.Either section is omitted when no corresponding criteria are available, and the response is instructed not to reveal the criteria.Source: §D.2 · p. 24

D.3. Data Generation Prompt Template

  • For datasets with ideal answers, rubric generation uses question-answer pairs; otherwise, GPT-4.1 first generates example solutions and then derives rubrics from them.The prompt asks for specific, measurable, binary-evaluable criteria with comprehensive coverage.Source: §D.3 · pp. 24–25
  • The data-generation prompt calls for 5–15 criteria covering factual accuracy, solution quality, answer consistency, and format compliance when applicable.Criterion points reflect relative importance, with positive scores from 1 to 10 and negative scores from -10 to -1.Source: §D.3 · pp. 25–26

E.1. Standard Deviations of Experiments

The appendix reports variability across repeated experiments and illustrates how rubric-based evaluation addresses shallow compliance. It contrasts keyword-only rules with evaluation across complementary response-quality criteria.

E.1. Standard Deviations of Experiments

  • Experiments are repeated three times with different random seeds, and Tables 14 and 15 report mean performance and empirical standard deviations.The tables extend the main results for models and training variants.Source: §E.1 · p. 26, Table 14 · p. 26, Table 15 · p. 26

F.1. Shallow Compliance Risk

  • A keyword-count rule can reward responses that satisfy the literal instruction while undermining its intended meaning.The appendix illustrates this risk with keyword-based instructions involving “delicious” and “innovation.”Source: §F.1 · p. 27
  • Rubric criteria for informativeness, coherence, genuine sentiment, and other complementary qualities can penalize shallow keyword spamming.The examples contrast keyword-rule success with rubric-aligned responses.Source: §F.1 · p. 27

F.2. Initial Model vs. RuscaRL-Trained Model

A HealthBench case study compares the initial Qwen2.5-7B-Instruct response with its RuscaRL-trained counterpart under the same query and rubric. The trained response receives a higher normalized rubric score and satisfies more positive criteria.

F.2. Initial Model vs. RuscaRL-Trained Model

  • The RuscaRL-trained response scores 1.00 versus 0.29 for the initial model on the HealthBench case, satisfying more positive criteria while only the initial response meets penalty criterion c4.Both responses satisfy c2 and c6; the trained response notably satisfies c1, c5, c9, and c10.Source: §F.2 · p. 27, Table 16 · p. 30
  • The rubric assigns positive points to appropriate context-seeking, hedging, emergency guidance, and accuracy, while penalizing inappropriate emergency escalation and failure to hedge.The listed criteria include context-seeking at +7, appropriate hedging at +6, and inappropriate emergency recommendations at -6.Source: §F.2 · pp. 27–28
  • The trained response asks for relevant symptoms and exposures and advises immediate care for red flags, whereas the initial response gives general self-care steps and suggests seeing a provider.The initial response includes hydration and avoiding solid foods; the trained response asks about symptoms and recommends immediate care for red-flag symptoms.Source: §F.2 · p. 28
  • The rubric marks criteria met individually and assigns each criterion’s point value when met, otherwise zero.Table 16 presents the criterion-by-criterion comparison between the two responses.Source: §F.2 · p. 27, Table 16 · p. 30
Loading 2508.16949v7…