PaperlayerView exact revision on arXiv

Source-linked AI summary

RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning

Yukun Chen, Jiaming Li, Longze Chen, Ze Gong, Jingpeng Li, Zhen Qin, Hengyu Chang, Ancheng Xu, Zhihao Yang, Hamid Alinejad-Rokny, Qiang Qu, Bo Zheng, Min Yang

arXiv:2602.21628v2cs.CL

TL;DR

Outcome-only rewards can encourage spurious reasoning, and rubric-based methods face costly instance-level generation and treat rubrics as equally learnable. RuCL stratifies generalized rubrics by competence and dynamically adjusts their weights during training. Across seven benchmarks it reports a 7.83% average performance gain and state-of-the-art performance among 7B-scale reasoning models.

  • Problem

    Outcome-based rewards emphasize final-answer correctness over intermediate reasoning, while rubric-based approaches face costly instance-level generation and treat rubrics as equally learnable.

    Source: §1 · pp. 1–2

  • Method

    RuCL applies curriculum learning to reward design by stratifying rubrics by model competence and dynamically shifting emphasis from foundational to advanced reasoning.

    Source: §1 · p. 2

  • Results

    RuCL reports a 7.83% average performance gain across seven benchmarks and a new state-of-the-art among 7B-scale reasoning models.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Integrated with GRPO, RuCL is reported to mitigate reward hacking and training instability.

    Source: §5 · p. 8

    RuCL relies on proprietary teacher LLMs and large-scale judges, and its static initial stratification assumes rubric difficulty remains constant during training.

    Source: §I · p. 28

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a prevailing paradigm for enhancing reasoning in Multimodal Large Language Models (MLLMs). However, relying solely on outcome supervision risks reward hacking, where models learn spurious reasoning patterns to satisfy final answer checks. While recent rubric-based approaches offer fine-grained supervision signals, they suffer from high computational costs of instance-level generation and inefficient training dynamics caused by treating all rubrics as equally learnable. In this paper, we propose Stratified Rubric-based Curriculum Learning (RuCL), a novel framework that reformulates curriculum learning by shifting the focus from data selection to reward design. RuCL generates generalized rubrics for broad applicability and stratifies them based on the model's competence. By dynamically adjusting rubric weights during training, RuCL guides the model from mastering foundational perception to tackling advanced logical reasoning. Extensive experiments on various visual reasoning benchmarks show that RuCL yields a remarkable +7.83% average improvement over the Qwen2.5-VL-7B model, achieving a state-of-the-art accuracy of 60.06%.

1. Introduction

Outcome-only RLVR can reward correct answers despite spurious intermediate reasoning, while rubric-based methods face costly instance-level generation and treat rubrics as equally learnable. RuCL applies curriculum learning to reward design, progressively aligning rubric difficulty with model competence.

1. Introduction

  • Outcome-based rewards overemphasize final-answer correctness, leaving models prone to spurious reasoning patterns and superficial shortcuts.Such reward hacking can produce contradictory or hallucinatory intermediate steps that still arrive at correct answers.Source: §1 · p. 1
  • Existing rubric-based methods incur high costs from instance-level rubric generation and treat rubrics as equally challenging throughout training.This can penalize complex logical failures before models master basic visual perception, producing noisy gradients and hindering convergence.Source: §1 · p. 2
  • RuCL shifts curriculum learning from data selection to reward design by organizing and scheduling rubrics according to their learnability.It aims to support structured, progressive acquisition of reasoning skills.Source: §1 · p. 2
  • RuCL generates generalized rubrics for shared reasoning primitives, stratifies them by initial competence, and dynamically shifts weight from foundational to advanced rubrics.Final-answer and rubric-based rewards are combined.Source: §1 · p. 2
  • The framework combines a data-driven rubric-construction pipeline, applicability-aware evaluation, and a performance-triggered scheduler for scalable rubric-based reward design.The contributions also describe RuCL as dynamically aligning rubric difficulty with model competence.Source: §1 · p. 2
  • RuCL achieves an average performance gain of 7.83% across seven benchmarks.The paper also reports detailed ablation studies validating its effectiveness.Source: §1 · p. 2

2. Related Work

Prior work explores scalable MLLM post-training, rubric-based assessment of reasoning, and curriculum learning, but the cited approaches leave gaps in reward design. RuCL applies curriculum learning to rubric weights and summarizes its framework as a two-stage process of rubric stratification and dynamic scheduling.

2. Related Work

  • Prior MLLM reasoning methods use rationale distillation, human preferences, iterative DPO, or RLVR, while sparse outcome-based rewards leave models prone to reward hacking.Examples of RLVR approaches include teacher-generated chain-of-thought, combined supervised and RL training, and reflection-aware rewards.Source: §2 · p. 2
  • Rubric-based rewards assess intermediate reasoning with explicit criteria, but existing approaches typically use instance-specific rubrics and treat them as equally learnable.Prior rubric applications span medical reasoning, code generation, and instruction following.Source: §2 · p. 2
  • Prior curriculum-learning approaches focus on data-level curricula, whereas RuCL dynamically adjusts rubric weights during reinforcement learning.The cited examples include multi-stage training and an easy-to-hard RL curriculum with online difficulty weighting.Source: §2 · p. 2
  • RuCL has two stages: construct and stratify generalized rubrics by empirical difficulty, then dynamically weight rubric rewards to shift focus from basic skills to complex reasoning.The scheduler adjusts the weight λ based on real-time performance.Source: Figure 2 · p. 3

3. Stratified Rubric-based Curriculum Learning (RuCL)

RuCL applies curriculum learning to reward design, organizing generalized rubrics by learnability rather than treating them as equally difficult. It combines rubric-based process supervision with outcome verification and progressively shifts emphasis toward advanced reasoning.

3.1. Problem Formulation

  • RuCL turns curriculum learning into a time-varying weighting scheme over rubric rewards, prioritizing learnable objectives before harder reasoning criteria.The curriculum changes reward design rather than selecting training data.Source: §3 · p. 3, §3.1 · p. 4

3.2. Phase I: Generalized Rubric Construction and Stratification

  • RuCL constructs reusable generalized rubrics for diverse reasoning tasks instead of generating ad hoc, instance-specific rubrics.The shared rubric pool supports principled difficulty stratification and scheduling.Source: §3.2 · p. 4
  • RuCL reduces rubric-generation overhead from O(N × Cgen) for instance-level methods to O(1 × Cgen) with a shared generalized pool.The passage states that this removes repetitive per-instance LLM calls without compromising evaluation density.Source: §3.2 · p. 4
  • The applicability-aware Judge separately records whether each rubric applies and whether the response satisfies it, assessing satisfaction only when applicable.This separation keeps non-applicable rubrics from distorting coverage and proficiency statistics.Source: §3.2 · p. 4
  • Applicability filtering removes low-coverage rubrics, while pass rates divide retained rubrics into high-pass foundational skills and low-pass advanced reasoning.The two tiers support reinforcing foundational skills before pivoting to more challenging reasoning.Source: §3.2 · pp. 4–5

3.3. Phase II: Dynamic Curriculum Learning

  • The hybrid reward combines stratified rubric-based reasoning evaluation with strict rule-based verification of final-answer correctness.A fixed α controls the trade-off between outcome correctness and rubric-based process supervision.Source: §3.3 · pp. 5–6
  • Advanced rubrics activate only after foundational performance exceeds a proficiency threshold throughout a sliding window, guarding against transitions after transient lucky guesses.The schedule then ramps the advanced-rubric weight over a defined duration and holds it at its peak after ramp-up.Source: §3.3 · pp. 5–6
  • RuCL modulates evaluative signals over the output space, with early foundational-rubric prioritization intended to reduce gradient interference from currently unachievable objectives.The paper describes this design as alleviating cold-start instability and supporting more stable, efficient optimization.Source: §3.3 · p. 6

4. Experiments

RuCL is evaluated on seven visual-reasoning benchmarks using ViRL-39K and Qwen2.5-VL-7B-Instruct. It outperforms the baseline across mathematical benchmarks and reports gains on general and logical tasks; ablations examine rubric scheduling, reward balance, and curriculum timing.

4.1. Experiment Setup

  • ViRL-39K supplies approximately 39,000 verifiable training pairs, and evaluation covers seven benchmarks spanning mathematical, general visual, logical, counting, and multidisciplinary reasoning.Training starts from Qwen2.5-VL-7B-Instruct; comparisons include proprietary, general-purpose open-source, and reasoning-focused models.Source: §4.1 · pp. 6–7

4.2. Main Results

  • 60.06% is RuCL’s highest average score across seven tasks, ahead of leading open-source reasoning models; it improves over Qwen2.5-VL-7B on all mathematical benchmarks.On WeMATH, RuCL rises from 58.52% to 71.49%; on MathVerse, from 48.98% to 54.14%.Source: §4.2 · p. 7
  • LogicVista improves from 39.26% to 49.66%, while MMMU gains 5.67% and Counting gains 12.00% over the baseline.The authors report that RuCL surpasses other open-source 7B competitors on LogicVista and reaches 85.50% on Counting.Source: §4.2 · p. 7
  • Training first prioritizes foundational rubrics, then introduces advanced constraints after proficiency stabilizes above 0.9, while foundational performance remains robust.The authors also report that rubric-based judging penalizes spurious reasoning chains that outcome-only supervision may miss.Source: §4.2 · p. 7

4.3. Ablation Study

  • The reported 57.13% result is exceeded by Uniform Averaging (57.56%) and Linear Stratification (58.41%); RuCL’s sigmoid schedule beats linear on Counting (85.50% vs. 79.50%) and LogicVista (49.66% vs. 47.43%).The comparison keeps the GRPO backbone and training data fixed; Table 3 reports ablations on General benchmarks.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • At α = 0.7, performance peaks at 71.49% on WeMath and 85.50% on Counting; moving to α = 0.5 lowers Counting to 79.00%, while α = 0.9 lowers WeMath to 53.78%.The authors attribute these drops respectively to excessive auxiliary constraints and diminished fine-grained supervision.Source: §4.3 · p. 8, Figure 3 · p. 8
  • w = 20 achieves the best overall performance, w = 30 is close, and w = 10 underperforms; the authors use w = 20 as the default.They suggest shorter windows are more vulnerable to transient fluctuations and premature curriculum transitions.Source: §4.3 · p. 8, Table 4 · p. 8

5. Conclusion

RuCL shifts curriculum learning from data selection to reward design by stratifying rubrics into foundational and advanced categories. Across seven benchmarks, it outperforms the base model and establishes a new state-of-the-art among 7B-scale reasoning models.

5. Conclusion

  • RuCL reframes curriculum learning around reward design, stratifying rubrics so reward signals align with evolving model capabilities.The authors report experiments across seven benchmarks and state that RuCL outperforms the base model and is state-of-the-art among 7B-scale reasoning models.Source: §5 · p. 8

Impact Statement

The impact statement presents RuCL as a curriculum framework for MLLM reasoning that progresses from foundational perception to advanced deduction. The appendix describes GRPO as a policy-optimization approach that uses group statistics and a clipped objective with a KL penalty.

Impact Statement

  • RuCL shifts curriculum focus from data selection to reward design, guiding MLLMs from foundational perception toward advanced deduction.The impact statement says training and evaluation use publicly available datasets under their licenses and usage policies.Source: Impact Statement · p. 9

A. Group Relative Policy Optimization (GRPO)

  • GRPO estimates each sampled output’s advantage by normalizing its reward against group statistics, avoiding a separate value-function critic.Its objective uses a PPO-style clipped loss and penalizes deviation from a reference model with a KL-divergence term.Source: A. Group Relative Policy Optimization… · p. 13
  • The policy objective uses rewards derived from the stratified rubrics to optimize the model.The formulation is described as stable and efficient without the memory burden of a critic model.Source: A. Group Relative Policy Optimization… · p. 13

B. Theoretical Derivation and Analysis of Gradient Variance

The derivation isolates the rubric component of RuCL’s policy-gradient estimator to analyze its variance under curriculum modulation. It treats the outcome reward as a fixed-weight addend that does not affect variance with respect to λt.

B. Theoretical Derivation and Analysis of Gradient Variance

  • The analysis considers the curriculum-modulated rubric reward while omitting baselines and advantage normalization; its reasoning also applies to advantage-based estimators.The outcome term is a fixed-weight addend and is excluded from the variance decomposition with respect to λt.Source: B. Theoretical Derivation and Analysis… · p. 13

B.1. Gradient Estimator Decomposition

  • The derivation begins from the standard policy-gradient objective, whose expected reward is taken over trajectories sampled from the policy.The gradient estimator is then expressed for training step t.Source: §B.1 · p. 13
  • RuCL represents the rubric reward as a dynamic convex combination of foundational and advanced rubric scores.The gradient estimator is decomposed into stochastic components induced by those two rubric groups.Source: §B.1 · p. 13

B.2. Variance Analysis

The schedule delays advanced rubrics until their gradient signals become more reliable, while progressively introducing harder objectives. The supplied material also describes rubric filtering, reward assessment, implementation settings, and case-study evaluation.

B.2. Variance Analysis

  • The variance analysis quantifies gradient-estimator variability using the trace of its covariance matrix and applies the covariance property for linear combinations.The resulting decomposition motivates the proposed curriculum schedule.Source: §B.2 · p. 13, §B.3 · p. 14

B.3. Justification of Curriculum Schedule

  • Early training sets λt = 0 to suppress sparse, noisy advanced-rubric gradients, then gradually increases λt as advanced-rubric satisfaction and gradient reliability improve.The schedule is also intended to reduce destructive interference between perception-oriented and reasoning-oriented gradients before foundational skills are established.Source: §B.3 · p. 14

C. Configuration Details

  • Training uses verl with GRPO and a constant learning-rate schedule, while curriculum parameters K and τth are chosen in preliminary experiments to balance stability and learning speed.Table 5 summarizes the hyperparameter settings.Source: C. Configuration Details · p. 14, Table 5 · p. 14

D. Rubric Construction and Filtering

  • Gemini 3 Pro generates 20 rubric candidates covering visual faithfulness, logical coherence, constraint satisfaction, and mathematical accuracy.The candidates are evaluated for applicability and model accuracy before six are selected for the RL pipeline.Source: D. Rubric Construction and Filtering · p. 14, §D.2 · p. 21

D.2. Applicability and Accuracy of Rubric Candidates

  • Low-coverage candidates Cand 09 and Cand 03 apply to only 9.7% and 18.8% of samples, respectively, and could produce erroneous failure signals in over 80% of instances without filtering.The final pipeline selects six rubrics based on applicability, accuracy, and the need for automated reward computation.Source: §D.2 · p. 21, Table 6 · p. 21

F. Reward Signal Generation Prompts

G. Case Studies

  • The appendix presents two single-instance case studies using model rollouts and the Judge’s raw JSON rubric scores.These cases illustrate reward generation for both a reward-hacking example and an aligned reasoning trajectory.Source: G. Case Studies · p. 23

G.1. Case Study 1: Mitigation of Reward Hacking

In this case, the model reaches the correct answer BC = 20, but its reasoning contains geometric and logical errors. RuCL’s rubric rewards penalize those gaps despite the correct final answer.

G.1. Case Study 1: Mitigation of Reward Hacking

3. Judge Output (Rubric Reward Signal)

  • The rollout receives scores of 1 for visual presence, key-entity extraction, and question-intent alignment, but 0 for coherence, evidence grounding, and conclusion match.The zero scores target errors in the reasoning trajectory rather than the final answer’s requested format.Source: §3 · pp. 25–26
  • The Judge identifies a geometric error: using the distance from D to AB to calculate the area of △ABC instead calculates the area of △ABD.It also flags the model’s equation of that distance with the altitude of △ABC.Source: Case Study 1: Mitigation of Reward… · p. 25, §3 · p. 26
  • The model’s reasoning ends with BC dependent on unknown AC, so its conclusion of 20 is not derived from the preceding steps.The Judge assigns zero for reasoning-conclusion match.Source: Case Study 1: Mitigation of Reward… · p. 25, §3 · p. 26

G.2. Case Study 2: Alignment of Foundational and Advanced Reasoning

The second case presents an aligned reasoning trajectory in which visual perception supports logical deduction. The model’s foundational and advanced rubric scores are both high.

G.2. Case Study 2: Alignment of Foundational and Advanced Reasoning

  • The model correctly extracts coordinates and identifies the linear function, satisfying foundational rubrics R01–R04.Its derivation is mathematically sound, and the final answer follows from the steps, satisfying advanced rubrics R05–R06.Source: §G.2 · p. 26
  • The case’s high scores across stratified rubrics are presented as evidence that the model has integrated perception and reasoning rather than treating them as separate tasks.The example is described as a successfully aligned reasoning trajectory.Source: §G.2 · p. 26

Case Study 2: Alignment of Foundational and Advanced Reasoning

The case study works through a geometry problem and evaluates the response against rubrics for visual grounding, relevance, coherent reasoning, and answer consistency.

  • The solution finds the line equation y = x + 2, then uses point B’s y-coordinate of −1 to determine its coordinates.Source: Case Study 2: Alignment of… · p. 27
  • Rubric checks score the response for visible evidence, selecting the relevant entities, answering the requested question, coherent steps, and grounding in the image.Each of these checks receives a score of 1.Source: Case Study 2: Alignment of… · p. 27
  • The response gives the triangle’s area as 5, and the reasoning-conclusion rubric confirms that the boxed answer matches the calculation.Source: Case Study 2: Alignment of… · p. 27

H. Additional Evaluation on Out-of-Distribution Robustness

On EvadeBench, all evaluated models score below 46%; RuCL performs slightly better than the Qwen2.5-VL-7B-Instruct baseline and Vanilla GRPO, while overall performance remains limited.

I. Limitations

RuCL has computational overhead from proprietary teacher-LLM rubric generation and large-scale reward judges, and uses static stratification based on initial statistics.

  • RuCL incurs moderate computational overhead from proprietary teacher-LLM generation and large-scale judges used for reward calculation.Source: §I · p. 28
  • Static stratification assumes rubric difficulty remains constant during training, and the authors identify online difficulty updates as future work.Source: §I · p. 28
  • The paper also examines RuCL on specialized out-of-distribution scenarios, including evasive content detection, with detailed results in Appendix H.Source: §I · p. 28
Loading 2602.21628v2…