PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning

Mengzhao Jia, Zhihan Zhang, Ignacio Cases, Zheyuan Liu, Meng Jiang, Peng Qi

arXiv:2510.14738v2cs.CL

TL;DR

Answer-only RLVR can reward flawed intermediate reasoning whenever the final answer is correct, while reliable rubric construction for multimodal reasoning is difficult because ground-truth reasoning trajectories are usually unknown. AutoRubric derives problem-specific checkpoints from consistent steps in successful model trajectories and combines rubric-based rewards with outcome rewards. It reports improved accuracy and faithfulness across six benchmarks, with added training-time overhead from the judge.

  • Problem

    Answer-only RLVR can reward flawed intermediate reasoning when the final answer is correct, while multimodal rubric design is difficult because ground-truth reasoning trajectories are usually unknown.

    Source: §1 · p. 1

  • Method

    AutoRubric aggregates consistent reasoning steps from successful model-generated trajectories into problem-specific rubrics and combines judge-based rubric rewards with outcome rewards.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    AutoRubric improves accuracy and reasoning faithfulness across six multimodal benchmarks, with 53.24 average Strict Accuracy, 8.00 points above Qwen2.5-VL-7B (45.24 →53.24).

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    The results demonstrate that rubric-based process supervision can be combined with verifiable rewards to improve multimodal reasoning accuracy and faithfulness.

    Source: §5 · p. 8

    The LLM judge adds computational overhead, increasing training time per step by approximately 40% compared with vanilla RLVR.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have rapidly advanced from perception tasks to complex multi-step reasoning, yet reinforcement learning with verifiable rewards (RLVR) often leads to spurious reasoning since only the final-answer correctness is rewarded. To address this limitation, we propose AutoRubric, a framework that integrates RLVR with process-level supervision through automatically collected rubric-based generative rewards. Our key innovation lies in a scalable self-aggregation method that distills consistent reasoning checkpoints from successful trajectories, enabling problem-specific rubric construction without human annotation or stronger teacher models. By jointly leveraging rubric-based and outcome rewards, AutoRubric achieves state-of-the-art performance on six multimodal reasoning benchmarks and substantially improves reasoning faithfulness in dedicated evaluations.

1 Introduction

Answer-only RLVR can reward flawed reasoning whenever the final answer is correct, motivating process-level supervision for multimodal reasoning. AutoRubric builds problem-specific rubrics from successful model trajectories and combines their rewards with outcome rewards; it reports improved benchmark performance and faithfulness.

1 Introduction

  • Prior work on rubric-based rewards faces a multimodal reasoning challenge: the ground-truth reasoning trajectory is usually unknown, making reliable rubric design and integration non-trivial.Pre-trained process reward models can also produce unreliable estimates under distribution shifts.Source: §1 · p. 1
  • Answer-only RLVR gives the same reward to correct-answer traces even when one contains flawed or contradictory reasoning.The figure contrasts an inconsistent trace with a coherent step-by-step trace, though both reach the correct answer.Source: §1 · p. 1, Figure 1 · p. 2
  • AutoRubric combines rubric-based process supervision with outcome rewards to improve multimodal reasoning faithfulness and accuracy.Its problem-specific rubrics are automatically derived from consistent steps in the model’s own successful trajectories, without human annotation or stronger teacher models.Source: §1 · p. 2
  • Across six multimodal reasoning benchmarks, AutoRubric achieves state-of-the-art results and substantially more faithful reasoning in a dedicated evaluation.The authors also report that problem-specific rubrics are necessary relative to general judging criteria and that the framework stabilizes training.Source: §1 · p. 2

2 Related Work

Prior work improves multimodal reasoning through teacher-derived chain-of-thought or added supervision, while rubric-based rewards have largely focused on instruction following. AutoRubric applies automatically derived problem-specific rubrics alongside verifiable rewards to multimodal reasoning.

2 Related Work

  • Earlier multimodal RL work enriched supervision through annotated steps, visual perception rewards, or reflection-based rewards, but often emphasized isolated reasoning aspects and costly proprietary annotations.Other approaches distilled multimodal chain-of-thought data from teacher models before reinforcement learning.Source: §2 · pp. 2–3
  • AutoRubric derives problem-specific rubrics from multiple successful trajectories without proprietary supervision, aggregating consistent steps and filtering spurious ones.The authors describe this as improving reasoning fidelity and discouraging shortcut solutions.Source: §2 · p. 3
  • Existing rubric generation uses manual annotation, teacher responses, or contrastive response analysis, primarily for instruction-following tasks without verifiable rewards.Rubrics may be query-agnostic or tailored to a prompt and are commonly used by an LLM-as-a-judge.Source: §2 · p. 3
  • The work combines rubrics with verifiable rewards in multimodal reasoning to enhance the accuracy and faithfulness of intermediate reasoning chains.Source: §2 · p. 3

3 Method

AutoRubric augments RLVR with rubric-based trajectory rewards and uses GRPO as a representative policy-optimization framework. Its problem-specific rubrics are distilled from recurring steps across correct model-generated trajectories.

3.1 Preliminary

  • The task setup trains a policy to map visual input and textual query to the labeled answer while generating intermediate reasoning steps.Source: §3.1 · p. 3
  • GRPO optimizes the policy by comparing multiple responses sampled for the same query, without a separately trained value function.The paper uses GRPO as a representative example and says the approach can complement various policy-optimization frameworks.Source: §3.1 · p. 3, §3 · p. 3

3.2 Integrating LLM-As-a-Judge into RLVR

  • Rubric-based rewards score the fraction of problem-specific reasoning checkpoints satisfied by a trajectory, then combine with final-answer correctness as a weighted reward.The judge compares the trajectory with explicit checkpoints and need not reprocess the visual input; the weighting parameter controls the rubric reward’s impact.Source: §3.2 · p. 4

3.3 Aggregation-based Rubric Generation

  • For each problem, the method samples multiple trajectories, retains those with correct answers, and prompts an LLM to summarize their common steps as ordered checkpoints.These checkpoints form structured, problem-specific rubrics for judging trajectories during training.Source: §3.3 · p. 5

4 Experiments

Experiments assess answer accuracy and reasoning consistency across multimodal reasoning benchmarks, then examine reward components, rubric coverage, and training dynamics. AutoRubric leads the reported benchmark comparisons and shows stronger faithfulness-aware outcomes.

4.1 Experimental Setup

  • Experiments cover general and mathematical multimodal reasoning, using Standard Accuracy for answer correctness and Strict Accuracy to check reasoning-answer consistency.Rubrics are built by aggregating shared steps from successful trajectories; the procedure yields 26,144 rubric sets covering 67.3% of training data.Source: §4.1 · pp. 5–7

4.2 Experimental Results

  • 53.24 Strict Accuracy average (+8.00 points over Qwen2.5-VL-7B, 45.24 → 53.24) and 55.26 Standard Accuracy average lead the Table 2 comparisons.AutoRubric also has the highest Strict Accuracy across all benchmarks and maintains a −2.02 average inconsistency penalty.Source: Table 2 · p. 6, §4.2 · p. 7

4.3 Ablation Study

  • Increasing rubric coverage improves both accuracies, especially Strict Accuracy; even 20% coverage gains over answer-only supervision.The full setup uses 67.26% rubric coverage.Source: Figure 3 · p. 7, §4.3 · p. 8
  • AutoRubric achieves the best averaged Standard and Strict Accuracy; removing rubrics or judge rewards degrades performance, with larger gaps under Strict Accuracy.Judge rewards without problem-specific rubrics improve Strict Accuracy over answer-only rewards, while rubric-based judge supervision yields the strongest faithfulness-aware performance.Source: Table 3 · p. 7, §4.3 · pp. 7–8

4.4 Training Dynamics

  • As training proceeds, vanilla RLVR develops oscillating rewards and rising inconsistency and false-positive rates, while AutoRubric trains more smoothly and maintains lower rates.AutoRubric also generates longer responses than vanilla RLVR, consistent with more extended reasoning trajectories under rubric-based rewards.Source: Figure 4 · p. 8, §4.4 · p. 8

5 Conclusion

AutoRubric addresses reasoning–answer inconsistency in answer-only RLVR with rubric-based process supervision, automatically induced from consistent reasoning trajectories. Across six multimodal reasoning benchmarks, it improves accuracy and substantially enhances reasoning faithfulness.

5 Conclusion

  • Across six multimodal reasoning benchmarks, AutoRubric improves standard accuracy and substantially enhances reasoning faithfulness under stricter evaluation metrics.The framework induces problem-specific rubrics from consistent reasoning trajectories and combines rubric-based supervision with an LLM-as-a-judge, without human annotation or stronger teacher models.Source: §5 · p. 8
  • AutoRubric identifies reasoning–answer inconsistency as a failure mode of answer-only RLVR that leads to unstable training dynamics and unfaithful reasoning.It addresses this through rubric-based process supervision that provides structured, trajectory-level feedback.Source: §5 · p. 8
  • The conclusion presents rubric-based process supervision as an effective and scalable approach for training more reliable multimodal reasoning models.Source: §5 · p. 8

Limitations

AutoRubric’s LLM-as-a-judge adds training overhead, while the authors argue that the faithfulness gains justify the computation in settings where correct answers alone are insufficient.

Limitations

  • In the experiments, adding the judge increases training time per step by approximately 40% compared with vanilla RLVR.Source: Limitations · p. 9
  • The judge introduces additional computational overhead during training.Source: Limitations · p. 9
  • The authors argue that faithful reasoning matters in decision-support and safety-sensitive settings, and that the added computation is a justified trade-off.They report improved reasoning-trajectory consistency alongside accuracy, while RLVR without judge supervision may exacerbate inconsistency.Source: Limitations · p. 9
  • The authors identify reducing the overhead as an important direction and describe aggregating rubric evaluations into one LLM call as more efficient than evaluating each rubric independently.Parallel judge replicas can further increase reward-computation throughput when additional GPU resources are available.Source: Limitations · p. 9

A Evaluation Protocol

The evaluation protocol scores multiple-choice answers by extracting option letters and comparing them with ground truth, while a proxy judge evaluates semantic matches for open-ended answers. Baselines and the proposed model are re-evaluated under a unified protocol.

A Evaluation Protocol

  • For multiple-choice questions, regular expressions extract the predicted option letter, which is compared directly with the ground-truth label.Source: A Evaluation Protocol · p. 11
  • For open-ended answers, Qwen3-30B-A3B-Instruct-25074 judges whether predictions match ground truth in meaning, addressing exact-match failures for semantically equivalent phrasing.Source: A Evaluation Protocol · p. 11
  • Because reported zero-shot scores for the same model and benchmark can vary across studies, the authors re-evaluate open-source baselines and AutoRubric under one evaluation system while following original study prompts.Source: A Evaluation Protocol · p. 11

B Supplementary Analysis of Rubrics

The supplementary analysis details how AutoRubric constructs and evaluates problem-specific rubrics, contrasts their expressiveness with R1-VL key steps, and reports GRPO details and training on another dataset. The rubric comparison and cross-dataset results provide evidence about rubric structure and performance beyond the primary training setting.

B.1 Rubric Construction and Statistics

  • Rubrics are generated from shared reasoning steps among successful trajectories; samples with no correct trajectory receive no rubric, and synthesis requires more than three correct trajectories.The process uses eight trajectories per problem from a model trained for one epoch with standard RLVR, then an open-source text-only LLM synthesizes complete semantic criteria.Source: §B.1 · pp. 11–12

B.2 Comparison with Key Steps in R1-VL

  • Figure 5 contrasts concise R1-VL key steps on the left with AutoRubric’s corresponding rubric sets on the right for two geometry problems.The qualitative comparison concerns the representations’ expressiveness and supervisory effectiveness.Source: Figure 5 · p. 12, §B.2 · p. 12

B.1 Rubric Construction and Statistics

  • Generated rubric sets average 3.47 criteria, with criteria averaging 23.25 words and the training set containing over 2.1 million words of rubric text.Some criteria reach 198 words.Source: §B.1 · p. 12
  • A joint evaluation of 100 randomly sampled rubric sets by two graduate students gives an average score of 4.18 on a five-point scale, with an average absolute rating difference of 0.82.Source: §B.1 · p. 12

B.2 Comparison with Key Steps in R1-VL

  • R1-VL key steps can reduce evaluation to keyword matching, making logical coherence, completeness, and justification of intermediate conclusions difficult to assess.Source: §B.2 · pp. 12–13
  • AutoRubric’s complete, semantically rich criteria specify correct reasoning at different stages and evaluate meaningful conditions rather than keyword presence.The authors describe this formulation as providing clearer, more interpretable standards and more informative, reliable reward signals for complex multimodal trajectories.Source: §B.2 · p. 13

C Details of Group Relative Policy Optimization

  • GRPO assigns each response a scalar reward and computes advantages by group-wise normalization; its formulation includes PPO clipping and KL regularization toward a fixed reference policy.The clipping parameter is ϵ, and β controls the strength of KL regularization.Source: C Details of Group Relative Policy… · p. 13

D.1 Training on Other Datasets

  • AutoRubric gains an average of +1.36 points over the vanilla RLVR baseline across the reported benchmarks when trained on Geometry3K.The authors also report clear gains on MMMU and MMMU-Pro; Table 4 covers comparisons on mathematical and general reasoning benchmarks.Source: Table 4 · p. 12, §D.1 · p. 13

D.2 Data Efficiency

AutoRubric improves data efficiency over vanilla RLVR and the strict-accuracy judge detects reasoning–answer inconsistencies in model outputs, with reliability checked against manual review.

D.2 Data Efficiency

  • 53.38 to 55.28: AutoRubric improves as training data scales from 10% to 100%, while vanilla RLVR changes from 52.42 to 52.96.With 10% of the data, AutoRubric matches or surpasses full-data vanilla RLVR on several benchmarks, including WeMath and MMMU.Source: §D.2 · p. 13, Table 5 · p. 14

E.1 Reasoning Inconsistency Observation

  • The strict-accuracy judge detects cases where the reasoning-derived conclusion conflicts with the final answer in representative MathVista outputs.It produces a reasoning trace followed by a structured consistency judgment.Source: §E.1 · p. 14
  • Manual verification across five benchmarks found over 85% recall and over 90% precision for the judge's inconsistency detections.The review sampled 100 judged predictions, with 20 from each benchmark.Source: §E.1 · p. 14

E.2 Additional Analysis of Faithfulness

Additional faithfulness analyses compare inconsistency rates and reasoning quality over training. AutoRubric maintains lower inconsistency and higher quality than vanilla RLVR in the reported evaluation.

E.2 Additional Analysis of Faithfulness

  • The quality check assesses trajectory soundness across three dimensions, beyond the inconsistency check's focus on contradictions or unjustified answer shifts.A strong judge model, GPT-4o, assigns a holistic score between 0 and 1.Source: §E.2 · p. 14
  • AutoRubric keeps inconsistency around 10%, while vanilla RLVR remains above 20% and rises during training.The evaluation sampled 100 MathVision trajectories at each training step from 160 to 280.Source: §E.2 · p. 14
  • AutoRubric's reasoning-quality scores gradually improve and consistently exceed vanilla RLVR's, whose scores are flat or slightly declining.The quality assessment considers unfounded transitions, calculation errors, and logical connectivity.Source: §E.2 · p. 14

F Case Study

The case study illustrates how rubric-based rewards distinguish sound reasoning from flawed reasoning even when both trajectories reach the correct answer. The judge also handles different symbolic definitions between rubrics and trajectories.

F Case Study

  • Figure 6 shows MathVista examples where intermediate reasoning supports a different conclusion from the final answer, which the strict-accuracy judge identifies.The judge produces an explicit reasoning analysis followed by a structured consistency verdict.Source: Figure 6 · p. 15
  • Rubric-based evaluation distinguishes a trajectory with logical mistakes from a sound one, although both reach the same correct final answer.The example includes conflating BC with CD after defining AB = x and writing AC = x + CD.Source: F Case Study · p. 15
  • The judge aligns meaning across different symbol systems, such as rubric lengths a and b versus trajectory variable x.The authors attribute this to semantic understanding beyond surface-level keyword matching, as in R1-VL (Zhang et al., 2025).Source: F Case Study · p. 15

G Reproducibility Statement

The authors describe steps intended to support reproducibility and provide prompts for the rubric and evaluation procedures. The appendix includes prompts for rubric construction, reward judging, strict accuracy, and reasoning quality.

G Reproducibility Statement

  • The authors state that training code and evaluation scripts will be released, and that random seeds were fixed across training and evaluation runs.Model configurations and training and evaluation setups are described in the main paper and appendix.Source: G Reproducibility Statement · p. 15

H Prompts

  • The appendix presents prompts for rubric construction, rubric-based reward judging, strict-accuracy evaluation, and reasoning-quality evaluation.The prompts are shown in Figures 9–12, including a quality assessment that scores reasoning from 0 to 1 across specified dimensions.Source: H Prompts · pp. 15, 18, Figure 9 · p. 16, Figure 10 · p. 17
Loading 2510.14738v2…