PaperlayerView exact revision on arXiv

Source-linked AI summary

GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning

Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Junhao Cheng, Ying Shan, Xihui Liu

arXiv:2506.16141v1cs.CVcs.AIcs.CLcs.LG

TL;DR

MLLM post-training lacks rigorous evaluation that balances perception, reasoning, and generalization, while outcome-supervised GRPO can produce incoherent reasoning. The paper introduces SEED-Bench-R1 and GRPO-CARE, which outperforms standard GRPO and improves consistency on the benchmark.

  • Problem

    MLLM post-training evaluation needs benchmarks that balance perception and reasoning while assessing generalization, and outcome-supervised GRPO can leave reasoning inconsistent with correct answers.

    Source: §1 · p. 2

  • Method

    The paper introduces SEED-Bench-R1's three-level video evaluation and GRPO-CARE, which combines answer-correctness rewards with reference-model-calibrated consistency bonuses.

    Source: §1 · pp. 2–3

  • Results

    GRPO-CARE outperforms standard GRPO across SEED-Bench-R1 difficulty levels, with 6.7% higher Level-3 performance and a 24.5% higher consistency rate.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present SEED-Bench-R1 and GRPO-CARE as tools for advancing robust MLLM post-training and improving reasoning coherence and performance without explicit process supervision.

    Source: §1 · p. 3, §5 · p. 9

    Outcome-supervised GRPO often reaches correct answers while its reasoning lacks coherence, undermining transparency.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Recent reinforcement learning approaches, such as outcome-supervised GRPO, have advanced Chain-of-Thought reasoning in large language models (LLMs), yet their adaptation to multimodal LLMs (MLLMs) is unexplored. To address the lack of rigorous evaluation for MLLM post-training methods, we introduce SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced perception and reasoning. It offers a large training set and evaluates generalization across three escalating challenges: in-distribution, cross-environment, and cross-environment-task scenarios. Using SEED-Bench-R1, we find that standard GRPO, while improving answer accuracy, often reduces logical coherence between reasoning steps and answers, with only a 57.9% consistency rate. This stems from reward signals focusing solely on final answers, encouraging shortcuts, and strict KL penalties limiting exploration.To address this, we propose GRPO-CARE, a consistency-aware RL framework optimizing both answer correctness and reasoning coherence without explicit supervision. GRPO-CARE introduces a two-tiered reward: (1) a base reward for answer correctness, and (2) an adaptive consistency bonus, computed by comparing the model's reasoning-to-answer likelihood (via a slowly-evolving reference model) against group peers.This dual mechanism amplifies rewards for reasoning paths that are both correct and logically consistent. Replacing KL penalties with this adaptive bonus, GRPO-CARE outperforms standard GRPO on SEED-Bench-R1, achieving a 6.7% performance gain on the hardest evaluation level and a 24.5% improvement in consistency. It also shows strong transferability, improving model performance across diverse video understanding benchmarks. Our work contributes a systematically designed benchmark and a generalizable post-training framework, advancing the development of more interpretable and robust MLLMs.

1 Introduction

SEED-Bench-R1 provides a three-level evaluation of multimodal video post-training, and GRPO-CARE jointly targets answer correctness and reasoning coherence. The reported results show stronger SEED-Bench-R1 performance and consistency than standard GRPO.

1 Introduction

  • Prior RL advances in LLM reasoning motivate MLLM post-training, but existing MLLM evaluations tend to isolate perception or reasoning or use broad data without structured generalization assessment.The cited examples include perception tasks [8], multimodal math reasoning [10], and general-domain training datasets.Source: §1 · p. 2
  • SEED-Bench-R1 evaluates video understanding requiring balanced perception and reasoning across in-distribution, cross-environment, and cross-environment-task levels.It uses realistic egocentric videos, large-scale training data, and verifiable ground-truth answers.Source: §1 · p. 2
  • Outcome-supervised GRPO improves MLLM perception and answer accuracy but yields only 57.9% consistency between reasoning chains and final answers.The paper attributes this to answer-only rewards encouraging shortcuts and strict KL penalties constraining exploration.Source: §1 · p. 2
  • GRPO-CARE jointly optimizes answer correctness and logical consistency without explicit process supervision, using a consistency bonus alongside the base correctness reward.A slowly updated reference model calibrates reasoning-to-answer likelihoods, and an adaptive group-relative bonus rewards accurate, coherent samples in place of a KL penalty.Source: §1 · pp. 2–3
  • 6.7% higher performance on Level-3 and a 24.5% higher consistency rate accompany GRPO-CARE's stronger results than standard GRPO across all difficulty levels.The paper also reports strong transferability to diverse general video understanding benchmarks.Source: §1 · p. 3

2 Related Work

Prior work applies outcome-based RL to reasoning but notes coherence problems and alternative supervision strategies. For MLLM post-training, video benchmarks have lacked large-scale training data, structured generalization tiers, and questions balancing perception and reasoning.

RL for LLMs/MLLMs

  • Outcome-only supervision can yield inconsistent reasoning despite correct answers, motivating process reward models, LLM judges, adaptive regularization, and approaches to MLLM Thought Collapse [20] [22] [23] [24] [25] [26].GRPO-CARE uses a slowly updated reference model to reward accurate, logically consistent responses without extra annotations or stronger correctors.Source: §2 · p. 3

Benchmarks for MLLM Post-training

  • Prior video RL efforts were limited by narrow tasks or scarce training data, while general-domain video benchmarks lacked rigorous generalization assessment [31] [32] [33] [34] [35] [37].SEED-Bench-R1 addresses these gaps with large-scale training data and validation sets across three generalization tiers.Source: §2 · p. 3
  • Image-focused MLLM post-training work spans perception and reasoning tasks [7] [8] [10] [29], whereas video understanding remains underexplored.The paper identifies a need for real-world multimodal questions that balance perception and reasoning alongside structured generalization evaluation.Source: §2 · p. 3

3 Pilot Study with SEED-Bench-R1

The pilot introduces SEED-Bench-R1 for evaluating multimodal video post-training and compares supervised fine-tuning with outcome-supervised GRPO. GRPO improves performance across in-distribution and out-of-distribution questions, while the analysis finds stronger perception but incoherent reasoning-answer traces.

3.1 SEED-Bench-R1

  • SEED-Bench-R1 combines realistic egocentric videos, inference-heavy next-action questions, automatically constructed training data, and three validation levels spanning new environments and tasks.Its ground-truth answers are the actual next actions in the uncropped video, and negative options come from the same video.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Experiment Setup

  • The pilot uses Qwen2.5-VL-Instruct-7B and 6k of 50k training samples to compare GRPO with SFT, which uses rejection-sampled CoT annotations.GRPO uses rule-based outcome rewards rather than explicit CoT annotations.Source: §3.2 · pp. 4–5

3.3 Result Analysis

  • GRPO significantly improves data efficiency and MLLM performance over SFT on in-distribution and OOD questions, using a simple outcome reward without specialized CoT annotations.The comparison covers L1 and OOD levels L2 and L3.Source: §3.3 · p. 5, Table 2 · p. 5
  • Attention-map analysis finds that GRPO-generated CoT attends more thoroughly to visual content, especially in OOD settings, while the authors hypothesize that SFT relies on more superficial, memorized CoT.The authors suggest broader visual exploration may underpin GRPO’s stronger generalization.Source: §3.3 · p. 5
  • GRPO often produces reasoning chains whose later steps diverge or contradict earlier steps, even when the final answer is correct, undermining transparency.The analysis also notes that reasoning-answer mismatches complicate interpretability.Source: §3.3 · p. 5
  • Limited reasoning can prevent GRPO from combining perception with world knowledge, as when it recognizes running water but misses that the next step is turning off the faucet.Source: §3.3 · p. 5

4 Consistency-Aware Reward-Enhanced GRPO for MLLMs (GRPO-CARE)

GRPO-CARE addresses outcome-supervised GRPO’s tendency to weaken reasoning coherence by adding a consistency-aware reward to answer correctness. It uses a slowly evolving reference model and group-relative likelihoods, and is evaluated on SEED-Bench-R1 and general video benchmarks.

4 Consistency-Aware Reward-Enhanced GRPO for MLLMs (GRPO-CARE)

  • GRPO-CARE combines an answer-correctness reward with an adaptive, group-relative consistency bonus estimated using a slowly evolving reference model.The bonus is assessed for high-accuracy samples and rewards reasoning traces whose likelihood exceeds the group-relative consistency baseline; the KL penalty is removed from model updates.Source: §4 · pp. 5–7

4.1 Evaluation on SEED-Bench-R1

  • Nearly 10%: GRPO-CARE improves over GRPO on the most challenging L3 evaluation, including Hobbies and Work, which diverge substantially from training data.The method also outperforms GRPO across all three difficulty levels.Source: §4.1 · p. 7
  • KL-based variants do not outperform NoKL, while DenseCons improves L1 and L2 but slightly underperforms on L3, and RefGen raises consistency but reduces overall performance.GRPO-CARE’s sparse consistency rewards improve across all levels; the authors attribute greater reliability to avoiding overfitting to imperfect likelihoods or sampling noise.Source: §4.1 · pp. 8–9

4.2 Generalization to General Video Understanding Benchmarks

  • CARE-7B (SB-R1) significantly improves over the base model across the six general video understanding benchmarks evaluated.The evaluation covers spatial reasoning, knowledge-intensive QA, and general video understanding; for MMVU it uses multiple-choice questions, and for VideoMME it uses the subtitle-free setting.Source: §4.2 · p. 9
  • With GRPO-CARE training on general-domain data using 16-frame inputs and testing with 32-frame inputs, the model is competitive with or superior to Video-R1-7B on most benchmarks.Video-R1-7B uses explicit temporal-order grounding constraints and supplementary supervised fine-tuning with additional data, whereas this model is trained solely with RL.Source: §4.2 · p. 9
  • The authors suggest that improved reasoning-answer consistency can encourage alignment between responses and visual grounding, with efficacy comparable to explicit visual-perception constraints.They present this implicit approach as a possible alternative for enhancing MLLM performance.Source: §4.2 · p. 9

5 Conclusion

The paper presents SEED-Bench-R1 for evaluating balanced perception and reasoning with three-level hierarchical generalization, and GRPO-CARE for jointly optimizing answer correctness and logical consistency. The conclusion reports better performance and interpretability from likelihood calibration with a slow-evolving reference model.

  • SEED-Bench-R1 evaluates MLLMs on tasks requiring balanced perception and reasoning through three-level hierarchical generalization.Source: §5 · p. 9
  • The paper’s analysis finds that standard GRPO improves answer accuracy but often reduces reasoning coherence.Source: §5 · p. 9
  • GRPO-CARE jointly optimizes correctness and logical consistency using likelihood calibration with a slow-evolving reference model, leading to better performance and interpretability.Source: §5 · p. 9
Loading 2506.16141v1…