PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking

Desen Meng, Rui Huang, Zhilin Dai, Xinhao Li, Yifan Xu, Jun Zhang, Zhenpeng Huang, Meng Zhang, Lingshu Zhang, Yi Liu, Limin Wang

arXiv:2506.01725v1cs.CV

TL;DR

Open-ended video captioning remains underexplored as a reinforcement-learning task, despite the cost of assembling high-quality caption-training data. VideoCap-R1 uses structured reasoning and separate rewards for reasoning and caption quality to train with GRPO. It improves over Qwen2-VL-7B with 1.5k samples across three benchmarks and outperforms SFT counterparts across benchmarks.

  • Problem

    Open-ended video captioning remains underexplored in multimodal reinforcement learning, while existing captioning approaches commonly depend on costly, high-quality description data for instruction tuning.

    Source: §1 · p. 2

  • Method

    VideoCap-R1 uses GRPO to train a two-step process that reasons about video subjects, attributes, and actions before captioning, with separate reasoning and caption-quality rewards.

    Source: §1 · p. 2

  • Results

    With 1.5k samples, VideoCap-R1 improves over Qwen2-VL-7B by +4.4 event F1 on DREAM-1K, +4.2 accuracy on VDC, and +3.1 action F1 and +6.9 object F1 on CAREBENCH, and outperforms SFT counterparts across benchmarks.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that GRPO-based post-training with structured thinking and separate reasoning and caption rewards improves video action and event descriptions even with limited training samples.

    Source: §5 · p. 9

    Naturalness scoring is subjective and may reflect judge-model biases, potentially encouraging reward hacking that inflates scores while deviating from the desired caption.

    Source: Event Score (Escore) · p. 5

Abstract

from arXiv · show

While recent advances in reinforcement learning have significantly enhanced reasoning capabilities in large language models (LLMs), these techniques remain underexplored in multi-modal LLMs for video captioning. This paper presents the first systematic investigation of GRPO-based RL post-training for video MLLMs, with the goal of enhancing video MLLMs' capability of describing actions in videos. Specifically, we develop the VideoCap-R1, which is prompted to first perform structured thinking that analyzes video subjects with their attributes and actions before generating complete captions, supported by two specialized reward mechanisms: a LLM-free think scorer evaluating the structured thinking quality and a LLM-assisted caption scorer assessing the output quality. The RL training framework effectively establishes the connection between structured reasoning and comprehensive description generation, enabling the model to produce captions with more accurate actions. Our experiments demonstrate that VideoCap-R1 achieves substantial improvements over the Qwen2VL-7B baseline using limited samples (1.5k) across multiple video caption benchmarks (DREAM1K: +4.4 event F1, VDC: +4.2 Acc, CAREBENCH: +3.1 action F1, +6.9 object F1) while consistently outperforming the SFT-trained counterparts, confirming GRPO's superiority in enhancing MLLMs' captioning capabilities.

1 Introduction

Although reinforcement learning has improved reasoning in LLMs and MLLMs, open-ended video captioning remains underexplored. VideoCap-R1 combines structured, two-step caption generation with GRPO and specialized rewards, outperforming SFT counterparts and improving benchmark scores with 1.5k training samples.

1 Introduction

  • Prior GRPO work extended to visual perception and video reasoning, but open-ended captioning remained underexplored despite reinforcement learning’s demonstrated value for LLM reasoning.Existing multimodal work focused largely on verifiable problems such as math, multiple-choice questions, and grounding.Source: §1 · pp. 1–2
  • SFT reasoning training produced the requested format without meaningful links to final descriptions, which the authors attribute to a train–inference discrepancy.Training uses ground-truth prefixes, while inference relies on the model’s own predictions.Source: §1 · p. 2
  • VideoCap-R1 first reasons about video subjects, attributes, and actions, then generates a complete caption, using GRPO with separate reasoning and caption-quality rewards.The reasoning scorer is LLM-free and the caption scorer is LLM-assisted.Source: §1 · p. 2
  • VideoCap-R1 improves over Qwen2-VL-7B with 1.5k samples: DREAM-1K +4.4 event F1, VDC +4.2 accuracy, and CAREBENCH +3.1 action F1 and +6.9 object F1.The paper presents this as the first successful application of GRPO to open-ended video captioning.Source: §1 · p. 3
  • GRPO outperforms SFT-trained counterparts on identical training datasets, whereas SFT learns superficial reasoning patterns, including with structured-thinking augmentation.The comparison holds regardless of whether structured-thinking augmentation is used.Source: §1 · p. 3

2 Related work

Video-captioning approaches commonly use costly, high-quality instruction-tuning data, while reinforcement-learning work on video has mostly targeted other tasks. This paper explores GRPO for captioning and reports better data efficiency than SFT under the same data budget.

Video caption models

  • Specialized video-captioning models rely predominantly on large, manually annotated instruction-tuning datasets that are costly and time-consuming to produce.The cited approaches include ShareGPT4Video, Shot2Story, Vript, LLaVA-video, and Tarsier2.Source: §2 · p. 3
  • Under the same data budget, the paper reports that its reinforcement-learning approach outperforms supervised fine-tuning, demonstrating superior data efficiency.This contrasts with approaches centered on large instruction-tuning datasets.Source: §2 · p. 3

Reinforcement learning for MLLMs

  • Prior GRPO studies covered video grounding, sentiment, tracking, and visual question answering, while open-ended captioning remained understudied and VideoChat-R1’s event-recall reward left captions unsatisfactory.The authors describe their work as designing and evaluating caption-specific rewards to adapt GRPO to the task.Source: §2 · p. 3

3 Methodology

VideoCap-R1 uses structured reasoning about video subjects, attributes, and actions before caption generation, with GRPO rewards for both reasoning and caption quality. Its method combines group-relative policy optimization with specialized scoring and a curated 1.5k-sample training set.

3.1.1 Group Relative Policy Optimization

  • GRPO compares multiple sampled responses to the same prompt, normalizes their rewards into relative advantages, and updates the policy to favor higher-advantage responses.Its objective uses clipping and KL regularization against excessive deviation from the reference model.Source: §3.1.1 · pp. 3–4

3.2 Caption Reward Modeling

  • Caption-specific reward functions extend format and accuracy rewards to guide policy optimization for video captioning.The final reward combines format_score, Tscore, and Escore.Source: §3.2 · p. 4, §3.3 · p. 6

3.2.1 Two-step Caption Generation Strategy

  • The two-step strategy has the model identify video subjects, attributes, and actions in structured reasoning, then synthesize them into coherent captions.The paper frames captioning as requiring coverage of significant elements across potentially complex video sequences.Source: §3.2.1 · p. 4

3.2.2 LLM-Free Think Scorer

  • The LLM-free think scorer extracts predicted entities and their attributes and actions, then compares them with ground truth using matching and F1-based metrics.It uses embedding similarity for action matching and one-to-one entity correspondences before computing action and attribute scores.Source: §3.2.2 · pp. 4–5

3.2.3 LLM-Assistant Caption Scorer

  • The caption scorer uses Qwen2.5-72B to explore direct rule-based scoring and LLM-assisted event extraction for evaluating captions.The authors motivate multiple scoring dimensions because valid captions can differ substantially from ground truth wording.Source: §3.2.3 · p. 5

Completeness-Naturalness Score (CNscore)

Event Score (Escore)

  • Because judge-model naturalness scores are subjective and susceptible to bias and reward hacking, the method avoids direct judge scoring and evaluates predicted captions by ground-truth event coverage.It assigns 0 when event_coverage < δ1, 0.5 when δ1 ≤ event_coverage < δ2, and 1 when event_coverage ≥ δ2; event coverage is the proportion of ground-truth events entailed by the predicted caption.Source: Event Score (Escore) · pp. 5–6

3.3 Enhancing Video Description Capabilities of Video MLLMs via GRPO

  • The training set was curated for dynamic motion, balanced action categories, and paired captions and reasoning annotations, yielding 1.5k samples.The final GRPO reward combines format_score, Tscore, and Escore.Source: §3.3 · p. 6

4 Experiments

Across three video-captioning benchmarks, VideoCap-R1 improves on the Qwen2-VL-7B baseline with 1.5k training samples, and GRPO outperforms SFT across benchmarks. Ablations, training curves, and qualitative comparisons examine its rewards, learning dynamics, and descriptions.

4.1 Experiment Setups

  • Evaluation uses DREAM1K, VDC, and CAREBENCH, which assess fine-grained events, detailed descriptions, and static objects plus dynamic actions, respectively.Experiments follow each benchmark's specified settings; implementation uses Qwen2-VL-7B-Instruct and samples up to 32 frames per video.Source: §4.1 · p. 6

4.2 Main Results and Analysis

  • +4.4 event F1 on DREAM-1K, +4.2 accuracy on VDC, +3.1 action F1 and +6.9 object F1 on CAREBENCH versus Qwen2-VL-7B; VideoCap-R1 also beats specialized models on VDC and CAREBENCH.It trails Tarsier-7B by 0.4% event F1 on DREAM-1K; Table 1 reports results across the three benchmarks.Source: §4.2 · p. 6, Table 1 · p. 7

4.2.1 Superiority of RL to SFT

  • GRPO's two-stage strategy consistently outperforms SFT across benchmarks, while adding structured thinking to SFT data degrades average performance compared with standard SFT.The authors attribute the SFT result to pattern imitation without a genuine reasoning-caption relationship.Source: Table 2 · p. 7, §4.2.1 · p. 7

4.3 Ablation Study

  • Combining think and caption scores yields optimal results; Escore performs best overall, while CNscore performs best on VDC but can reward judge-appealing descriptions lacking objective video relevance.The authors adopt Escore as the default because its event-based scoring evaluates factual video descriptions more objectively.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Training Dynamics

  • Think score and total reward increase stably, while response length first decreases, then rises and stabilizes with fluctuations that the authors associate with varying video information density.The authors interpret length variation as adaptation to video content.Source: §4.4 · p. 8, Figure 2 · p. 8

4.5 Qualitative Results

  • In qualitative comparisons, VideoCap-R1 identifies a stick that Qwen2-VL and its SFT version call an umbrella, and describes finer actions and details than the baselines.The authors report that its structured reasoning aligns with its final descriptions.Source: §4.5 · pp. 8–9, Figure 3 · p. 9

5 Conclusion and Future Work

The conclusion presents VideoCap-R1 as a GRPO post-training approach for video MLLMs, combining structured thinking with rewards for reasoning and final descriptions. It reports improvements with 1.5k samples and identifies scaling training data as future work.

5 Conclusion and Future Work

  • VideoCap-R1 combines two-stage structured thinking, Tscore for reasoning, and a caption score for final descriptions in GRPO post-training.The authors report improvements over baselines and SFT-trained counterparts across benchmarks with 1.5k training samples, and plan to scale training data.Source: §5 · p. 9

Appendix

The appendix lists supplementary material on implementation, data curation, qualitative comparisons, and prompts used for scoring and training.

Appendix

  • Appendix A provides additional implementation details, and Appendix B describes training-data curation.These are listed as supplementary sections.Source: Appendix · p. 13
  • Appendix C provides further qualitative comparisons between VideoCap-R1 and baseline models.The appendix outline identifies this as additional comparison material.Source: Appendix · p. 13
  • Appendices D and E provide the Qwen2.5-72B scoring prompt and the Qwen2-VL-7B training prompt, respectively.The appendix outline names both prompts.Source: Appendix · p. 13

A More Implementation Details

The implementation details specify GRPO rollout, sampling, KL-penalty, event-coverage, and GPU-allocation settings.

A More Implementation Details

  • GRPO uses 7 rollouts per prompt, temperature 1.0, KL penalty β = 0.001, and event-coverage thresholds δ1 = 0.28 and δ2 = 0.35.GRPO uses 7 GPUs for training and reserves 1 for rollouts; SFT uses all 8 GPUs for training.Source: A More Implementation Details · p. 13, Table 4 · p. 13

B Training Data Curation

Training data curation selects dynamic videos with diverse actions and pairs them with structured reasoning annotations and detailed descriptions.

B Training Data Curation

  • Videos are retained for high optical-flow dynamicity and durations of 10–30 seconds.The source is Tarsier2-Recap-585K [43], selected for accurate, detailed descriptions and comprehensive action annotations.Source: B Training Data Curation · p. 13
  • Structured annotations cover main subjects, keep attributes and actions consistent, and align described actions with video progression.Qwen2.5-72B [38] is prompted to produce the reasoning annotations.Source: Figure 4 · p. 13, B Training Data Curation · p. 14, Figure 5 · p. 14
  • Incremental action-based filtering yields 1.5K videos with high dynamicity, diverse action types, paired reasoning annotations, and detailed descriptions.Videos are added when they introduce action types not already represented in the dataset.Source: B Training Data Curation · p. 14

C Qualitative Results

Qualitative comparisons across DREAM-1K, VDC, and CAREBENCH highlight VideoCap-R1's fine-grained descriptions, with examples annotated for accuracy and hallucination.

C Qualitative Results

  • Across DREAM-1K, VDC, and CAREBENCH, examples show advantages in fine-grained action characterization, object attributes, and event coverage.The comparisons are presented as representative qualitative samples.Source: C Qualitative Results · p. 14, Figure 6 · p. 15, Figure 7 · p. 16
  • The comparison annotations mark hallucinated content in red, accurate descriptions in green, and correct details uniquely captured by VideoCap-R1 or Qwen2-VL-7B-SFT in highlights.These annotations compare the models against Qwen2-VL-7B.Source: Figure 6 · p. 15, Figure 7 · p. 16, Figure 8 · p. 17

D The scoring prompt for Qwen2.5-72B

The appendix identifies a prompt template for the completeness score.

D The scoring prompt for Qwen2.5-72B

  • Figure 11 presents the prompt template for the completeness score.The supplied passage does not describe the prompt's contents.Source: Figure 11 · p. 20

E The Training Prompt for Qwen2-VL-7B used in GRPO

The appendix presents prompt templates for naturalness scoring and training, with the training template used for both GRPO and SFT.

E The Training Prompt for Qwen2-VL-7B used in GRPO

Loading 2506.01725v1…