MVRCap: Towards Comprehensive Video Captioning via Hierarchical Multi-Dimensional Verifiable Rewards

1 Introduction

• Existing models often enumerate salient entities and primary actions but miss logical intent, viewpoint shifts, lighting, and holistic narrative structure.The authors characterize this as a limitation of approaches centered on local recognition rather than global scene reasoning.Source: §1 · p. 1
• The proposed taxonomy organizes video narratives into six dimensions, covering physical composition, dynamics, and observational context.The dimensions are Entity Attributes, Scene and Environment, Actions and Interactions, Temporal Information, Camera Language, and Style and Atmosphere.Source: §1 · p. 1
• MVRCap-SFT provides structured multi-dimensional annotations, but supervised fine-tuning alone remains insufficient for reliable visual grounding.The authors report that models can imitate synthetic-data complexity without robust grounding, with hallucinations particularly affecting temporal descriptions.Source: §1 · p. 2
• Its Fine-Grained Verifiable Reward combines hierarchical QA-based concept coverage, factuality checks, and a conciseness constraint.The QA reward probes multi-level question-answer pairs across six dimensions; factuality checks penalize inconsistencies across objects, actions, and temporal dynamics.Source: §1 · p. 2
• MVRCap achieves state-of-the-art performance on VDC, CAPability, and VC4VG.The authors say gains across benchmarks reflect fine-grained visual detail capture with semantic coherence and structural integrity.Source: §1 · pp. 2–3

2.1 Video Captioning

• Video-captioning methods have expanded annotation density and introduced reasoning, but often remain at the level of isolated entities and primary motions.The related-work discussion contrasts these approaches with MVRCap’s structured visual reasoning and multi-dimensional taxonomy.Source: §2.1 · p. 3

2.2 Reinforcement Learning with Verifiable Rewards

• For verifiable reinforcement learning in open-ended captioning, MVRCap uses hierarchical QA to probe narrative structure and causal logic rather than matching isolated semantic units.The passage notes that captioning lacks definitive ground truth and that event coverage and atomic-unit set matching remain coarse-grained.Source: §2.2 · p. 3

3.1 Data Construction Workflow

• The workflow defines a taxonomy, generates MVRCap-SFT captions, and constructs MVRCap-QA pairs for verifiable reward signals.Figure 2 depicts QA construction on the left and SFT generation on the right; only three QA dimensions are shown.Source: §3.1 · p. 3, Figure 2 · p. 4

3.1.1 Multi-Dimensional Annotation Paradigm.

• The six-dimensional annotation paradigm structures video understanding around entities, actions, time, camera language, environment, and style or atmosphere.It transforms captioning into structured visual reasoning, with dimensions addressing physical composition, dynamic evolution, and observational context.Source: §3.1.1 · p. 4

3.1.2 MVRCap-SFT Dataset Construction

• MVRCap-SFT uses MLLM-generated, multi-dimensional captions from sampled open-source videos, with cross-model checks, human review, and prompt refinement.The process uses Gemini 2.5 Pro to generate descriptions and GPT-5 to compare captions with source videos; it yields 4K video-caption pairs.Source: §3.1.2 · pp. 4–5

3.1.3 MVRCap-QA Dataset Construction

• The first MVRCap-QA level checks foundational grounding, including entity presence, spatial layout, behavior, event order, and camera-technique classification.Its stated aim is to prevent generative hallucinations through accuracy checks on explicit visual content.Source: §3.1.3 · p. 5
• The second QA level evaluates descriptive depth and causal reasoning, including interaction logic, environmental impact, temporal coherence, and aesthetic intent.The dataset contains 205K QA pairs across the two levels; GPT-5 verification and expert review support visual grounding.Source: §3.1.3 · p. 5

3.2 Training Strategy

• The training framework first establishes multi-dimensional instruction following with supervised fine-tuning, then uses GRPO to enforce visual-semantic alignment.Source: §3.2 · p. 6

3.2.1 Supervised Fine-Tuning

• Fine-tuning on MVRCap-SFT adapts the base MLLM to the schema and establishes a stable reference policy for reinforcement learning.The stage trains fluent, structured output from video-caption pairs.Source: §3.2.1 · p. 6

3.2.2 Group Relative Policy Optimization

• GRPO samples groups of captions, standardizes their composite rewards into advantages, and uses clipping and KL regularization to stabilize policy updates.Source: §3.2.2 · p. 6

3.2.3 Fine-Grained Verifiable Reward Formulation

• Coverage reward uses MVRCap-QA questions across six dimensions, scoring captions by whether a frozen Judge LLM can answer them from the caption alone.Dimension-level accuracy scores are averaged to balance semantic aspects.Source: §3.2.3 · pp. 6–7
• Factuality reward grades object, action, and temporal contradictions against the ground-truth caption, allowing valid paraphrases while penalizing factual errors.Source: §3.2.3 · p. 7
• A soft linear length penalty applies only above a target threshold, and the final reward weights coverage, factuality, and conciseness components.Source: §3.2.3 · p. 7

4.1 Implementation Details.

• Training uses Qwen2.5-VL-7B-Instruct as the baseline; RL uses 2.0 FPS, 8 rollouts per iteration, reward weight 1.0, and a Qwen3-30B-A3B-Instruct judge.SFT uses Adam at 1 × 10−5, and experiments run on 16 NVIDIA H100 GPUs.Source: §4.1 · p. 7

4.2 Main Results

• 74.6% Total Score on VC4VG for MVRCap-32B exceeds LLaVA-Video-Gen’s 57.0%; MVRCap-7B scores 65.4% versus its Qwen2.5-VL baseline’s 42.6%.The results also report consistently high scores across dynamic precision and Necessity-L1/L2.Source: Table 1 · p. 8, §4.2 · p. 8
• 55.18 All Avg. on CAPability exceeds the Qwen2.5-VL baseline by 5.61 points; Video Avg. is 37.29, with gains of +7.56 in Action and +8.61 in Camera Movement.MVRCap achieves state-of-the-art performance among 7B open-source models.Source: Table 2 · p. 8, §4.2 · p. 9
• 49.7% Average Accuracy and a Score of 2.6 on VDC set new state-of-the-art results; MVRCap leads in four of five categories, including Background at 53.6% and Object at 52.5%.The results also report a lead over OwlCap-7B in Detailed descriptions.Source: Table 3 · p. 8, §4.2 · p. 9

4.3 Ablation Study

• Removing Rcov lowers VDC Score from 2.60 to 2.42, while omitting Rfact lowers CAPability Video F1 from 37.29 to 34.12; all three rewards yield 55.18 All F1.The ablation evaluates coverage, factuality, and conciseness reward components.Source: Table 4 · p. 9, §4.3 · p. 9
• Applying RL directly to the baseline underperforms the SFT-plus-RL pipeline, which uses SFT to establish multi-dimensional semantic coverage before RL refines grounding and information density.The authors attribute the direct-RL gap to a cold start without the schema for comprehensive video description.Source: Table 5 · p. 9, §4.3 · p. 9

• MVRCap pairs MVRCap-SFT for semantic foundations with MVRCap-QA for fine-grained policy optimization, using rewards for hierarchical concept coverage and precise factuality.The framework is introduced for comprehensive video captioning.Source: §5 · p. 9