Source-linked AI summary
Reviso: Not All Visual Evidence Deserves Equal Attention in Video Captioning
Tengfei Liu, Yang Shi, Yuran Wang, Yuqi Tang, Yizhuo Jia, Zhuoran Zhang, Qixun Wang, Shujie Li, Xuanyu Zhu, Xinyu Liu, Minghui Guo, Shixuan Liu, Haotian Wang, Yuanxing Zhang, Pengfei Wan
TL;DR
Video captioning models can produce detailed captions yet miss essential context and key events. Reviso plans and carries out targeted visual inspections, then integrates the evidence into captions. Across four benchmarks, it outperforms evaluated open-source baselines, and its captions better support downstream video question answering.
Problem
Detailed video captions may still overemphasize minor details and overlook essential context or key events.
Source: §1 · p. 1Method
Reviso uses hierarchical Context-Focus-Points planning and active, tool-assisted observation, learning the strategy from verified trajectories and tailored GRPO rewards.
Source: §1 · pp. 2–3Results
Across four video captioning benchmarks, Reviso outperforms evaluated open-source baselines, and its captions better support downstream video question answering.
Source: §1 · p. 3, §6 · p. 11Takeaways & Limitations
The findings support priority-guided active observation as an effective approach to video captioning.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video. Motivated by this observation, we propose Reviso, an agentic video captioning framework that learns what to observe and how to gather relevant visual evidence. From a low-frame-rate global preview, Reviso constructs a hierarchical Context--Focus--Points (CFP) observation plan that identifies the video context, prioritizes key aspects, and defines concrete observation targets and corresponding tools. Guided by this plan, Reviso performs goal-directed active perception by selectively revisiting temporal segments and combining tools for fine-grained inspection. It interprets the acquired evidence, updates its plan as needed, and integrates global and local information to generate the final caption. To learn this observation strategy, we develop a verification-guided pipeline that constructs high-quality training trajectories through factual verification and teacher-guided refinement. We further train Reviso using Group Relative Policy Optimization (GRPO), incorporating a key-fact reward for caption quality, a Plan Alignment Reward (PAR) for identifying important observation targets, and an Unproductive Tool-Call Penalty (UTCP) that adjusts observation-cost penalties based on caption quality. Experiments across four video captioning benchmarks show that Reviso outperforms the evaluated open-source baselines, while its generated captions better support downstream video question answering. These findings highlight the effectiveness and potential of priority-guided active observation for video captioning.
1 Introduction
Reviso frames video captioning as a priority-guided observation problem: it plans what to inspect, gathers targeted evidence, and learns this strategy through verified trajectories and tailored rewards. Experiments across four benchmarks show improved caption quality over open-source baselines.
1 Introduction
- Video-captioning models may provide detail while missing essential context or key events, motivating priorities tailored to each video's content.The observation challenge includes choosing important targets and selecting tools for fine-grained inspection.Source: §1 · pp. 1–3
- Reviso uses hierarchical Context-Focus-Points planning to guide goal-directed active perception and tool-assisted evidence gathering.The plan identifies context, prioritizes important aspects, and specifies observation targets and tools; Reviso updates it as evidence arrives.Source: §1 · pp. 2–3
- A verification-guided pipeline builds agentic training trajectories, and tailored GRPO rewards train observation planning, tool use, caption quality, and observation-cost regulation.The rewards include Key-Fact Reward, Plan Alignment Reward, and Unproductive Tool-Call Penalty.Source: §1 · pp. 2–3
- Reviso improves caption quality over strong open-source baselines across four video captioning benchmarks.Its generated captions also better support downstream video understanding.Source: §1 · p. 3
2 Related Work
Prior captioning work improves fine-grained descriptions and temporal representations, while video-agent research develops iterative observation. Reviso addresses open-ended captioning by learning video-specific observation plans and gathering evidence without a supplied question.
Video Captioning
- Video-captioning methods improve fine-grained information and temporal dynamics, but primarily generate captions from available visual representations rather than learning video-specific targets and acquiring further evidence.Examples include structured descriptions, trajectory records, reinforcement learning, and persistent identity galleries.Source: §2 · p. 3
Agentic Video Understanding
- Many video agents guide observation with a question; Reviso instead learns observation planning and tool-assisted evidence gathering for open-ended captioning.Omni-Detective and OmniFysics-Agent gather evidence for captions, but their captioners learn primarily from resulting descriptions rather than observation trajectories.Source: §2 · p. 3
3 Method
Reviso plans video-specific observations, uses the plan to gather and integrate local evidence, and learns this strategy from verified trajectories and reward-guided training. Its data pipeline retains trajectories after factual verification and teacher-guided refinement.
3.1 Priority-Guided Active Observation
- A low-frame-rate preview informs a Context–Focus–Points plan that identifies the setting, prioritizes relevant aspects, and specifies targets, temporal segments, granularity, and tools.Reviso iteratively inspects evidence, updates unresolved targets, and combines local findings with global context for caption generation.Source: §3.1 · pp. 3–4
3.2.1 Supervised Fine-Tuning
- Verified teacher trajectories initialize supervised fine-tuning, with token-level supervision for plans, tool calls, interpretations, plan updates, and captions.Environment responses are masked from the loss.Source: §3.2.1 · p. 4
3.2.2 GRPO for Priority-Guided Observation
- GRPO combines caption-quality, plan-alignment, and observation-cost rewards; UTCP penalizes costly observations more when captions are lower quality.The cost penalty activates only if a trajectory in the sampled group reaches a fixed KFR threshold; tool calls after the shared preview are normalized by a fixed call budget.Source: §3.2.2 · pp. 4–6
- Key-Fact Reward (KFR) scores F1 between deduplicated facts in generated and reference captions, using one-to-one semantic matching for precision and recall.It encourages key-information coverage while penalizing unmatched claims.Source: §3.2.2 · p. 5
- Plan Alignment Reward (PAR) measures whether the initial plan’s targets match key-information targets that remain unresolved after the global preview.One-to-one semantic matching computes precision and recall, rewarding relevant targets while limiting unmatched ones.Source: §3.2.2 · p. 5
3.3 Data Construction
- Training trajectories are generated from human-prepared ground-truth captions, then checked for factual consistency, reference-fact coverage, and evidence support.Failed attempts receive teacher-guided recovery and refinement; verified trajectories from both the original model and teacher are retained for supervised fine-tuning.Source: §3.3 · pp. 6–7
4.1 Experimental Setup
The experiments compare Reviso with open-source and proprietary MLLMs across captioning and downstream-understanding benchmarks. Models are evaluated under a shared maximum frame budget.
Data and training
- Training uses 10,000 supervised trajectories and 20,000 reinforcement-learning samples from four sources, with no overlapping videos between the datasets.The model receives two SFT epochs followed by 200 GRPO steps; its visual encoder remains frozen.Source: §4.1 · p. 7
Baselines
- Reviso is compared with 15 MLLMs, including open-source baselines and proprietary reference models.Qwen3.5-9B serves as the backbone baseline.Source: §4.1 · p. 7
Evaluation Benchmarks
- Evaluation covers four captioning benchmarks and two downstream video-understanding benchmarks, using metrics tailored to factual accuracy, coverage, detail, and question answering.DREAM-1K and MCTS-VCB use precision, recall, and F1; CaReBench uses action and object F1; VDC uses TP score and TP accuracy; LVBench and MMVU assess understanding accuracy.Source: §4.1 · p. 7
Implementation details
- All methods have a maximum of 128 input frames per video; Reviso divides that budget between its global preview and temporal revisits, while baselines sample uniformly.This keeps the maximum frame budget consistent across evaluated methods.Source: §4.1 · p. 7
4.2 Main Results
Reviso achieves the best overall results among evaluated open-source models across four video captioning benchmarks, consistently outperforming its Qwen3.5-9B backbone.
4.2 Main Results
- Reviso achieves the best overall results among evaluated open-source models across four video captioning benchmarks, consistently outperforming its Qwen3.5-9B backbone.It records the highest precision on DREAM-1K, highest action and object F1 on CaReBench, and best VDC Detailed metrics.Source: §4.2 · p. 7, Table 1 · p. 8
4.3 Ablation Study
Ablations show that active observation and the full GRPO reward objective improve captioning metrics, with PAR removal causing the largest reduction among reward ablations.
4.3 Ablation Study
- The full GRPO objective improves all three metrics over SFT, while removing PAR causes the largest reward-ablation reduction, including a 4.2-point drop in object F1.Removing KFR or UTCP also lowers performance on both benchmarks; the variant without active captioning remains close to the base model.Source: §4.3 · p. 8, Table 2 · p. 8
4.4 Analysis and Findings
Reviso’s captions support stronger caption-based question answering, and targeted observation benefits longer clips. Across comparisons, uniform frame-rate increases and access to general-purpose agentic tools alone do not ensure better caption quality.
4.4.1 Caption-To-QA Evaluation
- Reviso achieves the highest accuracy across all four caption-based QA metrics, improving over its Qwen3.5-9B backbone by 4.9 points on LVBench and 2.1 points on MMVU Overall.Gemini answers benchmark questions from captions without access to the videos, and captions are generated independently of the questions.Source: §4.4.1 · p. 9, Table 3 · p. 9
4.4.2 Benefits of Tool-Assisted Observation
- Tool-assisted observation yields larger object F1 improvements for longer videos, reaching approximately 10 percentage points for 24–27-second clips.For shorter videos, additional gains are relatively limited, potentially because the frame budget already provides dense temporal coverage.Source: §4.4.2 · p. 9
4.4.3 Does Higher FPS Improve Caption Quality
- Reviso outperforms evaluated direct-captioning configurations despite increased sampling rates, while denser sampling produces inconsistent gains and can lower some models’ performance at 10 FPS.Molmo2-8B and Qwen3.8-27B continue to benefit from denser sampling; Qwen3.5-9B and GPT-6 Astra decline at 10 FPS.Source: §4.4.3 · pp. 9–10
4.4 Analysis and Findings
- In the qualitative comparison, Reviso uses OCR to track game state and produces a more accurate caption than Qwen3.5-9B.Figure colors mark incorrect facts in red, key facts in green, and fine-grained details in blue.Source: Figure 6 · p. 10
4.4.4 Comparison of Captioning Strategies
- General-purpose agentic workflows improve caption quality for several models, but Reviso outperforms most model-harness combinations; only GPT-6 Astra with Codex scores higher.The comparison covers direct captioning and Codex and Claude Code workflows for five VLMs.Source: §4.4.4 · p. 10
5 Case Study
In a basketball clip, Reviso verifies the shot outcome by revisiting the sequence and using cropping and OCR on the scoreboard, avoiding an unsupported conclusion that the game has ended.
5 Case Study
- Reviso verifies the three-pointer’s outcome with scoreboard OCR, correctly describing Denver’s 74–72 lead with 1:45 remaining instead of concluding the game has ended.Qwen3.5-9B mistakes the player’s gesture for a victory celebration; Reviso revisits the shot sequence and extracts the score and remaining time.Source: §5 · p. 10
6 Conclusion
Reviso prioritizes important visual information through adaptive, tool-assisted observation. Across four video captioning benchmarks, it outperforms the evaluated open-source baselines, and its captions better support downstream video question answering.
6 Conclusion
- Reviso uses a hierarchical Context–Focus–Points plan to guide multi-round, tool-assisted observation and adapts the plan as it acquires evidence.Source: §6 · p. 10
- Across four video captioning benchmarks, Reviso outperforms the evaluated open-source baselines, and its captions better support downstream video question answering.Source: §6 · p. 11
- Analyses highlight the benefits of observation planning and targeted evidence gathering for improving caption quality.Source: §6 · p. 11