PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting

Daeun Lee, Shoubin Yu, Yue Zhang, Mohit Bansal

arXiv:2603.14659v2cs.CVcs.AI

TL;DR

Reliable spatio-temporal grounding remains difficult because existing approaches rely on costly data scaling or inference-time perception tools. VisionCoach uses input-adaptive visual prompting during RL, self-distillation, and grounding-aware rewards, achieving strong cross-benchmark performance while retaining a single inference pathway.

  • Problem

    Video reasoning models still struggle to reliably align intermediate reasoning with spatio-temporal evidence, while stronger grounding often requires costly annotations or inference-time tools.

  • Method

    VisionCoach selectively applies visual prompts to challenging inputs during RL, trains a Visual Prompt Selector and Spatio-Temporal Reasoner, and uses self-distillation to remove prompt requirements at inference.

  • Results

    VisionCoach achieves consistent gains across video reasoning, video understanding, and temporal grounding benchmarks, including +15.0 mAM and +25.1 mLGM over Qwen2.5-VL-7B on V-STAR.

  • Takeaways & Limitations

    Training-time visual guidance can improve grounded reasoning while preserving direct reasoning on raw videos through a single efficient inference pathway.

  • Takeaways & Limitations

    The framework relies on grounding annotations for reward construction, and its effectiveness depends on a predefined, relatively simple prompt candidate pool.

Abstract

from arXiv · show

Video reasoning requires models to locate and track question-relevant evidence across frames. While reinforcement learning (RL) with verifiable rewards improves accuracy, it still struggles to achieve reliable spatio-temporal grounding during the reasoning process. Moreover, improving grounding typically relies on scaled training data or inference-time perception tools, which increases annotation cost or computational cost. To address this challenge, we propose VisonCoach, an input-adaptive RL framework that improves spatio-temporal grounding through visual prompting as training-time guidance. During RL training, visual prompts are selectively applied to challenging inputs to amplify question-relevant evidence and suppress distractors. The model then internalizes these improvements through self-distillation, enabling grounded reasoning directly on raw videos without visual prompting at inference. VisonCoach consists of two components: (1) Visual Prompt Selector, which predicts appropriate prompt types conditioned on the video and question, and (2) Spatio-Temporal Reasoner, optimized with RL under visual prompt guidance and object-aware grounding rewards that enforce object identity consistency and multi-region bounding-box overlap. Extensive experiments demonstrate that VisonCoach achieves state-of-the-art performance under comparable settings, across diverse video reasoning, video understanding, and temporal grounding benchmarks (V-STAR, VideoMME, World-Sense, VideoMMMU, PerceptionTest, and Charades-STA), while maintaining a single efficient inference pathway without external tools. Our results show that visual prompting during training improves grounded video reasoning, while self-distillation enables the model to internalize this ability without requiring prompts at inference time.

1 Introduction

VisionCoach addresses unreliable spatio-temporal grounding by shifting guidance from costly data scaling and inference-time tools to input-adaptive visual prompting during RL. Self-distillation internalizes this guidance, while selector and reasoner components support efficient grounded inference and strong benchmark performance.

  • Motivation: Existing video reasoning models struggle with reliable spatio-temporal grounding, while tool-based methods remain dependent on external perception modules at inference.Grounded reasoning models can still produce inaccurate object references and hallucinated bounding boxes.
  • Motivation: Improving grounding commonly requires expensive dense annotations, scaled training data, or computationally costly inference-time perception tools.VisionCoach instead targets training-time guidance to improve intrinsic perception while preserving a lightweight inference pipeline.
  • Approach: VisionCoach selectively applies visual prompts to challenging inputs during RL, then uses self-distillation to internalize improved grounding without prompts at inference.The approach amplifies question-relevant evidence and suppresses distractors during training.
  • Approach: The framework combines a Visual Prompt Selector that predicts prompt types with a Spatio-Temporal Reasoner trained using prompt-guided RL and grounding-aware rewards.A proxy-reasoner-based pipeline constructs visual-prompt candidates for selector training.
  • Evaluation: VisionCoach evaluates spatio-temporal reasoning, general video understanding, and temporal grounding across V-STAR, VideoMME, WorldSense, VideoMMMU, PerceptionTest, and Charades-STA.The evaluation includes broad benchmark coverage and analyses of attention, ablations, selector generalization, and adaptive prompting.
  • Results: VisionCoach achieves state-of-the-art performance across diverse video reasoning, understanding, and temporal grounding benchmarks.The reported results support improved grounding and reasoning under comparable settings.

2 Related Work

Video reasoning research increasingly uses reinforcement learning with verifiable or rule-based rewards to strengthen multimodal reasoning, but reliable grounded reasoning remains challenging.

  • Video Reasoning: Large multimodal models support complex video question answering and reasoning across diverse application areas.Video understanding has advanced rapidly alongside these models.
  • Video Reasoning: Reliable grounded video reasoning remains difficult because models must track object states, events, and interactions over time.This challenge motivates methods that improve temporal and spatial evidence tracking.
  • Video Reasoning: A growing body of work applies reinforcement learning with verifiable or rule-based rewards to strengthen multimodal reasoning.These approaches are increasingly relevant to video reasoning.

3 Method

VisionCoach combines input-adaptive visual prompting with reinforcement learning and self-distillation to improve spatio-temporal grounding. Its VP-Selector guides hard inputs, while the ST-Reasoner learns from grounding-aware rewards and internalizes prompted reasoning for inference on raw videos.

  • Overall framework: The framework contains a VP-Selector for input-adaptive prompt prediction and an ST-Reasoner optimized under prompt-guided perception.The selector is trained from proxy-reasoner prompt candidates and then frozen during RL training.
  • Visual Prompting Guided RL: Hard samples are identified by average rollout reward, then receive a selected visual prompt applied to key frames and a textual hint appended to the question.The prompted input is used for additional rollouts whose rewards determine candidate trajectories for self-distillation.
  • Visual Prompting Guided RL: Self-distillation selects prompted rollouts whose overall rewards exceed the original average and reinforces the top N candidates with token-level negative log-likelihood.If no prompted rollout exceeds the original average reward, self-distillation is skipped for that sample.
  • Overall framework: VisionCoach uses visual prompting during RL training and self-distillation to improve grounding without prompts at inference.Challenging inputs receive visual guidance, whose improved reasoning trajectories are reinforced by the model.
  • Reward Design: The reward combines answer accuracy, format correctness, temporal grounding, and newly introduced object-aware spatial grounding.Rewards are group-normalized across rollouts for GSPO updates.
  • Reward Design: Object-aware spatial reward jointly evaluates soft object identity matching and all predicted box IoUs under temporal gating.This design addresses single-box predictions and object-agnostic hallucinations by retaining temporally aligned, identity-consistent predictions.

4 Experiments

VisionCoach is evaluated across spatio-temporal reasoning, general video understanding, and temporal and spatio-temporal grounding benchmarks. Results show strong performance, while ablations and analyses examine adaptive prompting, grounding, latency, and transferability.

  • Setup: VisionCoach uses object-aware spatial grounding rewards within a two-stage SFT-then-RL training pipeline, with VP-Selector trained separately and frozen during ST-Reasoner training.Rewards are group-normalized across rollouts to compute advantages for GSPO updates.
  • Main Results: +15.0 mAM and +25.1 mLGM over Qwen2.5-VL-7B on V-STAR, with VQA accuracy improving from 33.5 to 61.1.VisionCoach also outperforms proprietary and open-source baselines across V-STAR evaluation dimensions.
  • Main Results: VisionCoach consistently outperforms prior open-source VideoLLMs across general video understanding and temporal grounding benchmarks.Gains are especially reported for WorldSense Recognition and VideoMMMU Perception, while evaluation includes VideoMME, WorldSense, VideoMMMU, PerceptionTest, and Charades-STA.
  • Main Results: VisionCoach achieves substantial gains over Open-o3-video across all HCSTVG evaluation metrics and remains competitive with GPT-5.4 despite a smaller open-source backbone.The results are presented as evidence for fine-grained spatio-temporal reasoning requiring accurate localization of relevant objects and events.
  • Ablation Study: Input-adaptive prompting with self-distillation achieves the best performance across all ablation metrics, whereas fixed prompting degrades grounding through indiscriminate perturbations.VP-guided training also yields higher rewards and faster convergence; 58% of samples are identified as hard, with prompt-specific reward gains of +56% to +66%.
  • Ablation Study: VP-Selector consistently improves performance across backbone models on in-domain TVQA+ and out-of-domain PerceptionTest subsets.These results support VP-Selector as a plug-and-play module for grounded visual reasoning.
  • Additional Analysis: Attention-map analysis shows visual prompting concentrating temporal attention on the correct key frame and spatial attention on queried regions while suppressing distractors.The analysis aggregates question-to-visual-token attention across heads at frame and patch levels.

5 Conclusion

The conclusion presents VisionCoach as an instance-adaptive RL framework that uses training-time visual prompting to expose question-relevant evidence. Across multiple tasks, it reports consistent gains in reasoning accuracy and grounding quality while preserving a single efficient inference pathway.

  • Conclusion: VisionCoach applies visual prompting during RL through VP-Selector and ST-Reasoner to expose question-relevant evidence during learning.The framework is described as an instance-adaptive approach for grounded video reasoning.
  • Conclusion: Experiments across multiple challenging benchmarks show consistent gains in reasoning accuracy and grounding quality while preserving a single efficient inference pathway.The conclusion summarizes the cross-task outcome without external inference tools.

Contents of Supplementary Materials

The supplementary materials cover baselines, metrics, training and inference details, visual prompting, data and architecture, backbone and selector details, and additional ablations.

  • Supplementary Contents: The supplement includes baseline descriptions and evaluation metrics.
  • Supplementary Contents: Training datasets and additional training and inference details are provided.
  • Supplementary Contents: Supplementary sections describe visual prompting types, data collection, data statistics, and model architecture.
  • Supplementary Contents: Further details cover the Qwen3-VL backbone, VP-Selector training with open-source data, and VP-Selector inference with visual prompting.
  • Supplementary Contents: Additional ablations examine hyperparameters, rewards, visual prompting assignment, and visual prompting combinations.
  • Supplementary Contents: The experiments compare proprietary, open-source, text-based, tool-calling, and temporal-grounding VideoLLM baselines.For Ego-R1, dataset-specific RAG tools are disabled while the remaining tools are retained.

A.2 Evaluation Metrics

V-STAR evaluates what, when, and where reasoning with dedicated accuracy and localization metrics, then aggregates them across reasoning chains. The appendix also specifies metrics for general video understanding and temporal grounding.

  • V-STAR Metrics: V-STAR decomposes reasoning into what, when, and where dimensions across two chain orders.Chain1 follows what–when–where, while Chain2 follows what–where–when.
  • V-STAR Metrics: The three V-STAR dimensions use answer accuracy, mean temporal IoU, and mean visual IoU, respectively.
  • V-STAR Metrics: V-STAR aggregates these metrics using Arithmetic Mean and Geometric Mean, then reports mean AM and mean LGM across reasoning chains.The logarithmic geometric mean avoids collapse when any metric is zero, and epsilon prevents ln(0) near metric value one.
  • Other Benchmarks: General video understanding benchmarks report multi-choice question accuracy, while Charades-STA uses standard temporal-grounding metrics over predicted and ground-truth intervals.

B.2 More Training and Inference Details

VisionCoach constructs adaptive visual-prompt supervision by generating and scoring multiple prompt candidates for each video-question pair. Its prompts highlight objects, temporal cues, or attention-relevant regions, while proxy reasoning supplies structured grounding signals.

  • Training setup: Training uniformly samples 16 frames per video and uses one RL epoch with a learning rate of 1 × 10−6.The default setup uses 8 NVIDIA GPUs, per-device batch size 1, and four GSPO rollouts per input.
  • Visual prompt types: Key-object prompts use ground-truth boxes to highlight target objects through red circles or darkening.The red circle is centered on the object box and covers its region.
  • Visual prompt types: Frame numbering overlays explicit indices, while attention prompts use CLIP-ViT-L-14-336 to compute question-conditioned spatial relevance maps.The key-frame index appears as a red label at the bottom-right corner.
  • Prompt selection: The pipeline generates multiple visual-prompt candidates, produces grounded proxy reasoning, and derives pseudo-labels for selecting the most effective prompt.Reasoning is required to include object, bounding-box, and timestamp tags.
  • Prompt scoring: Prompt selection evaluates answer correctness, spatial grounding, and object consistency from parsed answers, boxes, and object names.Answer quality uses accuracy for multiple-choice questions and ROUGE for open-ended answers; spatial quality uses IoU.

C.3 Data Statistics

VP-Selector training combines annotated multiple-choice and open-ended video-question data, then predicts a prompt class for inference-time application. Experiments examine prompt selection, backbone transfer, and open-source supervision.

  • Data composition: The training data includes 5,000 TVQA+ samples, 1,382 VideoEspresso multiple-choice samples, and 3,000 open-ended VideoEspresso samples.The datasets provide annotated keyframes and key-object bounding boxes where specified.
  • Data composition: The selected-prompt distribution indicates that explicit visual prompting is generally beneficial, while the optimal strategy varies across samples.This supports adaptive rather than uniform prompt assignment.
  • Selector training: VP-Selector is trained as a generative prediction task by fine-tuning Qwen2.5-VL-Instruct (3B) with LoRA while freezing backbone parameters.The input includes the question, optional key object, and video key frames combined into a contact sheet.
  • Inference: At inference, VP-Selector predicts a prompt class, which is applied before each reasoner processes the original input.When key-object annotations are unavailable, Gemini-2.5-Flash extracts key objects for key-object prompting.
  • Backbone transfer: With Qwen3-VL-8B-Instruct, VisionCoach improves VQA from 63.0 to 65.9, mAM from 35.7 to 37.0, and mLGM from 48.3 to 50.9.The evaluation is conducted on V-STAR against Open-o3-video.

D.2 VP-Selector Training with Open-Source Data

Replacing proprietary proxy labels with Qwen3-VL-generated annotations still improves performance across TVQA+ and PerceptionTest. Ablations further show that moderate self-distillation and sufficiently challenging samples are important for training.

  • Open-source supervision: Qwen3-VL-generated proxy annotations still let VP-Selector consistently improve performance across TVQA+ and PerceptionTest.Combining open-source and proprietary supervision yields additional gains.
  • Inference behavior: Visual prompting substantially benefits the original Qwen2.5-VL model, with gains of +3.1% on TVQA+ and +8.6% on PerceptionTest.The comparison evaluates whether visual-prompt behavior is internalized through self-distillation.
  • Hard-sample selection: Selecting a balanced set of sufficiently challenging samples is important because low thresholds add weak supervision while high thresholds reduce training signals.The threshold is based on the reward distribution of Open-o3-video.
  • Self-distillation: At α = 0.1, self-distillation reaches 60.7 Acc, 31.3 mAM, and 43.1 mLGM across the reported metrics.Increasing the weight to α = 0.5 slightly degrades performance by reducing exploration during training.

E.2 Ablation of Reward

Reward ablations show that answer, temporal, spatial, and identity signals jointly improve grounded reasoning, with average multi-object overlap and identity matching providing richer supervision than maximum IoU alone. Prompt assignment also benefits from learned input adaptation rather than fixed or coarse heuristic choices.

  • Overall reward design: Adding answer reward to format, temporal, and spatial rewards increases accuracy from 56.9 to 58.5 while improving grounding performance.The ablation evaluates the contribution of reward components in combination.
  • Reward formulation: The max-IoU reward assigns credit using the best overlap with any ground-truth box, whereas the full reward averages IoU over temporally aligned predictions with valid identity matches.The intermediate design averages overlap across ground-truth objects at matched frames.
  • Spatial reward design: Average IoU improves grounding-related metrics over max IoU, and identity matching adds consistent gains across benchmarks.These results support aggregating multiple objects while enforcing object-level consistency.
  • Spatial reward design: Max-IoU reward produces 0.25 predicted objects and 0.38 boxes per reasoning, whereas Average IoU increases these values to 0.70 and 1.18.The increase indicates richer multi-object spatial reasoning.
  • Prompt assignment: Applying one fixed prompt type to all hard samples slightly degrades overall reasoning, while reward-based gating provides only limited gains because its assignment heuristic is coarse.The fixed and gating strategies are compared with learned VP-Selector assignment.
  • Prompt candidate pool: A diverse prompt pool improves grounding, with combined frame indices, circles, and darkening providing complementary signals.Single visual cues also improve grounding over the vanilla model, while the full combination achieves the strongest reported evaluation metrics.
  • Qualitative comparison: VisionCoach correctly identifies the nail color as “pink” and grounds the answer spatially across the woman’s hands, unlike Open-o3-video’s incorrect “white” answer.The example is presented as evidence of more reliable visual grounding.

G Limitation and Future Work

VisionCoach’s remaining limitations concern its dependence on grounding annotations for reward construction and on a predefined, relatively simple visual-prompt pool. Future work targets more expressive guidance and scalable training for larger models and datasets.

  • VisionCoach relies on grounding annotations, including object boxes and temporal locations, to construct spatial and temporal rewards.
  • Scaling VisionCoach to domains with scarce precise grounding supervision may require additional strategies for obtaining reliable reward signals.
  • The framework’s training effectiveness depends on a predefined and relatively simple prompt candidate pool, including circles, darkening, and frame indicators.
  • More expressive visual guidance or learned prompt generation may improve training signals for complex reasoning scenarios.
Loading 2603.14659v2…