PaperlayerView exact revision on arXiv

Source-linked AI summary

Watch Before You Answer: Learning from Visually Grounded Post-Training

Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang, Penghui Du, Yiming Jia, Dongfu Jiang, Xuan He, Shenhui Zhang, Ping Nie, Peter West, Kelsey R. Allen

arXiv:2604.05117v1cs.CVcs.AIcs.CL

TL;DR

Video understanding benchmarks and post-training datasets often contain questions answerable from text alone, limiting what their scores and training signals measure. The paper introduces VidGround, which retains visually grounded questions for RL-based post-training and achieves stronger video-understanding performance with less data. It concludes that curating evaluation and post-training data for visual grounding is important, while its approach relies on RL for post-training.

  • Problem

    Popular video benchmarks contain 40–60% text-only answerable questions, while video understanding still lags behind text-based reasoning.

  • Method

    VidGround curates post-training data by retaining only visually grounded questions and combines this dataset with RL-based post-training.

  • Results

    VidGround improves performance by up to 6.2 points over full-dataset post-training while using 69.1% of the original data, and outperforms five strong baselines.

  • Takeaways & Limitations

    Data quality is a major bottleneck for improving VLM video understanding, making visual grounding relevant to post-training and evaluation curation.

  • Takeaways & Limitations

    The post-training approach uses reinforcement learning, selected because prior evidence suggests it improves visual recognition and forgets less than supervised fine-tuning.

Abstract

from arXiv · show

It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.

1 Introduction

Video understanding benchmarks and post-training datasets contain substantial linguistic shortcuts, allowing questions to be answered without video. The paper introduces VidGround, which selects visually grounded questions for RL-based post-training and improves video understanding despite using less data.

  • Motivation: Benchmark gains increasingly come from answering questions without video, while video-based question performance can remain flat or worsen.Scaling model size or version improves text-only reasoning, but visual gain often remains flat or decreases.
  • Motivation: Post-training datasets also contain pervasive linguistic biases, potentially limiting improvements in video understanding.The paper identifies the same issue in commonly used post-training data, not only evaluation benchmarks.
  • Results: VidGround with RL-based post-training outperforms several more complex post-training techniques, indicating that data quality is a major bottleneck.The comparison includes methods using token-level importance weighting, long-video sequence scaling, and adaptive test-time frame selection.
  • Motivation: 40–60% of questions in popular benchmarks can be answered from text alone across multiple frontier models.This makes benchmark performance partly reflect language-based answering rather than genuine visual understanding.
  • Approach: VidGround post-trains VLMs using only questions that genuinely require visual understanding.The approach is presented as a simple data-curation strategy for removing linguistic shortcuts from post-training.

2 Related work

Prior work established linguistic shortcutting in VQA and documented persistent weaknesses in vision-centric reasoning. Video studies examined modality bias, while this paper extends the investigation to question-text-only shortcuts and uses visually grounded data with RL-based post-training.

  • Linguistic shortcutting: VQA research shows that models can rely more on textual cues than visual information when answering questions.This linguistic shortcutting problem has been documented since early VQA work.
  • Linguistic shortcutting: Modern VLMs remain weak on spatial reasoning, object counting, geometric perception, visual analogy, and fine-grained recognition.These weaknesses motivate continued study of visual capabilities beyond language-based answering.
  • Video understanding: Video research has found linguistic modality bias when subtitles are available, but has rarely examined question-text-only bias without subtitles.The paper addresses this comparatively underexplored setting for video understanding.
  • Mitigation: Earlier VQA mitigation methods reweighted training data or changed objectives to prioritize visual information.These approaches targeted the influence of text-only answerability during training.
  • Post-training: The paper applies RL-based post-training with visually grounded data to improve VLM video understanding.It positions this combination against existing SFT and RL post-training approaches in the video domain.

3 Analyzing linguistic biases in video understanding datasets

Video understanding benchmarks and post-training datasets contain substantial text-only answerable questions, allowing models to exploit linguistic shortcuts instead of genuine video reasoning. The analysis identifies four shortcut categories and motivates curating visually grounded data.

  • 40–60% of questions in popular video understanding benchmarks can be answered from text alone, revealing substantial linguistic bias.
  • Video-R1-260K contains 30.9% text-only answerable questions, meaning nearly one-third of its post-training data may not require visual understanding.
  • Text-only answerability can inflate benchmark scores and may cause post-training models to learn textual patterns and world knowledge instead of robust vision-language associations.
  • The four shortcut categories are textual cues, external knowledge, inferential or elimination strategies, and imagined video content.

4 VidGround: a simple approach to post-training

VidGround combines RL-based post-training with data curation that retains only questions requiring visual understanding. It selects 69.1% of the original data and improves video understanding over training on the full dataset.

  • VidGround combines reinforcement learning post-training with curation of questions that genuinely require visual understanding.
  • The RL objective uses GRPO with DAPO techniques, temporal-aware rewards, token-level policy gradients, and asymmetric clipping.
  • 181,710 visually grounded samples—69.1% of the original dataset—are retained because GPT-5-mini cannot answer them from text alone.
  • 85% of GPT-5-mini-selected visually grounded questions are also unanswerable by Qwen2.5-VL-7B in text-only mode, while circular evaluation yields 97% agreement across option permutations.
  • VidGround compares a 263,071-sample full dataset with a 181,710-sample visually grounded variant.

5 Experiments

Experiments compare VidGround with strong 7B post-training baselines across three video benchmarks, frame counts, data variants, and qualitative reasoning behavior. Visually grounded curation improves accuracy, preserves gains as frames increase, and encourages reference to video evidence.

  • Across post-training approaches: VidGround improves over Video-R1 by 4.8, 4.6, and 6.2 points on Full Avg at 16, 32, and 64 frames, respectively, using 69.1% of the training data.On VG Avg, the corresponding gains are 3.5, 4.5, and 5.0 points.
  • Across post-training approaches: VidGround maintains the highest Full Avg performance among all baselines at every frame setting.
  • Across datasets: On MMVU, VidGround outperforms Qwen2.5-VL-7B by 3.0 points at 64 frames and achieves the highest VideoMME performance across all frame settings.VidGround provides consistent gains across datasets and temporal resolutions.
  • Across frames: VidGround generally improves from 16 to 64 frames, whereas LongVILA-R1-7B and Video-R1 decrease by 3.1 and 2.7 points on MMVU from 32 to 64 frames.Video-RTS drops by 0.4 points on VideoMMMU over the same range.
  • Visually grounded training enables consistent frame-scaling: GRPO with VG data improves Full Avg from 56.8 to 58.5 to 59.5 across 16, 32, and 64 frames, while GRPO with Full data reaches 52.0, 53.9, and 53.3.On VG Avg, GRPO with VG improves from 45.2 to 47.6 to 47.9, while GRPO with Full stalls at 41.7 to 4.
  • Qualitative reasoning analysis: VidGround references specific visual elements such as lines, shapes, and colors, whereas Video-R1 reasons abstractly about art concepts and answers incorrectly.

6 Discussion

The discussion identifies pervasive text-only answerability in video benchmarks and post-training data as a serious measurement and training problem. VidGround addresses this by retaining visually grounded questions, yielding stronger performance and more consistent use of visual evidence.

  • Linguistic shortcuts: VideoMME and VideoMMMU contain 40–60% of questions answerable from question text alone, while over 30% of Video-R1-260K is text-only answerable.
  • VidGround: VidGround uses only visually grounded questions for post-training and outperforms five strong baselines on visually grounded evaluation splits and standard benchmark performance.
  • Frame scaling: Models trained on visually grounded data continue improving as more frames are provided, whereas baselines trained on unfiltered data plateau or degrade.
  • Additional analyses: The appendix adds evidence on multi-model agreement, cross-task generalization, benchmark-wide text-only answerability, implementation, and qualitative reasoning comparisons.
  • Multi-model agreement: GPT-5-mini, Qwen2.5-VL-7B, and Gemini-3.1-Pro show high agreement when identifying questions requiring visual input, with a 68.7% Jaccard index.All three models fail on 145,486 questions, forming a robust visually grounded core.

A.2 Alternative data curation strategies

Alternative curation strategies test progressively stricter multi-model filtering against VidGround’s single-model approach. The curation procedure uses text-only evaluation and retains questions judged to require visual input, while on-policy filtering targets shortcut reinforcement.

  • Consensus threshold: A question is retained as VG when fewer than two models answer it correctly in text-only mode, balancing curation quality with data retention.
  • Curation pipelines: All three curation pipelines evaluate approximately 263K Video-R1-260K samples in text-only mode before retaining visually grounded questions.GPT uses single-pass evaluation; Qwen and Gemini use permutation-based or repeated evaluation procedures.
  • Curation variants: VidGround retains 181,710 samples (69.1%), VidGround-M1 retains 160,837 (61.1%), and VidGround-M2 is a stricter 148K-sample variant.
  • On-policy curation: On-policy curation uses Qwen2.5-VL-7B as both curator and post-training base, targeting questions it cannot answer without visual input.Questions solvable by the training model from text alone are unlikely to improve its visual grounding.
  • Results: All three VidGround variants improve over the base model and Video-R1, while Video-R1 degrades the base model by 2.6 and 2.2 Avg. Full points at 16 and 32 frames.

B Cross-task generalization

Cross-task and broad benchmark analyses test whether visually grounded post-training preserves image QA while exposing linguistic shortcuts across video evaluations. VidGround improves or maintains non-video performance, whereas text-only models and VLMs both perform substantially above chance on video benchmarks.

  • Image QA: VidGround improves over Qwen2.5-VL-7B on MME (648.9 vs. 624.3), MMMU (58.7 vs. 56.7), and MMBench (84.5 vs. 84.2).
  • Benchmark-wide answerability: All VLMs and LLMs exceed chance performance by more than 20 points, with Gemini-3.1-Pro reaching +42.7 points above random.
  • Text-only performance: Gemini-3.1-Pro reaches 58.2%, 61.1%, and 63.4% on VideoMME, VideoMMMU, and MMVU without visual information.Gemini-2.5-Pro also reaches approximately 50% or higher on each benchmark.
  • Prompt sensitivity: Text-only performance varies substantially with prompting: Gemini-1.5-Pro reaches 41.4% on VideoMME in these experiments, versus a previously reported value below 15%.The authors attribute the discrepancy likely to prompt sensitivity.
  • Model scaling: Scaling within model families improves performance even in text-only settings, including GPT-5 over GPT-5-mini by +3.3 average points and Qwen2.5-VL-72B over 7B by +8.9 points.
  • LLM–VLM comparison: Text-only LLMs achieve competitive or superior performance to VLMs, with GPT-OSS-120B reaching 45.4% average versus GPT-4o’s 44.1%.

C.2 Analysis of linguistic biases in video benchmarks

Video benchmarks contain widespread linguistic shortcuts that let models answer without visual grounding. Text-only answerability is especially prevalent for video questions, causing benchmark accuracy to overstate genuine video understanding.

  • Four shortcut categories—textual cues, external knowledge, elimination, and hallucinated content—let models answer benchmark questions without observing the video.
  • 74.5% of video questions in Video-R1-260K achieve Pass@10 > 0 without video input, compared with 33.4% of image questions.Pass@10 is the fraction of questions receiving at least one correct answer across 10 independent text-only samples.
  • Video questions show 25.5% never answered correctly and 7.1% answered correctly in all 10 runs, suggesting greater susceptibility to linguistic shortcuts.
  • Overall, 48.4% of questions are never answered correctly and 3.7% are answered correctly in all 10 runs without visual input.
  • When 30–50% or more of performance comes from linguistic shortcuts, benchmarks systematically exaggerate progress in video comprehension.
  • Training on the full dataset performs worse than training on visually grounded data alone, indicating that text-only answerable questions undermine visual grounding.

D.1 Training configuration

VidGround post-training uses GRPO on a filtered set of visually grounded samples, with fixed frame sampling and standard evaluation prompting procedures. The filtering retains 69.1% of the original dataset.

  • Training uses 16 uniformly sampled video frames, 8× NVIDIA H100 GPUs, and 700 steps.
  • Evaluation uses 4× NVIDIA L40S GPUs with 16, 32, and 64 uniformly sampled frames per video.
  • VidGround filters Video-R1-260K by removing questions answerable without visual content, retaining 181,710 of 263,071 samples.GPT-5-mini evaluates question-answer pairs without visual access; correctly answered questions are classified as text-only answerable.
  • 69.1% retention removes 30.9% of linguistically biased examples from the post-training dataset.
  • Enhanced prompts prevent model refusals during text-only evaluation when models otherwise decline to answer without video access.
  • GRPO trains Qwen2.5-VL-7B-Instruct for one epoch on 181K visually grounded samples from Video-R1-260K.

E Additional qualitative analysis

Qualitative comparisons show VidGround consistently grounds reasoning in video content, whereas Video-R1 more often analyzes questions and options using textual cues and prior knowledge.

  • VidGround systematically grounds its reasoning in video content, while Video-R1 primarily relies on text-based analysis without referencing visual context.
  • VidGround responses commonly establish the video’s content before analyzing the question, including visual elements, the pelvis, eosinophils, and influence lines.
  • Grounding reasoning in observed video evidence makes responses less reliant on linguistic priors.
  • Video-R1 typically applies generic problem-solving templates, textual shortcuts, and general domain knowledge to analyze questions and answer options.

E.2 Beyond accuracy metrics: identical answers through different reasoning paths

Identical answers can arise from different reasoning paths: VidGround references video evidence, while Video-R1 may reach the same conclusion through textual analysis and prior knowledge. This exposes a limitation of accuracy-only evaluation.

  • Both models correctly answer the structural engineering question, but VidGround derives the answer from video content while Video-R1 analyzes the options linguistically.
  • Video-R1 analyzes all four answer choices through textual reasoning and never engages with the video content.
  • VidGround identifies the influence-line concept and unit-load demonstration before reaching the answer.
  • Accuracy-based evaluation can miss whether correct answers result from visual understanding or linguistic shortcuts, making reasoning-path analysis important.
  • Across the qualitative comparisons, VidGround improves performance and produces more visually grounded reasoning with less reliance on linguistic shortcuts.
  • Qualitative examples across art, psychology, medicine, public health, chemistry, and engineering illustrate the contrast between video-grounded and prior-knowledge reasoning.
Loading 2604.05117v1…