PaperlayerView exact revision on arXiv

Source-linked AI summary

Can I Trust Your Answer? Visually Grounded Video Question Answering

Junbin Xiao, Angela Yao, Yicong Li, Tat Seng Chua

arXiv:2309.01327v2cs.CVcs.AIcs.MM

TL;DR

The paper investigates whether strong VideoQA models are grounded in relevant video evidence or exploit language and spurious correlations. It introduces NExT-GQA and a weakly supervised Gaussian-mask method, finding that grounding improves alongside QA while current models remain far behind humans.

  • Problem

    The central problem is determining whether strong VideoQA predictions are anchored in relevant visual content rather than language shortcuts or spurious vision-language correlations.

  • Method

    The paper constructs NExT-GQA with 10.5K temporal labels and learns differentiable Gaussian temporal masks using VQA and question-video supervision without temporal labels.

  • Results

    Current VLMs show strong QA performance but weak visual grounding: a 69%-accurate SoTA model grounds only 16% of correctly predicted answers.

  • Takeaways & Limitations

    The findings support continued work on interpretable and trustworthy VLMs because predictions often lack strong connections to pertinent visual information.

  • Takeaways & Limitations

    NExT-GQA focuses on weakly supervised temporal grounding and leaves spatio-temporal grounding for future exploration.

Abstract

from arXiv · show

We study visually grounded VideoQA in response to the emerging trends of utilizing pretraining techniques for video-language understanding. Specifically, by forcing vision-language models (VLMs) to answer questions and simultaneously provide visual evidence, we seek to ascertain the extent to which the predictions of such techniques are genuinely anchored in relevant video content, versus spurious correlations from language or irrelevant visual context. Towards this, we construct NExT-GQA -- an extension of NExT-QA with 10.5$K$ temporal grounding (or location) labels tied to the original QA pairs. With NExT-GQA, we scrutinize a series of state-of-the-art VLMs. Through post-hoc attention analysis, we find that these models are extremely weak in substantiating the answers despite their strong QA performance. This exposes the limitation of current VLMs in making reliable predictions. As a remedy, we further explore and propose a grounded-QA method via Gaussian mask optimization and cross-modal learning. Experiments with different backbones demonstrate that this grounding mechanism improves both grounding and QA. With these efforts, we aim to push towards trustworthy VLMs in VQA systems. Our dataset and code are available at https://github.com/doc-doc/NExT-GQA.

1. Introduction

The paper asks whether strong VideoQA systems genuinely use relevant video evidence or instead rely on language shortcuts and spurious correlations. It introduces weakly grounded VideoQA, diagnoses current VLMs, and proposes Gaussian-mask grounding to improve both evidence localization and QA.

  • Motivation: Current VLMs can answer VideoQA questions accurately while relying on language shortcuts or spurious vision-language correlations instead of relevant video content.The concern is whether predictions are grounded in the pertinent visual evidence.
  • Motivation: 62.5% of SoTA VLM predictions overlap with standalone language-model predictions, including 66% of correct and 79% of wrong predictions.Adding a coarse single-frame signal increases the overlap, although that frame often falls outside key moments.
  • Task: Grounded VideoQA requires models to answer questions and simultaneously identify relevant video moments, here under weak supervision.The paper presents this as the first weakly supervised formulation targeting visual explainability in VideoQA.
  • Dataset: NExT-GQA extends NExT-QA with 10.5K manually annotated start and end timestamps for validation and test QA pairs.The labels were checked as key to comprehending questions and determining correct answers.
  • Findings: Only 16% of correctly predicted answers from a 69%-accurate SoTA model are grounded in the video, compared with humans grounding 82% of their 93% correct answers.This gap motivates evaluating visual grounding alongside QA accuracy.
  • Method: The proposed method learns differentiable temporal Gaussian masks with lightweight transformer layers under VQA and question-video supervision, without temporal labels.Experiments with different QA backbones improve both grounding and QA, especially for questions requiring temporal understanding.
  • Contributions: The grounding mechanism improves existing VLMs in visual grounding and contributes to new state-of-the-art QA performance, reaching 73.1% on NExT-QA test.The contribution is framed as a simple mechanism for more trustworthy VLMs.

2. Related Work

Prior work advances VideoQA with transformers, pretraining, and powerful language models, but generally neglects visual-evidence grounding. This paper positions weakly grounded VideoQA and Gaussian mask learning as responses to that gap.

  • Grounded VQA: Existing grounded VQA benchmarks use full supervision, while no prior work addressed weakly grounded VideoQA according to the authors.Existing supervised benchmarks are also biased toward subtitles or limited object sets, making them unsuitable for general visual-evidence grounding.
  • VideoQA techniques: Strong VideoQA systems use shared, dual, or stacked transformers with image-text, video-text, or combined pretraining.These systems commonly rely on language models such as BERT, T5, GPT, or LLaMA.
  • Grounding techniques: Two-stage weakly supervised grounding methods rank temporal proposals by language-query similarity, but are described as inefficient and sub-optimal for multi-granular temporal modeling.The paper therefore turns to end-to-end Gaussian mask learning.
  • Language priors: Prior work addresses language priors through paired data, prediction-score regularization, or spatial-grounding-oriented methods.The paper’s stated distinction is defining weakly grounded VideoQA to encourage more interpretable and trustworthy techniques.

3. NExT-GQA Dataset

NExT-GQA extends NExT-QA with manually annotated temporal evidence for question answering in untrimmed videos, enabling models to provide and evaluate visual support. Its design emphasizes fine-grained, weakly supervised grounding rather than subtitle or object localization.

  • Dataset Construction: NExT-GQA augments NExT-QA with temporal labels for causal and temporal questions while excluding descriptive questions focused on global content.The dataset uses NExT-QA as its source because many other VideoQA datasets contain short videos already trimmed around relevant content.
  • Dataset Construction: 10,531 valid temporal segments were collected through multi-annotator labeling, checking, and refinement over approximately two months.Each QA pair was annotated by at least two people before an additional quality-control pass.
  • Dataset Analysis: Most temporal segments last under 15 seconds, average 7 seconds, occupy less than half of each video, and are distributed across early, middle, and late positions.The average segment-to-video length ratio is 0.2, and near 90% of QAs ground on a single segment.
  • Benchmark Comparison: NExT-GQA requires models to jointly ground visual evidence and answer questions, making it more challenging than text-only NExT-QA and trimmed-video benchmarks.It also permits visual answers and helps distinguish localization failures from answer-conversion failures.
  • Benchmark Comparison: Unlike declarative-sentence video grounding, NExT-GQA requires models to infer unstated visual evidence and refine moments through temporal or causal reasoning.A valid grounding need only contain sufficient evidence for the answer rather than exactly match all query-related content.
  • Benchmark Comparison: NExT-GQA uses weak temporal supervision and leaves fine-grained spatio-temporal grounding for future exploration.The authors distinguish this setting from fully supervised benchmarks that provide training labels or focus on subtitles and limited object sets.

4. Weakly-Supervised Grounding in VideoQA

The paper introduces model-agnostic weakly supervised grounding methods that make VideoQA models identify answer-relevant temporal segments alongside predictions. Gaussian temporal masks provide differentiable localization, while NG+ adds cross-modal supervision to improve grounding and QA.

  • Task and Approaches: Weakly grounded VideoQA requires models to answer questions and explicitly estimate a relevant video segment as visual evidence.The paper presents post-hoc, Naive Gaussian, and NG+ solutions.
  • Post-hoc Grounding: Post-hoc grounding extracts temporal segments from attention values after prediction, without influencing the model’s answers.The implementation uses attention pooling for dual architectures and averaged multi-head attention for stacked architectures.
  • Naive Gaussian: Naive Gaussian inserts a grounding module that estimates a key moment before the QA module predicts from localized video content.The moment is represented by differentiable Gaussian weights with learnable mean and standard deviation, enabling end-to-end learning.
  • Naive Gaussian: Naive Gaussian learns and outputs grounding information, but its improvement over post-hoc grounding is limited by weak QA supervision.Its dual-transformer version adds a Gaussian mask prediction head and Gaussian-weighted token learning and aggregation.
  • NG+: NG+ regularizes the VQA objective with cross-modal self-supervision by pulling the grounded video representation toward its question and away from negative questions.Negatives include questions from the same video and other videos, while some positive questions are rephrased for enrichment.
  • NG+: The combined grounding and grounded-QA objectives enforce video content that is relevant to both questions and answers.The grounding term identifies a question-relevant moment, while grounded QA refines that moment using answer supervision.

5. Experiments

Experiments show that strong VideoQA accuracy does not reliably indicate visual grounding, while Gaussian mask optimization improves both grounding and question answering across backbones.

  • 5.2.1 Q1: Are the answers visually grounded?: Models exceed 50% QA accuracy but achieve only 12–16% grounded QA accuracy, revealing a large gap between answering and grounding.FrozenBiLM reaches 69% QA accuracy versus 16% grounded QA accuracy; poor temporal grounding is the main source of the disparity.
  • 5.2.1 Q1: Are the answers visually grounded?: BlindQA reproduces much of standard VQA performance, indicating that language shortcuts contribute substantially to model predictions.BlindQA reaches 50.3% versus 59.4% for dual models and 56.7% versus 69.1% for stacked models.
  • 5.2.2 Q2: Does better QA imply better grounding?: Better QA does not necessarily imply better grounding, as FrozenBiLM has the strongest QA performance but worse temporal grounding than VGT.The authors associate this pattern with frozen language models relying heavily on common-sense knowledge rather than provided videos.
  • 5.2.2 Q2: Does better QA imply better grounding?: Grounding generally improves QA, although correct grounding does not guarantee a correct answer.PosQA consistently outperforms NegQA, and grounding improvements also improve QA performance, but grounding and answering remain imperfectly coordinated.
  • 5.2.3 Q3: Is Gaussian masking solution effective?: Gaussian grounding methods improve both grounding and QA, with NG+ generally outperforming NG and gaining most on questions requiring video and temporal grounding.NG+ is especially effective for dual-style architectures and focuses on the correct instance when multiple visual moments could match an answer.
  • 5.2.4 Method Comparison: Existing methods show stronger grounding than random selection, but the results motivate better coordination between QA and grounding.SeViLA has stronger standalone grounding than other methods, while FrozenGQA achieves higher grounded QA accuracy; the authors suggest localization pretraining as a future direction.

6. Conclusion

Current VLMs can answer visual questions well but often lack strong connections to pertinent visual information, and localizing temporal actions remains difficult. The proposed solution improves grounding and QA, yet remains below human performance.

  • Current VLMs often rely on language shortcuts and irrelevant visual context despite strong visual-question answering performance.
  • Localizing questions involving temporal actions and events remains a difficult open challenge.
  • The proposed solution improves both grounding and question answering but still leaves a large gap compared with human performance.
  • NExT-GQA is presented as a benchmark intended to support progress toward more trustworthy visually grounded VideoQA.

A.1. Dataset Construction

The dataset construction criteria are designed to reduce annotation ambiguity by covering answer-bearing temporal segments and preserving context needed to interpret questions.

  • Each annotation should cover the entire temporal segment containing the answer and enough context to interpret the question.
  • When mentioned visual content is not simultaneous or contiguous with the answer, annotation focuses on the answer.
  • If visual evidence appears multiple times, annotations include all relevant video moments.

A.2. Implementation Details

The implementation combines Gaussian temporal masking, attention-based post-hoc localization, and question-video correspondence learning within existing VLM architectures. Training and inference use architecture-specific settings and temporal-window fusion.

  • Post-hoc: Post-hoc temporal attention uses the frame with maximal attention as a pivot and searches nearby frames to form a time span.
  • Naive Gaussian: Naive Gaussian applies a frame-length Gaussian mask to self-attention weights before aggregating value vectors.
  • Naive Gaussian: In stacked-style transformers, Gaussian-weighted visual tokens are reinserted into the multimodal sequence for answer prediction.
  • Video-Question Correspondence Learning (NG+): Video-Question Correspondence Learning is trained in two stages, first with the grounding term and then with both objectives.
  • Others: During inference, overlapping temporal windows from Gaussian masking and temporal attention are fused by selecting their overlap area.

A.3.1 Can multiple Gaussian masks help?

Using multiple Gaussian masks improves grounding according to IoU but harms QA accuracy, while five masks achieve the best grounded-QA accuracy with negligible improvement over one mask.

  • Multiple Gaussian masks increase grounding performance according to IoU but reduce QA accuracy.
  • 15.8% is the best grounded-QA accuracy with five Gaussian masks, versus 15.5% with one mask.
  • The improvement from one to five Gaussian masks is negligible, so the method defaults to a single mask.

A.3.2 Does the generated questions help?

Generated positive questions improve QA results, but their grounding benefits are mixed, while the grounding method adds little parameter overhead at higher training cost. Video-LLaMA experiments further show that grounding improvements do not eliminate the gap between QA and grounded QA.

  • Generated questions: Generated positive questions improve Acc@QA but not Acc@GQA, with slightly higher IoU and lower IoP than models without them.The authors use the generated questions in final experiments because they improve QA, while noting that rephrasing more questions could increase the benefit.
  • Generated questions: Rephrasing currently covers only 10% of training questions and adds compute cost.
  • Efficiency: The grounding module introduces little additional parameter overhead, but NG+ takes longer to train than the backbone models.Temp[CLIP] trains and runs substantially faster than FrozenBiLM in the reported efficiency comparison.
  • Video-LLaMA generalization: NG and NG+ consistently improve over post-hoc grounding on Video-LLaMA, yet a large gap remains between QA and GQA accuracy.Video-LLaMA's pretrained Video Q-Former improves QA over image-text-pretrained CLIP but not video grounding.
  • Video-LLaMA generalization: Video-LLaMA achieves higher GQA than Temp[CLIP] through stronger QA performance rather than better grounding, and generally underperforms FrozenBiLM.The authors attribute this difference to generative answer decoding versus FrozenBiLM's candidate classification for multi-choice QA.
  • Result visualization: Visualization examples show correct answers can coexist with incorrect evidence, especially when answer concepts appear throughout the video.The examples also include failures to answer questions correctly or localize the relevant segments.
Loading 2309.01327v2…