PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-Backed Video Question Answering

Shijie Wang, Honglu Zhou, Ziyang Wang, Ran Xu, Caiming Xiong, Silvio Savarese, Chen Sun, Juan Carlos Niebles

arXiv:2607.11862v1cs.CVcs.AI

TL;DR

Video LLMs often answer video questions without verifiable visual grounding. E-VQA couples semantic answers with temporal segments and dense tracked masklets, and ST-Evidence-Instruct fine-tuning improves both QA accuracy and dense grounding.

  • Problem

    Video LLMs provide textual answers without verifiable spatio-temporal evidence, while existing rationales and sparse boxes inadequately ground complex video understanding.

  • Method

    E-VQA jointly predicts semantic answers, temporal evidence, and dense tracked spatio-temporal masks, supported by human-verified ST-Evidence and scalable pipelines for ST-Evidence-Instruct.

  • Results

    High QA accuracy does not correlate with grounding proficiency, while fine-tuning on ST-Evidence-Instruct improves QA accuracy and dense grounding.

  • Takeaways & Limitations

    E-VQA provides a foundation for explainable Video LLMs that align answers with visual evidence across time and space.

  • Takeaways & Limitations

    Evaluation may be constrained by reliance on a proxy segmenter, a concern examined through ablations.

Abstract

from arXiv · show

Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.

1 Introduction

E-VQA requires Video LLMs to jointly produce semantic answers and verifiable temporal and dense spatial evidence, addressing the black-box limitations of textual QA. The authors introduce human-verified ST-Evidence benchmarks and a 160k-triplet instruction dataset to evaluate and improve evidence-grounded video understanding.

  • Task formulation: E-VQA jointly outputs a semantic answer, supporting temporal segments, and dense tracked spatio-temporal segmentation masks.This formulation explicitly couples high-level reasoning with low-level grounding.
  • Benchmark: ST-Evidence is the first E-VQA benchmark, built through sampling, human dense-evidence annotation, and independent verification.The pipeline allows low-quality samples to be rejected and annotations to be flagged for re-annotation.
  • Benchmark variants: ST-Evidence-Gen requires complete triplet generation, whereas ST-Evidence-MCQ requires answer, temporal-segment, and candidate-mask selection.The generative variant targets dense pixel-level grounding, while the multiple-choice variant emphasizes discriminative and sparse grounding.
  • Evaluation: High QA accuracy does not correlate with grounding proficiency, with most open-source models performing near random-guess levels in spatial evidence selection.The evaluation includes Qwen3-VL, OpenAI-o3, and Gemini-2.5-Pro.
  • Instruction tuning: 160k triplets comprise ST-Evidence-Instruct, bridging semantic QA with aligned temporal and spatial evidence through fully automatic generation pipelines.The dataset contains QA pairs, temporal evidence, and spatial evidence, including masklet-annotated videos repurposed for reasoning tasks.

2 Related Work

Prior Video LLMs improve multimodal comprehension and reasoning but largely provide textual outputs, while grounding efforts remain limited to temporal segments, sparse boxes, or separate spatial and temporal capabilities. Existing video QA datasets likewise lack comprehensive dense evidence annotations combining semantic reasoning with pixel-level grounding.

  • Video Large Language Models: Video LLMs demonstrate strong video comprehension, but their responses remain purely textual without visual justification.Reasoning Video LLMs use reinforcement learning and test-time scaling to generate multi-step reasoning chains.
  • Video Large Language Models: Grounded reasoning models target temporal segments or sparse bounding boxes, but do not provide comprehensive dense spatio-temporal evidence.VITED and Time-R1 focus on temporal segments, whereas Open-O3 Video and Seg-R1 incorporate sparse boxes or segmentation-oriented strategies.
  • Spatio-Temporal Grounded Video LLMs: Temporally grounded Video LLMs predict segment boundaries without spatial grounding, while spatial models add boxes or dense masks through segmentation decoders.Representative temporal models include VTimeLLM, Momentor, and VTG-LLM; spatially grounded models include VideoMolmo, NumPro, VGR, Sa2VA, UniPixel, VideoLISA, VideoGLaMM, and GLUS.
  • Video QA and Grounding Datasets: Existing video QA datasets separately emphasize causal reasoning, grounded video understanding, or referential comprehension, leaving dense evidence-backed reasoning unsupported.NeXT-QA and STAR lack grounding annotations, while NeXT-GQA, V-STaR, VideoEspresso, CG-Bench, SAMA, VideoRefer, and Strefer represent grounded or referential datasets.

3 Evidence-Backed Video QA

E-VQA extends video question answering by requiring answers alongside temporal segments and spatio-temporal masklets that ground the reasoning. ST-Evidence evaluates this unified triplet through human-verified generative and multiple-choice grounding benchmarks.

  • E-VQA Task Definition: E-VQA requires models to output a unified triplet: answer A, non-overlapping temporal evidence Et, and spatial evidence Es as spatio-temporal masklets.Temporal evidence contains critical time spans, while spatial evidence identifies relevant objects or regions.
  • E-VQA Task Definition: Grounding reasoning in specific spatio-temporal regions mitigates static language bias and enhances video-understanding explainability.
  • ST-Evidence Benchmark: ST-Evidence uses a semi-automatic Sample–Annotate–Verify pipeline to construct high-quality, human-verified evidence from existing semantic Video QA benchmarks.Source benchmarks include NeXT-QA, Perception Test, STAR, CLEVRER, and Ego4D.
  • ST-Evidence Benchmark: ViCaS supplements dense-mask coverage with human-annotated masks and captions, while VLM-generated QA pairs are filtered and evidence is annotated against existing object masks.The VLM used is Qwen3-VL-235B-A22B.
  • Benchmark Variants: ST-Evidence-Gen evaluates generated triplets using answer accuracy, temporal tIoU and IoP, and spatial J&F, whereas ST-Evidence-MCQ evaluates discriminative grounding through structured selection.The generative format targets open-ended synthesis; the multiple-choice format supports Video LLMs not explicitly trained for generative localization.

4 Instruction Tuning Data Construction

The authors construct E-VQA instruction-tuning data by combining semantic and grounded video sources through scalable, bidirectional automated pipelines. The resulting ST-Evidence-Instruct dataset contains aligned question-answer and spatio-temporal evidence tuples for explainable video reasoning.

  • 4 Instruction Tuning Data Construction: E-VQA unifies semantic reasoning with fine-grained spatio-temporal grounding, while annotating both jointly from scratch is prohibitively challenging and costly.The proposed pipelines leverage existing semantic and grounded video-understanding datasets.
  • 4 Instruction Tuning Data Construction: The training data aggregates 20k video-question pairs from Perception Test, STAR, and CLEVRER, plus 20k captioned videos with segmentation masks from ViCaS.These sources represent Video Question Answering and Video Segmentation benchmark categories.
  • 4.1 Automatic Data Construction Pipeline: Each E-VQA sample is a tuple (V, Q, A, C, Et, Es) containing video, question, answer, optional choices, temporal evidence, and dense spatial masklets.The construction paradigm is bidirectional and multi-step, conditioned on the source data type.
  • 4.1 Automatic Data Construction Pipeline: The grounding-to-semantics pathway uses Qwen3-VL and Gemini to generate grounded questions, answers, distractors, evidence objects, confidence scores, and explanations from ViCaS captions and masks.Each model typically generates three to four candidate questions per video, yielding typically six to eight candidates.
  • 4.1 Automatic Data Construction Pipeline: A two-stage quality-control process removes formatting errors and low-confidence samples, then uses Gemini-2.5-Pro for text-only validation of captions, objects, generated QA, evidence, and explanations.The first stage discards samples with confidence scores < 3.5.
  • 4.1 Automatic Data Construction Pipeline: For accepted ViCaS samples, Qwen3-VL predicts temporal evidence at 4 FPS, while spatial evidence is directly derived from the corresponding pre-existing mask annotations.Question generation processes videos at 2 FPS, whereas evidence assignment uses 4 FPS to capture finer temporal dynamics.
  • 4.1 Automatic Data Construction Pipeline: The semantics-to-grounding pathway identifies critical evidence objects with Qwen3-VL-235B-A22, generates and filters bounding boxes, then uses SAM-3 to propagate dense masks across each video.Filtering removes boxes with area ratios < 1% or > 90%, anomalous aspect ratios, or severe inconsistencies; masklets with IoU > 0.9 are deduplicated.
  • 4.1 Automatic Data Construction Pipeline: ST-Evidence-Instruct is a 160k-scale dataset containing aligned (Q, A, Et, Es) tuples that provide a foundation for explainable, evidence-backed video reasoning.The dataset is produced by the fully automated construction pipelines.

5 Experiments

Experiments on ST-Evidence show a substantial gap between answer quality and precise visual grounding in existing Video LLMs. Fine-tuning UniPixel on ST-Evidence-Instruct improves temporal and spatial evidence, while validation studies support the quality of the annotations and identify language-model reasoning as the main bottleneck.

  • Baseline performance: Gemini-2.5-Pro leads QA and Temporal Evidence, but its t-mean is only 49.9.Among open-source General Video LLMs, the Qwen3-VL series improves substantially over Qwen2.5-VL.
  • Baseline performance: 44.0 J &F is the strongest General Video LLM spatial-evidence result, achieved by Gemini-2.5-Pro.Gemini-2.5-Flash reaches 42.9, while Qwen3-VL-235B-A22B reaches 41.9, indicating limited grounding despite strong language modeling.
  • ST-Evidence-Instruct fine-tuning: 26.9 T-Evidence is achieved by the fine-tuned 3B model, outperforming Qwen2.5-VL-72B; both fine-tuned 3B and 7B models significantly outperform other baselines on S-Evidence.Compared with UniPixel, fine-tuning yields slight QA gains and substantial improvements in temporal and spatial evidence.
  • ST-Evidence-Instruct fine-tuning: The fine-tuned 7B model consistently surpasses all baseline models on segmentation and general video-understanding benchmarks.The model retains both pixel-level and semantic-level capabilities beyond the proposed E-VQA task.
  • Validation and bottlenecks: 44.7 J &F from manually annotated referring expressions versus 25.6 from Gemini-2.5-Pro shows that the primary E-VQA bottleneck lies mainly in the language model.The proxy segmenter performs substantially better with manually annotated expressions than with expressions generated by top models.
  • Validation and bottlenecks: 62.8 J &F is achieved by the automated annotation pipeline, exceeding the fine-tuned 7B model at 54.1 and the 235B baseline at 41.9.This comparison directly validates the effectiveness of the automatic annotation pipeline against the human-verified ST-Evidence-Gen benchmark.

6 Conclusion

E-VQA unifies video question answering with dense spatio-temporal evidence grounding through the ST-Evidence benchmark and ST-Evidence-Instruct dataset. Fine-tuning on ST-Evidence-Instruct improves both answer accuracy and grounding, establishing a foundation for explainable, evidence-grounded Video LLMs.

  • 6 Conclusion: E-VQA unifies question answering with dense spatio-temporal evidence grounding.The framework is designed for Video LLMs to reason transparently over time and space.
  • 6 Conclusion: ST-Evidence is a human-verified benchmark, while ST-Evidence-Instruct aligns answers with visual evidence for instruction tuning.
  • 6 Conclusion: Fine-tuning on ST-Evidence-Instruct enhances both answer accuracy and visual grounding.
  • 6 Conclusion: The work establishes a foundation for explainable and evidence-grounded Video LLMs.
Loading 2607.11862v1…