PaperlayerView exact revision on arXiv

Source-linked AI summary

MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering

Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, Mike Zheng Shou

arXiv:2212.09522v1cs.CV

TL;DR

Long-form VideoQA requires reasoning across multiple events and visual granularities, challenges that dense attention and sparse sampling do not address together. MIST selects question-relevant segments and regions and iteratively attends to visual concepts; it reports state-of-the-art performance on four datasets alongside efficiency and interpretability advantages.

  • Problem

    Long-form VideoQA requires multi-event and multi-granularity reasoning, while dense attention is costly and sparse-sampling approaches struggle with these demands.

    Source: §1 · pp. 1–2

  • Method

    MIST decomposes dense spatial-temporal attention into question-conditioned segment and region selection followed by iterative attention over multi-modal, multi-granularity features.

    Source: §1 · p. 2

  • Results

    MIST achieves state-of-the-art performance on AGQA, NExT-QA, STAR, and Env-QA, with higher efficiency and interpretability.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    MIST's results support its effectiveness and efficiency and interpretability advantages for long-form VideoQA across the four evaluated datasets.

    Source: §1 · p. 2, §5 · p. 8

    Patch features have limitations in complex object-level reasoning, and high-frequency motion may still need improvement.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must. Existing multi-modal VQA models achieve promising performance on images or short video clips, especially with the recent success of large-scale multi-modal pre-training. However, when extending these methods to long-form videos, new challenges arise. On the one hand, using a dense video sampling strategy is computationally prohibitive. On the other hand, methods relying on sparse sampling struggle in scenarios where multi-event and multi-granularity visual reasoning are required. In this work, we introduce a new model named Multi-modal Iterative Spatial-temporal Transformer (MIST) to better adapt pre-trained models for long-form VideoQA. Specifically, MIST decomposes traditional dense spatial-temporal self-attention into cascaded segment and region selection modules that adaptively select frames and image regions that are closely relevant to the question itself. Visual concepts at different granularities are then processed efficiently through an attention module. In addition, MIST iteratively conducts selection and attention over multiple layers to support reasoning over multiple events. The experimental results on four VideoQA datasets, including AGQA, NExT-QA, STAR, and Env-QA, show that MIST achieves state-of-the-art performance and is superior at computation efficiency and interpretability.

1. Introduction

Long-form VideoQA must handle diverse activities, multiple events, and interactions across visual granularities, while adapting pre-trained models without dense-attention costs or sparse-sampling gaps. MIST uses question-conditioned selection and iterative attention, and reports state-of-the-art results on four datasets with efficiency and interpretability advantages.

1. Introduction

  • VideoQA aims to help people solve everyday problems by answering questions about diverse events in videos of their activities.Examples include helping users find something or remember what they did.Source: §1 · p. 1
  • Long-form questions require multi-event temporal and causal reasoning, as well as interactions among objects, relations, and events across different granularities.The videos record more events than short clips, expanding the reasoning demands of the questions.Source: §1 · p. 1
  • Dense attention is costly for long videos, while sparse 3- or 4-frame sampling risks a domain gap and segment aggregation struggles with multi-granularity interactions.Sparse frames that form coherent actions in short-video pre-training may become random event shots in long videos.Source: §1 · p. 2
  • MIST decomposes dense spatial-temporal attention into question-conditioned segment and region selection followed by attention over multi-modal, multi-granularity features.Iterating selection and attention over layers supports reasoning across multiple events and temporal and causal reasoning.Source: §1 · p. 2
  • MIST achieves state-of-the-art performance on AGQA, NExT-QA, STAR, and Env-QA, with higher efficiency and interpretability.The evaluated datasets have average video durations ranging from 12 to 44 seconds.Source: §1 · p. 2

2. Related Work

Prior VideoQA work spans activity clips, subtitled movie and television videos, and long-form benchmarks targeting spatial-temporal reasoning. Related methods address video efficiency or adapt pre-trained models, while MIST uses questions to guide iterative attention toward long-form, multi-event and multi-granularity reasoning.

2. Related Work

  • Long-form benchmarks target spatial-temporal reasoning such as causality and sequential order, while some VideoQA datasets focus on daily activities and others on subtitled movie or television plots and social interactions.The cited long-form datasets are; TVQA, MovieQA, and Social-IQ are [29],, and.Source: §2 · p. 2
  • VideoQA methods include LSTM- or graph-based cross-modal and motion-appearance modeling, graph modeling of objects and relations, and pre-trained Transformer fine-tuning [10] [46].Hierarchical processing of videos with pre-trained Transformers is proposed in.Source: §2 · pp. 2–3
  • Video-transfer methods aggregate CLIP features or model temporal differences [9], whereas MIST adapts pre-trained models to long-form VideoQA's multi-event and multi-granularity reasoning focus.The related transfer tasks include object detection, image generation, and video-text retrieval [9] [55] [60].Source: §2 · p. 3
  • Long-form video methods reduce computation through short-term features and long-term memory [7], audio cues [34], or a structured multi-scale temporal decoder [17]; MIST instead uses questions to guide computation.MIST considers QA-task characteristics to reduce computation and benefit performance.Source: §2 · p. 3
  • Unlike iterative attention work on recurrent image classification, Perceiver, and TimeSformer, MIST uses visual-question correspondence to guide iterative attention in VideoQA.The related approaches target efficiency or performance through recurrent region attention, asymmetric attention, or video-classification attention schemes.Source: §2 · p. 3

3. Method

MIST adapts pre-trained vision-language features to long-form VideoQA by iteratively selecting question-relevant video segments and regions for spatial-temporal reasoning, then matching video-question representations to candidate answers.

3.1. Input Representation

  • The video is split into K uniform segments of T frames, each frame into N patches; frozen vision-language encoders produce patch features with temporal-position embeddings.Source: §3.1 · p. 3

3.2. Iterative Spatial-Temporal Attention Layer

  • ISTA pools patch features into frame and segment representations, then uses question-conditioned temporal attention and differentiable top-k selection to choose segments; cross-modal attention selects relevant patches within sampled frames.Source: §3.2 · pp. 4–5
  • For spatial-temporal reasoning, ISTA attends over selected frames and regions while retaining all K segment features, limiting patch cost while supporting broader multi-event reasoning.Source: §3.2 · p. 5
  • Across L ISTA layers, updated segment and word features feed the next layer for multi-event interaction; outputs from all layers are mean-pooled for answer prediction.Source: §3.2 · p. 5, §3.3 · p. 5

3.3. Answer Prediction

  • The model mean-pools ISTA layer outputs and ranks pre-trained candidate-answer features by similarity; the highest-similarity candidate is predicted, with softmax cross-entropy training.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate MIST on four longer-video QA benchmarks and report state-of-the-art results, alongside component ablations, computation comparisons, and qualitative attention examples.

4.1. Datasets

  • The evaluation uses AGQA v2, NExT-QA, STAR, and Env-QA, whose videos average 12–44 seconds and whose tasks include compositional, causal, temporal, situated, and dynamic-environment reasoning.AGQA v2 has 2.27M QA pairs over 9.7K videos; the other benchmarks range from 5,440 to 23K videos.Source: §4.1 · p. 5

4.2. Implementation Details

  • Experiments use CLIP (ViT-B/32) and All-in-One-Base backbones, sampling 32 frames into eight segments and setting Topk=2, Topj=12, and ISTA depth L=2.The model is trained with AdamW on NVIDIA RTX A5000 GPUs and implemented in PyTorch.Source: §4.2 · p. 5

4.3. Comparison with State-of-the-arts

  • MIST outperforms state-of-the-art methods on all four datasets; gains are especially strong on questions involving multi-grained concepts, causality, or multi-event reasoning.The reported comparisons cover AGQA v2, NExT-QA, STAR, and Env-QA; gains are relatively limited on Env-QA, whose virtual-environment videos differ from CLIP pretraining data.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6

4.4. Comparison with Baselines

  • ISTA improves overall accuracy by more than 3% over alternatives; region-aware models outperform frame-only attention, and integrating multiple visual granularities adds further benefit.The baselines include mean pooling, frame-level Transformer attention, patch-level Transformer attention, and divided space-time attention.Source: §4.4 · pp. 6–7

4.5. Ablation Study

  • Removing Segment Selection drops accuracy by more than 3%, removing spatial-temporal attention significantly hurts both datasets, and removing spatial attention has little effect.Segment Selection also matters for Causal and Temporal questions; small Topk values of 1 or 2 work well, while performance can saturate or decline with more selected regions or deeper ISTA. Spatial selection may reduce computation on future high-resolution or extremely complex videos.Source: §4.5 · pp. 7–8, Figure 5 · p. 7

4.6. Computation Efficiency

  • Sampling more frames significantly raises accuracy, while MIST achieves better performance than frame-based methods with only a little extra computation and no larger model size.Patch-level standard Transformer attention is computationally expensive, whereas the frame-based method is lightweight but has limited performance.Source: Figure 5 · p. 7, §4.6 · p. 8

4.7. Qualitative Results

  • On NExT-QA, visualizations show question-relevant clip and region selection, but selection is difficult for questions about social emotions.The authors attribute this difficulty to pretrained models’ limited understanding of correspondences between abstract concepts and videos.Source: §4.7 · p. 8, Figure 6 · p. 8

5. Conclusion and Future Work

MIST uses cascaded segment and region selection with attention over visual concepts to support efficient long-form VideoQA and multi-event reasoning. The authors report effectiveness, efficiency, and interpretability across four datasets, while noting remaining limitations in motion and object-level reasoning.

5. Conclusion and Future Work

  • MIST decomposes dense self-attention into cascade segment and region selection, then uses attention over visual concepts and iterates these steps to support multi-event reasoning.Source: §5 · p. 8
  • MIST achieves state-of-the-art performance on four VideoQA datasets, with reported advantages in efficiency and interpretability.Source: §5 · p. 8
  • The authors identify high-frequency motion capture and complex object-level reasoning as remaining limitations, and suggest exploring action- and object-focused pre-trained models.Source: §5 · pp. 8–9

Appendix

The appendix provides supplementary discussions, experiment-setting details, additional experimental results, and prediction visualizations.

Appendix

  • The appendix is presented as supplementary material to the main paper.Source: Appendix · p. 9
  • The appendix adds discussion of MIST’s top-k selector and further details on experimental settings.Source: Appendix · p. 9
  • It also includes additional experimental results and visualizations of prediction results.Source: Appendix · p. 9

A. More Discussions of Top-k Selector

The appendix explains the differentiable top-k selection used in MIST and why its sampling choice fits long-form VideoQA, where questions may concern only one segment.

A. More Discussions of Top-k Selector

B. More Details about Experiment Setting

The appendix describes evaluation datasets, metrics, and an Env-QA-specific answer-prediction adjustment. QA accuracy is used for AGQA, NExT-QA, and STAR, while Env-QA uses role-value answers and an IOU-like score.

B. More Details about Experiment Setting

C. Experimental Results

On AGQA v1, MIST outperforms prior state-of-the-art methods, including methods designed for compositional questions. Selector ablations favor sampling with replacement and show an advantage for two ISTA layers over non-parametric attention.

Performances on AGQA v1

  • MIST obtains about 4% performance boost over the previous SOTA using the same Temp[ATP] feature and performs better on compositional reasoning than specialized designs.The comparison is reported for AGQA v1.Source: C. Experimental Results · p. 9

Comparison of Different Top-k Selectors

  • The evaluated selectors include segment sampling with and without replacement, plus non-parametric attention based on pretrained visual-question feature matching.Sampling without replacement sets selected segments’ next-round sampling probabilities to zero; non-parametric attention cannot be trained and therefore selects the same segments across iterations, so this variant has one ISTA layer.Source: C. Experimental Results · pp. 9–10
  • Sampling with replacement slightly outperforms sampling without replacement, while non-parametric attention is worse than two-layer G-S w. Replacement.Non-parametric attention performs similarly to one-layer G-S w. Replacement; Table 8 reports selector accuracies on the AGQA v2 test set.Source: C. Experimental Results · p. 10, Table 8 · p. 10

D. More Visualizations of MIST

MIST’s visualized predictions reveal difficulties with partial visibility and with distinguishing numerous similar events.

More Visualizations of MIST

  • When an object is partially visible early and fully visible later, MIST may select the later segment for a question about its earlier appearance.The example concerns a lady who puts a box aside at the beginning of the video.Source: D. More Visualizations of MIST · p. 10
  • MIST has difficulty distinguishing numerous similar events, such as a boy stirring, a lady taking over, and the boy pulling her arm to reclaim the stirrer.The passage describes locating the correct segments for these events as relatively hard.Source: D. More Visualizations of MIST · p. 10
Loading 2212.09522v1…