Source-linked AI summary
Video as Conditional Graph Hierarchy for Multi-Granular Question Answering
Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, Tat-Seng Chua
TL;DR
VideoQA needs models that reason over hierarchical video elements and multi-granular language rather than treating both as simple sequences. The paper builds a query-conditioned graph hierarchy that aggregates visual facts across levels, and reports strong cross-dataset performance, generalization, and interpretability, while acknowledging limitations in object-level motion and complex action recognition.
Problem
Existing VideoQA methods focus on sequential video and language representations, leaving hierarchical video structure and multi-granular linguistic concepts insufficiently accounted for.
Method
The model constructs a language-conditioned graph hierarchy that level-wise aggregates visual elements from entities and actions to activities and events using query cues.
Results
The model achieves state-of-the-art results across datasets and question types, with enhanced generalizability and interpretability; multi-level query conditioning improves performance, especially for complex videos.
Takeaways & Limitations
Hierarchical, query-conditioned reasoning provides a unified way to align visual elements of different granularity with VideoQA questions and produce visual-textual evidence for answers.
Takeaways & Limitations
The current model relies seriously on object appearance features, lacks region-level motion, and faces gaps in action or activity recognition for complex multi-object videos.
Abstract
from arXiv · showhide
Video question answering requires the models to understand and reason about both the complex video and language data to correctly derive the answers. Existing efforts have been focused on designing sophisticated cross-modal interactions to fuse the information from two modalities, while encoding the video and question holistically as frame and word sequences. Despite their success, these methods are essentially revolving around the sequential nature of video- and question-contents, providing little insight to the problem of question-answering and lacking interpretability as well. In this work, we argue that while video is presented in frame sequence, the visual elements (e.g., objects, actions, activities and events) are not sequential but rather hierarchical in semantic space. To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues. Despite the simplicity, our extensive experiments demonstrate the superiority of such conditional hierarchical graph architecture, with clear performance improvements over prior methods and also better generalization across different type of questions. Further analyses also demonstrate the model's reliability as it shows meaningful visual-textual evidences for the predicted answers.
Introduction
VideoQA must handle hierarchical video content and multi-granular language cues, which sequential models do not fully capture. The paper proposes a language-conditioned graph hierarchy and reports strong results across datasets and question types.
- Motivation: The paper identifies a mismatch between hierarchical video semantics, multi-granular questions, and models that encode both as sequences.Video elements range from entities and local interactions to activities and global events, while questions may require one or multiple granularities.
- Approach: A conditional graph hierarchy level-wise aggregates visual facts while injecting local and global query cues to identify relevant objects, actions, activities, and events.The architecture uses graph aggregation and pooling to connect low-level entities with higher-level video elements.
- Conceptual framework: The bottom-up view organizes video from visual entities through local interactions to activities and global events, while the top-down view links question needs to different granularities.A single question may invoke multiple levels of visual resources.
- Results: The model is evaluated on four VideoQA datasets covering varied aspects of video understanding and achieves consistently strong results.The introduction reports state-of-the-art results across datasets and question types, with enhanced generalizability and interpretability.
Related Work
Prior VideoQA methods use cross-modal, memory, graph, or hierarchical architectures, but important limitations remain in their representation and reasoning structures. The proposed direction addresses flat or overly narrow modeling of video relations by emphasizing hierarchical, fine-grained interactions.
- Sequential and fusion models: Cross-modal attention and motion-appearance memory methods fuse video and question information but treat both modalities mainly as holistic frame and word sequences.This sequential formulation does not directly represent video topology or compositional structure.
- Graph-structured models: Graph-structured models improve relation reasoning or interpretability, yet monolithic graphs can be cumbersome and flat query-specific graphs miss fine-grained interactions and hierarchy.Some graph models use detected regions, while others construct query-specific graphs over coarse video segments.
- Hierarchical architectures: HCRN models hierarchical temporal relations with conditional blocks and mean pooling, but generalizes poorly to multiple objects interacting in space-time.HOSTR extends the design with nested graphs over object trajectories for spatio-temporal reasoning.
Method
The method models video as a query-conditioned hierarchy of graphs that progressively aggregates objects, frame interactions, and clip interactions into a global representation for VideoQA.
- Data Representation: Video inputs combine motion, frame-appearance, and region-of-interest features, while question inputs use contextualized token embeddings and a global query representation.Object features jointly encode RoI appearance, bounding-box location, and temporal position.
- Conditional Graph Reasoning and Pooling: The QGA unit contextualizes visual nodes through semantic and geometric neighborhood reasoning, then pools the refined nodes into a global descriptor.Dynamic graph edges represent node similarities, and skip-connected graph attention refines node representations before self-attention pooling.
- Conditional Graph Reasoning and Pooling: Query conditioning uses token-level language representations to emphasize visual nodes corresponding to video elements mentioned in the question.Cross-attention gives stronger responses to visual nodes whose represented elements have textual correspondences in the query.
- Hierarchical Architecture: QGA units hierarchically aggregate object features into frame-level, clip-level, and global video representations while conditioning each level on the question.The hierarchy captures object interactions, short-term dynamics, and higher-level video elements before answer decoding.
- Answer Decoder: The final aggregated video representation is fused with the global query representation and passed to the answer decoder.The architecture is designed to infer video elements at different granularities and provide query-instantiated visual evidence.
Experiments
Experiments evaluate the model across four VideoQA datasets, compare it with established approaches, and analyze hierarchy, query conditioning, interpretability, and limitations. The results support hierarchical graph modeling while revealing weaker action-recognition performance and dependence on object appearance features.
- Datasets: The model is evaluated on four datasets covering action repetition, state transition, frame QA, and recognition of video elements.TGIF-QA, MSRVTT-QA, MSVD-QA, and NExT-QA probe different aspects of video understanding.
- State-of-the-Art Comparison: The model compares against cross-attention, memory, graph-structured, and hierarchical VideoQA methods across benchmark accuracy tables.The cited comparison covers established methods including ST-VQA, HME, L-GCN, GMIN, HCRN, and others.
- Hierarchy: Removing both lower-level graph units reduces accuracy by 1.5% on NExT-QA and 2.6% on MSRVTT-QA, showing that clip-level modeling alone is insufficient.Removing GO or GF individually also lowers accuracy, with GF removal causing drops over 1.2% on both datasets.
- Multi-level Condition: Removing language conditioning at any single level harms performance, while a global query representation reaches 37.52% versus 37.03% on MSRVTT-QA but loses this benefit on NExT-QA.The analysis also reports small benefits from global context features Fa and Fm.
- Discussion: The method’s current version relies on object appearance features, leaving a 4.6% MSRVTT-QA and 7.5% MSVD-QA gap between action/activity and object/attribute recognition for ‘what’ questions.The authors identify absent region-level motion as a limitation and propose jointly modeling appearance and motion.
- Qualitative Analysis: Learned attention traces identify query-relevant clips, frames, and objects, while conditional weights align high-level action words with graph nodes and lower-level object words with object nodes.The learned adjacency matrices become sparse despite fully connected graph construction, filtering relations relative to the query.
Conclusion
The work models video as a conditional graph hierarchy for level-wise, multi-granular video-language alignment in VideoQA. Experiments and analyses validate the method's effectiveness, while future work targets object-level motion and pretraining.
- A conditional graph hierarchy reasons from low-level visual resources to high-level video elements while injecting language queries at multiple granularity levels.A reusable query-conditioned graph attention unit is stacked to build the hierarchical architecture.
- Experiments and analyses validate the effectiveness of the proposed method for video question answering.
- Future work could incorporate object-level motion information or exploit pretraining techniques to improve performance.
A. Datasets
The experiments use four VideoQA datasets covering recognition, state transition, action repetition, frame QA, and deeper causal and temporal multi-object interactions. Official splits are followed where available, with a random validation split for TGIF-QA.
- Four datasets challenge video understanding from object and action recognition to state transition and causal-temporal interactions among multiple objects.The datasets are TGIF-QA, MSRVTT-QA, MSVD-QA, and NExT-QA.
- Official training, validation, and testing splits are used for NExT-QA, MSRVTT-QA, and MSVD-QA; TGIF-QA uses a random 10% validation split because none is officially provided.
- Table 5 reports dataset details and distinguishes open-ended from multiple-choice tasks.
B. Implementation Details
The implementation combines pretrained motion, appearance, object-detection, and language features for VideoQA. Answer spaces are dataset-dependent and treat out-of-set predictions as failures during evaluation.
- Motion features use a Kinetics-pretrained 3D ResNeXt-101, while sparse-stream appearance features use an ImageNet-pretrained ResNet-101.
- Object regions are obtained with an object detector pretrained on Visual Genome.
- NExT-QA uses official BERT features, while other multiple-choice tasks fine-tune BERTbase on question-choice pairs and open-ended tasks classify questions into answer categories.
- MSRVTT-QA uses the 4,000 most frequent answers plus an out-of-set class, whereas MSVD-QA and TGIF-QA use all training answers as predefined classes.
- Predictions outside the predefined answer sets are counted as failures during evaluation.
Study of Hyper-Parameters
The hyper-parameter study examines graph-attention depth, sampled clips, and candidate regions on NExT-QA and MSRVTT-QA. Two graph-attention layers perform best, while dataset-specific sampling choices affect performance.
- A two-layer graph attention setting, H = 2, produces the best results on both NExT-QA and MSRVTT-QA.One layer may underfit complex relations, whereas three layers may over-smooth visual components in the stacked hierarchy.
- K = 8 sampled video clips provide good performance, although the default K = 16 setting is optimal on the NExT-QA test set.
- NExT-QA benefits from N = 20 candidate regions, whereas MSRVTT-QA uses N = 10.The passage relates this difference to longer NExT-QA videos containing multiple interacting objects.
Analysis of Efficiency
The model uses a lightweight three-level graph hierarchy with shared parameters at each level. This design improves time efficiency relative to prior RNN-based approaches, although feature extraction is excluded and end-to-end speed may be lower than segment-only models.
- Three stacked 2-layer graphs make the model light-weight, with parameter sharing among graphs at the same level.
- Graph-hierarchy modeling improves time efficiency compared with previous RNN-based approaches.
- Overall runtime with feature extraction may be slower than models using only segment-level features, despite fast execution once features are ready.
Qualitative Analysis
Qualitative analyses show that the hierarchical conditional graph aligns query phrases with video elements at matching granularity, abstracts local components into global events, and diagnoses errors through visual evidence. The examples also reveal sensitivity to incomplete sampling and distractor content.
- Successful predictions: The model aligns dynamic phrases with higher graph levels and contact actions with object-level representations, then traces evidence to the relevant frame and region.“After bending down” is grounded at GC and GF, while “feed horse” is grounded at GO and traced to O18.
- Successful predictions: The hierarchy aggregates local elements such as singers and instruments into a global “singing performance” answer without explicit referring cues.
- Failure analysis: A failure resulted when incomplete sampling omitted the transient “fell off swing” content and the model was distracted by a visually similar clip and answer.The missed content should have appeared between clips C6 and C7, while clip C5 resembled “fell off.”
- Overall interpretation: The analyses indicate that the model supports multi-granular visual-textual matching and provides evidence for interpreting correct and incorrect predictions.
- Visualization: Figure 6 visualizes ground-truth answers, predictions, self-attention, and conditional graph weights over object bounding boxes.