Source-linked AI summary
Hierarchical Conditional Relation Networks for Video Question Answering
Thao Minh Le, Vuong Le, Svetha Venkatesh, Truyen Tran
TL;DR
VideoQA requires representations that capture dynamic visual artifacts, distant relations, and linguistic concepts, while existing architectures are often specialized across modalities and tasks. The paper introduces reusable Conditional Relation Networks and stacks them hierarchically into HCRN for multimodal video reasoning. HCRN demonstrates competitive reasoning capability across multiple VideoQA datasets and supports robust visual reasoning through native multimodal interaction.
Problem
VideoQA needs question-specific representations of dynamic visual properties and distant temporal relations, but existing specialized architectures are not readily adaptable across modalities, video lengths, and question types.
Method
The paper builds HCRN by replicating and stacking CRNs that encode high-order relations among video objects while conditioning them on motion and linguistic context.
Results
HCRN demonstrates competitive reasoning capability on the TGIF-QA, MSVD-QA, and MSRVTT-QA VideoQA datasets.
Takeaways & Limitations
Generic conditional relational units support hierarchical, iterative interaction among appearance, motion, relations, and language for VideoQA.
Abstract
from arXiv · showhide
Video question answering (VideoQA) is challenging as it requires modeling capacity to distill dynamic visual artifacts and distant relations and to associate them with linguistic concepts. We introduce a general-purpose reusable neural unit called Conditional Relation Network (CRN) that serves as a building block to construct more sophisticated structures for representation and reasoning over video. CRN takes as input an array of tensorial objects and a conditioning feature, and computes an array of encoded output objects. Model building becomes a simple exercise of replication, rearrangement and stacking of these reusable units for diverse modalities and contextual information. This design thus supports high-order relational and multi-step reasoning. The resulting architecture for VideoQA is a CRN hierarchy whose branches represent sub-videos or clips, all sharing the same question as the contextual condition. Our evaluations on well-known datasets achieved new SoTA results, demonstrating the impact of building a general-purpose reasoning unit on complex domains such as VideoQA.
1. Introduction
VideoQA requires question-specific modeling of dynamic visual properties and temporal relations, while specialized architectures struggle across modalities, video lengths, and question types. HCRN addresses this with reusable, hierarchically stacked CRNs and performs well across diverse VideoQA questions.
- VideoQA must represent object permanence, motion, prolonged actions, and varying-length temporal relations under linguistic guidance.
- Handcrafted VideoQA subsystems are often specialized for a modality or purpose, making them less adaptable to changing modalities, video lengths, and question types.
- CRN transforms object arrays into context-conditioned arrays, encoding sparse high-order relations and enabling reusable hierarchical HCRN construction.
- HCRN hierarchically integrates frame appearance, clip motion, linguistic context, inter-clip relations, and whole-video motion across successive CRN stages.
- A four-layer HCRN achieves favorable accuracy across VideoQA tasks and performs consistently on appearance, motion, state transition, temporal-relation, and action-repetition questions.
- HCRN handles representative frame-relation cases that challenge a flat visual-question baseline, including near-term action counting and long transitions.
2. Related Work
Prior VideoQA systems use recurrent, memory, fusion, attention, and temporal-hierarchy approaches, but often separate modalities or rely on heterogeneous components. HCRN instead enables iterative multimodal interaction through reusable conditional relational building blocks.
- HCRN represents appearance, motion, and relations as complementary video factors that interact iteratively within a hierarchical multi-scale framework.
- Recurrent and memory networks improve modeling for longer or structurally complex videos, including movies and television programs with additional modalities.
- HCRN separates conditioning factors from refined information, making multimodal fusion more efficient and more adaptable to conditioning types than costly joint multiplicative operators.
- Although temporal hierarchy has been studied with recurrent and graph networks, HCRN introduces hierarchical interaction of multimodalities including linguistic cues for VideoQA.
- Unlike prior visual-linguistic interaction methods, HCRN conditions video features with linguistic clues at every stage of multi-level refinement.
- CRN is an array-to-array block supporting conditional relational reasoning that can be reused to build networks for vision and language processing.
3. Method
The method builds VideoQA representations with reusable Conditional Relation Networks, stacked hierarchically across video granularities and conditioned on linguistic cues. This hierarchy progressively integrates appearance, motion, and question information before classification.
- VideoQA maps a video V and question q to an answer ã in a predefined answer space A.
- Conditional Relation Network: CRN transforms an array of tensorial objects into same-dimensional outputs while encoding high-order relations under a conditioning feature.Inputs may be vectors in R^d or tensors in R^(W×H×d), and the output preserves the object-array form.
- Conditional Relation Network: CRN reduces redundancy and computation by sampling subsets of mutually informative input objects rather than considering every combination.The implementation can use k_max = n − 1, with average-pooling aggregation and parameter-tied conditioning subnetworks across equal-sized subsets.
- Hierarchical Conditional Relation Networks: HCRN stacks reusable CRN blocks to model temporal relations, motion, video hierarchy, and question-guided reasoning.The architecture is explicitly named Hierarchical Conditional Relation Networks and is built by stacking core units layer by layer.
- Hierarchical Conditional Relation Networks: At clip level, CRNs relate frame appearances using clip motion and linguistic context; at video level, they relate clip encodings using video motion and the question.Clip motion features are passed through an LSTM to form video-level motion features, while the video-level input array is the clip-level output.
- Hierarchical Conditional Relation Networks: For very long videos, deeper hierarchy groups short-term clips into hyper-clips to densely model relations while reducing computational cost.The proposed grouping creates a three-level hierarchical network instead of only limiting the maximum subset size.
- Answer decoding: The final visual representation is averaged using the question-conditioned video-level outputs and combined with the question representation for answer prediction.Open-ended tasks use a classifier over visual and question features with cross-entropy, while repetition-count tasks use regression with MSE.
4. Experiments
Experiments evaluate HCRN across VideoQA benchmarks, ablations, and hierarchy settings. HCRN is competitive or superior across datasets, while high-order relations, hierarchical structure, motion, and question conditioning support temporal reasoning and efficient long-video processing.
- Datasets and metrics: TGIF-QA evaluates repetition count, repeating action, state transition, and Frame QA, covering temporal reasoning and single-frame answering.Accuracy is used throughout except for repetition count, which uses MSE.
- Benchmarking against SoTAs: HCRN consistently outperforms or matches state-of-the-art methods across all tasks and datasets, with particularly noticeable gains on action and transition questions.These tasks require strong temporal reasoning, including near-term and far-term relations.
- Benchmarking against SoTAs: 36.1% accuracy on MSVD-QA and 35.6% on MSRVTT-QA improve results by 1.7 and 0.6 points, respectively.The datasets are open-ended benchmarks, with MSRVTT-QA containing longer and more complex videos.
- Ablation Studies: Ablating design components or CRN units degrades performance on temporal reasoning tasks, including actions, transitions, and action counting.The ablation study uses TGIF-QA configurations reported in Table 4.
- Ablation Studies: High-order relations are critical for temporal reasoning, whereas sampling resolution has less effect when relations are modeled.Frame QA is an exception because it can be answered from a single frame, making temporal information potentially confusing.
- Deepening model hierarchy: A deeper hierarchy reduces training and inference time by 4 times while maintaining the same performance when moving from 2-level to 3-level HCRN.The long-video evaluation uses 24 clips and compares similarly sized models of approximately 50M parameters.
5. Discussion
The paper presents CRNs as reusable relational transformers for hierarchical, context-conditioned VideoQA reasoning. It emphasizes relational modeling over object selection and identifies broader multimodal reasoning applications.
- CRNs map arrays of tensorial objects to new arrays conditioned on contextual features while encoding and modulating high-order relations.Their encapsulating design supports flexible stacking, hierarchy, and iterative reasoning.
- HCRN concentrates on modeling relations and hierarchy rather than selecting objects through temporal attention.The authors note that attention mechanisms can augment CRNs for improved object selection in related tasks such as frame QA.
- The examination of CRN highlights generic neural reasoning units with native multimodal interaction as important for robust visual reasoning.The authors identify extensions involving linguistic channels as relevant to TVQA and MovieQA.