Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering

• VideoQA remains challenging because videos have dynamic spatiotemporal dependencies and questions have sophisticated compositional semantics.Source: §1 · p. 1
• Prior attention methods model temporal, spatial, or spatiotemporal visual relationships but rely on positional relationships rather than in-depth semantics.This makes sophisticated appearance-motion and visual-question relations difficult to capture.Source: §1 · p. 1
• Global question features can miss video-word relations, while local approaches associate visual and word information through co-occurrence rather than question compositional semantics.The example contrasts linking “woman” to all women with using “in the red” to identify the left woman.Source: §1 · p. 2
• Structured representations have been explored for VQA and image-text matching, but remain underexplored for VideoQA.Source: §1 · p. 2
• Bridge to Answer builds appearance, motion, and question graphs, using question-to-visual interactions and the question graph as a bridge for visual-to-visual interactions.These interactions model visual-question and appearance-motion relations while incorporating the question’s compositional semantic structure.Source: §1 · p. 2
• The paper evaluates Bridge to Answer with ablation studies and state-of-the-art comparisons on TGIF-QA, MSVD-QA, and MSRVTT-QA.Source: §1 · p. 2

2. Related Works

• Temporal attention methods model appearance-question relations [20], with co-attention, self-attention enhancement [34], and object- or frame-aware interactions [10].The cited approaches use object-aware temporal attention or frame and object features to capture fine-grained interactions.Source: §2 · p. 2
• Following two-stream appearance and motion features [11], VideoQA methods developed multimodal fusion, co-memory [6], hierarchical, multi-head [16], and multi-step attention [14].Hierarchical attention also captures appearance-question and motion-question relations from frame to segment level [18].Source: §2 · p. 2
• Unlike methods relying on positional relationships, this work graph-models appearance, motion, and question to learn question-conditioned visual representations and mutually enhance appearance and motion.The authors identify the use of question compositional semantics to reason between appearance and motion as the work’s novel contribution.Source: §2 · pp. 2–3
• Graph-based vision-language methods include image-question graphs [33], region-to-phrase matching [21], paired visual and textual semantic graphs [26], and hierarchical video-caption graphs [1].These approaches learn structured visual-text relationships, but the paper notes they cannot be directly applied to VideoQA.Source: §2 · pp. 2–3

3.1. Feature Extraction and Graph Construction

• The method represents video appearance with frame features and motion with clip features, while question words are embedded and encoded with a BiGRU.Appearance and motion graphs are fully connected; the question graph connects words with semantic dependencies identified by Stanford CoreNLP, with edge weights based on representation affinity.Source: §3.1 · pp. 3–4

3.2. Question-to-Visual Interactions

• Question-to-visual interactions associate question nodes with appearance and motion nodes, aggregate question representations, and propagate them along visual graph edges to learn question-conditioned visual representations.The method describes question-to-appearance in detail and applies the symmetric operation to motion.Source: §3.2 · p. 4

3.3. Visual-to-Visual Interactions

• The visual-to-visual interaction uses the question graph as a bridge to propagate question-conditioned information between appearance and motion graphs along question dependencies.The process bridges one visual graph to the question graph, propagates and aggregates representations through graph convolution, then delivers them to the other visual graph; the method applies this symmetrically for motion-to-appearance and appearance-to-motion.Source: §3.3 · p. 5

3.4. Answer Decoder and Loss Functions

• The decoder treats open-ended questions as multi-label classification, repetition counts as rounded linear regression, and multiple-choice questions as candidate scoring.Open-ended questions use cross-entropy loss, repetition counts use mean squared error, and multiple-choice candidate scores use hinge loss against incorrect answers.Source: §3.4 · pp. 5–6

4.1. Datasets

• The evaluation covers TGIF-QA’s four video-question tasks, MSVD-QA’s five question types, and the open-ended questions of MSVD-QA and MSRVTT-QA.TGIF-QA includes repetition count, repeating action, state transition, and Frame QA; MSVD-QA and MSRVTT-QA are described as having what, who, how, when, and where questions.Source: §4.1 · p. 6

4.2. Implementation Details

• Videos are divided into eight clips of 16 frames by default; MSRVTT-QA videos are additionally divided into 24 clips, and results use the epoch with best validation accuracy.Training runs for 25 epochs, with batch size 16 for TGIF-QA and MSVD-QA and 4 for MSRVTT-QA; the learning rate is 10^-4 and is halved every five epochs.Source: §4.2 · p. 6

4.3. Experimental Results

• On TGIF-QA, the model achieves state-of-the-art performance and outperforms existing methods on every task except FrameQA.Source: §4.3 · p. 7
• On MSVD-QA and MSRVTT-QA, the model achieves 37.2% and 36.9% accuracy, outperforming existing approaches by 1.1% and 1.3%, respectively.These open-ended-question benchmarks are described as highly challenging compared with TGIF-QA.Source: Table 3 · p. 6, §4.3 · p. 7

4.4. Ablation Studies

• Ablations find that removing any interaction direction degrades performance across tasks, with Q2V a primary component and Q2M important except for FrameQA.Removing either appearance or motion is critical, while motion contributes more on action-related tasks and appearance matters more for FrameQA.Source: §4.4 · p. 7
• The visualization shows relevant nodes connected with relatively high interaction values, and the question node is connected to an appearance node through semantic relation; uniform motion-question and question-appearance values are 0.09 and 0.008, respectively.For example, the first clip–“man” and “man”–fourth-frame connections each have an interaction value of 0.15.Source: §4.4 · p. 8
• The bridged architecture improves task results by 0.5%, 0.9%, and 0.6% in accuracy, and by 0.12 in MSE.The ablation compares bridged visual-to-visual interactions with interactions performed without the question bridge.Source: §4.4 · p. 8
• Larger λ performs better than λ = 1; λ = 10 is best on MSVD-QA, while λ = 20 is best on MSRVTT-QA.The paper suggests the datasets’ different optimal values might result from their different video lengths.Source: Table 5 · p. 7, §4.4 · p. 8

• Bridge to Answer achieves state-of-the-art performance on TGIF-QA, MSVD-QA, and MSRVTT-QA.Source: §5 · p. 8
• Question-visual interactions encode semantic relations between visual and question graphs in appearance- and motion-based representations.Source: §5 · p. 8
• Question semantics bridge appearance and motion graphs, modulating their relations to enhance visual representations and support robustness to scene composition and complex questions.Source: §5 · p. 8