Source-linked AI summary
Location-aware Graph Convolutional Networks for Video Question Answering
Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, Chuang Gan
TL;DR
Video QA must recognize relevant actions, object interactions, and temporal order despite complex backgrounds and multiple actions. L-GCN builds a location-aware graph over detected objects, applies graph convolution, and combines the result with encoded question features through attention. It achieves state-of-the-art performance on TGIF-QA, Youtube2Text-QA, and MSVD-QA.
Problem
Video QA requires reasoning over multiple actions, temporal cues, and complex videos with irrelevant backgrounds, while prior methods can neglect direct interactions between nonadjacent objects.
Method
L-GCN represents detected objects as nodes with appearance, spatial, and temporal location features, applies graph convolution, and combines graph and question features through visual-question attention.
Results
L-GCN achieves state-of-the-art results on TGIF-QA, Youtube2Text-QA, and MSVD-QA, including 53.2% accuracy on the who task versus 29.4%.
Takeaways & Limitations
Object graphs let the model focus on foreground action content while directly modeling object relationships and event location for video question answering.
Takeaways & Limitations
Recurrent processing of object sequences can neglect direct interactions between nonadjacent objects, motivating the graph-based design.
Abstract
from arXiv · showhide
We addressed the challenging task of video question answering, which requires machines to answer questions about videos in a natural language form. Previous state-of-the-art methods attempt to apply spatio-temporal attention mechanism on video frame features without explicitly modeling the location and relations among object interaction occurred in videos. However, the relations between object interaction and their location information are very critical for both action recognition and question reasoning. In this work, we propose to represent the contents in the video as a location-aware graph by incorporating the location information of an object into the graph construction. Here, each node is associated with an object represented by its appearance and location features. Based on the constructed graph, we propose to use graph convolution to infer both the category and temporal locations of an action. As the graph is built on objects, our method is able to focus on the foreground action contents for better video question answering. Lastly, we leverage an attention mechanism to combine the output of graph convolution and encoded question features for final answer reasoning. Extensive experiments demonstrate the effectiveness of the proposed methods. Specifically, our method significantly outperforms state-of-the-art methods on TGIF-QA, Youtube2Text-QA, and MSVD-QA datasets. Code and pre-trained models are publicly available at: https://github.com/SunDoge/L-GCN
1 Introduction
Video question answering must identify relevant actions amid complex, changing video content and reason about their temporal order and object interactions. L-GCN addresses these challenges with a location-aware object graph and achieves state-of-the-art results on three benchmarks.
- Motivation: Video QA is more challenging than image QA because videos contain many frames, irrelevant backgrounds, multiple actions, and temporal cues.Questions may concern only a subset of actions and require reasoning about their order.
- Limitations of Prior Work: Existing frame-based methods use spatiotemporal attention, while object-sequence methods can neglect direct interactions between nonadjacent objects.Recurrent processing also depends on arranging the input object sequence.
- Proposed Approach: L-GCN constructs a fully connected graph over detected objects, encodes spatial and temporal locations in nodes, and uses graph convolution for interaction-based action reasoning.Question and graph outputs are combined through a visual-question interaction module for answer prediction.
- Results: L-GCN achieves state-of-the-art results on TGIF-QA, Youtube2Text-QA, and MSVD-QA.The paper reports this outcome across all three benchmark datasets.
- Contributions: The method explicitly excludes irrelevant background content and models relationships among detected objects for video question answering.These design choices are listed as core contributions of the proposed method.
2 Related Work
Related work spans image and video question answering, attention-based visual reasoning, and graph-based relation modeling. Existing video QA methods commonly model frame-level motion, appearance, attention, or long-range dependencies, but neglect object interactions.
- VQA Tasks: Visual question answering is divided into image QA, which focuses on spatial information, and video QA, which addresses visual content that changes over time.Image QA commonly uses attention to connect question words with relevant image regions.
- Video QA: Video QA methods use motion and appearance features, self-attention, and location encoding to represent videos and model long-range dependencies.These approaches target understanding of untrimmed or dynamically changing videos.
- Video QA: Existing video QA methods neglect interactions between objects, although such relationships are described as vital for the task.This gap motivates object-centered graph reasoning.
- Graph-Based Reasoning: Graph convolutional networks have been applied to relation reasoning and can relax sequence restrictions in text-based tasks by learning graph weights through attention.Prior graph-based work also includes methods for dynamically learning graph structures.
- Attention Mechanisms: Attention mechanisms support video recognition, language translation, co-attention, and video QA by selecting or relating visual and language information.The paper positions its visual-question interaction module within this broader attention-based literature.
3 Proposed Method
L-GCN processes questions and videos in separate streams, representing detected objects with appearance and location information in a graph. Graph reasoning and visual-question attention are then used to predict answers.
- Notation and Problem Definition: The video contains N frames and K detected objects per frame, with object features obtained by RoIAlign and spatial locations represented separately.The total number of objects is denoted T=N×K.
- Notation and Problem Definition: Video QA requires recognizing relevant actions in complex content and determining their temporal order.Questions may involve multiple actions and strong irrelevant backgrounds.
- Overall Framework: The framework uses a Bi-LSTM question encoder and a video encoder based on a location-aware object graph.The two streams are combined before answer prediction.
- Location-Aware Graph: The location-aware graph integrates spatial and temporal location information into object-node features to model relationships relevant to answer reasoning.This design makes the graph aware of event order as well as spatial placement.
- Training Process: The training process constructs the graph, extracts question and object features, updates the adjacency matrix, performs graph reasoning, and predicts answers.It iterates these operations until convergence.
3.3 Question Encoder Stream
The question encoder combines character and word representations to handle vocabulary variation and misspellings. A highway network and Bi-LSTM then produce contextual question features.
- Question Representation: Each question is represented with character embeddings and word embeddings for its κ words.The two embedding types have separate dimensions and are concatenated through later processing.
- Question Representation: A two-layer highway network combines the character and word embeddings into a question embedding.Word embeddings are initialized with pretrained 300-dimension GloVe vectors, while character embeddings are randomly initialized.
- Question Representation: Character embeddings are processed by a function containing a two-dimensional convolutional layer.This processing contributes to the question embedding.
- Contextual Encoding: A bidirectional LSTM processes the question embedding and stacks hidden states from both directions to produce the question feature FQ.The resulting feature preserves representations at each question-word time step.
3.4 Location-aware Graph Construction
L-GCN constructs a fully connected graph over detected objects, augmenting appearance features with spatial and temporal location information so the graph preserves where and when actions occur.
- Graph construction: The method constructs a fully connected graph whose nodes are detected video objects and whose edges represent object relationships.Object features alone would omit location information important for temporally related answer reasoning.
- Spatial location encoding: Each detected object’s spatial location is encoded with an MLP from its bounding-box coordinates.The box is represented by its top-left coordinate, width, and height.
- Temporal location encoding: Temporal location features encode an object’s frame position using sine and cosine functions at different frequencies.The temporal feature is indexed by its entries and dimension.
- Node representation: Each graph node concatenates appearance, spatial-location, and temporal-location features.Thus, nodes contain both object appearance and location information.
3.5 Reasoning with Graph Convolution
The reasoning module applies multilayer graph convolution to location-aware node features, learns edge weights from node representations, and adds a skip connection from the input to the final regional features.
- Graph convolution: A P-layer GCN transforms the location-aware graph into regional features through graph convolution.The hidden representation at each layer is updated using the graph structure and trainable weights.
- Regional features: The regional features combine the final GCN output with the original node features through a skip connection.This design is described as improving training performance similarly to ResNet.
- Learnable graph structure: The adjacency matrix is learnable, allowing the model to infer edge weights for all object relationships during reasoning.The matrix is calculated from node features using projected representations and row-wise softmax.
3.6 Visual Encoder Stream
The visual encoder extracts object features from detected regions, performs graph-based message passing, and combines regional object features with temporally processed global frame context.
- Object feature extraction: The encoder extracts frame features with a fixed feature extractor and obtains detected-object features using RoIAlign followed by dimension reduction.Each frame contributes K detected bounding boxes to the object representation.
- Object interaction: A location-aware graph is built over detected objects, and graph convolution enables message passing between them.The graph-convolution output is called the regional feature representation.
- Global context: The encoder also applies global average pooling to frame features to generate global features that provide video context.These global features complement the object-centered regional features.
- Visual feature fusion: A 1D convolution merges neighboring-frame context, after which replicated global features are combined with regional features through an MLP to produce visual features.The resulting visual features are used by the subsequent question-interaction module.
3.7 Visual-question Interaction Module
The visual-question interaction module aligns visual and question representations, attends to question words for each visual representation, and aggregates the resulting cross-modal features for answer prediction.
- Feature alignment: Independent fully connected layers map visual and question features into a shared subspace before interaction.The resulting representations have dimensions T×d_s and L×d_s.
- Cross-modal attention: A row-wise softmax over visual-question dot products produces a similarity matrix that identifies relevant question words for each visual feature.This matrix supplies the attention weights for cross-modal reasoning.
- Cross-modal representation: The model computes weighted question features and combines them with visual features into a cross-modal representation.The cross-modal representation is formed using the attention-derived question features.
- Answer representation: A Bi-LSTM followed by max pooling produces the final representation used for answer prediction.Pooling is applied across the temporal dimension.
3.8 Answer Reasoning and Loss Function
The model uses task-specific answer reasoning for multiple-choice, open-ended, and counting questions, with losses and prediction rules tailored to each type.
- Multiple-choice question: Multiple-choice answers are scored after interacting each answer feature with visual features, and the highest-scoring choice is selected.Each choice is independently embedded and produces a weighted answer feature before scoring.
- Open-ended question: Open-ended answers are selected from a predefined candidate-word set using a fully connected layer and softmax.The model is trained with cross-entropy loss and predicts the word with the highest score.
- Counting question: Counting answers are predicted as numbers from 0 to 10 using a fully connected layer and mean square error loss.At test time, predictions are rounded to the nearest integer and clipped to the 0–10 range.
4 Experiments
Experiments evaluate L-GCN on three video QA benchmarks and compare its performance with state-of-the-art methods and ablated variants. Results favor object-based graph reasoning, location features, and a two-layer graph with five detected objects.
- Datasets and evaluation: L-GCN is evaluated on TGIF-QA, Youtube2Text-QA, and MSVD-QA using MSE for TGIF-QA Count and accuracy for other tasks.TGIF-QA contains Action, Transition, FrameQA, and Count tasks; the other datasets include open-ended and multiple-choice question types.
- Comparison with state-of-the-art results: L-GCN achieves the best performance on all four TGIF-QA tasks, despite competing methods using additional C3D and optical-flow features.The authors attribute this result to object graphs capturing object-object interactions for reasoning.
- Comparison with state-of-the-art results: 53.2% accuracy versus 29.4% improves L-GCN over baselines on Youtube2Text-QA open-ended who questions.L-GCN also achieves better overall accuracy for both multiple-choice and open-ended questions, while the others category represents only 2% of QA pairs.
- Comparison with state-of-the-art results: L-GCN achieves the most promising overall accuracy on MSVD-QA compared with ST-VQA, Co-Mem, AMU, and HME.The comparison targets non-trivial video question-answering scenarios.
- Ablation study: Adding object features, GCNs, and location features progressively improves TGIF-QA performance, whereas FC or LSTM replacements provide minor gains or performance drops.Using both spatial and temporal location information performs better than using either type alone, especially for Transition and Count.
- Ablation study: Two-layer GCNs and five detected objects per frame perform best on three TGIF-QA tasks.Two detected objects may omit important objects, while too many objects can hurt performance because many questions concern only a few salient objects.
- Qualitative analysis: The similarity visualization shows that detected salient objects help exclude irrelevant background and that the graph captures cross-frame and semantic relationships.Examples include airplane-boy and man-motorcycle interactions associated with the questions.
5 Conclusion
The paper proposes a location-aware graph for modeling relationships between detected objects in video question answering. L-GCN excludes irrelevant background content, incorporates spatial and temporal event locations, and outperforms state-of-the-art techniques on three benchmark datasets.
- L-GCN models relationships between detected objects with a location-aware graph for video question answering.
- The method explicitly removes influences from irrelevant background content compared with existing spatio-temporal attention mechanisms.
- Spatial and temporal event-location awareness supports predicting the correct answer.
- L-GCN outperforms state-of-the-art techniques on three benchmark datasets.