Source-linked AI summary
Videos as Space-Time Region Graphs
Xiaolong Wang, Abhinav Gupta
TL;DR
The paper addresses why video-recognition systems struggle to recognize actions requiring temporal object-state changes and functional interactions. It represents long-range videos as space-time region graphs with similarity and spatial-temporal edges, then applies Graph Convolutional Networks for reasoning. The approach achieves state-of-the-art results on Charades and Something-Something, including a 4.4% boost on Charades.
Problem
Current video-recognition frameworks struggle to jointly model temporal shape dynamics and human-object or object-object interactions needed for actions such as opening a book.
Method
The model represents long-range videos as space-time region graphs whose object-region nodes use similarity and spatial-temporal relations, then applies Graph Convolutional Networks.
Results
4.4% boost is reported on Charades, and the model achieves state-of-the-art results on both Charades and Something-Something.
Takeaways & Limitations
The framework improves action recognition in complex environments and captures object interactions, pose changes, and actions across sequences.
Abstract
from arXiv · showhide
How do humans recognize the action "opening a book" ? We argue that there are two important cues: modeling temporal shape dynamics and modeling functional relationships between humans and objects. In this paper, we propose to represent videos as space-time region graphs which capture these two important cues. Our graph nodes are defined by the object region proposals from different frames in a long range video. These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects. We perform reasoning on this graph representation via Graph Convolutional Networks. We achieve state-of-the-art results on both Charades and Something-Something datasets. Especially for Charades, we obtain a huge 4.4% gain when our model is applied in complex environments.
1 Introduction
The paper argues that video recognition must model both how object shapes change over time and how humans and objects interact. It represents long-range videos as space-time region graphs and reasons over them with graph convolutions, achieving strong results on challenging action-recognition datasets.
- Motivation: Recognizing actions such as opening a book requires modeling temporal shape changes and human-object or object-object interactions.The book changes from closed to open, while interactions help explain those state changes.
- Limitations of Existing Methods: Existing approaches struggle because frame-level or local-motion methods miss shape dynamics, while whole-scene temporal models overlook relationships between object instances.The paper identifies limitations in two-stream ConvNets, RNNs, and 3D ConvNets.
- Limitations of Existing Methods: Static relationship reasoning also fails to capture temporal interaction dynamics, object-state changes, and their causes and effects.The limitation applies to approaches modeling human-object or object-object interactions in static images.
- Proposed Approach: The proposed space-time region graph connects object regions using appearance-similarity and spatial-temporal proximity relations.Similarity edges capture long-range dependencies, while spatial-temporal edges connect nearby objects and encode interaction and state-change ordering.
- Proposed Approach: Graph Convolutional Networks perform reasoning over the graph to infer actions from long-range video sequences.The model targets videos exceeding 100 frames or 5 seconds and uses region-level nodes.
- Results: 4.4% boost is obtained on Charades, alongside significant improvements over state-of-the-art action-recognition results on Charades and Something-Something.The paper reports state-of-the-art performance and highlights gains in complex environments.
2 Related Work
Prior video-understanding methods learn spatio-temporal appearance features but struggle to model object-instance relationships over long ranges. The paper combines object-level graph reasoning with explicit spatial-temporal ordering and long-range dependencies.
- Video Understanding Models: Two-stream ConvNets use short optical-flow inputs, while RNNs and 3D ConvNets target longer-term information but primarily extract whole-scene features.The related-work discussion contrasts local motion modeling with longer-range scene representations.
- Visual Relationships: Visual-relationship methods improve tasks such as detection and recognition by modeling pairwise human-object or object-object relationships.These relationships have also been applied in visual question answering and intuitive physics.
- Visual Relationships: Non-local Neural Networks reason over pixel-level feature-space locations but do not explicitly encode temporal ordering.The paper contrasts this with object-level graph reasoning and spatial-temporal relations.
- Graphical Models: Graphical-model approaches capture long-range relationships, including CRF-based mean-field inference and feedforward graph neural networks.The paper places its graph-based reasoning within this broader line of work.
- Object Graph Models: Unlike prior object-graph methods focused on local relations, the proposed representation connects object pairs across space and time to encode long-range dependencies.The distinction is made against approaches such as Structural-RNN.
3 Overview
The model converts a long video clip into object-level graph features, propagates information through two relation-specific graphs, and combines the resulting object representation with a global video feature for classification.
- Feature Extraction: More than 5 seconds of video frames are processed by a 3D ConvNet into a T × H × W × d feature map.T denotes time, H × W spatial dimensions, and d the channel number.
- Feature Extraction: A Region Proposal Network extracts object bounding boxes, and RoIAlign produces a d-dimensional feature for each proposal on every feature frame.Each proposal feature is aligned to 7×7×d and max-pooled to 1 × 1 × d.
- Graph Construction: The graph contains N nodes representing object proposals aggregated over T frames.The construction uses the same nodes for two relation-specific sub-graphs.
- Graph Construction: The two sub-graphs encode similarity relations and spatial-temporal relations between object nodes.The graph is decomposed into a similarity graph and a spatial-temporal graph.
- Graph Reasoning: Graph Convolutional Networks propagate information over the object graph while preserving N × d output features.The GCN output remains aligned with the input object-feature dimensionality.
- Video Classification: Average-pooled GCN features are concatenated with average-pooled global video features for video-level classification.Both representations have d dimensions before concatenation.
4 Graph Representations in Videos
The model represents long video clips as space-time region graphs built from object proposals. Similarity edges capture long-range semantic dependencies, while spatial-temporal edges encode nearby-object relations and temporal ordering.
- Video Representation: The video representation begins with 3D ConvNet features extracted from sampled frames of a long clip.The baseline uses a ResNet-50 I3D backbone with 32×224×224 input dimensions.
- Video Representation: Object bounding boxes are generated by a class-agnostic RPN and converted into object features with independently applied RoIAlign.RoIAlign produces 7×7×d features that are max-pooled to 1×1×d dimensions.
- Similarity Graph: The similarity graph connects any pair of object proposals that may represent the same object in different states or correlated objects useful for action recognition.Learned feature transformations model both cross-frame state correlations and relations between different objects.
- Similarity Graph: Similarity affinities are computed from transformed proposal features and row-normalized with softmax to form the adjacency matrix Gsim.The affinity function uses two learned transformations, φ(x)=wx and φ′(x)=w′x.
- Spatial-Temporal Graph: The spatial-temporal graph links overlapping proposals across adjacent frames with directed IoU-weighted edges.Forward and backward graphs encode temporal ordering and provide bidirectional propagation neighborhoods.
5 Convolutions on Graphs
Graph convolutions perform message passing over object nodes using the proposed graph relations. The model separately processes learned similarity relations and fixed spatial-temporal relations before combining their outputs for video classification.
- Graph Convolution: A graph convolution updates each object node from its graph-defined neighbors, producing features that retain the N×d dimensionality.G may be Gsim, Gfront, or Gback, while X and W have dimensions N×d and d×d respectively.
- Graph Convolution: The network applies Layer Normalization and ReLU after each graph-convolution layer before forwarding features to the next layer.Multiple graph-convolution layers can be stacked.
- Graph Convolution: The model relates graph convolution to non-local reasoning while operating on object-level features and adding convolutional processing and residual connections.The first layer applies a convolutional operator to the input, and residual connections extend each GCN layer.
- Multiple Graphs: Directly combining Gsim, Gfront, and Gback in every layer hurts performance relative to using a single similarity graph.The authors attribute this to optimization difficulty from learning Gsim alongside the non-learned spatial-temporal graphs.
- Multiple Graphs: The final architecture uses separate GCN branches for Gsim and for Gfront plus Gback, fusing their results only after the graph convolutions.The summed graph features are combined with global video features for classification.
6 Experiments
Experiments on Charades and Something-Something evaluate the graph-based model, its components, robustness, efficiency, and performance across action conditions. The model improves over I3D and achieves stronger results when actions involve long-term sequences, object interactions, or pose changes.
- Experimental Setup: The model is evaluated on Charades and Something-Something using long video clips and graph-based components built on I3D features.Training uses pretraining and fine-tuning of an I3D backbone before target-dataset evaluation.
- Charades Results: Naive proposal-feature aggregation yields only a 0.3% boost, whereas relation-aware graph reasoning produces substantially larger gains.The spatial-temporal graph reaches 34.2% mAP, the similarity graph reaches 35.0%, and their combination reaches 36.2%.
- Charades Results: 4.4% mAP improvement is achieved on Charades when combining spatial-temporal and similarity graphs in the Joint GCN.The Joint GCN reaches 36.2% mAP, compared with 31.8% for the I3D baseline.
- Robustness and Efficiency: The method remains stable as proposals per frame change, achieving 35.9% mAP with 25 proposals and 36.1% with 100 proposals.With 50 proposals per frame, the model achieves 36.2% mAP.
- Robustness and Efficiency: 3% increase in FLOPs is observed for I3D + Joint GCN, with 158 × 10^9 FLOPs versus 153 × 10^9 for baseline I3D.The paper reports barely observable differences in training and inference time between the baseline and proposed model.
- Error Analysis: The method improves most for actions in sequences, involving objects, or exhibiting high pose variance, while achieving 8%–9% gain at high pose variability.For non-isolated actions, the gain exceeds 5%; the approach also performs slightly better when objects are involved.
- Something-Something Results: On Something-Something, I3D + Joint GCN improves top-1 accuracy by 1.7%, while combining it with Non-local I3D reaches 46.1% top-1 accuracy.The final single-RGB model achieves 45.0% on the official test leaderboard.
7 Conclusions
The paper presents a graph-based network for modeling long-range relationships in videos for action recognition. Its error analysis indicates improved handling of object interactions, pose changes, and actions occurring in sequence, with potential beyond video classification.
- The proposed network models long-range relationships in videos for action recognition.
- Error analysis shows improved handling of object interactions, pose changes, and actions occurring in a sequence.
- The authors identify potential applications beyond video classification, including detection and tracking in videos.