PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Temporal Transformer for Dynamic Scene Graph Generation

Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, Michael Ying Yang

arXiv:2107.12309v2cs.CV

TL;DR

Dynamic scene graph generation must model changing object relationships and temporal dependencies across video frames. STTran combines a spatial encoder with a temporal decoder to perform this task, achieving state-of-the-art results on Action Genome and showing positive effects from temporal context.

  • Problem

    Dynamic scene graph generation is challenging because object relationships vary across frames and static image-based methods do not directly model temporal dependencies.

  • Method

    STTran uses a spatial encoder to extract within-frame context and a temporal decoder to capture dependencies between frames for relationship prediction.

  • Results

    STTran achieves state-of-the-art results on the Action Genome dynamic scene graph generation task.

  • Takeaways & Limitations

    Temporal context has a positive effect on relationship prediction, with ablations supporting the contribution of the temporal decoder and learned frame encoding.

Abstract

from arXiv · show

Dynamic scene graph generation aims at generating a scene graph of the given video. Compared to the task of scene graph generation from images, it is more challenging because of the dynamic relationships between objects and the temporal dependencies between frames allowing for a richer semantic interpretation. In this paper, we propose Spatial-temporal Transformer (STTran), a neural network that consists of two core modules: (1) a spatial encoder that takes an input frame to extract spatial context and reason about the visual relationships within a frame, and (2) a temporal decoder which takes the output of the spatial encoder as input in order to capture the temporal dependencies between frames and infer the dynamic relationships. Furthermore, STTran is flexible to take varying lengths of videos as input without clipping, which is especially important for long videos. Our method is validated on the benchmark dataset Action Genome (AG). The experimental results demonstrate the superior performance of our method in terms of dynamic scene graphs. Moreover, a set of ablative studies is conducted and the effect of each proposed module is justified. Code available at: https://github.com/yrcong/STTran.

1. Introduction

Dynamic scene graph generation extends image-based scene graphs to videos, where object relationships vary across frames and require temporal dependencies. STTran addresses this challenge by combining spatial context and temporal reasoning, achieving state-of-the-art results on Action Genome.

  • Scene graphs represent objects as nodes and relationships as edges, supporting tasks including retrieval, detection, segmentation, and vision-language understanding.
  • Dynamic scene graph generation is harder than image-based generation because object consistency and relationships can change across video frames.
  • Action recognition treats activities as monolithic events, whereas decomposing activities into object relationships can represent their evolving structure.
  • STTran encodes spatial context within frames and decodes relationship representations using temporal dependencies across frames.
  • STTran applies multi-label relationship prediction and introduces a strategy for generating dynamic scene graphs with confident predictions.
  • STTran achieves state-of-the-art results on the Action Genome benchmark.

2. Related Work

Related work spans image-based scene graph generation, dynamic scene graphs for videos, Transformer models, and established spatial-temporal video architectures. These lines of research provide the context for modeling object relationships across space and time.

  • Scene Graph Generation: Image scene graphs represent object interactions and support applications including image retrieval, captioning, visual question answering, and image generation.
  • Scene Graph Generation: Image-based methods improve scene graph quality through spatial context, graph structure, optimization, reinforcement learning, semi-supervised training, and contrastive loss.
  • Scene Graph Generation: Most prior image-based methods use single-label edge prediction, limiting their direct applicability to videos with multiple relationships over time.
  • Scene Graph Generation: Dynamic scene graph work extends scene graphs to videos by decomposing activities and using them to improve video action recognition.
  • Transformer for Computer Vision: Transformer variants have been applied successfully to language modeling and vision-language tasks such as image captioning and visual question answering.
  • Spatial-Temporal Networks: Spatial-temporal video architectures include RNN/LSTM, 3D ConvNet, two-stream CNN, I3D, and non-local self-attention approaches.

3. Method

STTran uses a Transformer architecture specialized for dynamic scene graphs: a spatial encoder refines relationships within each frame, while a temporal decoder models dependencies across frames. Multi-label prediction and Semi Constraint graph generation accommodate co-occurring relationships.

  • Architecture: STTran models a dynamic scene graph as a static graph augmented with a temporal index and uses Transformer properties to integrate spatial-temporal context.
  • Relationship Representation: The relationship representation combines visual appearance, spatial information, and semantic embeddings for each subject-object proposal pair.
  • Spatial Encoder: The spatial encoder applies self-attention within a single frame to refine relationship representations before passing them to the temporal decoder.
  • Frame Encoding: Learned frame encodings inject temporal position, and their fixed window size keeps encoding length independent of video length.
  • Temporal Decoder: The temporal decoder uses sliding windows over spatially contextualized representations to pass information between adjacent frames while limiting interference from distant frames.
  • Prediction and Graph Generation: A multi-label margin loss supports multiple predicates for a subject-object pair, while Semi Constraint accepts predicates whose confidence exceeds a threshold.

4. Experiments

Experiments on Action Genome evaluate STTran across standard scene-graph tasks, constraint strategies, threshold settings, and temporal-dependency ablations. STTran consistently outperforms image-based baselines, while sequence disruption and removing temporal modeling reduce performance.

  • Experimental Setup: Action Genome provides frame-level labels for 234,253 frames, with 476,229 bounding boxes across 35 object classes and 1,715,568 instances across 25 relationship classes.The dataset is built on Charades and separates attention, spatial, and contact relationships.
  • Experimental Setup: The evaluation uses PredCLS, SGCLS, and SGDET, with successful object detection requiring at least 0.5 IoU.PredCLS predicts predicates from ground-truth objects, SGCLS classifies ground-truth boxes, and SGDET detects objects while predicting relationships.
  • Quantitative Results: STTran outperforms state-of-the-art image-based methods across all metrics under With Constraint, Semi Constraint, and No Constraint.All methods use the same object detector for a fair comparison.
  • Quantitative Results: Under With Constraint, STTran improves GPS-Net by 1.9% on PredCLS-R@20, 1.0% on SGCLS-R@20, and 1.0% on SGDET-R@20.These results concern predicting the most important relationship between an object pair.
  • Quantitative Results: Under Semi Constraint, STTran improves PredCLS-R@20 by 1.9%, SGCLS-R@20 by 1.5%, and SGDET-R@20 by 0.6%.Semi Constraint allows multiple relationships between a subject-object pair when confidence exceeds a threshold.
  • Threshold Analysis: STTran consistently outperforms three comparison models across Semi Constraint thresholds from 0.7 to 0.95.The threshold used in the experiments is 0.9; higher thresholds suppress R@20 except in SGDET, where more pair proposals are available.
  • Temporal Dependency Analysis: Reversing one-third of training videos lowers PredCLS-R@20 from 71.8% to 71.0%, while shuffling lowers it to 70.6%.The results support that STTran learns temporal dependencies from frame sequence order.

5. Conclusion

STTran combines spatial encoding within frames with temporal decoding across frames for dynamic scene graph generation. Experiments show temporal context benefits relationship prediction and yield state-of-the-art results on Action Genome.

  • STTran uses an encoder to extract spatial context within each frame and a decoder to capture temporal dependencies between frames.
  • The model applies a multi-label margin loss and introduces a new strategy for generating dynamic scene graphs.
  • Experiments demonstrate that temporal context has a positive effect on relationship prediction.
  • STTran achieves state-of-the-art results for dynamic scene graph generation on the Action Genome dataset.

6. Appendix

The appendix details STTran’s implementation, Action Genome annotations and evaluation comparisons, then examines qualitative behavior and common failure cases. Results highlight temporal and multi-label relationship prediction, while limitations include missed objects, annotation errors, ambiguity and class imbalance.

  • Implementation Details: STTran adds frame encodings to relationship representations so relationships from the same frame receive the same temporal position information.The appendix represents each frame’s relationships as an unordered tensor and adds the same encoding across that frame’s representations.
  • Implementation Details: Object representations combine detector features, semantic embeddings, and position embeddings before classification.The detector provides a 2048-d feature vector; box coordinates produce a 128-d position embedding, and the concatenated representation is projected to 37 classes including background.
  • Action Genome Analysis: Action Genome annotations organize human-object relationships into attention, spatial, and contact types, with multiple spatial or contact labels permitted.The appendix also specifies distinct subject-object ordering for attention, contact, and spatial relationships.
  • Additional Results: The three generation strategies trade off relationship coverage and noise: With Constraint restricts labels, whereas No Constraint can produce substantial noise when few pairs are present.Semi Constraint permits multiple spatial or contact labels above a confidence threshold, while No Constraint selects the top-K relationships regardless of type.
  • Additional Results: STTran outperforms GPS-Net for attention and spatial predicate prediction, while GPS-Net performs better for holding and not contacting.The comparison evaluates the ten most frequent Action Genome relationships using APpred.
  • Failure Cases: Common failures arise from missed small objects, incorrect annotations, human-ambiguous relationships, and predictions of majority relationships over correct minority relationships.The appendix attributes these cases respectively to object detection, labeling, perceptual ambiguity, and imbalanced relationship distributions.
Loading 2107.12309v2…