Source-linked AI summary
Action Genome: Actions as Composition of Spatio-temporal Scene Graphs
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, Juan Carlos Niebles
TL;DR
Video action recognition often treats actions as monolithic events, leaving temporal object-relationship structure underexplored. Action Genome addresses this gap with a dataset and spatio-temporal scene-graph representation, improving action recognition, supporting few-shot recognition, and benchmarking scene-graph prediction. The paper reports 44.3% action-recognition performance on Charades versus 42.5% for the prior state of the art, and 42.7% mAP with 10 few-shot examples.
Problem
Computer-vision action recognition usually treats actions as monolithic events, while temporal decompositions into object relationships remain limited.
Method
Action Genome decomposes action segments into temporally changing scene graphs centered on actors, relevant objects, and their pairwise relationships.
Results
The paper improves Charades action recognition from 42.5% to 44.3%, achieves 42.7% mAP with 10 examples, and benchmarks spatio-temporal scene graph prediction.
Takeaways & Limitations
The representation supports more decomposable video understanding across action recognition, few-shot recognition, and spatio-temporal scene graph prediction.
Takeaways & Limitations
The paper identifies future work around the rich hierarchy of events and additional research directions enabled by Action Genome.
Abstract
from arXiv · showhide
Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However in Computer Vision, few explorations on representations encoding event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.
1. Introduction
Action Genome addresses the limited temporal decomposition of video actions by representing changing human-object interactions as spatio-temporal scene graphs. It uses this representation for action recognition, few-shot recognition, and scene graph prediction.
- Motivation: Action recognition models commonly treat long video actions as monolithic events rather than explicitly decomposing object interactions.Temporal event decompositions have received comparatively little attention in computer vision.
- Motivation: Hierarchical event structures can support predicting future or past scene graphs and learning actions from few examples.The paper illustrates this with a person transitioning from next to, to in front of, to sitting on a sofa.
- Representation: Action Genome decomposes actions into spatio-temporal scene graphs whose relationships evolve across frames.For sitting on a sofa, the sequence progresses from next to, to in front of, to sitting on the sofa.
- Representation: 476K object bounding boxes and 1.72M relationships across 234K video frames cover 157 action categories in Action Genome.The dataset is built upon Charades and provides frame-level structure for action components.
- Results: 44.3% replaces 42.5% on Charades, while oracle scene graphs reach 60.3% in the reported action-recognition experiment.The method extends an existing action-recognition model with spatio-temporal scene graphs as feature banks.
- Results: 42.7% mAP is achieved with 10 training examples in few-shot action recognition, alongside a benchmark for spatio-temporal scene graph prediction.The paper also evaluates existing scene graph models using video-specific metrics.
2. Related work
Prior video datasets and models largely emphasize clip-level action labels, while structured scene graphs have mainly supported image understanding. Action Genome extends structured annotation toward temporal action decomposition and complements existing few-shot strategies.
- Action recognition: Many action-recognition backbones learn video representations for classification, but common datasets provide single action labels for clips.These annotations generally treat actions as monolithic events.
- Action recognition: Existing datasets add actor localization, concurrent actions, or interacted-object annotations, but do not generally track object relationships throughout actions.Action Genome is positioned as a complementary video dataset with spatio-temporal scene graph labels.
- Scene graphs: Image scene graphs represent objects as nodes connected by pairwise relationships and have supported captioning, retrieval, question answering, and generation.The paper transfers this structured representation from static images toward temporal events.
- Few-shot prediction: Few-shot prediction methods either transfer knowledge from frequent categories or learn invariances and decompositions for novel categories.Action Genome relates to the second strategy through its compositional event representation.
3. Action Genome
Action Genome adapts scene graphs into an action-focused, temporally changing representation centered on the actor’s interactions with relevant objects. Its annotations organize relationship types and show broad object-relationship coverage with reduced dataset bias.
- Representation: Action Genome represents actions as temporally changing action-object units involving objects and their pairwise interactions with the actor.Unlike Visual Genome, it annotates action segments and objects involved in those actions rather than densely representing entire scenes.
- Dataset statistics: Most relationships have at least 1K instances and objects have at least 10K instances in the dataset.The occurrence distributions are shown separately for relationships and objects.
- Relationship types: Attention relationships indicate what a person looks at, spatial relationships describe relative layout, and contact relationships describe how the person contacts an object.Contact changes can mark actions such as picking up a book.
- Annotation framework: Five frames are uniformly sampled across each action interval, with annotators drawing boxes and selecting relationships using neighboring clips for disambiguation.Action-oriented sampling produces labels for each action, including concurrent actions.
- Dataset statistics: Objects are densely interconnected across attention, spatial, and contact relationships, and Action Genome does not show the strong object-category relationship bias reported for Visual Genome.Most objects are described as fairly evenly involved across the three relationship types.
- Relationship types: Action Genome decomposes clip-level verbs into frame-level relationship sequences, such as lying on, sitting on, and not contacting a bed.This separates Charades action labels from the finer-grained relationship annotations.
4. Method
SGFB predicts frame-level spatio-temporal scene graphs, encodes them as feature banks, and combines them with 3D CNN features for action recognition.
- Scene graph prediction: SGFB jointly learns scene graphs and action labels from video sequences.Its scene graph predictor generates objects and actor-centered attention, spatial, and contact relationships for each frame.
- Scene graph prediction: The scene graph predictor outputs confidence scores for predicted objects and relationships.These scores are used to construct the object-relationship confidence matrix.
- Scene graph feature banks: Each frame’s object-relationship confidence scores form a flattened feature vector.The confidence matrix has dimensions |O| × |R|, with Action Genome using 35 object classes and 25 relationship classes.
- Scene graph feature banks: SGFB aggregates scene graph features across frames using feature-bank operators such as mean/max pooling or non-local blocks.The resulting contextual representation is combined with short-term 3D CNN features before predicting video action labels.
5. Experiments
Experiments evaluate Action Genome for action recognition, few-shot recognition, and spatio-temporal scene graph prediction, showing gains from scene graph representations and remaining benchmark gaps.
- Action recognition on Charades: 16% increase on mAP is observed with SGFB Oracle using ground-truth objects and visual relationships.The reported oracle result uses annotations for only the few frames with ground-truth scene graphs and is not an upper bound.
- Few-shot action recognition: SGFB outperforms LFB in all 1-, 5-, and 10-shot Charades experiments.SGFB Oracle provides a 10.9% 10-shot mAP improvement when ground-truth scene graphs are available.
- Spatio-temporal scene graph prediction: Existing image-based scene graph models leave significant room for improvement on the video-based prediction task.These models were designed for single-frame inputs and do not consider the entire video sequence.
- Spatio-temporal scene graph prediction: RelDN marginally outperforms IMP on spatio-temporal scene graph prediction, while IMP outperforms numerous newer methods.The small PredCls–SGCls performance gap suggests similar behavior across those evaluation settings.
6. Future work
Action Genome supports future work on grounded localization, explainable action models, and video generation from spatio-temporal scene graphs.
- Research directions: The representation opens additional research directions beyond scene graph prediction and compositional action recognition.The authors describe Action Genome as enabling future studies using its hierarchy of events.
- Spatio-temporal action localization: Action Genome can support localization of both actors and interacting objects in spatio-temporal action localization.It also permits weakly supervised variants using action labels while localizing actors and objects.
- Explainable action models: Frame-level labels of objects being looked at or interacted with can support explainable action models.These labels provide an attention signal for training explainable visual models.
7. Conclusion
Action Genome decomposes actions into spatio-temporal scene graphs that represent changing objects and relationships, supporting action recognition, few-shot recognition, and scene graph prediction.
- Conclusion: Action Genome represents how objects and their relationships change as an action occurs.The paper uses this representation to improve state-of-the-art action recognition and few-shot action recognition, and to benchmark spatio-temporal scene graph prediction.