Source-linked AI summary
Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks
Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, Trevor Darrell
TL;DR
The paper studies whether action-recognition models can generalize to unseen verb-object combinations, a capability humans exhibit. It introduces a spatial-temporal interaction model using object grounding and geometric relations, and reports stronger performance across compositional settings than appearance-based baselines.
Problem
Current action-recognition models struggle to capture compositionality when tested on verb-object combinations unseen during training.
Method
The paper introduces a Spatial-Temporal Interaction Network that models subject-object interaction dynamics from geometric configurations and object grounding annotations.
Results
The STIN approach outperforms baselines on the proposed compositional and few-shot settings, with detection boxes yielding 5.6% higher performance than I3D.
Takeaways & Limitations
Modeling interaction dynamics and object configurations supports generalization across different object appearances and unseen action combinations.
Takeaways & Limitations
The approach is constrained for actions without clearly defined spatial object-agent interactions, where it relies on baseline scene-level spatio-temporal representations.
Abstract
from arXiv · showhide
Human action is naturally compositional: humans can easily recognize and perform actions with objects that are different from those used in training demonstrations. In this paper, we study the compositionality of action by looking into the dynamics of subject-object interactions. We propose a novel model which can explicitly reason about the geometric relations between constituent objects and an agent performing an action. To train our model, we collect dense object box annotations on the Something-Something dataset. We propose a novel compositional action recognition task where the training combinations of verbs and nouns do not overlap with the test set. The novel aspects of our model are applicable to activities with prominent object interaction dynamics and to objects which can be tracked using state-of-the-art approaches; for activities without clearly defined spatial object-agent interactions, we rely on baseline scene-level spatio-temporal representations. We show the effectiveness of our approach not only on the proposed compositional action recognition task, but also in a few-shot compositional setting which requires the model to generalize across both object appearance and action category.
1. Introduction
The paper frames action recognition as compositional: actions can be identified from changing subject-object geometry even with unseen object combinations. It introduces Something-Else and STIN to model these relations explicitly.
- Humans recognize “taking something out of something” from changing relative hand-object positions, including with previously unseen objects.
- Existing scene-level and I3D-based models struggle with compositionality because they emphasize spatial appearance rather than object-specific geometric transformations.
- STIN builds sparse, semantically grounded subject-object graphs from detected and tracked boxes, then reasons over spatial and temporal interactions.
- Something-Else extends Something-Something with dense object-box annotations and a split where action categories use unseen object combinations at test time.
- The paper evaluates compositional recognition in standard and few-shot settings and reports gains over appearance-based models.
2. Related Work
Related work spans deep spatio-temporal action models, visual compositionality, relational reasoning, and few-shot recognition in images and videos.
- Action-recognition systems include recurrent networks with 2D convolutions and 3D ConvNets that learn temporal representations.
- Visual compositionality research has studied assembling visual primitives and concepts, including pictorial structures and deformable part-based models.
- Visual-relationship methods use object and motion features or deep relational reasoning to model interactions among entities.
- Few-shot recognition research covers both image and video settings, including comparisons between simple baselines and meta-learning approaches.
3. Spatial-Temporal Interaction Networks
STIN represents hands and objects with tracked, object-centric features, reasons spatially within frames and temporally across tracklets, and can combine these interactions with scene appearance.
- STIN uses a generic detector and tracker to construct sparse graphs containing hand and constituent-object nodes.
- 3.1. Object-centric Representation: Object representations combine box center, height, and width with learned subject, object, or null identity embeddings.
- 3.1. Object-centric Representation: Combining coordinate and identity features significantly improves compositional generalization across verb-noun combinations.
- 3.2. Spatial-temporal interaction reasoning: Spatial reasoning aggregates other objects in each frame with the current object, while temporal reasoning links features along the same tracklet.
- 3.2. Spatial-temporal interaction reasoning: Tracklet information is aggregated for classification, using either averaging or non-local blocks that encode pairwise trajectory relationships.
- 3.2. Spatial-temporal interaction reasoning: Scene-level 3D ConvNet appearance features can be combined with interaction reasoning, especially for actions lacking prominent inter-object dynamics.
4. The Something-Else Task
The Something-Else task reorganizes Something-Something V2 with annotations and compositional splits that test recognition across unseen verb–object combinations. It also defines a few-shot setting for generalization to novel action categories.
- Task and dataset: The Something-Else task adds new annotations and splits to Something-Something V2 for compositional action recognition.The dataset contains 174 interaction categories and 12,554 object descriptions; the annotations include hand and object bounding boxes.
- Compositional Action Recognition: Training and testing use the same actions with different object categories, so tested verb–noun combinations are unseen during training.The compositional split is formed by pairing action group 1 with object group A and action group 2 with object group B for training, then flipping these pairings for validation.
- Compositional Action Recognition: The compositional split divides frequent object categories and action categories into disjoint groups while preserving action super-class assignments.Frequent objects appear in more than 100 videos, and related action categories are assigned to the same group.
- Task statistics: Table 1 compares task variants and reports their dataset statistics, including few-shot and n-shot settings.The table caption identifies FS as few-shot and n-S as n-shot.
- Few-shot Compositional Action Recognition: The few-shot compositional setting evaluates generalization to 86 novel action classes after training on 88 base classes.The protocol samples k labeled examples per novel category while retaining the remaining novel videos for evaluation.
5. Experiments
Experiments evaluate STIN on standard, compositional, few-shot, and one-object settings, using ground-truth or detected boxes and combinations with I3D and STRG. STIN improves recognition especially when generalization depends on object interactions, object appearance, or compositional transfer.
- Experimental setup: STIN uses bounding-box coordinates, object identity embeddings, and non-local aggregation, with variants combined with I3D or STRG.The experiments use either ground-truth or detector-generated boxes; I3D and STRG provide appearance or graph-based comparison models.
- Original Something-Something split: 5% improvement over the I3D baseline is achieved by combining STIN with I3D on the original Something-Something V2 split.The STIN model with ground-truth annotations is also comparable with TRN using only coordinates.
- Compositional action recognition: 11.3% improvement over I3D is achieved by the ensemble on compositional action recognition, while joint combination yields 7.8% improvement over the baseline.On the compositional split, I3D top-1 accuracy drops by almost 15% relative to the shuffled split, and OIE improves STIN by 4.3%.
- Compositional action recognition: 3.9% relative improvement over STRG is achieved by combining STRG with STIN + OIE + NL, showing complementarity between the methods.With detected boxes, OIE boosts STIN by 3.1%, while combining I3D with STIN improves over I3D by 1.4%.
- Few-shot compositional action recognition: Few-shot ensembles improve over I3D by 12.2% in 5-shot and 13.9% in 10-shot learning with ground-truth boxes.With detection boxes, ensemble gains are 5.5% in 5-shot and 5.9% in 10-shot learning; STRG combinations improve by 4.3% and 4.7%.
- One-object training: 5.6% better performance than I3D is achieved with detection boxes in the one-object training experiment, while ground-truth boxes almost double I3D performance.Training uses only videos involving the object category “box,” testing generalization across different object appearances.
- Category analysis: STIN performs best on actions describing object movement, such as “put something” and “take something,” but trails I3D on intrinsic-property changes such as “poking” and “tearing”.The category analysis links the strongest differences to whether actions are represented by geometric-relational transformations.
6. Conclusion
The paper proposes STIN, a sparse semantically grounded subject-object graph model, and validates it on compositional and few-shot settings in Something-Else. STIN models object interaction dynamics and outperforms all baselines.
- STIN uses sparse, semantically grounded subject-object graph representations for action recognition.The model is motivated by appearance bias in current activity recognition models.
- The approach is validated on novel compositional and few-shot settings in the Something-Else dataset.Training uses new constituent object grounding annotations.
- STIN models the interaction dynamics of objects composed in an action and outperforms all baselines.