Source-linked AI summary
Attend and Interact: Higher-Order Object Interactions for Video Understanding
Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan AlRegib, Hans Peter Graf
TL;DR
Fine-grained video understanding must capture interactions among multiple objects, while pairwise modeling over many video frames is computationally difficult. The paper introduces efficient recurrent higher-order interaction modules with attention and temporal reasoning, and reports state-of-the-art results for action recognition and video captioning at a maximum 1 FPS. The gains are stronger on Kinetics than on ActivityNet Captions, and the comparison is subject to feature and proposal differences.
Problem
Existing video-understanding and visual-relationship methods often use single-object or pairwise representations that do not efficiently capture complex multi-object interactions.
Method
The paper uses recurrent modules with dot-product attention and temporal reasoning to discover higher-order interactions among selected object subgroups, combining them with coarse video context.
Results
SINet and SINet-Caption achieve state-of-the-art performance on Kinetics and ActivityNet Captions using RGB videos sampled at maximum 1 FPS.
Takeaways & Limitations
Modeling higher-order object relationships provides high accuracy at low computation costs across the paper’s two video-understanding tasks.
Takeaways & Limitations
Higher-order interactions contribute less substantially on ActivityNet Captions than on Kinetics, and comparison with LSTM-A3 is constrained by different temporal proposals and feature representations.
Abstract
from arXiv · showhide
Human actions often involve complex interactions across several inter-related objects in the scene. However, existing approaches to fine-grained video understanding or visual relationship detection often rely on single object representation or pairwise object relationships. Furthermore, learning interactions across multiple objects in hundreds of frames for video is computationally infeasible and performance may suffer since a large combinatorial space has to be modeled. In this paper, we propose to efficiently learn higher-order interactions between arbitrary subgroups of objects for fine-grained video understanding. We demonstrate that modeling object interactions significantly improves accuracy for both action recognition and video captioning, while saving more than 3-times the computation over traditional pairwise relationships. The proposed method is validated on two large-scale datasets: Kinetics and ActivityNet Captions. Our SINet and SINet-Caption achieve state-of-the-art performances on both datasets even though the videos are sampled at a maximum of 1 FPS. To the best of our knowledge, this is the first work modeling object interactions on open domain large-scale video datasets, and we additionally model higher-order object interactions which improves the performance with low computational costs.
1. Introduction
The paper targets fine-grained video understanding by modeling object relationships and interactions that coarse scene representations and existing relationship methods do not adequately capture. It introduces an efficient recurrent approach for higher-order interactions and reports state-of-the-art results on action recognition and video captioning with videos sampled at up to 1 FPS.
- Motivation: Existing video methods often use coarse scene representations or 3D convolutions, while attention-based captioning methods also do not model object interactions.These approaches miss fine-grained relationships among multiple objects in a scene.
- Motivation: Open-domain video makes relationship modeling difficult because many objects appear across time, creating substantial temporal and computational demands.The paper identifies adapting image-based relationship techniques to video as an unresolved challenge.
- Approach: The proposed recurrent module dynamically discovers higher-order object interactions using efficient dot-product attention combined with temporal reasoning.It is designed for fine-grained video understanding across open-domain tasks.
- Results: SINet and SINet-Caption achieve state-of-the-art performance on Kinetics action recognition and ActivityNet Captions while using RGB frames sampled at maximum 1 FPS.The method combines coarse- and fine-grained information for both tasks.
2. Related work
Prior work primarily aggregates compact video representations, detects relationships in images, or models limited video interactions. This paper instead targets higher-order object interactions while addressing the inefficiency of enumerating object pairs in videos.
- Higher-order interactions: The proposed approach models higher-order interactions among arbitrary subgroups of objects instead of restricting relationships to pairs.Its formulation includes pairwise and triplet relationships as special cases when attention selects a single object.
- Video understanding: Earlier action-recognition models using region proposals extract object features but do not learn interactions between objects in the scene.The paper distinguishes its approach by combining coarse context with fine-grained higher-order interactions.
- Visual relationships: Image relationship methods use scene graphs, separate object and interaction branches, or pairwise object combinations to detect visual relationships.These methods provide precedents for explicit relationship modeling but are mainly developed for images.
- Visual relationships: Enumerating all object pairs becomes inefficient when scenes contain many objects, making conventional relationship methods difficult to apply to video.The paper frames this combinatorial cost as a central obstacle for video-domain relationship modeling.
3. Model
The model combines coarse video context with recurrent, higher-order object interactions to support fine-grained video understanding. It uses attention to select object groups over time, then models their interactions efficiently with temporal recurrence.
- The model jointly exploits high-level video representations and explicit higher-order object interactions for fine-grained video understanding.
- 3.1.2 Fine-grained higher-order object interactions: Unlike pairwise modeling, the method learns interactions among arbitrary object subgroups, with pairwise and triplet relationships as special cases.
- 3.1.2 Fine-grained higher-order object interactions: Recurrent HOI dynamically selects important object candidates, concatenates their combinations into higher-order interactions, and processes them recurrently for temporal reasoning.
- 3.1.2 Fine-grained higher-order object interactions: The number of selected groups K depends on scene complexity and task requirements, while dynamically selecting K remains future work.
- 3.1.2 Fine-grained higher-order object interactions: Attention uses projected objects, current image context, and previous object interactions to select groups, while dot-product attention incorporates inter-object relationships during selection.
- 3.1.2 Fine-grained higher-order object interactions: The selected interactions are aggregated through an LSTM, whose hidden representation encodes object interactions across the video sequence.
4. Datasets and Implementations
The evaluation uses Kinetics for action recognition and ActivityNet Captions for video captioning, with videos sampled at low temporal rates and object features extracted from detected regions. Training and feature extraction combine fine-tuned image representations with ROI-based object representations.
- Datasets: Kinetics contains 400 action classes and approximately 300k clips, and videos are sampled at 1 FPS rather than 25 FPS.The dataset includes human-object and human-human interactions.
- Datasets: ActivityNet Captions contains 20k videos, 849 video hours, and 100K descriptions for fine-grained event captioning.The setup uses ground-truth temporal segments rather than evaluating temporal proposal identification.
- Implementation Details: Image features come from a ResNeXt-101 fine-tuned on Kinetics, producing 2048-dimensional vectors from center-cropped frames.Training uses SGD with Nesterov momentum and standard cropping and flipping augmentation.
- Implementation Details: Object features are extracted from ROIs generated by a Deformable R-FCN pretrained on MS-COCO with a ResNet-101 backbone.Non-maximum suppression uses an IoU threshold of 0.2, and the method balances image and object features.
- Implementation Details: SINet and SINet-Caption are trained with Adam using learning rates of 1e−5 and 1e−3, respectively, with automatic 10x reductions when validation loss saturates.Batch sizes are 64 for Kinetics and 32 for ActivityNet Captions.
5. Evaluation
Experiments show that temporal attention and higher-order object interactions improve action recognition and captioning, while attentive selection avoids the substantial cost of exhaustive pairwise modeling. Comparisons are qualified by sampling rates, feature choices, and ground-truth temporal proposals.
- 5.1. Action recognition on Kinetics: Temporal SDP-Attention outperforms traditional LSTM aggregation and performs comparably with a 3D ConvNet using a much higher sampling rate.The comparison is reported on Kinetics with RGB videos sampled at 1 FPS.
- 5.1. Action recognition on Kinetics: 74.2% top-1 and 91.7% top-5 accuracy are achieved on Kinetics with dynamically discovered higher-order object interactions.This exceeds the 73.1% top-1 accuracy of direct image-and-object feature combination, while dot-product selection outperforms α-attention.
- 5.1. Action recognition on Kinetics: Attentive selection improves over pairwise or triplet interactions while saving significant computation time.Reducing the maximum objects per frame from 30 to 15 produces no substantial accuracy difference, indicating that the top 15 ROI-scored objects suffice.
- Scope: The study focuses on object interactions and treats optical flow and audio as complementary modalities rather than integrating them.These modalities are acknowledged as possible sources of further action-recognition improvement.
- 5.2. Video captioning on ActivityNet Captions: The method reports better validation-set results across nearly all language metrics than LSTM-A3 but does not claim superiority because the methods use different proposals and feature representations.The comparison includes METEOR, ROUGE-L, CIDEr-D, and BLEU scores.
- 5.2. Video captioning on ActivityNet Captions: On ActivityNet Captions, SINet-Caption improves across nearly all evaluation metrics after adding its efficient object-interaction selection module.The method uses ground-truth temporal proposals, and co-attention effects vary across the two validation sets.
6. Conclusion
The paper presents an efficient approach for discovering higher-order object interactions in fine-grained video understanding. Across action recognition and video captioning, it reports high accuracy at low computation using RGB videos sampled at no more than 1 FPS.
- Conclusion: The proposed approach discovers higher-order object interactions with computational efficiency.This is the paper’s central methodological contribution.
- Conclusion: Learning higher-order object relationships provides high accuracy over existing methods at low computation costs on large-scale action-recognition and captioning datasets.The conclusion spans both evaluated tasks.
- Conclusion: The method achieves state-of-the-art performance on both tasks using only RGB videos sampled at maximum 1 FPS.The stated result covers action recognition and video captioning.
7. Supplementary
The supplementary analyses show that SINet focuses on fine-grained object interactions for action recognition and captioning, improving ambiguous-class recognition while revealing weaker gains and annotation challenges on ActivityNet Captions.
- 7.2. Qualitative analysis on ActivityNet Captions: SINet-Caption distinguishes interactions involving a shared object, such as riding horses versus playing polo, rather than defaulting to one horse-related caption.The qualitative analysis reports successful distinction among human–horse interactions.
- 7.2.2 Discussion on ActivityNet Captions: ActivityNet captions can contain semantically different annotations for similar video content, which may cause SINet-Caption to focus on one described interaction instead of another.The paper suggests associating differing captions with distinct region sequences as future work.
- 7.3. Performance improvement analysis on Kinetics: 136/400 classes improved by more than 5% and 46 by more than 10% in top-1 accuracy over baseline with SINet (K = 3).The improvements particularly affect closely related action classes.
- 7.3. Performance improvement analysis on Kinetics: 21% top-1 improvement on washing hair and 16% on getting a haircut demonstrate gains among visually similar hair-related actions.The analyzed group contains seven ambiguous hair actions.
- 7.3. Performance improvement analysis on Kinetics: 16%, 14%, and 13% improvements were obtained for juggling fire, juggling balls, and contact juggling, respectively.
- 7.4. ActivityNet Captions on 1st and 2nd val set: Combining coarse- and fine-grained representations improves all ActivityNet Captions evaluation metrics, although higher-order interactions yield relatively minor gains versus Kinetics.Interaction-based captioning consistently outperforms coarse-grained image representation, while co-attention differs across validation sets.