Source-linked AI summary
Object Level Visual Reasoning in Videos
Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, Greg Mori
TL;DR
Fine-grained activity recognition needs models that understand detailed interactions between actors and objects, beyond global motion, object classes, and scene context. The paper performs spatiotemporal reasoning over semantically meaningful detected objects and reports state-of-the-art results on three datasets. Its conclusion emphasizes object-level interactions and temporal dependencies as central to the approach.
Problem
Fine-grained video activity recognition requires detailed understanding of interactions between actors and objects beyond global motion, object classes, and scene context.
Method
The model integrates object detection with an Object Relation Network that reasons about semantic object instances and their relationships across space and time.
Results
The method achieves state-of-the-art results on Twenty-BN Something-Something, VLOG, and EPIC Kitchens.
Takeaways & Limitations
Reasoning over semantically defined objects and temporal relationships supports activity recognition across three difficult video datasets.
Takeaways & Limitations
Reported failure cases arise from errors in object mask prediction or in the Object Relation Network itself.
Abstract
from arXiv · showhide
Human activity recognition is typically addressed by detecting key concepts like global and local motion, features related to object classes present in the scene, as well as features related to the global context. The next open challenges in activity recognition require a level of understanding that pushes beyond this and call for models with capabilities for fine distinction and detailed comprehension of interactions between actors and objects in a scene. We propose a model capable of learning to reason about semantically meaningful spatiotemporal interactions in videos. The key to our approach is a choice of performing this reasoning at the object level through the integration of state of the art object detection networks. This allows the model to learn detailed spatial interactions that exist at a semantic, object-interaction relevant level. We evaluate our method on three standard datasets (Twenty-BN Something-Something, VLOG and EPIC Kitchens) and achieve state of the art results on all of them. Finally, we show visualizations of the interactions learned by the model, which illustrate object classes and their interactions corresponding to different activity classes.
1 Introduction
The paper targets video understanding beyond object and scene recognition, focusing on reasoning about semantic object interactions and their temporal effects. It proposes object-level relational reasoning to connect detected objects with activity recognition.
- Fine-grained activity recognition requires understanding the manner, order, and effects of human interactions with scene elements.
- Human visual reasoning can infer complex activities between frames, whereas this capability remains unattainable for contemporary computer vision algorithms.
- The approach uses time as an explicit causal signal to identify object-level relations, such as an earlier knife action affecting a later chopped-carrot state.
- Object detection supplies semantic object instances, but activity understanding must reason about which objects interacted, how, and with what results as scene structure changes.
- The proposed Object Relation Network reasons over detected object instances across space and time to infer pairwise relationships at different moments.
2 Related work
Prior work spans spatiotemporal architectures, attention, pose and semantic-object models, and relational reasoning, but challenging fine-grained videos still require long temporal dependencies and human-object interaction understanding.
- Video recognition: Deep video models use temporal pooling, 3D or separable convolutions, and recurrent networks to aggregate information across frames.
- Video recognition: Two-stream recognition combines RGB and optical-flow inputs but relies on relatively small-scale optical-flow computations.
- Attention and structured representations: Pose-based and Structural RNN approaches exploit human or semantic-object structure, while the latter integrates object and relationship features for activity recognition.
- Attention and structured representations: Attention models focus selectively on pose joints, feature-map cells, time, or raw-RGB parts without requiring predefined object positions.
- Motivation: Raw-video aggregation methods fail on challenging fine-grained datasets that require long temporal dependencies and human-object interaction modeling.
- Relational reasoning: Relational reasoning includes Interaction Networks, Relation Networks, and unsupervised object-interaction discovery across visual and physical-system tasks.
- Relational reasoning: Unlike feature-map Relation Networks, this work treats objects as semantically meaningful entities rather than discrete cells, avoiding interactions that grow with feature-map resolution.
- Relational reasoning: Object-set methods such as PointNet and DeepSet provide related permutation-invariant representations, whereas this work directly reasons over object relations.
3 Object-level Visual Reasoning in Space and Time
The model combines global motion and object-level representations to reason about semantically meaningful spatiotemporal interactions for activity recognition. Its ORN processes detected object instances across frames, models pairwise relations recurrently, and integrates object reasoning with activity representations.
- Model overview: The architecture uses activity and object heads that share features, with the activity head extracting global motion and the object head reasoning over predicted object instances.The object head uses detected masks and object-class predictions to form object-level inputs, while the activity head provides global motion context.
- Object Relation Network: The Object Relation Network performs spatiotemporal reasoning between detected object instances, modeling how objects move, appear, disappear, and interact across frames.ORN receives object detection masks spanning categories and temporal occurrences and infers pairwise relationships between objects at different moments.
- Temporal object relations: The model samples object sets from different frames and treats the combined objects as unordered apart from their frame membership, enabling reasoning over temporal changes.The temporal sampling uses a prior frame t′<t, so relations are inferred between objects from two distinct time points.
- Interaction modeling: ORN directly models inter-frame pairwise object interactions, while higher-order interactions are left to the mappings’ shared output space and global recurrent function.The global mapping is recurrent, allowing long-range reasoning over variable-length sequences; pairwise mappings are implemented as an MLP.
- Design choices: Unlike feature-map relational models, ORN defines relationships over semantically meaningful object instances and integrates relations over time with recurrent reasoning.The design explicitly distinguishes object instances from feature-map cells and uses recurrence to aggregate object relations over sequences.
- Object features: Object-instance features are computed from local regions predicted by a mask predictor, using binary masks, object-class distributions, appearance, temporal change, and shape information.The implementation uses Mask R-CNN to detect instances and produce segmentation masks, while the dedicated object head supplies features that include motion from shared spatiotemporal convolutions.
- Recognition: The final recognition stage independently classifies the global-motion representation and the ORN object-reasoning representation.The two representations are hidden states of separate GRUs: h provides global motion context and r provides the ORN reasoning state.
4 Network Architectures and feature dimensions
The model uses joint activity and object heads built on a shared ResNet50-based convolutional backbone, with higher-resolution object features supporting local descriptors. Activity features capture global motion, while object features encode pooled appearance, masks, and semantic classes.
- Backbone: The joint network uses an ImageNet-pretrained ResNet50 with Conv1 and Conv5 blocks inflated to 2.5D convolutions.The 2.5D convolutions separate the temporal dimension while retaining convolutional processing.
- Feature maps: The activity and object heads produce feature maps of dimensions 2048×T×7×7 and 2048×T×14×14, respectively.The object head uses higher spatial resolution for more precise local descriptors.
- Feature representations: Global spatial pooling produces 2048-dimensional activity vectors for a 512-dimensional GRU, while ROI pooling produces 2048-dimensional object vectors.The activity stream summarizes global motion, whereas the object stream extracts local object features.
- Object features: A one-hidden-layer MLP encodes each binary mask into a 100-dimensional mask embedding.The mask embedding is part of the object representation.
- Object features: The object representation has 2229 dimensions because it combines features with information from 80 object classes.The architecture therefore incorporates semantic object-category information alongside visual features.
- Reasoning module: The object reasoning module uses an MLP with two 512-unit hidden layers, while its temporal component is a GRU with a 256-dimensional hidden state.ReLU activations follow each network layer.
5 Training
Training combines activity classification with an auxiliary object-semantic objective, then fine-tunes the full network after initializing the object head. Optimization uses Adam with validation-based early stopping.
- Objective: The loss has two terms: supervised activity-class losses and an auxiliary loss aligning object features with semantic object classes.The activity losses compare predictions from both the activity and object heads with ground-truth activity labels.
- Objective: The auxiliary object loss uses cross-entropy between mask-predictor class labels and class predictions from ROI-pooled object features.This encourages object features to represent both motion and semantic object information without ground-truth object classes.
- Training procedure: Training first optimizes the object head alone and then trains the full network, which the authors found performed better.The backbone is modified from ImageNet-pretrained ResNet50 using inflated 2.5D filters before fine-tuning.
- Optimization: The model is trained with Adam at an initial learning rate of 10^-4 for 30 epochs, using validation-set early stopping for hyper-parameter optimization.Training takes approximately 50 minutes per epoch on four Titan XP GPUs with eight-frame clips.
6 Experimental results
The method is evaluated on three challenging fine-grained activity-recognition datasets using standard comparisons and ablations. Results show gains from object-level reasoning, temporal recurrence, complementary object cues, and temporal CNN integration.
- Datasets: The evaluation covers Something-Something, VLOG, and EPIC Kitchens, representing fine-grained activity-recognition tasks.Something-Something contains 108,000 videos and 157 classes; VLOG contains 114,000 videos and 30 human-object interaction classes; EPIC Kitchens contains 55 hours of non-scripted first-person activity video.
- Comparison with other approaches: The full method reaches 40.89 accuracy on EPIC, outperforming CNN-2D and I3D ResNet-18 baselines by approximately 6.4 and 7.9 points.EPIC results are reported on the validation set because the test set was unavailable.
- Effect of object-level reasoning: Adding the object head and ORN improves the baseline by 0.8, 2.5, and 2.4 points on Something-Something, EPIC, and VLOG, respectively.The ablation uses Top-1 accuracy for EPIC and Something-Something and mAP for VLOG.
- Temporal reasoning: Removing temporal recurrence decreases performance by 1.1 points on VLOG and 1.4 points on EPIC.The recurrent fφ captures long-range reasoning over time; removing it changes the module to an MLP.
- Object representations: Object-region visual features are most discriminative, while object shapes and labels provide complementary information; pairwise cliques outperform unary and size-three cliques.Even unary cliques model object interactions, but the model must identify interaction-specific subspaces in hidden representations.
- CNN architecture and kernel inflations: Temporal convolutions increase performance by up to 100% relative to pure 2D baselines, and 2.5D kernels perform best when used on both input and output sides.The results suggest temporal integration is useful at both low-level motion estimation and high-level reasoning, while recurrent integration outperforms global average pooling.
- Interaction visualizations: The learned interactions correspond to activity-relevant object classes, including person-bed, human-bowl, and human-cup interactions.The visualizations aggregate object-instance mask predictions across samples for each activity class.
7 Conclusion
The paper presents activity recognition through object-instance detections and visual reasoning over object interactions across time. It reports state-of-the-art results on three difficult datasets.
- Object-instance detections provide semantically defined entities for visual reasoning about video activity.The approach focuses on object interactions rather than grid-level convolutional feature-map cells.
- Temporal dependencies and causal relationships are modeled by integrating relationships between different time instants.
- The method achieves state-of-the-art results on three difficult datasets where standard approaches do not perform well.