Source-linked AI summary
Objects2action: Classifying and localizing actions without any video example
Mihir Jain, Jan C. van Gemert, Thomas Mensink, Cees G. M. Snoek
TL;DR
The paper addresses action recognition and localization without video examples, avoiding predefined attribute classifiers and class-to-attribute mappings. It introduces objects2action, an object-based semantic embedding with multiword modeling and responsive-object selection, and reports promising results across four action datasets and related retrieval tasks.
Problem
The paper addresses zero-shot action recognition without video examples while avoiding the need to design attribute classifiers and class-to-attribute mappings.
Method
Objects2action represents actions through a semantic embedding of object categories, models multiword descriptions, and selects the most responsive objects for classification and localization.
Results
The approach achieves promising action classification and localization results on four action datasets and is demonstrated for action and event retrieval.
Takeaways & Limitations
Object images, labels, and freely available text can support zero-shot action recognition without video examples or action annotations.
Takeaways & Limitations
The sparse-selection hyperparameters depend on the dataset, semantic representation, and action description.
Abstract
from arXiv · showhide
The goal of this paper is to recognize actions in video without the need for examples. Different from traditional zero-shot approaches we do not demand the design and specification of attribute classifiers and class-to-attribute mappings to allow for transfer from seen classes to unseen classes. Our key contribution is objects2action, a semantic word embedding that is spanned by a skip-gram model of thousands of object categories. Action labels are assigned to an object encoding of unseen video based on a convex combination of action and object affinities. Our semantic embedding has three main characteristics to accommodate for the specifics of actions. First, we propose a mechanism to exploit multiple-word descriptions of actions and objects. Second, we incorporate the automated selection of the most responsive objects per action. And finally, we demonstrate how to extend our zero-shot approach to the spatio-temporal localization of actions in video. Experiments on four action datasets demonstrate the potential of our approach.
1. Introduction
The paper targets action recognition without video examples or action annotations, avoiding hand-designed attribute classifiers and mappings. It proposes objects2action, which transfers knowledge through object-based semantic representations and extends zero-shot recognition toward localization.
- Zero-shot action recognition is difficult because attributes such as “torso twist” and “look-down” are difficult to define and annotate.
- Existing approaches also assume a large labeled set of action classes for transferring knowledge to unseen classes.
- The approach builds on object scores from image-pretrained networks, whose object categories have linguistic correspondences through WordNet.
- Objects2action classifies actions without video data or action annotations by relying on object annotations, images, and textual descriptions.
- The paper incorporates multiword descriptions, automated selection of responsive objects, and zero-shot action classification plus spatiotemporal localization.
2. Related work
Related work spans supervised action recognition, action localization, attribute-based zero-shot recognition, metadata-driven event recognition, and semantic links based on visual concepts or word embeddings. The paper positions object-based semantic embeddings as a flexible alternative for multiword action descriptions.
- Supervised action recognition aggregates spatiotemporal descriptors or learned video representations before classification with labeled examples.
- Action localization extends classification by detecting where an action occurs, but existing methods combine proposal sampling with labeled video data.
- Traditional zero-shot recognition commonly uses human-provided class-to-attribute mappings, whose design is nontrivial and often domain-specific.
- Zero-shot event recognition can exploit event metadata, multimodal concepts, pseudo-relevance feedback, or relevant external tags.
- Semantic word embeddings provide a flexible link between known and novel labels while supporting multiword object and action descriptions.
3. Objects2action
Objects2action transfers object-based visual representations from ImageNet images to unseen action classes in videos, without action-labeled visual examples. It combines semantic embeddings, multiword descriptions, and sparse object selection to classify actions and support localization.
- Zero-shot prediction combines video object-class scores with affinities that relate training object classes Y to unseen action classes Z.The setup transfers from ImageNet-labeled images to videos and translates object semantics into action semantics.
- Objects2action encodes each video using deep-network probabilities over 15,293 ImageNet object categories, averaged across sampled frames.The object vocabulary contains categories with more than 100 available examples.
- Skip-gram word embeddings place objects and actions in a shared semantic space, where textual contextual similarity supports object-to-action affinities.The approach uses a convex combination of known object classifiers and accommodates objects and actions described by multiple words.
- Fisher Word Vectors aggregate multiword embeddings through a generative topic model, enabling word-sense-sensitive descriptions beyond Average Word Vectors.The Fisher representation models a distribution over words and can distinguish contexts such as infarct and swimming for stroke.
- Action sparsity retains the T_z most responsive objects for each action, while video sparsity retains the T_v most prominent predicted objects to reduce unrelated noise.Both thresholds are dataset- and representation-dependent hyperparameters, with T typically much smaller than the full object vocabulary.
4. Experiments
Experiments evaluate Objects2action across four action datasets, testing semantic embeddings, sparsity choices, zero-shot classification, event retrieval, and localization. The results consistently favor Fisher Word Vectors and action sparsity, while object transfer supports zero-shot recognition without video examples.
- Experimental setup: Experiments cover four action datasets and evaluate semantic embeddings, sparsity, zero-shot classification, event retrieval, and spatio-temporal localization.The datasets include UCF101, THUMOS14, HMDB51, and UCF Sports.
- Sparsity analysis: Action sparsity consistently outperforms video sparsity, with the strongest action-sparsity range selecting 5 to 30 related object classes.The experiments fix Tz = 10 and Tv = 100 for subsequent evaluations.
- Sparsity analysis: Applying sparsity to both video and actions does not improve performance beyond the best action-sparsity setting.This indicates that selecting the most prominent objects per action is sufficient for zero-shot classification in this analysis.
- Zero-shot classification: Fisher Word Vectors always outperform Average Word Vectors, and action sparsity with FWV performs best across the four datasets.The comparison also includes supervised upper bounds using object scores as the video representation.
- Zero-shot classification: Fully supervised classification requires 4 object-representation samples or 10 motion-representation samples per class to surpass the zero-shot approach.This comparison is reported for mean average precision in Figure 4.
- Zero-shot classification: Object transfer outperforms action transfer because thousands of object classes provide more related descriptors than available action labels or attributes.The comparison uses no video examples for training the object-transfer setup.
5. Conclusion
The paper presents zero-shot action recognition without video examples by relating actions to object embeddings built from object images, labels, and text. Experiments show promising classification and localization results across four action datasets, with sparsity and Fisher Vector modeling improving the embedding.
- 5. Conclusion: The method avoids video annotations by using object images and labels plus a text corpus to map unseen actions into an object embedding.This enables zero-shot action recognition without video examples.
- 5. Conclusion: Fisher Vector modeling of embedded-word distributions provides a more precise representation of unseen action-class topics than simple averaging.
- 5. Conclusion: Sparsity in both the object embedding and unseen action class improves performance over relying only on feature dimensionality.
- 5. Conclusion: The approach achieves promising results for action classification and localization on four action datasets.
- 5. Conclusion: The method is also demonstrated for action retrieval on THUMOS14 and event retrieval on TRECVID13 MED.