Source-linked AI summary
Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions
Pascal Mettes, Cees G. M. Snoek
TL;DR
Zero-shot action localization and classification must distinguish actions whose global semantic embeddings miss actors’ interactions with their environments. The paper builds a spatial-aware object embedding from actor and object information and their relations. It reports state-of-the-art results in zero-shot action localization and classification, and demonstrates spatio-temporal retrieval with composite queries.
Problem
Global attribute or object-classification embeddings miss actors’ spatial interactions, making similar actions such as throwing and hitting a baseball hard to distinguish within one video.
Source: §1 · p. 1Method
A spatial-aware object embedding combines word embeddings, actor and object box locations, spatial relations, and global-object context for action tubes and composite retrieval queries.
Source: §1 · pp. 1–2, §4.2 · p. 4Results
The embedding achieves state-of-the-art zero-shot action localization and classification results, and is demonstrated for spatio-temporal retrieval with queries containing object positions and sizes.
Source: §7 · p. 8Takeaways & Limitations
The embedding supports retrieval queries that specify objects, spatial relations, and relative size, alongside zero-shot action localization and classification.
Source: §1 · p. 2, §4.2 · p. 4
Abstract
from arXiv · showhide
We aim for zero-shot localization and classification of human actions in video. Where traditional approaches rely on global attribute or object classification scores for their zero-shot knowledge transfer, our main contribution is a spatial-aware object embedding. To arrive at spatial awareness, we build our embedding on top of freely available actor and object detectors. Relevance of objects is determined in a word embedding space and further enforced with estimated spatial preferences. Besides local object awareness, we also embed global object awareness into our embedding to maximize actor and object interaction. Finally, we exploit the object positions and sizes in the spatial-aware embedding to demonstrate a new spatio-temporal action retrieval scenario with composite queries. Action localization and classification experiments on four contemporary action video datasets support our proposal. Apart from state-of-the-art results in the zero-shot localization and classification settings, our spatial-aware embedding is even competitive with recent supervised action localization alternatives.
1. Introduction
Zero-shot action recognition based on global attributes or object scores misses actors’ spatial interactions, making visually similar actions hard to distinguish. The paper proposes a spatial-aware object embedding for action localization and classification, and extends it to spatio-temporal retrieval.
1. Introduction
- Global attribute or object-classification embeddings miss actor–environment spatial interactions, making actions such as throwing and hitting a baseball difficult to distinguish within one video.The paper frames this as a limitation of semantic embeddings built from image or video classifiers.Source: §1 · p. 1
- The proposed spatial-aware object embedding targets zero-shot action localization and classification without action-video training examples.It builds on actor and object detectors and incorporates word embeddings.Source: §1 · p. 1
- Actor and object box locations and their spatial relations generate action tubes for video classification and precise action localization.The embedding also extends to contextual awareness from global objects.Source: §1 · p. 2
- The embedding extends to spatio-temporal retrieval queries involving objects, spatial relations, and object sizes.The paper reports localization and classification experiments on four contemporary action video datasets.Source: §1 · p. 2
2. Related work
Prior work includes supervised representations and localization methods, attribute- or action-mapping-based zero-shot classification, and holistic object-score approaches. This paper situates its approach as using local actor and object information to model spatial interaction for zero-shot localization, classification, and retrieval.
2.1. Supervised action localization and classification
- Recent supervised localizers use Faster R-CNN trained on action-frame bounding boxes, while this work uses commonly available image-trained detectors for zero-shot localization.Earlier methods also used local features or trajectories, and action proposals built from supervoxels, trajectories, or actor detections.Source: §2.1 · p. 2
2.2. Zero-shot action localization and classification
- Attribute-based zero-shot methods require manual action-to-attribute mappings and cannot localize actions or generalize to arbitrary queries.These approaches learn attributes from training videos of other actions.Source: §2.2 · p. 2
- Action-to-action mapping methods and relatedness-based unseen-action classifiers avoid attributes but still require other-action videos and only classify.Source: §2.2 · p. 2
- Holistic object-score methods support zero-shot classification of arbitrary actions, whereas local object information models actor–object spatial interaction for localization, classification, and retrieval.Jain et al. [19] previously localized zero-shot actions using proposals and object scores, without object detectors or explicit spatial relations.Source: §2.2 · p. 2
3. Spatial-aware object embeddings
The method builds zero-shot action tubes by scoring actor boxes for actor presence, relevant objects, and expected actor–object spatial relations. It links high-scoring boxes over time to localize actions.
3. Spatial-aware object embeddings
- The framework uses per-frame actor and local-object detector boxes and scores to construct spatial-aware embeddings for zero-shot action classification and localization.Its three stages gather prior knowledge, score bounding boxes, and link boxes into action tubes.Source: §3 · p. 2, §3.1 · p. 2, §3.3 · p. 3
3.1. Prior knowledge
- Object relevance is selected using word2vec similarity to the action name, while prior knowledge represents each object’s expected location relative to actors in nine spatial positions.Examples include skateboards on or below actors, bicycles to their left or right, and traffic lights above them.Source: §3.1 · p. 3, Figure 2 · p. 3
3.2. Scoring actor boxes with object interaction
- For each bounding box, the score function incorporates actor-detector probability, relevant-object evidence, and the match between observed and preferred actor–object spatial relations.The object search considers nearby boxes and favors those with both object-detection probability and a spatial-relation match; the match is one minus the Jensen–Shannon divergence between distributions.Source: §3.2 · p. 3
3.3. Linking spatial-aware boxes
- Action tubes are formed by linking actor boxes with high spatial-aware scores and overlap over time, using Viterbi dynamic programming and then removing each selected tube’s boxes.The score combines actor presence, relevant local-object presence, and expected actor–object spatial relations.Source: §3.3 · p. 3
4. Local and global object interaction
The method combines local actor–object interactions with video-level global-object context to score action tubes, classify videos, and retrieve tubes using composite spatial queries.
4.1. Scoring videos with global objects
- For each action, relevant global objects are selected by textual embedding, and their word2vec similarities are combined with video-level classifier probabilities.The global-object set consists of objects with corresponding classifiers and names; p(g|V) is the probability of object g appearing in video V.Source: §4.1 · p. 4
4.2. Spatial- and global-aware embedding
- Global-object scores distinguish tubes across videos, but not tubes within the same video; localization ranks tubes using the combined scores.The method selects top-scoring tubes per video before ranking tubes across the collection.Source: §4.2 · p. 4
- Video classification max-pools tube scores, combines that score with global-object video scores, and predicts the class with the highest combined score.For classification, precise tube locations are not required.Source: §4.2 · p. 4
- Retrieval ranks tubes against queries specifying objects, spatial relations, and relative object size, with size matched using bounding-box size.Changing the queried object size can retrieve different localizations of the same action.Source: §4.2 · p. 4
5. Experimental setup
The experiments use four action datasets for classification and localization, with detector-based inputs, word embeddings, and video-level object classifiers; performance is assessed with localization and classification metrics.
5.1. Datasets
- UCF Sports has 150 videos across 10 sport actions, while UCF 101 has 13,320 videos across 101 actions.UCF Sports uses the test split suggested in; UCF 101 uses its provided test splits unless otherwise stated.Source: §5.1 · p. 4
- J-HMDB contains 928 videos across 21 actions, and Hollywood2Tubes contains 1,707 Hollywood2 videos with spatiotemporal localization annotations.J-HMDB localization uses bounding boxes around binary action masks; both datasets use their suggested test splits.Source: §5.1 · pp. 4–5
5.2. Implementation details
- Action and object names are represented by averaged 500-dimensional word2vec vectors trained on YFCC100M metadata.The embeddings are used to map action semantics to objects.Source: §5.2 · p. 5
- Faster R-CNN detects actors and local objects, while GoogLeNet supplies global-object probabilities averaged over sampled video frames.Actor-object spatial relations are estimated from MS-COCO; the top 100 relevant global objects are selected per action.Source: §5.2 · p. 5
- Localization is evaluated with spatiotemporal intersection-over-union, mean Average Precision, and AUC; classification uses mean class accuracy.Table 1 compares actor-only, actor-plus-object, and spatial-aware embeddings on UCF Sports.Source: §5.2 · p. 5, Table 1 · p. 5
6. Experimental results
The experiments evaluate spatial-aware embeddings, global-object context, composite-query retrieval, and comparisons with zero-shot and supervised action methods. Results show benefits from spatial and global object information, with performance varying by task and available relevant objects.
6.1. Spatial-aware embedding properties
- Adding local objects improves both tasks, and the spatial-aware embedding performs best among actor-only, actor-plus-object, and spatial-aware settings.The benefit varies by action: Riding a horse rises from 0.03 to 0.75 mAP, while Running is unchanged when no relevant object is among the available detectors.Source: §6.1 · pp. 5–6
- 0.221 mAP at overlap 0.5 is reached with the top-1 local object for localization, versus 0.083 using only the actor; classification instead benefits from aggregating more objects.The experiments use the top-1 object for localization and top-5 for classification; finer grids slightly reduce localization scores from 0.199 with a 3x3 grid to 0.170 and 0.171 with 4x4 and 5x5 grids.Source: §6.1 · p. 6
6.2. Local and global object interaction
- Global-object scores improve localization across overlap thresholds, with the largest gains at low thresholds, and improve UCF Sports classification accuracy.Classification accuracy is 0.645 with GoogleNet global classifiers and 0.374 using Jain et al.'s network, compared with 0.264 reported in [19].Source: §6.2 · pp. 6–7, Table 2 · p. 7
6.3. Spatial-aware action retrieval
- Composite queries on J-HMDB retrieve action localizations reflecting specified objects, spatial relations, and object sizes.Changing the requested object size yields different retrievals, including baseball and soccer-ball examples.Source: §6.3 · p. 7, Figure 6 · p. 7
6.4. Comparison to state-of-the-art
- On UCF-101, the method reports an accuracy of 0.328 versus 0.303 for Jain et al. [19], and 0.404 on the 51/50 split versus the next-best zero-shot result of 0.268.For 20 test actions, it reports 0.512, compared with 0.255 and 0.311 [11].Source: §6.4 · pp. 7–8, Table 3 · p. 8
- For localization, the method is competitive with several supervised alternatives on UCF Sports and Hollywood2Tubes, and reports the first zero-shot localization results on J-HMDB.The state-of-the-art supervised approach performs better but requires thousands of video-tube annotations.Source: §6.4 · p. 8, Table 4 · p. 8
7. Conclusions
The paper presents a spatial-aware embedding for zero-shot action localization and classification, combining actors, relevant local objects, and their spatial relations with global object context. It reports state-of-the-art zero-shot results and demonstrates spatio-temporal retrieval using queries with object positions and sizes.
7. Conclusions
- The embedding localizes and classifies actions without action-video training, using actors, relevant local objects, their spatial relations, and global-object context.Experiments report state-of-the-art zero-shot localization and classification.Source: §7 · p. 8
- The embedding achieves state-of-the-art results in zero-shot action localization and classification.Source: §7 · p. 8
- The paper demonstrates spatio-temporal action retrieval with queries containing object positions and sizes.Source: §7 · p. 8