PaperlayerView exact revision on arXiv

Source-linked AI summary

Transductive Zero-Shot Action Recognition by Word-Vector Embedding

Xun Xu, Timothy Hospedales, Shaogang Gong

arXiv:1511.04458v2cs.CV

TL;DR

The paper addresses zero-shot action recognition under growing annotation demands and severe domain shift between disjoint training and testing classes. It uses word-vector semantic embeddings with complementary transductive and auxiliary-data strategies, achieving competitive performance and enabling analysis of when transfer works. The study remains limited in its treatment of mixed known-and-novel testing classes and transferability prediction.

  • Problem

    Growing action-category coverage makes exhaustive video annotation difficult, while disjoint training and testing classes create a severe domain-shift problem for zero-shot action recognition.

  • Method

    The paper maps visual action features into an unsupervised word-vector semantic space and combines data augmentation, manifold regularization, self-training, and hubness correction.

  • Results

    The four strategies are complementary and together produce a highly effective system competitive with existing attribute-based approaches and state-of-the-art performance under supported settings.

  • Takeaways & Limitations

    Inter-class affinity analysis supports predicting the efficacy of a zero-shot scenario in advance and guiding training-set construction for target classes.

  • Takeaways & Limitations

    Recognizing novel classes when test instances also include known classes remains an open, little-studied setting, and transferability prediction is not fully addressed.

Abstract

from arXiv · show

The number of categories for action recognition is growing rapidly and it has become increasingly hard to label sufficient training data for learning conventional models for all categories. Instead of collecting ever more data and labelling them exhaustively for all categories, an attractive alternative approach is zero-shot learning" (ZSL). To that end, in this study we construct a mapping between visual features and a semantic descriptor of each action category, allowing new categories to be recognised in the absence of any visual training data. Existing ZSL studies focus primarily on still images, and attribute-based semantic representations. In this work, we explore word-vectors as the shared semantic space to embed videos and category labels for ZSL action recognition. This is a more challenging problem than existing ZSL of still images and/or attributes, because the mapping between video spacetime features of actions and the semantic space is more complex and harder to learn for the purpose of generalising over any cross-category domain shift. To solve this generalisation problem in ZSL action recognition, we investigate a series of synergistic strategies to improve upon the standard ZSL pipeline. Most of these strategies are transductive in nature which means access to testing data in the training phase.

1 Introduction

Zero-shot learning addresses the growing cost and difficulty of labeling increasingly numerous and fine-grained action categories by constructing recognizers for unseen classes from semantic descriptions. This work considers unsupervised semantic embeddings, including word vectors, as an alternative to manually specified attributes.

  • Action datasets are expanding in size and category complexity, increasing the effort and cost required to collect and annotate training videos.Action annotation often requires spatio-temporal segmentation, while finer-grained categories also create ontological difficulties.
  • The paper explores constructing recognition models for new action categories by combining language knowledge with information transferred from labeled categories.
  • Zero-shot learning constructs recognizers for unseen categories from semantic descriptions without labeled visual training samples.It learns a general mapping between visual features and semantic descriptors using known categories.
  • Attribute-based semantic descriptors represent categories with manually specified binary or continuous attributes, but unsupervised semantic embeddings offer an automatically constructed alternative.Word-vector representations are produced from large text corpora and avoid manually defining attributes for each category.

The domain shift problem for ZSL of actions

Zero-shot action recognition is difficult because mappings learned on disjoint known classes must generalize across a severe domain shift in complex video space-time features. The paper addresses this challenge with complementary transductive and cross-dataset strategies, reporting broad gains over conventional zero-shot approaches.

  • The domain shift problem for ZSL of actions: Disjoint training and testing categories create domain shift, causing visual-to-semantic mappings learned on known actions to generalize poorly to novel actions.The problem is especially severe for actions because their space-time feature distributions and mappings to semantic space are complex.
  • Our Solutions: The paper combines manifold regularization, data augmentation, self-training, and hubness correction to improve generalization across the train-test semantic gap.Manifold regularization uses unlabeled testing data, while data augmentation adds auxiliary datasets and the other methods adapt or re-rank predictions.
  • Results: The word-vector embedding approach consistently outperforms conventional zero-shot approaches across HMDB51, UCF101, Olympic Sports, CCV, and USAA.
  • Our Solutions: In this context, data augmentation means adding data from additional datasets rather than synthesizing examples through transformations.
  • Transductive Setting: Manifold regularization, self-training, and hubness correction require access to the full unlabeled test set during training, defining a transductive setting.The paper motivates this setting for batch processing of unlabeled video repositories.
  • New Insights: The study analyzes relationships between training and testing classes to investigate why and when zero-shot recognition works.It reports a causal connection between known-category and novel-category recognition performance.
  • Contributions: The model uses a closed-form visual-to-semantic mapping and unsupervised semantic embeddings while achieving state-of-the-art performance on contemporary action and event datasets.The authors also report a simple, fast implementation and analysis of factors affecting zero-shot efficacy.

2 Related Work

Related work covers video and event recognition, zero-shot architectures, semantic representations, and the domain-shift problem. The paper positions word-vector embeddings and transductive strategies as responses to scalability and generalization challenges in action and event recognition.

  • Human Action Datasets: Action-recognition datasets progressed from isolated actions to larger online-video collections, while event datasets target longer multimedia events.Examples include KTH, Weizmann, OlympicSports, HMDB51, CCV, and TRECVID MED.
  • Feature Representation: Local space-time features became prevalent because they avoid non-trivial object tracking and segmentation while capturing texture, shape, and motion.
  • Zero-Shot Learning: Zero-shot learning dynamically constructs classifiers for novel classes from semantic descriptors rather than labeled examples.
  • ZSL Architectures: Sequential zero-shot architectures map visual features to semantic representations and then recognize classes from the predicted representations.The visual mapping is learned from training data and assumed to generalize to novel classes.
  • Attribute Embeddings: Attribute embeddings require categories to be described using binary or continuous attributes, creating a need for a shared ontology and manual class specification.
  • Word-Vector Embeddings: Word-vector embeddings generate class descriptors automatically from unstructured text knowledge bases such as Wikipedia.The vectors encode class names through their distributional relationships in a text corpus, and recognition commonly uses nearest-neighbor matching.
  • Domain-Shift: Every zero-shot method faces domain shift because mappings are trained on one set of classes and tested on a disjoint set.The paper addresses this through dataset augmentation, manifold regularization, self-training, and hubness correction.
  • ZSL Insights: Prior work examined class-label embeddings, scalability, and hubness, while this study focuses on when positive transfer occurs.

3 Methodology

The method maps video features and action labels into a shared word-vector space, then improves zero-shot generalization with transductive manifold regularization and auxiliary-data augmentation.

  • Problem formulation: Zero-shot learning recognizes testing videos whose classes are disjoint from training classes by mapping visual features to semantic class descriptors.The training and testing label sets satisfy ytr ∩ yte = ∅.
  • Semantic embedding space: Word-vector embeddings encode action labels, with compound names represented by averaging the vectors of their constituent words.The embedding is constructed from a neural-network word representation, while compound labels receive a single averaged vector.
  • Visual-to-semantic mapping: Kernel ridge regression learns a mapping from low-level visual features to class-name embeddings and projects videos into the semantic space.The regression uses training features and their corresponding embedding targets, with a closed-form solution.
  • Manifold-regularized regression: Manifold regularization uses unlabelled testing videos to preserve visual-neighbour relationships in semantic space during transductive training.A K-nearest-neighbour graph over labelled and unlabelled instances supplies the graph-Laplacian regularizer.
  • Data augmentation: Data augmentation merges target training data with auxiliary datasets, whose class names are also embedded, to provide more training examples for the regressor.The auxiliary datasets need not share classes with the target dataset, and their instances and embeddings are combined with target data.
  • Optimization: The combined manifold-regularized augmentation objective remains convex and has a closed-form solution that can be efficiently implemented.Setting γI = 0 reduces the formulation to kernel ridge regression with data augmentation only.

4 Experiments

Experiments evaluate the approach across action-recognition and event-detection datasets using word-vector embeddings, controlled component combinations, and established zero-shot baselines. The results show complementary gains from manifold regularization, data augmentation, self-training, and hubness correction.

  • Datasets and settings: Experiments cover HMDB51, UCF101, Olympic Sports, CCV, and TRECVID MED 2013, spanning human-action recognition and event detection.The datasets include classification and detection benchmarks with differing numbers of categories and videos.
  • Feature and embedding representations: Videos are represented with improved trajectory descriptors encoded as Fisher Vectors, while labels use 300-dimensional skipgram word vectors.The word vectors come from a model trained on approximately 100 billion words.
  • Evaluation protocol: A 50/50 split holds out half the categories for zero-shot testing, with mean accuracy and standard deviation computed over 50 independent splits.Mappings are trained on the remaining seen categories.
  • Ablation design: The component study varies manifold regularization, self-training, hubness correction, data augmentation, matching strategy, and transductive access to unlabelled test data.Exact class matches are excluded from auxiliary datasets to preserve the zero-shot assumption, while semantically related classes may remain.
  • Metrics: Average accuracy is reported for HMDB51, UCF101, and USAA, while mean average precision is reported for Olympic Sports and CCV.For detection benchmarks, normalization does not change retrieval rankings, so NN and NRM have identical performance.
  • Results: Manifold regularization, data augmentation, self-training, and hubness correction each improve results, with the first two complementary to the post-processing methods.The direct feature-to-embedding baseline already performs reasonably well, supporting semantic space as a zero-shot representation.
  • Baselines: The study also compares word-vector methods including SJE, ConSE, and SVE with state-of-the-art attribute-embedding approaches.Attribute embeddings are evaluated only on datasets where attributes are available.

1. Direct Attribute Prediction (DAP) We imple-

The DAP implementation uses Fisher Vector visual features with linear-kernel SVM attribute classifiers and performs recognition from attribute posteriors and manually specified class attributes.

  • Direct Attribute Prediction: DAP uses linear-kernel SVM classifiers to estimate attributes from Fisher Vector visual features.Recognition combines attribute posteriors with manually specified attribute descriptors for each class.
  • Recognition: Recognition is performed using attribute posteriors together with manually specified attribute descriptors.
  • Role in evaluation: The method provides an attribute-embedding comparison point for the paper’s word-vector embedding approach.

2. Indirect Attribute Prediction (IAP) (Lampert

IAP differs from DAP by learning category classifiers first and then using training-category attribute dependencies to estimate attributes. Table 3 reports component-wise zero-shot results across classification and detection benchmarks.

  • Indirect Attribute Prediction: IAP first learns per-category classifiers p(y|x) from training data, then uses category–attribute dependencies p(a|y) to estimate attributes.
  • Attribute estimation: The IAP procedure obtains attribute estimators from training-category attribute-prototype dependencies rather than directly classifying attributes.
  • Evaluation: Table 3 evaluates individual components using average accuracy for HMDB51, UCF101, and USAA and mean average precision for Olympic Sports and CCV.A dash indicates no performance difference between NN and NRM.

3. Human Actions by Attributes (HAA) (Liu et al,

HAA uses nearest-neighbour matching in a manually defined attribute embedding space, while M2LATM combines user-defined and discovered latent attributes with multiple visual modalities.

  • HAA: HAA performs nearest-neighbour matching in attribute embedding space for zero-shot recognition.Its testing procedure projects samples into attribute space and assigns the closest testing-class prototype using cosine distance.
  • M2LATM: M2LATM combines user-defined and discovered latent attributes with SIFT, STIP, and MFCC features.The multimodal design uses static, motion, and audio information and is evaluated on the USAA dataset.

6. Transductive Multi-View Bayesian Label Propagation (TMV-BLP) (Fu et al, 2014a). This

The paper compares mixed semantic embeddings and transductive strategies across zero-shot benchmarks, finding strong performance and analyzing when transfer succeeds or fails. Results also show that visual clustering, class relatedness, and transductive adaptation shape recognition quality.

  • Embedding comparisons: Mixed embedding concatenates attribute and word-vector representations and is compared with TMV-BLP, UDA, and TMV-HLP.The paper evaluates this simple concatenation despite multi-view embedding not being its main focus.
  • Experimental results: The method’s non-transductive RR model beats SJE and ConSE on UCF101, CCV, and USAA with word-vector embeddings.It also beats DAP, IAP, and HAA on Olympic Sports and USAA.
  • Experimental results: With transductive testing access, MR-X-✓-W outperforms most word-vector alternatives and is competitive with attribute-embedding models.Combining manifold regularization, data augmentation, self-training, and hubness correction remains competitive or significantly better than attribute-centric methods depending on the dataset.
  • Transductive strategies: Hubness correction improves HMDB51 and UCF101 across three models, while self-training generally helps ConSE and HAA but harms SJE.The paper attributes SJE’s weaker self-training response to ranking-loss projections that do not form neat word-vector clusters.
  • Complex events: The direct embedding approach exceeds the 0.28% random-guess baseline for TRECVID MED 2013, with self-training and manifold regularization providing further improvement.The evaluation reports mean average precision for detecting 20 events and an average over all events.
  • Qualitative mechanisms: Manifold regularization tightens same-class visual clusters, data augmentation moves unseen samples toward prototypes, and self-training adapts prototypes toward sample centers.These mechanisms explain how the proposed components improve zero-shot accuracy.
  • Transferability: The correlation between class-name relatedness and zero-shot efficacy is 0.548, while transfer can vary across related category pairs.For example, ride horse and ride bike transfer symmetrically, whereas cartwheel supports handstand without the reverse relation.
  • Transferability: Class-name affinity predicts transfer: a 30% subset of related training classes outperforms a 70% subset of unrelated classes.The maximal class-name affinity measure is the most predictive among the tested alternatives, and related classes benefit zero-shot performance when labels are limited.

1. Low-Level Feature (Wang and Schmid, 2013)

The semantic embedding representation is comparable to strong low-level and attribute-based approaches while avoiding supervised manual attribute definition and annotation.

  • The semantic embedding model maps data into a word-vector space before training category classifiers on the mapped representations.It uses regularized linear regression and then trains a linear SVM for each category.
  • Unlike attribute-based representations, the semantic embedding requires no supervised manual attribute definition or annotation.
  • The implementation has favorable efficiency, requiring about 300 seconds for 50 HMDB51 splits, or 520 seconds with data augmentation.The runtime is dominated by matrix inversion.

5 Detailed Parameter Sensitivity Analysis

The analysis examines sensitivity to embedding, regression, manifold, and self-training parameters, finding generally robust performance except under unregularized regression and with some neighborhood choices.

  • Experimental settings: The main experiments use γA = 10^-6, γI = 40, manifold KNN K = 5, and self-training K = 100.
  • Word-vector dimension: A reasonable word-vector dimension is between 256 and 2048, although performance trends differ across datasets.Performance generally increases with dimension on HMDB51, UCF101, and Olympic Sports, but shows no clear trend for CCV.
  • Ridge regression regularization: Zero-shot performance is close to random without ridge regularization because zero or collinear kernel rows or columns cause numerical inversion problems.
  • Manifold regression: The framework is not very sensitive to manifold parameters, with a slight preference for moderately low graph-neighborhood values.The analysis varies γI and the manifold KNN parameter jointly.
  • Self-training: Self-training performance is robust when the self-training neighborhood parameter K is above 20.This parameter concerns the neighboring distribution around testing-time prototypes rather than the training-time manifold graph.

6 Conclusion

The paper applies unsupervised word-vector embeddings and four strategies to zero-shot action recognition, showing that transductive access to testing data makes them complementary and effective. It also identifies unresolved challenges in mixed known/novel testing and transferability prediction.

  • The study investigates unsupervised word-vector embeddings for zero-shot action recognition under disjoint training and testing classes.
  • Data augmentation, manifold regularization, self-training, and hubness correction are complementary strategies for addressing domain shift.
  • The resulting system is competitive with attribute-based approaches, has a closed-form solution, and is simple and efficient to implement.
  • The analysis supports predicting zero-shot efficacy in advance and guiding training-set construction for target classes.
  • Recognizing novel classes when testing data also contain known classes remains an open problem, and transferability prediction is not fully addressed.The paper notes that class interactions may make pairwise semantic relatedness insufficient for selecting training data.
Loading 1511.04458v2…