PaperlayerView exact revision on arXiv

Source-linked AI summary

A Generative Approach to Zero-Shot and Few-Shot Action Recognition

Ashish Mishra, Vinay Kumar Verma, M Shiva Krishna Reddy, Arulkumar S, Piyush Rai, Anurag Mittal

arXiv:1801.09086v1cs.CV

TL;DR

Growing action-class vocabularies make annotation cumbersome, while zero-shot recognition targets classes missing from training. The paper predicts visual-space class distributions from attributes and seen-class data; it reports state-of-the-art performance across zero-shot settings, with generative sampling and unlabeled unseen-class data supporting generalized and transductive use.

  • Problem

    As action categories grow, annotating videos becomes cumbersome and can restrict scalability; zero-shot learning addresses recognition of classes absent from training using information such as attributes.

    Source: §1 · p. 1

  • Method

    The method represents each action class by a probability distribution whose parameters are attribute-weighted combinations of basis parameters learned from seen classes.

    Source: §1 · pp. 1–2

  • Results

    The approach achieves state-of-the-art performance in inductive, transductive, and generalized zero-shot settings, with significant gains reported across three benchmark datasets and all three settings.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Sampling from predicted unseen-class distributions can provide novel examples for generalized zero-shot classifier training, and leveraging unlabeled unseen-class data helps address domain shift.

    Source: §1 · p. 2, §5 · p. 8

    The Gaussian formulation assumes a diagonal covariance matrix to reduce estimated parameters and prevent overfitting, though other covariance forms can also be used.

    Source: §2 · p. 3

Abstract

from arXiv · show

We present a generative framework for zero-shot action recognition where some of the possible action classes do not occur in the training data. Our approach is based on modeling each action class using a probability distribution whose parameters are functions of the attribute vector representing that action class. In particular, we assume that the distribution parameters for any action class in the visual space can be expressed as a linear combination of a set of basis vectors where the combination weights are given by the attributes of the action class. These basis vectors can be learned solely using labeled data from the known (i.e., previously seen) action classes, and can then be used to predict the parameters of the probability distributions of unseen action classes. We consider two settings: (1) Inductive setting, where we use only the labeled examples of the seen action classes to predict the unseen action class parameters; and (2) Transductive setting which further leverages unlabeled data from the unseen action classes. Our framework also naturally extends to few-shot action recognition where a few labeled examples from unseen classes are available. Our experiments on benchmark datasets (UCF101, HMDB51 and Olympic) show significant performance improvements as compared to various baselines, in both standard zero-shot (disjoint seen and unseen classes) and generalized zero-shot learning settings.

1. Introduction

The growing number of action categories makes full annotation cumbersome, motivating recognition of classes absent from training. The paper models action classes with attribute-weighted visual-space distributions and extends the approach to transductive, few-shot, and generalized zero-shot settings.

1. Introduction

  • Each action class is modeled as a visual-space probability distribution whose parameters are a linear combination of basis parameters weighted by the class attributes.The contribution describes a Gaussian distribution, while noting that other distributions can be used.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Basis vectors learned from seen classes predict unseen-class distribution parameters, while a reverse-direction regularizer encourages reconstruction of class attributes from those parameters.The loss brings seen-class basis combinations close to maximum-likelihood estimates and admits a closed-form solution.Source: §1 · p. 2
  • The approach generalizes to unseen classes inductively, improves over state of the art, and reports significant gains across three benchmark datasets and all three settings.The settings include inductive, transductive, and few-shot learning.Source: §1 · p. 2
  • The approach extends to transductive learning by using unlabeled data from unseen classes during training.Source: §1 · p. 2
  • In the few-shot extension, a few additional labeled examples from an unseen class can update its distribution parameters.Source: §1 · p. 2
  • Sampling from an unseen class distribution synthesizes novel examples that can support classifier training in generalized zero-shot learning.The generalized setting can include labeled data from both seen and unseen classes.Source: §1 · p. 2

2. Methodology

The method predicts visual-space class distributions from class attributes, learning the mapping from labeled seen classes so it can estimate parameters for unseen classes. It extends this approach to transductive, few-shot, and other-distribution settings.

2. Methodology

  • Each class is modeled by a distribution whose parameters depend on its attributes; a diagonal Gaussian covariance reduces parameter count and overfitting risk when examples are scarce.The framework can also use other covariance forms.Source: §2 · p. 3

2.1. Linear Regression

  • Linear regression learns attribute-to-mean and attribute-to-variance projections from seen-class estimates using multi-output ridge regression.The ridge problems have closed-form solutions.Source: §2.1 · p. 3

2.2. Nonlinear Regression

  • Nonlinear regression maps attributes through a kernel and regularizes the visual-space projection by reconstructing attributes from visual features.The reconstruction term acts as an autoencoder-style regularizer.Source: §2.2 · pp. 3–4

2.2.1 Optimization

  • The optimization conditions form Sylvester equations, which can be solved efficiently with the Bartels–Stewart algorithm.The paper also notes that off-the-shelf solvers are available.Source: §2.2.1 · p. 4

2.3. Transductive setting

  • In the transductive setting, inductive estimates initialize unseen-class distributions that are fine-tuned with unlabeled test data using EM to address domain shift.With Gaussian class distributions, this procedure is equivalent to a GMM with one component per unseen class.Source: §2.3 · pp. 4–5

2.4. Few-shot Action Recognition

  • For few-shot recognition, the zero-shot parameter estimates serve as priors that can be updated using a small number of labeled examples from each unseen class.The Gaussian conjugacy makes these updates straightforward.Source: §2.4 · p. 5

2.5. Extension to Other Distributions

  • The framework is not restricted to Gaussians: it applies to parametric distributions, and other density-estimation methods such as deep generative models [32] can also be used.Gaussian or exponential-family conjugacy makes the transductive and few-shot estimation procedures particularly simple.Source: §2.5 · p. 5

3. Related Work

The related work places zero-shot learning at the intersection of visual features, class semantics, and mappings between them. This paper instead maps semantic features to distribution parameters, combining generative modeling with reconstruction regularization for action recognition.

Related work

  • Where many zero-shot methods map between visual and semantic features, this approach maps semantic features to parameters of class distributions.The related-work discussion identifies this as the key difference from such methods.Source: §3 · pp. 5–6
  • The model combines the generative approach of [29] with the autoencoder-style reconstruction regularizer of [9], unlike the latter’s inductive, non-generative method.Source: §3 · p. 6
  • Prior transductive action-recognition work [35] uses unlabeled unseen-class data with manifold-regularized regression and data augmentation.Source: §3 · p. 6
  • The paper evaluates its approach for both generalized and standard zero-shot learning on three standard datasets.The passage names no dataset-specific results.Source: §3 · p. 6

4. Experiments

Experiments evaluate zero-shot recognition in inductive, transductive, generalized, and few-shot settings across UCF101, HMDB51, and Olympic. The reported results favor the approach in most settings, with a UCF101 exception in transductive ZSL and few-shot performance comparable to supervised learning.

4. Experiments

  • Evaluation covers UCF101 (51/50), HMDB51 (26/25), and Olympic (8/8) seen/unseen splits, reporting mean accuracy and standard deviation over 30 random splits.Source: §4 · p. 6
  • Experiments use averaged 4096-dimensional C3D fc6 video features and human attributes or 300-dimensional word2vec; HMDB51 uses word2vec only.Source: §4 · p. 6

4.1. Inductive and Transductive ZSL

  • Inductively, the model improves on the state of the art by 3% on Olympic, leads on UCF101 with attributes, and outperforms the state of the art on HMDB51 with word2vec.HMDB51 results use word2vec because class attributes are unavailable.Source: §4.1 · p. 7, Table 2 · p. 7
  • Transductively, the model outperforms the state of the art on Olympic and HMDB51, while [31] performs best on UCF101.The approach outperforms [31] on UCF101 in the inductive setting.Source: §4.1 · p. 7, Table 3 · p. 7

4.2. Generalized ZSL

  • In generalized ZSL, the model significantly outperforms the state of the art on all three datasets using transductively estimated distributions and 200 synthesized unseen-class examples.The synthesized examples are combined with labeled seen-class data to train an SVM.Source: §4.2 · p. 8, Table 4 · p. 8

4.3. Few-shot action recognition

  • With a few labeled examples per unseen class and no unseen-class unlabeled data, performance improves to become comparable to supervised learning.The experiments vary the number of examples per class; Figure 3 plots accuracy against the number of data points.Source: §4.3 · p. 8, Table 5 · p. 8, Figure 3 · p. 8

5. Conclusion

The paper concludes that its probabilistic generative framework performs well across standard and generalized zero-shot settings, and extends to few-shot recognition. It reports state-of-the-art performance and few-shot results comparable to fully supervised learning using only a few examples from unseen classes.

  • The approach performs well in inductive and transductive standard and generalized ZSL, with state-of-the-art experimental performance.Source: §5 · p. 8
  • Its generative model can synthesize unseen-class examples, while leveraging unlabeled data in the transductive setting helps address domain shift between seen and unseen classes.The framework yields a closed-form parameter solution that the authors describe as fast and easy to implement.Source: §5 · p. 8
  • In few-shot action recognition, the method achieves results comparable to fully supervised learning using only a few synthesized examples from each unseen class.Source: §5 · p. 8
Loading 1801.09086v1…