Source-linked AI summary
Out-of-Distribution Detection for Generalized Zero-Shot Action Recognition
Devraj Mandal, Sanath Narayan, Saikumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shahbaz Khan, Ling Shao
TL;DR
Generalized zero-shot action recognition must classify both seen and unseen action categories, but standard classifiers are biased toward seen classes. The paper introduces an entropy-based out-of-distribution detector, trained with GAN-synthesized unseen features, to route samples to separate classifiers. It outperforms the f-CLSWGAN baseline across Olympic Sports, HMDB51, and UCF101.
Problem
Generalized zero-shot action recognition must recognize unseen action categories alongside seen categories, while learned classifiers are biased toward seen classes and misclassify unseen samples.
Method
An entropy-based out-of-distribution detector separates seen and unseen features, using GAN-generated unseen features to train the detector and separate classifiers.
Results
The proposed approach outperforms f-CLSWGAN by 7.0% on Olympic Sports, 3.4% on HMDB51, and 4.9% on UCF101 in classification accuracy.
Takeaways & Limitations
Separating seen and unseen classification reduces the bias toward seen classes in generalized zero-shot action recognition.
Abstract
from arXiv · showhide
Generalized zero-shot action recognition is a challenging problem, where the task is to recognize new action categories that are unavailable during the training stage, in addition to the seen action categories. Existing approaches suffer from the inherent bias of the learned classifier towards the seen action categories. As a consequence, unseen category samples are incorrectly classified as belonging to one of the seen action categories. In this paper, we set out to tackle this issue by arguing for a separate treatment of seen and unseen action categories in generalized zero-shot action recognition. We introduce an out-of-distribution detector that determines whether the video features belong to a seen or unseen action category. To train our out-of-distribution detector, video features for unseen action categories are synthesized using generative adversarial networks trained on seen action category features. To the best of our knowledge, we are the first to propose an out-of-distribution detector based GZSL framework for action recognition in videos. Experiments are performed on three action recognition datasets: Olympic Sports, HMDB51 and UCF101. For generalized zero-shot action recognition, our proposed approach outperforms the baseline (f-CLSWGAN) with absolute gains (in classification accuracy) of 7.0%, 3.4%, and 4.9%, respectively, on these datasets.
1. Introduction
Generalized zero-shot action recognition extends zero-shot learning to test samples from both seen and unseen categories, but learned classifiers are biased toward seen classes. The paper addresses this by separating seen and unseen classification with an out-of-distribution detector trained using synthesized unseen features.
- Problem: Generalized zero-shot learning evaluates samples from both seen and previously unavailable unseen categories, making it harder than standard zero-shot learning.The paper focuses on generalized zero-shot action recognition in videos and treats standard ZSL as a special case of GZSL.
- Motivation: Fully supervised action recognition assumes all test categories and training instances are available, limiting its realism for applications such as tagging web videos.Zero-shot action recognition addresses cases where information about some categories is unavailable during training.
- Motivation: Existing transductive approaches use unlabelled test data for fine-tuning, yet bias persists because seen and unseen categories receive similar treatment.The paper seeks a non-transductive treatment that separates the two category groups.
- Approach: The proposed detector uses entropy to distinguish seen from unseen features and dynamically routes each test feature to a classifier trained for the corresponding category group.Seen inputs produce peaked, non-uniform outputs, whereas unseen inputs are trained toward uniform outputs; inference compares entropy with a threshold.
- Approach: A conditional Wasserstein GAN synthesizes unseen action features from seen features and class embeddings, enabling joint training of the detector and separate seen/unseen classifiers.Cosine embedding and cycle-consistency losses are added to improve feature generation across action categories.
2. Related Work
Prior zero-shot action-recognition work largely relies on transductive data or feature-generation strategies, while generalized zero-shot action recognition remains less explored. This paper instead trains an entropy-based out-of-distribution detector with GAN-generated unseen features and avoids prior data-distribution assumptions.
- Zero-shot learning: Zero-shot methods address categories unseen during training, while generalized zero-shot learning must classify both seen and unseen categories.These settings target real-world applications such as automatic tagging of images and videos with new categories.
- Existing approaches: Earlier image-based approaches used attribute mapping or combinations of seen classes, whereas GZSL methods also generate unseen features with GAN variants.Synthesizing realistic, discriminative unseen features is identified as a challenge because of classifier bias toward seen classes.
- Action recognition: Video action-recognition studies have received less attention and commonly use transductive training with unlabelled test data, unlike this work.Prior work also leaves challenging GZSL comparisons across feature types unexplored.
- Proposed approach: The proposed method trains an entropy-based out-of-distribution detector with GAN-generated unseen action features to distinguish seen from unseen groups without assuming a prior data distribution.The GAN is trained on real seen-class features and conditioned on class attributes.
3. Proposed Approach
The framework synthesizes unseen-class video features with a conditional WGAN, then separates seen and unseen inputs using an entropy-based out-of-distribution detector and routes them to dedicated classifiers.
- Proposed Approach: The framework reformulates GZSL as three classifiers: an out-of-distribution detector routes features to separate seen- and unseen-class classifiers.The detector identifies whether a feature is in-distribution or out-of-distribution before class prediction.
- Generating unseen class features: Unseen-class features are synthesized by conditioning a WGAN on class embeddings, using Wasserstein, cycle-consistency, and cosine embedding losses.Cycle consistency reconstructs the class embedding, while cosine embedding encourages matched features to be similar and unmatched features dissimilar.
- Out-of-distribution detector for unseen class: The generator produces features for every unseen category after training, providing synthetic out-of-distribution samples for detector learning.Real seen features and generated unseen features jointly train the detector without assuming a prior seen-feature distribution.
- Out-of-distribution detector for unseen class: The detector maps seen inputs to low-entropy, class-emphasized outputs and unseen inputs to near-uniform, high-entropy outputs.Entropy is used to separate the two groups in entropy space rather than relying on a binary classifier.
- Out-of-distribution detector for unseen class: At inference, features whose detector entropy is below a threshold go to the seen classifier, while the remaining features are handled by the unseen classifier.A spatio-temporal CNN first computes the test video features before detector-based routing.
4. Experiments
Experiments on Olympic Sports, HMDB51, and UCF101 evaluate the proposed CEWGAN-OD framework against baselines and variants for generalized and standard zero-shot action recognition. CEWGAN-OD consistently improves performance and reduces bias toward seen classes.
- Experimental setup: Evaluations use Olympic Sports, HMDB51, and UCF101 with mean per-class accuracy averaged over 30 independent test runs.Video evaluation includes I3D and C3D features; the main GZSL baseline comparison uses concatenated I3D features.
- Baseline comparison: The proposed ODent detector outperforms the binary ODbin detector, while ODbin generally performs worse than CEWGAN as the number of classes increases.ODbin performs better than CEWGAN only on Olympic Sports, which has eight seen and unseen classes.
- Baseline comparison: 7.0%, 3.4%, and 4.9% accuracy gains over f-CLSWGAN are achieved on Olympic Sports, HMDB51, and UCF101, respectively, using word2vec.These gains are reported for generalized zero-shot action recognition with the proposed framework.
- State-of-the-art comparison: 16% and 19.8% gains over GGM are achieved on HMDB51 and UCF101, respectively, for generalized zero-shot recognition using word2vec.For UCF101 with manual attributes, CEWGAN-OD also gains 25.8% over GGM.
- State-of-the-art comparison: 6.6%, 4.9%, and 1.5% accuracy gains over the best existing ZSL results are obtained on Olympic Sports, HMDB51, and UCF101, respectively.The comparison uses CEWGAN with I3D features and word2vec embedding.
- Bias towards seen categories: 6.2% and 10.1% gains over CEWGAN improve unseen-class classification on HMDB51 and UCF101, respectively, using word2vec.The experiment treats seen and unseen features as two classes to quantify bias reduction by the standalone OD detector.
5. Conclusion
The paper integrates an out-of-distribution detector into generalized zero-shot action recognition to separate seen and unseen category classification and reduce seen-class bias. Across three human-action video datasets, it outperformed the baseline using word2vec embeddings.
- 7.0%, 3.4% and 4.9% absolute gains over the baseline were achieved on Olympic Sports, HMDB51 and UCF101, respectively, using word2vec embeddings.The approach was evaluated in generalized zero-shot action recognition across three human action video datasets.
- The proposed out-of-distribution detector separates classification of seen and unseen categories, reducing the bias towards seen classes present in baseline approaches.The detector uses real features from seen categories and GAN-generated features from unseen categories during training.