PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Zero-shot Recognition

Yanwei Fu, Tao Xiang, Yu-Gang Jiang, Xiangyang Xue, Leonid Sigal, Shaogang Gong

arXiv:1710.04837v1cs.CVcs.AIcs.LGcs.MMstat.ML

TL;DR

Zero-shot recognition seeks to scale recognition to categories with few or no labelled examples. This review synthesizes semantic representations, recognition models, related tasks, datasets, and evaluation issues, while identifying limitations and future directions.

  • Problem

    Recognition systems commonly require many labelled samples per class, motivating recognition of unseen categories through knowledge transferred from source data.

  • Method

    The paper reviews semantic representations, zero-shot recognition models, one-shot and open-set extensions, datasets, and evaluation settings.

  • Results

    The review compares representation and model families, discusses dataset and evaluation issues, and identifies future research directions.

  • Takeaways & Limitations

    Zero-shot recognition relies on shared semantic representations and prototype-based transfer to recognize target classes without labelled target instances.

  • Takeaways & Limitations

    Zero-shot methods can suffer projection domain shift because source-trained visual-to-semantic projections may be biased for target classes.

Abstract

from arXiv · show

With the recent renaissance of deep convolution neural networks, encouraging breakthroughs have been achieved on the supervised recognition tasks, where each class has sufficient training data and fully annotated training data. However, to scale the recognition to a large number of classes with few or now training samples for each class remains an unsolved problem. One approach to scaling up the recognition is to develop models capable of recognizing unseen categories without any training instances, or zero-shot recognition/ learning. This article provides a comprehensive review of existing zero-shot recognition techniques covering various aspects ranging from representations of models, and from datasets and evaluation settings. We also overview related recognition tasks including one-shot and open set recognition which can be used as natural extensions of zero-shot recognition when limited number of class samples become available or when zero-shot recognition is implemented in a real-world setting. Importantly, we highlight the limitations of existing approaches and point out future research directions in this existing new research area.

I. INTRODUCTION

Zero-shot recognition addresses the need to recognize novel categories without labelled target examples by transferring knowledge through shared semantic representations. The review situates this problem alongside one-shot and open-set recognition and surveys its models, datasets, evaluation issues, and future directions.

  • Existing computer-vision methods commonly require hundreds or thousands of labelled samples per object class, unlike human category learning from descriptions or few examples.
  • Zero-shot, one-shot, and open-set recognition target novel categories with no samples, few examples, or tests containing both seen and unseen categories.
  • Zero-shot recognition transfers knowledge from source data because target categories have few or no labelled instances, making domain-invariant and generalizable representations central challenges.
  • The review covers semantic representations, recognition models, related one-shot and open-set tasks, datasets, evaluation problems, and future research directions.
  • Shared semantic representations encode class names as attribute or word vectors, enabling knowledge transfer between source and target datasets.
  • Recognition typically learns a projection from visual features to semantic space, then matches projected target instances with unseen-class prototypes.

III. SEMANTIC REPRESENTATIONS IN ZERO-SHOT RECOGNITION

Semantic representations provide intermediate descriptions that connect visual data with class concepts and support transfer under sparse or zero training examples. The reviewed representations include attributes, ontologies, and properties derived from visual or textual information.

  • Semantic Attributes: Attributes describe intrinsic characteristics, properties, or annotations of instances and classes rather than merely naming the instance.
  • Semantic Attributes: Attribute descriptions bridge low-level visual features and high-level class concepts, supporting recognition when class-level training data are sparse.
  • Semantic Attributes: Compound attributes extend unary and binary attributes to richer combinations useful for retrieval and identification.
  • Semantic Attributes: Subjective visual properties treat attributes as continuous values, including interestingness, memorability, aesthetics, and estimated age.
  • Semantic Attributes: User-defined attributes may come from experts or concept ontologies and vary across facial, clothing, biological, product, and 3D-shape tasks.
  • Semantic Attributes: Generative models have represented elementary color and geometric properties as unary attributes of individual segments or binary attributes involving pairs of segments.

2) Relative Attributes:

Relative attributes represent richer visual semantics by ranking the strength of properties, while data-driven attributes seek discriminative representations beyond manually defined vocabularies. Both approaches address limitations of sparse or incomplete semantic descriptions.

  • Relative Attributes: Relative attributes support zero-shot recognition and applications including interactive search, semi-supervised learning, and active learning.
  • Relative Attributes: Relative attributes learn ranking functions that estimate the comparative strength of visual properties from pairwise image judgments.
  • Relative Attributes: Weakly supervised pairwise comparisons can train deep models to localize and rank relative attributes jointly.
  • Data-driven Attributes: Manually defined attributes can be costly to create and may be descriptive without being sufficiently discriminative, motivating data-driven attribute discovery.
  • Data-driven Attributes: Exhaustive attribute vocabularies are unlikely because ontology creation is expensive, human-obvious attributes may be undetectable, and collected labels can remain noisy.
  • Data-driven Attributes: Existing data-driven attribute studies often learn user-defined and mined attributes separately, and their discovered attributes lack semantic names for direct zero-shot use.

4) Video Attributes:

Video attributes represent concepts, actions, scenes, and events for zero-shot video understanding, while textual and ontology-based representations provide alternative semantic spaces for transferring knowledge.

  • Video Attributes: Video attributes encompass objects, scenes, actions, and events and support video concept detection and zero-shot event recognition.
  • Video Attributes: Video concept detection can be treated as video attribute learning for zero-shot video event detection, with tagging spaces constrained by concept ontologies.
  • Video Attributes: Video events are higher-level semantic entities composed of multiple concepts or video attributes, such as blowing candles and birthday cake in a birthday party.
  • Concept Ontology: Concept ontologies organize lexical items into cognitive synonym sets and define semantic distances used to transfer information between categories.
  • Semantic Word Vectors: Semantic word-vector spaces encode relationships through distributed linguistic representations, allowing class names to serve as prototypes for zero-shot recognition.
  • Embedding Models: Embedding models project visual features near semantic prototypes, after which recognition compares projected instances with unseen-class prototypes.

A. Embedding Models

Bayesian zero-shot models use semantic attributes to connect visual inputs with unseen class categories. DAP learns attribute classifiers directly, whereas IAP derives them by combining probabilities over known classes.

  • Bayesian Models: Bayesian generative models integrate prior knowledge about attributes to compensate for limited supervision of novel classes.The reviewed formulation first models p(z|x) from attribute probabilities and class-conditional relationships.
  • Bayesian Models: DAP and IAP were early zero-shot models that learned embeddings with SVMs and performed recognition using Bayesian formulations.They established a single-model pipeline for categories without training examples.
  • DAP Model: DAP trains attribute classifiers from attribute annotations of known classes using a binary class–attribute association matrix.The matrix records whether each attribute is present in each known or unseen class, with associations supplied by experts, ontologies, or semantic relatedness.
  • IAP Model: IAP builds attribute classifiers by combining the probabilities of associated known classes rather than learning them directly from the known classes.It estimates p(y_k|x) for training classes, combines those probabilities into p(a|x), and then applies the result as in DAP.
  • Label Embedding: Label embedding reframes attribute-based image classification as minimizing a compatibility function between an image and a label embedding.The reviewed work derives a modified ranking objective from WSABIE.

2) Semantic Embedding:

Semantic embedding maps visual features into spaces defined by attributes, text, labels, or shared representations. Recognition then compares projected instances with class prototypes, while multi-view, manifold, latent, and feature-synthesis models extend this framework.

  • Semantic Embedding: Joint embedding methods learn a common space linking visual information with attributes, text, and hierarchical relationships.Examples include jointly embedding attributes, text, and hierarchy, or using text to predict convolutional and fully connected layer weights.
  • Semantic Embedding: Multi-view embedding fuses complementary semantic representations, including class-label or label-relation graphs, to improve recognition performance.The reviewed methods combine scores or relations across semantic views.
  • Semantic Embedding: CCA-based models embed images and associated text into comparable spaces, with three-view extensions using additional views for retrieval.Inter-modal label transfer from text and image features can be generalized to zero-shot recognition.
  • Semantic Embedding: Deep models such as DeViSE learn visual–semantic embeddings using labeled images and semantic information from unannotated text.ConSE instead maps images into semantic space through convex combinations.
  • Recognition in the Embedding Space: After projection, nearest-neighbor recognition assigns labels by comparing testing instances with class prototypes in the embedding space.Other recognition models use hypergraphs, label propagation, synthesized classifiers, latent variables, or synthesized unseen visual features.
  • Projection Domain Shift: The projection domain shift figure contrasts red zero-shot prototypes with blue predicted attribute projections for Pig and Zebra sharing the hasTail attribute.The shared attribute corresponds to different visual appearances, illustrating why projected instances can deviate from target prototypes.

C. Problems in Zero-shot Recognition

Zero-shot recognition faces projection domain shift and hubness, while conventional evaluation assumes testing classes are unseen only. Generalized zero-shot recognition relaxes that assumption by including seen classes at test time.

  • Projection Domain Shift: Projection domain shift occurs when source-trained visual-to-semantic projections do not transfer cleanly to target classes with different data distributions.Pig and Zebra may share hasTail while exhibiting different visual appearances, creating a discrepancy between target prototypes and predicted projections.
  • Projection Domain Shift: Transductive methods use unseen-class manifold information to alleviate projection shift, but require access to all testing data before learning.That assumption is invalid when new unseen classes appear dynamically and were unavailable during model learning.
  • Hubness: Hubness is the emergence of universal neighbors, or hubs, in the embedding space.Its causes remain debated: one account treats it as inherent to high-dimensional distributions, while another attributes it to boundary or density-gradient effects.
  • Hubness: Regression-based zero-shot methods can suffer from hubness, addressed through transductive unseen-data distributions or inductive prototype embedding.The reviewed approaches use different information regimes to reduce the phenomenon.
  • Evaluation Settings: Conventional zero-shot recognition assumes source and target classes are disjoint and that test instances come only from unseen classes.This closed separation simplifies recognition relative to settings where seen and unseen classes coexist.
  • Evaluation Settings: Generalized zero-shot recognition includes training classes among test data and is recommended as a more practical evaluation setting.Directly extending existing zero-shot methods to this setting is nontrivial and ineffective.

1) Conventional open set recognition:

Conventional open-set recognition determines whether inputs belong to training or unseen classes without identifying the specific unseen class. Related generalized open-set and one-shot settings expand recognition to semantic prediction or limited target examples.

  • Conventional Open-Set Recognition: Conventional open-set recognition identifies whether a testing image comes from training classes or unseen classes.It does not explicitly predict which unseen class contains the instance.
  • Generalized Open-Set Recognition: Generalized open-set recognition additionally predicts the semantic class of testing instances from unseen novel categories.The paper characterizes it as a most general version of zero-shot recognition under limited training data.
  • One-Shot Recognition: One-shot learning assumes one or a few training samples per new class instead of only a textual description.It shares with zero-shot recognition the motivation of learning new categories from limited information.
  • One-Shot Recognition: Early one-shot approaches train standard classifiers directly on target classes without auxiliary related classes supplying transferable knowledge.Nonparametric models such as kNN avoid dependence on sample count, but their distance metric is not learned.
  • Transfer Learning-Based One-Shot Recognition: Transfer-learning approaches use auxiliary classes and meta-learning to transfer models, features, attributes, or contextual information to target classes with few examples.The approaches differ in both the transferred knowledge and its representation.
  • Transfer Learning-Based One-Shot Recognition: Deep meta-learning can adapt models to new classes with one or a few gradient steps, but lifelong learning with progressively added few-shot classes remains unsolved.Model-agnostic meta-learning trains on auxiliary data for effective fine-tuning on new classes.

VI. DATASETS IN ZERO-SHOT RECOGNITION

The paper surveys datasets used for zero-shot recognition, spanning general and fine-grained image settings and datasets with attribute annotations. It highlights benchmark standardization through unified evaluation protocols and data splits.

  • Xian et al. unified evaluation protocols and data splits into a benchmark for comparing zero-shot recognition methods.
  • AwA contains 30,475 images from 50 animal categories, with 10 designated test classes and 40 classes available for training.AWA2 provides 37,322 publicly licensed images from the same 50 classes and 85 attributes.
  • aPascal-aYahoo combines 12,695 PASCAL images and 2,644 Yahoo images across 32 object classes, with 64 binary attributes per image.
  • CUB-200-2011 contains 11,788 images from 200 bird classes and provides bounding boxes, part locations, and attribute labels for fine-grained recognition.
  • OSR contains 2,688 images from 8 categories with 6 attributes and sparse pairwise attribute annotations collected through majority voting.

4) Outdoor Scene Recognition (OSR) Dataset [183]:

The section describes datasets for outdoor scenes, social activities, large-scale image recognition, flowers, and human actions. These datasets vary in scale, granularity, modalities, and annotation structure.

  • PubFig-sub uses 772 images from 8 people and 11 attributes, with pairwise attribute labels collected from five Amazon Mechanical Turk workers per pair.
  • The SUN subset contains 14,340 images from 717 scene classes, with 102 binary attributes describing materials, lighting, functions, affordances, and layout.
  • USAA provides videos from 8 social activity classes with 69 attributes spanning actions, objects, scenes, sounds, and camera movement.Using ground-truth attributes directly with an SVM yields 86.9% classification accuracy, while instance-level attributes remain insufficient for perfect classification.
  • ImageNet contains over 15 million high-resolution labeled images across roughly 22,000 categories.
  • Oxford 102 contains 8,189 images organized into 102 flower groups, while UCF101 contains 13,320 video clips across 101 annotated action classes.

11) Fudan-Columbia Video Dataset (FCVID) [192]:

The section presents large-scale video datasets and cautions that zero-shot comparisons depend on dataset choices, feature representations, auxiliary data, and evaluation splits.

  • FCVID contains 91,223 manually annotated web videos across 239 categories, with 45,611 videos for training and 45,612 for testing.
  • ActivityNet contains 27,801 clips from 203 activity classes totaling 849 hours, with both trimmed and untrimmed video settings.
  • Table II divides zero-shot datasets into general image classification, fine-grained image classification, and video classification groups.
  • Different deep feature types and fine-tuning choices can change representation ability, making comparisons unfair when methods use different features.Performance improvements may largely reflect better deep features rather than the zero-shot method itself.
  • Auxiliary source data and resources used to learn concept ontologies or semantic word vectors can substantially affect zero-shot recognition performance.
  • Many datasets lack agreed source-target splits, motivating unified benchmarks that standardize evaluation protocols and data splits.

VII. FUTURE RESEARCH DIRECTIONS

The paper identifies future directions beyond restrictive zero-shot settings, including lifelong class expansion, joint zero- and few-shot learning, richer visual concepts, and curriculum-based class selection.

  • Existing methods mainly classify known unseen object classes, whereas practical systems should add new classes progressively without harming recognition of existing classes.
  • Zero-shot and few-shot learning are closely related, but existing methods generally do not jointly use few examples and textual or human knowledge about new classes.
  • Future zero-shot research should model visual concepts beyond object categories, including attributes whose meanings depend on the associated objects.The paper gives ‘yellow face’ and ‘yellow banana’ as examples of context-dependent attribute meaning.
  • In lifelong learning, the order of adding classes affects model performance, motivating curriculum-learning principles for designing zero-shot learning sequences.
  • The review covers semantic representations, zero-shot models, one-shot and open-set recognition, datasets, evaluation issues, and future research directions.
Loading 1710.04837v1…