Source-linked AI summary
From Zero-shot Learning to Conventional Supervised Classification: Unseen Visual Data Synthesis
Yang Long, Li Liu, Ling Shao, Fumin Shen, Guiguang Ding, Jungong Han
TL;DR
Collecting sufficient real images for expanding or rare classes is difficult, limiting conventional recognition and existing ZSL approaches. The paper synthesizes unseen visual features from semantic attributes, converts ZSL into supervised classification, and reports improved state-of-the-art performance across four benchmark datasets.
Problem
Object recognition requires many real images, but collecting labeled data for expanding and rare unseen classes is difficult, while existing ZSL methods cannot expand training data.
Method
UVDS synthesizes unseen visual features from semantic attributes and uses them as training data for conventional classifiers, with regularisation addressing structural difference, training bias, and variance decay.
Results
The approach outperformed state-of-the-art methods on all four benchmark datasets.
Takeaways & Limitations
Synthesizing unseen visual data enables ZSL recognition to be handled as conventional supervised classification without acquiring real images.
Takeaways & Limitations
The method relies on the Local Invariance assumption to estimate visual and semantic data structures through a spectral Dual-Graph approach.
Abstract
from arXiv · showhide
Robust object recognition systems usually rely on powerful feature extraction mechanisms from a large number of real images. However, in many realistic applications, collecting sufficient images for ever-growing new classes is unattainable. In this paper, we propose a new Zero-shot learning (ZSL) framework that can synthesise visual features for unseen classes without acquiring real images. Using the proposed Unseen Visual Data Synthesis (UVDS) algorithm, semantic attributes are effectively utilised as an intermediate clue to synthesise unseen visual features at the training stage. Hereafter, ZSL recognition is converted into the conventional supervised problem, i.e. the synthesised visual features can be straightforwardly fed to typical classifiers such as SVM. On four benchmark datasets, we demonstrate the benefit of using synthesised unseen data. Extensive experimental results suggest that our proposed approach significantly improve the state-of-the-art results.
1. Introduction
Zero-shot learning addresses the difficulty of collecting labeled images for expanding and rare classes by using semantic descriptions. This paper synthesizes unseen visual features from attributes, converts ZSL into supervised classification, and targets structural, bias, and variance-decay problems.
- Motivation: Conventional object-recognition systems depend on many labeled images, but annotation costs, expanding fine-grained categories, and rare classes make data collection difficult.The introduction cites newly released handbag patterns and ancient species as examples of these constraints.
- Motivation: Zero-shot learning recognizes unseen classes from semantic descriptions, but fixed seen-class data cannot expand to represent ever-growing concepts.Existing methods map visual data to semantic representations without adding training data for unseen classes.
- Approach: The proposed framework synthesizes unseen visual features from semantic attributes without acquiring real images, converting ZSL into conventional supervised classification.Synthesized data can be fed directly to conventional classifiers.
- Approach: The framework introduces Diffusion Regularisation to address variance decay by diffusing information across synthesized-data dimensions through an orthogonal rotation problem.The method also addresses structural difference and training bias.
2. Related Work
Prior ZSL methods use semantic side information to recognize unseen classes, commonly by mapping visual features into semantic space. This paper instead synthesizes visual features from semantic representations, while attribute-based methods remain prominent because attributes are compact and human-understandable.
- Zero-shot Recognition Schemes: Zero-shot learning avoids acquiring visual data for novel classes because collecting well-labeled examples is expensive.Early approaches include Direct-Attribute Prediction, which uses attributes as intermediate clues for classification.
- Zero-shot Recognition Schemes: Embedding-based frameworks map visual features to attributes jointly and infer class labels in semantic space using similarity measurements.Transductive variants may use unseen-class data and visual similarity or distribution information.
- Zero-shot Recognition Schemes: The proposed method reverses the usual inference direction by synthesizing visual feature vectors from semantic instances rather than mapping visual data to label space.This preserves the advantages of semantic embedding while changing the synthesis direction.
- Semantic Side Information: Text, pseudo-concepts, and ontological hierarchies are alternative forms of semantic side information for ZSL.The paper situates attributes within a broader set of side-information approaches.
- Semantic Side Information: Attribute-based ZSL remains popular because it often performs strongly and represents image properties compactly with human-understandable words.Relative, class-similarity, and augmented attributes are cited as extensions.
3. Approach
UVDS learns an intermediate latent embedding and projections that synthesise unseen visual features from semantic attributes. Dual-graph structure preservation and diffusion regularisation address visual-semantic structural differences, training bias, and variance decay before unseen features are classified by conventional supervised models.
- Problem formulation: The framework represents training samples as visual features, semantic attributes, and seen-class labels, while unseen visual samples remain unavailable during training.It supports either class-level or image-level attributes and aims to learn a classifier for unseen classes.
- Unseen feature synthesis: UVDS maps semantic attributes into synthesised visual features so unseen data can be generated before testing and supplied to conventional classifiers such as SVM.The method learns a mapping through an auxiliary latent space rather than requiring real unseen images.
- Latent embedding: The latent embedding V reconciles visual and semantic spaces, while projection matrices P and Q connect attributes, latent representations, and visual features.V is designed to preserve intrinsic structure from both spaces instead of relying only on a direct attribute-to-visual mapping.
- Structure preservation: Dual-graph regularisation combines visual and attribute graphs to preserve local structure across the two spaces despite their visual-semantic gap.The visual graph and semantic graph encode relationships among training instances, and their mean defines the combined graph.
- Diffusion regularisation: Variance decay occurs when D ≫ M causes projections to concentrate information in a few synthesised dimensions, making the representation redundant.The method uses an orthogonal rotation to diffuse information across dimensions while preserving overall variance.
- Diffusion regularisation: Diffusion regularisation maximises the sum of diffused standard deviations through an orthogonal rotation and an ℓ2,1-based objective.The overall loss jointly reconstructs visual features from attributes, preserves data structure, and diffuses information across dimensions.
- Optimisation strategy: UVDS is optimised alternately through V-, Q-, and P-steps, using a Sylvester-equation solution for V, Cayley transformation updates for Q, and least squares for P.The Q-step maintains the orthogonality constraint, while the alternating procedure seeks a local optimum.
2 Repeat
The optimisation alternates updates to the visual features, auxiliary matrix, and projection matrix, using specified equations until convergence.
- The V-step fixes P and Q, then updates V using Eq. (12).
- The Q-step fixes P and V, then updates Q through the prescribed iterative procedure.
- The procedure computes the Cayley matrix H^t using Eq. (15) during the Q update.
- It computes Q^{t+1} using Eq. (14) and stops when convergence is reached.
- The P-step fixes V and Q, then updates P using Eq. (18).
4. Experiments
Experiments evaluate the method on four benchmark datasets and compare it with published methods and baselines. Results support robust performance across attribute settings, classifiers, and feature types, while diffusion regularisation addresses variance concentration.
- Settings: The evaluation uses four benchmark datasets with published seen/unseen splits and excludes methods using different settings or semantic information.
- Comparison with the State-of-the-art methods: The method steadily outperforms state-of-the-art methods in conventional ZSL scenarios and can also exceed two transductive results.
- Comparison with the State-of-the-art methods: Deep features improve performance, but the method also achieves acceptable results with low-level features, while SVM often further improves recognition rates.
- Detailed Evaluations: The method prevents training bias, whereas unregularised linear regression suffers 30% average performance degradation from seen to unseen classes.
- Detailed Evaluations: Diffusion regularisation complements graph regularisation, and the method applies reliably to both class-level and image-level attributes.
- Detailed Evaluations: Without diffusion regularisation, synthesised variance concentrates in a few dimensions; with it, variance becomes more balanced than in real data.
- Detailed Evaluations: Nearest-neighbour retrieval shows both successful same-class matches and failures caused by semantic ambiguity or visually nondistinct classes.
5. Conclusion
The paper concludes that semantic attributes can synthesise unseen-class visual data, while regression-based embedding faces structural difference, training bias, and variance decay. The proposed approach addresses these problems and outperforms state-of-the-art methods on four benchmark datasets.
- The approach outperformed state-of-the-art methods on all four benchmark datasets.
- The proposed algorithm synthesises visual data for unseen classes using semantic attributes.
- Direct regression-based embedding can produce low recognition rates because of structural difference, training bias, and variance decay.