Source-linked AI summary
HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning
Shiming Chen, Guo-Sen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, Ling Shao
TL;DR
Existing ZSL common-space methods mainly address distribution disagreement, although visual and semantic representations also differ structurally. HSVA performs hierarchical structure and distribution adaptation, and reports consistent improvements across conventional and generalized ZSL benchmarks, with limitations tied to its Gaussian prior and training setup.
Problem
Existing common-space ZSL methods use one-step distribution alignment despite heterogeneous visual and semantic representations differing in both distribution and structure.
Method
HSVA uses two partially-aligned variational autoencoders, supervised adversarial discrepancy for structure adaptation, and Wasserstein alignment through a common encoder for distribution adaptation.
Results
HSVA achieves consistent improvement over current state-of-the-art methods on four ZSL benchmarks in conventional and generalized settings.
Takeaways & Limitations
Aligning both manifold structure and distribution yields a more discriminative and accurate common representation than distribution alignment alone, according to the paper's qualitative evidence.
Takeaways & Limitations
HSVA assumes standard normal prior distributions for the visual and semantic latent variables and trains using seen visual classes with unseen semantic classes.
Abstract
from arXiv · showhide
Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. Typically, to guarantee desirable knowledge transfer, a common (latent) space is adopted for associating the visual and semantic domains in ZSL. However, existing common space learning methods align the semantic and visual domains by merely mitigating distribution disagreement through one-step adaptation. This strategy is usually ineffective due to the heterogeneous nature of the feature representations in the two domains, which intrinsically contain both distribution and structure variations. To address this and advance ZSL, we propose a novel hierarchical semantic-visual adaptation (HSVA) framework. Specifically, HSVA aligns the semantic and visual domains by adopting a hierarchical two-step adaptation, i.e., structure adaptation and distribution adaptation. In the structure adaptation step, we take two task-specific encoders to encode the source data (visual domain) and the target data (semantic domain) into a structure-aligned common space. To this end, a supervised adversarial discrepancy (SAD) module is proposed to adversarially minimize the discrepancy between the predictions of two task-specific classifiers, thus making the visual and semantic feature manifolds more closely aligned. In the distribution adaptation step, we directly minimize the Wasserstein distance between the latent multivariate Gaussian distributions to align the visual and semantic distributions using a common encoder. Finally, the structure and distribution adaptation are derived in a unified framework under two partially-aligned variational autoencoders. Extensive experiments on four benchmark datasets demonstrate that HSVA achieves superior performance on both conventional and generalized ZSL. The code is available at \url{https://github.com/shiming-chen/HSVA} .
1 Introduction
ZSL common-space methods must align heterogeneous visual and semantic representations that differ in both distribution and structure. HSVA addresses this with sequential structure and distribution adaptation in a unified framework.
- Motivation: Existing common-space methods use one-step distribution alignment, neglecting structural variation between heterogeneous visual and semantic representations.This can leave the domains in different manifolds and limit performance.
- Proposed approach: HSVA learns an intrinsic common space through sequential structure adaptation and distribution adaptation under two partially-aligned variational autoencoders.The framework targets both structure- and distribution-aligned visual and semantic representations.
- Structure adaptation: Structure adaptation uses task-specific encoders and supervised adversarial discrepancy to reduce prediction discrepancy and bring visual and semantic manifolds closer.The SAD module operates through two task-specific classifiers.
- Distribution adaptation: Distribution adaptation uses a common encoder to map structure-aligned features into a distribution-aligned space while preserving their structure.The method minimizes Wasserstein distance between latent multivariate Gaussian distributions.
- Evaluation: HSVA reports consistent improvement over existing common-space learning methods across four benchmark datasets in conventional and generalized ZSL.The evaluated datasets are CUB, SUN, AWA1, and AWA2.
2 Related Work
ZSL methods learn relationships between visual features and semantic information in an embedding space, while domain adaptation provides alignment strategies for source and target representations. HSVA is framed against these approaches as a method for heterogeneous visual-semantic alignment.
- Zero-Shot Learning: ZSL uses semantic side information such as attributes, word vectors, and sentences to transfer knowledge from seen to unseen categories.Common-space learning is one of three embedding strategies alongside direct mapping and model parameter transfer.
- Zero-Shot Learning: Direct mapping projects visual features to semantic representations but faces high visual intra-class variability, reduced projected variance, and hubness.These limitations arise from mapping into a lower-dimensional semantic space.
- Zero-Shot Learning: Model parameter transfer operates in visual space and includes generative ZSL, but may be limited because unseen-class inter-class relationships are not considered.Generative methods learn semantic-to-visual mappings and augment data.
- Domain Adaptation: Domain adaptation learns domain-invariant representations by reducing source-target distribution discrepancy through alignment, adversarial learning, or task-specific methods.The field distinguishes homogeneous settings with shared feature spaces from heterogeneous settings.
- HSVA: HSVA uses two partially-aligned variational autoencoders to perform structure and distribution adaptation for visual-semantic feature alignment.Its framework is presented as a unified response to heterogeneous domain differences.
3 Hierarchical Semantic-Visual Adaptation
HSVA learns a common space for ZSL by adapting heterogeneous visual and semantic representations hierarchically. It first aligns their manifold structures, then aligns their distributions with partially-aligned variational autoencoders.
- Framework: HSVA addresses ZSL by learning an intrinsic common space for visual and semantic representations through structure and distribution adaptation.The framework uses two partially-aligned variational autoencoders and jointly optimizes their adaptation objectives.
- Variational Autoencoders: The visual and semantic branches use partially-aligned variational autoencoders with domain-specific encoders and decoders plus a shared common encoder.The visual branch contains Ex, Ez, and Dx, while the semantic branch analogously contains Ea, Ez, and Da; cross reconstruction further constrains the branches.
- Structure Adaptation: Structure adaptation uses task-specific encoders and classifiers to learn discriminative, structure-aligned visual and semantic embeddings.Classification losses train the encoders and classifiers, while adversarial discrepancy learning alternates classifier discrepancy maximization with encoder discrepancy minimization.
- Distribution Adaptation: Distribution adaptation maps structure-aligned features into a common space using a shared encoder while addressing remaining distribution variation.The method uses distribution-alignment losses, including correlation alignment, and minimizes discrepancies between encoded visual and semantic representations.
- Optimization: The full model jointly optimizes VAE, reconstruction, classification, discrepancy, and alignment losses with shared loss weights across datasets.The authors report consistently significant results on all datasets and characterize the model as robust and easy to train.
- Classification: After training, HSVA encodes seen and unseen visual features and semantic embeddings into the distribution-aligned space for supervised ZSL classification.The classifier uses encoded seen visual features and unseen semantic features for training, and the method is inductive because unseen visual features are not used during training.
4 Experiments
Experiments on four ZSL benchmarks evaluate HSVA under conventional and generalized settings, with quantitative, ablation, and qualitative evidence supporting its hierarchical alignment strategy.
- Experimental setup: Experiments use four benchmark datasets, CNN visual features, and top-1 accuracy for conventional ZSL plus seen, unseen, and harmonic-mean scores for generalized ZSL.The benchmarks include CUB, SUN, AWA1, and AWA2; generalized ZSL uses H = (2 × S × U)/(S + U).
- Conventional zero-shot learning: At least 5.3%, 3.0%, and 2.0% improvements over common-space methods are reported on AWA1, CUB, and SUN, respectively, for conventional ZSL.The comparison uses HSVA with a softmax classifier.
- Generalized zero-shot learning: At least 2.7%, 1.0%, 2.9%, and 2.7% harmonic-mean improvements over common-space models are reported on AWA1, AWA2, CUB, and SUN, respectively, for generalized ZSL.Against SGAL, the harmonic-mean gains are 5.3%, 1.0%, 8.3%, and 8.4% on those datasets; HSVA reaches a harmonic mean of 43.3 on SUN.
- Ablation study: Removing structure-adaptation constraints reduces harmonic mean by 4.5% on AWA1 and 2.1% on CUB, while omitting distribution-adaptation constraints produces very poor results.The ablation isolates structure and distribution adaptation components.
- Ablation study: HSVA’s iCORAL cooperation improves harmonic mean by 4.4%, 3.0%, 1.4%, and 1.5% on AWA1, AWA2, CUB, and SUN, respectively.LiCORAL pushes unseen classes away from seen classes to address seen-unseen bias.
- Qualitative results: t-SNE visualizations show CADA-VAE placing visual and semantic features in different manifolds, whereas HSVA learns discriminative common representations through sequential structure and distribution adaptation.The visualization covers 10 classes from CUB and AWA1; colors denote classes, with circles for visual and crosses for semantic features.
5 Conclusion
The conclusion presents HSVA as a common-space framework that addresses heterogeneous visual and semantic representations through structure and distribution adaptation. Across four benchmarks, it reports consistent improvements and qualitatively better alignment than methods using distribution alignment alone.
- Conclusion: HSVA uses two partially-aligned variational autoencoders to perform structure and distribution adaptation for an intrinsic visual-semantic common space.The framework is designed to address heterogeneous representations and bridge the visual-semantic gap.
- Conclusion: Across four ZSL benchmarks, HSVA achieves consistent improvement over current state-of-the-art methods.The conclusion reports qualitative verification alongside the benchmark improvements.
- Conclusion: HSVA aligns visual and semantic domains in both structure and distribution, whereas other common-space methods consider distribution alignment alone.The conclusion connects this difference with more discriminative and accurate learned representations.