Source-linked AI summary
FREE: Feature Refinement for Generalized Zero-Shot Learning
Shiming Chen, Wenjie Wang, Beihao Xia, Qinmu Peng, Xinge You, Feng Zheng, Ling Shao
TL;DR
GZSL methods often rely on ImageNet-trained features despite cross-dataset bias that can produce poor visual features and limit recognition of seen and unseen classes. FREE jointly refines features and semantic→visual mapping with class- and semantic-relevant losses, and experiments on five benchmarks show advantages over its baseline and current state-of-the-art methods.
Problem
Cross-dataset bias between ImageNet and GZSL benchmarks can limit knowledge transfer, producing poor-quality visual features for GZSL recognition.
Method
FREE jointly learns semantic→visual mapping and feature refinement in a unified generative model, using SAMC-loss and semantic cycle-consistency to learn discriminative representations.
Results
FREE demonstrates advantages over its baseline and current state-of-the-art methods on five GZSL benchmark datasets.
Takeaways & Limitations
Feature refinement is presented as an effective approach for addressing cross-dataset bias in generative GZSL methods.
Abstract
from arXiv · showhide
Generalized zero-shot learning (GZSL) has achieved significant progress, with many efforts dedicated to overcoming the problems of visual-semantic domain gap and seen-unseen bias. However, most existing methods directly use feature extraction models trained on ImageNet alone, ignoring the cross-dataset bias between ImageNet and GZSL benchmarks. Such a bias inevitably results in poor-quality visual features for GZSL tasks, which potentially limits the recognition performance on both seen and unseen classes. In this paper, we propose a simple yet effective GZSL method, termed feature refinement for generalized zero-shot learning (FREE), to tackle the above problem. FREE employs a feature refinement (FR) module that incorporates \textit{semantic$\rightarrow$visual} mapping into a unified generative model to refine the visual features of seen and unseen class samples. Furthermore, we propose a self-adaptive margin center loss (SAMC-loss) that cooperates with a semantic cycle-consistency loss to guide FR to learn class- and semantically-relevant representations, and concatenate the features in FR to extract the fully refined features. Extensive experiments on five benchmark datasets demonstrate the significant performance gain of FREE over its baseline and current state-of-the-art methods. Our codes are available at https://github.com/shiming-chen/FREE .
1. Introduction
GZSL must classify both seen and unseen classes, but cross-dataset bias can degrade transferred visual features. FREE addresses this issue by jointly refining visual features and semantic→visual mapping, guided by class- and semantic-relevant objectives.
- GZSL predicts both seen and unseen classes, making it more realistic and challenging than conventional ZSL.
- Cross-dataset bias between ImageNet and GZSL benchmarks causes distribution mismatch, limiting knowledge transfer and producing poor-quality visual features.The bias is larger for fine-grained datasets such as CUB than for coarse-grained datasets such as AWA.
- FREE jointly optimizes feature refinement with semantic→visual mapping in a unified generative model to improve visual features for GZSL.
- SAMC-loss encourages intra-class compactness and inter-class separability, while semantic cycle-consistency guides semantically relevant feature learning.
- FREE concatenates discriminative features from multiple refinement layers to extract fully refined features for classification.
- Experiments on five benchmarks demonstrate advantages over the f-VAEGAN baseline and current state-of-the-art methods.
2. Related Work
Prior GZSL work addresses the visual-semantic domain gap and seen-unseen bias through embedding, mapping, generative, and distribution-alignment methods. The paper frames cross-dataset bias as an additional obstacle and proposes unified feature refinement to address it.
- Visual-Semantic Domain Gap: Generative GZSL methods learn visual-semantic mappings and generate unseen-class features for data augmentation.
- Cross-Dataset Bias: Cross-dataset bias is distribution mismatch between datasets that limits knowledge transfer and yields poor-quality features on new GZSL benchmarks.
- Cross-Dataset Bias: Domain adaptation is unsuitable for this setting because GZSL features are extracted before sequential learning, while fine-tuning can cause inefficiency and overfitting.
- Cross-Dataset Bias: FREE uses feature refinement in a unified network to improve semantic→visual mapping while circumventing cross-dataset bias.
3. Method
FREE refines GZSL visual features within a unified generative framework to address cross-dataset bias. Its FR module uses adaptive class-center supervision, semantic cycle consistency, and concatenated representations to produce class- and semantically-relevant features for classification.
- Motivation: Cross-dataset bias can produce poor-quality GZSL visual features and limit semantic→visual learning, feature synthesis, and classification.The method targets the mismatch between ImageNet-pretrained feature extraction and GZSL benchmark data.
- Overview: FREE jointly optimizes f-VAEGAN and feature refinement to couple semantic→visual mapping with discriminative representation learning.Stage 1 learns semantic-conditioned visual synthesis while FR processes real and synthesized seen features.
- Self-Adaptive Margin Center Loss: SAMC-loss balances intra-class compactness and inter-class separability using class centers, a margin, and an adaptive coefficient γ.The coefficient is larger for fine-grained datasets and smaller for coarse-grained datasets.
- Feature Refinement: Semantic cycle-consistency loss guides FR toward semantically relevant representations, while residual information preserves discriminative content discarded by dimensionality reduction.FR combines latent and semantically relevant embeddings with original residual information to form fully refined features.
- Extracting Fully Refined Features: Fully refined features concatenate FR’s latent and semantically relevant embeddings for seen and unseen samples before classifier training and testing.Refined real seen and synthetic unseen features train the classifier, while refined real unseen features are used for testing.
- Optimization: The unified network jointly trains the encoder, generator, discriminator, and FR using a weighted combination of objective losses.Alternating updates couple semantic→visual mapping and visual feature refinement during optimization.
4. Experiments
FREE is evaluated on five benchmark datasets using unified GZSL protocols, component ablations, feature analyses, and hyperparameter studies. It consistently improves harmonic-mean performance and benefits from refined feature components, while dataset granularity affects some hyperparameter trends.
- Comparison with State of the Arts: FREE attains the best harmonic mean on AWA1, AWA2, and CUB, with scores of 66.0, 67.1, and 57.7, respectively.It achieves second-best harmonic means of 41.7 on SUN and 75.0 on FLO.
- Ablation Study: The complete FREE model improves harmonic mean over its baseline by 4.6% on CUB, 10.1% on FLO, and 4.4% on AWA2.The combined SAMC-loss and semantic cycle-consistency loss are reported as mutually complementary for feature refinement.
- Ablation Study: FR independently improves baseline harmonic mean on CUB, FLO, and AWA2, with SAMC-loss gains of 3.4%, 7.1%, and 2.5%.The corresponding FR(LR_a) gains are 2.6%, 3.2%, and 1.7%; FR(L_SAMC) performs better than FR(L_R_a).
- Feature Components: Adding hidden features and learned semantically relevant features to FR features produces further improvements, while FR improves unseen accuracy and harmonic mean even on coarse-grained AWA2.The analysis attributes these gains to improved visual-feature quality and reduced cross-dataset bias.
- Hyperparameter Analysis: As γ increases, S, U, and H consistently improve on fine-grained CUB but decrease on coarse-grained AWA2.The reported explanation contrasts stronger gains from intra-class compactness on confused fine-grained classes with inter-class separability benefits for ambiguous coarse-grained classes.
- Hyperparameter Analysis: FREE is generally insensitive to the number of synthesized unseen-class features, while more synthetic features slightly reduce seen accuracy and improve unseen accuracy.The authors interpret this trend as evidence that FREE can alleviate seen-unseen bias.
5. Discussion
FREE uses feature refinement to address cross-dataset bias in GZSL, combining refinement with semantic→visual mapping and discriminative losses. The discussion reports improved feature separation and competitive results relative to fine-tuning, while noting fine-tuning’s seen-unseen overfitting risk.
- FR for Cross-Dataset Bias: FR significantly improves seen and unseen visual features, reducing ambiguity between categories and preserving similar class relationships between synthesized and real unseen features.These observations support the paper’s claim that FR learns a promising semantic→visual mapping.
- Feature Refinement vs Fine-tuning: FREE jointly combines FR with semantic→visual mapping, while SAMC-loss and semantic cycle-consistency loss guide discriminative feature refinement.The authors describe these components as mutually beneficial for alleviating limited knowledge transfer and reducing cross-dataset bias.
- Feature Refinement vs Fine-tuning: Fine-tuning can alleviate cross-dataset bias but may cause seen-unseen overfitting, with a reported seen-unseen bias of 29.1% on FLO.FR achieves competitive results over fine-tuning on FLO and AWA2.
6. Conclusion
FREE couples semantic→visual mapping with feature refinement to alleviate cross-dataset bias in GZSL. It uses SAMC-loss, semantic cycle-consistency, and multi-layer features to learn refined representations for classification.
- FREE couples semantic→visual mapping and feature refinement to alleviate cross-dataset bias in GZSL.
- SAMC-loss and semantic cycle-consistency encourage FR to learn class- and semantically-relevant feature representations.
- FREE extracts features from multiple FR layers as fully refined features for classification.
- Competitive results on five popular benchmarks demonstrate the superiority and potential of the approach.