Source-linked AI summary
Transductive Unbiased Embedding for Zero-Shot Learning
Jie Song, Chengchao Shen, Yezhou Yang, Yang Liu, Mingli Song
TL;DR
Existing ZSL methods are strongly biased toward seen source classes, hurting generalized ZSL when test images include both source and target categories. QFSL uses transductive training with labeled source and unlabeled target data, mapping them toward source- and target-specified semantic points. Across the reported benchmarks, the method outperforms existing ZSL approaches in generalized and conventional settings.
Problem
Strong bias toward seen source classes causes existing ZSL methods to perform poorly in generalized settings.
Method
QFSL jointly trains on labeled source and unlabeled target data, mapping source images to source-specified points and target images to other target-specified points.
Results
QFSL significantly outperforms existing ZSL methods in both generalized and conventional settings.
Takeaways & Limitations
Using unlabeled target data helps QFSL alleviate source-class bias while building visual-semantic connections.
Takeaways & Limitations
QFSL assumes labeled source and unlabeled target data are available during training and leaves inductive solutions for future work.
Abstract
from arXiv · showhide
Most existing Zero-Shot Learning (ZSL) methods have the strong bias problem, in which instances of unseen (target) classes tend to be categorized as one of the seen (source) classes. So they yield poor performance after being deployed in the generalized ZSL settings. In this paper, we propose a straightforward yet effective method named Quasi-Fully Supervised Learning (QFSL) to alleviate the bias problem. Our method follows the way of transductive learning, which assumes that both the labeled source images and unlabeled target images are available for training. In the semantic embedding space, the labeled source images are mapped to several fixed points specified by the source categories, and the unlabeled target images are forced to be mapped to other points specified by the target categories. Experiments conducted on AwA2, CUB and SUN datasets demonstrate that our method outperforms existing state-of-the-art approaches by a huge margin of 9.3~24.5% following generalized ZSL settings, and by a large margin of 0.2~16.2% following conventional ZSL settings.
1. Introduction
ZSL addresses recognition with limited labeled data by connecting source and target categories through a shared semantic space. The paper targets strong source-class bias in generalized settings with transductive QFSL, using labeled source and unlabeled target data together.
- Motivation: ZSL uses labeled source categories and a shared semantic space to recognize novel target categories without sufficient labeled images for each category.The semantic space provides a common representation for projected images and class names.
- ZSL Settings: Transductive ZSL trains with both labeled source data and unlabeled target data, whereas inductive ZSL uses only source data.Transductive methods aim to use information from both data sources for ZSL.
- ZSL Settings: Generalized settings evaluate images from both source and target classes, unlike conventional settings that restrict test images to target classes.The broader search space makes generalized evaluation more practical than the conventional assumption.
- Problem: Existing methods exhibit strong bias because source-class anchor points dominate the visual-to-semantic mapping, causing target images to be classified as source classes.This bias contributes to much poorer generalized-setting performance than conventional-setting performance.
- Proposed Method: QFSL uses labeled source data to learn visual-semantic relationships and unlabeled target data to map inputs to target-related points rather than only fixed source anchors.The model is trained end-to-end to recognize source and target data without target labels.
- Results: QFSL is reported to significantly outperform existing ZSL methods in both generalized and conventional settings.The contribution statement presents this as the method’s principal empirical finding.
2. Related Work
Related work organizes ZSL by semantic-space choice, visual-semantic projection pathway, and access to unlabeled target data. QFSL belongs to transductive ZSL and learns an unbiased embedding from labeled source and unlabeled target data.
- Semantic Spaces: ZSL semantic spaces include attributes, word vectors, text descriptions, and human gaze; attributes are effective but require expensive human labor.Word vectors are presented as an alternative receiving increasing attention.
- Embedding Pathways: Existing methods relate visual and semantic spaces through visual-to-semantic mapping, semantic-to-visual mapping, or projection of both into a shared intermediate space.Classification can then use nearest-neighbor search in the resulting representation.
- Transductive ZSL: Transductive ZSL uses labeled source and unlabeled target data in semi-supervised learning, including label propagation, cross-domain association, and shared-model-space approaches.Prior methods exploit manifold structure, CCA, sparse coding, or attribute-based knowledge transfer.
- QFSL: QFSL leverages labeled source and unlabeled target data to learn an unbiased embedding space for ZSL.The proposed architecture uses the same model for both labeled and unlabeled data, although the streams are depicted separately for clarity.
- Generalized Settings: Generalized ZSL evaluates recognition across seen and unseen classes, relaxing conventional ZSL’s assumption that test instances belong only to unseen classes.This setting reflects the practical need to recognize both source and target categories at test time.
3. Quasi-Fully Supervised Learning
QFSL uses a transductive deep neural network to classify source and target classes jointly, while an additional bias loss uses unlabeled target data to reduce source-class bias.
- Problem formulation: QFSL assumes labeled source data, unlabeled target data, and semantic embeddings are available during training for conventional and generalized settings.The source and target class sets are disjoint, with S source classes and T target classes.
- Network architecture: The model combines visual embedding, visual-semantic bridging, scoring, and classification modules in an end-to-end deep neural network.The visual embedding subnet maps images to visual embeddings, the bridging subnet maps them to semantic embeddings, and the scoring subnet produces class scores.
- Visual embedding: The visual embedding function is jointly optimized with other modules, unlike methods that keep deep CNN visual features fixed.A pre-trained CNN supplies the visual embedding, typically using the first fully connected layer's output.
- Visual-semantic bridging: A nonlinear bridging function maps visual embeddings to semantic embeddings, where classification uses inner products with normalized class semantic embeddings.The bridging function uses fully connected layers followed by ReLU activations.
- Scoring and classification: The scoring subnet is initialized with normalized semantic vectors for all S + T classes and remains frozen during training.The classifier produces probabilities over all source and target classes, and prediction selects the class with the highest probability.
- QFSL optimization: The QFSL loss adds a bias loss L_b to conventional classification and regularization losses, encouraging unlabeled target instances to receive probability mass among target classes.This discourages target instances from being mapped to source classes; training mixes labeled and unlabeled data in stochastic-gradient-descent batches.
4. Experiments
Experiments evaluate QFSL on AwA2, CUB, and SUN using conventional and generalized ZSL settings, examining architecture, visual-subnet optimization, loss weighting, and comparisons with existing methods. QFSL generally improves performance, especially in generalized settings, while data scarcity affects the benefit of optimizing the visual embedding subnet.
- Experimental settings: Experiments use AwA2, CUB, and SUN, with class-level attributes and four ImageNet-pretrained CNN architectures considered.GoogLeNet is used for comparisons with existing methods.
- Experimental settings: MCA is used for evaluation; conventional settings report target-class accuracy, while generalized settings report MCAt, MCAs, and their harmonic mean H.Generalized evaluation includes both source and target classes in the search space and test data.
- Optimization of the visual embedding subnet: Optimizing the visual embedding subnet improves QFSL on CUB and AwA2 but worsens performance on SUN, where only about 20 training images are available per class.The authors associate the SUN result with scarce source-class training data; AwA2 and CUB have about 750 and 60 images per class, respectively.
- Classification loss and bias loss: MCAs decreases steadily as λ increases, while the best overall MCAt results occur for λ ∈[0.5, 2].Values below 0.5 leave bias insufficiently addressed, whereas values above 2 harm the image–semantic embedding relationship.
- Comparisons in conventional settings: 4.5 ∼16.2%: QFSL outperforms other state-of-the-art methods on CUB and AwA2 in conventional settings.The comparison includes the inductive baseline QFSL−.
- Comparisons in generalized settings: 9.3 ∼24.5%: QFSL improves generalized-setting harmonic mean H across the three datasets, mainly through higher MCAt without much MCAs decline.The results are attributed to using unlabeled target instances to alleviate source-class bias.
5. Further Study and Discussions
On SUN, QFSL is evaluated as the number of source classes varies, revealing a trade-off between source and target performance and increasing advantages over QFSL− under greater class imbalance.
- Experimental Setup: Seven source-class subsets—100, 200, 300, 450, 550, 600, and 645—are evaluated against fixed 72 target classes on SUN.The experiments use GoogLeNet and the standard split.
- Results: As source classes increase, MCAs decreases because source classification becomes more difficult, while MCAt increases from additional visual-semantic mapping knowledge.These trends are reported for generalized settings.
- Results: QFSL achieves comparable MCAs to QFSL− despite additionally addressing the bias problem.QFSL− is trained using only labeled source data.
- Results: As source-target imbalance grows, target instances are more likely to be classified as source classes, making the bias problem more severe.The passage connects this increasing bias with larger superiority of QFSL over QFSL−.
- Results: QFSL’s superiority over QFSL− becomes larger as the number of source classes increases.The reported comparison concerns the generalized setting and increasing source-target imbalance.
6. Conclusions and Future Work
QFSL learns unbiased ZSL embeddings by using labeled source and unlabeled target data, mapping them to source- and target-specified semantic points. Experiments report large-margin improvements over state-of-the-art methods in both conventional and generalized settings, while future work considers alternative semantic spaces and inductive learning.
- Conclusions: QFSL uses labeled source data to learn visual-semantic relationships and unlabeled target data to reduce bias toward source classes.Source and target images are projected to points specified by their respective classes in the semantic space.
- Conclusions: Experiments on different benchmarks show that QFSL outperforms state-of-the-art ZSL methods by a large margin in conventional and generalized settings.The conclusion states the cross-setting result without specifying a dataset or numerical margin.
- Future Work: Future work will explore word vectors and inductive learning as alternatives to attribute-based semantic spaces and transductive bias mitigation.The paper identifies both directions as further study.