Source-linked AI summary
Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zheng-Jun Zha, Yongdong Zhang
TL;DR
GZSL methods can bias recognition toward seen classes because semantic labels are weakly discriminative, while semantic-free representations are underused. DVBE combines semantic-free and semantic-aligned representations, using adaptive second-order embedding, entropy-based domain detection, and auto-searched alignment. Across classification and segmentation benchmarks, DVBE reports improved performance, with limitations including unavailable visualization code and planned refinement of its domain detector.
Problem
Weak semantic labels can make aligned visual representations hard to distinguish, causing unseen images to be recognized as seen classes.
Method
DVBE combines semantic-free and semantic-aligned representations, using AMSE for discriminative domain detection and auto-searched semantic-visual embedding for unseen-class transfer.
Results
DVBE improves object classification by an average 4.3% across CUB, AWA2, aPY, and SUN, and semantic segmentation by 11% on Pascal VOC.
Takeaways & Limitations
Complementary visual representations let DVBE separate domain detection from unseen-class recognition, reducing the biased recognition problem toward the seen domain.
Takeaways & Limitations
The authors identify unavailable ZS3Net code as preventing visualization and plan to explore a more elaborate domain detector.
Abstract
from arXiv · showhide
Recent methods focus on learning a unified semantic-aligned visual representation to transfer knowledge between two domains, while ignoring the effect of semantic-free visual representation in alleviating the biased recognition problem. In this paper, we propose a novel Domain-aware Visual Bias Eliminating (DVBE) network that constructs two complementary visual representations, i.e., semantic-free and semantic-aligned, to treat seen and unseen domains separately. Specifically, we explore cross-attentive second-order visual statistics to compact the semantic-free representation, and design an adaptive margin Softmax to maximize inter-class divergences. Thus, the semantic-free representation becomes discriminative enough to not only predict seen class accurately but also filter out unseen images, i.e., domain detection, based on the predicted class entropy. For unseen images, we automatically search an optimal semantic-visual alignment architecture, rather than manual designs, to predict unseen classes. With accurate domain detection, the biased recognition problem towards the seen domain is significantly reduced. Experiments on five benchmarks for classification and segmentation show that DVBE outperforms existing methods by averaged 5.7% improvement.
1. Introduction
GZSL aligns visual images with semantic labels, but weak label discrimination can bias unseen images toward seen classes. DVBE addresses this with complementary semantic-free and semantic-aligned representations for domain detection and knowledge transfer.
- Weakly discriminative semantic labels reduce inter-class visual divergence after alignment, causing unseen images to be recognized as seen classes.
- DVBE learns an extra semantic-free representation that improves seen-class prediction and filters unseen images using prediction entropy.
- DVBE constructs complementary semantic-free and semantic-aligned representations to treat seen and unseen domains separately.
- AMSE uses second-order visual statistics, cross-attentive channel interaction, and adaptive margin constraints to increase semantic-free representation discrimination.
- 4.3% average improvement on CUB, AWA2, aPY, and SUN and 11% improvement on Pascal VOC were reported.
2. Related Works
Embedding-based GZSL commonly transfers knowledge by aligning images with semantic information, whereas generative methods synthesize unseen features. DVBE instead emphasizes semantic-free representations and entropy-based domain detection to address alignment-related bias.
- Embedding-based GZSL aligns image representations with attributes, word2vec, or text descriptions in a joint embedding space.
- Generative methods train GANs on seen data to generate unseen visual features, while DVBE focuses on embedding-based GZSL without synthetic training data.
- Semantic-aligned embedding methods remain affected by weak semantic discrimination, motivating DVBE’s semantic-free visual representation.
- Unlike COSMO’s elaborate domain detector, DVBE learns discrimination with AMSE and uses a simple entropy-based detector for end-to-end training.
3. Domain-aware Visual Bias Eliminating
DVBE separates seen- and unseen-domain recognition with complementary semantic-free and semantic-aligned representations. It uses entropy-based domain detection for unseen images, discriminative second-order features with adaptive margins, and an automatically searched semantic-visual embedding.
- Formulation: DVBE introduces semantic-free and semantic-aligned representations to handle seen and unseen domains separately in generalized zero-shot learning.The semantic-free representation supports seen-class discrimination and domain detection, while the semantic-aligned representation transfers knowledge to unseen classes.
- Formulation: Entropy from the seen-class predictions of fd(x) determines whether an image is treated as seen or routed to unseen-class recognition.Higher entropy indicates an unseen-domain image; images with entropy above τ are classified using the unseen search space.
- Adaptive Margin Second-order Embedding: AMSE compacts second-order visual statistics through cross-attentive channel interaction to reduce noisy-feature effects and improve semantic-free discrimination.The method uses compressed subspaces, spatial and channel attention, and complementary attention inputs; bilinear pooling otherwise magnifies noisy elements.
- Adaptive Margin Second-order Embedding: Adaptive margin Softmax imposes larger inter-class constraints on difficult samples while avoiding unnecessarily strict margins for easy samples.The margin coefficient depends on the predicted class probability: easy samples approach standard Softmax, whereas hard samples receive smaller coefficients and larger margins.
- Auto-searched Semantic-Visual Embedding: AutoS2V automatically searches a task-specific semantic-visual embedding architecture instead of relying on manually designed processing for semantic signals.Its operation set includes fully connected layers, graph convolution, skip connection, and none; the searched architecture is intended to bridge the visual-semantic gap.
- Overall: The overall DVBE objective combines semantic-visual alignment, adaptive-margin classification, and an auxiliary cross-entropy loss that prevents fv(x) collapse.The auxiliary loss is weighted by γ and is used alongside the classification and alignment objectives.
4. Experiments
DVBE is evaluated on four generalized zero-shot classification benchmarks and Pascal-VOC segmentation, with comparisons, ablations, and domain-detection analyses. It improves classification and segmentation performance while reducing seen-domain bias through semantic-free representations and related design components.
- Object Classification: DVBE surpasses non-generative methods by 2.5%, 4.7%, 8.3%, and 1.8% on CUB, AWA2, aPY, and SUN, respectively.Compared with AREN, DVBE achieves an averaged 5.6% improvement across the four classification datasets.
- Comparison with state-of-the-art methods: DVBE outperforms COSMO by 6.3% on CUB and achieves comparable performance on SUN through stronger semantic-free visual discrimination and entropy-based domain detection.The end-to-end DVBE variant surpasses COSMO by 18.3% on CUB and 1.8% on SUN.
- Comparison with state-of-the-art methods: DVBE obtains comparable performance to f-VAEGAN-D2 without using unseen-domain knowledge.The comparison concerns a generative method that uses prior unseen semantic labels to synthesize training data.
- Semantic Segmentation: DVBE improves unseen-domain performance over ZS3Net on Pascal-VOC, reaching 45.4% versus 26.1% mIoU.The reported hIoU improvement over ZS3Net is 11.0%.
- Ablation Studies: Adaptive margin Softmax outperforms standard and fixed-margin Softmax, while auto-searched g(·) improves MCAu over the hand-designed architecture.The adaptive strategy also shows fast and stable convergence, and searched architectures differ between CUB and AWA2.
5. Conclusion
DVBE addresses biased recognition in generalized zero-shot learning by combining semantic-free and semantic-aligned visual representations for classification and segmentation. The paper identifies future exploration of a more elaborate domain detector as a target.
- DVBE tackles biased recognition in generalized zero-shot learning using complementary semantic-free and semantic-aligned visual representations.These representations support the paper’s treatment of seen and unseen domains.
- DVBE combines cross-attentive channel interaction with adaptive margin Softmax to improve semantic-free visual discrimination.The semantic-free representation is used for unseen-image detection based on seen-class prediction entropy.
- DVBE automatically searches semantic-visual architectures to produce robust semantic-aligned representations.
- DVBE outperforms existing methods in both classification and segmentation.
- Future work targets an elaborate domain detector to improve DVBE.