Source-linked AI summary
Goal-Oriented Gaze Estimation for Zero-Shot Learning
Yang Liu, Lei Zhou, Xiao Bai, Yifei Huang, Lin Gu, Jun Zhou, Tatsuya Harada
TL;DR
ZSL needs fine-grained attribute localization because global features can miss discriminative regions shared across seen and unseen classes. GEM-ZSL uses attribute-guided gaze estimation with localized and global features, achieving superior or competitive results across three benchmarks and supporting the value of gaze supervision.
Problem
Global visual features and existing attention models insufficiently capture fine-grained discriminative attribute regions needed for zero-shot recognition.
Method
GEM-ZSL combines attribute-guided gaze estimation, localized discriminative attributes, and a cosine-metric ZSL backbone using joint global and local features.
Results
GEM-ZSL achieves superior or competitive performance on CUB, AWA2, and SUN, with ablations showing consistent gains in ZSL and GZSL accuracy.
Takeaways & Limitations
The results support using human gaze data and automatic gaze estimation to improve discriminative attribute localization in ZSL.
Abstract
from arXiv · showhide
Zero-shot learning (ZSL) aims to recognize novel classes by transferring semantic knowledge from seen classes to unseen classes. Since semantic knowledge is built on attributes shared between different classes, which are highly local, strong prior for localization of object attribute is beneficial for visual-semantic embedding. Interestingly, when recognizing unseen images, human would also automatically gaze at regions with certain semantic clue. Therefore, we introduce a novel goal-oriented gaze estimation module (GEM) to improve the discriminative attribute localization based on the class-level attributes for ZSL. We aim to predict the actual human gaze location to get the visual attention regions for recognizing a novel object guided by attribute description. Specifically, the task-dependent attention is learned with the goal-oriented GEM, and the global image features are simultaneously optimized with the regression of local attribute features. Experiments on three ZSL benchmarks, i.e., CUB, SUN and AWA2, show the superiority or competitiveness of our proposed method against the state-of-the-art ZSL methods. The ablation analysis on real gaze data CUB-VWSW also validates the benefits and accuracy of our gaze estimation module. This work implies the promising benefits of collecting human gaze dataset and automatic gaze estimation algorithms on high-level computer vision tasks. The code is available at https://github.com/osierboy/GEM-ZSL.
1. Introduction
Zero-shot learning transfers attribute-based semantic knowledge from seen to unseen classes, but global features often miss fine-grained discriminative regions. GEM-ZSL addresses this by using attribute-guided gaze estimation and localized attribute learning alongside a ZSL backbone.
- ZSL classifies unseen images using semantic descriptions shared between seen and unseen classes.
- Global image features often underrepresent fine-grained information, while existing attention models neglect discriminative attribute localization.
- Its attention transition module calibrates learned regions using real gaze data and a gaze loss before combining local and global features for recognition.
- GEM-ZSL uses attribute word vectors as queries and seen-class attributes as supervision to learn localized discriminative attributes.
- Across CUB, AWA2, and SUN, GEM-ZSL achieves superior or competitive ZSL performance, while gaze experiments validate the GEM.
2. Related Works
Prior ZSL methods progressed from global visual-semantic mappings to attribute-guided part representations, but GEM-ZSL adds goal-oriented gaze estimation for discriminative localization without training on unseen-class attributes.
- Early ZSL methods map visual features to semantic spaces, while later end-to-end models learn more discriminative global features from seen-class attributes.
- Part-based ZSL methods use attribute descriptions to guide learning of discriminative part features and improve ZSL performance.
- APN improves GZSL during testing through calibrated stacking, whereas RGEN uses a balance loss requiring unseen-class attributes during training.
- GEM-ZSL introduces goal-oriented gaze estimation for discriminative attribute localization while training only with seen-class attributes.
- Human-gaze estimation research has evolved from bottom-up saliency modeling toward deep-learning approaches for predicting where people look.
3. The Proposed Method
GEM-ZSL combines a plain cosine-metric ZSL backbone with goal-oriented gaze estimation to localize discriminative attributes and support zero-shot recognition.
- Overall architecture: The model uses an Image Encoder, GEM, and cosine metric space to combine global image features, localized attribute features, and class semantics.GEM includes word encoding, attention, attribute localization, and attention transition modules.
- Cosine metric learning: Cosine similarity maps visual features into the semantic space and empirically improves unseen-class accuracy over dot-product similarity.The classification score uses a scaling factor σ, and feature and class-embedding normalization reduces intra-class variance.
- Gaze estimation module: Attribute word vectors guide bilinear attention over image feature maps to learn regions associated with localized discriminative attributes.The word encoder converts GloVe-based attribute descriptions into visual attribute features used as attention queries.
- Gaze estimation module: Attribute localization applies global max pooling to attention maps and minimizes MSE between predicted attribute responses and ground-truth class attributes.A distance loss further concentrates each attention map around its maximum-response coordinate.
- Gaze estimation module: The attention transition converts attribute feature maps into gaze maps, which are matched one-to-one with human gaze maps using the Hungarian algorithm before gaze-loss optimization.The transition uses a learnable 1 × 1 convolution and sigmoid function; gaze supervision is omitted when ground truth is unavailable.
- Zero-shot recognition: After training, zero-shot recognition embeds a test image in the cosine space and selects the class embedding with highest compatibility; calibrated stacking reduces seen-class bias in GZSL.The calibration factor is set to 3.5 for AWA2 and 0.7 for CUB and SUN in the reported implementation.
4. Experiments
Experiments on CUB, SUN, and AWA2 compare GEM-ZSL with existing methods and analyze its modules, scaling factor, training strategy, and gaze-estimation quality. GEM-ZSL is competitive overall, improves over the baseline in both ZSL and GZSL, and produces more accurate attribute-focused gaze predictions.
- Comparison with the State-of-the-Art: GEM-ZSL outperforms all compared methods on CUB for both ZSL and GZSL, while remaining competitive on SUN and AWA2.On AWA2, it is only slightly below RGEN; on SUN, feature-generation methods perform strongly because the dataset contains more than 700 categories.
- Ablation Study: 10.0% (CUB), 8.1% (SUN), and 3.8% (AWA2) are the ZSL T1 improvements over the baseline after adding GEM.The corresponding GZSL harmonic-mean improvements are 8.9% (CUB), 6.0% (SUN), and 7.9% (AWA2).
- Ablation Study: 8.9% (CUB), 6.0% (SUN), and 7.9% (AWA2) are the GZSL H improvements over the baseline after adding GEM.The ablation progressively adds the gaze module, mean square error loss, distance loss, and cosine similarity.
- Ablation Study: Attention transition improves ZSL accuracy when evaluated with gaze loss on unseen categories from the CUB-VWSW dataset.The experiment constructs gaze heatmaps from captured gaze points and repeats the seen/unseen split four times before averaging T1.
- Scaling Factor: The best scaling factor is σ = 20 on CUB and AWA2 and σ = 25 on SUN.With learnable σ initialized at 20, performance increases with slight fluctuation during training; experiments therefore fix dataset-specific values.
- Training Method: Episode-based training outperforms random sampling, with the highest accuracy at M = 16 and N = 2.Each episode samples M categories and N images per category, while the random baseline uses a mini-batch of 64.
- GEM Evaluation: GEM achieves better gaze-estimation results than GBVS and GP on unseen CUB-VWSW images using AUC and NSS evaluation.AUC measures threshold-based true-positive versus false-positive performance, while NSS measures normalized saliency at fixated locations.
- GEM Evaluation: Qualitative visualizations show GEM concentrating more accurately on discriminative attribute regions than GBVS and GP.The predicted gaze maps are more concentrated than the ground-truth maps, and the localized regions help distinguish categories.
5. Conclusion
The paper introduces GEM-ZSL, which uses semantic query-guided attention and human-gaze supervision to localize discriminative attributes for zero-shot learning. Across three benchmarks, it achieves superior or competitive performance and supports gaze-based attribute localization as a promising direction.
- GEM learns discriminative attributes through semantic query-guided attention and real human-gaze supervision when available.The localized attributes are used to improve global image feature representation for ZSL.
- GEM-ZSL achieves superior or competitive performance on three ZSL benchmarks.The benchmarks are CUB, AWA2, and SUN.
- The work highlights potential benefits from collecting human-gaze datasets and developing automatic gaze-estimation algorithms for computer vision.
- Further investigation of discriminative attribute localization for ZSL remains an important future direction.