Source-linked AI summary
Gaze Embeddings for Zero-Shot Image Classification
Nour Karessli, Zeynep Akata, Bernt Schiele, Andreas Bulling
TL;DR
Zero-shot classification relies on auxiliary information, but expert-annotated attributes require costly domain expertise. This paper uses human gaze and three gaze embeddings as auxiliary information, reporting class-discriminative gaze that is competitive with attributes and improves over other baselines.
Problem
Zero-shot image classification needs auxiliary information for classes absent from training, while compiling discriminative attributes requires domain experts and repeated revision as classes change.
Source: §1 · p. 1Method
The paper collects gaze during a comparison-and-classification task and learns image–gaze compatibility using GH, GFG, and GFS embeddings.
Source: §1 · pp. 1–2Results
Human gaze is class-discriminative, improves over expert-annotated attributes and mouse-click data on CUB-VW, and gaze results on PET show generalization to another domain.
Source: §6 · p. 8Takeaways & Limitations
Non-expert human gaze provides a competitive alternative to expert-provided attributes and improves over several zero-shot baselines in the reported evaluations.
Source: §1 · p. 2On larger fine-grained datasets such as CUB-VWSW, results might benefit from alternative data collection because novice annotators did not compare the relevant species groups.
Source: §5.2 · p. 7
Abstract
from arXiv · showhide
Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification.
1. Introduction
The paper replaces expert-compiled attributes with human gaze as auxiliary information for zero-shot classification, using a discrimination-focused collection task and three gaze descriptors. It evaluates these descriptors on fine-grained bird and pet data and reports class-discriminative gaze and competitive performance against attributes and other baselines.
Motivation and approach
- Human gaze is proposed as zero-shot auxiliary information because expert-compiled attribute lists require domain knowledge and revision as classes are added.The collection paradigm has observers compare exemplars from two classes before classifying another exemplar, to focus gaze on discriminative object parts.Source: §1 · p. 1
Contributions
- The work introduces three class-discriminative gaze descriptors—GH, GFG, and GFS—and a structured joint embedding with deep image features.It also provides multi-observer gaze data for two CUB subsets and one PET subset.Source: §1 · p. 2
- Evaluations report that non-expert gaze is class-discriminative, improves over several baselines, and is competitive with expert-provided attributes for zero-shot learning.Source: §1 · p. 2
2. Related Work
Prior zero-shot work uses auxiliary information to connect images with classes, most often attributes, but fine-grained classes can require many costly attributes. The paper positions gaze as a way to obtain class-discriminative representations from observers’ visual judgments.
Zero-Shot Learning
- Zero-shot learning evaluates classes absent from training, so auxiliary information such as attributes, text representations, or class hierarchies is used to associate classes.Source: §2 · p. 2
- Fine-grained objects may require many attributes, which are costly to obtain; automatically extracted text representations and hierarchical embeddings are alternatives, but the introduction says they do not outperform attributes.Source: §2 · p. 2
Gaze-Supported Computer Vision
- Earlier gaze-supported vision work used gaze for tasks including detection, segmentation, and captioning, while related approaches also used mouse-click bubbles or saliency maps.Source: §2 · p. 2
- The paper distinguishes its approach from prior gaze work by collecting real eye-tracking data to form class-discriminative representations for zero-shot classification.Source: §2 · p. 2
3. Gaze Tracking and Datasets
The paper collects gaze during a fine-grained comparison-and-classification task and constructs class-level gaze embeddings from multiple observers. Its three descriptors encode spatial counts, spatially arranged gaze features, or gaze-feature sequences.
3.1. Gaze Data Collection
- Observers compare two fine-grained class examples for six seconds, fixate centrally for one second, then classify a new example from either class within five seconds.Gaze is recorded during the classification screen; eye tracking logs valid binocular gaze locations, pupil diameter, and timestamps.Source: §3.1 · pp. 2–3
- The datasets cover 14 bird classes and 464 images in CUB-VW, 1,882 images in the expanded CUB-VWSW, and 720 images across 24 pet classes in PET, with five observers per image.Bird comparisons are made at the sub-species level because broader family comparisons are too easy and yield fewer gaze points.Source: §3.1 · p. 3
3.2. Gaze Embeddings
- GH bins gaze points in an m × n spatial grid and represents each class by its mean histogram, encoding where observers look most often.The paper notes that these histograms capture class-specific spatial differences in attention.Source: §3.2 · p. 3
- GFG averages six-dimensional gaze features within each spatial grid cell and concatenates the cell features into a class-level vector.The features include location, duration, scan-path angles, and pupil diameter; spatial ordering can also represent behavioral patterns.Source: §3.2 · pp. 3–4
- GFS orders gaze points by time, selects a fixed number from each sequence, and averages the resulting vectors by class.Unlike grid-based embeddings, it encodes temporal sequence rather than spatial layout.Source: §3.2 · p. 4
- The paper combines per-class observer embeddings by averaging them, concatenating them for early fusion, or averaging separate models’ classification scores for late fusion.Source: §3.2 · p. 4
4. Gaze-Supported Zero-Shot Learning
Zero-shot learning trains on classes disjoint from the test classes, so auxiliary information links them. This section uses gaze embeddings in a Structured Joint Embedding model to score image–class compatibility.
4. Gaze-Supported Zero-Shot Learning
- Training and test classes are disjoint, so auxiliary information supplies a structure for associating classes without labeled test-class images.During training, the model sees images and gaze embeddings only for training classes.Source: §4 · p. 4
- The model learns a function from images to classes by minimizing empirical risk with the Structured Joint Embedding model.The empirical risk uses a cost for predicting a label when the true label differs.Source: §4 · p. 4
- Image features from a DNN and gaze class embeddings are preprocessed, then combined through a learned bilinear compatibility function.At test time, the model selects the class whose gaze embedding gives the highest joint compatibility score.Source: §4 · p. 4
5. Experiments
Across CUB-VW, CUB-VWSW, and Oxford Pets, the experiments evaluate gaze embeddings against alternative representations and examine how encoding choices affect zero-shot classification. Gaze performs strongly on CUB-VW and PET; results on CUB-VWSW show a boundary when novice observers compare similar bird species without cross-family comparisons.
5.1. Gaze Embeddings on CUB-VW
- GFS outperforms GH and GFG, and participant fusion reaches 69.8% with EARLY versus 56.6% with AVG and 69.6% with LATE.The authors report that participant gaze embeddings differ and may contain complementary information.Source: §5.1 · pp. 5–6, Figure 5 · p. 5
- With GFS EARLY, [x, y, d, α1, α2] reaches 73.9%; duration and sequence improve results, while pupil diameter adds no further improvement.The features are added cumulatively, starting with location.Source: §5.1 · p. 6, Figure 6 · p. 6
- 73.9% accuracy with GFS EARLY and [x, y, d, α] on CUB-VW exceeds the listed baselines and expert attributes (72.9%); combining gaze and attributes reaches 78.2%.The comparison includes saliency histogram (35.8%), random points (39.5%), central point (41.5%), bubbles (43.2%), and Wikipedia embeddings (55.2%).Source: §5.1 · p. 6, Table 2 · p. 6
- Reducing gaze information in the bubble ablation lowers accuracy from 73.9% to 69.7%, 64.0%, 55.0%, and 49.2% across the reported variants.The variants use the same images as bubbles, gaze points inside bubbles, averaged points, and one random point inside bubbles, respectively.Source: Table 3 · p. 6, §5.1 · p. 7
5.2. Gaze Embeddings on Other Datasets
- On CUB-VWSW, novice observers did not compare bird families, and similar-looking vireos, sparrows, and warblers constrained gaze results.The authors suggest cross-species comparisons or expert annotation might improve results.Source: §5.2 · p. 7
- On CUB-VWSW, gaze beats random points, bubbles, and bag-of-words, but expert attributes outperform gaze; on PET, gaze reaches 46.6% versus 33.5% for bag-of-words.PET comprises cat and dog breeds; its random-point chance is 16%.Source: §5.2 · p. 7, Table 4 · p. 7
5.3. Qualitative Results
- For the qualitative Vireo examples, gaze ranks Black capped Vireo images in the first three positions, while attributes and bag-of-words make mistakes.The reported attribute error is a Blue headed Vireo; the bag-of-words mismatch is linked to the frequent word “head” in both classes’ Wikipedia articles.Source: §5.3 · pp. 7–8
- For cats and dogs, gaze avoids the cat/dog confusions seen with bag-of-words, but retrieves Abyssinian images for Russian blue queries when color distinguishes the breeds.The paper describes this as a gaze-embedding failure case.Source: §5.3 · p. 8, Figure 7 · p. 8
6. Conclusion
The paper uses human gaze as auxiliary information to learn compatibility between images and labels for zero-shot learning. Its gaze embeddings show class discrimination and gains over attributes and mouse-click data, while results on larger fine-grained datasets suggest data collection may need to account for species hierarchy.
6. Conclusion
- On CUB-VW, gaze is class-discriminative and improves over expert-annotated attributes and mouse-click data; PET results show generalization to another domain.The authors note that results on larger fine-grained datasets might benefit from alternative collection paradigms.Source: §6 · p. 8
- Three gaze embeddings use spatial layout and first-order statistics, combine gaze location, duration, sequence, and concentration with spatial ordering, or sequentially sample gaze features.The embeddings learn compatibility between image and label space for zero-shot learning.Source: §6 · p. 8
- On larger fine-grained datasets such as CUB-VWSW, results might benefit from collection paradigms that let annotators view both super-species and sub-species.The authors identify further investigation of gaze behavior across fine-grained images as future work.Source: §6 · p. 8