Source-linked AI summary
Improving zero-shot learning by mitigating the hubness problem
Georgiana Dinu, Angeliki Lazaridou, Marco Baroni
TL;DR
Zero-shot labeling depends on nearest neighbors after mapping vectors into linguistic space, but mapped neighborhoods can be dominated by hubs that displace correct labels. The paper corrects retrieval globally using proximity information from many mapped pivots, and reports consistent improvements across translation, image labeling, and image retrieval.
Problem
Mapped-vector neighborhoods contain hubs that are near many items, pushing correct zero-shot labels down the neighbor list.
Method
The paper replaces standard nearest-neighbor queries with globally adjusted retrieval based on hubness and the topology of additional mapped, unlabeled source vectors.
Results
The corrected methods achieved significant improvements of 7% to 14% over standard nearest-neighbor retrieval in English-to-Italian translation, with consistent improvements across the evaluated domains.
Takeaways & Limitations
Hubness correction improves zero-shot retrieval across cross-lingual translation, image labeling, and image retrieval without requiring more advanced mapping functions.
Abstract
from arXiv · showhide
The zero-shot paradigm exploits vector-based word representations extracted from text corpora with unsupervised methods to learn general mapping functions from other feature spaces onto word space, where the words associated to the nearest neighbours of the mapped vectors are used as their linguistic labels. We show that the neighbourhoods of the mapped elements are strongly polluted by hubs, vectors that tend to be near a high proportion of items, pushing their correct labels down the neighbour list. After illustrating the problem empirically, we propose a simple method to correct it by taking the proximity distribution of potential neighbours across many mapped vectors into account. We show that this correction leads to consistent improvements in realistic zero-shot experiments in the cross-lingual, image labeling and image retrieval domains.
1 INTRODUCTION
Zero-shot learning uses vector representations and mappings to reduce manual annotation, but mapped-vector neighborhoods are polluted by hubs. The paper proposes globally adjusting neighbor retrieval to improve labeling while retaining simple regression mappings.
- Zero-shot learning maps source-domain vectors to linguistic-space vectors so nearest-neighbor words can provide labels with less manual annotation.
- Mapped-vector neighborhoods contain universal neighbors, or hubs, that occur near many different mapped vectors and degrade zero-shot labeling quality.
- Hubness is more severe after regression-based mapping, although its theoretical explanation is left for future work.
- The proposed post-processing method adjusts the similarity matrix, allowing simple least-squares regression mappings while producing consistent improvements across tasks.
2 HUBNESS IN ZERO-SHOT MAPPING
The paper formalizes zero-shot mapping as regularized linear regression followed by nearest-neighbor retrieval, then measures how frequently targets appear in pivot neighborhoods. Empirically, mapped vectors exhibit substantially stronger hubness than original target vectors.
- The Zero-shot setup: Zero-shot training pairs source vectors with linguistic target vectors, while test instances use labels whose target vectors were unseen during training.
- The Zero-shot setup: The mapping is learned as a linear function with an l2-regularized least-squares objective, using concatenated source and corresponding target-vector matrices.
- Target space label retrieval: At retrieval time, a source vector is mapped into the target domain and its nearest target label is selected using cosine similarity.
- Hubness: Hubness counts how often a target appears in the k-nearest-neighbor lists of pivot vectors; large counts identify hubs.
- Original vs. mapped vectors: In English–Italian mapping with 5,000 training pairs and 1,500 test pivots, mapped targets reached maximum N20 values above 40, versus at most 11 for original targets.
- Original vs. mapped vectors: Regularization did not significantly mitigate hubness, indicating that the effect was not simply caused by overfitting to training vectors.
3 A GLOBALLY CORRECTED NEIGHBOUR RETRIEVAL METHOD
The paper introduces globally corrected neighbour retrieval, which uses neighbourhood statistics across mapped pivots to suppress hubs. Across translation and cross-modal zero-shot tasks, this correction improves retrieval accuracy and reduces hubness.
- Globally corrected retrieval: Globally corrected retrieval ranks targets using nearest-neighbour statistics across the entire mapped pivot set rather than independently querying each pivot.The method reverses the query direction: it selects the target for which the pivot receives the highest rank, reducing the influence of hubs.
- English to Italian word translation: Both corrected methods improve En→It translation accuracy over standard NN by 7% to 14%.The gains are strongest for medium- and low-frequency words; corrected measures are also robust when training data exceed 5K examples.
- English to Italian word translation: GC consistently pushes high-hubness translation candidates down neighbour lists, allowing correct translations such as arretratezza to replace polluted NN outputs.The corrected method reduces the recurrence of hubs such as 11/09/2002, while also correcting more ordinary words such as dio and violentatori.
- Zero-shot image labeling and retrieving: GC significantly improves standard NN in both image labeling and image retrieval, whereas cosine normalization performs worse than no correction in these cross-modal settings.The experiments use 5,000-word label and 500,000-image search spaces, with chance accuracy of 1/5,000 in both directions.
- English to Italian word translation: With a stochastic-gradient-descent mapping, En→It accuracy rises from 38.4 (NN) to 40.6 (GC retrieval).This result indicates that GC is not limited to least-squares error estimation settings.
4 CONCLUSION
The basic zero-shot setup is strongly affected by hubness, but globally adjusted nearest-neighbour queries provide a simple correction that consistently improves performance across diverse domains.
- Strong hubness negatively affects zero-shot experiments using regularized least-squares linear mappings across representational vector spaces.
- Globally adjusted nearest-neighbour queries provide a simple way to correct the hubness problem.
- The correction requires additional unlabeled source-space data beyond the test instances.
- Consistent improvements are obtained across very different domains while retaining the basic setup’s simplicity, generality, and high performance.
- Future work will examine whether hubness affects other word representations, learning objectives, and mapping setups.