Source-linked AI summary
Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing
Hengtong Hu, Lingxi Xie, Richang Hong, Qi Tian
TL;DR
Cross-modal hashing seeks efficient cross-modal retrieval, while supervised approaches often achieve greater accuracy than unsupervised ones but require more annotation. UKD uses an unsupervised model’s outputs to guide a supervised student; on MIRFlickr and NUS-WIDE, it outperforms existing unsupervised methods by a significant margin.
Problem
Unsupervised cross-modal hashing is easier to deploy but often less accurate than supervised methods, which depend on costly instance-level annotations.
Source: §1 · p. 1, §2.3 · p. 2Method
UKD uses feature outputs from an unsupervised CMH teacher to estimate pairwise similarities that guide a supervised student.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The paper presents unsupervised guidance of a supervised method as a direction for cross-modal hashing.
Source: §1 · p. 2, §5 · p. 8The described calculation assumes that the vector has an ℓ2-norm, to ease subsequent calculations.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
In recent years, cross-modal hashing (CMH) has attracted increasing attentions, mainly because its potential ability of mapping contents from different modalities, especially in vision and language, into the same space, so that it becomes efficient in cross-modal data retrieval. There are two main frameworks for CMH, differing from each other in whether semantic supervision is required. Compared to the unsupervised methods, the supervised methods often enjoy more accurate results, but require much heavier labors in data annotation. In this paper, we propose a novel approach that enables guiding a supervised method using outputs produced by an unsupervised method. Specifically, we make use of teacher-student optimization for propagating knowledge. Experiments are performed on two popular CMH benchmarks, i.e., the MIRFlickr and NUS-WIDE datasets. Our approach outperforms all existing unsupervised methods by a large margin.
1. Introduction
Cross-modal hashing maps different modalities into a shared Hamming space using fixed-length binary codes, enabling efficient retrieval. UKD uses an unsupervised model’s features to estimate pairwise similarities for a supervised model, seeking to combine easier deployment with stronger accuracy.
1. Introduction
- Cross-modal hashing maps contents from different modalities into a common Hamming space using fixed-length binary codes for retrieval.Hash-code indexing reduces storage cost and makes retrieval time complexity constant.Source: §1 · p. 1
- Supervised cross-modal hashing often achieves more accurate results, while unsupervised methods are easier to deploy because they do not require instance-level annotations.Both learn to shrink the distribution gap between training data from different modalities.Source: §1 · p. 1
- UKD estimates cross-modal pairwise similarity from feature distances produced by a trained unsupervised CMH model, providing guidance for a supervised method without labels.The approach treats pairwise similarity information—not instance labels themselves—as the signal used by supervised methods.Source: §1 · p. 1
- UKD comprises an unsupervised CMH module followed by a supervised module, and both can be replaced by newer models.Source: §1 · p. 2
- On MIRFlickr and NUS-WIDE, the approach achieves state-of-the-art performance and outperforms existing unsupervised CMH methods by a significant margin.Source: §1 · p. 2
2. Related Work
Cross-modal hashing uses binary codes to retrieve semantically similar items across modalities. Prior work divides into supervised methods that use semantic labels and unsupervised methods that rely on paired-data correlations; the latter motivate using unsupervised outputs to guide supervised training.
2.1. Cross-Modal Retrieval and Hashing
- Cross-modal retrieval searches for semantically similar instances in one modality using a query from another; hashing compresses outputs into fixed-length binary vectors to address high computation costs.The paper focuses on image-text retrieval, with separate mapping models trained and evaluated using image-text pairs.Source: §2.1 · p. 2
2.2. Supervised Cross-Modal Hashing Methods
- Supervised CMH methods use semantic labels to capture cross-modal correlations, but obtaining large amounts of labels is often expensive and can make these approaches infeasible in real-world applications.Deep approaches include DCMH, SSAH, and an adversarial hashing network with attention.Source: §2.2 · p. 2
2.3. Unsupervised Cross-Modal Hashing Methods
- Unsupervised CMH relies on correlations in paired data and is easier to deploy, but its accuracy is often below supervised methods because it lacks pairwise similarity knowledge.The paper’s motivation is that an unsupervised model’s outputs contain such semantic information, albeit somewhat inaccurately, and can guide a supervised model.Source: §2.3 · p. 2
3. Our Approach
UKD uses an unsupervised teacher to estimate pairwise similarity and guide a supervised student. The approach defines how that similarity is constructed and used, and relates it to prior knowledge-transfer methods.
3. Our Approach
- UKD trains a teacher without supervision, distills its knowledge into similarity matrix S, and uses S to supervise a student model.The method guides a supervised cross-modal hashing model with an unsupervised model's output.Source: §3 · pp. 2–3, Figure 1 · p. 3
3.1. Supervised and Unsupervised Baselines
- The learning challenge is defining S: instance-level labels estimate semantic relevance more accurately, while raw features avoid the burden of additional annotation.S_ij = 1 denotes a positive pair; the original strategy treats only paired instances as positive.Source: §3.1 · p. 3
3.2. Unsupervised Knowledge Distillation
- UKD estimates pair relevance from teacher-model features; S can guide supervised learning or improve unsupervised training's sampling of positive pairs.The authors compare similarity definitions and report that merging image- and text-based retrieval lists gives the best stated precision among the described options.Source: §3.2 · pp. 3–4
- 76.1% precision at top-5000 instances is reported when image- and text-feature retrieval lists are merged, exceeding either feature source alone.The authors fix this merging strategy for the rest of the paper.Source: Table 2 · p. 4, §3.2 · p. 4
3.3. Models and Implementation Details
- Implementation pairs a VGG19-based image network and SSAH text network with UGACH as teacher; relevant-instance counts were 10,000 for the supervised student and 20 for the unsupervised student.The teacher with the highest precision was selected using 128-bit codes.Source: §3.3 · pp. 4–5
3.4. Relationship to Previous Work
- Unlike DarkRank's ranking-distribution matching and methods using a distillation loss, UKD selects relevant instances without an extra loss.The authors also relate the approach to DeepCluster and Gomez et al., which guide learning using unsupervised outputs.Source: §3.4 · p. 5
- The authors connect UKD to self-learning for semi-supervised learning and to coarse-to-fine approaches that train a stronger model in a second stage.They distinguish UKD by how it obtains additional supervision.Source: §3.4 · p. 5
4. Experiments
UKD outperforms existing unsupervised cross-modal hashing methods across both benchmarks and all tested code lengths. Experiments also examine supervision, iterative distillation, teacher strength, feature transfer, and pair selection.
4.1. Datasets, Evaluation, and Baselines
- Experiments use MIRFlickr and NUS-WIDE, evaluate retrieval with mAP, and compare UKD with supervised and unsupervised baselines.The benchmarks use different dataset sizes, query splits, and image and text features; Table 3 reports mAP across two datasets and four code lengths.Source: §4.1 · p. 5, Table 3 · p. 6
4.2. Unsupervised Student vs. Supervised Student
- UKD-SS’s advantage over UKD-US is larger on MIRFlickr than NUS-WIDE, while teacher-estimated similarities can introduce noise into the supervised student.The reported SSAH-over-UGACH advantage is typically 9% on MIRFlickr but 1%–4% on NUS-WIDE; SSAH is the upper bound of UKD-SS.Source: §4.2 · p. 6
4.3. Comparison to the State-of-the-Arts
- UKD significantly outperforms existing unsupervised cross-modal hashing methods on both datasets at every code length, including against UGACH.The comparison reports average mAP gains over UGACH across image→text and text→image retrieval.Source: §4.3 · p. 6, Table 3 · p. 6
4.4. Does Iteration Help?
- First-round distillation yields larger gains than a second round, whose average image→text gain is 0.33% versus 0.60% for the first.The authors attribute the limited later gain to little improvement in intra-modal retrieval and training pairs without a clear quality advantage.Source: §4.4 · p. 7, Table 4 · p. 7
4.5. Diagnostic Experiments
- A 16-bit teacher still improves accuracy over baseline, but less than a 128-bit teacher; UKD also outperforms UGACH and UCH with CNN-F features.The weaker-teacher experiment links the reduced gain to a weaker similarity matrix, while the feature-transfer experiment reports UKD-SS above UKD-US.Source: §4.5 · p. 7, Table 5 · p. 7, Table 6 · p. 7
- As more relevant pairs are selected, mAP gains tend to saturate or dip, while top-ranked-pair precision drops and non-top-ranked pairs can add noise.The authors describe this as a quantity–quality tradeoff in pair selection.Source: §4.5 · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 8
- For a dog text query with few labeled correspondences, teacher-mined extra training pairs improve predictions; the image query is more semantically complex.The qualitative comparison shows top-five retrievals for text and image queries, with relevant results marked in the figure.Source: §4.5 · p. 8, Figure 5 · p. 8
5. Conclusions
The paper proposes unsupervised knowledge distillation to guide a supervised cross-modal hashing method with outputs from an unsupervised method, using teacher-student optimization. On MIRFlickr and NUS-WIDE, the authors report performance that outperforms state-of-the-art methods.
5. Conclusions
- The proposed approach uses outputs from an unsupervised method to guide a supervised cross-modal hashing method.The authors use teacher-student optimization to propagate knowledge.Source: §5 · p. 8
- The approach uses relevance information from an unsupervised teacher model to improve the supervised student model's performance.Source: §5 · p. 8
- On MIRFlickr and NUS-WIDE, the authors report that their method outperforms state-of-the-art methods.Source: §5 · p. 8