Source-linked AI summary
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, Sanja Fidler
TL;DR
Cross-modal retrieval requires ranking correct image-caption matches above competing items. VSE++ introduces hard negatives into the embedding loss and combines this with augmented data and fine-tuning, achieving state-of-the-art MS-COCO performance with 8.8% and 11.3% R@1 improvements in caption and image retrieval, respectively.
Problem
The paper addresses ranking correct image-caption matches above negative items in cross-modal retrieval.
Method
VSE++ incorporates hard negatives into the multi-modal embedding loss and combines this change with augmented data and fine-tuning.
Results
On MS-COCO, VSE++ improves R@1 by 8.8% for caption retrieval and 11.3% for image retrieval over reported state-of-the-art comparisons.
Takeaways & Limitations
The proposed loss improves visual-semantic retrieval on MS-COCO and Flickr30K and can guide stronger or fine-tuned image encoders.
Takeaways & Limitations
Increasing mini-batch size produces harder negatives but can weaken SGD benefits, causing local optima or longer training times.
Abstract
from arXiv · showhide
We present a new technique for learning visual-semantic embeddings for cross-modal retrieval. Inspired by hard negative mining, the use of hard negatives in structured prediction, and ranking loss functions, we introduce a simple change to common loss functions used for multi-modal embeddings. That, combined with fine-tuning and use of augmented data, yields significant gains in retrieval performance. We showcase our approach, VSE++, on MS-COCO and Flickr30K datasets, using ablation studies and comparisons with existing methods. On MS-COCO our approach outperforms state-of-the-art methods by 8.8% in caption retrieval and 11.3% in image retrieval (at R@1).
1 Introduction
VSE++ addresses cross-modal image-caption retrieval by changing the ranking loss to emphasize hard negatives, alongside augmented data and fine-tuning. These changes improve retrieval performance and extend the benefits of stronger image encoders.
- Problem: VSE++ targets cross-modal retrieval, mapping images and captions into a shared space for retrieving either modality from the other.The paper evaluates image-to-caption and caption-to-image retrieval using recall at K.
- Results: The combined modifications produce a significant increase in caption retrieval performance over the baseline ranking loss on benchmark datasets.The paper studies these modifications on MS-COCO and Flickr30K.
- Approach: Fine-tuning and the stronger loss function amplify the benefit of a more powerful image encoder.The paper presents this as an additional benefit of the combined approach.
- Approach: The method incorporates hard negatives into the loss function, rather than relying only on the baseline ranking loss.This change is combined with augmented data and fine-tuning.
2 Learning Visual-Semantic Embeddings
The paper learns joint visual-semantic embeddings by projecting image and caption representations into a normalized shared space, then ranking positive pairs above negatives. Its MH loss emphasizes the hardest negatives, approximated within mini-batches to improve training signals while balancing robustness and optimization efficiency.
- Visual-Semantic Embedding: Training minimizes empirical loss over image-caption pairs using a symmetric hinge-based triplet ranking objective over negative captions and images.The margin parameter α determines when violations contribute nonzero loss.
- Visual-Semantic Embedding: Image and caption encoders produce representations that are linearly projected into a shared embedding space and normalized before inner-product similarity is computed.The image and caption mappings use separate projection matrices, with optional fine-tuning of the image encoder.
- Emphasis on Hard Negatives: Hard negatives are the highest-similarity incorrect image or caption for each query, and the MH loss focuses on these negatives rather than summing violations across all negatives.The authors report that MH performed best along the spectrum from Sum of Hinges to Max of Hinges.
- Emphasis on Hard Negatives: Mini-batch hard-negative mining preserves the SH loss’s quadratic complexity while often sampling negatives harder than 90% of the full training set.Under the stated IID assumption, the probability of missing the 90th-percentile negatives falls below 1% for M ≥44.
- Probability of Sampling the Hardest Negative: Larger mini-batches provide harder negatives but can reduce SGD’s optimization benefits, potentially causing local optima or extremely long training times.Mini-batches as small as 128 are described as providing strong training signal and robustness to label errors.
3 Experiments
Experiments evaluate VSE++ against VSE0 and prior methods on MS-COCO and Flickr30K, varying encoders, training data, augmentation, fine-tuning, and loss functions. The results show gains from the MH loss, especially with stronger encoders and additional data, while also revealing a short warm-up period and overfitting under limited training data.
- Experimental setup: VSE++ is compared with a VSE0 baseline and other state-of-the-art approaches on MS-COCO and Flickr30K.The experiments use both standard benchmarks and ablation studies.
- Experimental setup: The experiments use VGG19 or ResNet152 image encoders, GRU caption embeddings, multiple crop strategies, and optional image-encoder fine-tuning.Image features are evaluated with 1C, 10C, and random-crop training settings.
- MS-COCO results: 21.2% higher R@1 caption retrieval and 21% higher R@1 image retrieval than UVS are reported for the best MS-COCO VSE++ configuration.The best result uses ResNet152 and fine-tunes the image encoder.
- MS-COCO results: 8.8% higher R@1 caption retrieval than 2WayNet and 11.3% higher R@1 image retrieval than sm-LSTM are reported on MS-COCO.These comparisons use VSE++ with ResNet152 and fine-tuning.
- MS-COCO ablations: 5.9% and 5.1% R@1 caption-retrieval improvements over VSE0 occur with RC and RC+rV training, respectively, whereas 1C yields no caption-retrieval improvement.The results indicate stronger gains when additional or augmented training data are used.
- Flickr30K results: On Flickr30K, VSE++ improves R@1 by 23.1% for caption retrieval and 17.6% for image retrieval, but can over-fit with pre-computed 1C features.Overfitting is avoided with RC training, and MH-loss improvements persist across datasets and models.
- Improving order embeddings: Replacing SH with MH improves Order0 to Order++ by 4.5% in R@1 caption retrieval on MS-COCO.The authors report that MH may improve similar loss functions used in retrieval and ranking tasks.
- Loss behavior: The MH loss can take approximately five epochs to surpass SH loss, because its gradient is influenced by a smaller set of triplets early in training.Mini-batches of size 128 or 256 exceeded SH performance within the same training time, unlike larger mini-batches that trained extremely slowly.
4 Conclusion
The paper reports that VSE++ significantly improves cross-modal image-caption retrieval on MS-COCO and Flickr30K through its loss, stronger image encoding, fine-tuning, and augmented data. It achieves state-of-the-art performance on MS-COCO and remains slightly below the best recent model on Flickr30K.
- VSE++ significantly improves image-caption retrieval performance on the MS-COCO and Flickr30K datasets.
- The improved loss better guides the more powerful ResNet152 image encoder and image-encoder fine-tuning.
- VSE++ achieves state-of-the-art performance on MS-COCO.
- VSE++ is slightly below the best recent model on Flickr30K.
- The loss can train more sophisticated models that use a similar ranking loss.
A Examples of Hard Negatives
The appendix compares VSE++ and VSE0 on example MS-COCO test images by showing each model’s top retrieved caption and the rank of the highest-ranked ground-truth caption.
- Figure A.1 shows MS-COCO test images alongside the top 1 retrieved captions from VSE0 and VSE++ (ResNet)-finetune.
- Bracketed values indicate the rank of the highest-ranked ground-truth caption, while GT identifies a ground-truth caption sample.