Source-linked AI summary
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Jiuxiang Gu, Jianfei Cai, Shafiq Joty, Li Niu, Gang Wang
TL;DR
Cross-modal retrieval needs representations that bridge heterogeneous image and text modalities while capturing detailed correspondences beyond global semantics. The paper incorporates image-to-text and text-to-image generation into cross-modal embedding to learn global abstract and local grounded representations. Experiments on MSCOCO show improved retrieval performance, with the complete model generally achieving the best results and outperforming state-of-the-art methods.
Problem
Heterogeneous image and text modalities, and global representations that miss local similarity, make detailed cross-modal retrieval difficult.
Method
The framework combines conventional cross-modal embedding with image-to-text and text-to-image generative models to learn abstract and locally grounded representations using max-margin learning-to-rank.
Results
The complete GXN model, incorporating both image and text generations, generally achieves the best performances, while the framework significantly outperforms state-of-the-art methods on MSCOCO.
Takeaways & Limitations
Combining grounded and abstract representations improves matching of images and sentences with complex content in cross-modal retrieval.
Abstract
from arXiv · showhide
Textual-visual cross-modal retrieval has been a hot research topic in both computer vision and natural language processing communities. Learning appropriate representations for multi-modal data is crucial for the cross-modal retrieval performance. Unlike existing image-text retrieval approaches that embed image-text pairs as single feature vectors in a common representational space, we propose to incorporate generative processes into the cross-modal feature embedding, through which we are able to learn not only the global abstract features but also the local grounded features. Extensive experiments show that our framework can well match images and sentences with complex content, and achieve the state-of-the-art cross-modal retrieval results on MSCOCO dataset.
1. Introduction
Cross-modal retrieval must bridge heterogeneous image and text data while capturing both global semantics and fine-grained correspondences. The proposed framework adds generative, locally grounded embeddings to conventional global cross-modal representations and improves retrieval on MSCOCO.
- Heterogeneous image, text, and video data make effective cross-modal information retrieval difficult.
- Conventional methods map separately encoded modalities into a common semantic space and compare their distances, typically using ranking loss.
- Global representations alone may miss detailed local image similarity, spatial layout, and word-level sentence similarity.
- The framework incorporates image-to-text and text-to-image generative models to learn additional local grounded features alongside global semantic features.
- Extensive MSCOCO experiments show that combining grounded and abstract representations significantly improves state-of-the-art cross-modal image-caption retrieval.
2. Related Works
Prior textual-visual retrieval methods largely learn high-level shared spaces with ranking objectives, while some work explicitly aligns visual objects and textual words. This paper distinguishes its approach by learning both global and local common spaces through generative models.
- Supervised cross-modal embedding commonly uses pairwise ranking to learn similarity or distance metrics between matched and mismatched image-caption pairs.
- Earlier approaches encode images and captions into a shared space and optimize matched pairs to be closer than mismatched pairs.
- Several studies improve retrieval by aligning visual objects or image fragments with textual words, sometimes combining local and global alignment.
- Most existing studies focus on a high-level common space learned with ranking loss.
- The proposed approach additionally learns a local common space through generative models.
3. Proposed Generative Cross-modal Learning Network
GXN combines global and grounded cross-modal embeddings with image-to-text and text-to-image generative training paths. The resulting representations support retrieval while generative losses regularize local features.
- System Overview: GXN uses two feature-embedding branches: abstract features (v_h, t_h) and grounded features (v_l, t_l).The grounded branch is regularized through the two generative learning paths.
- Testing: At test time, GXN uses both abstract and grounded image and text features as the final representations for cross-modal retrieval.Image-to-text and text-to-image generations are treated as byproducts rather than the paper’s main focus.
- Cross-modal Feature Embedding: The cross-modal embedding path maps image and caption representations into a common space using image and sentence encoders, then optimizes pairwise ranking.The mapped vectors represent images and captions, while positive pairs are encouraged to outrank negative pairs.
- Image-to-text Generative Feature Learning: Image-to-text learning decodes the grounded visual feature v_l into a sentence and trains it with cross-entropy followed by a mixed cross-entropy and reinforcement-learning objective.The reinforcement component optimizes sentence-level rewards such as BLEU or CIDEr, while cross-entropy supports readability and fluency.
- Text-to-image Generative Adversarial Feature Learning: Text-to-image learning generates images from the grounded text feature t_l with a conditional GAN comprising an image generator and discriminator.The discriminator distinguishes real image–true caption pairs from generated image–true caption and real image–wrong caption pairs.
4. Experiments
Experiments evaluate GXN on MSCOCO using standard cross-modal retrieval and caption-quality metrics, comparing encoders, generative components, and complete-model performance. Results show that combining image-to-text and text-to-image generation generally improves retrieval and achieves the best performance across reported MSCOCO metrics.
- Dataset and implementation: MSCOCO experiments use 113,287 training images, 5,000 validation images, and 5,000 test images, with VGG19 and ResNet152 image encoders.The MSCOCO retrieval setting assigns five captions to each training image; VGG19 features have dimension 4096 and ResNet152 features dimension 2048.
- Evaluation metrics: R@K measures the percentage of queries whose ground-truth match appears among the first K results, while lower Med r indicates better retrieval rank.Sum aggregates R@1 and R@10 scores to evaluate overall cross-modal retrieval performance.
- Baseline comparisons: GRUBi (VGG19) raises caption retrieval R@1 from 51.4 to 53.6 and image retrieval R@1 from 39.1 to 40.0 versus GRU (VGG19).The bidirectional GRU uses context from both directions.
- Baseline comparisons: Replacing VGG19 with ResNet152 raises caption retrieval R@1 from 53.6 to 64.0 and image retrieval R@1 from 40.0 to 53.6.This comparison uses GRUBi (VGG19) as the starting model.
- Generative-model effects: Image-to-text generation improves GXN retrieval over GXN (fine-tune), with mixed RL+XE training producing further improvements.The image-to-caption generation path is intended to learn grounded visual features that generate captions similar to ground-truth captions.
- Generative-model effects: Text-to-image generation significantly improves text-to-image retrieval over GXN (fine-tune), while GXN (i2t+t2i) achieves the best performances in general.The complete model combines both image and text generations and has the advantages of GXN (i2t, mix) and GXN (t2i).
- Caption quality: Generative GXN models retrieve captions closer to the ground-truth captions according to BLEU and CIDEr scores computed over five reference sentences.BLEU and CIDEr are sentence-level metrics reported for the image-to-text retrieval task.
- State-of-the-art comparison: On MSCOCO, GXN achieves the best performance in all metrics against state-of-the-art methods.The comparison is reported in Table 3, with additional experiments conducted on Flickr30K.
5. Conclusion
The paper proposes a generative cross-modal embedding framework that learns both abstract and locally grounded representations for image-text retrieval. It reports superior MSCOCO retrieval performance over state-of-the-art methods.
- The framework incorporates image-to-text and text-to-image generative models into conventional cross-modal feature embedding.
- It learns both high-level abstract and local grounded representations within a max-margin learning-to-rank framework.
- The combination of grounded and abstract representations significantly improves state-of-the-art cross-modal image-caption retrieval performance on MSCOCO.