Source-linked AI summary
Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
Tan Wang, Xing Xu, Yang Yang, Alan Hanjalic, Heng Tao Shen, Jingkuan Song
TL;DR
Image-text matching must reconcile different visual and textual representations while balancing accuracy and model complexity. The paper proposes MTFN for rank-constrained tensor-fusion similarity learning and adds cross-modal re-ranking at test time. MTFN-RR achieves state-of-the-art retrieval performance on two datasets with much less time consumption.
Problem
Image-text matching must handle incompatible visual and textual representations, while embedding and classification approaches do not balance matching accuracy and model complexity well.
Method
MTFN learns image-text similarity through rank-constrained image-text and text-text tensor fusion, followed by cross-modal re-ranking without additional training.
Results
MTFN-RR achieves state-of-the-art retrieval performance on two benchmark datasets with much less time consumption.
Takeaways & Limitations
Combining tensor-fusion similarity learning with cross-modal re-ranking provides an efficient framework that jointly improves I2T and T2I retrieval.
Takeaways & Limitations
The re-ranking strategy relies on the assumption that paired images and texts retrieve one another in both directions.
Abstract
from arXiv · showhide
A major challenge in matching images and text is that they have intrinsically different data distributions and feature representations. Most existing approaches are based either on embedding or classification, the first one mapping image and text instances into a common embedding space for distance measuring, and the second one regarding image-text matching as a binary classification problem. Neither of these approaches can, however, balance the matching accuracy and model complexity well. We propose a novel framework that achieves remarkable matching performance with acceptable model complexity. Specifically, in the training stage, we propose a novel Multi-modal Tensor Fusion Network (MTFN) to explicitly learn an accurate image-text similarity function with rank-based tensor fusion rather than seeking a common embedding space for each image-text instance. Then, during testing, we deploy a generic Cross-modal Re-ranking (RR) scheme for refinement without requiring additional training procedure. Extensive experiments on two datasets demonstrate that our MTFN-RR consistently achieves the state-of-the-art matching performance with much less time complexity. The implementation code is available at https://github.com/Wangt-CN/MTFN-RR-PyTorch-Code.
1 INTRODUCTION
Image-text matching must bridge incompatible visual and textual representations while balancing retrieval accuracy against model efficiency. MTFN-RR addresses this with tensor-based similarity learning during training and cross-modal re-ranking during testing.
- Challenge: Image-text matching supports image-to-text and text-to-image retrieval but must compare fundamentally different visual and textual representations.The two retrieval directions are I2T and T2I.
- Challenge: Classification-based methods struggle with cross-modal complexity, whereas embedding-based methods offer greater expressive potential at substantially higher model and algorithmic complexity.This motivates a framework that balances matching performance and efficiency.
- Framework: MTFN directly learns image-text similarity from global visual and textual features through rank-constrained tensor fusion rather than mapping instances into a shared embedding space.Its two fusion branches model image-text and text-text interactions.
- Framework: The training architecture uses Image-Text Fusion and Text-Text Fusion branches, with rank-constrained tensor blocks capturing bilinear interactions before similarity prediction.The fused vector is optimized with a ranking loss that encourages a large margin between ground-truth and negative pairs.
- Framework: Cross-modal re-ranking jointly refines I2T and T2I retrieval at testing time without additional training, using both image-text and text-text similarity.Text-text similarity also helps address datasets where one image is paired with five sentences.
- Reported outcome: MTFN-RR achieves state-of-the-art performance on two datasets, while MTFN alone remains competitive with lower time complexity and simpler feature extraction.The re-ranking scheme is also presented as applicable to other off-the-shelf matching methods.
2 RELATED WORK
Prior image-text matching work includes shared-embedding and direct classification formulations, while the proposed framework uses tensor fusion to learn similarity across multiple input types. Its architecture combines image-text and text-text fusion before cross-modal re-ranking.
- Image-Text Similarity: Embedding-based methods project multimodal features into a common space and measure similarity with conventional cosine or Euclidean distance.These methods can use global or local features.
- Image-Text Similarity: Classification-based methods instead predict whether an image-text pair matches, treating similarity as a binary decision.The paper contrasts this formulation with direct similarity learning.
- Re-ranking: The framework's testing stage uses cross-modal re-ranking to combine image-text and text-text similarities across retrieval directions.This extends the tensor-fusion similarity model with a retrieval refinement step.
- Tensor Fusion: MTFN applies tensor fusion to image-text and text-text inputs, directly learning similarity scores for both multimodal and unimodal relationships.The architecture is presented as a general tensor fusion design for varied inputs.
3 PROPOSED MODEL
The proposed model learns image-text similarity through rank-constrained tensor fusion and refines bidirectional retrieval with cross-modal re-ranking. It uses image-text and text-text relations to improve retrieval consistency without requiring a separate training procedure for re-ranking.
- Similarity learning: MTFN defines a similarity function S(I,T) that should score positive image-text pairs above negative pairs.The function supports ranking text for an image query and images for a text query.
- Multi-modal Tensor Fusion Network: MTFN fuses global image and sentence features through Image-Text Fusion and Text-Text Fusion branches.The text-text branch captures semantic relevance among sentences associated with the same image and supplies information for testing-time re-ranking.
- Multi-modal Tensor Fusion Network: Rank-constrained tensor fusion represents interactions as a sum of R rank-1 vectors from different common subspaces.Projected image and text features are combined with element-wise products before the rank-constrained fusion is formed.
- Similarity learning: The fused vector is transformed into the image-text score S_IT(I,T), then optimized with a bidirectional max-margin ranking loss using hardest negatives.This replaces direct match/mismatch classification with ranking constraints that separate positive pairs from confusing negatives by a margin.
- Cross-modal Re-ranking: Cross-modal re-ranking jointly uses retrieval results from both I2T and T2I directions to reduce the discrepancy between training and inference.For I2T, candidate texts are re-ordered using inverse T2I positions; for T2I, related texts selected through S_TT help refine candidate images.
- Cross-modal Re-ranking: T2I re-ranking replaces an individual query text with its unimodal nearest-neighbour text set and performs I2T retrieval for each candidate image.The approach uses text-text similarity to exploit associations among multiple sentences annotating the same image.
4 EXPERIMENT
Experiments on Flickr30k and MSCOCO evaluate retrieval accuracy, fusion strategies, model complexity, and cross-modal re-ranking. MTFN-RR generally improves retrieval while maintaining lower complexity than more elaborate alternatives.
- Datasets and Evaluation Metric: Experiments use Flickr30k and MSCOCO for I2T and T2I retrieval, reporting R@L and mR evaluation measures.MSCOCO results use both five 1k test folds and the full 5k test set.
- Comparisons with the State-of-the-arts: MTFN achieves competitive performance across both retrieval tasks and datasets, while SCAN remains slightly better on Flickr30k I2T.The reported performance gap is described as insignificant relative to SCAN’s greater algorithmic complexity.
- Comparisons with the State-of-the-arts: MTFN-RR improves both retrieval tasks over MTFN and reaches state-of-the-art performance in most cases by jointly exploiting I2T and T2I associations.Using text-text similarity is especially relevant to T2I retrieval under image-text data imbalance.
- Distribution of Fusion Vector: Rank-based MTFN embeddings preserve matching-pair structure with larger margins and higher match-versus-mismatch classification accuracy than logistic-regression fusion.The comparison visualizes fusion vectors with a learned linear SVM boundary.
- Analysis on Fusion Strategy: MTFN outperforms traditional linear fusion strategies with much less training and evaluation time, whereas attention improves prior schemes but increases complexity and harms MTFN performance.The analysis attributes this to rank-constrained tensor fusion already encoding bilinear interactions between global image and text features.
- Analysis on Model Complexity and Cross-modal Re-ranking: Global features consistently outperform local features across tested parameter settings, and cross-modal re-ranking improves I2T for all evaluated methods.For MTFN, re-ranking also helps T2I; performance remains stable for larger neighborhoods once K reaches 6.
- Analysis on Model Ensemble: Ensembling MTFN-RR models with M = 2 or 3 generally improves retrieval without increasing training complexity, and M = 3 outperforms SCAN’s best ensemble on T2I.SCAN’s comparison uses separately trained I2T and T2I models.
5 CONCLUSION
The paper combines MTFN with cross-modal re-ranking into MTFN-RR and reports state-of-the-art retrieval performance with much less time consumption on two benchmark datasets.
- MTFN directly learns image-text similarity through multi-modal tensor fusion of global visual and textual features without redundant training steps.
- MTFN-RR combines MTFN with cross-modal re-ranking to improve I2T and T2I retrieval using additional unimodal text-text similarity.
- Experiments on two benchmark datasets show state-of-the-art retrieval performance with much less time consumption.
- The paper reports model ensemble results for MTFN-RR and comparison methods on Flickr30k.