PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Two-Branch Neural Networks for Image-Text Matching Tasks

Liwei Wang, Yin Li, Jing Huang, Svetlana Lazebnik

arXiv:1704.03470v4cs.CV

TL;DR

Image-text matching requires measuring correspondence between visual data and language across retrieval and grounding tasks. The paper proposes embedding and similarity two-branch networks with distinct training objectives and reports strong performance across phrase localization and bidirectional retrieval, while noting later methods improved Recall@1 and that combining local and global models yielded only small gains.

  • Problem

    The paper addresses how to measure semantic similarity between images or regions and sentences or phrases for image-text matching tasks.

  • Method

    It proposes an embedding network with a shared latent space, ranking loss, neighborhood constraints, and improved triplet sampling, alongside a similarity network using element-wise fusion and direct score prediction.

  • Results

    The networks achieve state-of-the-art phrase localization on Flickr30K Entities and near state-of-the-art bidirectional image-sentence retrieval on Flickr30K and MSCOCO.

  • Takeaways & Limitations

    The embedding network supports cross-modal matching and within-modality retrieval, while the two network types trade training efficiency, flexibility, and model complexity.

  • Takeaways & Limitations

    Combining the region-phrase and image-sentence models produces only a very small improvement because the region-phrase model cannot make several fine-grained distinctions needed for retrieval gains.

Abstract

from arXiv · show

Image-language matching tasks have recently attracted a lot of attention in the computer vision field. These tasks include image-sentence matching, i.e., given an image query, retrieving relevant sentences and vice versa, and region-phrase matching or visual grounding, i.e., matching a phrase to relevant regions. This paper investigates two-branch neural networks for learning the similarity between these two data modalities. We propose two network structures that produce different output representations. The first one, referred to as an embedding network, learns an explicit shared latent embedding space with a maximum-margin ranking loss and novel neighborhood constraints. Compared to standard triplet sampling, we perform improved neighborhood sampling that takes neighborhood information into consideration while constructing mini-batches. The second network structure, referred to as a similarity network, fuses the two branches via element-wise product and is trained with regression loss to directly predict a similarity score. Extensive experiments show that our networks achieve high accuracies for phrase localization on the Flickr30K Entities dataset and for bi-directional image-sentence retrieval on Flickr30K and MSCOCO datasets.

1 INTRODUCTION

The paper studies how two-branch neural networks can measure semantic similarity between visual data and text for phrase localization and image-sentence retrieval. It proposes embedding and similarity networks, along with neighborhood-aware training, and reports strong results across these tasks.

  • Image-text matching requires connecting visual regions or images with corresponding phrases or sentences despite large visual-concept vocabularies and background clutter.
  • The embedding network maps image and text features into a shared latent space and trains them with bidirectional maximum-margin ranking.Additional constraints bring semantically related samples within each modality closer together.
  • The similarity network fuses transformed image and text features by element-wise product and directly predicts pair similarity with logistic regression loss.Positive and negative pairs receive labels +1 and −1, respectively.
  • The study systematically evaluates loss functions, feature fusion, and positive/negative sampling strategies for both network types.
  • The networks achieve state-of-the-art phrase localization on Flickr30K Entities and near state-of-the-art bidirectional image-sentence retrieval on Flickr30K and MSCOCO.
  • The journal version adds the similarity network, deeper experiments, and improved absolute accuracies, while retaining architectures applicable to other image-text tasks.

2 RELATED WORK

Related work spans correlation-based embeddings, deep multimodal representations, ranking and metric learning, Siamese networks, and classification-based image-text matching. The paper distinguishes its asymmetric cross-modal networks and non-exclusive pairwise classification from these approaches.

  • CCA-based methods: CCA-based methods learn linear or nonlinear projections that maximize correlation between image and text views, but deep CCA optimization can be unstable.
  • Newer methods identified after the experiments achieved better Recall@1, limiting the currency of the paper’s retrieval comparisons.
  • Deep multimodal representations: Unlike recurrent-text approaches, the paper relies primarily on handcrafted orderless text features and reports similar image-sentence matching performance to LSTMs.
  • Ranking-based methods: Ranking-based methods learn shared spaces by penalizing incorrect annotations that outrank correct ones, with bidirectional ranking extending this idea across retrieval directions.
  • Metric learning and Siamese networks: The embedding network adds within-modality neighborhood-preservation constraints so semantically similar images or sentences remain close in the learned space.
  • Metric learning and Siamese networks: The two-branch networks learn across modalities with asymmetric branches rather than tied weights for same-modality Siamese similarity learning.
  • Classification-based methods: The similarity network treats each phrase-region pair as an independent binary classification problem, allowing multiple positive regions for one phrase.
  • Classification-based methods: Following related classification-based work, image-text correspondence can be modeled with softmax or other fusion strategies, whereas this paper uses non-exclusive logistic regression for pairwise matching.

3 EMBEDDING AND SIMILARITY NETWORKS

The paper develops embedding and similarity two-branch networks for phrase localization and bidirectional image-sentence retrieval. The embedding model learns a normalized shared space with ranking and neighborhood constraints, while the similarity model fuses branches by element-wise product and predicts pair scores.

  • 3.1 Overview of Image-Text Tasks: Phrase localization ranks candidate regions for an entity mention, while bidirectional retrieval searches matching sentences or images from a database.Both tasks use learned image-text similarity; retrieval uses nearest-neighbor search in the latent embedding space.
  • 3.2 Embedding Network: The embedding network maps image and text features into a common latent space where matched pairs have high cosine similarity.Its branches use fully connected transformations, nonlinearities, batch normalization, and L2 normalization; pre-computed features are used in this study.
  • 3.2.2 Learning Cross-Modal Matching by Ranking: A bidirectional margin-based ranking loss requires positive matches to be closer than negative matches in both image-to-text and text-to-image directions.The distance is Euclidean in the normalized embedding space, and the image-side constraint is d(x_i, y_j) + m < d(x_i, y_k).
  • 3.2.3 Preserving Neighborhood Structure within Modalities: Neighborhood constraints preserve within-modality structure by placing samples sharing corresponding data closer than samples outside the neighborhood.The loss adds image- and text-side neighborhood terms, whose weights control regularization strength; small values perform best.
  • 3.2.3 Preserving Neighborhood Structure within Modalities: Neighborhood sampling constructs triplets from multiple positive matches sharing a target, and it can improve image-sentence retrieval even without neighborhood loss terms.The sampling strategy searches positive pairs containing the target and can be pre-computed with hash tables.
  • 3.3 Similarity Network: The similarity network combines branch outputs by element-wise product, applies further fully connected layers, and learns scores with logistic regression.Training samples positive and negative pairs rather than triplets, while balancing their counts in each mini-batch.

4 PHRASE LOCALIZATION EXPERIMENTS

Phrase localization experiments on Flickr30K Entities evaluate architectural, sampling, and loss-function choices for embedding and similarity networks. Nonlinear branches and positive-region augmentation improve retrieval, while neighborhood sampling adds further gains, with comparable recall between the two network types.

  • Experimental setup: The experiments evaluate network structure, training-set sampling, and embedding-loss components for phrase localization on Flickr30K Entities.The benchmark extends Flickr30K with entity-to-region annotations, and localization ranks 200 EdgeBox proposals using region-phrase similarity scores.
  • Embedding network variants: Embedding-network ablations compare linear versus nonlinear branches, single- versus bi-directional losses, positive-region augmentation, and neighborhood sampling or constraints.The nonlinear variant uses two fully connected layers with ReLU, batch normalization, and L2 normalization; positive augmentation uses regions with IoU ≥0.7.
  • Embedding network results: Nonlinear branches improve performance, while bi-directional loss differs little from single-directional loss because phrase localization emphasizes phrase-to-region retrieval.The bi-directional setting uses λ1 = 1, λ2 = 4, λ3 = 0, λ4 = 0; the single-directional setting uses λ1 = 0, λ2 = 1, λ3 = 0, λ4 = 0.
  • Sampling results: Almost 5%: positive region augmentation improves R@1, while neighborhood sampling adds about 2% over standard sampling with one positive region per query phrase.Neighborhood sampling ensures at least two positive regions for each query phrase in a mini-batch.
  • Neighborhood effects: Neighborhood constraints slightly reduce R@1 but produce minor R@5 and R@10 improvements; mini-batch composition appears more responsible for gains than the neighborhood-preserving penalty.Region neighborhoods matter more than phrase neighborhoods in this task, because multiple positive regions are easier to obtain than multiple positive phrases.
  • Similarity network results: The similarity network reaches R@1 = 51.05, nearly matching embedding-network values of 50.69 and 51.03, with nonlinear branches and positive augmentation improving results.Using one fully connected layer after element-wise fusion gives R@1 = 36.61, while two layers give 49.39, nearly matching three layers.
  • Discussion: The two networks have comparable recall but different trade-offs: similarity training uses fewer pairs, whereas embedding training uses more triplets and provides an explicit shared space.The similarity network averages 2138 pairs per mini-batch; the embedding network averages 5378 triplets without neighborhood sampling and 10756 with it.

5 IMAGE-SENTENCE RETRIEVAL

The section evaluates embedding and similarity networks for bidirectional image-sentence retrieval on Flickr30K and MSCOCO, examining architecture, losses, sampling, and neighborhood constraints. The embedding network benefits from neighborhood sampling, while the similarity network performs poorly on image-sentence retrieval despite phrase-localization strength.

  • Task and datasets: Bidirectional image-sentence retrieval searches for matching sentences given an image and matching images given a sentence, evaluated here on Flickr30K and MSCOCO.MSCOCO contains 123287 images with five sentences each.
  • Experimental variants: Embedding-network experiments compare branch depth, single- versus bidirectional losses, neighborhood sampling, and neighborhood constraints.The study reports results for one versus two fully connected layers and multiple loss and sampling variants.
  • Embedding-network findings: Neighborhood sampling generally improves R@1 over conventional triplet sampling, whereas adding neighborhood constraints usually provides no significant further improvement.On MSCOCO, neighborhood constraints improve R@1 in both directions but slightly reduce R@10.
  • Embedding-network findings: The full embedding network with LSTM sentence encoding is comparable to or slightly worse than its HGLMM-feature version.This comparison is reported for the image-sentence retrieval experiments.
  • Similarity-network findings: The similarity network achieves very low image-sentence retrieval results, contrasting with its competitive phrase-localization accuracy.Its element-wise-product architecture was described as simpler but less flexible than the embedding network.
  • Sentence-to-sentence retrieval: Within-view sentence-to-sentence retrieval is used to examine whether neighborhood constraints improve retrieval among sentences describing the same image.The evaluation uses R@K on Flickr30K and MSCOCO with the same training, validation, and testing splits.
  • Model combination: Combining global image-sentence and local region-phrase models yields only a very small improvement over the image-sentence model.The region-phrase scores are inconsistent across images and phrases, limiting their usefulness for fine-grained cross-image ranking.

6 CONCLUSION AND FUTURE WORK

The conclusion compares two-branch architectures for region-to-phrase and image-to-sentence matching. The embedding network works across both tasks, whereas the similarity network is competitive for region-phrase matching but fails for image-sentence retrieval, and model combination remains an open direction.

  • Summary: The paper studies two-branch architectures for region-to-phrase and image-to-sentence matching on Flickr30K and MSCOCO.The authors report their results as the best to date on both tasks.
  • Embedding network: The embedding network maps image and text features into a joint nonlinear latent space and works well for both image-sentence and region-phrase tasks.Its objective uses multiple terms and requires comparatively costly triplet sampling.
  • Training: Triplet-sampling choices can significantly affect performance even without changing the objective function.The paper specifically investigates sampling within mini-batches.
  • Similarity network: The similarity network is an attractive alternative for region-phrase matching but fails for image-sentence retrieval.The conclusion highlights this difference between the two tasks.
  • Future work: Preliminary attempts to combine image-sentence and region-phrase models were unsuccessful and indicate an important direction for future research.The combined model's Flickr30K retrieval improvement was very small.
Loading 1704.03470v4…