PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Convolutional Neural Networks for Matching Image and Sentence

Lin Ma, Zhengdong Lu, Lifeng Shang, Hang Li

arXiv:1504.06063v5cs.CVcs.CLcs.NE

TL;DR

Image–sentence matching requires representing and aligning semantic content across words, phrases, and whole sentences. The paper proposes m-CNNs that encode images, compose sentence fragments, and learn multi-level inter-modal interactions; experiments report state-of-the-art bidirectional retrieval performance on Flickr30K and Microsoft COCO.

  • Problem

    Image–sentence matching involves word-, phrase-, and sentence-level correspondences, but prior models did not fully exploit them together, and CNN-based multimodal matching had not been studied.

  • Method

    m-CNNs use an image CNN and matching CNNs to encode image content, compose words into semantic fragments, and learn interactions between images and fragments at different levels.

  • Results

    The proposed m-CNNs achieve consistent state-of-the-art performance on bidirectional image and sentence retrieval, including Flickr30K and Microsoft COCO.

  • Takeaways & Limitations

    Multi-level convolutional matching effectively captures information needed for image–sentence matching in bidirectional retrieval.

Abstract

from arXiv · show

In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence. Our m-CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content, and one matching CNN learning the joint representation of image and sentence. The matching CNN composes words to different semantic fragments and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results on benchmark databases of bidirectional image and sentence retrieval demonstrate that the proposed m-CNNs can effectively capture the information necessary for image and sentence matching. Specifically, our proposed m-CNNs for bidirectional image and sentence retrieval on Flickr30K and Microsoft COCO databases achieve the state-of-the-art performances.

1. Introduction

Image–sentence matching must capture correspondences at word, phrase, and whole-sentence levels. The proposed m-CNN framework uses convolutional architectures to represent images, compose sentence fragments, and learn their interactions for bidirectional retrieval.

  • Image–sentence association supports image annotation, captioning, and natural-image search through higher scores for semantically correlated pairs.
  • Matching relations occur at multiple levels: words denote objects, phrases describe attributes or activities, and whole sentences express complete image meaning.
  • Existing models typically focus on a specific word, phrase, or sentence level rather than jointly exploiting all inter-modal correspondences.
  • CNNs had demonstrated strong image and sentence representation abilities, but their use for image–sentence matching had not been studied.
  • The m-CNN framework uses convolutional architectures to summarize images, compose words into semantic fragments, and learn matching interactions between images and fragments.
  • The proposed models validate this multi-level matching approach through bidirectional image and sentence retrieval with performance superior to state-of-the-art approaches.

2. Related Work

Prior work modeled image–text association using word-, phrase-, global-sentence-, or region-fragment correspondences and developed deep representations for retrieval and captioning. These approaches motivate richer models that combine local and global matching information.

  • Earlier image–text models focused on correlations between images and annotating words or phrases, which did not fully capture complex natural-sentence relations.
  • KCCA sought a shared image–sentence feature space, while later deep architectures addressed nonlinear inter-modal relations through richer representations.
  • Global sentence representations capture whole-sentence matching but neglect local sentence fragments and their correspondences to image content.
  • Finer-grained approaches aligned sentence fragments with image regions and used entities to collect region-to-phrase correspondences.
  • Image captioning systems including RVP, m-RNN, MNLM, NIC, DVSA, and LRCN learn image–sentence relations while generating captions.
  • CNN, time-delay, recursive, and recurrent networks provide sentence representations used for retrieval, classification, language modeling, and text generation.

3. m-CNNs for Matching Image and Sentence

m-CNNs use an end-to-end convolutional framework in which an image CNN encodes image content, matching CNNs model interactions with word- and phrase-level fragments, and an MLP produces the matching score.

  • Overall Architecture: m-CNN takes an image and sentence as inputs and generates their matching score through image CNN, matching CNN, and MLP components.The image CNN generates image representations, the matching CNN produces a joint representation, and the MLP outputs the final score.
  • Overall Architecture: The matching CNN composes sentence words into semantic fragments and lets image representations interact with them at different levels.The framework targets roughly word-, phrase-, and sentence-level matching relations.
  • End-to-End Learning: All m-CNN components and word representations are jointly learned end to end under supervision from matching instances.Joint learning can tune the image representation, word representations, and matching CNN for composition and matching.
  • Word-Level Matching CNN: MatchCNNwd uses local receptive fields and shared weights to model word composition and image interactions, then alternates convolution and max-pooling to form a global joint representation.Zero padding handles variable-length sentences, while gating removes convolution outputs produced from zero vectors.
  • Word-Level Matching CNN: Multimodal convolution makes image features interact with word-level fragments, capturing correspondences as words compose into higher-semantic phrases.The image representation and word representations enter the multimodal convolution process together, producing local matching signals.
  • Phrase-Level Matching CNN: Phrase-level CNNs compose words into short or long phrases before image interaction, while max-pooling filters undesired compositions and reduces representation dimensionality.Long phrases express richer meanings about objects, activities, and relative positions than individual words or short phrases.

4. Implementation details

The m-CNN implementation combines pretrained image encoders with matching CNNs that build joint image–sentence representations, then trains them using a margin-based ranking objective.

  • Image CNN: OverFeat and VGG image CNNs are initialized with ImageNet-learned parameters, and their final fully connected outputs form the image representation.The top softmax and last ReLU layers are removed before using the final fully connected layer.
  • Matching CNN: All four matching CNN variants use three convolution layers, three max-pooling layers, and a two-layer fully connected MLP.Their multimodal convolution layers occur at different depths depending on the variant.
  • Matching CNN: The matching CNN joint representation is fed into an MLP with one hidden layer of size 400.
  • Training: The models learn with contrastive sampling and a ranking loss that compares correlated and randomly sampled uncorrelated image–sentence pairs.The objective forces the correlated pair’s matching score above the uncorrelated pair’s score by margin µ, set to 0.5.
  • Training: Optimization uses stochastic gradient descent with minibatches of 100∼150, while early stopping and dropout with probability 0.1 reduce overfitting.ReLU is used throughout the m-CNNs.

5. Experiments

The experiments evaluate m-CNNs on bidirectional image and sentence retrieval across Flickr8K, Flickr30K, and Microsoft COCO, comparing them with established models. Performance depends on training-data scale, image-CNN quality, and combining matching relations across semantic levels.

  • Experimental Setup: The evaluation covers bidirectional image retrieval and sentence retrieval on Flickr8K, Flickr30K, and Microsoft COCO.The reported metrics are median rank and R@K for K = 1, 5, 10.
  • Datasets: Flickr8K contains 8,000 images, Flickr30K contains 31,783 images, and Microsoft COCO provides 82,783 training and 40,504 validation images.Each image in these datasets is paired with five descriptive sentences.
  • Bidirectional Image and Sentence Retrieval: On Flickr8K, FV performs best, while m-CNN remains superior to the other compared methods.The paper attributes FV’s advantage mainly to beneficial sentence-modeling bias under scarce training data.
  • Bidirectional Image and Sentence Retrieval: On Flickr30K and Microsoft COCO, m-CNNENS with VGG outperforms most competitors across the reported retrieval metrics.On Flickr30K, only m-RNN-vgg, FV, and RTP exceed it on sentence retrieval measured by R@1; on image retrieval it is consistently better than all competitors.
  • Bidirectional Image and Sentence Retrieval: On Microsoft COCO, m-CNN performance improves over Flickr8K and Flickr30K, and m-CNNENS with VGG consistently outperforms competitors on image retrieval.Only DVSA exceeds it on sentence retrieval measured by median rank.
  • Performances of Different m-CNNs: The sentence-level m-CNNst consistently outperforms other individual m-CNN variants, while m-CNNENS performs best by combining complementary levels.The results indicate that word-, phrase-, and sentence-level matching relations jointly capture complicated image-sentence correspondences.
  • Influence of Image CNN: Retrieval performance depends heavily on the image CNN: VGG-initialized m-CNNs substantially outperform OverFeat-initialized models.The paper reports ImageNet top-5 classification errors of 7% for VGG and 14% for OverFeat, and proposes incorporating stronger region-based image CNN features in future work.
  • Composition Abilities of m-CNNs: Randomly reshuffling sentence words greatly lowers matching scores, supporting composition of naturally ordered words into semantic fragments.This experiment tests the models’ ability to compose words for inter-modal matching at different levels.

6. Conclusion

The paper proposes m-CNNs for image-sentence matching, using convolutional architectures to compose sentence fragments and model interactions at multiple levels. On bidirectional retrieval benchmarks, the models achieve consistent state-of-the-art performance.

  • 6. Conclusion: m-CNNs use convolutional architectures to compose semantic sentence fragments and learn their interactions with image representations at different levels.The framework is designed to exploit inter-modal matching relations between images and sentences.
  • 6. Conclusion: The proposed m-CNNs achieve consistent state-of-the-art performance on bidirectional image and sentence retrieval.The conclusion reports this outcome for the proposed models as a whole.
Loading 1504.06063v5…