PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, Yi-Dong Shen

arXiv:1711.05535v4cs.CVcs.MM

TL;DR

The paper targets the difficulty of fine-grained image-text matching across heterogeneous modalities and the limitations of ranking-loss-only training. It introduces instance loss with a dual-path CNN, achieving competitive results on generic retrieval datasets and a reported +18% improvement on CUHK-PEDES.

  • Problem

    Ranking loss focuses on cross-modal distances without explicitly modeling intra-modal distributions, while fixed visual and textual features may miss fine-grained matching cues.

  • Method

    Instance loss treats each image with its associated sentences as an unsupervised distinct class, combined with an end-to-end dual-path CNN for image-text embedding learning.

  • Results

    The method achieves competitive accuracy on Flickr30k, MSCOCO, and CUHK-PEDES, including a +18% improvement on CUHK-PEDES.

  • Takeaways & Limitations

    Instance loss provides initialization for ranking loss, while end-to-end learning directly trains image and text representations from the data.

  • Takeaways & Limitations

    Ranking-loss-only end-to-end training may get stuck in a local minimum and fail to separate subtly different instances.

Abstract

from arXiv · show

Matching images and sentences demands a fine understanding of both modalities. In this paper, we propose a new system to discriminatively embed the image and text to a shared visual-textual space. In this field, most existing works apply the ranking loss to pull the positive image / text pairs close and push the negative pairs apart from each other. However, directly deploying the ranking loss is hard for network learning, since it starts from the two heterogeneous features to build inter-modal relationship. To address this problem, we propose the instance loss which explicitly considers the intra-modal data distribution. It is based on an unsupervised assumption that each image / text group can be viewed as a class. So the network can learn the fine granularity from every image/text group. The experiment shows that the instance loss offers better weight initialization for the ranking loss, so that more discriminative embeddings can be learned. Besides, existing works usually apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So in a minor contribution, this paper constructs an end-to-end dual-path convolutional network to learn the image and text representations. End-to-end learning allows the system to directly learn from the data and fully utilize the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO), experiments demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language based person retrieval, we improve the state of the art by a large margin. The code has been made publicly available.

1 INTRODUCTION

The paper addresses fine-grained image-text matching by learning shared embeddings with an instance loss and an end-to-end dual-path CNN. It reports competitive accuracy across three retrieval datasets.

  • Motivation: Image-text matching must distinguish fine-grained visual and textual differences while mapping heterogeneous modalities into one shared feature space.Examples include distinguishing black, gray, and multiple dogs in both images and text.
  • Motivation: Ranking loss may overlook intra-modal feature distributions, produce similar descriptors for subtly different instances, and become trapped in a local minimum.It focuses on distances between visual-textual pairs rather than explicitly modeling differences within one modality.
  • Instance Loss: The paper treats each image with its associated sentences as a distinct unsupervised class, forming the basis of instance loss.This transfers the usefulness of class-label supervision to instance-level retrieval without annotated category labels.
  • Instance Loss: Instance loss classifies image-text groups to capture fine-grained intra-modal differences and provide initialization for ranking-loss training.On MSCOCO, it classifies 113,287 groups and converges without hyper-parameter tuning.
  • Model: A dual-path CNN learns image and text representations end to end instead of relying on fixed CNN features and word2vec inputs.The CNN+CNN structure supports direct fine-tuning from data and supervision.
  • Results: The method achieves competitive accuracy against state-of-the-art methods on Flickr30k, MSCOCO, and CUHK-PEDES.The conclusion separately reports a +18% improvement on CUHK-PEDES.

2 RELATED WORKS

The paper situates its approach within multimodal retrieval, contrasting category-level methods and fixed-feature pipelines with instance-level retrieval using an unsupervised instance loss. It also proposes a dual-path CNN that learns visual and textual representations end to end.

  • Unlike methods using fixed CNN image features or word vectors, end-to-end fine-tuning learns representations directly from the retrieval data.
  • The proposed instance loss treats each training image/text group as a distinct class and applies softmax training under an unsupervised assumption.
  • Instance loss is presented as improving ranking-loss training by providing better initialization and yielding consistent retrieval improvements.

3 PROPOSED CNN STRUCTURE

The proposed architecture jointly learns image and sentence descriptors with separate convolutional paths, then supports shared classification and retrieval objectives. Its visual path fine-tunes ResNet-50, while its text path converts padded sentence codes into convolutional representations with position-shift augmentation and residual blocks.

  • The dual-path CNN consists of one deep image CNN and one deep text CNN that learn visual and textual representations end to end.
  • 3.1 Deep Image CNN: The image branch fine-tunes an ImageNet-pretrained ResNet-50 and outputs a 2,048-dimensional visual descriptor after additional fully connected, batch-normalization, and ReLU layers.
  • 3.2 Deep Text CNN: The text branch represents sentences as n × d codes, filters rare words with word2vec, and uses a fixed length of 32 with padding or clipping.
  • 3.2 Deep Text CNN: Position shift randomly distributes zero padding before and after shorter sentences to provide text data augmentation.
  • 3.2 Deep Text CNN: The text CNN begins with a 1 × 1 × d × 300 convolution acting as a lookup table, followed by residual blocks using 1 × 2 filters and shortcut connections.
  • 3.2 Deep Text CNN: The image and text CNNs use analogous residual structures, with 3 × 3 image filters and 1 × 2 text filters preserving block input size through shortcut connections.

4 PROPOSED INSTANCE LOSS

The paper proposes instance loss for instance-level image-text matching by treating each image/text group as a distinct class. It complements ranking loss by modeling intra-modal differences and providing discriminative initialization.

  • Ranking-loss limitation: Ranking loss models cross-modal distances but does not explicitly account for feature distributions within a single modality.This can make images or texts with slightly different semantics receive similar descriptors and can produce hard negative pairs.
  • Instance-loss formulation: Instance loss treats each image/text group as a class and uses shared final-layer weights for image and text features.The shared weights encourage modalities from the same group to occupy nearby regions while separating different groups.
  • Feature geometry: Instance loss explicitly encourages fine-grained image discrimination, producing lower feature correlation and greater orthogonality than ImageNet-pretrained features after Stage I.The paper links this behavior to sensitivity to details such as balls, sticks, and frisbees.
  • Similarity distributions: 0.2405 versus 0.0237: instance loss alone yields higher mean intra-modal intra-class similarity Dp than inter-class similarity Dn.The reported distributions indicate separation between positive and negative intra-modal pairs.
  • Similarity distributions: Srank = 0.2563, Sinstance = 0.1633, and Sfull = 0.0914, with the full model showing the greatest feature separability.Lower S denotes less overlap between positive-pair and negative-pair similarity distributions.
  • Loss combination: The training design uses instance loss first and combines it with ranking loss later, with the two losses described as complementary in Stage II.Instance loss alone can provide a better optimization starting point, while its joint use with ranking loss has a regularization effect.

5 A TWO-STAGE TRAINING PROCEDURE

The network is trained in two stages: instance loss first initializes the model, followed by end-to-end fine-tuning with both instance and ranking losses.

  • Training policy: The procedure is explicitly divided into two training stages, and experiments show that this policy helps training.Stage I precedes Stage II after convergence.
  • Stage I: Stage I freezes pretrained image-CNN weights and uses only instance loss to tune the remaining components.The text CNN is largely learned from scratch, so simultaneous training could compromise the pretrained image CNN.
  • Stage II: Stage II fine-tunes the entire network end to end with instance loss and ranking loss combined.Image-CNN weights are also fine-tuned, and both classification and ranking errors are considered.

6 EXPERIMENT

Experiments evaluate the method on three retrieval datasets using standard retrieval metrics, implementation studies, and comparisons with prior work. Results show competitive or improved retrieval, with instance loss, joint losses, end-to-end fine-tuning, and discriminative word representations contributing to performance.

  • Datasets and evaluation: The experiments use Flickr30k, MSCOCO, and CUHK-PEDES, reporting Recall@K and Median Rank for bidirectional or person retrieval.Recall@K rewards true matches in the top K, while lower Median Rank is better.
  • Comparison with state of the art: 44.4% Recall@1 and Med r = 2 are achieved on CUHK-PEDES with ResNet-50 for language-based pedestrian retrieval.This exceeds the second-best method by 18.46% in Recall@1 accuracy.
  • Comparison with state of the art: 65.6% Recall@1 with Med r = 1 is obtained on MSCOCO 1K-test-image image queries, while text queries reach 47.1% Recall@1 with Med r = 2.On the 5K-test-image setting, image queries reach 41.2% Recall@1 and text queries 25.3%.
  • Mechanism study: Instance loss outperforms ranking loss during initial training because it learns intra-modal discriminative descriptors before cross-modal optimization.The first stage fixes the image CNN, whereas the second stage fine-tunes the entire network with ranking loss.
  • Mechanism study: Combining instance loss and ranking loss produces higher validation performance than using either loss alone and helps regularize the model.The comparison uses the same ResNet-50 network across loss settings.
  • Mechanism study: End-to-end fine-tuning improves rank-1 accuracy by approximately 10% on Flickr30k and MSCOCO and improves CUHK-PEDES Recall@1 by 29.37%.The improvement is attributed in the experiments to the differing data distribution between ImageNet and surveillance-camera pedestrian imagery.
  • Mechanism study: Word2vec initialization outperforms random initialization by 1% to 2% for text CNN training.This occurs even though word2vec is not trained on the target dataset.
  • Mechanism study: The model identifies objective and color words as discriminative, while removing conjunctions has little effect on matching confidence.Examples of discriminative words include basketball, swing, and purple.

7 CONCLUSION

The paper concludes that instance loss treats each image/text group as an unsupervised class, improving ranking-loss initialization and regularizing training. A dual-path CNN enables end-to-end image and text learning, yielding competitive generic retrieval and a +18% improvement on CUHK-PEDES.

  • Contributions: Instance loss views every image/text group as one class and provides initialization for ranking loss while regularizing training.The assumption is that each image/text group is distinct.
  • Contributions: The dual-path CNN supports end-to-end training on both image and text branches.The paper presents this as a minor contribution alongside instance loss.
  • Results: +18% improvement is reported on the person retrieval dataset CUHK-PEDES, alongside competitive results on Flickr30k and MSCOCO.The code is publicly available.

APPENDIX

The appendix provides additional bidirectional image-text retrieval examples for Flickr30k and MSCOCO. Image-query results show retrieved images, while image-query text retrieval shows ranked sentences with true and false matches marked.

  • Supplementary examples: The appendix identifies the additional examples as image-text bidirectional retrieval results, with true matches in green.This summarizes the role of Figures 12–15.
  • Image search: Additional image search examples use text queries on Flickr30k and MSCOCO, with top-five images ordered by confidence and true matches shown in green.The examples are presented as supplementary qualitative retrieval results.
  • Text search: Additional text search examples use image queries on Flickr30k and MSCOCO, with the top five sentences ranked by confidence.True matches are shown in green and false matches in red.
Loading 1711.05535v4…