PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Representation Learning for Text-to-Image Matching

Nikolaos Sarafianos, Xiang Xu, Ioannis A. Kakadiaris

arXiv:1908.10534v1cs.CV

TL;DR

Text-to-image matching must handle both variable, noisy textual descriptions and difficult cross-modal distance measurement. TIMAM addresses these challenges with adversarial modality-invariant embeddings and cross-modal objectives, while using a deep language model for improved word embeddings; adversarial learning improved performance by approximately 3% on CUHK-PEDES and 1.8% on Flickr30K.

  • Problem

    Text-to-image matching is challenged by substantial variability and errors in textual descriptions, alongside difficulty accurately measuring distances between text and image features.

  • Method

    TIMAM jointly trains an adversarial discriminator with identification and cross-modal matching objectives to learn discriminative modality-invariant embeddings, while applying a deep language model to improve textual embeddings.

  • Results

    Adversarial learning improved matching by approximately 3% on CUHK-PEDES and approximately 1.8% on Flickr30K in ablation studies.

  • Takeaways & Limitations

    Adversarial learning is well-suited for cross-modal matching, and pre-trained language models can improve feature representations for text-to-image matching.

  • Takeaways & Limitations

    Image reconstruction alternatives produced very blurry reconstructions on CUHK-PEDES and Flickr30K, limiting their usefulness for learning better features.

Abstract

from arXiv · show

For many computer vision applications such as image captioning, visual question answering, and person search, learning discriminative feature representations at both image and text level is an essential yet challenging problem. Its challenges originate from the large word variance in the text domain as well as the difficulty of accurately measuring the distance between the features of the two modalities. Most prior work focuses on the latter challenge, by introducing loss functions that help the network learn better feature representations but fail to account for the complexity of the textual input. With that in mind, we introduce TIMAM: a Text-Image Modality Adversarial Matching approach that learns modality-invariant feature representations using adversarial and cross-modal matching objectives. In addition, we demonstrate that BERT, a publicly-available language model that extracts word embeddings, can successfully be applied in the text-to-image matching domain. The proposed approach achieves state-of-the-art cross-modal matching performance on four widely-used publicly-available datasets resulting in absolute improvements ranging from 2% to 5% in terms of rank-1 accuracy.

1. Introduction

The paper targets text-to-image retrieval despite linguistic variability and difficult cross-modal distance measurement. TIMAM combines adversarial modality alignment, cross-modal objectives, and BERT-based word embeddings, improving rank-1 accuracy over prior methods.

  • Challenges: Text-to-image matching must retrieve relevant images from descriptions whose wording, length, mistakes, and emphasis can vary substantially.Descriptions of the same image may differ because annotators select different information or include irrelevant details.
  • Challenges: The best prior text-to-image matching method on CUHK-PEDES remained below 50% rank-1 accuracy.Existing losses bring same-identity cross-modal features together and separate different identities, but performance remained unsatisfactory.
  • Approach: TIMAM learns modality-invariant representations by combining adversarial discrimination with identification and cross-modal matching objectives.Its discriminator distinguishes visual from textual features, while the jointly trained objectives align features for matching.
  • Approach: BERT is fine-tuned to improve word embeddings for text-to-image matching.The paper applies a pretrained language representation model from NLP to this computer-vision task.
  • Results: 2% to 5% rank-1 accuracy improvements were observed over previous best-performing techniques.The paper reports these gains for the proposed learning approach and states that adversarial learning produces more discriminative embeddings.

2. Related Work

Related work applies cross-modal embeddings across several applications, while text-to-image matching commonly combines visual features, LSTM text features, and distance-based losses. TIMAM instead emphasizes adversarial modality-invariant representations.

  • Applications: Cross-modal embeddings support applications including facial-and-voice PINs, generative feature learning, and domain adaptation.Text-to-image matching is another established cross-modal application involving descriptions of objects, birds, flowers, and people.
  • Text-to-Image Matching: Many text-to-image matching methods extract image features, encode text with an LSTM, and optimize a loss measuring distance between embeddings.This procedure is used across applications including the newer person-search setting, which retrieves individual frames from textual descriptions.
  • TIMAM: TIMAM introduces a domain discriminator to learn more discriminative representations than prior cross-modal matching approaches.The related-work context motivates a shift beyond procedures centered on feature extraction and cross-modal distance losses.

3. Methodology

TIMAM jointly learns discriminative visual and textual representations for bidirectional cross-modal retrieval using identification, projection-matching, and adversarial objectives. Its matching loss aligns paired modalities through batchwise scalar projections, while adversarial training encourages modality-invariant features.

  • 3. Methodology: TIMAM learns visual and textual representations for text-to-image and image-to-text retrieval.Training targets accurate retrieval of an input identity or category from the other modality.
  • 3.2. Cross-Modal Matching: The identification objective uses norm-softmax cross entropy to separate identities or categories in each modality.It normalizes classification weights so learning focuses on angular relationships rather than feature magnitude.
  • 3.2. Cross-Modal Matching: Cross-modal projection matching compares every image-text pair in a batch, assigning larger scalar projections to more similar embeddings and matching predicted probabilities to true pair probabilities.The loss is defined as a KL divergence, with the same procedure applied in both image-to-text and text-to-image directions.
  • 3.3. Adversarial Representation Learning: Adversarial representation learning feeds visual and textual features to a discriminator and trains the feature generators to fool it.This objective is intended to produce modality-invariant representations capable of text-to-image matching.
  • 3.4. Training and Testing Details: The total TIMAM loss sums identification, cross-modal matching, and discriminator losses.Figure 3 presents these as the three learning objectives: classification, modality discrimination, and batchwise cross-modal matching.
  • 3.4. Training and Testing Details: Training uses SGD with momentum for image and discriminator networks and Adam for textual networks, with 512-dimensional feature vectors.The reported learning rate starts at 2 × 10^-4 and is reduced to 2 × 10^-6 when validation loss plateaus.

4. Experiments

TIMAM is evaluated on four widely used datasets spanning pedestrians, objects, scenes, birds, and flowers, with quantitative gains, component ablations, and qualitative retrieval analysis. The experiments show benefits from BERT and adversarial representation learning, while reconstruction-based alternatives are less effective on varied datasets.

  • Datasets and quantitative results: TIMAM achieves state-of-the-art matching on four publicly available datasets covering pedestrians, objects, scenes, birds, and flowers.The evaluation follows the established protocols for CUHK-PEDES, Flickr30K, CUB, and Flowers.
  • Datasets and quantitative results: More than 5% absolute rank-1 improvement over the previous best method is reported on CUHK-PEDES.The gain is attributed to identification, cross-modal matching, and adversarial learning objectives.
  • Datasets and quantitative results: 2.2% and 3.4% rank-1 gains, plus 3.6% and 2.4% AP@50 gains, are reported across the CUB and Flowers datasets.The passage reports these increases for the evaluated cross-modal matching results.
  • Ablation studies: BERT raises CUHK-PEDES rank-1 accuracy to 52.97% after the identification and cross-modal projection losses reach 49.85% together.Adding adversarial representation learning produces further relative improvements of 2.9% with BERT and 3% without BERT.
  • Ablation studies: ARL improves Flickr30K rank-1 performance from 51.2% to 53.1% for image-to-text and from 41.0% to 42.6% for text-to-image matching.These results support the contribution of adversarial representation learning beyond the other objectives.
  • Discussion of alternatives: Triplet or quadruplet losses were avoided because they add training overhead and may require hard-mining schemes, while reconstruction alternatives were ineffective on varied datasets.Reconstructions were very blurry on CUHK-PEDES and Flickr30K, unlike the more patterned Birds and Flowers datasets.

5. Conclusion

The paper addresses linguistic variability and cross-modal distance measurement with TIMAM, which jointly uses adversarial, identification, and cross-modal matching objectives. It reports modality-invariant embeddings, benefits from a deep language model, and state-of-the-art results across four datasets.

  • Conclusion: TIMAM addresses linguistic variability and the difficulty of measuring distances between multimodal features in cross-modal matching.These are identified as central challenges for learning discriminative representations.
  • Conclusion: TIMAM jointly trains an adversarial discriminator with identification and cross-modal matching objectives to produce discriminative modality-invariant embeddings.The discriminator distinguishes visual from textual inputs while adversarial training encourages modality-invariant representations.
  • Conclusion: A deep language model improves cross-modal matching by learning better textual embeddings, and TIMAM achieves state-of-the-art results on four widely used datasets.The conclusion also identifies adversarial learning as well-suited for text-image matching.

Discussion on Novelty

TIMAM’s novelty lies in adversarial representation learning, fine-tuned BERT embeddings, and a simpler reproducible training approach for cross-modal matching.

  • Adversarial representation learning: TIMAM uses an adversarial discriminator to learn modality-invariant, more discriminative representations for cross-modal matching.The discriminator distinguishes visual from textual inputs, while adversarial learning encourages features that can fool it.
  • Adversarial representation learning: Ablations improved performance by approximately 3% on CUHK-PEDES and 1.8% on Flickr30K when the adversarial discriminator was used.
  • BERT-based textual embeddings: TIMAM applies a pre-trained language model with fine-tuning to learn improved textual embeddings for text-to-image matching.
  • Comparison with prior methods: Compared with CMPM, TIMAM employs a domain discriminator and identification losses rather than cross-modal projections in the classification loss.The authors report that cross-modal projections made an insignificant contribution in their setting.
  • Comparison with prior methods: For Flickr30K, TIMAM combines better textual embeddings from fine-tuned BERT with its adversarial representation learning framework.
  • Reproducibility: TIMAM is described as easy to reproduce using an image backbone, a deep language model, and the procedure in Algorithm 1.The paper contrasts this with prior methods using complex attention mechanisms or text reconstruction objectives.

7 Repeat steps 4-6 for the text modality to compute LT

The training procedure computes an adversarial loss through the discriminator and produces the trained network weights as output.

  • The procedure computes adversarial loss by passing visual and textual embeddings through the discriminator.
  • The discriminator receives φ(Vi) and τ(Ti) as its visual and textual inputs.
  • The algorithm outputs the resulting network weights.

Implementation Details

The experiments use CUHK-PEDES and Flickr30K alongside BERT-based text processing, ResNet-101 image features, and a fully connected discriminator.

  • Datasets: CUHK-PEDES contains 40,206 images of 13,003 identities, while Flickr30K contains 31,783 images with five descriptions per image.
  • Text processing: BERT processing uses a maximum sentence length of 50 words for CUHK-PEDES and 30 words for the other datasets, with shorter sentences zero-padded.
  • Image domain: The image backbone is ResNet-101, whose pooled features are projected to 512-dimensional image embeddings.For 224 × 224 × 3 inputs, ResNet-101 produces 7 × 7 × 2,048 representations before pooling and projection.
  • Text domain: The text pipeline extracts 768-dimensional word vectors, processes them with a bidirectional LSTM, and projects the output to 512-dimensional text embeddings.
  • Discriminator: The discriminator uses two fully connected layers with batch normalization and LeakyReLU to predict the input modality from an embedding.
  • Training details: Training uses SGD with momentum for image and discriminator networks, Adam for textual networks, and initially freezes the deep language model before fine-tuning.

Extended Quantitative Results

Complete results show that TIMAM surpasses all listed methods for text-to-image matching but performs below GXN for image-to-text matching; qualitative results show attribute and object matching.

  • Quantitative results: TIMAM surpasses all tested methods in text-to-image matching on the CUHK-PEDES and Flickr30K evaluations.Tables 7 and 8 provide complete comparisons against all tested approaches.
  • Quantitative results: TIMAM demonstrates inferior image-to-text matching performance compared with GXN.
  • Qualitative results: Qualitative retrieval results show matching of soft-biometric attributes, clothing, hats, and backpacks on CUHK-PEDES and Flickr30K.
Loading 1908.10534v1…