PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval

Ding Jiang, Mang Ye

arXiv:2303.12501v1cs.CV

TL;DR

Text-to-image person retrieval requires effective alignment between visual and textual modalities despite intra-identity variation and modality heterogeneity. IRRA transfers full-CLIP knowledge, implicitly learns fine-grained token relations with MLM, and uses SDM for global alignment, achieving state-of-the-art results across three public datasets.

  • Problem

    Text-to-image person retrieval must align visual and textual representations despite intra-identity variations and vision-language heterogeneity.

  • Method

    IRRA uses full-CLIP initialization, implicit relation reasoning through multimodal interaction and MLM, and SDM based on KL divergence between similarity and label distributions.

  • Results

    IRRA achieves state-of-the-art performance across three public benchmark datasets, while IRR and SDM each improve Rank-1 accuracy in ablations.

  • Takeaways & Limitations

    Implicit fine-grained interaction can enhance global image-text representations without additional supervision or inference cost.

Abstract

from arXiv · show

Text-to-image person retrieval aims to identify the target person based on a given textual description query. The primary challenge is to learn the mapping of visual and textual modalities into a common latent space. Prior works have attempted to address this challenge by leveraging separately pre-trained unimodal models to extract visual and textual features. However, these approaches lack the necessary underlying alignment capabilities required to match multimodal data effectively. Besides, these works use prior information to explore explicit part alignments, which may lead to the distortion of intra-modality information. To alleviate these issues, we present IRRA: a cross-modal Implicit Relation Reasoning and Aligning framework that learns relations between local visual-textual tokens and enhances global image-text matching without requiring additional prior supervision. Specifically, we first design an Implicit Relation Reasoning module in a masked language modeling paradigm. This achieves cross-modal interaction by integrating the visual cues into the textual tokens with a cross-modal multimodal interaction encoder. Secondly, to globally align the visual and textual embeddings, Similarity Distribution Matching is proposed to minimize the KL divergence between image-text similarity distributions and the normalized label matching distributions. The proposed method achieves new state-of-the-art results on all three public datasets, with a notable margin of about 3%-9% for Rank-1 accuracy compared to prior methods.

1. Introduction

Text-to-image person retrieval must bridge intra-identity variation and vision-language heterogeneity while aligning images and text in a joint space. IRRA addresses limited intermediate modality interaction through implicit fine-grained relation learning, full-CLIP transfer, and distribution-based matching.

  • Text-to-image person retrieval matches a textual description to the corresponding person in a large image gallery.
  • Pose, viewpoint, illumination, descriptive order, ambiguity, and representation discrepancies make retrieval difficult.
  • Early global-matching methods align embeddings mainly through end-of-network losses, limiting middle-level modality interaction.
  • IRRA implicitly learns fine-grained visual-textual relations with self- and cross-attention plus masked language modeling, without additional supervision or inference cost.
  • IRRA initializes both encoders from full CLIP to transfer pretrained cross-modal alignment capabilities into person retrieval.
  • IRRA combines implicit relation reasoning with SDM, which minimizes KL divergence between similarity and normalized label distributions.
  • IRRA consistently outperforms state-of-the-art methods across CUHK-PEDES, ICFG-PEDES, and RSTPReid.

2. Related work

Related work progresses from separately encoded global image-text matching toward multimodal representation learning and cross-modal alignment. IRRA builds on this trajectory by learning more discriminative image-text embeddings.

  • Early person-retrieval methods used VGG and LSTM representations followed by matching-loss alignment.
  • Later methods improved visual and textual backbones with ResNet and BERT while designing losses for global image-text alignment.
  • IRRA extends prior work by learning more discriminative image-text embeddings.
  • Vision-language pre-training learns semantic correspondence from large-scale image-text pairs for downstream multimodal tasks.

3. Method

IRRA uses full CLIP encoders and implicit cross-modal relation learning to improve global image-text representations. It combines masked language modeling, similarity distribution matching, and identity classification during training.

  • Feature Extraction Dual-Encoder: IRRA directly initializes its dual-stream backbone with the full CLIP image and text encoders to enhance cross-modal alignment.The framework uses CLIP representations for both visual and textual modalities rather than separately pretrained unimodal encoders.
  • Implicit Relation Reasoning: Masked language modeling implicitly mines fine-grained visual-textual relations to learn more discriminative global image-text features.Masked textual tokens are predicted using both unmasked text and visual tokens, aligning contextualized representations with static masked-token embeddings.
  • Implicit Relation Reasoning: The multimodal interaction encoder fuses visual and masked textual representations through multi-head cross-attention followed by four transformer blocks.Text tokens are randomly masked before the text and vision transformer states are jointly processed.
  • Implicit Relation Reasoning: The IRR objective predicts the original tokens at masked positions from their multimodal contextual representations using an MLP classifier.The objective uses the masked-token set, vocabulary distributions, predicted token probabilities, and one-hot ground-truth distributions.
  • Similarity Distribution Matching: Similarity Distribution Matching minimizes KL divergence between cosine-similarity distributions for N × N image-text pairs and normalized matching-label distributions.Matching probabilities are computed with a temperature-scaled softmax over cosine similarities in a mini-batch.
  • Optimization: Identity identification is jointly optimized with IRR and SDM to cluster representations belonging to the same identity in the joint embedding space.The ID loss is a softmax classification loss over image or text identities and explicitly considers intra-modal distances.

4. Experiments

IRRA is evaluated on three public text-to-image person retrieval datasets using Rank-k metrics and ablations of its components, with consistent gains over prior methods. Component analyses show benefits from implicit relation reasoning, similarity distribution matching, identity loss, and the multimodal interaction module, while qualitative results expose a phrase-level limitation.

  • Comparison with State-of-the-Art Methods: 73.38% Rank-1 accuracy and 66.13% mAP establish IRRA's CUHK-PEDES results, where it outperforms all state-of-the-art methods.The directly fine-tuned CLIP Baseline achieves 68.19% Rank-1 accuracy on CUHK-PEDES.
  • Comparison with State-of-the-Art Methods: +2.63%, +3.69%, and +3.4% over Cfine on ICFG-PEDES are achieved by IRRA for Rank-1, Rank-5, and Rank-10, respectively.IRRA records 63.46%, 80.24%, and 85.82% on these metrics.
  • Comparison with State-of-the-Art Methods: +9.65%, +8.8%, and +6.6% over Cfine are achieved by IRRA on RSTPReid for Rank-1, Rank-5, and Rank-10, respectively.IRRA also surpasses IVT by +13.5%, +11.3%, and +9.4% on the same metrics.
  • Experimental Setup: IRRA is evaluated on CUHK-PEDES, ICFG-PEDES, and RSTPReid using Rank-1, Rank-5, Rank-10, mAP, and mINP metrics.Rank-k measures whether a matching person image appears within the top-k candidates; higher values indicate better performance.
  • Ablation Study: 3.04%, 4.22%, and 3.85% Rank-1 improvements result from adding IRR to the Baseline on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.IRR learns local relations through an MLM task for fine-grained cross-modal alignment.
  • Ablation Study: 11.11%, 6.62%, and 2.2% Rank-1 gains over CMPM are obtained by SDM across the three datasets, while the multimodal interaction module gives slight but consistent Rank-k gains.The SDM loss also improves the Baseline by 2.23%, 3.71%, and 3.15% on Rank-1 across the datasets.
  • Qualitative Results: IRRA retrieves more accurate top-10 results than the Baseline, but random single-token masking leaves phrase-level semantics unresolved.The authors attribute the limitation to not performing phrase-level masking and identify it as future work.

5. Conclusion

IRRA learns discriminative global image-text representations by combining implicit cross-modal relation reasoning with similarity distribution matching. Together, these components align images and text in a joint embedding space and yield significant gains on three benchmark datasets.

  • IRRA learns discriminative global image-text representations through cross-modal implicit relation reasoning and aligning.
  • The Implicit Relation Reasoning module uses masked language modeling to mine fine-grained relations between visual and textual tokens.
  • Similarity Distribution Matching enlarges variance between non-matching pairs and correlation between matching pairs.
  • The two modules collaborate to align images and text into a joint embedding space, producing significant gains on three popular benchmark datasets.
Loading 2303.12501v1…