PaperlayerView exact revision on arXiv

Source-linked AI summary

See Finer, See More: Implicit Modality Alignment for Text-based Person Retrieval

Xiujun Shu, Wei Wen, Haoqian Wu, Keyu Chen, Yiran Song, Ruizhi Qiao, Bo Ren, Xiao Wang

arXiv:2208.08608v2cs.CV

TL;DR

Text-based person retrieval needs fine-grained alignment between visual and textual modalities, but existing methods rely on separate models or salient alignments that can miss subtle pairs. IVT uses one shared network with MLA and BMM to learn common representations and implicit alignments. Experiments on three public datasets report state-of-the-art performance, while the paper notes privacy risks from surveillance data collection and use.

  • Problem

    Existing text-based person retrieval methods use separate models or salient alignment strategies that can limit modality interaction and miss subtle visual-textual pairs.

  • Method

    IVT uses a single network for both modalities, with multi-level alignment for sentence-, phrase-, and word-level matching and bidirectional mask modeling for additional semantic alignments.

  • Results

    IVT achieves state-of-the-art performance in experiments on CUHK-PEDES, RSTPReID, and ICFG-PEDES.

  • Takeaways & Limitations

    Unified networks and implicit alignments can mine fine-grained semantic correspondences for text-based person retrieval without explicit body-part alignment.

  • Takeaways & Limitations

    The paper identifies potential privacy breaches from pedestrian surveillance data, requiring consented collection and regulated use.

Abstract

from arXiv · show

Text-based person retrieval aims to find the query person based on a textual description. The key is to learn a common latent space mapping between visual-textual modalities. To achieve this goal, existing works employ segmentation to obtain explicitly cross-modal alignments or utilize attention to explore salient alignments. These methods have two shortcomings: 1) Labeling cross-modal alignments are time-consuming. 2) Attention methods can explore salient cross-modal alignments but may ignore some subtle and valuable pairs. To relieve these issues, we introduce an Implicit Visual-Textual (IVT) framework for text-based person retrieval. Different from previous models, IVT utilizes a single network to learn representation for both modalities, which contributes to the visual-textual interaction. To explore the fine-grained alignment, we further propose two implicit semantic alignment paradigms: multi-level alignment (MLA) and bidirectional mask modeling (BMM). The MLA module explores finer matching at sentence, phrase, and word levels, while the BMM module aims to mine \textbf{more} semantic alignments between visual and textual modalities. Extensive experiments are carried out to evaluate the proposed IVT on public datasets, i.e., CUHK-PEDES, RSTPReID, and ICFG-PEDES. Even without explicit body part alignment, our approach still achieves state-of-the-art performance. Code is available at: https://github.com/TencentYoutuResearch/PersonRetrieval-IVT.

1 Introduction

Text-based person retrieval seeks a shared visual-textual representation, but separate encoders and salient-only alignment limit modality interaction and subtle matching. IVT addresses these issues with one network plus multi-level and mask-based implicit alignment, achieving state-of-the-art performance in experiments.

  • Motivation: Text-based person retrieval learns a common latent space between visual and textual modalities for detailed person search.Text descriptions are accessible and can provide complementary information when images are missing.
  • Limitations of prior work: Separate visual and textual models lack full modality interaction, while attention-based local alignment can overlook subtle complementary cues.Existing approaches also rely on manually processed or salient parts, creating limitations for fine-grained matching.
  • Proposed approach: IVT uses a single network for both modalities, sharing parameters to support common-space mapping while retaining separate feature extraction for retrieval speed.The framework is designed to avoid the interaction limitations of separate models and the slow inference of cross-attention.
  • Proposed approach: MLA explores sentence-, phrase-, and word-level matching, while BMM mines additional semantic alignments without extra manual labeling.Together, the paradigms target finer and more numerous visual-textual correspondences.
  • Results: Experiments on three public datasets show that IVT achieves state-of-the-art performance.The reported contributions include the unified framework and the two implicit alignment paradigms.

2 Related Work

Prior text-based person retrieval methods commonly learn visual-textual alignment with separate encoders and local alignment mechanisms. This work instead emphasizes full modality interaction and simpler local alignment.

  • Text-based Person Retrieval: Early methods use ResNet50 and LSTM-based representations followed by matching losses to align visual and textual modalities.Later approaches add divergence-based alignment, identity cues, adversarial objectives, cross-modal matching, or scene graphs.
  • Text-based Person Retrieval: Most existing approaches exploit local alignments, while the proposed work studies full modality interaction and simpler local alignment.The framework is presented as addressing both issues through a different modality-alignment perspective.

3 METHODOLOGY

IVT combines a unified visual-textual network with implicit alignment paradigms that map both modalities into a common space while preserving modality-specific cues. Its inputs are image-text pairs represented through visual patches and textual token embeddings.

  • 3.1 Overview: The IVT framework contains a unified visual-textual network, MLA, and BMM for learning common-space mappings and fine-grained alignment.The network uses shared and modality-specific modules, while MLA and BMM target finer and more semantic correspondences.
  • 3.1 Overview: MLA explores local and global alignment through sentence-, phrase-, and word-level matchings, while BMM mines more alignments through random masking.The two paradigms are implicit and avoid manually processed parts or salient-part selection.
  • 3.2 Unified Visual-Textual Network: The model receives image-text pairs with identity labels, splitting each image into patches and converting text words into token vectors.Visual inputs are patch-embedded, while textual inputs use pretrained word embeddings and tokenization.
  • 3.2 Unified Visual-Textual Network: Visual patch embeddings are augmented with learnable class, position, and type embeddings before entering the network.The visual sequence includes a learnable class token alongside patch embeddings.
  • 3.2 Unified Visual-Textual Network: Text sequences use pretrained word embeddings and include start and end tokens, with M denoting the tokenized subword length.The textual input may contain one or several sentences composed of word sequences.

embedding.

The unified network is motivated by the need to combine modality interaction with efficient retrieval. Shared parameters help map visual and textual representations into a common space.

  • Embedding: Separate encoders lack full modality interaction, whereas cross-attention requires encoding every image-text pair at inference and slows retrieval.The unified network is introduced as a faster alternative for the text-based person retrieval task.

Visual-Textual Encoder.

The encoder uses shared layer normalization and multi-head self-attention across visual and textual streams, while retaining modality-specific feed-forward layers. The final class tokens provide normalized global representations for both modalities.

  • Shared encoder: Shared layer normalization and multi-head self-attention learn common visual-textual representations across the two modalities.The shared parameters are intended to capture statistically common values between visual and textual token embeddings.
  • Modality-specific processing: Modality-specific feed-forward layers preserve separate processing for visual and textual features within each encoder block.
  • Encoder blocks: The output of each block becomes the input to the next block, while the final block produces the visual or textual representation.
  • Global representations: The last-block class tokens serve as normalized global representations, with 768 dimensions for both modalities.

Output.

The method replaces manually specified or attention-selected parts with implicit fine-grained alignment guided by textual descriptions. MLA uses multiple textual granularities, while BMM masks tokens to recover subtler cross-modal correspondences.

  • Multi-Level Alignment: Phrase-level representations are formed by splitting text at periods and commas to describe partial human-body appearance characteristics.
  • Multi-Level Alignment: Implicit alignment avoids explicitly defining visual semantic parts and instead explores aligned visual regions through textual descriptions.
  • Alignment motivation: Top-k attention-based local alignment may focus on salient cues already captured globally, while subtle complementary cues remain underused.
  • Bidirectional Mask Modeling: BMM randomly masks image and text tokens and forces aligned outputs, encouraging the model to use remaining patches or words.
  • Bidirectional Mask Modeling: Unlike image reconstruction methods, BMM focuses on cross-modal matching rather than reconstructing images.

3.4 Loss Function

The loss function uses cross-modal projection matching to learn visual-textual alignment from matching probabilities and normalized target probabilities. It computes matching in both image-to-text and text-to-image directions.

  • CMPM loss: CMPM learns visual-textual alignment by comparing predicted matching probabilities with normalized true matching probabilities.The predicted and target distributions are defined over visual-textual feature pairs.
  • CMPM loss: The matching loss represents the KL divergence from the target distribution q to the predicted distribution p.
  • Bidirectional matching: The total matching loss is computed bidirectionally: image-to-text and text-to-image.

4 Experiment

Experiments evaluate IVT on three person-retrieval benchmarks, component ablations, masking ratios, qualitative alignments, and computational efficiency. IVT achieves the best reported performance across all three datasets while improving fine-grained matching and retaining competitive retrieval efficiency.

  • Comparison with State-of-the-art Methods: 65.59% R1, 83.11% R5, and 89.20% R10 are achieved on CUHK-PEDES, improving over the baseline by +9.84%, +7.43%, and +5.07%, respectively.The baseline records 55.75%, 75.68%, and 84.13% on the same metrics.
  • Comparison with State-of-the-art Methods: IVT yields the best performance on all metrics across CUHK-PEDES, RSTPReid, and ICFG-PEDES.The authors attribute this to the unified visual-textual network and implicit semantic alignment paradigms.
  • Ablation Study: 61.00% R1 and 56.88% mAP follow from adding MLA, while BMM raises baseline results to 60.43% R1 and 56.65% mAP on CUHK-PEDES.MLA improves R1 and mAP by +5.25% and +3.52%; BMM improves them by +4.68% and +3.29%.
  • Ablation Study: R1 and mAP peak at a masking ratio of 0.3, whereas larger ratios progressively reduce performance.With zero masking, equivalent to the baseline, performance is worst; the authors relate excessive masking to insufficient semantic alignment mining.
  • Qualitative Results: Qualitative results show IVT retrieves more positive samples and captures subtle attributes such as age, bags, grey shorts, and shoes than the baseline.Heat maps and word-level visualizations associate textual attributes with relevant body regions, including less salient parts.
  • Computational Efficiency: 42 seconds are required to test CUHK-PEDES with IVT, compared with 103,320 seconds for ViLT.The authors explain that ViLT encodes every image-text pair, whereas IVT extracts features once; IVT remains competitive in parameter count and retrieval efficiency.

5 Discussion

The discussion identifies incomplete and subtle textual cues as central challenges in text-based person retrieval. It concludes that unified networks and implicit alignments can address these challenges while achieving satisfying performance.

  • Text descriptions often cover only part of the visual features and overlook subtle features, especially in longer descriptions.
  • A unified network is effective for text-based person retrieval and may serve as a future backbone.
  • Fine-grained semantic alignments beyond salient part pairs should be mined for text-based person retrieval.

6 Conclusion

The paper addresses modality alignment through a unified visual-textual network and implicit semantic alignment paradigms. Experiments on three public datasets demonstrate the effectiveness of the IVT framework for text-based person retrieval.

  • IVT learns visual and textual representations with a single network that supports both retrieval speed and modality interaction.
  • BMM and MLA explore fine-grained alignment by mining more semantic alignments and using three-level matchings.
  • Experiments on three public datasets demonstrate the effectiveness of IVT for text-based person retrieval.

7 Broader Impact

Text-based person retrieval has surveillance applications, but pedestrian data collection and use raise privacy concerns. The paper calls for consent and regulated data utilization.

  • Text-based person retrieval may support surveillance applications such as finding suspects, lost children, or elderly people.
  • Surveillance data about pedestrians may cause privacy breaches, so collection should be consented to and utilization regulated.
Loading 2208.08608v2…