PaperlayerView exact revision on arXiv

Source-linked AI summary

RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search

Yang Bai, Min Cao, Daming Gao, Ziqiang Cao, Chen Chen, Zhenfeng Fan, Liqiang Nie, Min Zhang

arXiv:2305.13653v1cs.CV

TL;DR

Text-based person search retrieves person images from textual descriptions, with fine-grained cross-modal matching posing a challenge. RaSa combines relation-aware and sensitivity-aware representation learning. It reports Rank@1 gains of 6.94%, 4.45%, and 15.35% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.

  • Problem

    Text-based person search must retrieve a person’s image from a textual description despite fine-grained cross-modal matching challenges.

    Source: §1 · p. 1

  • Method

    RaSa combines relation-aware learning to distinguish strong and weak positive pairs with sensitivity-aware learning that detects MLM-based word replacements.

    Source: §1 · p. 2, Sensitivity-aware Learning · p. 4

  • Results

    6.94%, 4.45%, and 15.35% Rank@1 gains over existing state-of-the-art methods are reported on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    Experiments on multiple benchmarks demonstrate RaSa’s effectiveness for text-based person search.

    Source: §5 · p. 7

Abstract

from arXiv · show

Text-based person search aims to retrieve the specified person images given a textual description. The key to tackling such a challenging task is to learn powerful multi-modal representations. Towards this, we propose a Relation and Sensitivity aware representation learning method (RaSa), including two novel tasks: Relation-Aware learning (RA) and Sensitivity-Aware learning (SA). For one thing, existing methods cluster representations of all positive pairs without distinction and overlook the noise problem caused by the weak positive pairs where the text and the paired image have noise correspondences, thus leading to overfitting learning. RA offsets the overfitting risk by introducing a novel positive relation detection task (i.e., learning to distinguish strong and weak positive pairs). For another thing, learning invariant representation under data augmentation (i.e., being insensitive to some transformations) is a general practice for improving representation's robustness in existing methods. Beyond that, we encourage the representation to perceive the sensitive transformation by SA (i.e., learning to detect the replaced words), thus promoting the representation's robustness. Experiments demonstrate that RaSa outperforms existing state-of-the-art methods by 6.94%, 4.45% and 15.35% in terms of Rank@1 on CUHK-PEDES, ICFG-PEDES and RSTPReid datasets, respectively. Code is available at: https://github.com/Flame-Chasers/RaSa.

1 Introduction

Text-based person search must match open-form descriptions to person images despite fine-grained cross-modal challenges. RaSa addresses representation learning with relation-aware and sensitivity-aware tasks.

  • Weak positive pairs can mismatch image content despite sharing identity, but existing methods weight them like strong pairs, risking overfitting.Source: §1 · p. 2
  • RA combines probabilistic image-text matching, which retains weak-pair information, with positive-relation detection, which distinguishes strong from weak pairs.Source: §1 · p. 2
  • SA treats word replacement as a sensitive transformation and uses momentum-model-generated replacements for replaced-token detection.Source: §1 · p. 2
  • 6.94%, 4.45%, and 15.35% Rank@1 gains over existing state-of-the-art methods are reported on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.Source: §1 · p. 2

2 Related Work

Prior text-based person-search work explores cross-modal alignment and representation learning, while equivariant contrastive learning motivates sensitivity to transformations. RaSa applies these ideas through pair-type detection and MLM-based word replacement.

Text-based Person Search

  • Earlier text-based person-search methods emphasize cross-modal alignment or auxiliary representation learning; RaSa detects positive-pair type and sensitive transformations.Source: Text-based Person Search · pp. 2–3
  • RaSa’s architecture places intra- and cross-modal contrastive learning after unimodal encoders and RA and SA after cross-modal encoding.Source: Figure 2 · p. 3

Equivariant Contrastive Learning

  • Equivariant contrastive learning extends invariance-focused contrastive learning by encouraging representations to respond to sensitive transformations; RaSa uses MLM-based word replacement.Source: Equivariant Contrastive Learning · p. 3

3 Method

RaSa combines relation-aware and sensitivity-aware learning with contrastive learning to train multimodal representations for text-based person search. Its method pairs global image–text matching and positive-pair relation learning with token-level replacement detection.

3.1 Model Architecture

  • The model uses image and text unimodal encoders followed by a text-guided cross-modal encoder, with an EMA momentum model providing delayed, more stable representations.The cross-modal encoder uses text as the query and visual features as keys and values.Source: §3.1 · p. 3

Relation-aware Learning

  • RA combines probabilistic image–text matching with positive relation detection to balance weak positive pairs’ alignment value against their noise risk.p-ITM emphasizes consistency between strong and weak pairs, while PRD highlights their difference.Source: Relation-aware Learning · p. 4

Sensitivity-aware Learning

  • SA detects MLM-based word replacements using m-RTD, applying a fine-grained token-level objective complementary to RA’s global image–text correlation.The momentum model generates replacement words, and MLM also provides token-level optimization.Source: Sensitivity-aware Learning · p. 4

Contrastive Learning

  • Contrastive learning trains unimodal representations before cross-modal fusion, using cross-modal and intra-modal objectives and queues that provide additional negative samples.The intra-modal objective brings representations of the same person closer than those of different people within each modality.Source: Contrastive Learning · p. 5

Joint Learning

  • The joint objective combines RA, SA, and contrastive learning; at inference, unimodal similarity selects 128 images for cross-modal p-ITM ranking.This candidate filtering speeds inference before the cross-modal encoder’s quadratic interaction.Source: Joint Learning · p. 5

4 Experiments

RaSa is evaluated on three text-based person-search datasets using vision-language pretrained backbones and Rank@K and mAP metrics. It surpasses the prior best method on all three datasets, with ablations examining its objectives and hyperparameters.

4.1 Evaluation Protocol

  • Evaluation uses Rank@K (K=1, 5, 10) and complementary mAP, with RaSa tested on ALBEF, TCL, and CLIP and ALBEF used by default.A retrieval counts as successful when a corresponding identity appears among the top K images ranked by text-image similarity.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 Comparison with State-of-the-art Methods

  • 6.94%, 4.45%, and 15.35% R@1 gains over CFine on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively, with RaSa highest on all metrics.The comparisons are reported in Tables 1–3; the supplied table captions identify Tables 2 and 3 as the ICFG-PEDES and RSTPReid comparisons.Source: Table 2 · p. 6, Table 3 · p. 6, §4.3 · p. 6

Effectiveness of Optimization Objectives

Analysis of RA

  • s-ITM improves R@1 by 2.23% over vanilla ITM, while PRD consistently improves performance when added to vanilla ITM or p-ITM.p-ITM probabilistically retains weak positives, balancing their useful supervision against noise; PRD distinguishes strong from weak positives and regularizes p-ITM.Source: Analysis of RA · pp. 6–7

Analysis of SA

  • m-RTD reaches 76.51% R@1, ahead of o-RTD at 75.99% and f-RTD at 75.13%.The authors describe the momentum model as a slow-moving online model that bootstraps MLM to generate increasingly challenging tokens for detection.Source: Analysis of SA · p. 7

Hyper-parameters

  • RaSa performs best with pw = 0.1, pm = 0.3, and λ1 = λ2 = λ3 = 0.5.The authors explain that larger pw adds noise, smaller pw limits useful weak-positive information, and larger pm makes detection too difficult.Source: Hyper-parameters · p. 7, Figure 4 · p. 7

4.5 Extended Experiments and Visualization

  • Extended experiments cover coarse-grained Flickr30K and COCO retrieval, fine-grained CUB and Flowers retrieval, and domain generalization; the paper reports effectiveness and generalization ability.A qualitative retrieval visualization is also provided in the appendix.Source: §4.5 · p. 7

5 Conclusion

RaSa combines relation-aware and sensitivity-aware learning to improve multimodal representations for text-based person search. Experiments across multiple benchmarks demonstrate its effectiveness.

5 Conclusion

  • RaSa combines relation-aware learning (RA) and sensitivity-aware learning (SA) to learn multimodal representations for text-based person search.RA distinguishes strong and weak positive pairs to address noise from weak positives; SA encourages sensitivity to MLM-based token replacement.Source: §5 · p. 7
  • RA uses positive-pair distinction as a regularizer for p-ITM, which focuses on consistency between strong and weak positives.The approach targets overfitting associated with noise in weak positive pairs.Source: §5 · p. 7
  • Experiments on multiple benchmarks demonstrate RaSa’s effectiveness for text-based person search.Source: §5 · p. 7

A.1 Datasets

The appendix describes three text-based person-search datasets and identifies additional coarse- and fine-grained retrieval benchmarks used in extended experiments.

A.1 Datasets

  • CUHK-PEDES contains 40,206 images, 80,440 texts, and 13,003 identities, split into training, validation, and test sets.The splits contain 34,054/68,126, 3,078/6,158, and 3,074/6,156 images/texts, respectively.Source: §A.1 · p. 10
  • ICFG-PEDES has 54,522 images from 4,102 identities, with one text describing each image.Its training and test sets contain 34,674 images from 3,102 identities and 19,848 images from 1,000 identities, respectively.Source: §A.1 · p. 10
  • RSTPReid contains 20,505 images of 4,101 identities; each identity has five images, and each image has two textual descriptions.The train/validation/test splits contain 3,701/200/200 identities.Source: §A.1 · p. 10
  • Extended retrieval experiments include Flickr30K and COCO for coarse-grained retrieval and CUB and Flowers for fine-grained retrieval.Source: Table 5 · p. 11, Table 6 · p. 11

A.2 Implementation Details

Experiments use four NVIDIA 3090 GPUs and train RaSa for 30 epochs with batch size 52. The implementation specifies optimizer, input, augmentation, and objective settings.

A.2 Implementation Details

  • Training uses four NVIDIA 3090 GPUs, 30 epochs, batch size 52, and AdamW with weight decay 0.02.The classifier learning rate is 1e-4, versus 1e-5 for the model’s remaining parameters.Source: §A.2 · p. 11
  • Images are resized to 384 × 384 with random horizontal flips, and input texts are limited to 50 tokens.Source: §A.2 · p. 11
  • The implementation sets momentum coefficient m = 0.995, queue size R = 65,536, temperature τ = 0.07, pw = 0.1, pm = 0.3, and λ1 = λ2 = λ3 = 0.5.Source: §A.2 · p. 11

A.3 Extended Experiments

The extended experiments assess RaSa on coarse- and fine-grained retrieval and investigate domain generalization.

  • The extended experiments include coarse-grained and fine-grained retrieval evaluations.Source: §A.3 · p. 11
  • A separate set of experiments investigates RaSa's domain generalization ability.Source: §A.3 · p. 11

Coarse-grained Retrieval

For coarse-grained retrieval, the paper evaluates RaSa on Flickr30K and COCO, where positive pairs are all strong and RA is not used. RaSa with SA is reported to improve consistently over ALBEF across metrics.

  • Flickr30K and COCO provide the coarse-grained evaluation, with five sentences annotated per image and Karpathy splits used for both datasets.Source: Coarse-grained Retrieval · p. 11
  • Because each image-text pair is a unique class and every positive pair is strong, RA is not applied; the experiments use SA and vanilla ITM.Source: Coarse-grained Retrieval · p. 11
  • RaSa with SA consistently improves over ALBEF across all reported metrics on coarse-grained retrieval.Source: Coarse-grained Retrieval · p. 11

Fine-grained Retrieval

RaSa is evaluated on the fine-grained CUB and Flowers datasets. It is reported to outperform existing state-of-the-art methods on both datasets.

  • CUB has 11,788 bird images across 200 categories, while Flowers has 8,189 images across 102 categories; each image has 10 descriptions.Source: Fine-grained Retrieval · pp. 11–12
  • For inference, RaSa's same-identity multimodal embeddings are averaged because its one-stream output is multimodal rather than a text embedding.Source: Fine-grained Retrieval · p. 12
  • 14.0% and 12.0% improvements on CUB and 14.1% and 9.2% boosts on Flowers are reported over iVAD for R@1 and AP@50, respectively.Source: Fine-grained Retrieval · p. 12

Domain Generalization

RaSa is evaluated on domain generalization, alternative backbones, and retrieval examples. The reported results favor RaSa in the transfer evaluation and show consistent improvements with TCL and CLIP, while CLIP's performance is lower than TCL's.

Domain Generalization

A.4 Backbones and Experiments

  • RaSa brings consistent improvements across metrics with TCL and CLIP backbones; R@1 is 73.23% for RaSaTCL and 57.60% for RaSaCLIP.Source: §A.4 · pp. 12–13, Table 8 · p. 12
  • For CLIP, the authors conjecture that absent cross-modal deep fusion and a randomly initialized added transformer block may hurt performance.Source: §A.4 · p. 13

A.5 Visualization

  • In the top-10 CUHK-PEDES examples, RaSa retrievals are reported as more accurate than ALBEF's; Figure 5 marks correct and incorrect images in green and red.Source: §A.5 · p. 13, Figure 5 · p. 13

Related papers

Loading 2305.13653v1…