PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of CLIP for Text-based Person Search

Min Cao, Yang Bai, Ziyin Zeng, Mang Ye, Min Zhang

arXiv:2308.10045v2cs.CVcs.CL

TL;DR

TBPS requires fine-grained matching between natural-language descriptions and person images, yet prior approaches often rely on unimodal backbones or added modules. This paper empirically studies CLIP’s augmentation and loss choices and develops a lightweight baseline; it reports promising results across multiple benchmarks and probes generalization and compression.

  • Problem

    TBPS needs fine-grained cross-modal alignment and efficient retrieval, while many methods use unimodal backbones and CLIP-based work has not fully exploited CLIP’s pretrained knowledge.

    Source: §1 · p. 1

  • Method

    The paper empirically studies CLIP for TBPS through data augmentation and loss-function choices, adding practical training tricks without sophisticated modules.

    Source: §1 · pp. 1–2

  • Results

    TBPS-CLIP achieves promising results on multiple TBPS benchmarks and is evaluated for generalization and model compression.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The empirical study is intended to provide a practical guideline for future CLIP-based TBPS research.

    Source: §5 · p. 7

    Using multiple image augmentations simultaneously may distort the original image and hurt performance.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Text-based Person Search (TBPS) aims to retrieve the person images using natural language descriptions. Recently, Contrastive Language Image Pretraining (CLIP), a universal large cross-modal vision-language pre-training model, has remarkably performed over various cross-modal downstream tasks due to its powerful cross-modal semantic learning capacity. TPBS, as a fine-grained cross-modal retrieval task, is also facing the rise of research on the CLIP-based TBPS. In order to explore the potential of the visual-language pre-training model for downstream TBPS tasks, this paper makes the first attempt to conduct a comprehensive empirical study of CLIP for TBPS and thus contribute a straightforward, incremental, yet strong TBPS-CLIP baseline to the TBPS community. We revisit critical design considerations under CLIP, including data augmentation and loss function. The model, with the aforementioned designs and practical training tricks, can attain satisfactory performance without any sophisticated modules. Also, we conduct the probing experiments of TBPS-CLIP in model generalization and model compression, demonstrating the effectiveness of TBPS-CLIP from various aspects. This work is expected to provide empirical insights and highlight future CLIP-based TBPS research.

1 Introduction

TBPS requires fine-grained cross-modal alignment and efficient retrieval, while much prior work used unimodal backbones. This study probes CLIP’s potential through augmentation and loss choices, then evaluates a lightweight baseline’s generalization and compression.

1 Introduction

  • TBPS retrieves person images from textual descriptions and requires effective cross-modal alignment and efficient retrieval.The task has potential applications including searching for suspects and locating lost children.Source: §1 · p. 1
  • Prior TBPS methods commonly use unimodal backbones, while existing CLIP-based approaches add sophisticated modules rather than fully exploiting CLIP’s pretrained knowledge.The paper probes CLIP itself through empirical studies of data augmentation and loss functions.Source: §1 · p. 1
  • The study evaluates data augmentation strategies and probes loss functions inspired by vision-language pretraining communities.Prior methods typically apply random horizontal flips to images without augmenting text; the examined losses are generic across cross-modal tasks.Source: §1 · p. 2
  • TBPS-CLIP combines common training tricks, augmentations, and losses with CLIP, attaining competitive performance without sophisticated modules.The resulting architecture is described as lightweight and low-cost.Source: §1 · p. 2
  • The paper examines TBPS-CLIP’s generalization as a baseline and in few-shot settings, and investigates module contributions to guide model compression.The introduction reports that its superiority in the few-shot setting is experimentally proved.Source: §1 · p. 2

2 Related Work

Prior TBPS work often combines unimodal backbones with task-specific losses or fine-grained alignment, while newer VLP-based methods use CLIP or other pretrained models. This paper instead studies CLIP’s own potential without sophisticated added modules.

2.1 Text-based Person Search

  • TBPS methods commonly use unimodal backbones, task-specific losses, and fine-grained image or text features for cross-modal alignment.Examples include horizontal image stripes and noun phrases parsed with NLTK Toolbox.Source: §2.1 · p. 2

2.2 Vision-Language Pre-training

  • VLP-based TBPS work includes CLIP adaptations and ALBEF-based losses, whereas this paper empirically mines CLIP’s potential without sophisticated added modules.VLP also supports broad cross-modal tasks; CLIP-related work explores data efficiency and few-shot learning through methods such as SLIP, FILIP, DeCLIP, CoOp, and CLIP-Adapter.Source: §2.1 · p. 2, §2.2 · pp. 2–3

3 Empirical Studies

The empirical studies examine CLIP training tricks, data augmentation, and loss design, then probe TBPS-CLIP’s generalization and module importance for compression. They compare augmentation and loss strategies for retrieval and explore few-shot settings and model analysis.

3 Empirical Studies

  • The empirical program is grounded on CLIP and examines practical training tricks, data augmentations, losses, generalization, and compression.The paper organizes the studies across Sections 3.1–3.5.Source: §3 · p. 3

3.1 Training Tricks

  • The training-trick study investigates global-gradient back-propagation, dropout, locking bottom layers, and soft labels.The paper identifies these as four common CLIP training tricks and details them in the Appendix.Source: §3.1 · p. 3

3.2 Data Augmentations

  • Image augmentations remove or alter visual information, while text augmentation has a smaller available pool because language is abstract and discrete.The image study considers crop, erasing, grayscale, blur, color, flipping, and rotation; text methods include back translation, synonym replacement, insertion, swap, deletion, and EDA.Source: §3.2 · p. 3, Text Augmentation · p. 3

3.3 Loss Functions

Improving Data Efficiency

  • Self-supervision maximizes similarity between two augmentations of a sample, while multi-view supervision applies normalized contrastive loss across differently augmented image and text views.Self-supervision can use image, text, or both; multi-view variants separately add image, text, or joint views.Source: Improving Data Efficiency · pp. 3–4

Optimization for Retrieval

  • Reversed ITC optimizes the reversed KL direction to separate negative pairs, while cyclic ITC adds in-modality and cross-modality regularization to address irregular representation geometry.The figure contrasts C-ITC with N-ITC and R-ITC; the text describes a risk that a query retrieves a negative image despite sharing identity with a paired text.Source: Optimization for Retrieval · p. 4, Figure 2 · p. 4

3.4 Model Generalization

  • The study probes TBPS-CLIP transfer to other TBPS methods and few-shot learning, and evaluates module contributions and weight importance for compression.The few-shot experiments also compare CoOp and CLIP-Adapter; compression uses performance drops after module removal and weight proximity to initialization while maintaining performance.Source: §3.4 · p. 4, §3.5 · p. 4

3.5 Model Compression

  • Model compression analysis measures each module’s contribution by removal-related performance drops and its importance by weight movement from initialization under a performance constraint.The paper adopts two evaluation metrics for assessing module roles in real-world compression.Source: §3.5 · p. 4

4 Experiments

The experiments ablate training tricks, data augmentations, and loss functions, then compare TBPS-CLIP with other methods and probe its generalization and compression.

4.1 Ablations of Training Tricks

  • Table 1 reports ablations of training tricks on CUHK-PEDES and defines CLIP* as CLIP with all four training tricks.Source: Table 1 · p. 5

4.2 Ablations of Data Augmentation

  • Image crops, erasing, and most semantics-preserving alterations improve performance, while Gaussian blur, hue jitter, and vertical flips hurt it.Stacking effective image augmentations also improves performance.Source: §4.2 · p. 5, Table 2 · p. 5
  • Back translation and random deletion help text augmentation, and stacking them improves Rank-1 by 1.38%; synonym replacement, insertion, and swapping hurt performance.The authors associate the harmful operations with risks of changing meaning or disrupting sentence structure.Source: §4.2 · p. 5, Table 3 · p. 5
  • Combining the selected image and text augmentations increases Rank-1 accuracy by 2.44% over CLIP*+Aug, with the gain attributed to augmentation alone.The combination uses the image augmentation pool and stacked text augmentations.Source: §4.2 · p. 6, Table 4 · p. 6

4.3 Ablations of Loss Functions

  • Stacking the effective losses reaches 69.54% Rank-1 accuracy, a 2.76% increase over CLIP*+Aug; image-side data-efficiency losses outperform text-side ones.R-ITC alone improves Rank-1 by 1.41% over CLIP+Aug.Source: §4.3 · p. 6, Table 5 · p. 6

4.4 Comparisons with State-of-the-Art Methods

  • TBPS-CLIP with ViT-B/16 outperforms IRRA on ICFG-PEDES and RSTPReid and performs on par with it on CUHK-PEDES; the simplified model also beats IRRA on the first two datasets.The comparisons cover CUHK-PEDES, ICFG-PEDES, and RSTPReid; TBPS-CLIP retains CLIP’s two-stream architecture.Source: Table 6 · p. 6, Table 7 · p. 6, §4.4 · p. 7

4.5 More Probing Experiments of TBPS-CLIP

  • In the few-shot setting, TBPS-CLIP shows promising results, while its simplified version has an advantage; CLIP, CoOp, and CLIP-Adapter perform poorly.The authors attribute the difficulty to TBPS being fine-grained and person-specific, unlike CLIP’s pretraining data.Source: §4.5 · p. 7, Table 11 · p. 7
  • Freezing some text-encoder layers has little performance impact, whereas dropping layers can hurt performance, supporting compression by freezing during training.TBPS-CLIP is also applied as a baseline to other TBPS methods to test generalization.Source: §4.5 · p. 7, Figure 3 · p. 7

5 Conclusion

The paper presents TBPS-CLIP as a straightforward CLIP baseline that uses common augmentations, loss functions, and training tricks rather than sophisticated modules. It reports promising results across multiple benchmarks and examines generalization and compression.

5 Conclusion

  • Common augmentations, loss functions, and practical training tricks let CLIP achieve promising results on multiple TBPS benchmarks without sophisticated modules.Source: §5 · p. 7
  • The study probes TBPS-CLIP’s effectiveness from two further viewpoints: model generalization and model compression.Source: §5 · p. 7
  • The empirical study is intended to provide practical guidance for future CLIP-based TBPS research.Source: §5 · p. 7

A.1 Training Tricks

The training tricks include backpropagating gradients across GPUs, applying dropout in the text encoder, locking an image-encoder layer, and soft-labeling the loss.

A.1 Training Tricks

  • Backpropagating loss gradients gathered across GPUs yields a notable performance gain over using only local-GPU gradients.Source: §A.1 · p. 9
  • Dropout is applied to the text encoder’s self-attention modules with probability 0.05.Source: §A.1 · p. 9
  • Locking the image encoder’s first convolutional layer, which maps image patches to tokens, gives slightly better performance.Source: §A.1 · p. 9
  • Soft labels average the model’s image-text matching probability with the ground-truth label to stabilize training.Source: §A.1 · p. 9

A.2 Data Augmentation

The appendix describes image and text augmentation, two few-shot CLIP alternatives, and metrics for assessing module contributions and compression potential.

A.2 Data Augmentation

  • Image augmentations are grouped into removal and alteration strategies, while text augmentation examples include back translation, synonym replacement, insertion, swapping, and deletion.Source: §A.2 · pp. 9–10

A.3 Model Generalization

  • CoOp prepends learnable soft text tokens to the input while freezing CLIP, whereas CLIP-Adapter trains a residual two-layer image MLP with the backbone locked.CoOp’s prompt is initialized as “a pedestrian photo of”; CLIP-Adapter’s residual factor is set to 0.4.Source: §A.3 · p. 10

A.4 Model Compression

  • The first compression metric estimates a module’s contribution from the performance drop after removing it.Source: §A.4 · p. 10
  • The second metric assesses how far module weights can move toward their initial values while maintaining a specified performance level.It treats a smaller C2_i as indicating a lower module contribution.Source: §A.4 · p. 10

B Experiments

Experiments compare methods across three TBPS datasets, use Rank-k and mAP metrics, and describe the datasets and implementation setup.

B Experiments

  • CUHK-PEDES has an average of three images per person and two descriptions per image; ICFG-PEDES has more complex backgrounds and longer descriptions, while RSTPReid uses five camera-view images per identity.Source: B Experiments · pp. 10–11
  • The image-augmentation hyperparameter ablation is reported on CUHK-PEDES, with the underlined value identifying the selected setting.Source: Table 13 · p. 11
  • Performance is evaluated with Rank-k at k = 1, 5, and 10, and mean Average Precision (mAP).Source: B Experiments · p. 11
  • Experiments use four Nvidia A40 GPUs, ViT-B/32 or ViT-B/16, AdamW, a warmup-and-cosine learning-rate schedule, and five training epochs.Source: B Experiments · p. 11

B.1 Ablations of Data Augmentation

This section introduces the hyperparameters used for data augmentation.

B.1 Ablations of Data Augmentation

  • The section describes the hyperparameters used for data augmentation.The supplied passage does not specify their settings.Source: §B.1 · p. 11

Image Augmentation

Image augmentation varies crop, erasing, blur, color, and rotation settings; the study evaluates the resulting hyperparameter choices.

Image Augmentation

  • Random-resized crops vary aspect ratio and area, then resize augmented images to 224 × 224.The cropped-area scale is sampled from [xRRC, 1].Source: Image Augmentation · p. 11
  • Random erasing varies the erased-area proportion, while Gaussian blur uses kernel size k and a randomized standard deviation.The supplied descriptions do not provide all parameter bounds.Source: Image Augmentation · p. 11
  • ColorJitter varies brightness, contrast, saturation, and hue, while rotation samples degrees from [−degree, +degree].Image-augmentation hyperparameter results are reported in Table 13.Source: Image Augmentation · p. 11

B.2 More Probing Experiments of TBPS-CLIP

TBPS-CLIP transfers to other TBPS methods and performs strongly in few-shot settings, while module-contribution analyses identify distinct compression opportunities in its text and image encoders.

B.2 More Probing Experiments of TBPS-CLIP

  • TBPS-CLIP verifies generalization as an IRRA baseline on ICFG-PEDES and RST-PReid, and shows an absolute advantage across few-shot settings.The simplified version is more competitive in performance.Source: §B.2 · p. 12
  • The two contribution metrics agree: text-encoder top and bottom layers matter most, middle layers appear redundant, and self-attention outweighs feed-forward modules.The analysis assesses performance with Rank-1 and sets ε to 3 for metric C2.Source: §B.2 · p. 12, Figure 5 · p. 12
  • Image-encoder contributions are relatively even across layers, although the first two self-attention modules contribute less; the image encoder remains essential for strong TBPS performance.These patterns suggest text-layer removal may have little effect, unlike the essential image encoder.Source: §B.2 · p. 12, Figure 5 · p. 12
  • For text-encoder compression, the study drops or freezes 1–5 layers, selecting combinations with the minimum summed Whole scores from the two metrics.Table 17 lists the selected layer IDs and corresponding trainable-parameter counts.Source: §B.2 · p. 12, Table 17 · p. 12

C Further Discussions

The discussion highlights a gain from CLIP training tricks and frames augmentation, loss, compression, and few-shot findings as guidance for future TBPS work.

C Further Discussions

  • Four training tricks with CLIP yield about 4% improvement at Rank-1.The authors suggest these tricks may inspire future performance improvements.Source: C Further Discussions · p. 12
  • Studies of more than 20 augmentations and about 10 loss functions provide guidance for selecting effective options in future models.The paper also explores model properties as guidance for compression and reports a preliminary study of CLIP-based few-shot TBPS.Source: C Further Discussions · p. 13
  • The paper presents few-shot TBPS as a sparsely researched area and its own study as preliminary.The authors frame its observations as guidance for future research direction.Source: C Further Discussions · p. 13

Related papers

Loading 2308.10045v2…