PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark

Shuyu Yang, Yinan Zhou, Yaxiong Wang, Yujiao Wu, Li Zhu, Zhedong Zheng

arXiv:2306.02898v4cs.CVcs.MM

TL;DR

Text-based person retrieval is limited by scarce data and high-quality annotations, restricting large-scale cross-modal pretraining. The paper introduces the synthetic MALS benchmark and joint APTM framework, whose pretraining improves retrieval across three real-world benchmarks.

  • Problem

    Text-based person retrieval lacks sufficient data and high-quality annotations for effective large-scale cross-modal pretraining.

  • Method

    The paper constructs MALS with synthetic image-text pairs and rich attributes, then jointly pretrains attribute prompt learning and text matching learning through APTM.

  • Results

    APTM achieves competitive retrieval performance on CUHK-PEDES, ICFG-PEDES, and RSTPReid, with consistent Recall@1 improvements reported across all three benchmarks.

  • Takeaways & Limitations

    MALS pretraining is scalable to real-world text-based person retrieval and pedestrian attribute recognition scenarios.

Abstract

from arXiv · show

In this paper, we introduce a large Multi-Attribute and Language Search dataset for text-based person retrieval, called MALS, and explore the feasibility of performing pre-training on both attribute recognition and image-text matching tasks in one stone. In particular, MALS contains 1,510,330 image-text pairs, which is about 37.5 times larger than prevailing CUHK-PEDES, and all images are annotated with 27 attributes. Considering the privacy concerns and annotation costs, we leverage the off-the-shelf diffusion models to generate the dataset. To verify the feasibility of learning from the generated data, we develop a new joint Attribute Prompt Learning and Text Matching Learning (APTM) framework, considering the shared knowledge between attribute and text. As the name implies, APTM contains an attribute prompt learning stream and a text matching learning stream. (1) The attribute prompt learning leverages the attribute prompts for image-attribute alignment, which enhances the text matching learning. (2) The text matching learning facilitates the representation learning on fine-grained details, and in turn, boosts the attribute prompt learning. Extensive experiments validate the effectiveness of the pre-training on MALS, achieving state-of-the-art retrieval performance via APTM on three challenging real-world benchmarks. In particular, APTM achieves a consistent improvement of +6.96%, +7.68%, and +16.95% Recall@1 accuracy on CUHK-PEDES, ICFG-PEDES, and RSTPReid datasets by a clear margin, respectively.

1 INTRODUCTION

The paper addresses data scarcity and annotation limitations in text-based person retrieval by introducing synthetic, richly annotated data and a joint attribute-text learning framework.

  • Text-based person retrieval is limited by insufficient data and costly, privacy-constrained high-quality language annotation.
  • MALS uses diffusion models and image captioning to generate scalable image-text data with automatically acquired annotations.
  • MALS focuses on fine-grained pedestrian details rather than only coarse-grained category recognition, supporting text retrieval and attribute recognition.
  • APTM jointly learns attribute prompts and text matching, using complementary attribute recognition and retrieval tasks to regularize representation learning.
  • 37.5× more images than CUHK-PEDES and competitive retrieval performance on three real-world benchmarks characterize the proposed benchmark and framework.

2 RELATED WORK

Related work contrasts efficient attention-free retrieval with more accurate but costlier attention-based alignment, motivating a hybrid strategy.

  • Cross-modal attention-free methods align image and text representations in a shared feature space without pair-wise inputs.
  • O(M+N) complexity makes attention-free methods efficient for M gallery images and N queries, whereas attention-based methods require O(MN).
  • Attention-based approaches typically achieve superior retrieval performance by enabling more early cross-modality communication, despite higher complexity.

3 BENCHMARK

MALS is a large synthetic benchmark designed to provide high-fidelity, diverse, fine-grained image-text pairs and attribute annotations while supporting transfer to real-world data.

  • MALS uses generative models to create aligned pedestrian image-text pairs at a scale of 1.5M, addressing annotation-cost and privacy constraints.
  • MALS provides 27 annotated attributes covering a structured attribute space with categories, annotation names, and available label choices.
  • The benchmark generates pedestrian images from real-world textual descriptions, then uses BLIP to produce more diverse captions for each synthetic image.
  • Its synthetic images offer high fidelity and varied backgrounds, viewpoints, occlusions, clothing, and body poses.
  • MALS descriptions share substantial vocabulary with CUHK-PEDES, supporting transfer from synthetic pre-training to real-world data.
  • The dataset combines diverse captions, synthetic images, large-scale pairs, and rich attributes for text-image matching and joint attribute learning.

4 METHOD

APTM jointly pre-trains image-attribute and image-text streams with shared encoders, using attribute prompts and text matching to align modalities and learn fine-grained representations. Its architecture combines contrastive, matching, and masked-language objectives across both streams.

  • Training Pipeline: The overall pre-training jointly optimizes attribute prompt learning and text matching learning before downstream fine-tuning.The two stages learn common knowledge for text-based person retrieval and pedestrian attribute recognition, then adapt parameters to a specific task.
  • APTM Architecture: APTM is a multi-task pre-training framework with image-attribute and image-text streams sharing encoders and using MLP-based headers.It includes image, text, and cross encoders; the cross encoder fuses image-text embeddings for predictions.
  • Attribute Prompt Learning: Text-derived attribute annotations are converted into Attribute Prompts that provide inputs for image-attribute alignment.Explicit Matching and Implicit Extension acquire attributes from text before prompt construction.
  • Attribute Prompt Learning: Attribute Prompt Learning uses image-attribute contrastive, matching, and masked attribute language modeling objectives.Contrastive learning distinguishes matched from unmatched prompts, matching predicts pair compatibility, and masked modeling predicts masked prompt words.
  • Text Matching Learning: Image-Text Contrastive Learning aligns paired images and texts against unmatched pairs in a shared feature space.The method extracts image and text representations, computes bidirectional matching scores, and optimizes the ITC loss.
  • Text Matching Learning: Image-Text Matching uses hard negative mining, while Masked Language Modeling predicts masked words from image-text pairs.Hard negatives are selected using cross-modal similarity before positive and negative pairs pass through the cross encoder.

5 EXPERIMENT

Experiments evaluate APTM across person-retrieval and attribute-recognition benchmarks, along with ablations of MALS pre-training scale and attribute-oriented objectives. MALS pre-training improves retrieval performance, with gains increasing as data scale grows and APL outperforming naïve alternatives.

  • Experimental Setup: APTM is evaluated on CUHK-PEDES, RSTPReid, ICFG-PEDES, and PA-100K using retrieval and attribute-recognition metrics.Retrieval uses Recall@1,5,10 and mAP; attribute recognition uses accuracy, precision, recall, and F1.
  • Effectiveness of Pre-Training: MALS pre-training improves performance for all three evaluated retrieval methods, despite using 1.5M pairs versus 4M general image-text pairs.The comparison is reported for downstream text-based person retrieval.
  • Qualitative Results: Qualitative retrieval results compare APTM and a baseline by similarity order, marking correct matches in green and incorrect matches in red.Green-highlighted text identifies details successfully matched by the retrieved results.
  • Effectiveness of Pre-Training: 10.09%, 5.12%, and 3.39% improvements in CUHK-PEDES Recall@1, Recall@5, and Recall@10 follow MALS pre-training over the APTM baseline.The baseline reaches 66.44%, 84.92%, and 90.76% on Recall@1, Recall@5, and Recall@10, respectively.
  • Impact of Pre-Training Scale: Performance on CUHK-PEDES, ICFG-PEDES, and RSTPReid improves consistently as MALS pre-training scale increases from 0M to 1.5M.The improvement is noticeable from 0 to 0.3M pairs and gradually diminishes from 0.3M to 1.5M.
  • Effectiveness of APL Loss: Removing APL loss hurts performance, while APL outperforms both naïve CMAL variants in CUHK-PEDES Recall@1 comparisons.The compared variants are pre-trained on 0.03M MALS data and then fine-tuned on CUHK-PEDES.

6 CONCLUSION

The paper introduces MALS for multi-attribute recognition and language-based person search, and jointly learns text-based retrieval with pedestrian attribute recognition. Experiments support scalability to real-world scenarios and report competitive retrieval performance on three public benchmarks.

  • Conclusion: MALS contains 1,510,330 image-text pairs with rich attribute annotations, approximately 37.5 times larger than CUHK-PEDES.The dataset is intended as a large-scale benchmark for multi-attribute recognition and language-based person search.
  • Conclusion: APTM jointly learns text-based person retrieval and pedestrian attribute recognition to regularize model training.The paper describes these as complementary tasks.
  • Conclusion: APTM achieves competitive recall rates on CUHK-PEDES, ICFG-PEDES, and RSTPReid after MALS pre-training.The conclusion characterizes MALS pre-training as scalable to real-world scenarios.
  • Broader Impact: The generated MALS dataset avoids access to specific people and is presented as addressing privacy concerns.The paper also states that MALS can facilitate person retrieval with limited data and provide a large-scale pre-training dataset.

A NETWORK DETAILS

APTM uses separate image, text, and cross encoders together with two MLP-based headers. The reported implementation uses Swin-B and split Bert layers, with 4.8 ms inference time per text query.

  • Network Details: APTM consists of an Image Encoder, Text Encoder, Cross Encoder, and two MLP-based headers.The paper reports parameters and GFLOPs for each component and for the complete model.
  • Network Details: The Image Encoder is Swin-B, while the Text Encoder and Cross Encoder use Bert’s first six and last six layers, respectively.
  • Network Details: 4.8 ms is the reported inference time per text query for APTM.

B ATTRIBUTE PROMPT TEMPLATE

Attribute Prompt Learning maps binary attributes to prompt templates and aligns those prompts with images. The templates specify that prompts describe image attributes and often outperform using label text alone.

  • Attribute Prompt Construction: 27 binary attributes are mapped to 54 Attribute Prompts for alignment with the corresponding image.
  • Attribute Prompt Construction: Five prompt-template types include formulations such as “the person is { Label Text }” and “the person with { Label Text }”.Different templates are customized for different attributes.
  • Attribute Encoding: Age is treated as a binary attribute, “young” and “adult”, because MALS lacks sufficient data for “young” and “old” in the quaternary scheme.
  • Prompt Design Rationale: Attribute prompt templates often improve performance over using only label text by specifying image attribute content and bridging distribution differences with paired text.

C FURTHER EXPERIMENTS AND DISCUSSION

Further experiments show that APTM remains effective under masked text, with β=0.8 giving the best reported CUHK-PEDES performance. The section also documents the model’s parameters, prompt templates, and retrieval behavior.

  • Parameter Sensitivity: 71.38% Recall@1 is achieved with β=0.8, the best performance among the tested APL-loss weights on CUHK-PEDES.The model was pre-trained on 0.03M MALS data for 32 epochs before identical fine-tuning.
  • Robustness Against Broken Sentences: APTM still identifies correct matches when words describing clothing, carried objects, or locations are masked.The examples include masked terms such as “street,” jacket color, hand-held items, “white,” and “black.”
  • Model Complexity: Table 7 reports parameter counts and GFLOPs for the image, text, and cross encoders, as well as the complete APTM.The table provides computational descriptors for each encoder and the overall framework.
  • Robustness Against Broken Sentences: When “street” is masked, APTM remains sensitive to “crossing” and infers the right image.The retrieval results are ordered from left to right by matching probability, with green boxes marking correct matches.
  • Attribute Prompts: A whole attribute prompt combines one of five prompt templates with label text for one of 27 attributes.The prompt construction varies the template and attribute-specific label text.
Loading 2306.02898v4…