PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval

Tianlu Zheng, Yifan Zhang, Xiang An, Ziyong Feng, Kaicheng Yang, Qichuan Ding

arXiv:2509.09118v1cs.CV

TL;DR

Text-based person retrieval faces scarce and noisy person-centric image-text data and difficulty capturing localized semantics with global contrastive learning. The paper builds WebPerson and proposes GA-DMS, which combines gradient-attention-guided token masking with informative-token prediction. It reports state-of-the-art performance on several downstream datasets.

  • Problem

    Person retrieval with CLIP is challenged by scarce, noisy person-centric image-text data and global contrastive learning’s difficulty capturing fine-grained localized visual semantics.

    Source: §1 · p. 1

  • Method

    The paper constructs WebPerson from filtered and synthetically captioned web images and proposes GA-DMS, which masks noisy tokens using gradient-attention similarity and predicts informative tokens.

    Source: §1 · p. 2, Conclusion · p. 8

  • Results

    GA-DMS achieves state-of-the-art performance on several downstream datasets.

    Source: Conclusion · p. 8

  • Takeaways & Limitations

    The authors report text-based person retrieval performance using a person-centric dataset constructed solely from internet images.

    Source: Limitations · p. 8

    Limited by computational resources, the work constructs a 5M-scale WebPerson dataset and leaves further scaling for community exploration.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Although Contrastive Language-Image Pre-training (CLIP) exhibits strong performance across diverse vision tasks, its application to person representation learning faces two critical challenges: (i) the scarcity of large-scale annotated vision-language data focused on person-centric images, and (ii) the inherent limitations of global contrastive learning, which struggles to maintain discriminative local features crucial for fine-grained matching while remaining vulnerable to noisy text tokens. This work advances CLIP for person representation learning through synergistic improvements in data curation and model architecture. First, we develop a noise-resistant data construction pipeline that leverages the in-context learning capabilities of MLLMs to automatically filter and caption web-sourced images. This yields WebPerson, a large-scale dataset of 5M high-quality person-centric image-text pairs. Second, we introduce the GA-DMS (Gradient-Attention Guided Dual-Masking Synergetic) framework, which improves cross-modal alignment by adaptively masking noisy textual tokens based on the gradient-attention similarity score. Additionally, we incorporate masked token prediction objectives that compel the model to predict informative text tokens, enhancing fine-grained semantic representation learning. Extensive experiments show that GA-DMS achieves state-of-the-art performance across multiple benchmarks.

1 Introduction

The introduction identifies limitations in person-centric image-text data and CLIP’s global contrastive learning as challenges for text-based person retrieval. It presents WebPerson and GA-DMS as complementary data and modeling contributions, reporting state-of-the-art results across multiple datasets.

1 Introduction

  • CLIP performs suboptimally in text-based person retrieval, where limited, noisy person-centric data and global learning’s weak capture of localized attributes hinder fine-grained matching.Existing paired datasets are constrained by manual annotation, while LUPerson has millions of images but no corresponding text descriptions.Source: §1 · pp. 1–2
  • WebPerson provides 5 million high-quality image-text pairs through automatic filtering and annotation of web-sourced images.The pipeline is presented as addressing scarcity and noise in person-centric training data.Source: §1 · p. 2
  • GA-DMS masks noisy text tokens using gradient-attention similarity and predicts informative tokens to improve fine-grained visual-semantic correspondence.The framework combines these objectives to enhance cross-modal alignment.Source: §1 · p. 2
  • GA-DMS achieves state-of-the-art performance across multiple text-based person retrieval datasets.Source: §1 · p. 2

2 Related Works

Prior person-retrieval methods use separate encoders or CLIP-based extensions, while dataset work addresses annotation scalability and synthetic captions. The related work also notes that existing approaches can leave data noise or fine-grained semantics insufficiently addressed.

2.1 Person Representation Learning

  • Earlier retrieval methods used separate vision and language encoders with custom alignment losses, while CLIP established a unified vision-language embedding space.Later methods add cross-modal transformers, global feature learning, or explicit feature-space alignment; the discussion also describes approaches targeting noise or identity prototypes.Source: §2.1 · p. 2
  • Prototype aggregation is described as compromising fine-grained semantic learning, while data noise remains a concern for cross-modal alignment.The related work describes RDE as targeting noise and ProPOT as using identity-enriched prototypes.Source: §2.1 · p. 2

2.2 Person-centric Dataset

  • Manual annotation limits the scalability of paired person datasets, and large pedestrian datasets may lack text descriptions or rely on video sources.LUPerson-MLLM generates descriptions from existing LUPerson images, while the discussion notes its dependence on that image collection.Source: §2.2 · pp. 2–3

3 WebPerson Dataset

WebPerson is constructed by filtering web-crawled images for person and pose quality, then generating captions through a synthetic-caption pipeline. The described process yields five million person-centric images and uses curated templates to guide caption generation.

3.1 Person-Centric Image Filtering

  • YOLOv11 filters COYO700M images by size, aspect ratio, and human-detection confidence, then pose checks produce 5 million high-quality person-centric images.The retained images must show at least eight visible keypoints, including at least one hip and two head keypoints.Source: §3.1 · p. 3

3.2 Synthetic Caption Generation

  • Caption generation transforms existing dataset captions into structured templates, refines and expands them, then uses MLLMs to caption the selected images.The pipeline clusters templates, selects representative and sampled examples, reviews generated templates for bias and stereotypes, and produces a curated set of one thousand templates.Source: §3.2 · p. 3, Figure 2 · p. 3

4 GA-DMS Framework

GA-DMS combines a gradient-attention similarity score with dual masking to address noisy tokens and strengthen fine-grained semantic learning. Its framework uses the score to differentiate noisy and informative tokens during training, with a prediction task for informative tokens.

4.1 Gradient-Attention Similarity Score

  • The gradient-attention similarity score combines gradient and attention information to weight text tokens for masking, with multi-scale pooling enriching features with local context.The score quantifies each token’s contribution to image-text alignment.Source: §4.1 · p. 4

4.2.1 Noise Token Masking

  • Training dynamically masks text tokens according to probabilities based on their gradient-attention similarity scores, reducing the influence of noise tokens.The noise-token probability uses a maximum-probability hyperparameter, while slope and midpoint parameters control differentiation between noisy and semantically relevant tokens.Source: §4.2.1 · p. 5
  • Similarity Distribution Matching uses image-to-text and symmetric text-to-image losses to align the predicted and ground-truth matching distributions.The text-to-image loss is computed by swapping the image and text inputs.Source: §4.2.1 · p. 5

4.2.2 Masked Informative Token Prediction

  • The informative-token objective selectively masks tokens with strong image-semantic correlations and predicts them to improve fine-grained semantic representation.A cross-modal interaction module serves as the decoder, and an MLP predicts the original masked tokens from fused representations.Source: §4.2.2 · p. 5

5 Experiments

Across three retrieval benchmarks, GA-DMS reports state-of-the-art results, while WebPerson pretraining shows transfer gains and scaling benefits. Ablations compare masking components and similarity scores, and select settings for the two masking probabilities.

5.1 Comparison with Existing Methods.

  • Fine-tuning with IRRA improves RSTPReid Rank-1 accuracy and mAP by 10.10% and 7.72% over the IRRA baseline.Table 1 presents comparisons with state-of-the-art methods in traditional evaluation settings.Source: §5.1 · p. 6, Table 1 · p. 6
  • At 5.0M, GA-DMS reaches state-of-the-art Rank-1 accuracies of 77.6%, 69.51%, and 71.25% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.The comparison is reported for the three downstream datasets.Source: §5.1 · p. 6, Table 1 · p. 6
  • Against NAM, GA-DMS gains 0.2%, 2.02%, and 1.8% Rank-1 on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.The paper attributes the improvement to gradient-attention similarity distinguishing noisy from informative tokens.Source: §5.1 · p. 6, Table 1 · p. 6

5.2 Comparison with Existing Datasets.

  • In direct transfer, 1M-sample WebPerson performs better on CUHK-PEDES and RSTPReid but worse on ICFG-PEDES; 0.1M samples perform comparably to full-scale LUPerson-MLLM.The comparison covers pretraining datasets across three benchmarks.Source: Table 2 · p. 6, §5.2 · p. 7
  • At 1M data scale, WebPerson pretraining gains 0.43%, 1.89%, and 0.85% Rank-1 over LUPerson-MLLM on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.The paper reports state-of-the-art fine-tuning performance across in-domain and cross-domain scenarios.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Ablation Study

  • In a 0.5M WebPerson ablation, replacing noisy tokens with learnable embeddings achieves Rank-1 accuracy gains of 7.12%, 9.39%, and 6.8% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.Combining MTP with SDM enhances all metrics.Source: §5.3 · pp. 7–8
  • GASS consistently outperforms CSS; the selected maximum masking probabilities are αn = 0.2 for noise tokens and αi = 0.3 for informative tokens.Figure 5 reports the ablation used to choose these settings.Source: §5.3 · p. 8, Figure 5 · p. 8
  • At 5.0M samples, Rank-1 improves over the 0.1M baseline by 9.39%, 16.46%, and 10.50% across the three benchmarks as data volume increases.Figure 6 shows direct-transfer performance across data scales from 0.1M to 5.0M samples.Source: §5.3 · p. 8, Figure 6 · p. 8

Conclusion

The paper combines a web-image data pipeline with GA-DMS to advance CLIP for person representation learning. It reports state-of-the-art performance across several downstream datasets.

Conclusion

  • WebPerson contains 5M high-quality person-centric image-text pairs produced by a pipeline that filters and captions web-crawled images using MLLMs’ in-context learning.The conclusion describes the pipeline as noise-resistant.Source: Conclusion · p. 8
  • GA-DMS masks semantically irrelevant text tokens using gradient-attention similarity and predicts informative tokens to support fine-grained visual-semantic feature learning.The conclusion presents these as concurrent objectives.Source: Conclusion · p. 8
  • GA-DMS achieves state-of-the-art performance in several downstream datasets.Source: Conclusion · p. 8

Limitations

The authors limit WebPerson to a 5M-scale dataset because of computational resources, leaving further scaling for community exploration.

Limitations

  • Limited computational resources constrain WebPerson to 5M scale; further scaling is left for community exploration.Source: Limitations · p. 8

Ethics Statement

The authors state that they abide by the ACL Code of Ethics and that study data resources are publicly available.

Ethics Statement

A.1 Detail Experimental Settings

The paper directs readers to Table 5 for the GA-DMS pretraining settings and identifies the table as a hyperparameter listing.

A.1 Detail Experimental Settings

  • Table 5 presents the hyperparameters used for GA-DMS pretraining.The accompanying text identifies it as the location of the GA-DMS training settings.Source: Table 5 · p. 12, §A.1 · p. 12

A.2 Detail Instruction Prompt

The appendix details prompts for producing structured, visually grounded pedestrian captions and reports that using the text encoder’s last eight layers gives the strongest overall masking performance. It also describes WebPerson’s quality, diversity, scale, and intended human-oriented applications.

A.2 Detail Instruction Prompt

  • The Qwen2.5-72B-Instruct prompt for generating structured templates identifies pedestrian-attribute words in titles—including tops, pants, footwear, head features, accessories, age, gender, and actions—and replaces them with crossidentity generic terms.Source: §A.2 · p. 12
  • The Qwen2.5-VL-Instruct prompt requests concise captions focused on the dominant pedestrian elements and the colors of those elements.The specified elements include clothing, footwear, head features, accessories, and actions.Source: §A.2 · p. 12
  • Caption instructions restrict uncertain details, prohibit background or speculative descriptions, and require visually confirmed attributes.Generic clothing terms and vague color terms are reserved for cases where the specific type or color is unclear.Source: §A.2 · p. 12
  • Captions follow a structured template, prioritize significant pedestrian elements, stay below the specified word limit, and contain no added features or explanation.The prompt asks for common everyday phrasing with correct grammar and logic.Source: §A.2 · p. 12

A.4 Dataset analysis

  • WebPerson contains 5 million images and 10 million textual descriptions, making it the most extensive real-world dataset in the comparison.The comparison concerns person datasets; the table caption identifies WebPerson as the largest automatically generated text-described person dataset.Source: Table 6 · p. 13, §A.4 · p. 13

A.3 The Influence of Layers.

  • The last 8 text-encoder layers yield the highest overall performance, and gradient-based masking outperforms the baseline at every tested depth.The comparison varies the number of layers used to compute Gradient-Attention Similarity Score (GASS).Source: §A.3 · p. 12, Figure 7 · p. 13

A.4 Dataset analysis

  • WebPerson uses filtered web images and MLLM-generated captions to provide high-fidelity descriptions and variation in scenes, viewpoints, occlusions, clothing, and poses.Figure 8 presents representative examples of pedestrian-attribute descriptions.Source: §A.4 · p. 13, Figure 8 · p. 13

A.5 Broader Impact

  • The framework pairs gradient-based token-wise similarity scoring with noise suppression and WebPerson’s diverse web-sourced image-text pairs for human-oriented applications.The stated application examples are intelligent surveillance and autonomous retail systems.Source: §A.5 · p. 13
Loading 2509.09118v1…