PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Retrieval for Unsupervised Text-Based Person Search

Mang Ye, Yucheng Ji, Yang Bai, Min Cao, Siyuan Chai, Bo Du, Min Zhang

arXiv:2609.12965v1cs.CVcs.AI

TL;DR

The paper addresses unsupervised TBPS, where manually annotated image-text pairs are unavailable. GTR+ generates fine-grained, diverse pseudo-descriptions and performs confidence-weighted retrieval learning, while LargeFine-Person provides a large-scale benchmark; experiments report effectiveness and generalization across multiple TBPS benchmarks.

  • Problem

    Unsupervised TBPS seeks retrieval from person images without the manually annotated textual descriptions on which mainstream methods depend.

  • Method

    GTR+ generates fine-grained, diverse pseudo-descriptions through three tiers and trains retrieval with adaptive confidence weighting, alongside the LargeFine-Person dataset.

  • Results

    Experiments on multiple TBPS benchmarks demonstrate the effectiveness and generalizability of GTR+ and LargeFine-Person in unsupervised settings.

  • Takeaways & Limitations

    GTR+ and LargeFine-Person provide an unsupervised TBPS solution and pre-training benchmark based on automatically generated, fine-grained, diverse textual data.

  • Takeaways & Limitations

    The retrieval framework is demonstrated with BLIP as a baseline model, although the authors report it can be replaced by other retrieval models.

Abstract

from arXiv · show

Text-based person search (TBPS) aims to retrieve images of a target person from a large image gallery based on a given natural language description. Most existing methods rely on supervised learning with manually annotated image-text pairs. In this paper, we explore unsupervised TBPS, with only unlabeled images. We propose GTR+, a two-stage generation-then-retrieval framework. In the generation stage, we introduce a tiered description generation framework designed to produce fine-grained and stylistically diverse textual descriptions through a three-tier sequential process. The base tier leverages an automated question-and-answer mechanism to generate basic visual attribute descriptions; the intermediate tier enhances fine-grained detail using an inter-sample contrastive mechanism; the advanced tier further enriches textual diversity via a stylized expansion mechanism. In the retrieval stage, to mitigate the impact of noisy pseudo texts, we develop an adaptive confidence-weighted retrieval learning framework. We model image-text pairs as clean or noisy using a Gaussian Mixture Model, calibrated by real-time image-text similarity and static text generation probability from the prior stage, yielding adaptive sample weights during training. Beyond that, we also contribute LargeFine-Person, a large-scale TBPS dataset with high-quality, fine-grained, and diverse textual annotations, enabling a practical and generalizable TBPS pre-training benchmark under unsupervised setting. Experiments on multiple TBPS benchmarks demonstrate the effectiveness and generalization of both GTR+ and LargeFine-Person. Code is available at: https://github.com/Flame-Chasers/GTR.

1 INTRODUCTION

The paper targets unsupervised text-based person search using only image corpora, replacing costly manual annotations with GTR+, a generation-then-retrieval framework and the LargeFine-Person dataset.

  • 1 INTRODUCTION: GTR+ addresses unsupervised TBPS by generating pseudo descriptions from person images, then training retrieval with those image-text pairs.This setting uses only image corpora rather than manually annotated textual descriptions.
  • 1 INTRODUCTION: The tiered generator combines automated question-and-answering, inter-sample contrastive generation, and stylized expansion to produce semantically rich, fine-grained, and diverse descriptions.The tiers progressively capture basic attributes, fine-grained details, and stylistic variation.
  • 1 INTRODUCTION: Adaptive confidence-weighted retrieval learning reduces the influence of noisy pseudo-text pairs by incorporating confidence scores into retrieval training.The framework addresses hallucinations and inaccuracies introduced during description generation.
  • 1 INTRODUCTION: LargeFine-Person contains 3.6 million image-text pairs generated from person images and is intended as a practical, generalizable unsupervised TBPS pre-training benchmark.Its descriptions are characterized as fine-grained and diverse, with fully automated construction described in the supplied passages.
  • 1 INTRODUCTION: Experiments on multiple TBPS benchmarks demonstrate the effectiveness and generalizability of GTR+ and LargeFine-Person in unsupervised settings.The supplied introduction states the overall experimental conclusion without reporting benchmark values.

2 RELATED WORK

Related work spans cross-modal alignment, reduced-supervision TBPS, unsupervised vision-language learning, image captioning, and noisy-correspondence mitigation; the paper combines these concerns for unsupervised TBPS.

  • 2 RELATED WORK: TBPS research includes cross-modal alignment methods that map visual and textual features into shared spaces using increasingly fine-grained correspondences and external tools.The related work describes progression from global alignment to multi-granularity and self-adaptive semantic alignment.
  • 2 RELATED WORK: Weakly supervised, semi-supervised, and cross-domain TBPS reduce annotation requirements but still rely on image-text pairs or partial supervision.These settings differ from the paper’s unsupervised setting, which provides only person images.
  • 2 RELATED WORK: Unsupervised vision-language research covers vision-language pretraining, text-to-image synthesis, and unsupervised image captioning, including unpaired image-caption settings.The supplied passage presents these as neighboring research directions for reducing paired-data requirements.
  • 2 RELATED WORK: Image captioning has evolved from templates and detector-based attributes to CNN-RNN systems, Transformers, and multimodal large language models.TBPS person descriptions remain distinct because they emphasize fine-grained, individual-specific features rather than general image semantics.
  • 2 RELATED WORK: Noisy-correspondence research develops methods to identify, purify, or rectify mismatched image-text samples for cross-modal learning.This line of work motivates handling noise in synthetic image-text pairs used for retrieval training.

3 METHOD

The method solves unsupervised TBPS with a two-stage generation-then-retrieval pipeline that creates synthetic descriptions and uses them for noise-robust retrieval training.

  • 3 METHOD: Given only person images S_I, the method generates synthetic text data S_T^syn and then trains a retrieval model using the original images and synthetic texts.The two stages are tiered description generation followed by adaptive confidence-weighted retrieval learning.
  • 3 METHOD: The retrieval stage is designed to enable noise-robust training from synthetic image-text pairs produced by the generation stage.This formulation addresses the reliability problems associated with generated textual supervision.

3.1 Tiered Description Generation

The tiered description generation framework produces fine-grained, diverse person descriptions through sequential question-and-answer, contrastive, and stylized expansion tiers.

  • 3.1 Tiered Description Generation: The framework generates descriptions in three tiers: Q&A extracts basic attributes, inter-sample contrast adds fine-grained details, and stylized expansion diversifies sentence structures.The tiers are designed to progressively improve attribute coverage, informativeness, and linguistic diversity.
  • Base Tier: Question-and-Answer Mechanism: The base tier uses structured questions and answers to query global and fine-grained visible attributes before summarizing them into a description.The prompts cover clothing, shoes, hairstyle, age, accessories, and other visible details while avoiding assumptions about unclear attributes.
  • Intermediate Tier: Inter-Sample Contrastive Mechanism: The intermediate tier selects a visually similar hard negative from the top-10 nearest neighbors and prompts the model to contrast shared and distinctive features.Random selection among the top-10 neighbors increases negative-sample diversity, while the comparison highlights differences such as clothing or shoe colors.
  • Advanced Tier: Stylized Expansion Mechanism: The advanced tier analyzes existing descriptions to build a diverse template gallery, then generates detailed text from selected templates using the earlier tier outputs.The prompt explicitly instructs the model not to invent details absent from the image, and Qwen2-VL-7B-Instruct and InternVL2.5-8B provide additional generation diversity.

3.2 Adaptive Confidence-Weighted Retrieval Learning

The retrieval stage addresses noisy generated texts by weighting image-text pairs according to adaptive confidence estimates. These estimates combine image-text similarity and text-generation confidence, with a GMM separating clean from noisy data and updating weights during training.

  • Adaptive Confidence Weighting: Adaptive confidence weighting reduces the influence of noisy generated texts by assigning higher weights to clean image-text pairs and lower weights to noisy ones.The confidence score is incorporated into the retrieval objectives rather than treating every pair uniformly.
  • Baseline Retrieval Model: The baseline retrieval model uses BLIP with image, text, and image-grounded text encoders optimized through image-text contrastive and matching losses.BLIP can be replaced by other retrieval models because the proposed weighting framework is flexible.
  • Confidence Estimation: A two-component Gaussian Mixture Model represents clean and noisy pair distributions, with the higher-mean component treated as clean.The model estimates each pair’s posterior probability of belonging to the clean distribution as its confidence score.
  • Confidence Estimation: The adjusted similarity combines cosine image-text similarity with expected token-generation probability, using text confidence to stabilize early training when learned similarities are unreliable.The generation probability is the expectation of constituent token probabilities, while the combination is controlled by γ.
  • Adaptive Training: Confidence scores and GMM parameters are updated alternately during training, allowing noise estimates to respond to changing similarity distributions.This dynamic procedure is contrasted with using a fixed text-generation probability throughout training.

3.3 Pre-training Dataset Construction: LargeFine-Person

LargeFine-Person is an automatically constructed pre-training dataset for unsupervised TBPS, built from denoised person images and tiered generated descriptions. It contains 1.2 million images paired with 3.6 million texts without manual annotation.

  • Dataset Role: The dataset combines fine-grained, diverse descriptions with large scale to support unsupervised TBPS pre-training.Its construction uses person images collected and refined from LUPerson and LPW before generating corresponding descriptions.
  • Image Collection: 1.2 million images remain after denoising LUPerson and combining it with LPW, yielding the image base for LargeFine-Person.Denoising removes low-resolution, blurred, and abnormal multi-person or undetected instances; 0.61M LUPerson images are retained and combined with 0.59M LPW images.
  • Dataset Role: LargeFine-Person is constructed entirely without manual annotation through the tiered description generation framework.The dataset is intended as a practical and generalizable pre-training benchmark for unsupervised TBPS.

4 EXPERIMENT

The experiments evaluate GTR+ under unsupervised training on three established TBPS benchmarks and compare it with methods spanning supervised, weakly supervised, semi-supervised, and unsupervised settings. LargeFine-Person is included as the proposed pre-training dataset, while benchmark training excludes textual annotations.

  • Datasets and Evaluation: The evaluation uses CUHK-PEDES, ICFG-PEDES, and RSTPReid as open TBPS benchmarks, alongside the proposed LargeFine-Person pre-training dataset.CUHK-PEDES is described as the most commonly used TBPS dataset; the three benchmarks provide distinct dataset sizes and annotation structures.
  • Datasets and Evaluation: Table 2 compares TBPS methods across supervised, weakly supervised, semi-supervised, and unsupervised learning settings.GTR+(Pre-training) denotes pre-training GTR+ on LargeFine-Person.
  • Evaluation Protocol: Under the default unsupervised setting, only training images are used, while their corresponding textual annotations are excluded from training.This evaluation protocol tests training without textual annotations from the benchmark training sets.

4.2 Protocol

The protocol evaluates GTR+ with Rank@K and mAP across multiple TBPS benchmarks and compares unsupervised, supervised, weakly supervised, and semi-supervised settings. LargeFine-Person is also assessed as an unsupervised pre-training dataset with and without downstream fine-tuning.

  • 4.2 Protocol: Rank@K measures whether a correct identity appears among the top K retrieved images, while mAP summarizes comprehensive retrieval performance.The protocol uses R@1, R@5, and R@10 together with mAP.
  • 4.2 Protocol: GTR+ is compared on CUHK-PEDES, ICFG-PEDES, and RSTPReID against methods spanning four supervision paradigms.Unsupervised methods provide the fairest direct comparison because competing supervised paradigms use additional annotation signals.
  • 4.2 Protocol: With LargeFine-Person pre-training, GTR+ achieves stronger results while remaining unsupervised and can outperform a semi-supervised method.GTR+ still trails methods using more annotated supervision, whose richer labels provide an advantage.
  • 4.2 Protocol: LargeFine-Person is compared under unsupervised and supervised pre-training settings, with direct evaluation and downstream fine-tuning variants.Tables 3 and 4 compare pre-training datasets using retrieval baselines under both evaluation regimes.
  • 4.2 Protocol: LargeFine-Person consistently outperforms competing unsupervised pre-training datasets across baselines both without and with downstream fine-tuning.Under supervised fine-tuning, it remains competitive with unsupervised datasets but is slightly inferior to human-annotated HAM-PEDES, particularly on HAM-PEDES.

4.5 Ablation Study

Ablations support the tiered generation design and adaptive confidence weighting, showing benefits from hard negatives, image masking, two-component GMM modeling, and combined confidence signals.

  • 4.5.1 Effectiveness of the tiered description generation framework: Q&A improves over template-based generation, while hard-negative ISC outperforms random negative sampling for producing richer fine-grained descriptions.The framework combines Q&A, ISC, and stylized expansion as three sequential generation components.
  • 4.5.2 Effectiveness of the adaptive confidence-weighted retrieval learning framework: Image-patch masking slightly outperforms text-masking variants because images retain more semantic coherence after partial masking.The comparison includes similarity-based text masking, generation-probability text masking, and random image masking.
  • 4.5.2 Effectiveness of the adaptive confidence-weighted retrieval learning framework: AdaWeight achieves the best noise-robust retrieval performance, outperforming masking-based alternatives and fixed confidence weighting.The framework assigns adaptive sample confidence to mitigate noise from generated image-text pairs.
  • 4.5.3 Hyper-parameter analysis: K = 2 Gaussian components yields the best confidence estimation, whereas K = 1 is too coarse and K > 2 over-partitions the noise distribution.The GMM separates clean and noisy pair components during training.
  • 4.5.3 Hyper-parameter analysis: Combining image-text cosine similarity with text generation probability outperforms relying exclusively on either confidence signal.γ controls their relative contribution, while β controls confidence influence in the training objectives.

4.6 Generalization Experiment

The adaptive confidence-weighted framework is tested across domains and retrieval backbones, where it remains competitive despite unsupervised generated-text training.

  • 4.6 Generalization Experiment: GTR+ is evaluated for cross-domain transfer using source-to-target settings such as C→I, alongside supervised domain-generalization methods.GTR+ trains on generated texts without textual annotations, whereas the compared methods use human-annotated descriptions.
  • 4.6 Generalization Experiment: The framework’s cross-domain evaluation and multi-backbone results support generalization beyond a single training dataset or retrieval model.The supplied results establish transfer evaluation and baseline adaptability, without specifying numerical gains.
  • 4.6 Generalization Experiment: The adaptive confidence-weighted framework consistently achieves competitive performance across different retrieval models on CUHK-PEDES.This indicates compatibility with varied retrieval baselines rather than dependence on one architecture.

4.7 Visualization Analysis

Visual analyses compare retrieval examples and activation maps, showing that GTR+ focuses more on described persons and key textual attributes than the compared methods.

  • 4.7 Visualization Analysis: GTR+ produces top-5 retrieval examples that are presented as more effective and superior to the BLIP baseline.The comparison is illustrated in Figure 9.
  • 4.7 Visualization Analysis: Activation maps show GTR+ placing greater emphasis on the described object and key text attributes than BLIP and GTR.Figure 10 visualizes positive image-text pairs under the unsupervised setting.

5 CONCLUSION

GTR+ addresses unsupervised TBPS by generating pseudo-descriptions before retrieval training, while LargeFine-Person provides a challenging benchmark with fine-grained and diverse annotations.

  • 5 CONCLUSION: GTR+ generates pseudo-descriptions and then trains a retrieval model under the unsupervised TBPS setting.The framework targets retrieval without relying on manually annotated textual descriptions.
  • 5 CONCLUSION: The tiered generation framework produces fine-grained and stylistically diverse captions through a three-tiered sequential process.
  • 5 CONCLUSION: Adaptive confidence-weighted learning dynamically adjusts each sample’s training contribution to reduce the negative impact of noisy pseudo texts.
  • 5 CONCLUSION: LargeFine-Person contributes high-quality, fine-grained, and diverse textual annotations as a realistic and challenging TBPS benchmark.
  • 5 CONCLUSION: The reported experiments demonstrate GTR+ performance on multiple unsupervised TBPS benchmarks and validate the effectiveness of LargeFine-Person.
Loading 2609.12965v1…