Chat-Driven Text Generation and Interaction for Person Retrieval

1 Introduction

• TBPS retrieves target-person images from large galleries using natural-language descriptions, but producing rich, accurate text annotations is labor-intensive and unscalable.The introduction frames the central question as whether TBPS can work without manually crafted descriptions.Source: §1 · p. 1
• CTGI unifies annotation-free pseudo-caption generation and inference-time query refinement through multimodal dialogue.The framework pairs MTG for training supervision with MTI for inference-time refinement.Source: §1 · pp. 1–2
• MTG iteratively refines a baseline caption with attribute-targeted prompts to produce semantically dense, diverse, fine-grained pseudo-labels.To encode longer descriptions, the method retains the first 20 learned positions and interpolates the rest to support up to 248 tokens without retraining.Source: §1 · pp. 1–2
• Targeted multi-turn Q&A progressively enriches captions with fine-grained visual details, unlike single-pass LLM captioning that can omit details or hallucinate content.The comparison highlights clothing, accessories, and scene context as details that direct captioning may miss.Source: §1 · p. 1, Figure 1 · p. 2
• At inference, MTI uses an anchor image to ask about missing or ambiguous attributes, aggregates responses into a refined query, and filters for redundancy.The module is described as plug-and-play with pretrained vision-language retrieval models.Source: §1 · p. 2

2.1 Text-Based Person Search (TBPS)

• TBPS research progressed from shared-space global alignment to fine-grained matching and pretrained-model approaches, yet costly human annotations remain common.Prior methods also explored pose, part-level features, semantic knowledge, and multimodal interaction.Source: §2.1 · pp. 2–3
• Weakly supervised and synthetic labeling offer partial relief, but struggle with vague or conversational queries.Source: §2.1 · p. 3
• Unlike earlier interactive retrieval systems that often require task-specific data or retraining, CTGI supports open-ended queries and dynamically refines training captions and inference queries.The framework uses MLLMs as plug-and-play agents.Source: §2.1 · p. 3

Vision-Language Pre-training and Multimodal Large Language Models

• Building on vision-language pretraining and MLLM research, CTGI uses MLLMs to automate annotation generation and query clarification.The cited research directions include reasoning, cross-modal representations, and prompt-based learning.Source: Vision-Language Pre-training and… · p. 3

3.1 Multi-Turn Text Generation

• MTG starts with a general image caption, adds question-answer rounds for distinctive attributes, then reconstructs a concise caption that retains added detail while reducing repetition.Source: §3.1 · p. 4

3.2 More Text Positional Embeddings

• Preserving the first 20 positional embeddings and stretching the rest 4× extends CLIP to 248 tokens; Table 4 reports improved retrieval without sacrificing pretrained alignment.Source: §3.2 · p. 4

3.3 Multi-Turn Text Interaction (MTI)

• At inference, MTI selects an affirmative top-K anchor, asks diagnostic questions, merges responses with the query, and re-ranks; it skips refinement without an anchor and stops at ξ=0.85.Source: §3.3 · p. 5

3.4 Reconstructor

• The Reconstructor filters low-relevance or redundant training Q&A, turns structured logs into pseudo-captions for downstream supervision, and uses curated diagnostic templates at inference.CTGI is designed to work without human-written descriptions or task-specific engineering.Source: §3.4 · p. 5

4 Experiments

• The experiments compare retrieval models trained on generated pseudo-labels with original-annotation baselines, then examine generalizability, component ablations, and retrieval visualizations.The re-annotation and comparison strategy is applied across three public datasets.Source: §4 · p. 5

4.1 Datasets and Performance Measurements

• The evaluation covers CUHK-PEDES, ICFG-PEDES, and RSTPReid; training uses image data alone, while test-time retrieval uses dataset captions, measured by Rank-k and mAP.Higher Rank-k and mAP indicate better retrieval performance.Source: §4.1 · p. 5

4.2 Implementation Details

• IRRA and RDE use CLIP-ViT/B-16; Qwen2-VL-7B supports multimodal reasoning, while GPT-4o synthesizes pseudo-captions, with six MTG Q&A rounds and MTI examining 20 candidates.MTI can early-exit when the top-1 similarity exceeds 0.85 and the MLLM confirms it; final scores use weighted re-ranking.Source: §4.2 · pp. 5–6

4.3 Comparison with the State-of-the-Art

• Our+RDE reaches Rank-1/mAP of 67.82%/55.14% on CUHK-PEDES, 56.16%/32.40% on ICFG-PEDES, and 66.35%/51.51% on RSTPReid, leading unsupervised methods.The reported results show consistent gains over unsupervised methods across all three datasets.Source: Table 1 · p. 6, §4.3 · p. 7, Table 2 · p. 7

4.4 Ablation Study

• With IRRA, MTG alone reaches Rank-1 52.30% and MTI alone 55.50%; together, MTG and MTI reach 64.20% with IRRA and 66.35% with RDE.The authors report that the combined modules produce the highest gains across all metrics.Source: §4.4 · p. 7, Table 4 · p. 8

4.5 Visualization of Retrieval Results

• Figure 3 compares top-10 IRRA retrieval with IRRA+MTI, showing ground-truth images, refined queries, and correct-match borders; MTI refinement enables more accurate and robust ranking.The retrieval model is trained solely on MTG-generated pseudo-captions, and initial queries may be incompletely aligned with test captions.Source: §4.5 · pp. 7–8, Figure 3 · p. 8

• CTGI is an annotation-free TBPS framework that removes dependence on manually crafted descriptions by combining MTG training supervision with MTI inference-time refinement.Source: §5 · p. 8
• Across multiple TBPS benchmarks, CTGI achieves competitive or superior performance compared with fully supervised methods while adapting to existing retrieval pipelines.Source: §5 · p. 8
• Ablations and qualitative visualizations underscore the value of multi-turn interaction and MLLM-guided refinement for cross-modal alignment and retrieval robustness.Source: §5 · p. 9

• MTG pseudo-labels may contain semantic noise or redundancy, and models without noise-filtering may be more vulnerable to degraded performance.Source: Limitations · p. 9
• MTI’s multi-turn MLLM interactions add inference overhead that can limit deployment in latency-sensitive applications, even with early stopping and anchor validation.Source: Limitations · p. 9
• Both modules depend on the underlying MLLM’s generalization, which may yield suboptimal results in unfamiliar domains or on fine-grained attributes.Source: Limitations · p. 9