Source-linked AI summary
CONQUER: Context-Aware Representation with Query Enhancement for Text-Based Person Search
Zequn Xie
TL;DR
TBPS is challenged by cross-modal discrepancies and ambiguous or incomplete queries. CONQUER combines training-time representation enhancement with inference-time query refinement, and experiments report consistent gains over strong baselines across in-domain and cross-domain settings.
Problem
TBPS must retrieve pedestrian images from large galleries despite cross-modal discrepancies and ambiguous user queries.
Method
CONQUER uses CARE for multi-granularity, context-aware cross-modal alignment during training and plug-and-play IQE for anchor-based query enrichment during inference without retraining.
Results
CONQUER consistently surpasses strong baselines across three TBPS benchmarks in in-domain and cross-domain scenarios, with complementary effects from CARE and IQE.
Takeaways & Limitations
CONQUER provides a practical two-stage approach for improving TBPS retrieval under cross-domain and incomplete-query conditions.
Abstract
from arXiv · showhide
Text-Based Person Search (TBPS) aims to retrieve pedestrian images from large galleries using natural language descriptions. This task, essential for public safety applications, is hindered by cross-modal discrepancies and ambiguous user queries. We introduce CONQUER, a two-stage framework designed to address these challenges by enhancing cross-modal alignment during training and adaptively refining queries at inference. During training, CONQUER employs multi-granularity encoding, complementary pair mining, and context-guided optimal matching based on Optimal Transport to learn robust embeddings. At inference, a plug-and-play query enhancement module refines vague or incomplete queries via anchor selection and attribute-driven enrichment, without requiring retraining of the backbone. Extensive experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReid demonstrate that CONQUER consistently outperforms strong baselines in both Rank-1 accuracy and mAP, yielding notable improvements in cross-domain and incomplete-query scenarios. These results highlight CONQUER as a practical and effective solution for real-world TBPS deployment. Source code is available at https://github.com/zqxie77/CONQUER.
1. INTRODUCTION
TBPS must bridge cross-modal gaps while handling redundant features, noisy correspondence, and ambiguous queries. CONQUER addresses these challenges through CARE for training-time alignment and IQE for inference-time query refinement.
- TBPS retrieves target individuals from large image galleries using natural-language descriptions but faces redundant visual features and noisy cross-modal correspondence.
- Existing methods often rely on passive embedding alignment, manual annotations, or initial queries that limit adaptive handling of ambiguous descriptions.
- CONQUER combines CARE for robust cross-modal representation learning with IQE, which adaptively refines ambiguous or incomplete queries during inference.
- CARE combines multi-granularity representation encoding, complementary pair mining, and context-guided optimal matching via Optimal Transport.
- IQE uses candidate-image attributes to enrich user queries, while CONQUER achieves state-of-the-art performance and robustness across benchmarks and incomplete-query scenarios.
2. METHODOLOGY
CONQUER uses a two-stage design: CARE enhances cross-modal embeddings during training, while IQE refines user queries at inference.
- CONQUER comprises CARE during training and IQE at inference to improve cross-modal retrieval for TBPS.
2.1. Context-Aware Representation Enhancement
CARE improves cross-modal alignment through multi-granularity encoding, complementary pair mining, and context-guided optimal matching. It combines global and local similarity information to identify training-pair confidence and align fine-grained features with semantic context.
- CARE uses multi-granularity representation encoding, complementary pair mining, and context-guided optimal matching to improve cross-modal alignment.
- Multi-granularity Representation Encoding: Modality-specific encoders extract global and local image-text features, which are arranged into feature matrices for similarity computation.
- Multi-granularity Representation Encoding: The batch-wise similarity matrix computes cosine similarity between every image-text pair using global features.
- Complementary Pair Mining: Training pairs are divided into clean, uncertain, and refinable sets according to confidence, with refinable pairs supplying complementary negatives.These negatives increase diversity and sharpen decision boundaries.
- Context-guided Optimal Matching: Context-guided optimal matching formulates local alignment as an Optimal Transport problem and solves the regularized transport objective with Sinkhorn.
- Context-guided Optimal Matching: The supervision objective aligns the transport plan with the local similarity matrix through row-wise KL-divergence and combines alignment and negative losses with trade-off weights.
2.2. Interactive Query Enhancement
IQE is a plug-and-play inference module that identifies reliable candidate-image anchors, enriches ambiguous queries, and fuses the enhanced information for re-ranking. It controls computation through early stopping and fallback activation, while retaining the original ranking when validation alignment drops substantially.
- IQE refines ambiguous queries without retraining the backbone through anchor identification, interactive refinement, and query fusion for re-ranking.
- Anchor Identification: The backbone produces top-K candidates and similarity scores, while a multimodal reasoning model supplies verification confidences for anchor selection.
- Anchor Identification: IQE accepts the initial ranking when the top score exceeds a threshold and activates only for short, ambiguous, or low-confidence queries.
- Interactive Query Refinement: For each anchor, IQE generates diagnostic questions, retains high-confidence responses, and aggregates evidence across anchors using confidence-weighted voting.
- Query Fusion and Re-ranking: The final retrieval score combines original and enhanced similarities with an optional anchor bonus.
- Query Fusion and Re-ranking: A safeguard restores the original ranking when the enhanced query causes a substantial drop in validation alignment.
- CARE context: CARE jointly analyzes global and local similarities to classify pairs, mine hard negatives, and guide fine-grained Optimal Transport alignment.
3. EXPERIMENTS
CONQUER is evaluated on three TBPS benchmarks, cross-domain transfers, and component ablations. It achieves strong in-domain and transfer performance, with the full CARE–IQE combination producing the best reported ablation results.
- Datasets and Metrics: CONQUER is evaluated on CUHK-PEDES, ICFG-PEDES, and RSTPReid using Rank-k accuracy and mAP.The text-to-image retrieval evaluation follows official dataset splits.
- In-Domain Comparisons: 77.13% R@1 and 68.75% mAP are achieved on CUHK-PEDES, while ICFG-PEDES reaches 67.70% R@1 and 40.36% mAP.These results use CLIP-ViT as the backbone and outperform cited recent methods.
- In-Domain Comparisons: 68.40% R@1 and 51.73% mAP are achieved on RSTPReid, ranking highest among the compared methods.The reported comparison identifies RDE as an outperformed baseline.
- Cross-Domain Evaluation: 3.10% and 7.87% R@1 improvements over SEN occur for CUHK-PEDES→RSTPReid and RSTPReid→CUHK-PEDES transfers, respectively.CONQUER also surpasses representative methods across R@1, R@5, and mAP under the reported transfer settings.
- Ablation Studies: 68.40% Rank-1 and 51.73% mAP are obtained when CARE and IQE are integrated on RSTPReid.The baseline starts at 66.50% Rank-1 and 51.47% mAP, while either module alone yields only marginal improvements.
4. CONCLUSION
CONQUER combines training-time representation enhancement with inference-time query refinement for TBPS. Experiments show consistent gains in in-domain and cross-domain settings, while future work targets latency, attribute reliability, and broader retrieval tasks.
- Framework: CONQUER uses CARE during training and plug-and-play IQE during inference to address cross-modal representation and ambiguous-query challenges.CARE uses multi-granularity encoding, complementary pair mining, and context-guided Optimal Transport; IQE uses anchor selection and attribute enrichment without retraining.
- Findings: CONQUER consistently surpasses strong baselines across three public benchmarks in both in-domain and cross-domain scenarios.Ablation studies verify complementary effects between CARE and IQE.
- Future Work: Future work targets lower inference latency, more reliable attribute extraction, and extension to broader cross-modal retrieval tasks.These are the stated directions for extending the framework.