Source-linked AI summary
DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval
Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, Gang Hua
TL;DR
Text-based person retrieval must match detailed person descriptions to images despite noisy, misaligned multimodal information. DSSL separates person from surroundings information under a mutually exclusion constraint, refines textual features, and uses diverse alignments. It achieves state-of-the-art performance on CUHK-PEDES and introduces RSTPReid for real-scenario evaluation.
Problem
Unconstrained common-space mappings may miss discriminative person clues and retain misaligned information in high-dimensional multimodal data.
Method
DSSL separates and fuses person and surroundings information under a mutually exclusion constraint, using five alignment paradigms and textual signal denoising.
Results
DSSL achieves state-of-the-art performance on CUHK-PEDES and outperforms previous methods.
Takeaways & Limitations
RSTPReid extends evaluation toward real scenarios with multi-camera indoor and outdoor variation.
Takeaways & Limitations
Visually similar mismatched pedestrians remain difficult, motivating deeper semantic clustering as future work.
Abstract
from arXiv · showhide
Many previous methods on text-based person retrieval tasks are devoted to learning a latent common space mapping, with the purpose of extracting modality-invariant features from both visual and textual modality. Nevertheless, due to the complexity of high-dimensional data, the unconstrained mapping paradigms are not able to properly catch discriminative clues about the corresponding person while drop the misaligned information. Intuitively, the information contained in visual data can be divided into person information (PI) and surroundings information (SI), which are mutually exclusive from each other. To this end, we propose a novel Deep Surroundings-person Separation Learning (DSSL) model in this paper to effectively extract and match person information, and hence achieve a superior retrieval accuracy. A surroundings-person separation and fusion mechanism plays the key role to realize an accurate and effective surroundings-person separation under a mutually exclusion constraint. In order to adequately utilize multi-modal and multi-granular information for a higher retrieval accuracy, five diverse alignment paradigms are adopted. Extensive experiments are carried out to evaluate the proposed DSSL on CUHK-PEDES, which is currently the only accessible dataset for text-base person retrieval task. DSSL achieves the state-of-the-art performance on CUHK-PEDES. To properly evaluate our proposed DSSL in the real scenarios, a Real Scenarios Text-based Person Reidentification (RSTPReid) dataset is constructed to benefit future research on text-based person retrieval, which will be publicly available.
1 INTRODUCTION
Text-based person retrieval addresses image-query scarcity by matching textual descriptions to pedestrian images, but unconstrained common-space mappings can retain misaligned information. DSSL separates person and surroundings information, combines complementary alignments, and introduces RSTPReid for more realistic evaluation.
- Text-based person retrieval identifies a pedestrian in a large image database using an accessible textual description rather than a target image.
- General cross-modal retrieval methods often generalize poorly because text-based person retrieval contains one pedestrian per image and detailed person descriptions.
- CUHK-PEDES is large and diverse but usually captures each person with the same camera under similar time and space conditions.
- Unconstrained latent-space mappings may miss discriminative person clues and retain redundant or misaligned information in high-dimensional multimodal data.
- DSSL separates mutually exclusive person and surroundings information, using complementary alignments to guide information exchange under a mutually exclusion constraint.
- DSSL uses five alignment paradigms and introduces RSTPReid, a 15-camera benchmark with indoor and outdoor variation for more challenging real-scenario evaluation.
2 RELATED WORKS
Related work spans person re-identification, general cross-modal retrieval, and text-based person retrieval, motivating methods that capture local, multigranular, and full-scale image-text correspondences.
- Deep learning has become central to person re-identification, with prior work strengthening representations through interaction, aggregation, and non-local attention modules.
- Text-based person retrieval methods progressively model patch-word similarity, pose-guided regions, multigranular alignments, and full-scale image-text representations.
- DSSL’s framework incorporates global, person, reconstructed, and local features alongside five alignment paradigms for multimodal and multigranular matching.
3 METHODOLOGY
DSSL is organized around modules that separate, fuse, denoise, describe, and attend to person information across visual and textual modalities.
- DSSL consists of SPSM, SPFM, SDM, PDM, and SAM modules for surroundings-person separation, fusion, denoising, person description, and salient attention.
3.1 Feature Extraction And Refinement
DSSL extracts global and local visual-textual features, refines textual person representations through denoising and reconstruction, and uses ranking-based matching constraints.
- A ResNet-50 backbone produces global visual features through pooling, group normalization, and fully connected projection, while horizontal strips provide local features.
- A bi-GRU processes whole sentences and extracted phrases to form global textual vectors and local textual feature matrices.
- The Signal Denoising Module zeroes a fixed ratio of vector elements and reconstructs the inputs to obtain textual person and local person features.
- The zeroing and reconstruction mechanism requires effective information to be retained while redundant noise signals are discarded.
- A triplet ranking loss constrains matched pairs to be closer than mismatched pairs by margin α using cosine similarity.
- The ranking loss sums all within-mini-batch pairs rather than using only the furthest positive and closest negative pairs.
3.2 Deep Surroundings-Person Separation Learning
DSSL separates visual person and surroundings information, then uses fusion and five alignment paradigms to match multi-modal, multi-granular person features.
- Five alignment paradigms exploit multi-modal and multi-granular information for robust surroundings-person separation learning.The stated goal is to improve retrieval accuracy.
- The SPSM separates person feature V_P and surroundings feature V_S using two parallel MLPs followed by a tanh layer.The feature dimension conversion is p→2p→p.
- Person features extracted from both modalities are first aligned in Align I.
- The Mutually Exclusion Constraint makes V_P and V_S orthogonal so visual information is distributed between them without overlap.
- The SPFM fuses textual person features T_P with V_S and reconstructs them into visual modality features V_R for alignment with V_G.The module combines inputs by addition or concatenation before MLP processing.
- The PDM reconstructs V_P into textual features T_R, while SAM highlights person information in local visual features and cross-modal attention aligns fine-grained clues.
3.3 Loss Function for Training
DSSL training uses two stages: initial identification-based feature learning followed by joint fine-tuning with identification, alignment, and mutually exclusive separation objectives.
- The complete DSSL training process contains two stages.
- Stage 1: Stage 1 fixes the ResNet-50 backbone and trains the left feature-extraction part with identification loss.
- Stage 1: Stage 1 uses global visual and textual features V_G and T_G because global features provide more complete information for clustering.
- Stage 2: Stage 2 fine-tunes all DSSL parameters together and applies identification loss to person and reconstructed features.
- Stage 2: The five alignment losses are used to improve retrieval accuracy.
- Stage 2: Stage 2 combines the five alignment losses with the mutually exclusion loss in the complete training objective.
4.1 Experimental setup
Experiments evaluate DSSL on CUHK-PEDES and the newly constructed RSTPReid dataset using top-k retrieval accuracy, with a two-stage training setup and cross-modal re-ranking.
- DSSL is evaluated on CUHK-PEDES and the proposed RSTPReid dataset.
- Datasets: CUHK-PEDES contains 34,054 training images, 3,078 validation images, and 3,074 testing images, with textual descriptions for each split.
- Datasets: RSTPReid contains 20,505 images of 4,101 persons from 15 cameras, with five cross-camera images and two textual descriptions per person image.
- Metrics: Top-1, top-5, and top-10 accuracy are reported based on whether a corresponding-person image appears among the top-k ranked images.
- Implementation: DSSL uses feature dimension p=1024, six local strips, at most 26 sentence phrases, batch size 32, and ranking-loss margin α=0.2.
- Implementation: Training runs for 10 fixed-backbone epochs in Stage 1 and 30 additional epochs updating all parameters in Stage 2.
- Implementation: A cross-modal re-ranking scheme is used during testing and real application to further improve retrieval accuracy.
4.2 Ablation Analysis
Ablations show that DSSL’s separation, exclusion constraint, multi-granular alignments, and denoising components each contribute to retrieval performance. The experiments also identify challenging real-scene conditions and remaining semantic-clustering limitations.
- Dataset difficulty and remaining limitations: RSTPReid varies cameras, scenes, time periods, illumination, weather, viewpoints, and body positions, making retrieval more challenging and leaving space for further research.Examples also show visually similar mismatched pedestrians, while closer semantic clustering without mixing remains future work.
- Surroundings-person separation and fusion mechanism: Removing SPSM, SPFM, and MEC sharply reduces top-1, top-5, and top-10 performance by 4.46%, 2.79%, and 2.37%.The complete model better captures person clues while discarding misaligned information from high-dimensional multimodal data.
- Surroundings-person separation and fusion mechanism: Adding SPSM and SPFM without MEC improves top-1, top-5, and top-10 performance by 1.79%, 1.71%, and 1.67%.Performance remains 2.67%, 1.08%, and 0.70% below complete DSSL, supporting the role of mutually exclusive person and surroundings features.
- Alignment paradigms: Adding fine-grained alignments Align IV and Align V increases top-1, top-5, and top-10 performance by 1.12%, 0.71%, and 0.61% on CUHK-PEDES and 1.43, 1.25, and 0.56 on RSTPReid.The reported gains support using multiple granularities alongside multimodal alignment.
- Signal denoising module (SDM): Ranking loss in SDM yields 1.05% higher top-1 accuracy than Euclidean distance, while removing SDM lowers top-1, top-5, and top-10 performance by 2.52%, 0.74%, and 0.37%.The zeroing-rate study reports an initial increase followed by deterioration after a peak as the rate grows.
4.3 Comparison With Other State-of-the-art Methods
DSSL is compared with 15 state-of-the-art methods on text-based person retrieval using top-1, top-5, and top-10 accuracy. It achieves the highest reported performance and benefits from separating person and surroundings information.
- 59.98%, 80.41%, and 87.56% are DSSL’s top-1, top-5, and top-10 accuracies, respectively.
- DSSL outperforms the 15 compared state-of-the-art methods on the text-based person retrieval task.
- DSSL also surpasses NAFS when both methods use a cross-modal re-ranking method.
- Properly separating person and surroundings information enables DSSL to exceed methods that directly map data into a common space.
- DSSL significantly improves over attention-based similarity methods, indicating more effective capture of detailed person information.
5 CONCLUSION
The paper proposes DSSL to extract and match person information through surroundings-person separation and fusion under a mutually exclusive constraint. It reports state-of-the-art performance on CUHK-PEDES and introduces RSTPReid for real-scenario evaluation.
- DSSL extracts and matches person information using a surroundings-person separation and fusion mechanism under a mutually exclusive constraint.
- Five diverse alignment paradigms use multi-modal and multi-granular information to improve retrieval accuracy.
- DSSL achieves state-of-the-art performance on CUHK-PEDES, currently the only accessible text-based person retrieval dataset.
- The paper constructs the Real Scenarios Text-based Person Reidentification dataset to support future real-scenario research.