Source-linked AI summary
CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval
Zijie Wang, Aichun Zhu, Jingyi Xue, Xili Wan, Chao Liu, Tian Wang, Yifeng Li
TL;DR
Text-based person retrieval must match person images to natural-language descriptions, but existing methods can over-rely on color and miss other discriminative clues. CAIBC combines RGB, grayscale, and color branches through mutual learning. It achieves state-of-the-art performance across the reported supervised and weakly supervised settings on CUHK-PEDES and RSTPReid.
Problem
Existing text-based person retrieval methods rely heavily on color, which can distract from texture, structural, and other discriminative clues during cross-modal matching.
Source: §1 · pp. 1–2Method
CAIBC jointly optimizes RGB, grayscale, and color branches and uses mutual learning to let them communicate and learn from each other.
Source: §1 · p. 2Results
CAIBC achieves state-of-the-art performance on CUHK-PEDES and RSTPReid in supervised and weakly supervised text-based person retrieval settings.
Source: §1 · p. 2Takeaways & Limitations
In the weakly supervised setting, CAIBC is competitive with or slightly better than existing methods without clustering or pseudo-label generation.
Source: §4.3.2 · p. 6
Abstract
from arXiv · showhide
Given a natural language description, text-based person retrieval aims to identify images of a target person from a large-scale person image database. Existing methods generally face a \textbf{color over-reliance problem}, which means that the models rely heavily on color information when matching cross-modal data. Indeed, color information is an important decision-making accordance for retrieval, but the over-reliance on color would distract the model from other key clues (e.g. texture information, structural information, etc.), and thereby lead to a sub-optimal retrieval performance. To solve this problem, in this paper, we propose to \textbf{C}apture \textbf{A}ll-round \textbf{I}nformation \textbf{B}eyond \textbf{C}olor (\textbf{CAIBC}) via a jointly optimized multi-branch architecture for text-based person retrieval. CAIBC contains three branches including an RGB branch, a grayscale (GRS) branch and a color (CLR) branch. Besides, with the aim of making full use of all-round information in a balanced and effective way, a mutual learning mechanism is employed to enable the three branches which attend to varied aspects of information to communicate with and learn from each other. Extensive experimental analysis is carried out to evaluate our proposed CAIBC method on the CUHK-PEDES and RSTPReid datasets in both \textbf{supervised} and \textbf{weakly supervised} text-based person retrieval settings, which demonstrates that CAIBC significantly outperforms existing methods and achieves the state-of-the-art performance on all the three tasks.
1 INTRODUCTION
Text-based person retrieval must match discriminative clues across images and descriptions, but existing methods can over-rely on color and miss other useful cues. CAIBC combines RGB, grayscale, and color branches with mutual learning, and the paper reports state-of-the-art performance in supervised and weakly supervised settings.
1 INTRODUCTION
- Existing retrieval methods can favor color-matched images while overlooking small discriminative clues such as bags or high-top shoes.The paper also describes cases where color is associated with the wrong local image parts but the image still ranks highly.Source: §1 · pp. 1–2
- RGB and grayscale models focus on different local regions, suggesting their joint use could provide complementary information and alleviate color over-reliance.The paper argues that color remains useful, but excessive reliance can distract from texture and structural clues.Source: §1 · p. 2
- CAIBC jointly optimizes RGB, grayscale (GRS), and color (CLR) branches, using mutual learning to help them communicate and learn from one another.The GRS branch removes image color and masks color-related words, while the CLR branch enhances extracted color information.Source: §1 · p. 2
- CAIBC achieves state-of-the-art performance in both supervised and weakly supervised text-based person retrieval settings on CUHK-PEDES and RSTPReid.The method also reports competitive or slightly better weakly supervised performance without clustering or pseudo-label generation.Source: §1 · p. 2
2 RELATED WORKS
The related-work section situates text-based person retrieval alongside person re-identification and describes approaches to image-text matching, including methods using attention, multiple granularities, and structured representations.
2.1 Person Re-identification
- Person re-identification addresses large intra-class and small inter-class variation across views, poses, illumination, and occlusion; IA Block and MCLNet are cited as representation approaches and modality-confusion learning.Figure 2 presents RGB and grayscale images alongside feature response maps from models trained with and without color information.Source: §2.1 · pp. 2–3, Figure 2 · p. 3
2.2 Text-based Person Retrieval
- Text-based retrieval work includes pose attention, multi-granularity alignment, scene-graph representations, attribute and class spaces, and weakly supervised mutual training.Other cited approaches address matching redundancy, person-surroundings separation and the RSTPReid dataset, color reasoning, and cross-modal, cross-domain search.Source: §2.2 · p. 3
3 METHODOLOGY
CAIBC formulates text-based person retrieval as matching pedestrian images with descriptions, then combines RGB, grayscale, and color-focused branches to capture complementary information. Its branches use mutual learning alongside identification and triplet-ranking losses.
3.1 Problem Formulation
- Given a description, the task is to retrieve the most relevant pedestrian images from a gallery of image–text pairs.Each pair links a surveillance-camera image with its corresponding textual query.Source: §3.1 · p. 3
3.2 Branch Prototype
- The branch prototype extracts global and fine-grained visual and textual representations for cross-modal similarity matching.Visual local features come from horizontally partitioned image maps; text uses word attention to obtain local representations.Source: §3.2 · p. 3, §3.2.1 · p. 3, §3.2.2 · pp. 3–4
3.3 Color Deprivation and Masking
- Color Deprivation and Masking remove color cues by converting images to grayscale and replacing description color words with [CLR].The color bank collects frequently occurring color-related words, and both modules have no learnable parameters.Source: §3.3 · p. 4
3.4 Capturing All-round Information via Multi-branch Architecture
- The RGB branch uses raw inputs, the GRS branch uses grayscale images and masked text, and the CLR branch isolates visual color features and enhances textual color representations.The CLR visual feature is obtained by subtracting the GRS feature map from the RGB feature map; its text representation uses color-related words as a prior.Source: §3.4.1 · p. 4, §3.4.3 · p. 5, §3.4.2 · p. 5
3.5 Optimization
- Mutual learning uses cross-branch KL-divergence knowledge communication, while triplet-ranking and identification losses also train CAIBC.Each branch is optimized under a mutual-learning loss.Source: §3.5 · p. 5
4 EXPERIMENTS
Experiments compare CAIBC with prior systems under supervised and weakly supervised settings and ablate its components. The reported results favor CAIBC across these settings.
4.1 Experimental Setup; 4.2 Implementation Details
- Evaluation uses CUHK-PEDES and RSTPReid, reports R@1/5/10, and uses ResNet-50 visual backbones with fixed BERT text encoding.The setup also specifies image resizing, batch size, optimizer, and training duration.Source: §4.1.1 · p. 5, §4.1.2 · p. 5, §4.2 · p. 5
4.3 Results
- CAIBC achieves state-of-the-art performance on supervised CUHK-PEDES and RSTPReid, and competitive or slightly better weakly supervised performance without clustering- or pseudo-label-based methods.The supervised comparisons are reported for both datasets; the weakly supervised comparison is on CUHK-PEDES.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6
- The paper argues that identity labels or weaker priors can flag mistakes but do not identify the missed details, limiting their ability to address color over-reliance.It describes color over-reliance as more significant when only pairwise relationships, rather than identity information, are available.Source: §4.3.3 · p. 6
4.4 Ablation Analysis
- Combining RGB and GRS branches improves performance over single branches, and adding the CLR branch brings further gains.The authors attribute the gains to branches attending to different information and complementing one another.Source: §4.4.1 · p. 7
- Mutual learning improves Rank-1/5/10 accuracy in both datasets in the Exp.5-versus-6 and Exp.17-versus-8 comparisons.The reported increases are 0.91%, 0.42%, and 0.37% on CUHK-PEDES and 0.95%, 0.35%, and 0.40% on RSTPReid for Exp.5 versus Exp.6; Exp.17 versus Exp.8 also improves the reported accuracies.Source: §4.4.2 · p. 7
- Adding color prior information brings further performance gains in the (Exp.7, Exp.8) and (Exp.9, Exp.17) comparisons.The authors attribute this effect to enhancing the CLR branch.Source: §4.4.3 · p. 7
4.5 Visualization
- Figure 5 pairs RGB and grayscale inputs with their feature response maps, while Figure 6 contrasts retrieval examples from single-branch models trained with and without color.Figure 6 marks matched and mismatched images in green and red, respectively.Source: Figure 5 · p. 8, Figure 6 · p. 8, §4.5 · p. 8
5 CONCLUSION
CAIBC addresses color over-reliance in text-based person retrieval with a jointly optimized three-branch architecture and mutual learning. It achieves state-of-the-art performance across the reported supervised and weakly supervised tasks on CUHK-PEDES and RSTPReid.
5 CONCLUSION
- CAIBC significantly outperforms existing methods and achieves state-of-the-art performance on all three supervised and weakly supervised tasks evaluated on CUHK-PEDES and RSTPReid.The tasks are text-based person retrieval settings.Source: §5 · p. 8
- CAIBC uses jointly optimized RGB, grayscale (GRS), and color (CLR) branches with mutual learning so they can communicate and learn from each other.The branches attend to varied aspects of information.Source: §5 · p. 8
- The method targets color over-reliance, which can distract models from texture and structural clues and lead to sub-optimal retrieval performance.Source: §5 · p. 8