Source-linked AI summary
Learning Granularity-Unified Representations for Text-to-Image Person Re-identification
Zhiyin Shao, Xinyu Zhang, Meng Fang, Zhifeng Lin, Jian Wang, Changxing Ding
TL;DR
Text-to-image ReID faces a feature-granularity gap because visual features are fine-grained while textual descriptions are coarse, and existing works often overlook it. LGUR uses shared-dictionary reconstruction and shared-prototype extraction to unify representations, consistently outperforming state-of-the-art methods on CUHK-PEDES and ICFG-PEDES.
Problem
Text-to-image ReID must match fine-grained visual features with coarse textual descriptions, while existing works often overlook this feature-granularity modality gap.
Method
LGUR combines dictionary-based reconstruction with a Multi-modality Shared Dictionary and prototype-based extraction using shared learnable prototypes for both modalities.
Results
LGUR consistently outperforms state-of-the-art methods by large margins on CUHK-PEDES and ICFG-PEDES.
Takeaways & Limitations
LGUR provides granularity-unified representations while avoiding cross-modal attention between every image-text pair during testing.
Abstract
from arXiv · showhide
Text-to-image person re-identification (ReID) aims to search for pedestrian images of an interested identity via textual descriptions. It is challenging due to both rich intra-modal variations and significant inter-modal gaps. Existing works usually ignore the difference in feature granularity between the two modalities, i.e., the visual features are usually fine-grained while textual features are coarse, which is mainly responsible for the large inter-modal gaps. In this paper, we propose an end-to-end framework based on transformers to learn granularity-unified representations for both modalities, denoted as LGUR. LGUR framework contains two modules: a Dictionary-based Granularity Alignment (DGA) module and a Prototype-based Granularity Unification (PGU) module. In DGA, in order to align the granularities of two modalities, we introduce a Multi-modality Shared Dictionary (MSD) to reconstruct both visual and textual features. Besides, DGA has two important factors, i.e., the cross-modality guidance and the foreground-centric reconstruction, to facilitate the optimization of MSD. In PGU, we adopt a set of shared and learnable prototypes as the queries to extract diverse and semantically aligned features for both modalities in the granularity-unified feature space, which further promotes the ReID performance. Comprehensive experiments show that our LGUR consistently outperforms state-of-the-arts by large margins on both CUHK-PEDES and ICFG-PEDES datasets. Code will be released at https://github.com/ZhiyinShao-H/LGUR.
1 INTRODUCTION
Text-to-image person ReID is difficult because fine-grained visual features must be matched with coarse textual descriptions, while existing methods often overlook this granularity gap. LGUR addresses it with dictionary-based alignment and prototype-based unification, achieving strong benchmark performance.
- Motivation: Text-to-image ReID retrieves images of a target identity from natural-language descriptions, but the task faces a substantial vision-language modality gap.Descriptions are flexible and easier to obtain in some circumstances, yet cross-modal matching remains challenging.
- Motivation: Fine-grained visual information and coarse textual attributes create a feature-granularity gap that enlarges the modality gap and complicates retrieval.Similar image regions can share the same textual description while differing in visual details.
- Related limitations: Existing works often overlook this feature-granularity modality gap, instead focusing on correspondences between words and image patches of varying sizes.Their common strategy uses cross-modal attention to adapt to changing image-patch sizes.
- Proposed framework: LGUR maps visual and textual features into a granularity-unified space through Dictionary-based Granularity Alignment and Prototype-based Granularity Unification.The framework is designed to address the gap between fine-grained images and coarse textual descriptions.
- Dictionary-based Granularity Alignment: DGA reconstructs both modalities with a Multi-modality Shared Dictionary, using textual guidance to optimize dictionary granularity and align reconstructed visual features.The dictionary stores granularity-unified atoms, while matched textual features guide visual reconstruction toward textual granularity.
- Prototype-based Granularity Unification: PGU uses shared learnable prototypes as queries to extract diverse features independently from both modalities and further reduce their granularity gap.Shared prototypes also reduce testing-time computational cost relative to pairing every image and text with cross-modal attention.
- Results: LGUR consistently and significantly outperforms existing approaches on CUHK-PEDES and ICFG-PEDES while avoiding cross-modal attention for each test image-text pair.The authors also report strong performance in domain generalization tasks.
2 RELATED WORK
Related work in text-to-image person ReID contrasts efficient cross-modal attention-free methods with more accurate but costlier attention-based approaches. LGUR targets a distinct feature-granularity modality gap and avoids image-text cross-modal attention during retrieval.
- Vision-language transformers: Transformer-based vision-language methods extract joint vision-language features using either two-stream or single-stream architectures.Both categories are presented as existing approaches for vision-language pre-training.
- Text-to-image person ReID: Text-to-image person ReID is more challenging than general cross-modal retrieval because of its fine-grained nature, motivating attention-based and attention-free alignment strategies.The related-work taxonomy is organized around the alignment strategy used.
- Attention-free methods: Attention-free methods align modalities through model structures or objective functions in a shared feature space and are generally more efficient.The supplied passage identifies their shared-space alignment approach, while efficiency is stated in the comparison of methods.
- Computational trade-off: For N images and M sentences, attention-free methods have complexity O(M+N), whereas cross-modal attention methods scale as O(MN).Attention-based methods establish region-word or region-phrase correspondences but require image-text pairing.
- Granularity gap: Existing cross-modal attention methods typically define a granularity gap through variable image-patch sizes associated with words, not finer visual detail than corresponding text.LGUR identifies the latter as a distinct modality gap in feature granularity.
- LGUR: LGUR uses a modality-shared dictionary to avoid image-text cross-modal attention and thereby improve computational efficiency.This design supports efficient retrieval without cross-modal operations between each image and text.
3 METHODOLOGY
LGUR aligns visual and textual feature granularities through dictionary-based reconstruction, then unifies their representations with shared learnable prototypes for text-to-image ReID.
- Framework overview: LGUR uses feature-extraction backbones, Dictionary-based Granularity Alignment, and Prototype-based Granularity Unification to enhance image-text granularity unification.The framework separately extracts visual and textual features before applying its two granularity-focused modules.
- Dictionary-based Granularity Alignment: DGA reconstructs visual and textual features with a Multi-modality Shared Dictionary whose atoms are intended to match textual granularity.Transformer cross-attention uses the text or visual features as queries and the shared dictionary as keys and values.
- Dictionary-based Granularity Alignment: DGA uses foreground-centric reconstruction to suppress background clutter and alleviate the optimization difficulty of the shared dictionary.A spatial-attention foreground mask focuses reconstruction on the pedestrian body rather than image background.
- Dictionary-based Granularity Alignment: DGA also reconstructs visual features using paired textual features and applies ranking loss to reduce the difference between the two visual reconstructions.The text-guided reconstruction produces a more abstract feature whose granularity is closer to textual features.
- Prototype-based Granularity Unification: PGU uses shared learnable prototypes as queries to project textual and visual features into a unified format with diverse, semantically aligned outputs.Independent fully connected layers produce diverse prototype features, while each query shares its layer across modalities; a single shared transformer is used.
- Optimization & Inference: Training combines identification losses for reconstructed and guidance features with ranking losses that pull matched features together and separate different identities.At inference, the model separately extracts eT_re and eV_re for text-to-image retrieval and uses cosine similarity.
4 EXPERIMENTS
Experiments evaluate LGUR on two text-to-image ReID benchmarks using standard retrieval metrics, component ablations, efficiency comparisons, and qualitative analysis. LGUR consistently improves retrieval accuracy, efficiency, and granularity alignment across the reported evaluations.
- Experimental setup: LGUR is evaluated on CUHK-PEDES and ICFG-PEDES using Rank-1, Rank-5, and Rank-10 retrieval accuracies.The benchmarks contain large-scale pedestrian images and textual descriptions, with CUHK-PEDES covering 13,003 identities.
- Benchmark comparisons: LGUR achieves Rank-1 accuracies of 64.21% with ResNet-50 and 65.25% with DeiT-Small on CUHK-PEDES.It outperforms NAFS by up to 4.27 percentage points in Rank-1 accuracy while avoiding pairwise cross-modal attention during feature extraction.
- Efficiency comparison: LGUR offers lower retrieval cost than several cross-modal attention methods, requiring 26 ms per query on CUHK-PEDES versus 76 ms for SSAN.The comparison measures training time, inference time, and Rank-1 accuracy under matched image-size, batch-size, and GPU settings.
- Ablation study: Adding PGU improves baseline Rank-1 accuracy by 4.59% on CUHK-PEDES, while adding DGA improves baseline Rank-1 by 3.19% and 3.74% on CUHK-PEDES and ICFG-PEDES.These ablations support the usefulness of unified representations and dictionary-based granularity alignment.
- Ablation study: The complete LGUR configuration improves over the baseline by 6.58% on CUHK-PEDES and 6.93% on ICFG-PEDES in Rank-1 accuracy.The full configuration combines the shared dictionary, foreground mask, reconstruction guidance, and PGU.
- Design analysis: Shared dictionaries and shared prototypes outperform their respective unshared variants, while matched phrase-patch attention scores indicate that dictionary atoms represent both modalities similarly.Foreground reconstruction raises CUHK-PEDES Rank-1 from 64.28% to 64.64%, and guidance V_g improves ICFG-PEDES Rank-1 by 0.65%.
5 CONCLUSION
LGUR learns granularity-unified representations for text-to-image ReID through dictionary-based alignment and prototype-based unification. Experiments on two large-scale databases demonstrate its effectiveness.
- LGUR learns granularity-unified representations for text-to-image person re-identification.
- DGA reconstructs visual and textual features with a Multi-modality Shared Dictionary to unify their granularity.
- Cross-modal guidance and a foreground mask facilitate optimization of the shared dictionary parameters.
- PGU uses shared prototypes to extract diverse textual and visual features while further aligning their granularity.
- Extensive experiments on two large-scale databases demonstrate LGUR's effectiveness.