Source-linked AI summary
Pose-Guided Multi-Granularity Attention Network for Text-Based Person Search
Ya Jing, Chenyang Si, Junbo Wang, Wei Wang, Liang Wang, Tieniu Tan
TL;DR
Text-based person search must retrieve matching images from descriptions despite multi-granularity image-text relevance and irrelevant visual regions. PMA combines pose-guided coarse and fine-grained alignment, and it outperforms state-of-the-art methods on CUHK-PEDES, including a roughly 15% top-1 gain over Dual Path with VGG-16 features.
Problem
Text-based person search requires extracting description-corresponding visual contents despite multi-granularity semantic relevance and irrelevant regions in global image representations.
Method
PMA combines similarity-based coarse alignment of descriptions with image regions and pose-guided fine-grained alignment of noun phrases with visual body parts.
Results
About 15% top-1 improvement over Dual Path is reported with VGG-16 features, while PMA achieves the best top-1, top-5, and top-10 performances among compared methods.
Takeaways & Limitations
Multi-granularity alignment and pose-guided part matching provide an effective approach for text-based person search on CUHK-PEDES.
Takeaways & Limitations
Accurate human poses remain difficult on CUHK-PEDES under occlusion and lighting changes, including partially or completely missing keypoints.
Abstract
from arXiv · showhide
Text-based person search aims to retrieve the corresponding person images in an image database by virtue of a describing sentence about the person, which poses great potential for various applications such as video surveillance. Extracting visual contents corresponding to the human description is the key to this cross-modal matching problem. Moreover, correlated images and descriptions involve different granularities of semantic relevance, which is usually ignored in previous methods. To exploit the multilevel corresponding visual contents, we propose a pose-guided multi-granularity attention network (PMA). Firstly, we propose a coarse alignment network (CA) to select the related image regions to the global description by a similarity-based attention. To further capture the phrase-related visual body part, a fine-grained alignment network (FA) is proposed, which employs pose information to learn latent semantic alignment between visual body part and textual noun phrase. To verify the effectiveness of our model, we perform extensive experiments on the CUHK Person Description Dataset (CUHK-PEDES) which is currently the only available dataset for text-based person search. Experimental results show that our approach outperforms the state-of-the-art methods by 15 \% in terms of the top-1 metric.
Introduction
Text-based person search retrieves person images from descriptions, but matching is difficult because relevant visual content spans multiple semantic granularities. PMA addresses this with pose-guided coarse and fine-grained alignment.
- Textual descriptions are more accessible and can describe appearance in greater detail and more naturally than image- or attribute-based queries.
- Text-based person search retrieves corresponding person images from a textual description in a large-scale image database.
- Global image features can include irrelevant regions, while prior single-level attention methods neglect multi-granularity image-text relevance.
- PMA uses coarse alignment to select description-related regions and fine-grained alignment to match noun phrases with visual body parts using pose information.
- The paper identifies multi-granularity visual extraction and pose-guided part alignment as contributions, and reports best results on CUHK-PEDES with extensive ablation studies.
Related Work
Prior text-based person-search methods use recurrent, identity-aware, word-image, or CNN-RNN approaches, while related work also applies pose and attention mechanisms to person search.
- GNA-RNN applies recurrent neural networks with gated neural attention to text-based person search.
- Identity-aware two-stage methods use identity-level annotations, while PWM+ATH matches words with image patches to capture local similarity.
- Image-based person-search approaches use human pose to improve visual representations and address pose variation through pose-transferred sample augmentation.
- Person-search attention mechanisms may be soft or hard, selecting inputs with weight maps or preserving only a few parts.
Network
PMA combines pose-augmented visual representations with coarse global-description alignment and fine-grained phrase-to-body-part alignment. Its training uses ranking and part losses, while confidence maps retain useful body-part cues when pose keypoints are inaccurate or missing.
- Visual Representation Extraction: Pose confidence maps remain informative under inaccurate or missing keypoints caused by occlusion and lighting changes.The paper reports that superimposed confidence maps can still provide cues about the person and body parts when generated joint keypoints fail.
- Visual Representation Extraction: PMA uses pose confidence maps to augment the input image and learn human-related visual features.The maps are concatenated with the three-channel image, producing a 17-channel input for visual representation extraction.
- Textual Representation Learning: The textual branch embeds words into 300-dimensional vectors, models adjacent-word dependencies with a bi-LSTM, and extracts noun phrases for phrase-level matching.The global textual representation concatenates the final forward and backward hidden states, while noun phrases use the same bi-LSTM encoding.
- Coarse Alignment Network: The coarse alignment network selects description-related image regions by comparing local visual and textual representations with cosine similarity and hard attention.A threshold τ retains local similarities whose weights exceed τ, rather than summing all regions.
- Fine-Grained Alignment Network: The fine-grained alignment network uses 14 pose confidence maps grouped into six body parts to align visual parts with textual noun phrases.Pose CNN embeddings guide attention over noun phrases and image regions for part-level matching.
- Learning PMA: Training combines triplet ranking loss with an additional part loss that classifies the six pose parts into six categories.The ranking loss separates positive pairs from the hardest negative pairs with a margin, while part loss encourages different pose parts to attend differently.
Experiments
Experiments evaluate PMA on CUHK-PEDES using retrieval metrics, state-of-the-art comparisons, ablations, and attention visualizations. Results support the contributions of pose augmentation, coarse alignment, and fine-grained alignment, while analyzing region selection and textual encoding choices.
- Experimental Setup: Experiments use CUHK-PEDES, the only available text-based person-search dataset, with top-1, top-5, and top-10 retrieval accuracies.The reported split contains 34,054 training images, 3,078 validation images, and 3,074 test images.
- State-of-the-art Comparison: PMA achieves the best top-1, top-5, and top-10 performances against state-of-the-art methods using the same visual CNN.Against Dual Path, top-1 improves by about 15% with VGG-16 and 9% with ResNet-50.
- Ablation Experiments: Using all 9,408 vocabulary words negatively affects accuracy, whereas bi-LSTM improves performance over unidirectional LSTM.The experiments attribute the vocabulary effect to noise from low-frequency words.
- Ablation Experiments: Ablations show that pose concatenation, coarse alignment, and fine-grained alignment each improve text-based person search performance.Con-pose+CA improves top-1 by 2.2% over Con-pose, while adding FA improves top-1 by another 1.6%.
- Coarse Alignment Analysis: Similarity-based hard attention mainly selects 6 regions for positive pairs and 20 regions for positive and negative pairs.For positive pairs it selects description-related regions; for negative pairs it tends toward selecting all regions because corresponding regions are lacking.
- Attention Analysis: Performance rises as more top-m regions are selected and soon saturates, while attention visualizations show attended areas through lighter regions and higher values.These analyses indicate that only some description-related regions are useful for matching and visualize the selected regions and pose-guided attention maps.
Conclusion
The paper proposes PMA, combining coarse and fine-grained alignment networks to learn multi-granularity cross-modal relevance for text-based person search. Extensive experiments and ablation analysis on a challenging dataset show a large performance margin over state-of-the-art methods.
- PMA learns multi-granularity cross-modal relevance through coarse and fine-grained alignment networks.
- The coarse alignment network selects description-related image regions using similarity-based attention.
- The fine-grained alignment network uses pose information to guide attention toward phrase-related visual contents.
- Extensive experiments with ablation analysis on a challenging dataset show that PMA outperforms state-of-the-art methods by a large margin.