Source-linked AI summary
Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search
Chenyang Gao, Guanyu Cai, Xinyang Jiang, Feng Zheng, Jun Zhang, Yifei Gong, Pai Peng, Xiaowei Guo, Xing Sun
TL;DR
Text-based person search must match descriptive sentences to people in image galleries despite similar pedestrian semantics and methods that restrict alignment to selected scales. NAFS builds full-scale image and text features and adaptively aligns them across scales; it reports 5.53% and 5.35% gains in top-1 and top-5 over ViTTA on CUHK-PEDES.
Problem
Similar high-level semantics among pedestrian images create small inter-class variation, while existing methods often use limited scales or fixed scale-pair alignments.
Source: §1 · pp. 1–2Method
NAFS extracts full-scale image and text features and uses contextual non-local attention to adaptively align them across scales.
Source: §1 · p. 2, §3.3 · p. 4Results
5.53% and 5.35% performance improvements over ViTTA in top-1 and top-5, respectively, are reported on CUHK-PEDES.
Source: §4.2 · p. 6Takeaways & Limitations
Ablations on CUHK-PEDES demonstrate that NAFS outperforms state-of-the-art methods by a large margin.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Text-based person search aims at retrieving target person in an image gallery using a descriptive sentence of that person. It is very challenging since modal gap makes effectively extracting discriminative features more difficult. Moreover, the inter-class variance of both pedestrian images and descriptions is small. So comprehensive information is needed to align visual and textual clues across all scales. Most existing methods merely consider the local alignment between images and texts within a single scale (e.g. only global scale or only partial scale) then simply construct alignment at each scale separately. To address this problem, we propose a method that is able to adaptively align image and textual features across all scales, called NAFS (i.e.Non-local Alignment over Full-Scale representations). Firstly, a novel staircase network structure is proposed to extract full-scale image features with better locality. Secondly, a BERT with locality-constrained attention is proposed to obtain representations of descriptions at different scales. Then, instead of separately aligning features at each scale, a novel contextual non-local attention mechanism is applied to simultaneously discover latent alignments across all scales. The experimental results show that our method outperforms the state-of-the-art methods by 5.53% in terms of top-1 and 5.35% in terms of top-5 on text-based person search dataset. The code is available at https://github.com/TencentYoutuResearch/PersonReID-NAFS
1. Introduction
Text-based person search uses natural-language descriptions to retrieve people from image galleries, but similar pedestrian semantics make discriminative matching difficult. NAFS builds full-scale image and text representations and adaptively aligns them across scales.
1. Introduction
- Natural-language queries make person search more user-friendly and may support broader applications.Source: §1 · p. 1
- Small inter-class variation motivates representing both images and descriptions at all scales and aligning them within and across scales.Descriptions can range from short phrases to longer sub-sentences, while images can be divided into regions of different sizes.Source: §1 · pp. 1–2
- Existing methods often use only global or finest-scale representations, while fixed scale-pair alignment rules ignore other possible cross-scale matches.Some methods combine global and fine-grained features but omit mid-scale information.Source: §1 · p. 2
- NAFS uses a staircase network with stripe shuffling for visual locality, locality-constrained BERT for full-scale text features, and contextual non-local attention for adaptive cross-scale alignment.The attention jointly takes image and text representations from all scales as input rather than relying on predefined scale pairs.Source: §1 · p. 2
- A nearest-visual-neighbor re-ranking algorithm is proposed to further improve ranking quality.Source: §1 · p. 2
2. Related Work
Related work covers global and part-based person representations, text-based person-search matching and alignment methods, and general image-text matching approaches that align visual and textual features.
2.1. Person Re-identification (ReID).
- Person re-identification methods use global CNN features, local parts, coarse-to-fine pyramids, or semantic regions, with some approaches relying on pose estimation or semantic parsing accuracy.Source: §2.1 · p. 2
2.2. Text-Based Person Search.
- Text-based person-search work includes affinity learning, two-stage matching, patch-word matching, cross-modal objectives, attribute supervision, and multi-granularity alignment.The paper's framework combines a staircase visual network, locality-constrained BERT, and contextual non-local attention for joint alignment.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
2.3. Image Text Matching.
- General image-text matching methods align images with textual fragments, while BERT- or transformer-based binary-classification approaches improve retrieval at the cost of longer inference time.Source: §2.3 · p. 3
3. Our Method
NAFS extracts image and text representations at multiple scales, then adaptively aligns them across scales. Its method combines locality-focused representation modules, contextual non-local attention, and visual-neighbor re-ranking.
3.1. Extracting Visual Representation
- The staircase backbone extracts global, region, and patch features, while split-and-shuffle processing is used to improve locality in the finer-scale branches.The region and patch branches partition feature maps into horizontal stripes to produce local features.Source: §3.1 · pp. 3–4
3.2. Extracting Textual Representation
- BERT produces sentence-, comma-delimited sub-sentence-, and word-level representations, with locality-constrained attention limiting sub-sentence [CLS] attention to a token range.The whole sentence is represented by its [CLS] hidden state; sub-sentences use their own [CLS] states, and words use their final hidden states.Source: §3.2 · p. 4
3.3. Contextual Non-Local Attention Mechanism
- Contextual non-local attention takes multiscale visual and textual features and outputs an image-text similarity score, adaptively aligning features across scales rather than using fixed scale pairings.The method uses image-text and text-image attention directions, comparing visual values with attended textual contexts and averaging their similarities.Source: §3.3 · pp. 4–5
- Cross-Scale Alignment Loss optimizes the alignment module, while global-branch CMPM and CMPC losses are added to stabilize training.The final objective combines the cross-scale alignment objective with the two global-branch losses.Source: §3.3 · p. 5
3.4. Re-Ranking by Visual Neighbors
- Visual-neighbor re-ranking recalculates query-image similarity using Jaccard distance between their nearest-neighbor sets, then re-sorts the gallery using averaged original and Jaccard scores.Global features are used to find neighbors for computational efficiency.Source: §3.4 · pp. 5–6
4. Experiments
On CUHK-PEDES, NAFS outperforms the compared state-of-the-art methods and its ablations support using full-scale representations and joint cross-scale alignment. Visualizations show word and sentence alignments with image regions at different scales.
4.1. Experimental Setup
- CUHK-PEDES evaluation uses top-k accuracy (k = 1, 5, 10) over 3,074 test images, 1,000 person IDs, and 6,156 descriptions.The dataset has 40,206 images and 13,003 unique person IDs overall.Source: §4.1 · p. 6
4.2. Comparison with State-of-The-Art Methods
- 5.53%, 5.35%, and 3.99% are NAFS’s reported improvements over ViTTA in top1, top5, and top10 accuracy, respectively.NAFS achieves the highest performance with and without RVN.Source: §4.2 · p. 6, Table 1 · p. 6
4.3. Extensive Ablation Studies
- Top1 accuracy rises from 55.47 to 56.90 after adding local information, then to 59.94 after adding mid-scale information.The scale comparison evaluates representations at different scales.Source: Table 2 · p. 6, §4.3 · p. 7
- Joint alignment outperforms separate alignment of global-global, large-region/sub-sentence, and small-region/word pairs.The authors report that joint alignment boosts text-based person search performance.Source: §4.3 · pp. 7–8, Table 3 · p. 7
- Locality-constrained BERT improves top1 accuracy by 0.71, the staircase network by 2.12, contextual non-local attention by 2.04, and RVN by 1.56.Split-and-shuffle is also added to the staircase backbone; the component comparison includes the baseline and these additions.Source: §4.3 · p. 8
4.4. Visualization Analysis
- Figure 4 visualizes cross-scale alignment, with red and yellow regions marking the highest and second-highest attention weights for corresponding text.Examples include words aligned to whole images, image stripes, and regions at different scales.Source: Figure 4 · p. 7, §4.4 · p. 8
5. Conclusion
NAFS performs joint alignment over full-scale representations using multi-scale image and text features and contextual non-local attention; CUHK-PEDES ablations report that it outperforms state-of-the-art methods by a large margin.
5. Conclusion
- NAFS’s extensive ablation studies on CUHK-PEDES show that it outperforms state-of-the-art methods by a large margin.Source: §5 · p. 8
- A staircase CNN and locality-constrained BERT extract multi-scale image and textual representations.Source: §5 · p. 8
- Contextual non-local attention adaptively aligns learned representations across different scales.Source: §5 · p. 8