Source-linked AI summary
HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng, Jing Shao, Shuai Yi, Junjie Yan, Xiaogang Wang
TL;DR
Pedestrian analysis needs comprehensive features for fine-grained tasks because conventional representations may miss localized, low-level, semantic, and multi-scale information. HydraPlus-Net uses multi-directional attention to fuse such features and achieves state-of-the-art performance on pedestrian attribute recognition and person re-identification. The paper also introduces the PA-100K dataset for pedestrian analysis.
Problem
Feature representation learning for pedestrian images remains challenging because existing architectures may miss detailed localized features complementary to high-level global features needed for fine-grained analysis.
Method
HydraPlus-Net uses multi-directional attention modules to aggregate and fuse multi-level, multi-scale features from global and local contents.
Results
The HP-net achieves state-of-the-art performance with significant improvements against prior methods on pedestrian attribute recognition and person re-identification.
Takeaways & Limitations
The framework provides an attentive feature representation for fine-grained pedestrian analysis, and PA-100K supports various pedestrian analysis tasks.
Takeaways & Limitations
PETA samples are annotated only once using a randomly selected exemplar, so images share attributes even when some attributes may not be visible.
Abstract
from arXiv · showhide
Pedestrian analysis plays a vital role in intelligent video surveillance and is a key component for security-centric computer vision systems. Despite that the convolutional neural networks are remarkable in learning discriminative features from images, the learning of comprehensive features of pedestrians for fine-grained tasks remains an open problem. In this study, we propose a new attention-based deep neural network, named as HydraPlus-Net (HP-net), that multi-directionally feeds the multi-level attention maps to different feature layers. The attentive deep features learned from the proposed HP-net bring unique advantages: (1) the model is capable of capturing multiple attentions from low-level to semantic-level, and (2) it explores the multi-scale selectiveness of attentive features to enrich the final feature representations for a pedestrian image. We demonstrate the effectiveness and generality of the proposed HP-net for pedestrian analysis on two tasks, i.e. pedestrian attribute recognition and person re-identification. Intensive experimental results have been provided to prove that the HP-net outperforms the state-of-the-art methods on various datasets.
1. Introduction
Pedestrian analysis requires comprehensive representations that capture localized, low-level, semantic, and multi-scale patterns for fine-grained recognition and re-identification. HydraPlus-Net addresses this need with multi-directional attention and achieves state-of-the-art performance across the evaluated tasks and datasets.
- Motivation: Pedestrian analysis remains challenging because feature representations must capture detailed localized information alongside high-level global features.These representations support fine-grained pedestrian analysis tasks such as attribute recognition and person re-identification.
- Motivation: Low-level features capture patterns such as clothing stride, while semantic features support judgments such as gender from the whole pedestrian image.
- Approach: The framework combines global and local contents with multi-level feature fusion, targeting pedestrian attribute recognition and person re-identification.The two tasks assign attribute labels or associate images of one person across cameras or temporal shots.
- Approach: HydraPlus-Net uses multi-directional attention modules to aggregate feature layers within attentive regions extracted from multiple network layers.Attention maps from different semantic layers abstract different visual-pattern levels, while shared maps filter multiple feature levels for fusion.
- Results: The HP-net achieves state-of-the-art performance with significant improvements against prior methods on pedestrian attribute recognition and person re-identification.
- Resources: The PA-100K dataset provides diverse scenes, the largest number of samples and instances up-to-date, and information useful for various pedestrian analysis tasks.
2. Related Works
Prior work applied attention in several computer-vision tasks and developed pedestrian-analysis methods using convolutional, pose, and body-part information. HydraPlus-Net instead introduces multi-directional attention for pedestrian representation without requiring prior body-part or poselet knowledge.
- Attention models: Computer-vision attention methods include hard attention to selected regions and differentiable soft attention that weights feature maps.
- Attention models: Attention has been used for image caption generation, visual question answering, object detection, multi-scale features, and knowledge transfer.
- Pedestrian attribute recognition: Pedestrian attribute-recognition methods have incorporated pose and body-part information to capture pose-normalized appearance differences.
- Pedestrian attribute recognition: HydraPlus-Net uses attention to attend to important areas and align body parts without prior knowledge of body parts or poselets.
3. HydraPlus-Net Architecture
HydraPlus-Net combines a plain CNN Main Net with an Attentive Feature Net containing multi-directional attention modules across semantic levels. These modules generate attention maps at different layers, apply them across feature levels, and fuse the resulting attentive features.
- Architecture: HP-net consists of a plain CNN Main Net and an Attentive Feature Net with multiple multi-directional attention branches.The AF-net shares the M-net’s convolutional architecture while adding MDA modules at different semantic feature levels.
- Feature fusion: Attentive feature maps are passed through subsequent blocks, concatenated, and fused using global average pooling and fully connected layers.The final representation can be projected into attribute logits or re-identification feature vectors.
- Multi-directional attention: The network uses three MDA modules whose attention maps are generated from different inception blocks and applied to feature maps at multiple levels.Each MDA module contains three branches that mask adjacent inception blocks, producing multi-level attentive features.
- Attention hierarchy: Attention maps from lower layers capture local patterns such as edges and textures, whereas higher-layer maps capture semantic regions such as objects or identity.The architecture therefore combines complementary feature scales and semantic levels.
- Attention-map generation: Each attention map is generated from an inception-block feature map by a 1 × 1 convolution followed by batch normalization and ReLU.The number of attention-map channels is fixed at L = 8 for both evaluated tasks.
- Stage-wise training: Stage-wise training first learns a plain M-net, then fine-tunes three duplicated AF-net MDA modules, and finally trains the remaining pooling and fully connected layers.The M-net and AF-net are fixed before the final GAP and FC training stage.
4. Ablation Study On Attentive Deep Features
The ablation study shows that HydraPlus-Net’s multi-level, multi-scale attention components and multi-directional connections jointly enrich pedestrian representations. Removing attention channels, modules, or connections reduces performance, while complementary attention levels capture both low-level patterns and semantic regions.
- Multi-level Attention Maps: Earlier attention layers capture edges and textures, whereas higher layers identify semantic patterns such as handbags or pedestrian identity.
- Attention Diversity: Multiple attention channels localize different pedestrian regions, enriching representations for recognizing attributes and identity under substantial appearance variation.
- Attention Consistency: Consistent attention maps highlight recurring semantic parts or salient objects, making selective attention important for pedestrian identification.
- Multi-Directional Attentive Features: Higher-level attention maps can localize a phone near the ear, while lower-level maps capture texture patterns such as upper-clothing appearance.
- Component Analysis: Removing MDA modules or internal connections lowers performance, and the complete network surpasses both naïve multi-level attention modules and a single MDA module.
5. Pedestrian Attribute Recognition
The pedestrian attribute recognition experiments evaluate HP-net across existing datasets and the newly introduced PA-100K dataset. HP-net and its baseline M-net outperform prior methods, with especially strong gains on attributes requiring fine-grained localization, while global attributes such as “talking” remain challenging.
- PA-100K Dataset: PA-100K contains 100,000 pedestrian images from 598 scenes, providing a larger and more diverse benchmark than previous attribute datasets.
- PA-100K Dataset: PA-100K annotations cover 26 attributes, use an 8:1:1 train-validation-test split, and assign tracklet samples to one split to separate attribute learning from identity.
- Evaluation: The evaluation uses mean accuracy, accuracy, precision, recall, and F1-score, with weighted cross-entropy addressing imbalanced classes.
- Comparison with Prior Arts: The baseline M-net and HP-net significantly outperform state-of-the-art methods, with prominent HP-net gains on fine-grained attributes such as glasses and handbags.
- Qualitative Evaluation: Qualitative results show attention maps can localize multiple bags, but “talking” remains difficult because it requires global understanding rather than a specific region.
- Qualitative Evaluation: On PA-100K examples, HP-net reliably predicts region-based attributes including glasses, backpack, hat, shorts, and handcarry.
6. Person Re-identification
HP-net is evaluated for person re-identification on three public datasets using CMC accuracy under single-query testing, with quantitative and qualitative comparisons against reference methods and M-net.
- Experimental Setup: The evaluation uses CUHK03, VIPeR, and Market-1501, with CMC Top-1-5-10-20 accuracies reported for comparison.Testing uses a single query, and the procedure is repeated 100 times for an average result.
- Quantitative Evaluation: HP-net improves M-net's Top-1 accuracies by 3.6%, 5.0%, and 3.8% on CUHK03, VIPeR, and Market-1501, respectively.
- Qualitative Evaluation: Qualitative retrieval examples show HP-net improving M-net's rankings and obtaining correct results for a query person.HP-net attention maps locate T-shirt patterns that distinguish the query from identities with similar clothing.
7. Conclusion
The paper concludes that HydraPlus-Net uses multi-directional attention to learn attentive representations for fine-grained pedestrian analysis, and introduces PA-100K for further research.
- Ablation studies and experiments support HP-net's effectiveness for learning multi-level and multi-scale attentive features in attribute recognition and person re-identification.
- The paper introduces the large-scale PA-100K attribute dataset to facilitate pedestrian analysis tasks.