Source-linked AI summary
HAT: Hierarchical Aggregation Transformers for Person Re-identification
Guowen Zhang, Pingping Zhang, Jinqing Qi, Huchuan Lu
TL;DR
Person Re-ID must extract discriminative representations despite appearance variation and CNNs’ limited global modeling, while simple multi-level aggregation can underrepresent low-level features. HAT combines deeply supervised hierarchical aggregation with transformer-based feature calibration, and outperforms most state-of-the-art methods across four large-scale benchmarks.
Problem
Person Re-ID needs rich, discriminative, and robust features under variation, but CNNs and traditional feature aggregation provide limited global or low-level semantic information.
Method
HAT combines Deeply Supervised Aggregation for iterative hierarchical CNN-feature aggregation with Transformer-based Feature Calibration that integrates low-level detail as a global prior for high-level semantics.
Results
HAT achieves better performance than most state-of-the-art methods on four large-scale person Re-ID benchmarks.
Takeaways & Limitations
The framework supports multi-scale person retrieval by jointly preserving semantic information and detail information across hierarchical features.
Abstract
from arXiv · showhide
Recently, with the advance of deep Convolutional Neural Networks (CNNs), person Re-Identification (Re-ID) has witnessed great success in various applications. However, with limited receptive fields of CNNs, it is still challenging to extract discriminative representations in a global view for persons under non-overlapped cameras. Meanwhile, Transformers demonstrate strong abilities of modeling long-range dependencies for spatial and sequential data. In this work, we take advantages of both CNNs and Transformers, and propose a novel learning framework named Hierarchical Aggregation Transformer (HAT) for image-based person Re-ID with high performance. To achieve this goal, we first propose a Deeply Supervised Aggregation (DSA) to recurrently aggregate hierarchical features from CNN backbones. With multi-granularity supervisions, the DSA can enhance multi-scale features for person retrieval, which is very different from previous methods. Then, we introduce a Transformer-based Feature Calibration (TFC) to integrate low-level detail information as the global prior for high-level semantic information. The proposed TFC is inserted to each level of hierarchical features, resulting in great performance improvements. To our best knowledge, this work is the first to take advantages of both CNNs and Transformers for image-based person Re-ID. Comprehensive experiments on four large-scale Re-ID benchmarks demonstrate that our method shows better results than several state-of-the-art methods. The code is released at https://github.com/AI-Zhpp/HAT.
1 INTRODUCTION
Person Re-ID must extract discriminative features despite large image variations, but CNN-based aggregation can leave low-level features semantically weak. Transformers offer global-view modeling, motivating aggregation designs that better combine hierarchical information.
- Person Re-ID retrieves the same person across different cameras, places, and times despite occlusion, illumination, pose, and background clutter.
- CNN-based Re-ID methods explore feature aggregation to increase non-linearity, capacity, receptive field, and integration of multi-level features.
- Existing methods use skip connections, top-down feature pyramids, attention-based fusion, unified aggregation gates, and multi-branch structures to combine multi-scale features.
- Previous aggregation methods may underperform because low-level features contain less semantic information for person Re-ID.
- Transformers model global structural patterns, but pure-transformer Re-ID approaches are limited by pre-training data requirements and lack CNN properties such as hierarchical structure and invariance.
T Transformer-based Feature Calibration
HAT combines CNN hierarchical features with Transformer interactions to aggregate multi-scale information globally. Its DSA and TFC modules generate discriminative features by using multi-granularity supervision and cross-scale semantic-detail integration.
- HAT combines CNNs for hierarchical feature extraction with Transformers for global-view interaction across feature scales.
- Deeply Supervised Aggregation recurrently aggregates CNN backbone features and uses multi-granularity supervision to enhance discriminative multi-scale representations.
- Transformer-based Feature Calibration integrates cross-scale semantic information while preserving detail information from a global view.
- HAT is designed to aggregate multiscale features and is the first framework described here to combine CNNs and Transformers for image-based person Re-ID.
2 RELATED WORK
Related work covers discriminative feature learning, Transformer-based vision models, and hierarchical feature aggregation. It identifies a Re-ID-specific need to preserve high-level semantics while enriching low-level detail through adaptive, iterative aggregation.
- Person Re-ID methods use attention, body-part partitions, salient-region erasing, and semantic-detail fusion to learn discriminative representations.
- Vision Transformers have been applied across vision tasks, while person Re-ID work uses pure Transformers, part tokens, and related mechanisms to learn discriminative features.
- Hierarchical aggregation research includes deep architectures, auxiliary losses, feature pyramids, skip connections, cross-level correlation, and parallel subnetworks.
- Simple aggregation can perform poorly for Re-ID because shallow features contain less semantic information, making it difficult to preserve high-level semantics while adding low-level detail.
3 PROPOSED METHOD
HAT combines CNN-based hierarchical feature extraction with Transformer-based global aggregation for image-based person Re-ID. Its DSA progressively refines multi-level features with supervision, while TFC integrates detail and semantic information across scales.
- Overview: HAT uses a multi-scale feature extractor, DSA, and TFC in an end-to-end framework for hierarchical feature aggregation.The framework can use CNN backbones such as ResNet-50 and extracts features from Res2 through Res5.
- Transformer-based Feature Calibration: TFC integrates semantic and detail information from previous scales with current hierarchical features using Transformer-based global interaction.It is inserted at each hierarchical level and uses multi-head self-attention to capture long-range dependencies across scales.
- Transformer-based Feature Calibration: TFC converts hierarchical feature maps into patch sequences, adds a class token and positional embeddings, and processes them with self-attention and feed-forward layers.The resulting Transformer representation is reshaped and passed through neighborhood adjustment convolution layers to recover local information.
- Deeply Supervised Aggregation: DSA progressively aggregates shallower and deeper features with multi-granularity supervision to address the weak semantic information of low-level features.The design avoids direct concatenation without guidance and refines representations recurrently across the backbone hierarchy.
- Loss Functions: Training supervises the CNN backbones and Transformers with label-smoothed identification loss and hard triplet loss.Each identity is treated as a distinct class, while the triplet objective separates inter-class and intra-class distances.
4 EXPERIMENTS
Experiments on four large-scale benchmarks show that HAT is highly competitive, achieving best or near-best retrieval performance while its ablations validate the aggregation, supervision, and scaling choices.
- Comparison with State-of-the-arts: HAT achieves the best mAP on Market1501, with Rank-1 comparable to the best result from RGA-SC.The results support aggregating shallow-layer detail information with deeper-layer semantic information for retrieving hard samples.
- Comparison with State-of-the-arts: HAT achieves the best mAP/Rank-1 performance on DukeMTMC, exceeding state-of-the-art methods by at least 1.0% mAP and 0.3% Rank-1.The authors attribute this result to capturing more discriminative features.
- Comparison with State-of-the-arts: HAT achieves the best performance on CUHK03-NP, exceeding state-of-the-art methods by at least 2.2% mAP and 2.7% Rank-1 for labeled CUHK03.On detected CUHK03, it achieves the best mAP and second-best Rank-1.
- Comparison with State-of-the-arts: HAT ranks second in both mAP and Rank-1 on MSMT17, while remaining comparable to AAformer despite its stronger long-range modeling with a ViT backbone.MSMT17 contains complex lighting variations and spans a long period of time.
- Ablation Studies: The TFC aggregation methods improve performance by 5.3% mAP and 4.1% Rank-1 in the DukeMTMC ablation.The ablations also indicate that deeper features need more Transformers, while accuracy tends to converge as Transformer depth increases.
- Ablation Studies: Removing MFE supervision decreases HAT accuracy by 4.9% mAP and 2.2% Rank-1, while auxiliary losses improve it by 1.0% mAP and 0.9% Rank-1.The auxiliary losses strengthen semantic information, gradient propagation, and regularization; Neighborhood Adjustment further improves the framework.
5 CONCLUSION
HAT combines cross-level feature calibration with deeply supervised aggregation to fuse semantic and detail information for image-based person Re-ID. Across four large-scale benchmarks, it achieves better performance than most state-of-the-art methods.
- HAT’s TFC merges semantic and detail information across feature levels by using low-level details as a global prior for high-level semantics.This feature calibration is part of the proposed framework alongside DSA.
- DSA recurrently aggregates hierarchical CNN features with multiple losses to enhance multi-scale representations.
- HAT achieves better performance than most state-of-the-art methods on four large-scale person Re-ID benchmarks.