Source-linked AI summary
AAformer: Auto-Aligned Transformer for Person Re-Identification
Kuan Zhu, Haiyun Guo, Shiliang Zhang, Yaowei Wang, Jing Liu, Jinqiao Wang, Ming Tang
TL;DR
Person re-identification methods often align human parts coarsely or miss identifiable nonhuman parts. AAformer introduces [PART] tokens and OT-based online patch grouping within Transformer self-attention, and experiments validate the approach across standard benchmarks.
Problem
Existing person re-ID methods can align human parts coarsely or fail to locate identifiable nonhuman parts such as knapsacks.
Method
AAformer uses learnable [PART] tokens as prototypes and fast Optimal Transport to online group patches into human and nonhuman part subsets within self-attention.
Results
AAformer is reported superior to various state-of-the-art methods on Market-1501, CUHK03, DukeMTMC-reID, and MSMT17.
Takeaways & Limitations
The output [PART] tokens can be directly used as part features for person re-identification, while visualizations show attention to human and nonhuman parts.
Abstract
from arXiv · showhide
In person re-identification (re-ID), extracting part-level features from person images has been verified to be crucial to offer fine-grained information. Most of the existing CNN-based methods only locate the human parts coarsely, or rely on pretrained human parsing models and fail in locating the identifiable nonhuman parts (e.g., knapsack). In this article, we introduce an alignment scheme in transformer architecture for the first time and propose the auto-aligned transformer (AAformer) to automatically locate both the human parts and nonhuman ones at patch level. We introduce the "Part tokens ([PART]s)", which are learnable vectors, to extract part features in the transformer. A [PART] only interacts with a local subset of patches in self-attention and learns to be the part representation. To adaptively group the image patches into different subsets, we design the auto-alignment. Auto-alignment employs a fast variant of optimal transport (OT) algorithm to online cluster the patch embeddings into several groups with the [PART]s as their prototypes. AAformer integrates the part alignment into the self-attention and the output [PART]s can be directly used as part features for retrieval. Extensive experiments validate the effectiveness of [PART]s and the superiority of AAformer over various state-of-the-art methods.
I. INTRODUCTION
AAformer addresses coarse human-part alignment and missed nonhuman cues in person re-identification by introducing patch-level adaptive alignment in a Transformer.
- Person re-identification needs fine-grained part features because detection, pose, and viewpoint changes cause body-part misalignment.
- Existing Transformer re-ID methods had not yet addressed part alignment, while CNN-based approaches often align parts coarsely or miss identifiable nonhuman items.
- [PART]s are learnable vectors that attend only to local patch subsets and learn their representations as part features.
- Auto-alignment uses fast Optimal Transport to cluster patches online around [PART] prototypes, covering both human and nonhuman parts.
- AAformer’s experiments validate [PART]s and report superiority over state-of-the-art methods on Market-1501, CUHK03, DukeMTMC-reID, and MSMT17.
II. RELATED WORK
Prior aligned person re-ID methods use fixed stripes, rectangular regions, external semantics, or attention, each with limitations in precision, coverage, or consistency.
- Stripe-based approaches: Stripe-based methods partition person images into fixed horizontal regions, but their coarse, fixed geometry limits part localization.
- Bounding box-based Approaches: Bounding-box methods locate latent parts, but their rectangular shapes constrain the geometry of extracted regions.
- Extra semantics-based methods: Extra-semantics methods depend on pretrained parsing or pose models and can ignore identifiable belongings such as knapsacks and reticules.
- Attention-based methods: Attention-based methods enhance discriminative regions but do not explicitly locate semantic parts or guarantee cross-image semantic consistency.
- Attention-based methods: AAformer locates human and nonhuman parts at patch level while maintaining semantic consistency, unlike the described alternatives.
B. Visual Transformer
Visual Transformers represent images as patch sequences and extend Transformer tokenization with learnable grouping mechanisms, including AAformer’s OT-based assignment.
- ViT divides an image into patches, linearly maps them to vectors, adds a learnable class token, and processes the sequence with a Transformer encoder.
- GroupViT groups patches toward learnable group tokens using Gumbel Softmax, whereas AAformer assigns patches to [PART]s with Optimal Transport.
- AAformer’s OT assignment is designed to avoid the trivial solution in which all patches are assigned to one [PART].
A. The Main Architecture
AAformer builds on the standard patch-based Vision Transformer architecture, adding [PART] representations and replacing self-attention for them with auto-alignment.
- The input image is reshaped into flattened patches, linearly projected into D-dimensional embeddings, combined with a class token and position embeddings, then encoded by a Transformer.
- Standard self-attention computes query-key affinities, applies scaled softmax weighting, and combines value vectors across the sequence.
- AAformer adds learnable [PART] tokens alongside the class token to represent local parts and replaces the MSA layer with Multi-head Auto-Alignment.
- Auto-alignment operates in parallel across attention heads, concatenates their outputs, and uses the resulting [PART]s for person re-identification.
B. Auto-Aligned Transformer
AAformer uses [PART]s and optimal-transport auto-alignment to assign patches into mutually exclusive part groups, integrating alignment into self-attention and producing adaptive part representations.
- [PART]s are learnable vectors that attend only to assigned local patch subsets, converting each subset into a part representation.
- Auto-alignment clusters patch embeddings into P groups using [PART] query vectors as prototypes, including human and nonhuman parts.
- Optimal Transport constraints prevent trivial assignments by enforcing that each [PART] is selected at least N/P times on average per image.
- A fast Sinkhorn-Knopp variant computes assignments efficiently; grouping 576 patches into 5 [PART]s takes 0.46ms.
- Multi-head Auto-Alignment runs across attention heads, concatenates their outputs, and uses the resulting [PART]s for person re-ID.
- AAformer performs alignment and self-attention in every layer, jointly learning part assignments and representations online and end-to-end.
- Initial [PART]s become dataset-adaptive prototypes, while output [PART]s become instance-adaptive part features for input-image retrieval.
C. Objective Function
AAformer trains global and part representations with identity classification and separately mined triplet objectives, then concatenates CLS and [PART]s for testing.
- The output CLS token supplies the global feature, while output [PART]s supply part features for the re-ID heads.
- Cross-entropy loss trains separate classifiers for the tokens, with label smoothing applied to improve performance.
- The concatenated [PART]s are used to calculate the part triplet loss alongside the global triplet loss from CLS.
- Hard sample mining selects the least similar positive and most similar negative samples for each image.
- Global and part features can identify different hardest pairs, so their triplet losses are calculated separately.
- During testing, CLS and [PART]s are concatenated to represent each person image.
IV. EXPERIMENTS
The experiments compare AAformer with state-of-the-art methods on holistic and occluded re-ID benchmarks and use ablations to assess its components and design choices.
- Experiments evaluate AAformer against state-of-the-art methods on holistic and occluded person re-ID benchmarks.
- Ablations examine [PART]s, AAformer, Optimal Transport versus Nearest Neighbor, and AAformer visualizations.
- The authors state that the reported results validate the effectiveness of the proposed method.
A. Implementation Details and Datasets
AAformer is implemented with a ViT-Base backbone and standard re-ID preprocessing, optimization, datasets, and CMC/mAP evaluation protocols.
- AAformer uses ViT-Base with 12 Transformer layers, hidden size D=768, an MLP four times wider, and 12 attention heads.
- Images are resized to 256×128, divided into 16×16 patches, and augmented with random cropping, flipping, and random erasing at probability 0.5.
- Training uses 10-epoch linear warmup, cosine decay, 120 fine-tuning epochs, batch size 64, SGD momentum 0.9, and weight decay 1×10^-4.
- Experiments use DukeMTMC-reID, Market-1501, CUHK03 under the New Protocol, and MSMT17 benchmarks.
- Evaluation follows standard training/test ID splits with CMC and mAP metrics, using Euclidean feature distances.
B. Comparison with State-of-the-art Methods
AAformer is compared with state-of-the-art methods on holistic and occluded person re-ID benchmarks. It achieves leading performance across the reported datasets, with especially large gains on CUHK03, MSMT17, and Occluded-DukeMTMC.
- Holistic benchmarks: AAformer obtains the best DukeMTMC-reID results, exceeding other methods by at least 0.5% in Rank-1 accuracy and 0.8% in mAP accuracy.
- Holistic benchmarks: AAformer achieves the second-best Market-1501 results, remaining only slightly behind the first-place methods in a nearly saturated dataset.
- Holistic benchmarks: On CUHK03, AAformer achieves the best performance on both labeled and detected sets, outperforming the second-best methods by up to 4.0% in mAP.The reported gains are 1.4%/2.1% on labeled and 1.9%/4.0% on detected sets for Rank-1/mAP, respectively.
- Holistic benchmarks: On MSMT17, AAformer surpasses the second-best method by 1.3% in Rank-1 and 1.6% in mAP.The comparison removes side information for fairness.
- Occluded benchmark: On Occluded-DukeMTMC, AAformer sets the new state-of-the-art, outperforming other methods by at least 2.6% in Rank-1 and 4.6% in mAP.The [PART]s can adaptively focus on visible areas in occluded images and extract discriminative features from them.
C. Ablation Studies
The ablations evaluate [PART]s, patch-assignment strategies, and part granularity, showing benefits from layer-wise local extraction, constrained OT grouping, and adaptive part counts.
- Effectiveness of [PART]s: [PART]s with MGN’s partitioning surpass the baseline by 2.8%/4.5% in Rank-1/mAP on MSMT17.
- Effectiveness of [PART]s: [PART]s extract local features through self-attention in every Transformer layer, whereas part pooling obtains local features only through the last layer.
- Effectiveness of [PART]s: Adding more CLS tokens only slightly improves performance because their information is similar and redundant, separating [PART] gains from feature-dimension increases.
- Effectiveness of AAformer: Optimal Transport outperforms Nearest Neighbor because unconstrained nearest assignment tends to group most patch embeddings into the same [PART].
- Different numbers of [PART]s: The {2, 3} granularity setting usually obtains the best accuracy, while increasing the number of [PART]s is not uniformly beneficial.
- Visualization of AAformer: AAformer’s [PART]s focus on human and nonhuman parts, handle pose variation and occlusion, suppress background responses, and maintain semantic consistency across images.
V. CONCLUSION
AAformer introduces [PART]s and integrates part localization into Transformer self-attention, using Optimal Transport to align human and nonhuman parts online. The method’s experiments validate [PART] effectiveness and AAformer’s superiority over many state-of-the-art methods.
- [PART]s are learnable Transformer vectors that extract part-level features by interacting with local patch subsets.
- AAformer integrates localization into self-attention and uses Optimal Transport to online align human and nonhuman parts without extra semantics.
- Experiments validate the effectiveness of [PART]s and AAformer’s superiority over many state-of-the-art methods.