AAformer: Auto-Aligned Transformer for Person Re-Identification

Kuan Zhu; Haiyun Guo; Shiliang Zhang; Yaowei Wang; Jing Liu; Jinqiao Wang; Ming Tang

Source-linked AI summary

AAformer: Auto-Aligned Transformer for Person Re-Identification

Kuan Zhu, Haiyun Guo, Shiliang Zhang, Yaowei Wang, Jing Liu, Jinqiao Wang, Ming Tang

arXiv:2104.00921v3Apr 2, 2021cs.CV

TL;DR

Person re-identification methods often align human parts coarsely or miss identifiable nonhuman parts. AAformer introduces [PART] tokens and OT-based online patch grouping within Transformer self-attention, and experiments validate the approach across standard benchmarks.

Problem
Existing person re-ID methods can align human parts coarsely or fail to locate identifiable nonhuman parts such as knapsacks.
Source: §I · p. 1
Method
AAformer uses learnable [PART] tokens as prototypes and fast Optimal Transport to online group patches into human and nonhuman part subsets within self-attention.
Source: §I · p. 2, B. Auto-Aligned Transformer · p. 4
Results
AAformer is reported superior to various state-of-the-art methods on Market-1501, CUHK03, DukeMTMC-reID, and MSMT17.
Source: §I · p. 2
Takeaways & Limitations
The output [PART] tokens can be directly used as part features for person re-identification, while visualizations show attention to human and nonhuman parts.
Source: §I · p. 2, C. Ablation Studies · p. 8

Abstract

from arXiv · show

In person re-identification (re-ID), extracting part-level features from person images has been verified to be crucial to offer fine-grained information. Most of the existing CNN-based methods only locate the human parts coarsely, or rely on pretrained human parsing models and fail in locating the identifiable nonhuman parts (e.g., knapsack). In this article, we introduce an alignment scheme in transformer architecture for the first time and propose the auto-aligned transformer (AAformer) to automatically locate both the human parts and nonhuman ones at patch level. We introduce the "Part tokens ([PART]s)", which are learnable vectors, to extract part features in the transformer. A [PART] only interacts with a local subset of patches in self-attention and learns to be the part representation. To adaptively group the image patches into different subsets, we design the auto-alignment. Auto-alignment employs a fast variant of optimal transport (OT) algorithm to online cluster the patch embeddings into several groups with the [PART]s as their prototypes. AAformer integrates the part alignment into the self-attention and the output [PART]s can be directly used as part features for retrieval. Extensive experiments validate the effectiveness of [PART]s and the superiority of AAformer over various state-of-the-art methods.

I. INTRODUCTION

AAformer addresses coarse human-part alignment and missed nonhuman cues in person re-identification by introducing patch-level adaptive alignment in a Transformer.

Person re-identification needs fine-grained part features because detection, pose, and viewpoint changes cause body-part misalignment.Source: §I · p. 1
Existing Transformer re-ID methods had not yet addressed part alignment, while CNN-based approaches often align parts coarsely or miss identifiable nonhuman items.Source: §I · pp. 1–2
[PART]s are learnable vectors that attend only to local patch subsets and learn their representations as part features.Source: §I · p. 2
Auto-alignment uses fast Optimal Transport to cluster patches online around [PART] prototypes, covering both human and nonhuman parts.Source: §I · p. 2
AAformer’s experiments validate [PART]s and report superiority over state-of-the-art methods on Market-1501, CUHK03, DukeMTMC-reID, and MSMT17.Source: §I · p. 2

II. RELATED WORK

Prior aligned person re-ID methods use fixed stripes, rectangular regions, external semantics, or attention, each with limitations in precision, coverage, or consistency.

Stripe-based approaches

Stripe-based methods partition person images into fixed horizontal regions, but their coarse, fixed geometry limits part localization.Source: A. Aligned Person Re-identification · p. 2

Bounding box-based Approaches

Bounding-box methods locate latent parts, but their rectangular shapes constrain the geometry of extracted regions.Source: A. Aligned Person Re-identification · p. 2

Extra semantics-based methods

Extra-semantics methods depend on pretrained parsing or pose models and can ignore identifiable belongings such as knapsacks and reticules.Source: A. Aligned Person Re-identification · pp. 2–3

Attention-based methods

Attention-based methods enhance discriminative regions but do not explicitly locate semantic parts or guarantee cross-image semantic consistency.Source: A. Aligned Person Re-identification · p. 3
AAformer locates human and nonhuman parts at patch level while maintaining semantic consistency, unlike the described alternatives.Source: A. Aligned Person Re-identification · p. 3

B. Visual Transformer

Visual Transformers represent images as patch sequences and extend Transformer tokenization with learnable grouping mechanisms, including AAformer’s OT-based assignment.

ViT divides an image into patches, linearly maps them to vectors, adds a learnable class token, and processes the sequence with a Transformer encoder.Source: B. Visual Transformer · p. 3
GroupViT groups patches toward learnable group tokens using Gumbel Softmax, whereas AAformer assigns patches to [PART]s with Optimal Transport.Source: B. Visual Transformer · p. 3
AAformer’s OT assignment is designed to avoid the trivial solution in which all patches are assigned to one [PART].Source: B. Visual Transformer · p. 3

A. The Main Architecture

AAformer builds on the standard patch-based Vision Transformer architecture, adding [PART] representations and replacing self-attention for them with auto-alignment.

The input image is reshaped into flattened patches, linearly projected into D-dimensional embeddings, combined with a class token and position embeddings, then encoded by a Transformer.Source: A. The Main Architecture · p. 3
Standard self-attention computes query-key affinities, applies scaled softmax weighting, and combines value vectors across the sequence.Source: A. The Main Architecture · p. 3
AAformer adds learnable [PART] tokens alongside the class token to represent local parts and replaces the MSA layer with Multi-head Auto-Alignment.Source: Figure 2 · p. 4, A. The Main Architecture · p. 4
Auto-alignment operates in parallel across attention heads, concatenates their outputs, and uses the resulting [PART]s for person re-identification.Source: Figure 2 · p. 4, A. The Main Architecture · p. 4

B. Auto-Aligned Transformer

AAformer uses [PART]s and optimal-transport auto-alignment to assign patches into mutually exclusive part groups, integrating alignment into self-attention and producing adaptive part representations.

[PART]s are learnable vectors that attend only to assigned local patch subsets, converting each subset into a part representation.Source: B. Auto-Aligned Transformer · p. 4
Auto-alignment clusters patch embeddings into P groups using [PART] query vectors as prototypes, including human and nonhuman parts.Source: B. Auto-Aligned Transformer · p. 4
Optimal Transport constraints prevent trivial assignments by enforcing that each [PART] is selected at least N/P times on average per image.Source: B. Auto-Aligned Transformer · p. 5
A fast Sinkhorn-Knopp variant computes assignments efficiently; grouping 576 patches into 5 [PART]s takes 0.46ms.Source: B. Auto-Aligned Transformer · p. 5
Multi-head Auto-Alignment runs across attention heads, concatenates their outputs, and uses the resulting [PART]s for person re-ID.Source: B. Auto-Aligned Transformer · p. 5
AAformer performs alignment and self-attention in every layer, jointly learning part assignments and representations online and end-to-end.Source: B. Auto-Aligned Transformer · p. 5
Initial [PART]s become dataset-adaptive prototypes, while output [PART]s become instance-adaptive part features for input-image retrieval.Source: B. Auto-Aligned Transformer · p. 5, Figure 3 · p. 5

C. Objective Function

AAformer trains global and part representations with identity classification and separately mined triplet objectives, then concatenates CLS and [PART]s for testing.

The output CLS token supplies the global feature, while output [PART]s supply part features for the re-ID heads.Source: C. Objective Function · p. 5
Cross-entropy loss trains separate classifiers for the tokens, with label smoothing applied to improve performance.Source: C. Objective Function · p. 5
The concatenated [PART]s are used to calculate the part triplet loss alongside the global triplet loss from CLS.Source: C. Objective Function · p. 5
Hard sample mining selects the least similar positive and most similar negative samples for each image.Source: C. Objective Function · p. 5
Global and part features can identify different hardest pairs, so their triplet losses are calculated separately.Source: C. Objective Function · pp. 5–6
During testing, CLS and [PART]s are concatenated to represent each person image.Source: C. Objective Function · p. 6

IV. EXPERIMENTS

The experiments compare AAformer with state-of-the-art methods on holistic and occluded re-ID benchmarks and use ablations to assess its components and design choices.

Experiments evaluate AAformer against state-of-the-art methods on holistic and occluded person re-ID benchmarks.Source: §IV · p. 6
Ablations examine [PART]s, AAformer, Optimal Transport versus Nearest Neighbor, and AAformer visualizations.Source: §IV · p. 6
The authors state that the reported results validate the effectiveness of the proposed method.Source: §IV · p. 6

A. Implementation Details and Datasets

AAformer is implemented with a ViT-Base backbone and standard re-ID preprocessing, optimization, datasets, and CMC/mAP evaluation protocols.

AAformer uses ViT-Base with 12 Transformer layers, hidden size D=768, an MLP four times wider, and 12 attention heads.Source: A. Implementation Details and Datasets · p. 6
Images are resized to 256×128, divided into 16×16 patches, and augmented with random cropping, flipping, and random erasing at probability 0.5.Source: A. Implementation Details and Datasets · p. 6
Training uses 10-epoch linear warmup, cosine decay, 120 fine-tuning epochs, batch size 64, SGD momentum 0.9, and weight decay 1×10^-4.Source: A. Implementation Details and Datasets · p. 6
Experiments use DukeMTMC-reID, Market-1501, CUHK03 under the New Protocol, and MSMT17 benchmarks.Source: A. Implementation Details and Datasets · p. 6
Evaluation follows standard training/test ID splits with CMC and mAP metrics, using Euclidean feature distances.Source: A. Implementation Details and Datasets · p. 6

B. Comparison with State-of-the-art Methods

AAformer is compared with state-of-the-art methods on holistic and occluded person re-ID benchmarks. It achieves leading performance across the reported datasets, with especially large gains on CUHK03, MSMT17, and Occluded-DukeMTMC.

Holistic benchmarks

AAformer obtains the best DukeMTMC-reID results, exceeding other methods by at least 0.5% in Rank-1 accuracy and 0.8% in mAP accuracy.Source: B. Comparison with State-of-the-art… · p. 6
AAformer achieves the second-best Market-1501 results, remaining only slightly behind the first-place methods in a nearly saturated dataset.Source: B. Comparison with State-of-the-art… · p. 6
On CUHK03, AAformer achieves the best performance on both labeled and detected sets, outperforming the second-best methods by up to 4.0% in mAP.The reported gains are 1.4%/2.1% on labeled and 1.9%/4.0% on detected sets for Rank-1/mAP, respectively.Source: B. Comparison with State-of-the-art… · p. 6
On MSMT17, AAformer surpasses the second-best method by 1.3% in Rank-1 and 1.6% in mAP.The comparison removes side information for fairness.Source: B. Comparison with State-of-the-art… · p. 6

Occluded benchmark

On Occluded-DukeMTMC, AAformer sets the new state-of-the-art, outperforming other methods by at least 2.6% in Rank-1 and 4.6% in mAP.The [PART]s can adaptively focus on visible areas in occluded images and extract discriminative features from them.Source: B. Comparison with State-of-the-art… · pp. 6–7

C. Ablation Studies

The ablations evaluate [PART]s, patch-assignment strategies, and part granularity, showing benefits from layer-wise local extraction, constrained OT grouping, and adaptive part counts.

Effectiveness of [PART]s

[PART]s with MGN’s partitioning surpass the baseline by 2.8%/4.5% in Rank-1/mAP on MSMT17.Source: C. Ablation Studies · p. 7
[PART]s extract local features through self-attention in every Transformer layer, whereas part pooling obtains local features only through the last layer.Source: C. Ablation Studies · pp. 7–8
Adding more CLS tokens only slightly improves performance because their information is similar and redundant, separating [PART] gains from feature-dimension increases.Source: C. Ablation Studies · p. 8

Effectiveness of AAformer

Optimal Transport outperforms Nearest Neighbor because unconstrained nearest assignment tends to group most patch embeddings into the same [PART].Source: C. Ablation Studies · p. 8

Different numbers of [PART]s

The {2, 3} granularity setting usually obtains the best accuracy, while increasing the number of [PART]s is not uniformly beneficial.Source: C. Ablation Studies · p. 8

Visualization of AAformer

AAformer’s [PART]s focus on human and nonhuman parts, handle pose variation and occlusion, suppress background responses, and maintain semantic consistency across images.Source: C. Ablation Studies · p. 8

V. CONCLUSION

AAformer introduces [PART]s and integrates part localization into Transformer self-attention, using Optimal Transport to align human and nonhuman parts online. The method’s experiments validate [PART] effectiveness and AAformer’s superiority over many state-of-the-art methods.

[PART]s are learnable Transformer vectors that extract part-level features by interacting with local patch subsets.Source: §V · p. 8
AAformer integrates localization into self-attention and uses Optimal Transport to online align human and nonhuman parts without extra semantics.Source: §V · p. 9
Experiments validate the effectiveness of [PART]s and AAformer’s superiority over many state-of-the-art methods.Source: §V · p. 9