PaperlayerView exact revision on arXiv

Source-linked AI summary

TransReID: Transformer-based Object Re-Identification

Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, Wei Jiang

arXiv:2102.04378v2cs.CV

TL;DR

Robust feature representation is difficult in ReID because CNNs focus on local neighborhoods and lose details through downsampling. TransReID uses a pure transformer with JPM and SIE to address these limitations, achieving state-of-the-art performance on person and vehicle benchmarks. Its design also exposes a remaining limitation: local patch segments may not fully exploit global dependencies.

  • Problem

    CNN-based ReID methods process local neighborhoods and can lose detail through convolution and downsampling, limiting robust feature representation.

  • Method

    TransReID encodes images as patch sequences and combines a pure-transformer baseline with JPM patch rearrangement and learnable SIE for camera and viewpoint clues.

  • Results

    TransReID achieves state-of-the-art performance on several person and vehicle ReID benchmarks.

  • Takeaways & Limitations

    The results support further exploration of transformer-based networks for ReID.

  • Takeaways & Limitations

    Local patch segments may not fully exploit global dependencies because each segment considers only part of the continuous patch sequence.

Abstract

from arXiv · show

Extracting robust feature representation is one of the key challenges in object re-identification (ReID). Although convolution neural network (CNN)-based methods have achieved great success, they only process one local neighborhood at a time and suffer from information loss on details caused by convolution and downsampling operators (e.g. pooling and strided convolution). To overcome these limitations, we propose a pure transformer-based object ReID framework named TransReID. Specifically, we first encode an image as a sequence of patches and build a transformer-based strong baseline with a few critical improvements, which achieves competitive results on several ReID benchmarks with CNN-based methods. To further enhance the robust feature learning in the context of transformers, two novel modules are carefully designed. (i) The jigsaw patch module (JPM) is proposed to rearrange the patch embeddings via shift and patch shuffle operations which generates robust features with improved discrimination ability and more diversified coverage. (ii) The side information embeddings (SIE) is introduced to mitigate feature bias towards camera/view variations by plugging in learnable embeddings to incorporate these non-visual clues. To the best of our knowledge, this is the first work to adopt a pure transformer for ReID research. Experimental results of TransReID are superior promising, which achieve state-of-the-art performance on both person and vehicle ReID benchmarks.

1. Introduction

TransReID addresses limitations of CNN-based ReID by using a pure transformer with global receptive fields and preserved details. It adds JPM and SIE to improve robust feature learning under object, camera, and viewpoint variation.

  • Object ReID associates objects across scenes and camera views, while robust and discriminative feature extraction remains central to the task.
  • CNN-based ReID methods emphasize local regions, struggle to extract diversified discriminative parts, and lose detail through downsampling.
  • Transformer models use multi-head self-attention to capture long-range dependencies and avoid downsampling, preserving detailed information such as backpack features.
  • TransReID is a pure-transformer ReID framework with a strong baseline, JPM for shift-and-shuffle patch rearrangement, and SIE for learnable incorporation of non-visual clues.
  • The paper presents pure transformers as a first investigation in object ReID and reports state-of-the-art performance on person and vehicle ReID benchmarks.

2. Related Work

Related work covers CNN-based ReID pipelines, fine-grained local-part features, side-information modeling, and the development of transformers for vision.

  • CNN-based ReID commonly trains a CNN backbone with cross-entropy identity loss and triplet loss, with BNNeck combining these objectives.
  • Fine-grained Features: Fine-grained ReID methods aggregate local information using horizontal stripes, semantic parsing, or keypoint-based alignment for person and vehicle matching.
  • Side Information: Side-information methods use camera IDs or viewpoint information to learn invariant features despite cross-camera differences in pose, orientation, illumination, and resolution.
  • Transformer: Transformers originated in sequential-data modeling for NLP and were later surveyed and applied across computer-vision tasks.
  • Pure Transformer: Pure vision transformers apply transformers directly to image-patch sequences, while ViT requires large-scale pretraining to overcome its data demands.

3. Methodology

TransReID builds a pure transformer ReID framework with overlapping patch extraction, global and local feature learning, JPM patch rearrangement, and SIE for camera/viewpoint information. The modules are jointly trained to improve robust feature representation under object variation.

  • 3.1. Transformer-based strong baseline: The framework adapts transformer-based image classification for ReID with overlapping patches, learnable position embeddings, and ID plus triplet supervision.Overlapping patches preserve neighboring structures, while the transformer provides global receptive fields without downsampling.
  • 3.1. Transformer-based strong baseline: The transformer prepends a learnable [cls] token and uses its output as a global feature, with patch embeddings and position embeddings forming the input sequence.A linear projection maps patches to D dimensions before transformer processing.
  • 3.2. Jigsaw Patch Module: JPM shifts and shuffles patch embeddings, regroups them into several groups, and feeds the groups through a shared transformer layer to learn local features.The shuffled groups can cover patches from different body or vehicle parts, while a parallel standard branch retains global features.
  • 3.2. Jigsaw Patch Module: Global and local features are jointly supervised with ID and triplet losses, then concatenated during inference as the final representation.Using only the global feature reduces computation with slight performance degradation.
  • 3.3. Side Information Embeddings: SIE adds learnable camera and viewpoint embeddings to the transformer input, jointly encoding the two side-information types to reduce camera- or viewpoint-related feature bias.The side-information embedding is shared across patches for an image, and λ balances its contribution.

4. Experiments

Experiments across person and vehicle ReID benchmarks show that TransReID’s transformer baseline offers competitive speed–accuracy trade-offs, while JPM and SIE improve robustness and final performance. The complete framework achieves strong benchmark results across multiple datasets and comparisons.

  • Experimental setup: TransReID is evaluated on four person ReID datasets and two vehicle ReID datasets using CMC curves and mAP.The datasets are Market-1501, DukeMTMC-reID, MSMT17, Occluded-Duke, VeRi-776, and VehicleID.
  • Transformer baseline: ViT-B/16s=12 achieves a better speed-accuracy trade-off than ResNeSt200, while reducing the sliding-window step improves performance at higher inference cost.ViT-B/16s=12 is faster than ResNeSt200 (2.81x vs 3.12x) and performs slightly better on ReID benchmarks.
  • JPM ablation: +2.6% mAP and +1.0% mAP are obtained by JPM over baseline on MSMT17 and VeRi-776, respectively.The shift and shuffle operations add a further +0.5% mAP and +0.2% mAP over JPM without rearrangement on the same datasets.
  • SIE ablation: +1.4% mAP and +0.1% rank-1 accuracy are obtained on MSMT17 when SIE encodes camera IDs alone.On VeRi-776, encoding both camera IDs and viewpoints reaches 79.6% mAP, compared with 78.5% mAP when viewpoint information is encoded alone.
  • Overall ablation: 64.9% mAP on MSMT17 and 80.6% mAP on VeRi-776 are achieved by TransReID, improving over baseline by +3.9% and +2.4% mAP, respectively.JPM and SIE contribute separately before their combined use in the final framework.
  • Benchmark comparison: TransReID∗ outperforms prior methods by +5.5%/+2.1% mAP on MSMT17/DukeMTMC-reID and reaches 85.2% Rank-1 accuracy on VehicleID.It also achieves 55.6% mAP on Occluded ReID and 82.3% mAP on VeRi-776, while running 4.8% faster than MGN in the reported comparison.

5. Conclusion

TransReID is a pure-transformer ReID framework with JPM and SIE, achieving state-of-the-art performance across person and vehicle benchmarks.

  • TransReID combines a pure transformer framework with jigsaw patch and side information embedding modules for object ReID.JPM and SIE are proposed as the framework’s two novel modules.
  • TransReID outperforms other state-of-the-art methods by a large margin on several person and vehicle ReID datasets.The reported datasets include MSMT17, Market-1501, DukeMTMC-reID, Occluded-Duke, VeRi-776, and VehicleID.
  • The results support further exploration of transformer-based networks for ReID with stronger representation power and lower computational cost.The paper frames this as a prospective direction based on TransReID’s results and experience from CNN-based methods.

A.1. Study on Transformer-based Strong Baseline

The strong transformer baseline is highly sensitive to training choices, with position embeddings, stochastic depth, and soft triplet loss improving performance.

  • The baseline experiments analyze initialization, hyperparameters, optimizer choice, network configuration, regularization, and loss functions.Ablation studies evaluate training-setting variants on MSMT17 and VeRi-776.
  • Position embeddings are crucial: removing them decreases mAP by 38.6% on MSMT17 and 10.2% on VeRi-776.The passage describes position embeddings as a key ingredient of the training procedure.
  • Stochastic depth boosts mAP by about 1% and facilitates transformer convergence, especially in deep models.Dropout and attention dropout instead reduce performance in the reported experiments.
  • Soft triplet loss improves MSMT17 mAP by 0.7% over regular triplet loss.The best reported combination is soft triplet loss with cross-entropy loss without label smoothing.

A.2. More Ablation Studies of JPM and SIE

Ablations show that JPM and SIE consistently improve transformer baselines, while rearrangement and complementary camera-view information strengthen their benefits.

  • JPM: JPM performs better with rearrangement, indicating that shift and patch shuffle help learn discriminative features robust to perturbations.The comparison includes rearranged and non-rearranged patch schemes.
  • JPM: JPM consistently improves both ViT-B/16 and the stronger ViT-B/16s=12 baselines.The improvement is reported regardless of baseline strength.
  • SIE: Combining viewpoint and camera information produces the best performance because the two sources are complementary.The viewpoint experiments are conducted on VeRi-776.
  • SIE: SIE consistently improves the ViT-B/16 and ViT-B/16s=12 baselines.The ablation compares inclusion and exclusion of side information.

B. Analysis on Rearranging Patches in JPM

JPM rearrangement broadens local attention beyond contiguous patch groups, helping local branches capture long-range dependencies rather than isolated receptive fields.

  • Directly splitting patch embeddings into contiguous groups can make each local part focus on only a limited spatial region.The last-layer embeddings retain spatial location information, so isolated subsequences may correspond to localized image areas.
  • JPM visualizes local-feature attention weights averaged across heads in the transformer’s last layer; brighter regions indicate higher weights.Faces are masked for anonymization in the visualization.
  • Without rearrangement, separate JPM parts tend to attend to limited receptive fields, such as the head or bottom of a person.The global sequence is split into isolated subsequences in this setting.
  • With rearrangement, JPM captures long-range dependencies across the image.The comparison is made between JPM with and without patch-sequence rearrangement.

C. More Visualization of Attention Maps

Grad-CAM visualizations compare attention maps across CNN-based, attention-augmented CNN, transformer-based, and TransReID variants. The results indicate that transformer methods capture global context and discriminative parts, with these properties further enhanced by TransReID.

  • Visualization findings: Transformer-based methods capture global context information and more discriminative parts than the compared CNN-based approaches.The visualization results support the same conclusion as the main-paper attention-map analysis.
  • Visualization findings: These attention characteristics are further enhanced in TransReID for better performance.The comparison includes TransReID without rearrangement and the full TransReID model.
  • Visualization setup: Figure 9 compares Grad-CAM attention maps for CNN-based methods, CNN+Attention methods, a transformer baseline, and TransReID variants.The TransReID variants include versions with and without patch rearrangement.
Loading 2102.04378v2…