Source-linked AI summary
NFormer: Robust Person Re-identification with Neighbor Transformer
Haochen Wang, Jiayi Shen, Yongtuo Liu, Yan Gao, Efstratios Gavves
TL;DR
Person re-identification must handle large appearance variation, yet most methods learn representations from single images and can produce outlier features. NFormer models relations among all input images with efficient landmark-based affinity computation and sparse neighbor attention, achieving state-of-the-art performance on four datasets.
Problem
Most Re-ID methods learn from single images despite high intra-identity variation across cameras and conditions, which can produce unstable matching and outlier features.
Method
NFormer models relations among all input images using Landmark Agent Attention for efficient low-rank affinity computation and Reciprocal Neighbor Softmax for sparse neighbor attention.
Results
NFormer achieves a new state-of-the-art across four person Re-ID datasets and further boosts other state-of-the-art methods when combined with them.
Takeaways & Limitations
NFormer provides robust, discriminative representations that scale to large inputs and are easy to combine with third-party methods.
Takeaways & Limitations
NFormer expects a large enough number of images of the same person and is less relevant in smaller setups.
Abstract
from arXiv · showhide
Person re-identification aims to retrieve persons in highly varying settings across different cameras and scenarios, in which robust and discriminative representation learning is crucial. Most research considers learning representations from single images, ignoring any potential interactions between them. However, due to the high intra-identity variations, ignoring such interactions typically leads to outlier features. To tackle this issue, we propose a Neighbor Transformer Network, or NFormer, which explicitly models interactions across all input images, thus suppressing outlier features and leading to more robust representations overall. As modelling interactions between enormous amount of images is a massive task with lots of distractors, NFormer introduces two novel modules, the Landmark Agent Attention, and the Reciprocal Neighbor Softmax. Specifically, the Landmark Agent Attention efficiently models the relation map between images by a low-rank factorization with a few landmarks in feature space. Moreover, the Reciprocal Neighbor Softmax achieves sparse attention to relevant -- rather than all -- neighbors only, which alleviates interference of irrelevant representations and further relieves the computational burden. In experiments on four large-scale datasets, NFormer achieves a new state-of-the-art. The code is released at \url{https://github.com/haochenheheda/NFormer}.
1. Introduction
Person re-identification faces large intra-identity variation because images differ across cameras and conditions, making single-image representations vulnerable to outliers. NFormer addresses this by modeling relations among all input images while using efficient, sparse attention mechanisms.
- Different cameras, lighting, viewpoints, occlusions, and clothing produce high intra-identity variation, causing unstable matching and sensitivity to outlier features.
- Prior relation-modeling methods use only a few images during training and extract images independently at test time, creating a training-test interaction gap.
- NFormer models relations among all input images during both training and testing, aggregates representations using an affinity matrix, and suppresses intra-identity variation.
- Landmark Agent Attention reduces affinity-matrix computation by mapping high-dimensional representations into a lower-dimensional space with a handful of landmark agents.
- Reciprocal Neighbor Softmax applies sparse attention to relevant neighbors only, strengthening useful interactions while reducing interference and computational cost.
- Experiments on four person Re-ID datasets show general improvements, including a new state-of-the-art, while NFormer can further boost other state-of-the-art methods.
2. Releated Work
Prior Re-ID work mainly learns features from individual images or optimizes rankings after retrieval, while transformers provide a framework for modeling long-distance dependencies. NFormer applies transformer-style relation modeling across input persons and remains compatible with ranking optimization.
- Feature Representation Learning: Feature representation methods improve single-image descriptors through local parts, horizontal divisions, and attention modules to address occlusion and misalignment.
- Ranking Optimization: Ranking optimization methods refine initial query-gallery rankings using aggregation, feedback, or query-adaptive retrieval strategies during testing.
- Ranking Optimization: Unlike postprocessing methods that operate on ranking lists, NFormer models relations directly in the representation distribution and can combine with re-ranking methods.
- Transformer: Transformers use multi-head self-attention to jointly attend to representation elements and capture long-distance dependencies across vision tasks.
- Transformer: NFormer adopts transformer architecture to learn relations between input persons in the person re-identification task.
3. Neighbor Transformer Network
NFormer models relations among all input images during training and testing, aggregating representations to suppress intra-identity variation. Landmark Agent Attention reduces affinity computation, while Reciprocal Neighbor Softmax limits aggregation to reciprocal neighbors.
- Neighbor Transformer Network: NFormer aggregates representations across all input images during training and testing to suppress intra-identity variation and produce more robust features.It computes an affinity matrix between individual representations and performs relation-guided aggregation.
- Computational challenges: Conventional affinity computation scales as O(N^2d), making relation modeling expensive when the number of images is large.The computational burden grows with both the number of images N and representation dimension d.
- Module pipeline: The approximate affinity computation uses query and key projections mapped through sampled landmark agents before sparse attention weights aggregate the value matrix.The pipeline projects z into q, k, and v, constructs approximate affinities from lower-dimensional encodings, applies RNS, and outputs u.
- Landmark Agent Attention: Landmark Agent Attention factorizes affinity computation through low-dimensional landmark-agent representations, reducing complexity from O(N^2d) to O(N^2l).The method uses l landmark agents, typically much smaller than d; experiments use l = 5 and d ≥256.
- Reciprocal Neighbor Softmax: Reciprocal Neighbor Softmax retains attention only between reciprocal top-k neighbors, suppressing irrelevant interactions and making aggregation more focused and efficient.The reciprocal mask is formed from a top-k neighbor mask and its transpose; aggregation complexity decreases from O(N^2d) to O(Nkd).
4. Experiments
Experiments across four large-scale person Re-ID datasets evaluate NFormer’s effectiveness, efficiency, module behavior, compatibility, and dependence on available neighbor images. NFormer improves retrieval performance while constraining outliers, reducing computation, and integrating with other methods.
- Ablation Study: NFormer achieves larger gains than a regular Transformer over Res50, improving top-1/mAP by 1.5%/7.6% on Market-1501 and 3.3%/9.4% on dukeMTMC-reID.It also requires 0.0025 GFLOPs per person versus 0.088 GFLOPs for the regular Transformer.
- Ablation Study: NFormer’s representation distributions become more gathered and detached, constraining identity outliers and reducing negative persons near queries in ranking lists.The visualization compares representations before and after NFormer on Market-1501 and dukeMTMC-reID.
- Ablation Study: With l = 5 landmark agents, mAP reaches 91.1% on Market-1501 and 83.5% on dukeMTMC-reID, with only 0.3% drops while LAA uses 1.95% of original affinity-map computations.Increasing l further saturates similarity and mAP while increasing FLOPs.
- Ablation Study: RNS reaches maximum mAP at k = 20, achieving 91.1% and 83.5% on Market-1501 and dukeMTMC-reID, and outperforming normal Softmax by 7.3% and 8.9%.RNS also consistently outperforms the top-k mask variant RNS-Mk.
- Ablation Study: Combining NFormer with ABDNet and RP further boosts top-1/mAP by 1.0%/3.0% on Market-1501 and 1.7%/5.9% on dukeMTMC-reID.These results demonstrate compatibility with a stronger feature extractor and re-ranking method.
- Ablation Study: NFormer’s improvements decrease when the average images per identity fall from 20 to 5, with ∆mAP dropping from 7.3%/9.2% to 2.3%/3.8% on Market-1501/dukeMTMC-reID.The authors identify sufficiently many images of the same person as a practical requirement for useful neighbor information.
- Comparison with SOTA methods: NFormer outperforms state-of-the-art competitors across Market-1501, DukeMTMC-reID, MSMT17, and CUHK03, with especially strong mAP improvements on larger datasets.Reported gains include 2.4%/9.7% top-1/mAP over baseline on MSMT17 and 2.5%/4.2% and 3.9%/3.5% on CUHK03 labelled and detected sets.
5. Conclusion
NFormer models relations among all input images for person re-identification, using efficient and sparse attention mechanisms to produce robust, discriminative representations that scale to large inputs.
- NFormer interacts across all input images rather than only single images or a few training-batch images.
- Landmark Agent Attention efficiently models relations among large numbers of inputs.
- Reciprocal Neighbor Softmax restricts attention to relevant neighbors through sparse attention.
- NFormer scales well with large inputs and is robust to outliers.
- Extensive ablation studies show that NFormer learns robust, discriminative representations that are easy to combine with third-party methods.