PaperlayerView exact revision on arXiv

Source-linked AI summary

MHSA-Net: Multi-Head Self-Attention Network for Occluded Person Re-Identification

Hongchen Tan, Xiuping Liu, Baocai Yin, Xin Li

arXiv:2008.04015v4cs.CV

TL;DR

Occluded person Re-ID must identify people despite noisy, obstructed regions, while existing solutions may depend on manual crops or human models. MHSA-Net uses MHSAB to capture diverse key local features and ACM to suppress attention noise, achieving competitive standard-task performance and stronger results on occluded datasets. Its main scope limitation is that person-on-person occlusion is not considered.

  • Problem

    Occluded regions introduce noise and mismatching in person Re-ID, motivating adaptive extraction of key features from non-occluded regions without relying on human models.

  • Method

    MHSA-Net combines MHSAB, which captures key local information into multiple diversity embeddings, with ACM, which prunes non-important local information.

  • Results

    MHSA-Net achieves better performance than most state-of-the-art methods on four occlusion datasets and competitive performance on three standard person Re-ID datasets.

  • Takeaways & Limitations

    MHSAB and ACM both contribute to performance improvement, supporting MHSA-Net as a competitive approach for standard and occluded person Re-ID.

  • Takeaways & Limitations

    The method mainly considers object-based occlusion and does not consider person-occluding-person scenarios.

Abstract

from arXiv · show

This paper presents a novel person re-identification model, named Multi-Head Self-Attention Network (MHSA-Net), to prune unimportant information and capture key local information from person images. MHSA-Net contains two main novel components: Multi-Head Self-Attention Branch (MHSAB) and Attention Competition Mechanism (ACM). The MHSAB adaptively captures key local person information, and then produces effective diversity embeddings of an image for the person matching. The ACM further helps filter out attention noise and non-key information. Through extensive ablation studies, we verified that the Multi-Head Self-Attention Branch (MHSAB) and Attention Competition Mechanism (ACM) both contribute to the performance improvement of the MHSA-Net. Our MHSA-Net achieves competitive performance in the standard and occluded person Re-ID tasks.

I. INTRODUCTION

Occlusion remains a central challenge for person Re-ID because obstructed regions introduce noise and existing approaches may require manual crops or unstable human models. MHSA-Net addresses this with MHSAB and ACM, achieving stronger occluded-task performance while remaining competitive on standard datasets.

  • I. INTRODUCTION: Occluded regions introduce noise and mismatching, motivating discriminative feature extraction from unoccluded regions.Manual cropping is inefficient, while pose-based extraction can be unstable because pose-estimation and retrieval training datasets differ.
  • I. INTRODUCTION: MHSAB adaptively captures key local person information and produces multiple diversity embeddings for person matching.The design is based on multi-head self-attention, which generates multiple attention maps from different aspects of one image.
  • I. INTRODUCTION: ACM helps MHSAB prune attention noise from unimportant or non-key regions.The mechanism is introduced because attention noise can remain in occluded or non-key sub-regions.
  • I. INTRODUCTION: MHSA-Net outperforms most state-of-the-art methods on four occlusion datasets while producing similar results to state-of-the-art algorithms on standard datasets.The occlusion datasets are Occluded-DukeMTMC, P-DukeMTMC-reID, Partial-REID, and Partial-iLIDS; standard datasets include Market-1501, DukeMTMC-reID, and CUHK03.
  • I. INTRODUCTION: Unlike human-model-based approaches, MHSA-Net adaptively adjusts local attention to enhance key features and suppress information from occlusion regions.The attention mechanism uses a multi-parameter learning strategy and does not rely on an external human model.

B. Occluded Person Re-identification

Occluded person Re-ID methods seek key features directly from whole occluded images, avoiding manual cropping and other preprocessing. MHSA-Net combines a global branch, MHSAB, and ACM to generate and refine features for matching.

  • B. Occluded Person Re-identification: Occluded person Re-ID directly captures key features from whole occluded images, whereas partial-person methods require manually cropped probe images.Prior approaches include classification, feature reconstruction, and foreground-background masking to reduce occlusion or clutter effects.
  • B. Occluded Person Re-identification: MHSA-Net comprises the Global Feature Branch, MHSAB, and ACM for occlusion and standard person Re-ID.The Global Feature Branch computes global features used by MHSAB and person matching.
  • B. Occluded Person Re-identification: MHSAB captures key local information through MHSAM, FRM, SAFFM, and RLM, outputting attention weights and fused features.Its submodules generate multiple embeddings, regularize their diversity, adaptively fuse them, and combine local and global features through residual learning.
  • B. Occluded Person Re-identification: ACM updates attention weights and fusion features to enhance key person information and suppress non-key person information.It consists of loss functions and a regularization item.
  • B. Occluded Person Re-identification: For standard Re-ID, MHSA-Net concatenates local and global feature vectors, whereas occluded Re-ID uses only the local fused feature vector.Matching compares squared distances between feature vectors.

IV. GLOBAL FEATURE BRANCH (BASELINE)

The baseline uses a modified ResNet-50 and global feature branch, while the MHSAB adds adaptive multi-head attention to capture key local information from occluded images.

  • IV. GLOBAL FEATURE BRANCH (BASELINE): The baseline combines a modified ResNet-50 backbone with a Global Feature Branch for person-image encoding.The backbone outputs Q(x) ∈ R24×8×2048, and the baseline is composed of the backbone and GFB.
  • IV. GLOBAL FEATURE BRANCH (BASELINE): The baseline loss contains only cross entropy because adding triplet loss negatively impacted generic occluded person Re-ID performance.The authors associate the stricter global-feature constraint with reduced local feature capturing.
  • IV. GLOBAL FEATURE BRANCH (BASELINE): MHSAM addresses two design issues: concatenated multi-head embeddings create expensive high-dimensional descriptors, while separate embeddings may redundantly encode similar information.The method therefore seeks a low-dimensional descriptor and diverse embeddings for person matching.
  • IV. GLOBAL FEATURE BRANCH (BASELINE): MHSAB combines MHSAM, Feature Regularization Mechanism, and Self-Attention Feature Fusion Module to produce adaptive local representations.MHSAM computes multiple attention maps and embeddings, while FRM improves diversity and individual embedding quality.
  • IV. GLOBAL FEATURE BRANCH (BASELINE): MHSAM learns K pixel-level attention weights and computes K attention-weighted 512-dimensional embeddings from the feature map Q(x).The attention weights are normalized across heads at each pixel, and the resulting heads form the feature branches.
  • IV. GLOBAL FEATURE BRANCH (BASELINE): Higher learned attention weights indicate more important local regions, enabling MHSAM to emphasize unoccluded information and suppress occluded regions.The attention weights are learned toward the person-matching objective.

B. Feature Regularization Mechanism (FRM)

FRM regularizes MHSAM’s K embeddings for diversity and trains each embedding to support person matching, producing refined local representations.

  • B. Feature Regularization Mechanism (FRM): FRM combines Feature Diversity Regularization Term and Improved Hard Triplet Loss to refine the K embeddings P(x) into P⊥(x).FDRT promotes coverage of key information from different perspectives, while IHTL improves individual embedding utility.
  • B. Feature Regularization Mechanism (FRM): FDRT minimizes the deviation of the embeddings’ Gram matrix from the identity matrix, encouraging diverse feature vectors.P(x) is normalized before constructing G(x)=P(x)P(x)^T and applying the Frobenius-norm regularization.
  • B. Feature Regularization Mechanism (FRM): The resulting diverse embeddings can capture key information from different perspectives and enhance model robustness.This is the stated effect of applying FDRT to the K representations.
  • B. Feature Regularization Mechanism (FRM): IHTL uses K×K pairwise distances between image embeddings, selecting the largest positive-pair distance and smallest negative-pair distance.These selections define improved hard positive and hard negative pairs for the loss.
  • B. Feature Regularization Mechanism (FRM): During training, IHTL refines embeddings so each can perform person matching separately and focus on important information.The triplet units consist of anchor, positive, and negative feature samples organized by person identity.

C. Self-Attention Feature Fusion Module (SAFFM)

SAFFM adaptively fuses MHSAM’s multiple embeddings into a lower-dimensional descriptor for efficient person matching and training.

  • C. Self-Attention Feature Fusion Module (SAFFM): SAFFM learns K attentional weights and uses them to fuse P⊥(x) into a single 512-dimensional feature p*(x).This avoids directly concatenating the multiple embeddings and the resulting dimensional expansion.
  • C. Self-Attention Feature Fusion Module (SAFFM): The attention-weight matrix β(x) is computed after transposing P⊥(x), with softmax applied pixel-wise across the attention vectors.The two learned weight matrices are ω4 ∈ R512×1024 and ω5 ∈ R1024×512.
  • C. Self-Attention Feature Fusion Module (SAFFM): SAFFM reduces the dimensionality of multiple embeddings during both training and testing.During training, p*(x) is also supplied to cross entropy and hard triplet losses.

D. Residual Learning Module

The Residual Learning Module combines global and local features so the local branch focuses on information not already represented globally.

  • D. Residual Learning Module: RLM copies the global feature q*(x) K times and combines it with local features P⊥(x) through residual learning.The copied global representation Q*(x) matches P⊥(x)’s K×512 dimensions.
  • D. Residual Learning Module: RLM encourages P⊥(x) to capture only important local information rather than duplicating the global feature.The module’s stated purpose is to separate local information from global information.
  • D. Residual Learning Module: The residual embedding is normalized with layer normalization, producing Z(x)=Norm(Q*(x)+P⊥(x)).The normalization is explicitly identified as layer normalization.
  • D. Residual Learning Module: The MHSAB training objective sums SAFFM, feature-diversity, residual-learning, and improved-hard-triplet losses with balance parameters λ1 and λ2.The resulting Z(x) is used during training but does not participate in person matching at test time.

VI. ATTENTION COMPETITION MECHANISM

The Attention Competition Mechanism (ACM) refines MHSAB attention by suppressing non-key information while preserving or enhancing regions useful for person matching.

  • VI. ATTENTION COMPETITION MECHANISM: The mechanism was adapted from attention competition in text-to-image generation and introduced here for person Re-ID.The authors describe this as the first competition strategy designed for the Re-ID task.
  • VI. ATTENTION COMPETITION MECHANISM: ACM combines attention regularization with the person Re-ID loss to suppress non-key regions and strengthen important regions.The regularization term pushes weights below threshold γ toward zero, while the matching loss increases beneficial sub-region weights.
  • VI. ATTENTION COMPETITION MECHANISM: MHSA-Net evaluation covers three generic and four occluded person Re-ID benchmarks, with ablation studies assessing each component.The benchmarks include Market-1501, DukeMTMC-reID, CUHK03, Occluded-DukeMTMC, P-DukeMTMC-reID, Partial-REID, and Partial-iLIDS.

A. Datasets and Evaluation

MHSA-Net is evaluated using Rank-1 and mAP on generic and occluded person Re-ID benchmarks, with local and global features compared across model variants.

  • A. Datasets and Evaluation: Evaluation uses Rank-1 accuracy and mAP, with resized and normalized images processed by an ImageNet-pretrained ResNet-50 backbone.Training additionally uses random horizontal flipping.
  • C. Comparison with state-of-the-art Methods: MHSA-Net reports Rank-1 scores of 94.6, 87.3, 73.4, and 75.8 on the four generic benchmarks before re-ranking.The passage introduces these results for Market-1501, DukeMTMC-Re-ID, CUHK03-Labeled, and CUHK03-Detected, respectively.
  • C. Comparison with state-of-the-art Methods: 95.5, 90.7, 80.2, and 82.6 Rank-1 with 93.0, 87.2, and 80.9 mAP are reported after re-ranking on four generic datasets.The values correspond respectively to Market-1501, DukeMTMC-Re-ID, CUHK03-Detected, and CUHK03-Labeled where supplied.
  • Occluded Person Re-ID: 70.7 Rank-1 and 41.1 mAP are achieved by MHSA-Net* on P-DukeMTMC-reID, outperforming all previous methods.The model improves over the baseline by 9.7 Rank-1 and 14.1 mAP on this dataset.
  • Occluded Person Re-ID: MHSA-Net* and MHSA-Net† achieve the best performance on Partial-REID and Partial-iLIDS, while local-feature variants are favored under severe occlusion.The paper recommends the full model for general data, MHSA-Net† for certain occlusions, and MHSA-Net* for severe occlusions.

D. Ablation Study of MHSA-Net

Ablation studies show that the MHSA-Net components improve baseline performance, while attention visualizations indicate better localization and diverse feature extraction.

  • D. Ablation Study of MHSA-Net: The selected hyperparameters are λ1 = 1e-4, λ2 = 1.0, λ3 = 1e-3, γ = 1e-3, and K = 8.The settings were chosen through hyperparameter discussions and stable person-matching results.
  • D. Ablation Study of MHSA-Net: 2.6/7.2, 6.3/10.3, 16.5/16.3, and 17.0/17.5 Rank-1/mAP gains over baseline occur on four generic datasets with the full model.The gains are reported for Market-1501, DukeMTMC-Re-ID, CUHK03-Detected, and CUHK03-Labeled, respectively.
  • D. Ablation Study of MHSA-Net: Each individual MHSA-Net component improves the baseline, and each embedding branch produces large gains on Occluded-DukeMTMC and CUHK03-Detected.The eight-branch setting uses K=8; smaller K values such as 5 or 6 produce minor performance changes.
  • D. Ablation Study of MHSA-Net: MHSAB captures more key sub-regions than the baseline, while adding ACM suppresses some attention and highlights regions judged more important.The combined module better captures key information from unoccluded regions in occluded person images.
  • D. Ablation Study of MHSA-Net: FDRT helps the eight MHSAB heads capture diverse information, whereas removing FDRT and IHTL makes responses redundant, scattered, sparse, or weak.The authors connect these regularization and loss constraints with ensuring diversity-oriented key information across feature branches.

1) Multi-Head Self-Attention Mechanism (MHSAM):

The MHSAM analysis identifies effective head counts and shows that feature fusion, FDRT, and IHTL improve person Re-ID performance under selected hyperparameter settings.

  • 1) Multi-Head Self-Attention Mechanism (MHSAM):: K = 7 or 8 provides good MHSAM performance across Rank-1 and mAP on Market-1501 and CUHK03-Detected.All K > 0 settings improve the baseline in both datasets.
  • 1) Multi-Head Self-Attention Mechanism (MHSAM):: SAFFM performs best among the compared fusion operations for combining P⊥(x) embeddings on Occluded-DukeMTMC.The alternatives are direct concatenation and summation into a 512-dimensional vector.
  • 1) Multi-Head Self-Attention Mechanism (MHSAM):: λ1 = 10^-4 gives the highest Rank-1 and mAP, and adding FDRT improves MHSAM performance on Market-1501 and CUHK03-Detected.These findings are reported under the tested λ1 range and corresponding K settings.
  • 1) Multi-Head Self-Attention Mechanism (MHSAM):: IHTL performs best at λ2 = 1.0 for K = 7 and K = 8, and its inclusion improves MHSAM performance.The scores increase and then decline as λ2 varies from 0.01 to 100.

F. Attention Competition Mechanism (ACM)

The Attention Competition Mechanism (ACM) regularizes attention to suppress non-key regions and improves the MHSAB under suitable hyperparameter settings. The paper also identifies scope limitations involving person-person occlusion and deployment in end-to-end person search.

  • Mechanism: The ACM combines competition loss LC with MHSAB losses, while its individual effectiveness is tested by setting λ1 = 0 and λ2 = 0.The full MHSAB loss includes LSAFFM, LFDRT, LIHTL, LReN, and LC.
  • Hyperparameter analysis: γ ≤10^-2 and λ3 ≤10^-1 improve MHSAM performance, whereas overly large γ suppresses attention weights across most regions.The model sets λ3 = 10^-3 and γ = 10^-3 in MHSA-Net.
  • Limitations: The proposed model mainly addresses object-person occlusion and does not consider person-person occlusion.The authors identify person-person occlusion as a direction for future optimization.
  • Limitations: Person search requires combining person detection with person Re-ID because relying only on the Re-ID model cannot effectively find the target person.The paper therefore calls for an end-to-end framework combining MHSA-Net with detection models.
  • Overall contribution: MHSA-Net uses MHSAB to capture diverse local person information and ACM to prune non-important regions, with both components contributing to competitive Re-ID performance.The conclusion reports competitive results on three standard and four occluded person Re-ID datasets.
Loading 2008.04015v4…