Source-linked AI summary
Feature Erasing and Diffusion Network for Occluded Person Re-Identification
Zhikang Wang, Feng Zhu, Shixiang Tang, Rui Zhao, Lihuo He, Jiangning Song
TL;DR
Occluded person ReID must match occluded and holistic pedestrian images while limiting contamination from both non-pedestrian occlusions and non-target pedestrians. FED combines NPO augmentation and occlusion erasing with feature diffusion to simulate these disturbances during training. It achieves 86.3% Rank-1 on Occluded-REID, while its feature diffusion module is removed during inference.
Problem
Existing occluded ReID approaches face contamination from non-pedestrian occlusions and non-target pedestrians, while auxiliary pose and parsing models can fail on multi-pedestrian images and add substantial computation.
Method
FED uses NPO augmentation and an occlusion erasing module to remove NPO features, then a feature diffusion module to synthesize NTP characteristics in feature space.
Results
86.3% Rank-1 accuracy on Occluded-REID surpasses other methods by at least 4.7%.
Takeaways & Limitations
Jointly optimizing OEM and FDM improves the model's perception of target pedestrians and alleviates the influence of NPO and NTP.
Takeaways & Limitations
Auxiliary pose and human parsing approaches can fail on multi-pedestrian images, miss personal belongings, and impose substantial computation.
Abstract
from arXiv · showhide
Occluded person re-identification (ReID) aims at matching occluded person images to holistic ones across different camera views. Target Pedestrians (TP) are usually disturbed by Non-Pedestrian Occlusions (NPO) and NonTarget Pedestrians (NTP). Previous methods mainly focus on increasing model's robustness against NPO while ignoring feature contamination from NTP. In this paper, we propose a novel Feature Erasing and Diffusion Network (FED) to simultaneously handle NPO and NTP. Specifically, NPO features are eliminated by our proposed Occlusion Erasing Module (OEM), aided by the NPO augmentation strategy which simulates NPO on holistic pedestrian images and generates precise occlusion masks. Subsequently, we Subsequently, we diffuse the pedestrian representations with other memorized features to synthesize NTP characteristics in the feature space which is achieved by a novel Feature Diffusion Module (FDM) through a learnable cross attention mechanism. With the guidance of the occlusion scores from OEM, the feature diffusion process is mainly conducted on visible body parts, which guarantees the quality of the synthesized NTP characteristics. By jointly optimizing OEM and FDM in our proposed FED network, we can greatly improve the model's perception ability towards TP and alleviate the influence of NPO and NTP. Furthermore, the proposed FDM only works as an auxiliary module for training and will be discarded in the inference phase, thus introducing little inference computational overhead. Experiments on occluded and holistic person ReID benchmarks demonstrate the superiority of FED over state-of-the-arts, where FED achieves 86.3% Rank-1 accuracy on Occluded-REID, surpassing others by at least 4.7%.
1. Introduction
Occluded person ReID must match pedestrians across cameras despite non-pedestrian occlusions and non-target pedestrians, which can contaminate representations. FED addresses both disturbances through image-level NPO simulation, occlusion erasing, and feature-level NTP simulation.
- Occluded person ReID must handle non-pedestrian occlusions and non-target pedestrians that disturb target-pedestrian representations in real-world scenes.
- Pose estimation and human parsing can fail on multi-pedestrian images, allowing non-target pedestrian noise to contaminate representations.Human parsing also misses some personal belongings, including backpacks and umbrellas.
- FED simulates non-pedestrian occlusions on images and non-target pedestrians in feature space to improve perception of target pedestrians.
- The proposed system combines NPO augmentation, the occlusion erasing module, and the feature diffusion module to address both occlusion sources.
2. Related Works
Prior occluded ReID methods commonly use pose estimation or human parsing, but these approaches can struggle with multi-pedestrian scenes and additional-model costs. FED instead performs end-to-end image- and feature-level simulation without relying on extra models.
- Occluded person ReID methods commonly use pose estimation or human parsing to identify visible body information.
- The FED overview contains a shared-parameter two-branch network with a feature extractor, OEM, FDM, and NPO augmentation strategy.
- FED uses an end-to-end approach that does not rely on extra models and simulates NPO and NTP at image and feature levels.
3. Feature Erasing and Diffusion Network
FED combines image-level NPO simulation and feature-level NTP simulation to learn occlusion-robust pedestrian representations. Its OEM predicts and suppresses occluded parts, while its FDM injects memorized characteristics during training and is removed for inference.
- Overall Architecture: FED begins with NPO augmentation that creates occluded image pairs and masks, then extracts four pooled local features for subsequent processing.The architecture uses a ViT feature extractor and part pooling with N=4.
- NPO Feature Erasing: The augmentation strategy pastes selected patches onto pedestrian images and generates coarse masks from pixel differences to label occluded horizontal stripes.Vertical occlusions are ignored during mask generation, while stripes covered more than three-quarters are labeled occluded.
- NPO Feature Erasing: OEM uses four part-specific submodules to regress occlusion scores and multiplies them with part features to suppress NPO-contaminated representations.Each submodule contains fully connected layers, layer normalization, and a Sigmoid output; mask supervision uses MSE loss.
- Feature Diffusion Module: FDM searches the memory bank for nearest centers, uses cross attention to diffuse their characteristics into the OEM features, and simulates NTP in feature space.Queries come from the OEM output, while keys and values come from memory-bank features.
- Feature Diffusion Module: OEM occlusion scores restrict NTP characteristic injection mainly to human parts, improving diffused-feature quality while stabilizing training through a weighted residual operation.The FDM uses multi-head attention and applies the weighted combination after nonlinear transformation.
- Training and Inference: FDM is an auxiliary training module removed during inference, while FED trains with identity and contrastive losses applied across its feature branches.The method maintains two memory banks, with the lower branch excluded from memory initialization and updates because of NPO augmentation deficiencies.
4. Experiments
FED is evaluated on occluded and holistic ReID benchmarks using CMC and mAP, with component, memory-search, and qualitative analyses. It achieves strong occluded-dataset performance while remaining comparable on holistic datasets.
- Evaluation setup: Experiments use CMC and mAP under a single-query evaluation setting across occluded and holistic ReID datasets.The occluded datasets include Occluded-DukeMTMC, Occluded-REID, and Partial-REID; holistic datasets include Market-1501 and DukeMTMC-reID.
- Occluded datasets: FED achieves the highest Rank-1 and mAP on Occluded-DukeMTMC and Occluded-REID, including 86.3% Rank-1 and 79.3% mAP on Occluded-REID.On Occluded-DukeMTMC, FED reaches 68.1% Rank-1 and 56.4% mAP; on Partial-REID, it reaches 80.5% mAP but not the highest Rank-1.
- Occluded datasets: FED's combined occlusion sets improve Occluded-REID and Partial-REID Rank-1/mAP by at least 0.7%.The expanded occlusion set combines OS1 and OS2 to increase augmentation diversity.
- Holistic datasets: 84.9% Rank-1 accuracy on DukeMTMC-reID surpasses other CNN-based methods and is close to TransReID.On holistic datasets overall, FED achieves comparable performance, while TransReID performs better because FED's occlusion-oriented components are not fully functional there.
- Ablation studies: FED achieves the highest accuracy in the component ablation, indicating that its components work individually and cooperatively.The ablation compares random erasing, NPO augmentation, OEM, FDM, and the complete FED model.
- Qualitative analysis: OEM identifies crucial NPO but treats stripes in multi-pedestrian images as valuable, motivating FDM for NTP-related interference.Qualitative retrieval examples show FED improving recognition for both object-occluded and multi-pedestrian images.
5. Conclusion
FED addresses both NPO and NTP in occluded person ReID by erasing occlusion features and diffusing pedestrian representations with memorized features. Joint optimization improves the model’s perception of target pedestrians across person ReID benchmarks.
- FED combines an NPO augmentation strategy, an occlusion erasing module, and a feature diffusion module.These components jointly address distractions from non-pedestrian occlusions and non-target pedestrians.
- The occlusion erasing module removes NPO features using predicted occlusion scores, after image-level NPO augmentation guides its training.
- The feature diffusion module synthesizes NTP characteristics by diffusing erased pedestrian representations with memorized features in feature space.
- Jointly optimizing OEM and FDM improves the model’s perception ability on target pedestrians, as shown by experiments on multiple person ReID benchmarks.