Source-linked AI summary
Pose-guided Feature Disentangling for Occluded Person Re-identification Based on Transformer
Tao Wang, Hong Liu, Pinhao Song, Tianyu Guo, Wei Shi
TL;DR
Occluded person Re-ID must cope with incomplete body information, occlusion noise, and features that resemble human parts. PFD uses a transformer with pose-guided aggregation, semantic views, pose-view matching, and a push loss, and reports competitive results across five datasets and two Re-ID settings.
Problem
Occlusion makes person Re-ID difficult by hiding body information, introducing noise, and producing non-human features that resemble body parts.
Method
PFD combines a ViT-based transformer with pose-guided feature aggregation, learnable semantic views, Pose-View Matching, and Pose-guided Push Loss.
Results
Competitive results on five datasets for occluded and holistic Re-ID demonstrate that PFD performs favorably against state-of-the-art methods.
Takeaways & Limitations
PFD provides a pose-guided transformer framework that emphasizes visible body parts while separating occluded features.
Abstract
from arXiv · showhide
Occluded person re-identification is a challenging task as human body parts could be occluded by some obstacles (e.g. trees, cars, and pedestrians) in certain scenes. Some existing pose-guided methods solve this problem by aligning body parts according to graph matching, but these graph-based methods are not intuitive and complicated. Therefore, we propose a transformer-based Pose-guided Feature Disentangling (PFD) method by utilizing pose information to clearly disentangle semantic components (e.g. human body or joint parts) and selectively match non-occluded parts correspondingly. First, Vision Transformer (ViT) is used to extract the patch features with its strong capability. Second, to preliminarily disentangle the pose information from patch information, the matching and distributing mechanism is leveraged in Pose-guided Feature Aggregation (PFA) module. Third, a set of learnable semantic views are introduced in transformer decoder to implicitly enhance the disentangled body part features. However, those semantic views are not guaranteed to be related to the body without additional supervision. Therefore, Pose-View Matching (PVM) module is proposed to explicitly match visible body parts and automatically separate occlusion features. Fourth, to better prevent the interference of occlusions, we design a Pose-guided Push Loss to emphasize the features of visible body parts. Extensive experiments over five challenging datasets for two tasks (occluded and holistic Re-ID) demonstrate that our proposed PFD is superior promising, which performs favorably against state-of-the-art methods. Code is available at https://github.com/WangTaoAs/PFD_Net
1 Introduction
Occluded person Re-ID must handle incomplete body information, occlusion noise, and misleading non-human features. PFD combines transformer representations with pose-guided disentangling and matching to emphasize visible body parts.
- 1 Introduction: Occluded person Re-ID is difficult because obstacles hide body parts, introduce noise, and may resemble human body regions.These conditions can cause mismatching and failure of feature learning.
- 1 Introduction: PFD combines transformer features with pose information to disentangle semantic components and selectively match non-occluded parts.The method is designed to separate non-occluded and occluded features without spatial alignment.
- 1 Introduction: The method includes a visual context transformer encoder, pose estimator, PFA module, part-view transformer decoder, and PVM module.PFA embeds pose information into global-context and part features.
- 1 Introduction: Pose-guided Push Loss focuses learning on human body parts and alleviates interference from occlusion and noise.The loss is introduced to avoid failure of feature learning.
- 1 Introduction: Experiments on occluded and holistic Re-ID datasets show that PFD performs favorably against state-of-the-art methods.The contribution statement reports extensive experimental support for the method.
2 Related Work
Related work addresses occluded Re-ID through handcrafted splitting, external cues, and transformer architectures. PFD differs by combining pose information with transformers for feature disentangling, matching, and occlusion mitigation.
- 2 Related Work: Occluded person Re-ID is more challenging than holistic Re-ID because body information is incomplete.Existing approaches are grouped into handcrafted splitting, additional-clue, and transformer-based methods.
- 2 Related Work: Handcrafted methods address occlusion by partitioning feature maps or measuring similarity among aligned patches and visible regions.Examples include PCB, VPM, and MoS.
- 2 Related Work: External-cue methods use segmentation, pose estimation, or body parsing to locate and learn discriminative human body parts.Representative methods include mask-guided attention, PGFA, PVPM, and HOReID.
- 2 Related Work: Transformer methods provide strong feature extraction and disentangling capabilities for person Re-ID.TransReID uses camera perspective information, while PAT targets robust human-part discovery.
- 2 Related Work: PFD combines pose information and transformer architecture to disentangle discriminative features and alleviate occlusion-related feature-learning failure.Its design is presented as different from the preceding method categories.
- 2 Related Work: PFD’s architecture contains a visual context encoder, PFA, a part-view decoder, PVM, and Pose-guided Push Loss.The modules respectively encode context, indicate visible parts, disentangle views, match feature sets, and emphasize visible parts.
3 Proposed Method
PFD combines a ViT encoder with pose estimation, pose-guided aggregation, semantic-view decoding, pose-view matching, and a push loss to disentangle body features and emphasize visible parts. Its pipeline uses landmark confidence to separate high- and low-confidence features, then retains high-confidence views for the final representation.
- Visual Context Transformer Encoder: The visual-context encoder splits person images into patches, projects them into D-dimensional embeddings, adds positional and camera information, and produces global and grouped part features.Overlapping patches are allowed when the sliding-window step is smaller than the patch size, helping preserve spatial-neighborhood information.
- Pose Estimation: Pose estimation extracts landmark heatmaps and assigns each landmark a binary label according to whether its confidence score reaches threshold γ.The method retains low-confidence heatmaps rather than setting them to zero, while recording their confidence status through labels.
- Pose-guided Feature Aggregation: PFA matches pose-guided heatmap features with grouped local features to assemble a set whose elements correspond to different human keypoints.The number of split groups K is set equal to the number of keypoints M, and heatmaps are transformed to the grouped-feature dimension before element-wise multiplication.
- Part View Based Transformer Decoder: The decoder uses learnable semantic part views as cross-attention queries over pose-weighted encoder features, producing view features intended to capture discriminative body parts.The keys and values combine pose heatmaps with encoder output after heatmap average pooling and multiplication.
- Pose-View Matching: PVM matches semantic views to pose-guided features and uses landmark labels to divide view features into high- and low-confidence keypoint sets.The high-confidence set Fh contains views associated with labels li = 1, while Fl contains the remaining views.
- Decoder Supervision and Representation: The Pose-guided Push Loss penalizes similarity between high- and low-confidence view features, while identity and triplet losses supervise high-confidence decoder features.At test time, the representation concatenates encoder global, decoder global, grouped local, and high-confidence view features, omitting low-confidence views and padding Fh to length Nv.
4 Experiments
Experiments evaluate PFD on five datasets spanning occluded and holistic person Re-ID, using Rank-1 accuracy and mAP. PFD outperforms competing methods on occluded benchmarks, achieves competitive or state-of-the-art holistic results, and benefits from its pose-guided modules and semantic-view design.
- Comparison with the State-of-the-Art: 67.7%/79.8% Rank-1 accuracy and 60.1%/81.3% mAP are achieved on Occluded-Duke and Occluded-REID, respectively.On Occluded-Duke, PFD outperforms all compared method categories and surpasses keypoint-based PGFA, PVPM, and HOReID by at least +12.6% Rank-1 accuracy and +16.3% mAP.
- Comparison with the State-of-the-Art: 95.5%/90.6% Rank-1 accuracy and 89.5%/82.2% mAP are obtained on Market-1501 and DukeMTMC-reID, respectively, constituting SOTA performance there.These results are reported for the two holistic datasets in that order.
- Comparison with the State-of-the-Art: 82.7% Rank-1 accuracy and 65.1% mAP are achieved on MSMT17, where PFD is described as competitive.MSMT17 is the third holistic Re-ID dataset evaluated.
- Ablation Study: Adding pose-guided feature aggregation improves Rank-1 accuracy and mAP by +5.5% and +7.9%, while combining PFA and PVM improves them by +8.8% and +11.2%.The ablation is conducted on Occluded-Duke; the overall model achieves optimal performance when the Pose-guided Push Loss is included.
- Ablation Study: Performance improves with semantic views up to Nv=17, decoder depth performs best at two layers, and threshold γ performs best at 0.2.The authors associate 17 views with the 17 keypoints and report little improvement from additional decoder layers.
- Pose Estimation and Noise Analysis: PFD remains state-of-the-art with less reliable landmark estimators and is robust to pose noise when Gaussian-noise σ is less than 10.The pose-estimation study uses HRNet, AlphaPose, and OpenPose, while Gaussian noise is added to estimated heatmaps.
- Visualization Analysis: Fused decoder attention heatmaps from learnable semantic views can almost accurately localize unobstructed human-body regions.The visualization is presented as evidence for the effectiveness of the semantic-view mechanism.
5 Conclusion
PFD is a transformer-based method for occluded person re-identification that uses pose information to disentangle semantic components and selectively match non-occluded parts. Experiments across five datasets and two re-identification tasks show competitive results demonstrating its effectiveness.
- PFD uses pose information to disentangle semantic components for occluded person re-identification.
- PFD combines a transformer encoder-decoder architecture, PFA and PVM matching modules, and a Pose-guided Push Loss.
- PFA matches estimated pose heatmaps with patch features to indicate visible body parts, while PVM matches similar semantic-view and pose-guided features.
- The Pose-guided Push Loss reduces occlusion-noise interference by increasing the embedded-space distance between visible and occluded parts.
- Experiments on Occluded-Duke, Occluded-REID, Market-1501, DukeMTMC-reID, and MSMT17 show competitive results for occluded and holistic Re-ID.