PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Completion for Occluded Person Re-Identification

Ruibing Hou, Bingpeng Ma, Hong Chang, Xinqian Gu, Shiguang Shan, Xilin Chen

arXiv:2106.12733v1cs.CV

TL;DR

Occluded person reID remains challenging, and many methods discard hidden regions. RFC completes occluded-region features using spatial and temporal context. The authors report strong results on occluded and holistic benchmarks, with about 10% mAP improvement over state-of-the-art methods on occluded datasets.

  • Problem

    Many existing reID methods focus on holistic images, while occluded scenes can hide discriminative information and make identification challenging.

    Source: §1 · p. 1

  • Method

    RFC partitions feature maps into body regions and uses spatial and temporal region-feature completion to recover occluded-region features.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    RFCnet significantly outperforms state-of-the-art methods by about 10% mAP on occluded datasets and performs favorably on holistic benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that feature completion can recover occluded features and report effectiveness on image and video reID, especially on occluded datasets.

    Source: §7 · p. 16

    RFCnet may fail under severe occlusion when most body parts are invisible and little appearance information remains.

    Source: §6.8 · pp. 15–16

Abstract

from arXiv · show

Person re-identification (reID) plays an important role in computer vision. However, existing methods suffer from performance degradation in occluded scenes. In this work, we propose an occlusion-robust block, Region Feature Completion (RFC), for occluded reID. Different from most previous works that discard the occluded regions, RFC block can recover the semantics of occluded regions in feature space. Firstly, a Spatial RFC (SRFC) module is developed. SRFC exploits the long-range spatial contexts from non-occluded regions to predict the features of occluded regions. The unit-wise prediction task leads to an encoder/decoder architecture, where the region-encoder models the correlation between non-occluded and occluded region, and the region-decoder utilizes the spatial correlation to recover occluded region features. Secondly, we introduce Temporal RFC (TRFC) module which captures the long-term temporal contexts to refine the prediction of SRFC. RFC block is lightweight, end-to-end trainable and can be easily plugged into existing CNNs to form RFCnet. Extensive experiments are conducted on occluded and commonly holistic reID benchmarks. Our method significantly outperforms existing methods on the occlusion datasets, while remains top even superior performance on holistic datasets. The source code is available at https://github.com/blue-blue272/OccludedReID-RFCnet.

1 INTRODUCTION

Occluded person re-identification is important in real-world scenes, but methods that discard hidden regions can lose discriminative information. RFC completes region features using spatial and temporal context, and is reported to perform strongly on occluded and holistic benchmarks.

1 INTRODUCTION

  • Occluded person reID matters because real scenes contain obstacles, while many existing methods focus on holistic images and discard occluded parts.Discarding hidden parts can reduce retrieval reliability when visible regions look similar and the occluded parts contain key discriminative facts.Source: §1 · p. 1
  • Feature completion is lightweight, integrates into the reID network for end-to-end training, and can capture long-range spatial and long-term temporal context.Unlike image completion, reID feedback can supervise feature completion; region features also let a module propagate spatial information between distant positions.Source: §1 · pp. 1–2
  • RFC uses an Adaptive Partition Unit to divide feature maps into body regions, then SRFC recovers occluded features spatially and TRFC refines predictions with temporal context.SRFC uses a region-encoder and region-decoder to propagate information from correlated visible regions; TRFC uses long-term temporal context.Source: §1 · p. 2
  • RFCnet significantly outperforms state-of-the-art methods by about 10% mAP on occluded datasets and performs favorably on holistic benchmarks.The authors also report superior performance on both occluded and holistic reID compared with state-of-the-art methods.Source: §1 · p. 2
  • The paper introduces Occluded-DukeMTMC-VideoReID and notes that SRFC can also support image reID because images are single-frame videos.RFC is integrated with ResNet-50 to form RFCnet.Source: §1 · p. 2

2 RELATED WORK

Related work spans image reID, video reID, and methods targeting occlusion. The paper positions RFC as feature completion that recovers hidden-region features rather than discarding occluded content.

2.1 Image Person ReID

  • Image reID methods include Siamese and identity-classification models, external-clue guidance, and part-based matching; RFC instead recovers occluded-region features.Some occlusion-handling approaches discard hidden parts and rely on visible regions, which the paper says reduces retrieval reliability.Source: §2.1 · pp. 2–3

2.2 Video Person ReID

  • Video reID work exploits temporal cues through optical flow, recurrent aggregation, 3D convolution, non-local blocks, and temporal attention.Attention-based methods may discard occluded frames, while RFC performs feature completion and can be integrated for end-to-end training.Source: §2.2 · p. 3

2.3 Occluded Person ReID

  • Occluded reID methods include occlusion classification, feature reconstruction, and pose-guided alignment, while RFC targets video occluded reID by recovering hidden-part features from spatial-temporal context.The paper also states that its method can be used for image reID because images can be treated as single-frame videos.Source: §2.3 · p. 3

3 REGION FEATURE COMPLETION BLOCK

RFC completes occluded region features using complementary spatial and temporal contexts, with sequential SRFC and TRFC modules providing better results than a parallel arrangement. APU partitions features into body regions, and the completed features are projected back to the original CNN feature space.

3 REGION FEATURE COMPLETION BLOCK

  • RFC combines APU, foreground-guided feature extraction, and spatial and temporal completion modules; sequential completion performs better than parallel placement.The completed region features are projected back to the original feature space for compatibility with CNN architectures.Source: §3 · pp. 3–4

3.1 Adaptive Partition Unit

  • APU adaptively partitions each frame into six body regions, using finer divisions for the lower body, based on four predicted keypoints.Human keypoints guide the locator, and the authors report that APU uses four coarse keypoints rather than predicting all keypoints.Source: §3.1 · p. 4

3.2 Foreground-Guided Region Feature Extractor

  • The foreground-guided extractor uses body-mask guidance to produce a foreground probability map and region features less affected by background and occluders.A 1 × 1 convolution and sigmoid produce the map, which weights features for the divided regions.Source: §3.2 · p. 4

3.3 SRFC Module

  • SRFC assigns regions to clusters using appearance and position, then decodes clusters into region features so occluded regions can draw on correlated regions.Its encoder encourages similar assignments for regions with similar appearance or close positions; residual learning eases completion.Source: §3.3 · pp. 5–6

3.4 TRFC Module

  • TRFC uses each region feature as a query over corresponding regions in other frames, adding gated temporal context to refine SRFC predictions.The gate controls how much temporal context is passed to the query.Source: §3.4 · p. 7

3.5 Embedding with Networks

  • The embedding stage uses region masks to project completed features into the original feature-map space, then applies batch normalization and a residual connection.Batch normalization adjusts the scale of the projected features to the initial features.Source: §3.5 · p. 7

4 NETWORK DESIGN FOR RFC

RFCnet adds RFC blocks to a modified ResNet-50 backbone and aggregates frame features for video re-identification. Its objective combines identity, triplet, key-point, foreground-map, and assignment-regularization losses.

4.1 Network Architecture

  • RFC blocks are placed at the stage2 and stage3 bottlenecks of modified ResNet-50, with TRFC omitted for image re-identification.The video architecture averages frame features to produce one sequence representation.Source: §4.1 · p. 7

4.2 Objective Functions.

  • The total objective combines cross-entropy and triplet losses with weighted key-point, foreground-map, appearance-assignment, and position-assignment losses.The weights λ1, λ2, and λ3 balance these additional terms.Source: §4.2 · p. 7
  • Cross-entropy supervises identity classification, while batch-hard triplet loss selects the hardest positive and negative examples within each batch.The triplet objective aims to reduce intra-class variation and increase inter-class variation.Source: §4.2 · p. 7
  • Key-point loss trains APU to predict key-point locations using labels from a pose estimator.The key-point loss is defined using cross-entropy.Source: §4.2 · p. 8
  • Foreground-map loss classifies each pixel as background or occlusion versus body, using person-mask labels and binary cross-entropy.The mask is obtained from a human parsing model.Source: §4.2 · p. 8

5 OCCLUDED-DUKEMTMC-VIDEOREID DATASET

The authors reorganize DukeMTMC-VideoReID into Occluded-DukeMTMC-VideoReID, a video-based occluded re-identification dataset. Its query and gallery sets include substantial occlusion, with varied viewpoints and obstacles.

5.1 Data Reorganization

  • All query tracklets contain occlusion, and the query set is constructed so each query-gallery comparison includes at least one occluded tracklet.The testing set is re-split and occluded query tracklets are selected from the original query and gallery sets.Source: §5.1 · p. 8
  • The authors remove 494 training tracklets that contain the same obstacles as in the test set to avoid models remembering those test occlusion patterns.Source: §5.1 · p. 8
  • The reorganized dataset contains 1,702 training tracklets across 702 identities, 661 query tracklets, and 2,636 gallery tracklets.Source: §5.1 · p. 8

5.2 Properties of Occluded-DukeMTMC-VideoReID

  • Occluded-DukeMTMC-VideoReID is presented as the first video-based occluded re-identification dataset, with varied viewpoints and obstacles including cars, trees, bicycles, and people.The authors also describe it as the largest occluded person re-identification dataset to date.Source: §5.2 · p. 8
  • More than 70% of gallery tracklets contain occlusion; over 60% of query and 30% of gallery tracklets have more than 80% occluded frames.The authors characterize the dataset as containing serious occlusion.Source: §5.2 · p. 8

6 EXPERIMENTS

Experiments evaluate RFCnet on occluded and holistic image and video re-identification benchmarks. The paper reports CMC and mAP under a single-query setting and describes distinct image- and video-training procedures.

6.1 Datasets and Settings

  • Evaluation covers one image and one video occluded benchmark, four holistic image benchmarks, and two holistic video benchmarks.The datasets include Occluded-DukeMTMC, Occluded-DukeMTMC-VideoReID, Market-1501, DukeMTMC-reID, CUHK03, MSMT17, MARS, and DukeMTMC-VideoReID.Source: §6.1 · p. 9
  • Image training uses 256 × 128 inputs, random flipping and erasing, Adam, minibatches of 64, and 60 epochs.Random erasing implicitly adds occluded images to training.Source: §6.1 · p. 9
  • For video training, RFCnet samples four frames at stride eight and trains for 150 epochs; testing averages features from successive 64-frame clips.The four-frame training clips are used because of GPU-memory limits.Source: §6.1 · p. 9
  • The reported evaluation metrics are CMC curves and mean Average Precision, using a single-query setting.Source: §6.1 · p. 9

6.2 Evaluation on Occluded Person Datasets

RFCnet outperforms comparison methods on occluded image and video benchmarks, while remaining competitive or superior on holistic benchmarks. The reported gains are especially large on occluded video re-identification.

6.2 Evaluation on Occluded Person Datasets

  • Up to 8.8% mAP improvement over occluded re-identification methods is reported on Occluded-DukeMTMC.RFCnet also consistently achieves the best performance under the same conditions in the image-setting comparison.Source: §6.2 · p. 9, Table 4 · p. 10
  • 29.6% top-1 accuracy and 27.7% mAP are the reported maximum gains over holistic video re-identification methods on the occluded video benchmark.The authors report that RFCnet outperforms the best existing methods on Occluded-DukeMTMC-VideoReID.Source: Table 5 · p. 10, §6.2 · p. 10

6.3 Evaluation on Holistic Person Datasets

  • 2.0% and 2.2% mAP gains are reported on Market-1501 and DukeMTMC, respectively, against occluded re-identification methods.RFCnet is also reported as comparable or superior to methods designed for holistic re-identification.Source: §6.3 · p. 10, Table 6 · p. 11
  • About 4.6% mAP improvement over temporal-attention methods is reported on MARS, and RFCnet achieves the best performance on the holistic video benchmarks.RFCnet also gains 1.7% mAP over VRSTC on DukeMTMC-VideoReID with less computation.Source: §6.3 · pp. 10–11, Table 7 · p. 11
  • RFCnet shows greater superiority on occluded datasets than on holistic re-identification datasets.Source: §6.3 · p. 11

6.4 Ablation Study

The ablations find that RFC’s spatial and temporal completion modules, adaptive partitioning, and training constraints improve occluded re-identification. Multiple RFC blocks add further gains with modest computational overhead.

SRFC block

  • 4.7% and 8.8% mAP gains over foreground-only region discarding show SRFC’s completion advantage on Occluded-DukeMTMC and Occluded-DukeMTMC-VideoReID, respectively.The authors report that SRFC remarkably improves performance, while using foreground masks only for discarding occluded regions yields small improvements.Source: Table 8 · p. 11, §6.4 · p. 12, Table 9 · p. 12

TRFC block

  • On video occluded reID, TRFC alone improves baseline mAP and top-1 accuracy by above 10%, and combining SRFC and TRFC adds 7.2% on both metrics.The authors attribute the gains to long-term temporal context and complementary completion by the two modules.Source: §6.4 · p. 12, Table 9 · p. 12

Location of RFC block within the network

  • Adding RFC blocks at stages 2 and 3 improves over a single stage-2 block by about 2.7% mAP and 3.0% top-1 accuracy on the image occluded dataset.The authors describe multiple blocks as hierarchical: later blocks refine earlier predictions with complementary features.Source: Table 8 · p. 11, Table 9 · p. 12, §6.4 · p. 13

Adaptive Partition Strategy

  • Adaptive partitioning consistently outperforms fixed partitioning, and APU performs best with six regions on Occluded-DukeMTMC.The authors note that too few regions locate occluded parts poorly, while too many can split visible body parts and damage their semantics.Source: Figure 7 · p. 12, §6.4 · p. 13

Partition comparisons

  • APU exceeds STN by 6.4% mAP, while SRFC exceeds fixed keypoint-select clustering by 4.2% and 5.0% mAP on the image and video occluded datasets, respectively.The authors relate APU’s advantage over STN to human-keypoint guidance and the fixed keypoint-select strategy’s difficulty handling varied occlusion patterns.Source: §6.4 · pp. 12–13

Constraint ablations

  • RFCnet outperforms variants without key-point or foreground-map constraints, whose learned partitions or maps can lack body-part semantics or focus on background and occluded regions.The authors link these patterns to difficulty modeling body-part correlations or eliminating obstacle interference.Source: §6.4 · p. 13, Figure 8 · p. 13

Assignment regularization

  • Assignment regularization helps prevent degenerate clustering on severely occluded images by aligning assignment similarity with regional appearance and position similarity.Without the regularization terms, the model may assign all regions to one cluster; with them, appearance-dissimilar or distant regions can be separated.Source: §6.4 · p. 14, Figure 10 · p. 14

Complexity Comparisons

  • For image reID, RFCnet requires 4.34 GFLOPs (a 6.8% increase over the original model), introduces 2.1M parameters (8.9% over baseline), and takes 78 minutes to train and 119 seconds for inference (increases of 8.3% and 8.2% over baseline).Source: §6.4 · p. 14

6.5 Performance Analysis between video and image.

RFCnet performs better on the video occluded benchmark than on the image benchmark, where non-occluded frames can provide additional appearance information. The authors attribute additional video-task gains to TRFC’s use of those frames.

Performance Analysis between video and image

  • RFCnet performs much better on Occluded-DukeMTMC-VideoReID than Occluded-DukeMTMC, where non-occluded frames can partly restore a person’s integral appearance.The authors also report that the baseline achieves 15% higher performance on the video dataset, which they say implicitly shows that sequences with more frames can partially solve occlusion.Source: §6.5 · p. 14, Figure 11 · p. 14
  • On Occluded-DukeMTMC-VideoReID, SRFC brings 13.8% top-1 gain over baseline and TRFC brings a further 10.3% top-1 gain.The authors explain that TRFC can use visible information from non-occluded frames to recover appearance when a region has no appearance clues in the occluded frame.Source: §6.5 · p. 14, Figure 11 · p. 14

6.6 Parameter Analysis

The parameter analysis varies each of three hyperparameters while holding the others at selected values. It reports the best-performing settings and uses a held-out identity validation set for tuning.

Parameter Analysis

  • The authors tune the hyperparameters on a validation set made from 100 identities split from the original training set.They report that the pattern across hyperparameters on the validation set is consistent with the test-set pattern.Source: §6.6 · p. 14
  • RFCnet achieves its best performance at λ1 = 0.1, λ2 = 0.5, and λ3 = 0.05 on Occluded-DukeMTMC.The authors report that positive λ1 and λ3 settings outperform zero, while λ2 performance varies within a small range.Source: §6.6 · p. 14

6.7 Visualization RFC Block

The visualizations illustrate how SRFC uses region assignments to complete occluded features and how TRFC aggregates information across frames. They also show that SRFC’s assignments adapt to the input occlusion pattern.

Visualizing SRFC module

  • SRFC’s encoding groups partially occluded R5 with visually similar R3, and fully occluded R6 with positionally close R4, allowing decoding to distribute correlated features to occluded regions.The authors say the updated lower-body features become more distinguishable after completion.Source: §6.7 · p. 15, Figure 14 · p. 15
  • For a different occlusion pattern, SRFC assigns fully occluded R3 and R5 alongside their closest regions R4 and R6, respectively, and still recovers their features.The authors describe this assignment as adaptive to the input image’s occlusion mode.Source: §6.7 · p. 15, Figure 14 · p. 15

Visualizing TRFC module

  • TRFC aggregates information from other frames to recover occluded-region features, while assigning lower gating weights to occluded regions.The authors say suppressing those occluded features helps the final video feature describe the target person.Source: §6.7 · p. 15, Figure 15 · p. 15

6.8 Visualizing Retrieval Results

RFCnet retrieves correctly in the illustrated case by completing lower-body features, while PCB and PGFA make errors tied to obstacles or discarded regions. Severe occlusion remains a failure condition, especially when little of the target is visible.

6.8 Visualizing Retrieval Results

  • RFCnet completes lower-body features to form a fuller identity characteristic, succeeding where PCB is distracted by similar obstacles and PGFA by similar upper clothes.The comparison shows retrieval examples on Occluded-DukeMTMC.Source: §6.8 · p. 15, Figure 16 · p. 16
  • Severe occlusion by another person can make RFCnet mistake the disturbed person for the target and retrieve people resembling that person.The failure cases are illustrated in Fig. 17.Source: §6.8 · p. 15, Figure 17 · p. 16
  • When static objects hide most body parts, RFCnet has too little appearance information to predict occluded features and may miss the true person.These failures are also illustrated in Fig. 17.Source: §6.8 · p. 16, Figure 17 · p. 16

7 CONCLUSION

The conclusion presents RFC as a feature-completion block that combines spatial and temporal context to recover occluded features, and reports effectiveness especially on occluded datasets. It also introduces a video occluded reID dataset and recommends feature-level completion for this task.

7 CONCLUSION

  • RFC jointly captures long-range spatial and long-term temporal context to recover occluded regions, suppress noise, and alleviate information loss.The authors report experiments on image and video reID showing effectiveness, especially on occluded datasets.Source: §7 · p. 16
  • The paper introduces Occluded-DukeMTMC-VideoReID to facilitate research on video occluded reID.The conclusion reports extensive image and video reID experiments.Source: §7 · p. 16
  • Completion can provide a complete target representation, unlike discarding occluded parts, which measures partial similarity inconsistently and may confuse people with similar visible parts.The conclusion argues that feature-level completion is preferable to image-level completion because it captures longer-range spatial and longer-term temporal context and is lightweight and end-to-end insertable.Source: §7 · p. 16
  • Future work will explore better integration of spatial and temporal completion and apply the method to occlusion scenes in tracking and detection.Source: §7 · p. 16

Related papers

Loading 2106.12733v1…