PaperlayerView exact revision on arXiv

Source-linked AI summary

VRSTC: Occlusion-Free Video Person Re-Identification

Ruibing Hou, Bingpeng Ma, Hong Chang, Xinqian Gu, Shiguang Shan, Xilin Chen

arXiv:1907.08427v1cs.CV

TL;DR

Partial occlusion severely degrades video person re-ID, while existing approaches may discard occluded frames and lose useful appearance cues. The paper proposes STCnet, which recovers occluded parts from spatial and temporal information, and combines it with re-ID as VRSTC. Experiments on three challenging datasets show that the framework outperforms state-of-the-art video re-ID approaches.

  • Problem

    Partial occlusion severely degrades video person re-ID, and methods that discard occluded frames can lose useful visible appearance information.

  • Method

    STCnet recovers occluded pedestrian parts using spatial structure and adjacent-frame temporal patterns, while VRSTC combines it with a re-ID network.

  • Results

    The proposed framework outperforms state-of-the-art video re-ID approaches on three challenging datasets.

  • Takeaways & Limitations

    Recovered occluded regions are leveraged with unoccluded regions to support robust video re-ID under partial occlusion.

  • Takeaways & Limitations

    The authors identify extremely severe occlusion as a future setting requiring other deep generative architectures.

Abstract

from arXiv · show

Video person re-identification (re-ID) plays an important role in surveillance video analysis. However, the performance of video re-ID degenerates severely under partial occlusion. In this paper, we propose a novel network, called Spatio-Temporal Completion network (STCnet), to explicitly handle partial occlusion problem. Different from most previous works that discard the occluded frames, STCnet can recover the appearance of the occluded parts. For one thing, the spatial structure of a pedestrian frame can be used to predict the occluded body parts from the unoccluded body parts of this frame. For another, the temporal patterns of pedestrian sequence provide important clues to generate the contents of occluded parts. With the Spatio-temporal information, STCnet can recover the appearance for the occluded parts, which could be leveraged with those unoccluded parts for more accurate video re-ID. By combining a re-ID network with STCnet, a video re-ID framework robust to partial occlusion (VRSTC) is proposed. Experiments on three challenging video re-ID databases demonstrate that the proposed approach outperforms the state-of-the-art.

1. Introduction

Partial occlusion severely degrades video person re-ID because common methods treat frames equally or discard occluded frames, losing useful appearance information. STCnet instead recovers occluded body parts from spatial structure and adjacent-frame temporal patterns, and VRSTC combines it with re-ID.

  • Partial occlusion severely degrades video re-ID because any body part may be hidden by pedestrians or environmental objects.
  • Equal treatment of frames corrupts aggregated video features when partial occlusion affects typical video re-ID methods.
  • Attention-based methods tolerate occlusion but discard frames whose remaining visible portions may still provide strong re-ID cues.
  • STCnet predicts occluded parts from visible body structure and refines them using temporal information from adjacent frames.
  • VRSTC combines STCnet with a re-ID network and reports robustness to partial occlusion across three challenging video re-ID datasets.

2. Related Works

Prior video re-ID methods use temporal modeling and attention, but occlusion handling commonly discards affected parts or frames. The paper positions recovery of occluded content as an alternative that retains spatial and temporal information.

  • Early video re-ID pipelines extract frame features, incorporate temporal context with recurrent layers, and aggregate representations through temporal average pooling.
  • Attention-based approaches select discriminative frames or regions to reduce the influence of partial occlusion.
  • Existing occlusion-handling methods discard occluded parts, losing spatial and temporal information from videos.
  • Image completion research includes patch matching and GAN-based methods that synthesize missing regions conditioned on surrounding content.

3. Spatial-Temporal Completion network

STCnet completes occluded pedestrian content using spatial structure and adjacent-frame temporal patterns. Its generators, discriminators, and identity guider are jointly trained so completed frames preserve structure, realism, and identity cues.

  • Architecture: STCnet uses a spatial structure generator, temporal attention generator, two discriminators, and an ID guider subnetwork.
  • Spatial Structure Generator: The spatial generator predicts occluded content from visible frame regions, using an autoencoder with dilated convolutions and a decoder that restores resolution.
  • Temporal Attention Generator: The temporal attention layer compares current and adjacent-frame patches with cosine similarity, then updates each current patch through weighted aggregation.
  • Temporal Attention Generator: Three temporal-generator encoders process the occluded and adjacent unoccluded frames, whose features are attended, concatenated, and decoded into the final output.
  • Supporting Networks: Local and global discriminators respectively improve occluded-part detail and whole-frame structure, while the ID guider preserves identity cues in completed frames.
  • Objective Function: STCnet jointly optimizes reconstruction, adversarial, and guider losses, replacing generated non-mask pixels with the original unoccluded pixels.
  • Objective Function: The guider loss is cross-entropy, and λ1 and λ2 balance adversarial and guider losses in the overall objective.

4. Occlusion-Free Video Person Re-ID

VRSTC combines similarity-based occlusion localization, STCnet completion, and re-ID learning to use recovered and unoccluded regions. Its design integrates spatial and temporal information while retaining a relatively simple re-ID network.

  • Framework: VRSTC scores frame regions, replaces low-scoring occluded regions with STCnet outputs, and trains the re-ID network using recovered and unoccluded regions.Regions below threshold τ are treated as occluded and replaced before re-ID training.
  • Similarity Scoring: The similarity scoring mechanism generates attention scores for individual frame regions using cosine similarity between frame-region and video-region features.The score is designed to identify regions whose semantic features differ from the original body parts.
  • Similarity Scoring: Frames are divided into upper, middle, and lower regions, whose convolutional features are pooled over time to obtain video-region features.The supplied description specifies three fixed vertical regions and temporal average pooling.
  • Re-ID Network: STCnet can be combined with most re-ID networks and loss functions, while the described implementation uses average temporal pooling and cross-entropy loss.The framework uses a simple re-ID network for evaluating STCnet as a data-enhancement method.
  • Re-ID Network: The re-ID network embeds non-local blocks earlier in the network to capture richer hierarchical temporal dependencies combining local and non-local information.This differs from approaches that build temporal dependency only near the network end.

5.1. Datasets and evaluation protocols

The evaluation uses three video re-ID datasets with varied camera, identity, sequence, and tracklet characteristics. Performance is measured with mAP and CMC metrics.

  • Datasets: MARS contains 1,261 identities and around 20,000 video sequences captured from six cameras.Its bounding boxes are produced by a DPM detector and GMMCP tracker.
  • Datasets: DukeMTMC-VideoReID tracklets are generated by cropping pedestrian images from videos at 12 frames per second.The dataset is a video person re-ID subset of the DukeMTMC tracking dataset.
  • Evaluation: Evaluation uses mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) as the reported metrics.These metrics assess retrieval performance and cumulative matching accuracy.

5.2. Implementation Details

The implementation uses PyTorch, a ResNet-based identity guider, similarity scoring for occlusion localization, and randomly masked unoccluded frames for STCnet training. Completed regions are then used to train and test the re-ID network.

  • Implementation: All experiments are implemented in PyTorch.The supplied implementation description identifies PyTorch as the experimental framework.
  • Pre-training: A ResNet-50 trained with cross-entropy loss serves as the identity guider, using four-frame tracks and average temporal pooling.Input images are resized to 256 × 128, with batch size 32 and random horizontal mirroring for augmentation.
  • Occlusion Localization: The similarity scoring mechanism identifies occluded regions as those with scores below threshold τ, which is set to 0.89.Frames without detected occluded regions are designated unoccluded frames.
  • Evaluation: Table 1 reports rank-1 CMC accuracy and mAP for MARS and DukeMTMC-VideoReID in brackets.The table is used for comparative analysis of STCnet.
  • STCnet Training: STCnet is trained without ground-truth occluded frames by randomly masking regions of unoccluded training frames as inputs and using the original frames as targets.The ID guider parameters remain fixed during STCnet training.
  • Re-ID Training: Generated regions replace occluded regions to form a new dataset, which trains and tests the re-ID network with embedded non-local blocks.The non-local blocks capture temporal dependencies in the input sequence.

5.3. Ablation Study

The ablation study shows that spatial and temporal completion improve video re-ID, while temporal attention contributes beyond simply increasing model depth. Similarity-based occlusion handling performs best at a moderate threshold, and completed frames further improve identification.

  • Spatial structure generator: Spatial completion improves rank-1 accuracy by 1.3%, 0.9% and 1.1% on iLIDS-VID, MARS and DukeMTMC-VideoReID respectively.The spatial generator uses frame-level spatial information to recover occluded regions.
  • Temporal attention generator: Temporal completion further improves accuracy over spatial completion by using adjacent-frame information to make generated frames more semantically consistent.The resulting sequences provide better temporal information for extracting discriminative video representations.
  • Temporal attention generator: Temporal-generator gains are not explained by added depth: Spa+AE does not improve over Spa, while the temporal-attention model outperforms TAE.These comparisons support the specific contribution of temporal attention rather than merely additional layers.
  • Discriminators: Discriminators only slightly improve performance, suggesting they increase visual realism without adding much discriminative information for re-ID.Both local and global discriminators are trained with reconstruction and adversarial losses.
  • ID guider network: The ID guider improves generated-sample performance by preserving visual cues associated with identity labels.The final STCnet combines reconstruction, adversarial and guider losses.
  • Parameter sensitivity: The best performance occurs at λ1=0.001 and λ2=0.1; excessively large weights cause degradation because STCnet becomes difficult to converge.Figure 4 varies λ1 with λ2 fixed and λ2 with λ1 fixed on DukeMTMC-VideoReID.
  • Similarity scoring threshold: Increasing the occlusion threshold initially improves performance, but excessive thresholds discard unoccluded frames; the best setting is τ=0.89.At τ=0, all frames are retained, matching the baseline model.
  • Similarity scoring threshold: STCnet-completed frames further improve performance, demonstrating that restored contents help identify the person.The similarity scores also reflect region visibility, with occluded lower parts receiving small scores in the first four frames of one sequence.

5.4. Comparison with State-of-the-arts

VRSTC outperforms existing methods on MARS and DukeMTMC-VideoReID, while achieving slightly lower performance than Snippet+OF on iLIDS-VID but outperforming Snippet without optical flow.

  • 5.4. Comparison with State-of-the-arts: VRSTC outperforms the best existing methods on MARS and DukeMTMC-VideoReID.The authors attribute these improvements to recovering occluded contents and combining STCnet with re-ID.
  • 5.4. Comparison with State-of-the-arts: On DukeMTMC-VideoReID, the baseline outperforms by 7.8% in rank-1 and 11.7% in mAP.
  • 5.4. Comparison with State-of-the-arts: On iLIDS-VID, VRSTC performs slightly below Snippet+OF, which uses additional optical flow unavailable to the proposed framework.
  • 5.4. Comparison with State-of-the-arts: VRSTC significantly outperforms Snippet without optical flow, providing a fairer comparison.

5.5. Visualizing the effect of STCnet

Visualizations show that STCnet recovers contents in occluded regions and helps the re-ID model focus on more effective regions and discriminative clues.

  • 5.5. Visualizing the effect of STCnet: STCnet recovers contents in regions occluded by pedestrians, indicators, and bicycles.The visualizations compare original images, completed outputs, and their activation maps.
  • 5.5. Visualizing the effect of STCnet: Before completion, appearances of other pedestrians can activate the re-ID network and corrupt the target person's feature representation.
  • 5.5. Visualizing the effect of STCnet: After completion, the re-ID model considers more effective regions and discovers new discriminative clues for recognizing the person.

6. Conclusion

The paper combines STCnet with a re-ID network to address partial occlusion by recovering occluded regions and leveraging them with unoccluded regions. Experiments on three datasets show that the method outperforms state-of-the-art video re-ID approaches, while future work targets extremely severe occlusion.

  • 6. Conclusion: The proposed framework combines a re-ID network with STCnet for video re-ID under partial occlusion.
  • 6. Conclusion: STCnet recovers occluded-region appearance, which is leveraged with unoccluded regions to train the re-ID network.
  • 6. Conclusion: Experiments on three datasets show that the proposed method outperforms state-of-the-art video re-ID approaches.
  • 6. Conclusion: Future work will explore other deep generative architectures for frames with extremely severe occlusion.
Loading 1907.08427v1…