PaperlayerView exact revision on arXiv

Source-linked AI summary

MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask

Shengyu Zhao, Yilun Sheng, Yue Dong, Eric I-Chao Chang, Yan Xu

arXiv:2003.10955v2cs.CV

TL;DR

Occlusions make feature warping ambiguous by introducing useless or doubled information into warped features. MaskFlownet uses an asymmetric occlusion-aware matching module to learn an unsupervised mask and feeds it into a dual-pyramid cascade. The method reports superior optical-flow benchmark performance with negligible computational cost.

  • Problem

    Feature warping introduces unreliable and potentially doubled information in occluded areas, creating ambiguity for subsequent optical-flow inference.

  • Method

    AsymOFMM applies a learnable multiplicative occlusion mask after feature warping and uses asymmetric feature matching within an end-to-end two-stage MaskFlownet architecture.

  • Results

    MaskFlownet surpasses all published optical-flow methods on MPI Sintel, KITTI 2012, and KITTI 2015 while using two-frame inputs without occlusion-ground-truth training.

  • Takeaways & Limitations

    The approach provides an occlusion-aware feature-matching component with negligible computational cost and a promising basis for supervised and unsupervised optical-flow estimation.

Abstract

from arXiv · show

Feature warping is a core technique in optical flow estimation; however, the ambiguity caused by occluded areas during warping is a major problem that remains unsolved. In this paper, we propose an asymmetric occlusion-aware feature matching module, which can learn a rough occlusion mask that filters useless (occluded) areas immediately after feature warping without any explicit supervision. The proposed module can be easily integrated into end-to-end network architectures and enjoys performance gains while introducing negligible computational cost. The learned occlusion mask can be further fed into a subsequent network cascade with dual feature pyramids with which we achieve state-of-the-art performance. At the time of submission, our method, called MaskFlownet, surpasses all published optical flow methods on the MPI Sintel, KITTI 2012 and 2015 benchmarks. Code is available at https://github.com/microsoft/MaskFlownet.

1. Introduction

Optical flow relies on feature warping, but occlusions introduce ambiguous or doubled information in warped images and features. MaskFlownet learns an unsupervised occlusion mask and combines asymmetric matching with a two-stage dual-pyramid architecture.

  • Motivation: Feature warping resolves long-range matching but introduces unreliable, potentially doubled information in occluded areas.The ambiguity affects both warped images and warped feature maps, confusing subsequent flow inference.
  • MaskFlownet Architecture: MaskFlownet uses two stages: an end-to-end MaskFlownet-S network followed by a refinement cascade with dual feature pyramids.The learnable mask is predicted coarse-to-fine and fed into the occlusion-aware feature pyramid.
  • Learnable Occlusion Mask: The network learns to apply a multiplicative occlusion mask immediately after feature warping without explicit occlusion supervision.The resulting masked features have cleaner semantics for subsequent cost-volume processing.
  • Asymmetric Feature Matching: AsymOFMM combines occlusion-aware masking with asymmetric feature matching and integrates into end-to-end architectures at negligible computational cost.The paper reports performance gains from this combined design despite prior emphasis on feature symmetry.
  • Reported Results: At submission, MaskFlownet surpassed all published optical-flow methods on MPI Sintel and KITTI 2012 and 2015 using two-frame inputs without additional assumptions.The model was trained on standard optical-flow datasets without using occlusion ground truth.

2. Related Work

Prior optical-flow research developed efficient feature-pyramid, warping, and cost-volume systems, while occlusion estimation often required supervision or bidirectional flow. MaskFlownet addresses this context with asymmetric feature matching and jointly predicts occlusions without explicit supervision in one forward pass.

  • Optical Flow Estimation: Learning-based optical-flow methods advanced from end-to-end CNNs toward compact feature-pyramid, warping, and cost-volume architectures.PWC-Net and LiteFlowNet combined these components while preserving high efficiency.
  • Optical Flow Estimation: The paper focuses on feature matching before correlation and identifies asymmetric design as an underexplored source of improved performance.This process is described as independent of VCN’s cost-volume improvement.
  • Occlusions and Optical Flow: Earlier joint optical-flow and occlusion methods learned explicitly from ground-truth occlusions, requiring additional effort on the data.The supplied passage states this limitation for FlowNet, FlowNet2, and IRR-PWC.
  • Occlusions and Optical Flow: Unsupervised methods exclude occluded pixels from photometric losses, while forward-backward consistency requires bidirectional flow and can produce noisy predictions.The paper positions single-pass unsupervised occlusion estimation as a more flexible alternative.
  • Occlusion-Aware Techniques in Other Applications: Occlusion-aware techniques in other applications commonly use spatial attention masks that require additional supervising information.The related work mentions crowded pedestrian detection as an example.

3. Occlusion-Aware Feature Matching

Feature warping can create duplicated foreground information in occluded regions, confusing flow inference. OFMM and AsymOFMM address this by masking warped features and adding asymmetric processing.

  • The conventional feature matching module warps target feature maps using the flow displacement before correlation computes the cost volume.The feature extractor is shared across the two images at each pyramid level.
  • Warping can copy a moving foreground object into occluded background regions, introducing ambiguous features that confuse subsequent flow inference.The foreground moves substantially while the background remains still, producing a ghosting effect after warping.
  • OFMM element-wise multiplies warped features by a soft learnable occlusion mask θ and adds a feature tensor µ that supplies information in masked areas.θ has shape (B, 1, H, W), is broadcast across channels, and is constrained to [0, 1].
  • OFMM can learn a rough occlusion mask without explicit occlusion supervision, at negligible cost, for use in a subsequent occlusion-aware feature pyramid.The mask filters useless information immediately after feature warping.
  • AsymOFMM adds asymmetric processing only to the target feature path and implements it with a deformable convolution specialized around the flow displacement.The asymmetric design is motivated by warping-induced divergence between the two feature paths.

4. MaskFlownet

MaskFlownet uses cascaded subnetworks with asymmetric matching in its first stage and dual feature pyramids for refinement. The occlusion-aware pyramid complements the original pyramid during later flow inference.

  • MaskFlownet-S: MaskFlownet consists of two cascaded subnetworks, with MaskFlownet-S replacing PWC-Net’s feature matching modules by AsymOFMMs.The first stage predicts through a six-level shared feature pyramid in coarse-to-fine order.
  • MaskFlownet-S: At each level, the previous stage provides flow, mask, and trade-off predictions that are upsampled and fed into AsymOFMM.Flow and mask use bilinear interpolation, while the trade-off tensor is upsampled through deconvolution and convolution.
  • Occlusion-Aware Feature Pyramid: The learned occlusion mask is concatenated with the warped image and supplied to an occlusion-aware feature pyramid for subsequent flow refinement.The mask is shifted by 0.5 before concatenation, while a zero mask is paired with I1 for symmetricity.
  • Cascaded Flow Inference with Dual Pyramids: The cascade retains the original feature pyramid because the occlusion-aware pyramid cannot feed information back into new coarse-to-fine flow predictions.The two pyramids therefore provide complementary information during refinement.

5. Experiments

Experiments evaluate MaskFlownet through benchmark comparisons, qualitative visualizations, and ablations of feature matching, asymmetricity, masking, and network cascading. Results support the contributions of AsymOFMM, learnable occlusion masks, asymmetric features, and dual pyramids.

  • 5.2. Main Results: MaskFlownet outperforms all published optical flow methods on MPI Sintel, KITTI 2012, and KITTI 2015 benchmarks.Training-set values compare only models not trained on the corresponding set.
  • 5.2. Main Results: Qualitative comparisons show better separation of moving objects from backgrounds, weaker checkerboard artifacts, and preserved object boundaries.The comparisons include PWC-Net, MaskFlownet-S, and MaskFlownet on Sintel final-pass samples.
  • 5.2. Main Results: The learned occlusion mask matches the inverse ground-truth occlusion map fairly well despite being learned without explicit supervision.MaskFlownet jointly predicts optical flow and a rough occlusion mask.
  • 5.3. Ablation Study: AsymOFMM significantly outperforms symmetric feature-matching variants, while added depth has limited impact in the symmetric setting.The asymmetric design is evaluated against OFMM, OFMM + sym-conv, and OFMM + asym-conv.
  • 5.3. Ablation Study: Only the combination of the learnable mask and trade-off term produces performance gains, with the trade-off term supporting proper mask learning.Disabling either factor removes the observed gains.
  • 5.3. Ablation Study: Dual feature pyramids consistently benefit MaskFlownet over a single new pyramid, while concatenating the occlusion mask improves the Sintel final pass.The occlusion-aware pyramid smooths occluded areas during feature extraction, making them more distinguishable.

6. Conclusion

The paper proposes AsymOFMM, which learns an occlusion mask after feature warping without explicit supervision and integrates it into end-to-end architectures. MaskFlownet extends this module with dual pyramids and achieves superior performance across modern optical-flow benchmarks.

  • 6. Conclusion: AsymOFMM filters occluded areas immediately after feature warping using a learnable mask without explicit supervision.The module introduces negligible computational cost and integrates into end-to-end networks.
  • 6. Conclusion: MaskFlownet uses a two-stage architecture with dual pyramids and achieves superior performance on modern optical-flow benchmarks.

Appendix A. More Implementation Details

The appendix describes training schedules, augmentation procedures, and crop handling used across datasets and stages. It also references tabulated geometric and chromatic augmentation settings.

  • Training Schedule: MaskFlownet-S is trained on FlyingChairs and FlyingThings3D, with longer Sintel fine-tuning and a shorter second-stage FlyingChairs schedule.Sintel fine-tuning uses 1000k iterations; submission predictions average three independent runs.
  • Data Augmentation: Training uses geometric and chromatic augmentations with dataset-specific random crop sizes and valid-area constraints.Augmentation ranges are provided in separate geometric and chromatic tables.

Appendix B. More Visualizations

Visualizations reveal a scope boundary in the learned occlusion masks: image-boundary regions remain relatively vague because out-of-bound warped features are already zero.

  • Appendix B. More Visualizations: Learned occlusion masks are relatively vague at image boundaries because the network cannot mask out-of-bound features that are already zeros.The authors expect improvement from manually treating out-of-bound areas as occlusions.

Appendix C. Screenshots on Benchmarks

MaskFlownet ranks first on both MPI Sintel passes and surpasses published optical-flow methods on KITTI 2012 and 2015 at submission time. The appendix also visualizes its learned masks and benchmark comparisons, while noting that some KITTI entries are not directly comparable.

  • MPI Sintel: MaskFlownet ranks first on MPI Sintel's clean and final passes at the time of submission.The clean-pass comparison includes MR-Flow, and the final-pass comparison includes SelFlow, which use additional assumptions or inputs.
  • KITTI benchmarks: MaskFlownet surpasses all optical-flow methods on KITTI 2012 and 2015, excluding anonymous entries.The top three KITTI 2015 entries are scene-flow methods using stereo images and are therefore not comparable.
  • Qualitative comparisons: The appendix includes visualizations of learned occlusion masks and qualitative comparisons among PWC-Net, MaskFlownet-S, and MaskFlownet.The masks are expected to roughly match inverse ground-truth occlusion maps despite being learned without explicit supervision.
  • Benchmark reporting: KITTI evaluation tables report optical-flow outlier rates for background, foreground, and all evaluated pixels, together with density and runtime.The listed format includes Fl-bg, Fl-fg, Fl-all, Density, Runtime, and Environment.
Loading 2003.10955v2…