PaperlayerView exact revision on arXiv

Source-linked AI summary

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation

Junhwa Hur, Stefan Roth

arXiv:1904.05290v1cs.CVcs.LG

TL;DR

Optical-flow networks often refine estimates through stacked or coarse-to-fine modules, but this increases parameter counts. The paper introduces shared-weight iterative residual refinement with joint occlusion and bidirectional estimation. It improves flow accuracy across backbones while reducing PWC-Net size and achieves state-of-the-art flow and occlusion results.

  • Problem

    Stacked or pyramid-based refinement improves optical-flow estimates but uses separate trainable weights, increasing model parameters and complicating training.

  • Method

    IRR repeatedly refines residual flow with a single shared-weight network block and integrates occlusion and bidirectional flow estimation.

  • Results

    17.7% average flow-accuracy improvement is reported for PWC-Net while reducing parameters by 26.4%, and the full design reaches state-of-the-art flow and occlusion results.

  • Takeaways & Limitations

    IRR improves accuracy and generalization across representative flow networks while reducing parameters for PWC-Net.

  • Takeaways & Limitations

    On Sintel and KITTI, supervision uses only forward flow and, where available, the first-frame occlusion map.

Abstract

from arXiv · show

Deep learning approaches to optical flow estimation have seen rapid progress over the recent years. One common trait of many networks is that they refine an initial flow estimate either through multiple stages or across the levels of a coarse-to-fine representation. While leading to more accurate results, the downside of this is an increased number of parameters. Taking inspiration from both classical energy minimization approaches as well as residual networks, we propose an iterative residual refinement (IRR) scheme based on weight sharing that can be combined with several backbone networks. It reduces the number of parameters, improves the accuracy, or even achieves both. Moreover, we show that integrating occlusion prediction and bi-directional flow estimation into our IRR scheme can further boost the accuracy. Our full network achieves state-of-the-art results for both optical flow and occlusion estimation across several standard datasets.

1. Introduction

Deep optical-flow networks often refine estimates through stacked or coarse-to-fine modules, but separately trained weights increase model size and training complexity. The paper proposes shared-weight iterative residual refinement, extending it with occlusion and bidirectional estimation to improve accuracy and efficiency.

  • Motivation: Stacking FlowNet-family networks refines previous estimates but linearly increases parameters and requires sequential stage training.Later modules use their own trainable weights, creating a burden for adoption in other applications.
  • Proposed approach: The proposed IRR scheme repeatedly refines a previous output using one network block with shared weights.This design is inspired by iterative energy minimization and residual networks.
  • Backbone integration: IRR applies to FlowNet and PWC-Net, increasing FlowNet accuracy without added parameters and reducing PWC-Net parameters while improving accuracy.The scheme is presented as applicable to both backbones.
  • Joint estimation: Occlusion estimation and bidirectional flow are integrated into IRR, with the full model combining IRR, Bi, and Occ for improved accuracy and fewer parameters than PWC-Net.The paper also introduces lightweight bilateral filtering and occlusion upsampling layers for refined boundaries.
  • Results: 18.5% and 17.7% flow-accuracy improvements are reported for FlowNet and PWC-Net, respectively, across multiple datasets.For PWC-Net, the gain uses 26.4% fewer parameters.

2. Related Work

Related optical-flow methods use autoencoders, classical optimization, coarse-to-fine residual updates, or stacked networks, while occlusion reasoning supports more accurate correspondence handling. This paper investigates a broadly applicable iterative residual scheme that reuses one baseline network without increasing network size.

  • Optical-flow architectures: CNN optical-flow methods provide supervised end-to-end training and fast inference, while optimization-based approaches can yield less blurry results but have longer testing runtimes.The related approaches include both fully learned and hybrid CNN-optimization designs.
  • Occlusion reasoning: Occlusion modeling can support flow estimation through more accurate matching costs, bidirectional consistency, and pixel-level uniqueness constraints.Several prior methods jointly estimate occlusions and flow or use occlusion reasoning in energy formulations.
  • Iterative and residual refinement: SpyNet and PWC-Net residually update flow across pyramid levels, whereas FlowNet2 stacks multiple networks to refine previous estimates.These approaches are motivated by classical coarse-to-fine refinement.
  • Paper's approach: The proposed generic scheme repeatedly uses one baseline network to improve accuracy without increasing network size.For some networks, repetitive modules can be removed while retaining competitive or improved accuracy.

3. Approach

The approach applies iterative residual refinement with shared weights to flow backbones, then extends joint flow–occlusion estimation with bidirectional estimation, bilateral refinement, and full-resolution occlusion upsampling.

  • Core concepts & base networks: IRR reuses one network block with shared weights to iteratively refine the previous flow estimate.For FlowNet, the final flow is the sum of residual flows across iterations, with feature warping before decoding.
  • Core concepts & base networks: PWC-Net replaces its separate pyramid-level decoders with one shared decoder whose iterations match the number of pyramid levels.Estimating flow at each level’s native resolution and using fixed input-channel dimensions enables this shared design.
  • Joint optical flow and occlusion estimation: Joint estimation adds an occlusion decoder alongside the flow decoder, with the same configuration but one output channel for occlusion.The occlusion decoder receives the same input as the flow decoder.
  • Joint optical flow and occlusion estimation: Bidirectional estimation reuses shared decoders by switching input order, while forward-backward consistency is crucial for more accurate occlusion estimation.Bidirectional flow itself yields at most minor accuracy improvements, whereas its consistency supports occlusion estimation.
  • Joint optical flow and occlusion estimation: Bilateral filters refine flow and occlusion across iterations using shared weights, with separate kernels because motion and occlusion boundaries need not align.The filters are individualized to each pixel and applied separately to flow components and occlusion.
  • Joint optical flow and occlusion estimation: Quarter-resolution occlusion output loses fine occlusions, motivating an upsampling layer that refines occlusion at full output resolution.An oracle reconstruction from quarter-resolution Sintel occlusions achieves an F-score of 0.777.

4. Experiments

Experiments evaluate the proposed components on synthetic and public benchmarks for optical flow, occlusion estimation, and cross-dataset generalization. IRR and its extensions improve accuracy while reducing parameters and avoiding sequential multi-stage training.

  • 4.1. FlyingChairsOcc dataset: FlyingChairsOcc provides supervision for forward and backward flow plus occlusion maps at both frames.The dataset follows the FlyingChairs generation protocol and contains 22,232 training and 640 validation images.
  • 4.2. Training details: The model remains trainable with one-direction ground truth, using a shared unidirectional decoder for available supervision.On Sintel, supervision uses forward flow and the first-frame occlusion map; on KITTI, only forward flow is used.
  • 4.3. Ablation study: Up to 10% EPE improvement across datasets is attributed to IRR, while FlowNet with Bi, Occ, and IRR reaches up to 20% improvement on Sintel Clean.The ablation reports consistent training-domain gains and better generalization across datasets.
  • 4.3. Ablation study: 17.7% average accuracy improvement over the PWC-Net baseline is achieved with 26.4% fewer parameters.The full IRR-PWC includes bilateral refinement and occlusion upsampling; these add only 0.83M parameters.
  • 4.3. Ablation study: IRR continues improving with more FlowNet steps, whereas stacked FlowNetS overfits after three steps and is consistently outperformed at equal stage counts.IRR achieves better accuracy without linearly increasing the number of parameters.
  • 4.4. Optical flow benchmarks: On Sintel and KITTI, IRR-PWC outperforms published two-frame methods and improves over PWC-Net by 9.18% on Sintel Final and 12.36% on Clean.The model is trained all at once rather than using piecewise training for the refinement stages.
  • 4.5. Occlusion estimation: Occlusion estimation achieves state-of-the-art accuracy on Sintel Training using 6.00M parameters instead of 110M.No public occlusion benchmark is available, so evaluation uses the Sintel training set.

5. Conclusion

The paper concludes that weight-shared IRR improves optical-flow accuracy and generalization across FlowNet and PWC-Net, while reducing PWC-Net parameters. Joint occlusion and bidirectional estimation further improve performance and achieve state-of-the-art benchmark results.

  • 5. Conclusion: IRR applies weight-shared iterative residual refinement to generic optical-flow networks, with bidirectional and occlusion-estimation components.The scheme is demonstrated on FlowNet and PWC-Net.
  • 5. Conclusion: The approach improves flow accuracy and generalization while reducing parameters for PWC-Net.The conclusion also reports state-of-the-art results when occlusion is jointly estimated with flow.

– Supplementary Material –

The supplementary material provides implementation details for IRR-PWC and additional qualitative evaluations. It covers the model’s occlusion upsampling layer, ablation examples, and comparisons with other methods.

  • Supplementary Material: The supplement describes IRR-PWC’s occlusion upsampling layer and residual blocks.
  • Supplementary Material: Additional qualitative examples evaluate the ablation study and compare the model with the state of the art.

A. IRR-PWC

IRR-PWC jointly estimates optical flow and occlusion using bidirectional estimation, bilateral refinement, and an occlusion upsampling layer. It iteratively estimates these outputs through a feature pyramid and restores them to the input resolution.

  • A. IRR-PWC: IRR-PWC jointly estimates optical flow and occlusion with bidirectional estimation, bilateral refinement, and occlusion upsampling.
  • A. IRR-PWC: The model iteratively and residually estimates flow and occlusion to quarter resolution before upsampling to the original resolution.Applying the upsampling layer at the sixth and seventh pyramid levels restores the quarter-resolution estimate.

B. Details on the Occlusion Upsampling Layer

The occlusion upsampling layer refines coarse occlusion maps using weight-shared residual blocks, while sharing weights across directions, pyramid levels, and iteration steps. This design improves occlusion accuracy with minimal parameter growth.

  • Occlusion upsampling layer: The occlusion upsampling layer refines an upscaled occlusion map by predicting residual occlusion estimates from feature maps.Its residual subnetwork contains three residual blocks, each with three convolution layers.
  • IRR-PWC design: The full IRR-PWC jointly estimates flow and occlusion to quarter resolution, then upsamples both outputs to the original resolution while improving accuracy.The occlusion upsampling layer specifically upscales the outputs and improves accuracy during this final stage.
  • Weight sharing: Weights are shared across bi-directional estimations, pyramid levels or iteration steps, and the residual blocks themselves.Sharing the residual-block weights differs from the cited prior design, where those weights were not shared.
  • Accuracy and parameters: 2.99% training-domain and 4.08% cross-dataset occlusion-accuracy gains require only 0.031 M additional parameters.The gains are reported on FlyingChairsOcc and Sintel, respectively.

C. Additional Qualitative Examples

Qualitative experiments show that occlusion upsampling sharpens boundaries and recovers thin occlusions, while the proposed PWC-Net schemes improve flow accuracy and cross-dataset generalization.

  • Occlusion upsampling: Occlusion upsampling produces sharper estimates along motion boundaries and refines coarse occlusion predictions.The models were trained on FlyingChairsOcc and tested on Sintel Train Clean without additional fine-tuning.
  • PWC-Net ablation: The proposed PWC-Net schemes significantly improve accuracy over the PWC-Net baseline and generalize better across datasets.The qualitative ablation models were trained on FlyingChairsOcc and tested on Sintel Train Clean.

D.1. Occlusion estimation

Against prior methods, the proposed approach achieves stronger qualitative occlusion estimation on Sintel, balancing higher recall against somewhat lower precision than FlowNet-CSSR-ft-sd.

  • Occlusion estimation: The proposed method misses fewer occlusions than MirrorFlow and achieves a better F1-score than FlowNet-CSSR-ft-sd.It reports state-of-the-art occlusion results on Sintel Train Clean and Final.
  • Occlusion estimation: Compared with FlowNet-CSSR-ft-sd, the method favors recall over precision by trying not to miss occlusions.FlowNet-CSSR-ft-sd is described as detecting fine occlusion details, while also receiving additional ChairsSDHom training.

D.2. Bi-directional flows and occlusion maps

Qualitative comparisons examine occlusion upsampling, ablations, and bi-directional flow and occlusion maps. The full model improves motion-boundary detail and produces fewer artifacts than the compared methods.

  • Bi-directional flow and occlusion: Compared with MirrorFlow, the proposed model shows fewer artifacts and fewer missing details in both flow and occlusion estimation.The comparison uses models fine-tuned on Sintel and KITTI 2015 training sets and examples from their validation splits.
  • Occlusion upsampling: The occlusion upsampling layer makes occlusion boundaries sharper and detects additional thinly shaped occlusions.The examples compare overlapped inputs, ground truth, and predictions without versus with the layer.
  • PWC-Net ablation: The full IRR-PWC model improves flow estimation over original PWC-Net, with fewer missing details and clearer motion boundaries.The ablation includes bidirectional, occlusion, and IRR variants before the full model.
  • Occlusion estimation: Figure 13 encodes false positives in blue, false negatives in red, and correctly estimated occlusions in white, with each method’s F-score shown.The compared methods are MirrorFlow, FlowNet-CSSR-ft-sd, and the proposed approach.
  • Bi-directional flow and occlusion: Figure 14 compares ground-truth and predicted forward and backward flows alongside occlusion maps for both views.The results are overlaid on the corresponding first- or second-frame image.
Loading 1904.05290v1…