PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Accurate Dense Correspondences and When to Trust Them

Prune Truong, Martin Danelljan, Luc Van Gool, Radu Timofte

arXiv:2101.01710v2cs.CV

TL;DR

Dense correspondence estimates become unreliable with large displacements, appearance changes, homogeneous regions, and occlusions, yet downstream tasks need to know which matches are trustworthy. PDC-Net jointly models flow and uncertainty with a constrained mixture and specialized architecture and self-supervision, achieving state-of-the-art results across geometric matching and optical flow while improving pose estimation.

  • Problem

    Dense flow estimation can be inaccurate or ill-defined in difficult image regions, creating a need for reliable pixel-wise confidence estimates for downstream tasks.

  • Method

    PDC-Net jointly predicts dense flow and uncertainty using a constrained mixture of Laplace distributions, an uncertainty architecture, and improved self-supervised training.

  • Results

    PDC-Net sets a new state-of-the-art on multiple geometric matching and optical flow datasets and outperforms dense matching methods on pose estimation.

  • Takeaways & Limitations

    The predicted confidence identifies accurate matches for downstream use, including pose estimation and direct 3D reconstruction.

  • Takeaways & Limitations

    The probabilistic formulation assumes conditional independence across spatial locations and, for simplicity, a single variance parameter.

Abstract

from arXiv · show

Establishing dense correspondences between a pair of images is an important and general problem. However, dense flow estimation is often inaccurate in the case of large displacements or homogeneous regions. For most applications and down-stream tasks, such as pose estimation, image manipulation, or 3D reconstruction, it is crucial to know when and where to trust the estimated matches. In this work, we aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map indicating the reliability and accuracy of the prediction. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets. We further validate the usefulness of our probabilistic confidence estimation for the task of pose estimation. Code and models are available at https://github.com/PruneTruong/PDCNet.

1. Introduction

Dense correspondence estimation supports many vision applications but becomes unreliable under large displacements, appearance changes, homogeneous regions, and occlusions. PDC-Net jointly predicts correspondences and uncertainty, using probabilistic modelling and improved self-supervision to identify which matches can be trusted.

  • Dense pixel-wise correspondences support 3D reconstruction, video analysis, image registration, image manipulation, and texture or style transfer.
  • Large displacements, appearance changes, homogeneous regions, and occlusions make dense flow predictions inaccurate or ill-defined.
  • Robust confidence estimates are crucial because pose estimation, 3D reconstruction, and image-based localization require highly accurate matches.
  • PDC-Net jointly learns dense flow and uncertainty through a conditional flow-density model parametrized as a constrained mixture.
  • The method combines an architecture exploiting correlation information with improved self-supervised data generation for robust, generalizable uncertainty prediction.
  • PDC-Net reports state-of-the-art performance on Megadepth and KITTI-2015, outperforms previous dense methods for YFCC100M pose estimation, and supports direct 3D reconstruction.

2. Related work

Prior confidence-estimation methods for dense matching and optical flow are limited by resolution, task scope, or the type of uncertainty they model. The paper positions its approach as a unified treatment of reliable dense correspondences and uncertainty for challenging geometric matching.

  • Confidence estimation has received limited attention in dense geometric and semantic matching compared with correspondence prediction itself.
  • Some methods encode confidence in 4D correspondence volumes, but their usual low-resolution restriction can hinder accuracy.
  • DGC-Net predicts dense correspondence and matchability maps, but trains matchability only to identify out-of-view pixels rather than actual match reliability.
  • Optical-flow uncertainty methods include post-processing approaches and probabilistic models targeting epistemic or aleatoric uncertainty.
  • Existing optical-flow uncertainty methods are designed for optical-flow data and restricted to small displacements, motivating a broader approach.

3. Our Approach: PDC-Net

PDC-Net jointly estimates dense flow and pixel-wise uncertainty through a probabilistic predictive distribution. Its constrained mixture model separates accurate predictions from outliers, while its architecture and training strategy target robust uncertainty generalization for real image pairs.

  • Probabilistic flow regression: PDC-Net predicts a dense flow field together with a pixel-wise confidence map indicating correspondence reliability and accuracy.The probabilistic formulation jointly learns flow and uncertainty for downstream geometric matching tasks.
  • Probabilistic flow regression: A predictive density replaces a single flow output, allowing the network to encode uncertainty in addition to the estimated correspondence.The model predicts distribution parameters at each spatial location rather than only a deterministic flow vector.
  • Constrained mixture model: A constrained mixture models inliers and outliers with shared mean flow but distinct variance ranges, assigning components to progressively larger uncertainties.The smallest-variance component represents accurate predictions, while the largest-variance component models large errors and outliers.
  • Constrained mixture model: The variance ordering breaks mixture permutation invariance, simplifying learning by assigning each component a specified uncertainty interval.The constraints order component variances and let the network use component weights as probabilities over uncertainty intervals.
  • Uncertainty prediction architecture: The architecture independently processes each correlation-volume slice to assess local matching reliability, then combines that information with mean flow to predict mixture parameters.This design addresses overly confident predictions caused by relying mainly on neighborhood information.
  • Self-supervised training: Flow perturbations and improved self-supervised data generation target uncertainty robustness beyond synthetic motion patterns and global smoothness assumptions.The method addresses remaining generalization failures around object boundaries and on real image pairs.
  • Geometric matching inference: The confidence estimates identify accurate correspondences that can initialize a coarse transformation before a second pass estimates the final flow.The final field is formed by composing the fine flow with the coarse transformation.

4. Experimental results

PDC-Net is evaluated on geometric matching, optical flow, uncertainty estimation, pose estimation, and dense 3D reconstruction. It consistently improves correspondence quality and uncertainty-aware downstream performance across challenging datasets and ablations.

  • Geometric Correspondences and Flow: PDC-Net outperforms previous methods on MegaDepth and RobotCar at all reported PCK thresholds.It is also more accurate and robust than RANSAC-Flow and GLU-Net-GOCor* in pure flow accuracy.
  • Generalization to optical flow: PDC-Net outperforms generic matching and specialized optical-flow methods on KITTI without fine-tuning.It improves both F1 and AEPE and surpasses RAFT on KITTI-2015.
  • Pose estimation: PDC-Net achieves similar pose-estimation results to RANSAC-Flow while running 12.2 times faster, and its multi-scale strategy is 3.6 times faster while outperforming it.Unlike RANSAC-Flow, PDC-Net directly estimates robust confidence maps without an additional semantic-segmentation network.
  • Dense 3D reconstruction: Confidence-thresholded correspondences are used with COLMAP to build a dense 3D point cloud from Aachen city images.The reconstruction demonstrates qualitative usability for dense 3D reconstruction.
  • Ablation study: Constraining the Laplace mixture improves both flow and uncertainty metrics compared with an unconstrained mixture.The unconstrained mixture's permutation invariance produces poor uncertainty estimates, whereas the constrained formulation avoids this issue.
  • Ablation study: The final uncertainty architecture balances correlation-based filtering with mean-flow information, while perturbation data improves uncertainty and flow for PDC-Net-s.Correlation alone struggles with independently moving objects, and perturbations are especially beneficial for the probabilistic model.

5. Conclusion

The paper presents PDC-Net as a probabilistic network for dense correspondences and confidence estimation. It reports state-of-the-art results across geometric matching and optical flow, with further utility for pose estimation and dense 3D reconstruction.

  • Conclusion: PDC-Net predicts dense image-to-image correspondences and confidence through a constrained mixture of Laplace distributions.The architecture and self-supervised training strategy target robust and generalizable uncertainty prediction.
  • Conclusion: PDC-Net achieves state-of-the-art results on multiple geometric matching and optical-flow datasets.The paper also reports improvements over dense matching methods for pose estimation.
  • Conclusion: The method is also applied to pose estimation and dense 3D reconstruction.These applications use the predicted correspondence confidence to support downstream geometric tasks.

A. Detailed derivation of probabilistic model

The probabilistic model represents flow as a constrained mixture of Laplace distributions and derives a confidence value from the predictive distribution. Training uses a numerically stable negative log-likelihood that can down-weight large-error examples.

  • Probabilistic formulation: The flow predictive distribution is modeled as a constrained mixture of bivariate Laplace components with independent flow-direction factors.Equal variances are imposed across the two flow directions.
  • Confidence estimation: The network outputs the mean flow and variance parameters of the predictive distribution for each pixel.These parameters define the uncertainty associated with the estimated flow vector.
  • Confidence estimation: Confidence is computed as the probability that the true flow lies within radius R of the predicted mean flow.The resulting probability PR can be thresholded to identify accurate matches.
  • Probabilistic training: The negative log-likelihood provides the training framework for fitting the predictive distribution.A numerically stable logsumexp implementation is used for the mixture loss.
  • Probabilistic training: Compared with L1 regression, the probabilistic loss can down-weight large-error examples by assigning them high variance.This lets training focus more on accurate predictions while modeling heavy-tailed errors.

B.2. Training datasets

Training combines synthetic image warps, independently moving objects, local perturbations, and sparse real correspondences. The combined data is important for obtaining reliable uncertainty estimates, especially during the second training stage.

  • Synthetic training data: Self-supervised training uses random homographies, independently moving COCO objects, and local perturbations to diversify motion and appearance.The perturbations are designed to challenge confidence prediction in otherwise homogeneous regions.
  • Perturbation data: Local residual flows are applied to selected regions of the reference image while the query image remains unchanged.The perturbed flow is formed by composing the base flow with the residual deformation.
  • Perturbation data: Smooth Gaussian masks create localized perturbations with gradual transitions between perturbed and unperturbed regions.This makes the perturbations visually subtle while preserving their training effect.
  • MegaDepth training: MegaDepth training uses reconstructed scenes, sparse SfM correspondences, depth checks, and occlusion filtering to form real image pairs.The procedure samples overlapping images and removes occluded pixels before resizing the pairs.
  • MegaDepth training: Training solely on sparse MegaDepth correspondences produces less reliable uncertainty estimates.The second training stage therefore combines sparse real correspondences with the synthetic perturbation data.

C.1. Architecture of the uncertainty decoder

The uncertainty decoder processes correlation information per pixel and combines it with flow-decoder features to predict constrained mixture-distribution parameters across pyramid levels. Training and inference use multi-scale losses and confidence-based correspondence filtering.

  • Correlation uncertainty module: The correlation uncertainty module independently processes each pixel’s 2D correspondence slice over displacement dimensions to produce an uncertainty representation.This design uses the correlation slice’s matching information while avoiding spatial-neighborhood reasoning.
  • Uncertainty predictor: The uncertainty predictor combines decoder flow features, correlation uncertainty features, and predictions from the previous pyramid level.In practice, the second-last flow-decoder layer provides richer information than the final flow field alone.
  • Uncertainty predictor: The uncertainty predictor uses three 3×3 convolutions with 32, 16, and 2M channels, with normalization and leaky-ReLU in the first two layers.The final layer is a linear 2D convolution without activation.
  • Probabilistic decoder: PDC-Net inserts an uncertainty decoder at every pyramid level and uses a two-component constrained Laplace mixture to model accurate matches and outliers.The first component has fixed variance, while the second component’s variance is constrained and the predictor estimates the second variance and mixture weights.
  • Training and inference: Training applies weighted losses across pyramid levels, using negative log-likelihood for probabilistic models and retaining out-of-view synthetic regions in the loss.MegaDepth finetuning instead applies the loss only at sparse ground-truth locations.
  • Evaluation: Sparsification evaluates uncertainty quality by removing pixels in descending uncertainty order and measuring the remaining flow error against an oracle reference.The reported metrics include AEPE and PCK, with lower AUSE indicating better uncertainty calibration.

D.2. Evaluation datasets and set-up

The evaluation spans real and synthetic geometric-matching, optical-flow, pose-estimation, and reconstruction settings, using sparse or semi-dense correspondence supervision where dense ground truth is unavailable.

  • MegaDepth: MegaDepth evaluates 1600 image pairs from held-out scenes using sparse correspondences obtained from SfM reconstructions.The images contain extreme viewpoint changes and each pair shares more than 30 reconstructed points.
  • Sequence evaluation: The sequence-based evaluation samples image pairs at multiple intervals, producing about 500 pairs per interval and 600K to 1000K sparse correspondences.PCK is computed over the whole dataset separately for each interval.
  • KITTI: KITTI provides real stereo road scenes with semi-dense LiDAR ground truth, including dynamic scenes, large motion, illumination changes, and occlusions.The 2012 set is static, whereas the 2015 set includes dynamic scenes through human annotations.
  • YFCC100M: YFCC100M supports pose evaluation using ground-truth poses from 3D reconstructions and correspondences selected from the quarter-resolution flow output.Selected correspondences are scaled back to the original image resolution.
  • Aachen: Aachen evaluates dense correspondences for 3D reconstruction, retaining matches with confidence probability PR=1 above 0.3 before COLMAP processing.Matches are selected at quarter resolution and then scaled to the original image size.
  • Evaluation protocol: The reported benchmark table covers geometric matching, optical flow, and YFCC100M pose estimation with methods trained purely self-supervised.For optical flow, the multi-stage strategy is omitted because it adds runtime without improvement.

E.1. Detailed results on uncertainty estimation

On RobotCar, PDC-Net’s uncertainty estimates more closely match the underlying correspondence errors than the compared dense geometric methods.

  • RobotCar uncertainty estimation: PDC-Net estimates uncertainty maps that better fit the underlying errors on RobotCar than DGC-Net and RANSAC-Flow.The comparison uses sparsification error curves for uncertainty quality.

E.2. Results when trained on purely synthetic data

When trained only on synthetic image warps, PDC-Net outperforms the compared self-supervised methods across multiple datasets, particularly on accuracy metrics. Its probabilistic model also improves over the non-probabilistic baseline with a modest inference-time increase.

  • Synthetic-data results: PDC-Net outperforms DGC-Net, GLU-Net, GLU-Net-GOCor, and GLU-Net-GOCor* across multiple datasets using purely synthetic training data.The strongest gains are reported for PCK and F1 accuracy metrics.
  • Synthetic-data results: PDC-Net’s probabilistic modeling yields large flow-estimation accuracy gains over the non-probabilistic GLU-Net-GOCor* baseline.The comparison isolates the effect of the probabilistic approach because both models use the same synthetic training setting.
  • Efficiency: 14.3% is the reported single-forward-pass inference-time increase over GLU-Net-GOCor* for PDC-Net.The paper reports this increase alongside substantially better results.
  • Downstream pose estimation: Using all dense correspondences with non-probabilistic methods produces poor YFCC100M relative poses, emphasizing the need for confidence prediction in downstream matching tasks.The confidence estimates support selecting more reliable correspondences for pose estimation.

E.3. Qualitative results

PDC-Net produces precise dense correspondences and confidence maps that identify unreliable regions across challenging datasets. Qualitative comparisons show benefits for moving objects, difficult static scenes, and geometric matching.

  • PDC-Net clearly identifies independently moving objects and performs well on static KITTI-2015 scenes with a single moving object.
  • Across RobotCar and MegaDepth, PDC-Net produces highly precise correspondences while its uncertainty estimates identify inaccurate matching regions.
  • On YFCC100M, PDC-Net confidence maps exclude unreliable sky regions and identify small accurate flow regions within predominantly failing flow fields.
  • Sparsification Error plots compare AEPE confidence quality across uncertainty decoders and confidence measures on MegaDepth and KITTI-2015.Smaller AUSE is better; the plots use constrained Laplace mixtures trained only on the first stage.

F. Detailed ablation study

The ablations examine uncertainty propagation, mixture parametrization, component count, and confidence measures. Results generally favor propagated uncertainty, probability-interval confidence, and the simpler two-component mixture, with dataset-specific trade-offs.

  • Confidence measure: On MegaDepth and KITTI-2015, the probability of the confidence interval PR=1 performs better than inverse variance 1/V as the uncertainty measure.
  • Uncertainty propagation: Propagating uncertainty predictions across multi-scale levels boosts performance across the presented datasets and metrics, except for a slightly worse KITTI-2015 F1 score.
  • Mixture parametrization: The alternative σ2_1 = β−_1 = β+_1 = 1.0 has better AUSE on KITTI-2015, where ground-truth displacements have smaller magnitudes.
  • Mixture parametrization: Fixing σ2_1 = 1.0 improves AUSE on MegaDepth and significantly improves pose estimation on YFCC100M.
  • Mixture parametrization: Constraining β+_2 produces slightly better flow results but slightly worse uncertainty performance, while also yielding more stable training in practice.
  • Number of mixture components: Using three Laplace components improves KITTI-2015 F1 and slightly improves YFCC100M pose estimation, but degrades MegaDepth and AUSE performance.The third component is intended to identify outliers more clearly, so the authors adopt M = 2 for simplicity.
Loading 2101.01710v2…