PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses

Eric Brachmann, Carsten Rother

arXiv:1905.04132v2cs.CV

TL;DR

RANSAC struggles to find outlier-free minimal sets when observations contain many outliers, while prior guided methods rely on heuristic side information or limited differentiability. NG-RANSAC learns sampling guidance through task-loss optimization, supports self-supervised inlier-count training, and extends to differentiable RANSAC. Across classic computer-vision tasks, it achieves consistent improvements or competitive and state-of-the-art results against robust estimators.

  • Problem

    RANSAC requires exponentially many iterations as outliers increase, while existing guidance and differentiable methods provide limited or heuristic control over hypothesis sampling.

  • Method

    NG-RANSAC uses neural-network-predicted observation weights to sample minimal sets and can optimize arbitrary task losses with non-differentiable solvers; NG-DSAC adds this guidance to differentiable RANSAC.

  • Results

    Across epipolar geometry, horizon-line estimation, and camera re-localization, the paper reports superior or competitive results, including state-of-the-art camera re-localization.

  • Takeaways & Limitations

    Learned hypothesis sampling improves RANSAC across multiple classic computer-vision tasks and can be trained self-supervised by maximizing inlier count.

  • Takeaways & Limitations

    NG-DSAC inherits DSAC’s requirement for differentiable task loss, scoring function, and minimal solver.

Abstract

from arXiv · show

We present Neural-Guided RANSAC (NG-RANSAC), an extension to the classic RANSAC algorithm from robust optimization. NG-RANSAC uses prior information to improve model hypothesis search, increasing the chance of finding outlier-free minimal sets. Previous works use heuristic side-information like hand-crafted descriptor distance to guide hypothesis search. In contrast, we learn hypothesis search in a principled fashion that lets us optimize an arbitrary task loss during training, leading to large improvements on classic computer vision tasks. We present two further extensions to NG-RANSAC. Firstly, using the inlier count itself as training signal allows us to train neural guidance in a self-supervised fashion. Secondly, we combine neural guidance with differentiable RANSAC to build neural networks which focus on certain parts of the input data and make the output predictions as good as possible. We evaluate NG-RANSAC on a wide array of computer vision tasks, namely estimation of epipolar geometry, horizon line estimation and camera re-localization. We achieve superior or competitive results compared to state-of-the-art robust estimators, including very recent, learned ones.

1. Introduction

NG-RANSAC addresses RANSAC’s difficulty with many outliers by learning observation weights that guide minimal-set sampling. This learned guidance can be trained with arbitrary task losses, self-supervised inlier counts, and combined with differentiable RANSAC.

  • RANSAC estimates model parameters by repeatedly sampling minimal sets, ranking hypotheses by consensus, and returning the top-ranked model.
  • As the outlier ratio increases, RANSAC needs exponentially many iterations to find an outlier-free minimal set.Implementations often cap iterations and return the best model found so far.
  • NG-RANSAC uses a neural network to predict observation weights that guide the sampling of minimal sets.The method is introduced as a learned-guidance extension of RANSAC.
  • Unlike heuristic guidance or soft inlier labeling, NG-RANSAC learns sampling to minimize an arbitrary task loss, allowing non-differentiable losses and minimal solvers.The formulation also supports self-supervised training using inlier-count maximization.
  • NG-DSAC combines neural guidance with differentiable RANSAC so networks can learn observations and confidences while focusing on useful input parts.The paper reports competitive horizon-line estimation and state-of-the-art camera re-localization results.
  • The paper applies NG-RANSAC to epipolar geometry, horizon-line estimation, and camera re-localization, reporting superior or competitive results against robust estimators.

2. Related Work

Related work includes classic guided and improved RANSAC variants, learned robust estimators, and differentiable RANSAC. NG-RANSAC targets a gap left by methods that cannot optimize observation selection or require differentiable components.

  • USAC combines guided PROSAC sampling, locally optimized model fitting, and efficient hypothesis verification as a classic RANSAC framework.
  • Many machine-learning vision pipelines use RANSAC only at test time because its optimization is non-differentiable.
  • DSAC optimizes expected task loss through probabilistic hypothesis selection but differentiates with respect to observations, not observation selection.This prevents DSAC from optimizing matching probabilities directly.
  • Other learned robust estimators predict observation weights for model fitting, including iterative weighting reminiscent of IRLS for fundamental matrix estimation.

3. Method

The method guides RANSAC’s minimal-set sampling with learned observation probabilities and trains this distribution to favor low-task-loss hypotheses. It also extends the framework to differentiable RANSAC, where networks jointly learn observations and their confidences.

  • RANSAC formulation: RANSAC forms hypotheses by applying a minimal solver to small observation subsets, then selects the highest-consensus hypothesis.The minimal-set size depends on the model and solver; for example, the 8-point algorithm uses N = 8 observations.
  • Neural Guidance: NG-RANSAC replaces uniform observation sampling with a learned categorical distribution that increases the chance of sampling outlier-free minimal sets.The learned probabilities define the hypothesis-pool distribution through independently sampled observations and minimal sets.
  • Training NG-RANSAC: Training minimizes expected task loss over hypothesis pools, using K sampled pools because enumerating all possible pools is infeasible.The task loss may use ground-truth model quality or the self-supervised negative consensus score.
  • Training NG-RANSAC: NG-RANSAC’s gradient estimator does not require differentiable task loss, robust solver, or minimal solver components.These components only generate the training signal that steers sampling probabilities; sampling-based gradients can nevertheless have high variance.
  • NG-DSAC: NG-DSAC extends differentiable RANSAC by learning both predicted observations and a probability for each observation.Its objective combines expectations over neural-guided hypothesis-pool sampling and final-estimate selection; unlike NG-RANSAC, it requires differentiable loss, scoring, and minimal-solver functions.

4. Experiments

The experiments evaluate NG-RANSAC across epipolar geometry, horizon-line estimation, and camera re-localization, including side-information and self-supervised settings. Across these tasks, neural guidance improves hypothesis search and achieves superior or competitive accuracy against classical and learned baselines.

  • Essential Matrix Estimation: NG-RANSAC estimates essential matrices from SIFT correspondences and is evaluated on outdoor and indoor image-pair settings.The evaluation compares configurations with and without descriptor-ratio side information and ratio filtering.
  • Essential Matrix Estimation: NG-RANSAC elevates essential-matrix estimation to the leading position, outperforming the compared robust estimators.Its guiding weights are inferred solely from the spatial distribution of correspondences in this comparison.
  • Essential Matrix Estimation: NG-RANSAC finds good hypotheses much earlier than USAC and reaches reasonable accuracy with as few as 10 hypotheses.USAC eventually samples degenerate hypotheses, whereas NG-RANSAC learns to suppress them more effectively.
  • Essential Matrix Estimation: Self-supervised NG-RANSAC uses the final estimate’s inlier count as a stable training objective and matches USAC while slightly trailing supervised NG-RANSAC.Supervised training adapts more precisely to the evaluation measure used at test time.
  • Fundamental Matrix Estimation: For fundamental-matrix estimation, NG-RANSAC outperforms RANSAC and USAC and performs slightly better than Deep F-Mat.The choice of training objective has a small but significant influence, while inlier-count training remains competitive as a self-supervised objective.

5. Conclusion

NG-RANSAC uses learned probabilities to guide hypothesis sampling, supports non-differentiable training objectives and solvers, and can be trained self-supervised with inlier counts. Across several classic vision tasks, it consistently improves on RANSAC alone.

  • NG-RANSAC guides hypothesis sampling with learned probabilities.
  • The method supports non-differentiable task losses and minimal solvers during training.
  • Using inlier count as the training objective enables self-supervised NG-RANSAC.
  • Across multiple classic computer vision tasks, NG-RANSAC consistently improves over RANSAC alone.

A. Essential Matrix Estimation

For essential-matrix estimation, NG-RANSAC predicts correspondence weights from feature matches and uses them to guide RANSAC sampling. The network is initialized from ground-truth alignment distributions and evaluated with qualitative and training-protocol details.

  • The network processes feature correspondences and predicts weights used to guide RANSAC hypothesis sampling.It uses independently processed correspondence features with global context from normalization layers.
  • Initialization minimizes KL divergence between predicted sampling probabilities and a ground-truth essential-matrix alignment distribution.The target assigns higher probability to correspondences aligned with the ground-truth essential matrix.
  • The target distribution uses σ = 10^-3, matching the RANSAC inlier threshold.

B. Fundamental Matrix Estimation

For fundamental-matrix estimation, NG-RANSAC is evaluated on Kitti examples against RANSAC, with neural guidance visualized and the final model refined using the largest inlier set found.

  • The final fundamental matrix is re-fitted to the largest inlier set found by NG-RANSAC.This parallels comparison with fitting to the top 20 weighted correspondences from a competing method.
  • NG-RANSAC is compared with RANSAC on qualitative Kitti fundamental-matrix results.The visualizations include the neural guidance predicted by the network.

C. Horizon Lines

For horizon-line estimation, NG-DSAC predicts image observations and sampling probabilities, then fits a line with differentiable RANSAC. Training uses gradient blockage and robust task-loss handling for difficult images.

  • The network predicts 64 2D observations and corresponding sampling probabilities from each image.The image is processed through two output branches for observations and guided sampling.
  • Gradient blockage prevents sampling predictions from directly influencing the observation-prediction branch during training.This addresses the larger variance and magnitude of sampling gradients during convergence.
  • NG-DSAC scores line hypotheses with a differentiable soft inlier count based on point-line distances.
  • A robust task loss limits the influence of images whose horizon lies outside the image or lacks clear visual cues.The loss uses the square root after a magnitude of 0.25.

D. Camera Re-Localization

For camera re-localization, the network predicts dense scene coordinates and sampling probabilities, then trains these outputs through coordinate, reprojection, and expected task-loss objectives. NG-DSAC++ selects pixels using predicted sampling weights, while visualizations compare its learned representations with DSAC++.

  • Network Architecture: The fully convolutional network takes an RGB image and predicts dense scene coordinates and sampling probabilities for camera-pose estimation.Outputs are subsampled by a factor of 8; the sampling branch uses a Sigmoid followed by normalization.
  • Network Architecture: Gradient blockage prevents sampling predictions from directly influencing scene-coordinate learning in the shared base network.The paper motivates this choice by noting that sampling gradients have larger variance.
  • Training: Training first minimizes robust distance to rendered ground-truth scene coordinates while ignoring pixels without corresponding reconstructed 3D points.The distance uses Euclidean error up to 10m and square-root scaling afterward, with 500k Adam iterations at learning rate 10^-4.
  • Training: The second stage minimizes robust reprojection error relative to the ground-truth camera pose, using a 10px threshold and 300k Adam iterations.The learning rate is 10^-4 and the batch size is one image.
  • Training: The third stage optimizes expected camera-pose task loss under NG-DSAC, combining rotation-angle error in degrees with camera-position error in meters.It uses a soft inlier count for hypothesis scoring and trains for 200k iterations with learning rate 10^-6.
  • Qualitative Results: NG-DSAC++ visualizations compare its learned 3D representations with DSAC++ and sample training pixels among the top 1000 predicted-weight pixels.The horizon-line figure also shows estimated and true lines alongside observation points colored by sampling weight.
Loading 1905.04132v2…