PaperlayerView exact revision on arXiv

Source-linked AI summary

DSAC - Differentiable RANSAC for Camera Localization

Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, Carsten Rother

arXiv:1611.05705v4cs.CV

TL;DR

The paper addresses the inability to train RANSAC-based pipelines end-to-end because hypothesis selection is non-differentiable. It introduces DSAC, which makes hard selection probabilistic and differentiates the expected task loss, then applies it to camera localization. The resulting pipeline improves camera-localization accuracy and exceeds state-of-the-art results by 7.3%.

  • Problem

    RANSAC’s non-differentiable hypothesis selection prevents its direct use in end-to-end deep learning pipelines.

  • Method

    DSAC replaces deterministic hypothesis selection with probabilistic selection, allowing differentiation of the pipeline’s expected loss with respect to learnable parameters.

  • Results

    7.3% higher camera-localization accuracy than the state-of-the-art results is reported for the proposed pipeline.

  • Takeaways & Limitations

    DSAC is presented as a differentiable robust-optimization component for end-to-end deep learning pipelines, including camera localization, structure from motion, and SLAM.

  • Takeaways & Limitations

    Soft argmax changes RANSAC into robust hypothesis averaging and is reported to be prone to overfitting.

Abstract

from arXiv · show

RANSAC is an important algorithm in robust optimization and a central building block for many computer vision applications. In recent years, traditionally hand-crafted pipelines have been replaced by deep learning pipelines, which can be trained in an end-to-end fashion. However, RANSAC has so far not been used as part of such deep learning pipelines, because its hypothesis selection procedure is non-differentiable. In this work, we present two different ways to overcome this limitation. The most promising approach is inspired by reinforcement learning, namely to replace the deterministic hypothesis selection by a probabilistic selection for which we can derive the expected loss w.r.t. to all learnable parameters. We call this approach DSAC, the differentiable counterpart of RANSAC. We apply DSAC to the problem of camera localization, where deep learning has so far failed to improve on traditional approaches. We demonstrate that by directly minimizing the expected loss of the output camera poses, robustly estimated by RANSAC, we achieve an increase in accuracy. In the future, any deep learning pipeline can use DSAC as a robust optimization component.

1. Introduction

RANSAC provides robust global model fitting but its hypothesis selection prevents direct integration into end-to-end deep learning. The paper introduces differentiable alternatives, especially probabilistic selection, and applies them to end-to-end camera localization.

  • Motivation: RANSAC robustly fits global models from noisy local predictions, supporting applications including multi-view geometry, object retrieval, pose estimation, and SLAM.It handles data containing a substantial percentage of outliers.
  • Motivation: RANSAC cannot directly support end-to-end training because selecting the highest-scoring hypothesis is non-differentiable.The selection process blocks direct gradient propagation through the pipeline.
  • Differentiable RANSAC: The paper explores soft argmax, which averages hypotheses, but reports that it changes RANSAC’s selection principle and is prone to overfitting.The soft approach learns a robust average rather than selecting a good hypothesis.
  • Differentiable RANSAC: DSAC preserves hard hypothesis selection as a probabilistic process, enabling gradients of the pipeline’s expected task loss with respect to learnable parameters.The approach is motivated by policy-gradient methods for stochastic computation.
  • Application: The method is demonstrated in camera localization, estimating a 6D pose from a single RGB image in a known static scene using scene-coordinate predictions and RANSAC.The pipeline builds on scene coordinate regression forest methods and uses CNN components around differentiable RANSAC.
  • Results: The proposed pipeline reports camera-localization accuracy exceeding the state of the art by 7.3%.The introduction attributes the advantage of probabilistic selection to reduced sensitivity to overfitting while retaining broad decision distributions.

2. Method

The method makes RANSAC trainable within an end-to-end pipeline by learning scene-coordinate predictions and hypothesis scoring while addressing non-differentiable selection. It compares soft averaging with DSAC’s probabilistic hard selection, which differentiates expected loss.

  • RANSAC pipeline: RANSAC generates hypotheses from minimal subsets, scores their consensus, selects the best, and optionally refines it using additional correspondences.Refinement is optional but practically important for high accuracy.
  • RANSAC pipeline: Camera localization uses per-pixel scene-coordinate predictions as noisy 2D–3D correspondences from which RANSAC estimates a 6D camera pose.The pose comprises the camera’s 3D rotation and 3D translation relative to the scene coordinate frame.
  • Learning objective: End-to-end learning seeks CNN parameters for scene-coordinate predictions and scoring such that robustly fitted poses minimize a task-specific loss.Parameters w affect generated pose quality, while v affects hypothesis selection through the scoring function.
  • Learning objective: Vanilla RANSAC cannot provide complete learning gradients because hypothesis selection depends on scoring parameters and competing hypotheses, not only the selected hypothesis.Parameters v influence scoring directly, while w changes the quality of competing hypotheses.
  • Differentiable selection: SoftAM replaces argmax with a weighted average of hypotheses, enabling gradients but changing RANSAC’s hard-decision principle toward robust averaging.The authors also report that early attempts to differentiate through vanilla selection caused the training loss to increase without recovering.
  • Differentiable selection: DSAC preserves hard hypothesis selection by sampling according to a softmax score distribution and differentiating the expected loss of this stochastic process.The approach is inspired by policy-gradient methods for stochastic computation and is presented as a differentiable counterpart to RANSAC.

3. Differentiable Camera Localization

The camera-localization pipeline applies differentiable RANSAC to RGB images, replacing traditional learned and scoring components with CNNs. It predicts scene coordinates, forms pose hypotheses with PNP, and uses reprojection-based hypothesis scoring.

  • Pipeline: The system targets one-shot camera localization from an RGB image using a pipeline inspired by Brachmann et al.That earlier pipeline extends the SCoRF approach from RGB-D to RGB images.
  • Hypothesis generation: Minimal sets of four scene coordinates are randomly sampled and passed to PNP to create a pool of camera-pose hypotheses.The reference pipeline then iteratively refines, resc ores, and rejects hypotheses in a preemptive RANSAC scheme.
  • Hypothesis scoring: A Score CNN predicts hypothesis consensus from reprojection errors computed for the scene-coordinate predictions.For each prediction, the errors form a 40x40 reprojection-error image used by the scoring network.
  • Pipeline: A Coordinate CNN predicts point-estimate scene coordinates for 42x42 image patches, using a 13-layer VGG-style architecture with 33M parameters.At test time, the system processes 40x40 patches per image to reduce runtime.

4. Experiments

Experiments on 7-Scenes evaluate componentwise and end-to-end training for a differentiable camera-localization pipeline. DSAC improves accuracy after end-to-end training, while SoftAM decreases it relative to componentwise initialization.

  • Experimental setup: The pipeline is evaluated on RGB-only images from the 7-Scenes dataset using pose accuracy below 5° and 5cm.The complete dataset contains 17000 test frames, with separate validation blocks used to fix learning parameters.
  • Training setup: The pipeline uses a Coordinate CNN and a Score CNN, trained componentwise with surrogate losses before end-to-end training.The Coordinate CNN predicts scene coordinates, while the Score CNN is trained from synthetic pose perturbations to predict hypothesis scores.
  • Componentwise training: Componentwise training yields very similar results for RANSAC, SoftAM, and DSAC, with DSAC 0.7% lower accuracy than RANSAC on the complete dataset.The comparison uses non-differentiable argmax selection for RANSAC, soft argmax for SoftAM, and probabilistic selection for DSAC.
  • End-to-end training: +2.2% complete-dataset accuracy is achieved by DSAC after end-to-end training, while SoftAM decreases accuracy by 3.8% from componentwise initialization.DSAC’s strongest scene-level improvements are +3.3% on Pumpkin and +5.0% on Kitchen; SoftAM decreases Office accuracy by 14.7%.
  • End-to-end training: DSAC end-to-end training improves over Brachmann et al. by 4.9% on scene average and 7.3% on the complete set.The experiments also report improved median pose error for DSAC relative to that pipeline.
  • Insights and detailed studies: DSAC’s score distribution can remain broader because unlikely hypotheses do not affect the loss of likely selected poses, supporting greater training stability than SoftAM.SoftAM reduces entropy and must aggressively downweight high-error poses because they influence its pose average.

5. Conclusion

The conclusion presents DSAC as a differentiable RANSAC strategy and demonstrates its use for end-to-end camera-localization learning. It also identifies broader applications where robust optimization is beneficial.

  • DSAC uses probabilistic hypothesis selection to differentiate RANSAC within an end-to-end camera-localization pipeline.
  • The paper compares two differentiation strategies: a soft argmax operator and probabilistic selection.
  • The authors conclude that probabilistic selection is superior and name this approach DSAC.
  • DSAC is presented as applicable to other deep learning pipelines requiring robust optimization, including end-to-end structure from motion or SLAM.

A. Derivatives

The appendix provides derivative information for SoftAM and DSAC learning strategies. It also uses difficult 7-Scenes camera-localization cases to motivate a RANSAC-based approach.

  • The appendix derives derivatives for the task loss, or its expectation, under SoftAM and DSAC.
  • Its second part illustrates difficult camera-localization situations in the 7-Scenes dataset.
  • These 7-Scenes examples motivate using a RANSAC schema for camera localization.

A.1. Soft argmax Selection (SoftAM)

SoftAM makes hypothesis aggregation differentiable by computing a weighted average of hypotheses. The weights are obtained from a softmax distribution over hypothesis scores.

  • SoftAM represents the output as a weighted average of hypotheses.
  • The SoftAM hypothesis average can be differentiated with respect to the learnable parameters.
  • The hypothesis weights P(J|v, w) follow a softmax distribution over hypothesis scores.

A.2. Probabilistic Selection (DSAC)

DSAC learns the camera-localization pipeline by minimizing the expected task loss under probabilistic hypothesis selection. Its derivatives include both task-loss and log-probability terms.

  • DSAC trains the camera-localization pipeline by minimizing the expectation of the task loss.
  • The DSAC derivative differentiates the task loss using Eq. 10.
  • The derivation also differentiates the log probabilities of the selected hypotheses.

B. Difficulty of the 7-Scenes Dataset

The 7-Scenes dataset includes difficult visual conditions, while scene-coordinate inlier ratios vary widely across the dataset. DSAC estimates the correct pose in the illustrated difficult cases.

  • Scene-coordinate prediction inlier ratios range from 5% to 85% across 7-Scenes.A prediction is considered an inlier when it lies within 10cm of the ground-truth scene coordinate.
  • 7-Scenes contains textureless surfaces, motion blur, reflections, and repeating structures.
  • DSAC estimates the correct pose in all four illustrated difficult frames.

C. Calculation of the Camera Pose Error

Camera-pose evaluation must use the inverse of the alignment transformation because the optimized transformation represents the scene pose rather than the camera pose. Correct evaluation reduced the reported result to 45.5%, with PnP identified as a major limitation.

  • The alignment transformation h represents the scene pose, whereas the camera pose is h−1.
  • Figure 6 reports pose-estimation accuracy as a function of scene-coordinate inlier ratios.
  • Rotational and translational pose errors must therefore be calculated using h−1 for the camera pose.Using h instead changes the error calculation because rotational errors additionally contribute to translational errors for h−1.
  • 45.5% was obtained after correcting the camera-pose evaluation.An earlier approximately 56.8% result was produced with an evaluation error.
  • The PnP implementation used in the experiments was identified as a major limiting factor.
Loading 1611.05705v4…