PaperlayerView exact revision on arXiv

Source-linked AI summary

Expert Sample Consensus Applied to Camera Re-Localization

Eric Brachmann, Carsten Rother

arXiv:1908.02484v1cs.CV

TL;DR

Camera re-localization must estimate a 6D pose from noisy correspondences, while large or ambiguous domains challenge a single network. ESAC integrates DSAC with specialized experts and joint end-to-end training; the paper reports improvements in scalability and ambiguity handling, with state-of-the-art results on difficult indoor datasets.

  • Problem

    Noisy correspondences and large or ambiguous environments make accurate 6D camera-pose estimation difficult for a single network.

    Source: §1 · pp. 1–2

  • Method

    ESAC combines DSAC with specialized experts, distributing pose hypotheses according to a gating network and selecting by sample consensus.

    Source: §1 · p. 2

  • Results

    ESAC improves scalability and ambiguity handling and achieves state-of-the-art results on difficult public indoor re-localization datasets.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The paper demonstrates ESAC on synthetic model fitting and real-world camera re-localization, including environments with scalability and ambiguity challenges.

    Source: §1 · p. 2

    ESAC improves over DSAC++ outdoors but does not close the performance gap to classical sparse feature-based methods; SfM training ground truth also contains substantial outliers, particularly for Dubrovnik.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Fitting model parameters to a set of noisy data points is a common problem in computer vision. In this work, we fit the 6D camera pose to a set of noisy correspondences between the 2D input image and a known 3D environment. We estimate these correspondences from the image using a neural network. Since the correspondences often contain outliers, we utilize a robust estimator such as Random Sample Consensus (RANSAC) or Differentiable RANSAC (DSAC) to fit the pose parameters. When the problem domain, e.g. the space of all 2D-3D correspondences, is large or ambiguous, a single network does not cover the domain well. Mixture of Experts (MoE) is a popular strategy to divide a problem domain among an ensemble of specialized networks, so called experts, where a gating network decides which expert is responsible for a given input. In this work, we introduce Expert Sample Consensus (ESAC), which integrates DSAC in a MoE. Our main technical contribution is an efficient method to train ESAC jointly and end-to-end. We demonstrate experimentally that ESAC handles two real-world problems better than competing methods, i.e. scalability and ambiguity. We apply ESAC to fitting simple geometric models to synthetic images, and to camera re-localization for difficult, real datasets.

1. Introduction

Camera re-localization must infer a 6D pose despite noisy correspondences and ambiguous environments. ESAC distributes pose hypotheses among specialized experts and selects by sample consensus, with joint end-to-end training.

1. Introduction

  • Camera re-localization estimates a camera’s 6D pose from an image of a known 3D environment, using 2D-3D correspondences and robust estimation to handle erroneous predictions.The paper describes RANSAC and differentiable DSAC as robust estimators for noisy correspondences.Source: §1 · p. 1
  • Repeating structures can create multiple valid correspondences, making a single network insufficient for some ambiguous environments.MoE assigns experts to local parts of the domain and uses a gating network to assess which expert is responsible.Source: §1 · p. 2
  • ESAC distributes pose hypotheses among experts according to gating probabilities, then selects the final estimate by sample consensus across experts.The ensemble is trained jointly and end-to-end by minimizing expected task loss over hypothesis sharing and selection.Source: §1 · p. 2
  • ESAC is reported to improve scalability and ambiguity handling, with state-of-the-art results on difficult public indoor re-localization datasets.The introduction also reports robustness over naive expert selection on a synthetic line-or-circle fitting task.Source: §1 · p. 2

2. Related Work

Related work covers general ensembles, Mixture of Experts, and camera re-localization. ESAC differs by using specialized experts to predict representations for robust parametric fitting within a jointly trained system.

2. Related Work

  • Conventional ensemble methods combine base learners trained across the full problem domain, whereas Mixture of Experts divides the domain among specialists and weights their outputs with a gate.MoE training approaches described include expected-loss minimization, likelihood maximization under a Gaussian-mixture interpretation, and expectation-maximization.Source: §2 · p. 2
  • The paper identifies a gap in MoE methods: expert outputs had not been used as representations for robust parametric fitting while retaining joint end-to-end training.Conditional computation can save work but risks catastrophic errors when the gate selects the wrong expert; ESAC distributes budget based on soft gate predictions.Source: §2 · p. 3
  • Camera re-localization methods include retrieval, direct pose regression, sparse feature matching with RANSAC, and dense scene-coordinate regression with differentiable RANSAC.Retrieval and pose regression are described as efficient but limited in accuracy; feature methods are accurate and scale well but can struggle with texture-less surfaces and motion blur.Source: §2 · p. 3
  • Prior scene-coordinate ensembles use random forests or decision trees, but their base learners do not specialize in parts of the problem domain.The cited ensemble approaches include boosting-like diversification and mapping decision-tree ensembles to neural networks.Source: §2 · p. 3
  • Related systems combining environment-part classification or image retrieval with relative pose regression and [46] use one large model and cannot be trained end-to-end.The paper contrasts these systems with ESAC’s conditional computation and end-to-end training.Source: §2 · p. 3

3. Method

The method moves from differentiable robust fitting to a mixture of specialized experts, then distributes model hypotheses across experts so that consensus scoring is not determined by the gate’s single largest probability. It trains this ensemble efficiently by estimating gradients through sampling.

3.1. Differentiable Sample Consensus

  • DSAC makes hypothesis selection probabilistic over scores, allowing the network to minimize expected task loss and compute gradients for end-to-end training.The objective uses a soft inlier count because the ordinary inlier count is non-differentiable.Source: §3.1 · p. 4

3.2. Expert Selection

  • Sampling the gradient of the expert-selection objective avoids running DSAC for every expert when computing the outer expectation.With K = 1, training uses one DSAC run per input.Source: §3.2 · p. 5
  • The expert-selection MoE assigns each input to one expert using gate probabilities, but its accuracy is limited by gate accuracy because choosing the wrong expert can yield an unusable estimate.Experts are specialized to different inputs, illustrated by line- and circle-fitting networks.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3. Expert Sample Consensus

  • ESAC allocates the total hypothesis budget among experts according to the gating network, then selects the final hypothesis by consensus score across experts.Experts with no assigned hypotheses can be skipped, enabling conditional computation.Source: §3.3 · p. 5

4. ESAC for Camera Re-Localization

The camera re-localization pipeline predicts scene coordinates from an RGB image, estimates pose hypotheses from minimal correspondence sets, and refines the selected pose. Its ensemble uses specialized coordinate-prediction networks and a gating network.

4. ESAC for Camera Re-Localization

  • Four 2D–3D correspondences define each pose hypothesis through perspective-n-point, and a robust estimator selects among hypotheses using a soft inlier-count score.The score uses scene-coordinate reprojection error.Source: §4 · p. 6
  • The selected pose is refined by iteratively solving perspective-n-point on its inliers, with training loss combining rotation-angle and translation errors.The translation term is weighted by γ, set to 100 for degrees and meters.Source: §4 · p. 6
  • Each expert predicts scene coordinates on an 80 × 60 grid for a 640 × 480 image, while a whole-image gating network assigns experts.Experts use 16 layers and 6M parameters; the gate uses 10 layers and 100k parameters.Source: §4 · p. 6
  • Figure 4 reports toy-problem parameter-estimation and model-type accuracy, with qualitative panels showing ground truth in green and estimates in blue.The caption specifies the plotted measures but provides no numerical outcomes.Source: Figure 4 · p. 7

5. Experiments

The experiments test ESAC on synthetic model fitting and camera re-localization across indoor and outdoor environments. ESAC handles ambiguous expert assignments and scales to larger environments, with end-to-end training and runtime also evaluated.

5.1. Toy Problem

  • On 10,000 test images of lines and circles, ESAC outperformed Expert Selection by 3.9% in parameter accuracy (line error <3px; circle center and radius within 3px) and 11.5% in model classification accuracy.Source: §5.1 · p. 7

5.2. Camera Re-Localization

  • With scene ID known, expert accuracy is similar to DSAC++, slightly worse on 7Scenes and slightly better on 12Scenes, despite networks 7.5× smaller.The camera experiments use RGB-only input and one expert per scene.Source: §5.2 · p. 7, Figure 5 · p. 7
  • 88.1% of 19Scenes images are correctly re-localized by ESAC, versus 53.3% for DSAC++; ESAC consistently leads as environments grow.ESAC’s accuracy is hardly affected by increasing environment scale.Source: Figure 6 · p. 8, §5.2 · p. 8
  • >10% accuracy improvement appears at the 2cm and 2° threshold; jointly training the ensemble gives best accuracy and mainly improves precision.At the common 5cm and 5° threshold, the effect is significant but not large.Source: Figure 7 · p. 8, §5.2 · p. 8
  • For visually similar offices, ESAC distinguishes scenes reliably by combining global image context when distributing hypotheses with geometric consistency when selecting them.Expert Selection is particularly prone to confusing offices with ambiguous furniture and equipment.Source: §5.2 · p. 8
  • 6.1 experts are evaluated in 555ms on average; limiting ESAC to the top two experts yields +19.7% accuracy over Expert Selection with a minor computation increase.These timing measurements use a single Tesla K80 GPU and include data reading.Source: §5.2 · p. 8, Figure 8 · p. 9
  • Outdoors, ESAC improves over DSAC++ by a large margin but does not close the gap to classical sparse feature-based methods; more experts help only to some degree.The outdoor experiments cover Dubrovnik and Aachen Day.Source: Figure 9 · p. 9, §5.2 · p. 9

6. Conclusion

ESAC distributes model hypotheses among expert networks and selects among them using sample consensus, rather than relying on a single expert. It achieves state-of-the-art accuracy for large indoor camera re-localization and makes progress toward closing the gap to classical feature-based outdoor methods.

6. Conclusion

  • ESAC distributes model hypotheses among experts and scores them by sample consensus, making it more robust than choosing a single expert.On Aachen, ESAC hypotheses form a clear cluster at the query image’s true position, unlike DSAC++ hypotheses.Source: §6 · p. 9, Figure 10 · p. 9
  • For large indoor re-localization, where each expert specializes in one room, ESAC achieves state-of-the-art accuracy.Source: §6 · p. 9
  • For large-scale outdoor re-localization, ESAC makes progress toward closing the gap to classical feature-based methods.Source: §6 · p. 9

A. Scene Classification

The scene-classification comparison contrasts Expert Selection, which follows the gating network’s scene prediction, with ESAC, which considers multiple experts and selects by geometric consistency.

A. Scene Classification

B. Large-Scale Outdoor Re-Localization

The outdoor experiments apply ESAC to Dubrovnik and Aachen Day, using clustered expert initialization and joint training. ESAC exceeds DSAC++ but not ActiveSearch, and the discussion cautions that dataset reconstruction and selection complicate comparisons.

B. Large-Scale Outdoor Re-Localization

  • Figure 11 compares scene confusion matrices for Expert Selection and ESAC, while Figure 12 shows test frames from scenes Expert Selection often confuses.Figure 11 uses true scenes on the Y-axis and estimated scenes on the X-axis.Source: Figure 11 · p. 10, Figure 12 · p. 10
  • Dubrovnik comprises ca. 6k varied-camera photos and Aachen Day ca. 4.5k images; Aachen Night is omitted because no night training data exist and bridging that gap is out of scope.The experiments apply ESAC to Dubrovnik and Aachen Day.Source: B. Large-Scale Outdoor Re-Localization · pp. 9–10
  • Initialization clusters each area by median scene coordinates, soft-assigns images to experts, initializes the gating and expert networks, then jointly trains the ensemble end-to-end.Expert networks are initialized in two stages before 50k iterations of joint ESAC training.Source: B. Large-Scale Outdoor Re-Localization · p. 10
  • ESAC improves accuracy over DSAC++ [8] on both outdoor datasets but remains below ActiveSearch, especially on Dubrovnik even with M=50 experts.The authors note that the datasets’ reconstruction quality may affect comparisons with feature-based methods.Source: B. Large-Scale Outdoor Re-Localization · p. 11
  • Because SfM reconstruction filters out images difficult for feature-based approaches, the authors caution that comparing learning-based and classical methods on these datasets may be problematic.The models and training poses are derived from reconstructions run on both training and test images.Source: B. Large-Scale Outdoor Re-Localization · p. 11
  • Reconstruction outliers may make ESAC overfit visually similar patches with different ground-truth coordinates, while jointly optimized geometry and poses make Dubrovnik’s ≈1m ActiveSearch accuracy uncertain.The authors interpret ESAC’s poorer Dubrovnik accuracy than Aachen’s as consistent with differences in reconstruction quality.Source: B. Large-Scale Outdoor Re-Localization · p. 11
Loading 1908.02484v1…