PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Scene Coordinate Classification and Regression for Visual Localization

Xiaotian Li, Shuzhe Wang, Yi Zhao, Jakob Verbeek, Juho Kannala

arXiv:1909.06216v3cs.CV

TL;DR

Single-image scene-coordinate regression is difficult in large, visually ambiguous environments. The paper introduces a hierarchical coarse-to-fine network whose conditioned classification layers guide final coordinate prediction, achieving state-of-the-art localization across multiple benchmarks and improved large-scale performance.

  • Problem

    Existing scene-coordinate regression methods are difficult to apply to large-scale scenes because limited receptive fields create ambiguous local visual patterns.

  • Method

    A hierarchical network predicts progressively finer discrete location labels through conditioned output layers before producing scene coordinates from a single RGB image.

  • Results

    The method achieves state-of-the-art single-image RGB localization on three benchmark datasets and scales more robustly to larger environments than a regression-only baseline.

  • Takeaways & Limitations

    The architecture improves re-localization accuracy and supports compact models for large scenes, while a hybrid version reduces the gap to explicit feature matching outdoors.

  • Takeaways & Limitations

    The reported comparison is affected by differing data-augmentation settings, and the method is complementary to advanced training and RANSAC schemes rather than evaluated in combination with them.

Abstract

from arXiv · show

Visual localization is critical to many applications in computer vision and robotics. To address single-image RGB localization, state-of-the-art feature-based methods match local descriptors between a query image and a pre-built 3D model. Recently, deep neural networks have been exploited to regress the mapping between raw pixels and 3D coordinates in the scene, and thus the matching is implicitly performed by the forward pass through the network. However, in a large and ambiguous environment, learning such a regression task directly can be difficult for a single network. In this work, we present a new hierarchical scene coordinate network to predict pixel scene coordinates in a coarse-to-fine manner from a single RGB image. The network consists of a series of output layers, each of them conditioned on the previous ones. The final output layer predicts the 3D coordinates and the others produce progressively finer discrete location labels. The proposed method outperforms the baseline regression-only network and allows us to train compact models which scale robustly to large environments. It sets a new state-of-the-art for single-image RGB localization performance on the 7-Scenes, 12-Scenes, Cambridge Landmarks datasets, and three combined scenes. Moreover, for large-scale outdoor localization on the Aachen Day-Night dataset, we present a hybrid approach which outperforms existing scene coordinate regression methods, and reduces significantly the performance gap w.r.t. explicit feature matching methods.

1. Introduction

The paper addresses ambiguities that make direct scene-coordinate regression difficult in large environments by introducing hierarchical coarse-to-fine prediction for single-image RGB localization. It reports improved performance over regression-only networks and state-of-the-art results across several benchmarks.

  • Visual localization estimates precise 6-DoF camera pose relative to a known environment for applications including augmented reality and autonomous driving.
  • Feature-based methods establish 2D-3D correspondences by matching local descriptors between query images and pre-built 3D models.
  • Scene coordinate regression directly predicts dense 3D coordinates from image pixels, avoiding explicit feature matching and descriptor databases at test time.
  • Limited receptive fields create visually ambiguous local patterns in large-scale environments, making scene-coordinate regression harder and reducing test-time performance.
  • The proposed network conditions progressively finer classification layers on previous location predictions before producing scene coordinates.
  • The approach consistently outperforms a regression-only network and achieves state-of-the-art single-image RGB localization on 7-Scenes, 12-Scenes, and Cambridge Landmarks.
  • The method supports compact models that scale robustly to larger environments, including combined scenes, while a hybrid approach reduces the gap to feature-based methods.

2. Related Work

Prior localization systems use feature matching, image retrieval, pose regression, or scene-coordinate prediction, with each approach offering different accuracy, scalability, or input requirements. The paper situates its method within these alternatives and related classification-regression frameworks.

  • Structure-based localization estimates 6-DoF pose from 2D-3D correspondences obtained by matching local features such as SIFT.
  • Image retrieval scales to very large environments using compact image-level descriptors and can restrict the search space for subsequent precise localization.
  • Direct camera-pose regression is closer to image-retrieval pose approximation than accurate 3D-structure-based estimation and remains outperformed by structure-based methods.
  • The cited sequence-based approach requires image sequences rather than single images.
  • Scene-coordinate regression learns pixel-level 3D coordinates with random forests or neural networks, producing dense correspondences without explicit feature detection, description, and matching.
  • Related joint classification-regression methods combine discrete prediction with continuous estimation across tasks including human pose and scene-coordinate prediction.

3. Hierarchical Scene Coordinate Prediction

The method hierarchically partitions 3D scene points into coarse-to-fine labels and jointly predicts those labels with scene coordinates. Coarser predictions condition finer layers through spatial feature modulation, while training combines classification and regression losses.

  • Hierarchical joint learning framework: The ground-truth 3D point cloud is hierarchically partitioned, associating each training pixel with coarse-to-fine discrete location labels.The partitions can be constructed using k-means.
  • Hierarchical joint learning framework: Conditioned classification layers predict only the label branch at each level, reducing each level’s prediction to k labels when the branching factor is k.Each layer is conditioned on all preceding label maps.
  • Network architecture: The global-to-local architecture uses broader image context at coarse levels and smaller receptive fields at finer levels, where location labels disambiguate local appearances.At test time, finer layers still depend on location labels predicted from the input.
  • Conditioning layers: Conditioning layers feed coarse label predictions into finer levels by generating spatial scaling and shifting feature maps for element-wise modulation.The generators produce parameters with the same height, width, and channel dimensions as the conditioned feature maps.
  • Loss function: The three-level network uses two classification outputs and one regression output, trained jointly with cross-entropy losses and Euclidean scene-coordinate regression.The first classification branch predicts coarse labels and the second predicts fine labels.

4. Experimental Evaluation

Experiments across standard, combined, and outdoor datasets show that hierarchical scene coordinate learning improves localization over regression-only baselines and scales more robustly to larger environments. The method also supports compact models and a hybrid Aachen approach that narrows the gap to feature-based localization.

  • Datasets and setup: The evaluation uses 7-Scenes, 12-Scenes, Cambridge Landmarks, three synthesized combined scenes, and Aachen Day-Night.The combined scenes integrate seven, twelve, or nineteen individual indoor scenes into single coordinate systems.
  • Standard benchmarks: Across the three standard datasets, the method consistently outperforms the regression-only baseline and achieves better overall results than DSAC++ and NG-RANSAC where reported.The authors also report consistently lower regression training error, attributing this to disambiguation from discrete location labels.
  • Combined scenes: On combined scenes, the regression-only baseline degrades dramatically as environment size increases, whereas the proposed method remains substantially more robust and outperforms ESAC.Unlike ESAC’s multiple specialized networks, the proposed approach uses a single model.
  • Model capacity: The model size can be reduced by more than a factor of two with less than one percentage point of accuracy loss, while remaining more than seven times smaller than ESAC on i19-Scenes.The additional cluster-center storage is negligible, at less than 1MB per individual scene.
  • Detailed analysis: Increasing the receptive field harms performance, while the small-receptive-field variant performs slightly better on combined scenes, supporting hierarchical handling of local ambiguities.The result is consistent with the paper’s motivation that limited receptive fields create ambiguity in large environments.
  • Aachen and robustness: Without data augmentation, the method still matches or exceeds state-of-the-art results, and on Aachen its hybrid approach significantly outperforms ESAC while remaining comparable to Active Search.The Aachen approach uses sparse local features and image retrieval because dense reprojection optimization is not applicable to the sparse training data.

5. Conclusion

The paper introduces hierarchical coarse-to-fine scene coordinate learning for visual localization and reports improved accuracy, scalability, and outdoor performance.

  • The hierarchical network uses multiple output levels, with each level conditioned on previous outputs.Progressively finer localization labels are predicted through classification branches.
  • The final scene coordinates come from either a regression layer or cluster centers at the finest classification level.
  • The approach achieves state-of-the-art RGB localization on three benchmarks and three combined scenes.
  • A hybrid approach narrows the performance gap with feature-based methods on challenging large-scale outdoor localization.

—Supplementary Material—

The supplementary material provides implementation and evaluation details beyond the main paper.

  • The supplement provides additional details about the network architecture.
  • The supplement provides additional details about the training procedure.
  • The supplement includes additional qualitative results at the end.

A. Main Experiment Details

This section presents experiment details for 7-Scenes, 12-Scenes, Cambridge Landmarks, and combined scenes.

  • The experiments include the 7-Scenes dataset.
  • The experiments include the 12-Scenes dataset and Cambridge Landmarks.
  • The experiments include combined scenes.

A.1. Network Architecture

The network uses a VGG-style regression backbone with added conditioning and classification branches, while image preprocessing fixes the input size at 640 × 480.

  • Backbone and branches: The base regression network uses a VGG-style architecture with ELU activations instead of ReLU.The hierarchical network adds conditioning layers, generators, and two classification branches.
  • Resolution and receptive field: Three stride-2 convolutions reduce the regression branch resolution by a factor of 8.Strided, dilated, and upconvolution layers enlarge receptive fields and preserve classification output resolution.
  • Dataset-specific architecture: The architecture is unchanged across the three standard datasets, while combined-scene experiments increase selected channels and add conditioning layers.
  • Input preprocessing: The network always receives an input image of size 640 × 480.Larger images are resized to height 480 and cropped to width 640 when necessary.

A.2. Network Training

Training uses dataset-specific optimization settings, augmentation, and PnP-RANSAC pose estimation. Reported training and inference times vary with hardware and network size.

  • Training schedule: The Cambridge Landmark dataset uses longer training because imperfect dense reconstructions produce significant ground-truth scene-coordinate outliers.The network is trained for 600K iterations on Cambridge Landmarks, compared with 300K iterations for 7-Scenes and 12-Scenes.
  • Loss weighting: The regression loss receives weight 100,000 for the three combined scenes and weight 10 for the other datasets, while classification-loss weights remain 1.The larger regression weight reflects the reported importance of final regression accuracy for localization performance.
  • Data augmentation: Training augments images with affine transformations and additive brightness changes, or with random pixel shifts when augmentation is disabled.Translations, rotations, scales, shears, brightness changes, and small horizontal and vertical shifts are sampled within specified ranges.
  • Pose optimization: Pose estimation follows PnP-RANSAC with an inlier threshold of τ = 10, 256 initial hypotheses, and refinement for up to 100 iterations.The soft inlier-count softness factor is β = 0.5, and refinement continues until convergence or the iteration limit.
  • Runtime: Inference takes approximately 100ms, comprising 50–65ms for scene-coordinate prediction and 30–60ms for pose optimization.Training takes approximately 12 hours for 300K iterations on a Tesla V100 and 18 hours on a GTX 1080 Ti.

B. Experiments on the Aachen Dataset

The Aachen experiments generate hierarchical location labels from a COLMAP-built sparse point cloud and evaluate a classification-only architecture. The four-level classification-only network performs better than the tested classification-regression alternatives.

  • Label generation: Hierarchical k-means clustering on a COLMAP-built sparse point cloud generates the ground-truth location labels for Aachen.The sparse model was built using SuperPoint as the local feature detector and descriptor.
  • Architecture choice: A 4-level classification-only network is adopted for Aachen because it outperforms the two tested classification-regression networks.The comparison is reported in Table 5 of the main paper.

B.2. Network Architecture

For Aachen, the network processes sparse local features rather than regular RGB images and uses a compact architecture adapted to irregular descriptors. Localization combines descriptor processing, scene-coordinate prediction, and pose optimization.

  • Input representation: Aachen uses sparse feature lists as network input, represented as an N×1×256 descriptor map rather than a regular RGB image.The network uses 1×1 convolutions and adds a dummy spatial dimension for the sparse, irregular input.
  • Network architecture: Different hierarchy levels have no shared layers, while every output layer includes image-level contextual information.These design choices adapt the architecture to sparse descriptor input and provide contextual information at each level.
  • Feature extraction: Training descriptors are extracted densely only where corresponding 3D points exist, after grayscale conversion and image augmentations.Random affine transformations, brightness changes, and contrast changes are applied before feature extraction.
  • Runtime: Aachen localization takes approximately 1.1s by day and 1.4s by night, including descriptor extraction, prediction, and pose optimization.Descriptor extraction takes about 170ms, scene-coordinate prediction about 280ms, and pose optimization about 600/900ms for day/night.

C. Additional Qualitative Results

Additional visualizations inspect predicted scene coordinates and retained correspondences on indoor and outdoor localization examples. The figures also include invalid predictions and failure cases where applicable.

  • 7-Scenes visualizations: Figure 8 compares the method with the regression-only baseline on three 7-Scenes/i7-Scenes test images using scene-coordinate visualizations.XYZ coordinates are mapped to RGB values; invalid depth values and out-of-scope predictions are omitted from visualization.
  • Aachen visualizations: Figure 9 visualizes Aachen predictions as 2D-2D matches between query and database images, retaining only inlier matches.The figure includes challenging queries and failure cases, with the database image having the most inliers selected for each pair.
  • Aachen architecture: The Aachen qualitative results use a 4-level classification-only network architecture.The architecture is presented as a dedicated network for the Aachen dataset experiments.
Loading 1909.06216v3…