Source-linked AI summary
Pixel-Perfect Structure-from-Motion with Featuremetric Refinement
Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, Marc Pollefeys
TL;DR
Sparse SfM depends on independently detected keypoints that can be poorly localized, affecting reconstruction accuracy. The paper refines keypoints before SfM and points and poses afterward through dense feature alignment, reporting more accurate geometry and camera poses across challenging conditions while scaling to large scenes.
Problem
Independent keypoint detection can be inaccurate under appearance changes and discrete sampling, allowing localization errors to affect sparse reconstruction.
Method
The method uses featuremetric direct alignment of dense neural features to refine keypoints before SfM and 3D points and camera poses after reconstruction.
Results
The approach produces more accurate camera poses and scene structure across challenging conditions and different local features, while scaling to scenes with thousands of images.
Takeaways & Limitations
Keeping a sparse SfM representation while exploiting dense local information preserves scalability and compatibility with downstream visual-localization applications.
Takeaways & Limitations
Featuremetric updates can drift along the scene surface, so the method must limit drift while retaining repeatability with respect to unrefined detections.
Abstract
from arXiv · showhide
Finding local features that are repeatable across multiple views is a cornerstone of sparse 3D reconstruction. The classical image matching paradigm detects keypoints per-image once and for all, which can yield poorly-localized features and propagate large errors to the final geometry. In this paper, we refine two key steps of structure-from-motion by a direct alignment of low-level image information from multiple views: we first adjust the initial keypoint locations prior to any geometric estimation, and subsequently refine points and camera poses as a post-processing. This refinement is robust to large detection noise and appearance changes, as it optimizes a featuremetric error based on dense features predicted by a neural network. This significantly improves the accuracy of camera poses and scene geometry for a wide range of keypoint detectors, challenging viewing conditions, and off-the-shelf deep features. Our system easily scales to large image collections, enabling pixel-perfect crowd-sourced localization at scale. Our code is publicly available at https://github.com/cvg/pixel-perfect-sfm as an add-on to the popular SfM software COLMAP.
1. Introduction
Sparse SfM is scalable and robust to appearance changes, but independently detected keypoints can be inaccurate. The paper refines keypoints before reconstruction and points, poses, and bundles afterward using dense feature alignment.
- Motivation: Sparse SfM supports scalable reconstruction and appearance variation by associating local image features across views.It estimates camera parameters and 3D scene geometry from images.
- Problem: Independent single-view keypoint detection is inaccurate under appearance changes and discrete image sampling.CNN detectors can worsen the issue by favoring global context over local image information.
- Approach: The method adjusts keypoints before SfM and refines 3D points and camera poses after reconstruction through featuremetric alignment.The pipeline uses two stages: tentative-match keypoint refinement followed by post-SfM bundle refinement over dense feature maps.
- Approach: Featuremetric optimization uses locally dense information while remaining applicable to incremental and global SfM with different sparse or dense features.Deep features provide wider convergence in challenging conditions.
- Results: Experiments report improved 3D structure and camera-pose accuracy across hand-crafted and learned local features, challenging conditions, and scenes with thousands of images.The system uses off-the-shelf CNNs and is released as an extension to COLMAP and hloc.
2. Related work
Related work spans sparse and dense matching, geometric keypoint refinement, direct alignment, and featuremetric optimization. This paper combines dense local information with sparse multi-view SfM to refine observations throughout the pipeline while retaining large-scale applicability.
- Sparse and dense matching: Sparse image matching is efficient and robust, but small keypoint localization errors can produce large errors in pose estimation and 3D triangulation.The standard pipeline detects, describes, matches, and geometrically verifies a small set of interest points.
- Sparse and dense matching: Dense matching yields denser, more accurate correspondences but is computationally unsuitable for large-scale SfM because of redundant correspondences.Coarse-to-fine local refinement improves efficiency but remains limited to image pairs.
- Featuremetric SfM: The proposed approach leverages dense local information to refine sparse observations and optimizes all locations across multiple views in a track.This makes dense refinement compatible with SfM rather than restricted to pairwise matching.
- Geometric refinement: Geometric keypoint and bundle adjustments do not model observation uncertainty, so they require many observations to average out geometric noise.The paper instead unifies keypoint and bundle optimization with a featuremetric cost for more accurate and efficient refinement.
- Direct alignment: Photometric direct alignment is naturally subpixel accurate but fails under moderate displacements or appearance changes, limiting its use for wide-baseline SfM.The paper addresses these conditions by aligning learned multidimensional image representations.
- Featuremetric optimization: Featuremetric methods improve convergence and robustness, but prior work such as PixLoc targets single-image pose refinement and does not scale to multi-image large scenes.This paper extends featuremetric refinement to additional SfM stages and thousands of images using off-the-shelf features.
- Contribution: The paper presents robust featuremetric optimization for large-scale sparse reconstruction and reports significant benefits for visual localization.Its contribution is framed as combining sparse matching with locally accurate dense refinement.
3. Background
SfM depends on accurately localized keypoints, but conventional independent detection and geometric refinement leave substantial detection noise. Existing bundle and track adjustments improve locations yet remain limited by geometric observations and approximate flow.
- SfM estimates camera poses and sparse 3D structure from multi-view keypoint correspondences, making keypoint location accuracy critical.Tracks associate 2D observations of the same 3D point across images.
- Global refinement: Bundle adjustment refines camera poses and 3D points by minimizing robust reprojection error over observations in each track.It implicitly refines keypoints while enforcing geometric consistency.
- Global refinement: Bundle adjustment requires many observations to average geometric noise and cannot recover correctly matched observations discarded during geometric verification.
- Track refinement: Track refinement adjusts keypoints on tentative raw-match tracks before geometric estimation, using dense optical flow to improve their accuracy.A neural network regresses point flow and interpolates it from a sparse grid, but regression and interpolation remain approximate.
- Both bundle and keypoint adjustments rely on geometric observations without modeling their uncertainties, limiting accuracy in practice.
4. Approach
The approach replaces geometric keypoint and bundle adjustments with featuremetric direct alignment of dense local information, while retaining sparse correspondences for scalable global SfM. It refines keypoints before verification and jointly refines structure and poses afterward, with constraints and implementations designed for efficiency and robustness.
- 4. Approach: Featuremetric refinement uses locally dense image information to refine sparse observations while preserving scalable global data association and optimization.The framework replaces geometric counterparts within SfM pipelines.
- Direct alignment: Direct alignment compares feature or intensity values at corresponding observations, producing locally accurate, subpixel refinements without descriptor matching or approximate flow interpolation.The resulting flow can be optimized jointly across observation pairs for multi-view consistency.
- Learned representation: Deep CNN feature maps provide locally discriminative, invariant representations for refinement under changes in illumination, resolution, and camera models.The same representation supports both keypoint and bundle adjustments with one forward pass per image.
- 4.2. Keypoint adjustment: Before geometric verification, tentative tracks are separated using matching-graph connectivity and the single-projection-per-image property, then their keypoints are featuremetrically adjusted.Confidence-weighted optimization can split tracks connected by weak correspondences.
- 4.2. Keypoint adjustment: The keypoint adjustment compares precomputed features at sparse points and optimizes tracks independently, making it more scalable than dense patch-flow regression.A bounded update limits drift from initial detections.
- 4.3. Bundle adjustment: After geometric estimation, featuremetric bundle adjustment optimizes 3D points and camera poses through projected observations against a fixed reference feature.The fixed reference reduces point drift and the reference is chosen near the robust mean observation.
- 4.3. Bundle adjustment: Using one reference reduces bundle-adjustment residuals from O(N^2) to O(N), while Schur-complement reduction and embedded point iterations accelerate convergence.The refinement generally converges within a few camera updates.
- 4.4. Implementation: The method supports off-the-shelf CNN feature maps at input resolution and uses Levenberg–Marquardt optimization with bicubic feature lookup and robust loss.Feature patches are constrained to K=8 pixels around initial detections.
5. Experiments
Experiments show that featuremetric refinement improves reconstruction and pose accuracy across local features, datasets, and challenging conditions, while remaining scalable to large image collections.
- Evaluation: The method is evaluated on triangulation, single-image camera pose estimation, end-to-end stereo and multiview SfM, ablations, and scalability.Experiments use ETH3D, crowdsourced Image Matching Challenge imagery, and Aachen Day-Night subsets.
- 3D triangulation: Featuremetric refinement produces significantly more accurate and complete 3D reconstructions than geometric SfM across SIFT, SuperPoint, D2-Net, and R2D2.It is especially more accurate than Patch Flow at the strict 1cm threshold, while showing similar completeness.
- Camera pose estimation: Featuremetric optimization substantially improves query-camera localization for all local features and is consistently more accurate than geometric counterparts.SuperPoint benefits particularly strongly despite producing sparse 3D models.
- End-to-end Structure-from-Motion: In crowdsourced imagery, refinement improves stereo epipolar geometries and multiview SfM poses, especially when only a small number of images observe the scene.Geometric optimization cannot fully suppress detection noise when observations are few.
- Additional insights: Both featuremetric keypoint adjustment and bundle adjustment improve triangulation and localization, while S2DNet dense features perform best among tested image representations.Increasing image resolution improves triangulation but increases runtime and can worsen localization; letting bundle adjustment drift also worsens localization.
- Scalability: The refinement is an order of magnitude faster than Patch Flow, uses less than 10GB peak bundle-adjustment memory for 1000 images, and refines 7k images in under 2 hours.Precomputed distance maps reduce memory from 80GB to less than 10GB for 1000 images.
6. Conclusion
The paper combines globally discriminative sparse matching with locally accurate dense-feature refinement to improve camera poses and structure. It retains SfM’s sparse representation, supporting scalability and compatibility with visual-localization applications.
- Conclusion: Accurate large-scale SfM uses sparse local features for coarse estimation followed by locally accurate dense-feature refinement.Dense features can capture lower-level texture because they need only local discriminativeness.
- Conclusion: Extensive experiments show more accurate camera poses and scene structure in challenging conditions and across different local features.The conclusion summarizes the paper’s results across the evaluated settings.
- Conclusion: Keeping SfM’s sparse scene representation preserves scalability and compatibility with downstream applications such as visual localization.The refinement is also reported to work well with few observations without averaging out keypoint detection noise.
- Conclusion: The approach could improve benchmark ground-truth poses and enable benchmarking in crowdsourced scenarios beyond densely photographed landmarks.The paper frames this as a potential impact of reduced sensitivity to under-sampling.
A. Additional results on ETH3D
Featuremetric refinement improves triangulation accuracy and point-cloud completeness, especially for observations affected by noisy keypoint detections. It also improves camera localization through map and query-keypoint refinement.
- A.1. Triangulation: Featuremetric refinement most improves triangulation for points with low track length, where geometric bundle adjustment is dominated by keypoint-detection noise.For larger track lengths, the refined point cloud approaches the accuracy of Faro Focus X 330 laser-scanner ground truth.
- A.1. Triangulation: The refined point clouds contain fewer noisy keypoints and are more complete for both SuperPoint and D2-Net.
- A.1. Triangulation: Refinement reduces triangulation errors toward 2mm-accurate Lidar ground truth, particularly for short tracks.
- A.2. Pose estimation: Refining the map and query keypoints improves camera-pose accuracy, with query keypoint adjustment before pose estimation providing the largest improvement.The pre-estimation adjustment increases inlier matches and the likelihood of finding the correct RANSAC pose.
- A.3. Patch size: Ten-by-ten patches provide sufficient triangulation accuracy while requiring significantly less memory.
B.2. Image resolution
Refinement accuracy depends on image resolution, feature-map displacement, patch size, reference selection, feature levels, and feature dimensionality. The experiments identify settings that balance accuracy, memory, and runtime for different detectors.
- B.2. Image resolution: Accuracy drops significantly below 1600×1066px, while doubling resolution to 3200×2132px improves accuracy but increases extraction time and GPU VRAM use.Extracting fine-level S2DNet features at 3200×2132px requires around 10GB of GPU VRAM.
- B.1. Point movement: Most refined keypoints move only modestly: 60% travel fewer than 2 pixels and 99% remain within 8 pixels of their initial detections.
- B.3. Patch size: Feature-map patches of size 10×10 offer a good accuracy–memory trade-off.
- B.4. Reference selection: Topological-center references improve point-cloud accuracy and completeness over feature-center references because they allow tracks to merge.
- B.6. Feature levels: A single fine feature level suffices for SIFT, SuperPoint, and R2D2, whereas D2-Net requires fine and medium levels because its detection noise is larger.
- B.5. Dimensionality of the features: Reducing the 128-dimensional S2DNet features accelerates refinement with a moderate accuracy drop.
C. Cost map approximation
Precomputed feature-distance cost maps make featuremetric bundle adjustment more memory-efficient, while derivative augmentation improves convergence and keeps the approximation error small.
- C.1. Cost-map construction: Precomputed cost maps reduce peak bundle-adjustment memory by a factor of D, often D=128, because they replace D-dimensional residuals with scalar residuals.
- C.3. Approximation error: Swapping distance computation and sparse interpolation introduces approximation error that is zero on the discrete grid and increases with feature-space roughness.
- C.3. Approximation error: The approximation displaces the local cost minimum by at most 1 pixel, usually much less.
- C.4. Convergence improvement: Optimizing squared spatial derivatives improves convergence and supports a more accurate bicubic-spline interpolation of the cost landscape.
- C.2. Evaluation: The approximation reduces triangulation accuracy by less than 1% without changing completeness, while enabling bundle adjustment to scale to thousands of images.
D. Experimental details
ETH3D experiments use established implementations and fixed image-resizing, matching, filtering, and train–test protocols. Sparse and dense features are extracted at controlled image resolutions.
- D.1. Datasets and features: ETH3D experiments use SuperPoint, D2-Net, R2D2, and root-normalized SIFT implementations, with images resized to a longest dimension of 1600 pixels.
- D.2. Train–test protocol: Hyperparameters are tuned on Temple Nara Japan, Trevi Fountain, and Brandenburg Gate, then evaluated on Sacre Coeur, Saint Peter’s Square, and Reichstag.
- D.3. Matching protocol: Matching uses detector-specific mutual checks, ratio tests, and geometric inlier thresholds for SIFT, D2-Net, and SuperPoint+SuperGlue.
- D.4. Feature extraction: Sparse and dense features are extracted at full image resolution, generally no larger than 1024px.
D.3. Ablation study - Section 5.4
The ablation evaluates triangulation and localization on ETH3D using SuperPoint features, comparing featuremetric refinement with alternative baselines and feature representations. It also examines a fixed-source-frame variant whose apparent accuracy comes with repeatability problems.
- ETH3D triangulation metrics use the 76-image Facade scene, while localization AUC aggregates 13 scenes with 10 holdout images per scene.
- F-KA refines keypoints before triangulation and adjusts query keypoints, whereas F-BA refines the model and then the query pose.F-BA uses featuremetric bundle adjustment followed by qBA after PnP+RANSAC.
- The “w/ F-BA drift” variant fixes only each track’s source-frame index, so it does not account for drift during optimization.This variant reportedly has higher accuracy but suffers from repeatability problems during localization.
- PatchFlow + F-BA initializes keypoints with Dusmanu et al.’s refinement and applies the proposed featuremetric bundle adjustment using PatchFlow’s published parameters.
- The ablation includes double-resolution input images and a photometric NCC baseline using RGB patches warped at featuremap resolution.The higher-resolution setting uses 3200 pixels in the longest image dimension, while photometric patches use 4×4 pixels at 1600 pixels.
- Dense features are extracted from ImageNet-trained VGG-16 conv1_2, PixLoc’s MegaDepth-trained fine feature map, and DSIFT.The representations have 64, 32, and 128 channels respectively.
D.4. Scalability
The scalability setup targets large localization models and full Aachen reconstruction. It limits matching to highly covisible references for acceleration, then applies keypoint refinement, triangulation, and full featuremetric bundle adjustment.
- Experiments run on 8 CPU cores and one NVIDIA RTX 1080 Ti.
- Aachen matching keeps each image’s top 20 most covisible reference images and uses SuperPoint mutual-check matching with a 0.7 distance threshold.
- The full Aachen model is refined by applying KA to SuperPoint features matched with SuperGlue, triangulating with fixed reference poses, and then running full bundle adjustment.The bundle adjustment uses the proposed cost-map approximation.
- On ETH3D Courtyard, refined sparse point clouds better fit scene geometry than raw clouds, especially on planar walls, for both SuperPoint and D2-Net.The figure compares raw and refined clouds using 1 cm accuracy labels.