PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneAligner: 3D-Grounded Floorplan Localization in the Wild

Junhyeong Cho, Ruojin Cai, Hadar Averbuch-Elor

arXiv:2605.22581v1cs.CVcs.AIcs.LG

TL;DR

Existing floorplan localization assumes small, controlled environments and precise vectorized maps, limiting use in large real-world buildings. SceneAligner reconstructs a 3D scene, converts it into a density-map proxy, and aligns it with rasterized floorplans, achieving substantial gains over prior methods, including with a single image.

  • Problem

    Existing methods rely on precise vectorized floorplans and controlled small-scale environments, leaving localization in large buildings with rasterized or symbolic maps unresolved.

  • Method

    SceneAligner reconstructs a gravity-aligned 3D scene, projects it into a density-map floorplan proxy, and learns cross-modal correspondences for 2D similarity-transform alignment.

  • Results

    SceneAligner improves combined angular-positional recall over C3Po by 123.24% (73.58 vs. 32.96) and remains effective with a single input view.

  • Takeaways & Limitations

    The approach supports floorplan localization in the wild and alignment of disjoint interior and exterior reconstructions through a shared floorplan.

  • Takeaways & Limitations

    Cross-modal correspondence ambiguities can cause failures even when the 3D reconstruction is accurate.

Abstract

from arXiv · show

Many public buildings provide floorplans with a "you are here" indicator to help visitors orient themselves. Floorplan localization seeks to computationally replicate this capability by determining where visual observations were captured within a floorplan. However, existing methods typically assume controlled small-scale environments and precise vectorized floorplans, limiting their ability to operate in large-scale buildings and rasterized floorplans. In this work, we present an approach for performing floorplan localization in the wild by grounding the task in a reconstructed 3D representation of the scene. Given an unconstrained image collection, our method reconstructs a gravity-aligned 3D scene and projects it into a 2D density map that serves as a floorplan proxy. Floorplan localization is then formulated as aligning this proxy with the input floorplan via a 2D similarity transform. To bridge the appearance gap between density maps and architectural floorplans, we adapt a 2D foundation model to learn cross-modal correspondences, introducing a fine-tuning scheme that encourages semantically aligned matches while preserving structural consistency. Extensive experiments demonstrate substantial improvements over prior methods, including in extremely sparse settings with as little as a single input image. Our code and data will be publicly available.

1 Introduction

SceneAligner addresses floorplan localization in large, real-world buildings where rasterized plans, architectural complexity, and exhaustive pose search limit prior methods. It instead leverages 3D reconstruction, density-map alignment, and cross-modal feature learning, achieving substantial improvements while remaining effective with sparse image collections.

  • Motivation: Prior floorplan-localization methods exhaustively search discretized camera poses, but their assumptions break down in large-scale real-world settings.The limitations are especially pronounced in historic landmarks and monuments.
  • Motivation: Rasterized or symbolic floorplans and high architectural complexity make localization difficult when precise geometry is unavailable.These conditions motivate localization methods that operate in the wild without exhaustive pose enumeration.
  • Approach: SceneAligner revisits floorplan localization using high-fidelity 3D reconstructions from unconstrained image collections.Recent 3D foundation models enable reconstruction across large-scale environments with diverse viewpoints and illumination.
  • Approach: The method learns reliable cross-modal correspondences between a structurally grounded density-map proxy and the provided architectural floorplan.It adapts DINOv3 into a shared feature space with fine-tuning objectives designed to bridge the modalities’ appearance gap.
  • Results: Two to threefold improvements across most metrics are reported on the in-the-wild testbed, alongside gains over discretized-pose indoor methods.SceneAligner also remains effective for sparse image collections.

2 Related Work

Prior floorplan-localization methods use depth cues, shared image–floorplan embeddings, or probabilistic geometric predictions, while cross-modal correspondence research has progressed from handcrafted descriptors to learned dense matching. SceneAligner addresses the gap between unconstrained photographs and abstract floorplans through an intermediate density map derived from 3D scene reconstruction.

  • Floorplan Localization: Floorplan-localization methods have used LiDAR or depth-camera cues to compare room edges with 2D layouts, often assuming known camera heights.More recent methods embed images and floorplans jointly or predict depth rays and probability volumes.
  • Learning Cross-Modal Correspondences: Cross-modal correspondence methods evolved from handcrafted descriptors with geometric verification to learned keypoints, matchers, diffusion features, and self-supervised representations.Deep visual features have enabled dense prediction, including LoFTR and DUSt3R.
  • Novelty: SceneAligner bridges unconstrained photographs and abstract floorplans through an intermediate density map derived from 3D scene reconstruction.This intermediate representation provides the connection between the two modalities.
  • 3D Scene Understanding from Internet Photo Collections: Prior internet-photo research has studied spatial understanding of large environments using visual patterns, viewpoints, metadata, annotated maps, crowd flow, scene semantics, and semantic registration.These efforts include assembling disjoint indoor reconstructions and registering photo collections to 3D reference models.

3 Method

SceneAligner localizes unconstrained images by reconstructing a gravity-aligned 3D scene, converting it into a filtered 2D density-map floorplan proxy, and aligning that proxy to the input floorplan. It learns robust cross-modal correspondences with feature, coordinate, and structural objectives before estimating a similarity transform.

  • 3.1 3D Scene Reconstruction: The method estimates 3D points and relative camera poses from an unconstrained image collection, then gravity-aligns the reconstruction to share the floorplan’s horizontal plane.Gravity is estimated from per-image GeoCalib predictions transformed into the reference frame, using their medoid and Gram-Schmidt orthogonalization.
  • 3.2 Floorplan Proxy Extraction: The density-map proxy filters unreliable, spatially unbounded, and non-structural points before orthographic projection onto the horizontal xz-plane.Filtering suppresses floor and ceiling surfaces while preserving layout-defining vertical structures.
  • 3.2 Floorplan Proxy Extraction: After projection, grid-cell point counts undergo gamma correction and normalization to produce a top-down line-drawing density map comparable to the reference floorplan.The resulting density map is D ∈ R^H×W×1, while the floorplan is F ∈ R^H×W×3.
  • 3.3 Floorplan Alignment: It retains the top 50% of correspondences by confidence, applies mutual nearest-neighbor matching, and estimates the scale, rotation, and translation of a 2D similarity transform with RANSAC.The transform is then applied to horizontal 3D coordinates and vertical coordinates are scaled to preserve structural proportions.
  • 3.3 Floorplan Alignment: The alignment module fine-tunes cross-modal features with contrastive matching, sub-patch coordinate regression, and topology and geometry consistency losses.Coordinate regression uses differentiable soft-argmax to avoid patch-level quantization errors, while structural priors preserve angles and distance ratios.

4 Experiments

The experiments evaluate SceneAligner in in-the-wild and synthetic indoor settings using image-level pose and pixel-level correspondence metrics. Results show strong performance against prior methods, including under sparse-view and single-view conditions, while revealing ambiguity in some single-view cases.

  • Experimental scope: Experiments cover in-the-wild data, sparse-view robustness, and synthetic indoor evaluation, with additional appendix studies of hyperparameters and design choices.The appendix includes ablations of learning objectives, correspondence filtering, LoRA configurations, and 3D reconstruction models.
  • Datasets and metrics: Evaluation reports Angular Recall and Positional Recall for camera pose, plus PCK and RMSE for pixel-level correspondence.Pose recall uses yaw and horizontal-center errors, while correspondence metrics use floorplan-normalized distances.
  • Datasets and metrics: The C3 evaluation uses a curated clean subset of a large-scale in-the-wild dataset pairing diverse photographs with floorplans and camera-pose and correspondence annotations.Samples with severe errors, including optical distortions or non-photographic content, are pruned to improve test reliability.
  • In-the-wild results: 73.58 vs. 32.96: SceneAligner improves combined angular-positional recall over C3Po by 123.24%, and 73.58 vs. 18.28 yields a 302.52% improvement over pretrained DINOv3.For RMSE, C3Po’s error is 129.38% higher than SceneAligner’s, with values 0.1780 vs. 0.0776.
  • Sparse-view robustness: SceneAligner outperforms C3Po in the single-view setting and reaches accuracy comparable to the 150-view setting with 10 to 30 views.Sparse reconstructions can still produce structurally meaningful density maps, although single-view geometry can ambiguously match multiple similar floorplan structures.
  • Single-view localization: 51.6 vs. 37.6: SceneAligner surpasses UnLoc by 37.2% on combined angular-positional recall in the single-view localization evaluation.This result is reported as state-of-the-art accuracy under a more general setting alongside in-the-wild scenarios.

5 Conclusion · Appendix · A 360◦View Comparisons of Floorplan Alignment

SceneAligner reframes floorplan localization as reconstruction and alignment grounded in scene geometry, combining pretrained 3D reconstruction priors with 2D visual foundation models. The appendix supplements the main paper with dataset, implementation, evaluation, analysis, and visualization materials, including 360° comparisons before and after DINOv3 fine-tuning.

  • 5 Conclusion: SceneAligner uses geometric priors from pretrained 3D reconstruction models and representations from 2D visual foundation models.Together, these components support reframing floorplan localization as a reconstruction and alignment problem grounded in scene geometry.
  • 5 Conclusion: The method reframes floorplan localization as reconstruction and alignment grounded in scene geometry.The conclusion describes this as the central perspective of SceneAligner.
  • Appendix: The appendix provides dataset, implementation, qualitative, quantitative, analysis, limitations, applications, and broader-impact details omitted from the main paper.These materials are organized in Sections B–I.
  • Appendix: The accompanying viewer.html contains 360° comparisons of floorplan-aligned 3D reconstructions.The viewer is referenced for comparisons described in Appendix Section A.
  • A 360◦View Comparisons of Floorplan Alignment: The HTML file visually compares alignment results before and after fine-tuning DINOv3.The renderings show floorplan-aligned 3D reconstructions from both stages.
  • A 360◦View Comparisons of Floorplan Alignment: Figure A.1 compares floorplan alignment using pretrained DINOv3 correspondences with those from the fine-tuned model.The figure presents a preview of these correspondence-based alignment results.

B In-the-Wild Dataset Details · C Additional Implementation Details

The paper establishes a clean C3 benchmark by filtering noisy scene–floorplan samples and annotating views as interior or exterior. It also organizes additional implementation details around correspondence estimation, augmentation, training, and inference for in-the-wild evaluation.

  • B In-the-Wild Dataset Details: The clean C3 subset retains 463 of 479 training scenes and 111 of 118 test scenes after removing samples with severe reconstruction and floorplan-registration errors.The subset distinguishes scene and image statistics by interior and exterior view types.
  • B In-the-Wild Dataset Details: A multi-stage filtering pipeline removes duplicates, extreme-aspect-ratio images, insufficient-correspondence pairs, and excessive out-of-boundary correspondences.Algorithmic filters target obvious low-quality samples before Qwen2.5-VL identifies harder failure cases.
  • B In-the-Wild Dataset Details: Images receive interior, exterior, or unsure classifications from Qwen2.5-VL, with CLIP-based tentative labels, structural path-keyword overrides, and three rounds of visual inspection.The labeling process uses camera location, viewpoint, and structural keywords such as “facade” and “nave.”
  • B In-the-Wild Dataset Details: The clean-subset statistics report scene-level and image-level counts grouped by interior and exterior views, including scene-floorplan associations and view-type distinctions within pairs.A scene may correspond to multiple floorplans, and images from one scene can have different view-type labels.
  • B In-the-Wild Dataset Details: The filtering pipeline excludes samples with severe optical distortions, such as fisheye imagery, and non-photographic content, such as blueprints and illustrations.These examples are shown as removed samples from the original C3 dataset.
  • C Additional Implementation Details: The additional implementation details cover the correspondence-estimation network architecture, data augmentation, training, and inference settings for predicting a 2D similarity transform.These details extend the cross-modal correspondence estimation described in Section 4.1 and support in-the-wild evaluation.

C.1 Model Architecture · C.2 Data Augmentation

The model adapts a frozen DINOv3 ViT-B/16 encoder with LoRA for density-map/floorplan correspondence estimation. Data augmentation simulates reconstruction noise, sparse views, geometric and photometric variation, and mixed input resolutions.

  • C.1 Model Architecture: The correspondence network uses DINOv3 ViT-B/16 as a shared encoder for density maps and reference floorplans.The inputs are a one-channel density map and a three-channel reference floorplan.
  • C.1 Model Architecture: The backbone remains frozen while trainable LoRA layers are inserted into every transformer block’s attention and MLP modules.LoRA uses rank r = 16, scaling factor α = 16, and dropout rate 0.1.
  • C.1 Model Architecture: Density-map extraction varies the confidence percentile threshold ρconf, horizontal percentile threshold ρxz, and gamma value γ.The default settings are ρconf = 45%, ρxz = 2.5%, and γ = 0.5.
  • C.2 Data Augmentation: Density-map augmentation combines π3 and VGGT reconstructions with cropping or padding preprocessing to produce 518×518 inputs.For each reconstruction, the method randomly samples 50–150 images, or uses all available images when fewer exist.
  • C.2 Data Augmentation: The reconstruction configurations produce ∼140K density maps, supplemented by ∼140K single-view density maps for sparse-view scenarios.The configurations include percentile settings with ρconf = [20%, 95%].
  • C.2 Data Augmentation: Training applies random rotations from {0°, 90°, 180°, 270°} and random cropping with scale factor [0.7, 1.0] to density maps and floorplans.Cropping starts from a randomly selected top-left corner.
  • C.2 Data Augmentation: Photometric augmentation uses color jittering, with floorplans perturbed by brightness 0.5, contrast 0.5, saturation 0.5, and hue 0.1.The passage states that density maps also receive photometric adjustments, but the remaining values are truncated.

C.3 Training Details · C.4 Inference Settings · C.5 Structured3D Setup

The method uses staged training with specified loss weights and stable optimization, then aligns density maps to floorplans through confidence-filtered correspondences and RANSAC. Structured3D reuses the in-the-wild configuration with single-view, non-augmented samples and longer training.

  • C.3 Training Details: Training samples Q = 1024 correspondence pairs per iteration and uses λfeat = 1, λregr = 50, λtopo = 10, and λgeo = 10.The feature-matching temperature is τ = 0.07.
  • C.3 Training Details: A curriculum trains only with Lfeat during the first 10% of iterations to stabilize training.The supplied passage specifies this initial curriculum stage but ends before describing later stages.
  • C.3 Training Details: The reconstruction model uses weights from [20], which were trained for robust 3D reconstruction from noisy Internet photo collections.
  • C.3 Training Details: Optimization uses AdamW with learning rate 10−4, batch size 4, and gradient clipping at maximum norm 1.0.Training runs for 1 epoch on ∼280K augmented samples and takes ∼2 days on one NVIDIA RTX A6000 GPU.
  • C.4 Inference Settings: Scenes with N > 150 images are partitioned into ⌈N/150⌉ chunks, while smaller scenes are processed entirely under GPU memory constraints.Input images are padded before 3D reconstruction; density extraction uses ρconf = 45%, ρxz = 2.5%, and γ = 0.5.
  • C.4 Inference Settings: Similarity-transform estimation samples density-proportional query points, retains the top 50% confidence points, applies mutual nearest-neighbor matching, and uses the remainder in RANSAC.The estimated transform M is applied to reconstructed 3D points and their associated camera poses to produce a floorplan-aligned 3D scene.
  • C.5 Structured3D Setup: Structured3D uses the in-the-wild model hyperparameters, losses, and optimization settings with single-view training samples and inference-derived density maps without augmentations.Because this setup slows convergence, the network is trained for 10 epochs.

D Additional Qualitative Results · E Additional Quantitative Evaluations on In-the-Wild Data

Additional qualitative results show that SceneAligner aligns density maps with rasterized floorplans to recover correspondences and camera poses, including from minimal-context or single-view images. Additional in-the-wild evaluations investigate the contributions of the learning objectives, correspondence filtering, model configurations, training resolution, reconstruction models, matching strategy, and hyperparameters.

  • D Additional Qualitative Results: The method predicts density-map-to-floorplan correspondences to estimate a 2D similarity transform across floorplans with varying target-region sizes.Photo-to-density-map correspondences come from 3D reconstruction, while density-map-to-floorplan matches are estimated cross-modally.
  • D Additional Qualitative Results: On Structured3D, the method recovers camera poses from a single image by aligning an extracted density map to a reference floorplan despite limited visual context.The evaluation uses rasterized floorplans without a discretized pose space or floorplan preprocessing.
  • D Additional Qualitative Results: The qualitative visualizations include aligned density maps with reliable correspondences and aligned 3D scene reconstructions alongside floorplan localization results.Figure D.4 displays the input floorplan, aligned density map, and aligned scene for each sample.
  • D Additional Qualitative Results: Global scene context enables pixel-level localization of minimal-context photos after mapping observations through the density map to the aligned floorplan.The mapping proceeds from photo to density map via 3D reconstruction and from density map to floorplan via correspondence estimation.
  • E Additional Quantitative Evaluations on In-the-Wild Data: The quantitative evaluations ablate the learning objectives by progressively adding feature matching, coordinate regression, topology preservation, and geometry consistency losses.These objectives are evaluated for their contributions to cross-modal correspondence learning.
  • E Additional Quantitative Evaluations on In-the-Wild Data: Correspondence-filtering experiments compare confidence-ranked top50 retention and mutual nearest neighbor matching before RANSAC estimates the similarity transform.top50 retains the top 50% of correspondences ranked by confidence w_i.
  • E Additional Quantitative Evaluations on In-the-Wild Data: Further evaluations examine LoRA configurations, mixed-resolution training, 3D reconstruction models, density map–floorplan matching, and robustness to hyperparameters.These studies validate additional design choices on in-the-wild data.

E.1 Design Choices … F.2 Evaluation by Scene View Type

The ablations identify fine-tuning, correspondence filtering, LoRA configuration, mixed-resolution training, and π3 reconstruction as important design choices. Additional analyses show robustness to density-map hyperparameters, degradation with sparse views, and stronger performance on interior than exterior scenes.

  • E.1 Design Choices: Fine-tuning objectives improve cross-modal correspondence learning by adding semantic feature alignment, continuous coordinate regression, and structural regularization.The feature matching loss enhances pretrained DINOv3, while coordinate regression transitions predictions from discrete grid indices to continuous spatial coordinates.
  • E.1 Design Choices: Confidence-based top50 pruning and mutual nearest-neighbor matching improve the reliability of correspondences used for RANSAC-based similarity-transform estimation.The method retains the top 50% of correspondences ranked by confidence and enforces bidirectional consistency.
  • E.1 Design Choices: LoRA rank and scaling factor affect localization accuracy, with r = α = 16 producing the best evaluated results.Very low rank, such as r = 4, limits adaptation capacity and degrades image-level and pixel-level localization accuracy.
  • E.1 Design Choices: Mixed-resolution training improves image-level and pixel-level localization by supporting correspondence matching across varying floorplan target-region sizes.The strategy also acts as data augmentation that helps mitigate overfitting in the correspondence estimation network.
  • E.1 Design Choices: π3 consistently outperforms VGGT across all reported metrics for gravity-aligned 3D scene reconstruction and is therefore used by default.The comparison is reported in Table E.5.
  • E.2 Robustness to Density Map Hyperparameters: The method remains robust across confidence, horizontal, and gamma thresholds for density-map extraction, outperforming C3Po, DUSt3R, and LoFTR.This robustness indicates that exhaustive tuning of density-map hyperparameters is not necessary.
  • F.1 Sparse-View 3D Reconstruction Quality: Sparse-view reconstruction quality declines as fewer images are used, with single-view inputs showing the largest deviation from dense-view pseudo ground truth.Reconstructions using ≤10 images closely approach the dense-view reference, while fewer views create a substantially larger gap and greater alignment ambiguity.
  • F.2 Evaluation by Scene View Type: Interior scenes achieve higher camera-pose and correspondence accuracy than exterior scenes, whose noisier reconstructions and distant background points introduce density-map artifacts.These artifacts make cross-modal correspondence estimation more challenging on exterior scenes.

F.3 Inference Time Comparison

The method is compared with C3Po using amortized end-to-end inference time per image on an NVIDIA RTX A6000 GPU, decomposed into four processing stages. It is faster in correspondence and camera-pose estimation, with a maximum chunk size of N ≤ 10 achieving 143.7 ms versus C3Po’s 195.4 ms.

  • F.3 Inference Time Comparison: Inference time is measured end-to-end per image on a single NVIDIA RTX A6000 GPU and decomposed into four stages: reconstruction, density extraction, correspondence estimation, and camera-pose estimation.The reported cost is amortized per image to reflect deployment efficiency.
  • F.3 Inference Time Comparison: The method is substantially faster than C3Po in correspondence estimation and camera-pose estimation through lightweight, amortized processing.It uses ViT-B for correspondence estimation and estimates correspondences once per density map, amortizing cost across up to N images.
  • F.3 Inference Time Comparison: 143.7 ms vs. 195.4 ms: Ours (≤10) is faster than C3Po and provides the best trade-off among tested chunk sizes.At N = 1, sequential processing underutilizes the GPU; at N = 150, quadratic transformer complexity in 3D reconstruction becomes the bottleneck.

G Limitations · H Applications · I Broader Impacts

The method remains vulnerable to reconstruction errors and cross-modal ambiguities, especially with sparse views, while enabling floorplan-anchored alignment of separate 3D reconstructions and applications in navigation, XR, and architectural understanding. The authors anticipate no significant negative societal impacts but emphasize responsible use in private spaces.

  • G Limitations: 3D reconstruction errors can degrade density maps and floorplan alignment, while accurate reconstructions may still fail because density maps and floorplans have ambiguous cross-modal correspondences.These are distinct sources of failure: imperfect reconstruction and inherent correspondence ambiguity.
  • G Limitations: Structural ambiguity occurs when a local density-map region matches multiple visually similar floorplan structures, preventing unique correspondence without sufficient surrounding context.A doorway is given as an example of a local region that can produce incorrect alignments.
  • G Limitations: Both ambiguity types arise from insufficient global context and become more pronounced in sparse-view settings, motivating future work on resolving them.Sparse views may capture only a partial scene view, although the method remains effective in most scenarios.
  • H Applications: Using a reference floorplan as a geometric anchor, the method aligns separate interior and exterior reconstructions into one global coordinate system despite minimal visual overlap and large viewpoint differences.This addresses failures of joint 3D foundation-model reconstruction under drastic viewpoint changes and limited overlap.
  • H Applications: The method independently aligns disjoint reconstructions into a unified global coordinate system without visual overlap, including partial reconstructions of Santi Pietro e Paolo d’Agrò.The reference floorplan serves as the shared geometric bridge.
  • I Broader Impacts: Floorplan localization of in-the-wild photographs can support scene navigation, augmented or virtual reality, and architectural understanding.The method also facilitates downstream interior-exterior and disjoint 3D scene alignment.
  • I Broader Impacts: The authors do not expect significant negative societal impacts because the method uses widely available photographs and architectural floorplans, but they stress responsible deployment in private spaces.Users are encouraged to follow established guidelines in real-world applications.
Loading 2605.22581v1…