ARTDECO: Towards Efficient and High-Fidelity On-the-Fly 3D Reconstruction with Structured Scene Representation

1 INTRODUCTION

• Monocular ambiguity and scale sensitivity make it difficult for existing methods to balance accuracy, speed, robustness, and large-scene efficiency.Per-scene optimization is accurate but computationally expensive, while feed-forward methods are fast and robust but have lower rendering fidelity and weaker consistency.Source: §1 · p. 2
• ARTDECO unifies localization, reconstruction, and rendering, integrating foundation models for pose estimation, loop closure, and dense point prediction to improve localization accuracy and mapping stability while preserving efficiency.Feed-forward priors reduce monocular ambiguities, while loop detection and lightweight bundle adjustment address global inconsistency.Source: §1 · p. 2
• Its hierarchical semi-implicit Gaussian representation and LoD-aware densification trade reconstruction fidelity against rendering efficiency in large-scale, navigable environments.Source: §1 · p. 2
• Across indoor and outdoor experiments, ARTDECO achieves SLAM-level efficiency, feed-forward robustness, and near per-scene optimization quality for on-the-fly digitization.Source: §1 · p. 2

2.1 MULTI-VIEW RECONSTRUCTION AND RENDERING

• NeRF produces high-quality synthesis but expensive volume rendering and large networks hinder real-time use; 3D Gaussian Splatting supports high-fidelity reconstruction and real-time rendering.Many reconstruction methods still assume accurate poses from SfM, which imposes considerable preprocessing costs for large-scale or in-the-wild captures.Source: §2.1 · p. 3

2.2 STREAMING PER-SCENE RECONSTRUCTION

• Classical SLAM provides online tracking, mapping, and loop closure, while integrated 3DGS approaches still struggle to balance robustness, accuracy, and runtime in monocular settings.On-the-fly NVS reports interactive reconstruction with GPU-friendly mini-bundle adjustment and incremental 3DGS updates, but limited robustness on casual unposed videos.Source: §2.2 · p. 3

2.3 FEED-FORWARD MODELS

• Feed-forward models avoid per-scene optimization and offer fast inference, but generally trail per-scene methods in accuracy and face global-consistency, high-resolution, and long-sequence challenges.ARTDECO's described frontend classifies frames using correspondence ratio and pixel displacement; its backend performs loop detection and global pose optimization.Source: §2.3 · p. 3, Figure 2 · p. 4

3 METHOD

• ARTDECO targets static, rigid scenes, estimating camera poses and Gaussian primitives from monocular cues, with or without known camera intrinsics.The pipeline has frontend, backend, and mapping modules; image-wise pointmaps initialize Gaussians that are incrementally optimized.Source: §3 · p. 4

3.1 FRONTEND MODULE

• Frames become keyframes when valid correspondences fall below a threshold, mapper frames when the 70th-percentile displacement exceeds a threshold, or common frames otherwise.Keyframes go to backend pose refinement and mapping; mapper frames support multi-view reconstruction; common frames refine existing scene details without adding structure.Source: §3.1 · p. 5

3.2 BACKEND

• The backend builds a factor graph and performs global bundle adjustment; loop closures link a keyframe to three geometrically consistent predecessors to reduce drift.ASMK proposes loop candidates, and π3 pointmaps help select predecessors by angular error; pointmap confidence is based on reprojection error across prior keyframes.Source: §3.2 · p. 5

3.3 MAPPING MODULE

• Keyframes and mapper frames add Gaussians, while common frames refine existing ones; this uses all frame types for mapping and pairs dense supervision with LoD control.The design targets improved geometry and rendered-view fidelity while maintaining computational efficiency.Source: §3.3 · p. 5, Figure 3 · p. 6
• LoG-guided insertion selects regions for refinement, initializes Gaussian positions and colors from pointmaps and pixels, and sets opacity to 0.2 times pointmap confidence.A Gaussian is added when its insertion probability exceeds a threshold; the probability uses image and rendered-image information.Source: §3.3 · pp. 5–6
• The scale and rotation MLPs use concatenated local voxel-context and individual features; base scale also uses depth, focal length, and LoG-derived image spacing.The individual feature and newly added voxel-wise feature are initialized to zero; the hybrid design is intended to combine global consistency with local distinctiveness.Source: §3.3 · p. 6
• Gaussians are organized from finest to coarsest levels and included or faded according to viewing distance, with dmax controlling level-of-detail rendering.The distance-aware scheme is intended to suppress flickering and maintain stable rendering quality across scales while preserving efficiency.Source: §3.3 · p. 6, Figure 3 · p. 6
• Mapper and keyframes trigger K optimization iterations with new Gaussian insertion, common frames trigger K/2 iterations without insertion, and a global optimization follows streaming.Training samples current frames with probability 0.2 and past frames with probability 0.8; final pose optimization is joint with Gaussian parameters.Source: §3.3 · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation spans eight indoor and outdoor benchmarks, uses PSNR, SSIM, and LPIPS for reconstruction, ATE RMSE for pose accuracy, and FPS for efficiency.Sequence lengths range from 32–5577 frames indoors and 200–1363 frames per outdoor trajectory; experiments use an i9-14900K CPU and RTX 4090 GPU.Source: §4.1 · p. 7

4.2 COMPARISON

• ARTDECO outperforms baselines on indoor and outdoor reconstruction benchmarks, with strong results on challenging TUM and ScanNet and the best performance on VR-NeRF and ScanNet++.The qualitative comparison also reports preserved rendering detail in complex environments, especially in highlighted regions.Source: §4.2 · p. 7, Figure 4 · p. 8, Table 1 · p. 8
• ARTDECO achieves markedly higher localization accuracy than other 3DGS-based systems on TUM and ScanNet++, competitive performance on Waymo, and lower ATE RMSE than state-of-the-art non-3DGS SLAM methods on TUM.The tracking comparison covers SLAM- and SfM-based reconstruction methods on indoor and outdoor datasets, and SLAM systems on nine TUM fr1 scenes.Source: §4.2 · p. 7, Table 2 · p. 9
• ARTDECO runs faster than all compared 3DGS-based methods except OnTheFly-NVS, with extra time primarily spent on pose estimation.The authors describe this runtime trade-off as justified by superior pose accuracy.Source: §4.2 · p. 8

4.3 ABLATION STUDY

• Replacing MASt3R with π3 worsens pose estimation, disabling loop closure degrades localization, and using both mapper frames and keyframes unexpectedly reduces pose accuracy.The authors attribute the dense-input result to small-parallax inputs that can produce ghosting and blur, corrupting point clouds and feature correspondences.Source: §4.3 · p. 9, Table 3 · p. 9
• Mapper frames add multi-view constraints, while level-of-detail and structural Gaussians yield more compact, regularized representations that improve reconstruction fidelity and rendering quality.Source: §4.3 · p. 9, Table 3 · p. 9

• Foundation-model dependence and assumptions of consistent illumination and sufficient parallax constrain robustness; noise, blur, lighting changes, out-of-distribution inputs, low-texture or repetitive surfaces, and near-degenerate trajectories can cause reduced reliability, drift, or artifacts.The authors identify uncertainty estimation, adaptive model selection, and stronger priors as future directions.Source: §5 · p. 9

• ARTDECO combines feed-forward priors and structured Gaussian representations in one system to deliver accuracy and efficiency for on-the-fly monocular reconstruction.Source: §6 · p. 10

• The supplement covers implementation and evaluation protocols, Jacobians and covariance transformation, frontend and backend procedures, and extended experiments with per-scene metrics and qualitative results.It also includes π3-based multi-frame ablations and observations.Source: A SUPPLEMENTARY MATERIAL · p. 15

A.1 MORE IMPLEMENTATION DETAILS

• One frame in every eight is held out from mapping supervision and used only for pose optimization during inference.The same sampled frames are used as evaluation frames across all sequences.Source: §A.1 · p. 15
• Baseline scripts were adapted so all reported metrics use the identical protocol, avoiding evaluation-selection bias.This makes performance differences attributable to algorithmic design rather than evaluation-frame selection.Source: §A.1 · p. 15

A.2 JACOBIAN

• The projection uses log-depth parameterization for 3D points expressed in current-camera coordinates.The mapping differential describes how small 3D perturbations affect pixel-space.Source: §A.2 · p. 15
• The projection Jacobian propagates 3D point uncertainty into keyframe measurement-space covariance.Correspondences with determinant of projected covariance above τ are rejected, filtering high geometric uncertainty in a rotation-invariant way.Source: §A.2 · pp. 15–16
• When focal length is unknown or weakly calibrated, its direct and indirect projection effects are combined in a joint derivative for coherent pose-and-focal updates.The focal length affects projected pixel coordinates and the reconstructed 3D point through normalization.Source: §A.2 · p. 16

A.3 ADDITIONAL EXPERIMENTS

• The experiment targets π3’s potential for stronger global spatial constraints and denser, smoother pointmaps, but its non-metric-aware predictions complicate scale and temporal consistency in streaming SLAM.MASt3R supplies two-view correspondences and metric pointmaps but has limited temporal context.Source: §A.3 · p. 16
• π3 batches consecutive frames for joint pointmap and correspondence inference, followed by local Sim(3) bundle adjustment and keyframe selection.Overlapping temporal windows reuse recent keyframes as anchors to maintain continuity between inference batches.Source: §A.3 · p. 17
• The backend applies π3 inference to accumulated keyframes, then globally optimizes poses using non-adjacent keyframes selected for long-range loop-closure evidence.The ASMK module filters the inference window to keep runtime tractable.Source: §A.3 · p. 17
• The mapper remains unchanged and reconstructs from keyframes received from the backend.Source: §A.3 · p. 17

A.4 DETAILS IN FRONTEND: POSE AND FOCAL OPTIMIZATION

• MASt3R supplies dense two-view correspondences and 3D points; the frontend composes similarity transformation and projection into robust weighted residuals for Gauss–Newton pose and focal-length refinement.Correspondence weights derive from measurement covariance, and a Huber kernel suppresses outliers.Source: §A.4 · p. 17

A.5 DETAILS IN BACKEND: LOOP CLOSURE AND GLOBAL BUNDLE ADJUSTMENT

• The backend detects candidate loops with ASMK, validates them with π3 geometric consistency, then uses global bundle adjustment to refine connected keyframe poses.Confirmed loops require more than 0.15 geometrically consistent correspondences; optimized point confidence guides subsequent Gaussian updates.Source: §A.5 · pp. 18–19

A.6 MORE EXPERIMENTS

• ARTDECO reports the highest PSNR and SSIM and lowest LPIPS across indoor and outdoor datasets, from TUM and ScanNet to Waymo, KITTI, and MatrixCity.The reported per-scene tables span reconstruction and tracking results; the provided table captions identify metric and dataset but do not provide cell values.Source: §A.6 · p. 19, Table 4 · p. 19, Table 5 · p. 19
• Qualitative comparisons include reconstruction results and tracking trajectories on KITTI, ScanNet++, and TUM.Source: §A.6 · p. 19, Figure 6 · p. 26, Figure 7 · p. 27