Source-linked AI summary
Geometry-Aware Representation Denoising for Robust Multi-view 3D Reconstruction
Jin Hyeon Kim, Jaeeun Lee, Claire Kim, Kyoungjin Oh, Paul Hyunbin Cho, Jaewon Min, Yeji Choi, Jihye Park, Hyunhee Park, Minkyu Park, Seungryong Kim
TL;DR
Degraded multi-view observations challenge feed-forward 3D reconstruction because corrupted features disrupt geometric consistency, while existing restoration spaces may lose cross-view or fine-grained structure. GARD performs diffusion-based restoration in the reconstructor’s geometry-aware feature space and decodes the refined representations into both 3D geometry and RGB images. Experiments on the DA3 benchmark report consistent improvements over existing restoration approaches across pose estimation and 3D reconstruction.
Problem
Real-world degradations impair feature extraction and cross-view geometric consistency, while existing image- and VAE-space restoration approaches cannot reliably preserve the information needed for robust reconstruction.
Method
GARD learns a diffusion-based multi-view denoiser directly in a feed-forward reconstructor’s geometry-aware feature space and uses an RGB decoder alongside the geometry decoder.
Results
GARD consistently outperforms existing single-view and multi-view restoration methods, achieving the best 3D reconstruction performance across all evaluated DA3 datasets under severe motion blur.
Takeaways & Limitations
Representation-level denoising enables joint restoration of scene structure and appearance from degraded multi-view inputs within a unified framework.
Takeaways & Limitations
Because GARD is diffusion-based, it requires iterative denoising steps that limit efficiency in latency-sensitive settings.
Abstract
from arXiv · showhide
Multi-view 3D reconstruction has achieved remarkable progress with the advent of feed-forward 3D reconstruction models. However, these models are typically trained and evaluated under ideal, degradation-free imaging conditions, whereas real-world observations often contain degradations that differ significantly from such settings. Improving robustness for multi-view 3D reconstruction under degraded conditions therefore remains an important challenge. We present Geometry-Aware Representation Denoising (GARD), a novel framework that performs diffusion-based multi-view restoration directly in the feature space of a feed-forward 3D reconstruction model. This design exploits the geometry-aware feature representations of the 3D reconstructor to effectively recover accurate scene geometry. Furthermore, by employing an additional RGB image decoder, the refined representations can also be used to restore high-quality RGB images, thereby enabling the simultaneous recovery of 3D scene geometry and high-quality imagery. Comprehensive experiments on the Depth Anything 3 (DA3) benchmark demonstrate the effectiveness of the proposed GARD framework.
1 Introduction
Feed-forward multi-view reconstruction models work well under ideal imaging but become vulnerable to real-world degradations that disrupt feature extraction and geometric consistency. GARD addresses this by denoising geometry-aware representations, jointly recovering imagery and 3D geometry.
- Motivation: Feed-forward models infer scene geometry directly from multi-view inputs using cross-view information encoded in geometry-aware representations.These end-to-end architectures replace traditional multi-stage reconstruction pipelines and perform accurate, scalable reconstruction under ideal imaging conditions.
- Motivation: Real-world degradations such as motion blur obscure textures and structural cues, making features less discriminative and disrupting geometric consistency across views.Because feed-forward models lack explicit error-correction mechanisms, degraded features can propagate and accumulate in the final reconstruction.
- Motivation: Image-space restoration typically processes views independently, limiting its use of multi-view information and preventing enforcement of cross-view geometric consistency.This motivates moving restoration into a representation space that retains multi-view structure.
- Motivation: VAE-based compressed latents can lose fine-grained details and structural fidelity needed for accurate multi-view 3D reconstruction.High-dimensional representations are motivated as a way to better retain global structure and local detail while supporting high-fidelity image decoding.
- GARD: GARD applies diffusion-based multi-view denoising directly to a feed-forward reconstructor’s geometry-aware feature space.A dedicated RGB decoder converts refined representations into high-quality images while the reconstruction model recovers scene geometry.
- Evaluation: Experiments on the Depth Anything 3 benchmark evaluate restoration and reconstruction under controlled motion-blur degradations against feature-space, VAE-latent, and restore-then-reconstruct alternatives.The study also compares dedicated image-restoration methods using standard image-quality metrics.
2 Related Work
Related work has advanced image restoration and visual representation learning, while feed-forward reconstruction models have improved direct scene-geometry inference. However, these approaches still face robustness and information-preservation challenges in degraded multi-view settings.
- Robust multi-view 3D reconstruction: Feed-forward 3D reconstruction models directly infer scene geometry in one forward pass, replacing conventional multi-stage optimization pipelines.Their performance nevertheless degrades in real-world settings containing noise and other degradations.
- Multi-view image restoration: Image restoration methods recover clean images from degraded observations through tasks including deblurring, denoising, and super-resolution.Transformer-based methods improve long-range dependency modeling, while language conditioning supports unified restoration in InstructIR.
- Representation space learning: Pretrained visual encoders provide high-dimensional representations that generalize across downstream tasks, whereas latent diffusion commonly uses compact VAE-based spaces for efficiency.The related-work discussion contrasts the efficiency of compact latents with the representational considerations motivating richer feature spaces.
3 Method
GARD restores degraded intermediate representations directly in the geometry-aware feature space of a frozen feed-forward multi-view reconstructor. Its multi-view denoiser and task-specific decoders jointly support robust geometry estimation and RGB image restoration.
- Motivation: Pixel-space restore-then-reconstruct pipelines cannot fully exploit multi-view information or enforce cross-view geometric consistency during restoration.Existing video restoration methods also rely heavily on short-range temporal coherence, limiting generalization to sparse multi-view scenarios.
- GARD framework: GARD inserts a diffusion-based denoiser into the feature space of a pretrained, frozen feed-forward multi-view encoder.The denoiser refines degraded intermediate features rather than restoring images before reconstruction.
- Task outputs: Refined multi-level features are decoded by geometry and RGB decoders to produce 3D scene geometry and restored images simultaneously.The geometry decoder remains part of the feed-forward reconstructor, while the RGB decoder is adapted and fine-tuned for GARD.
- Geometry-aware feature analysis: DA3 features achieve the highest PCK across thresholds on clean inputs and remain more robust than VAE- and DINOv2-based features under increasing degradation.The analysis compares feature-based cost volumes under high-quality and mild, moderate, and heavy degradation settings.
- GARD denoiser: The denoiser combines frame-level attention with global cross-view attention to capture within-view structure, correspondences, and geometric consistency.This architecture is designed for denoising high-dimensional feature representations.
- Training objective: Attention alignment loss is jointly optimized with flow matching to encourage geometrically consistent correspondences and sharper attention patterns.The total objective is L = Lflow + λattnLattn.
4 Experiments
GARD is evaluated as a geometry-aware multi-view restoration framework for robust pose estimation, 3D reconstruction, and image restoration under severe motion blur. Experiments also examine training components and the effect of increasing input views.
- Experimental settings: The experiments compare GARD with single-view, multi-view, and video restoration baselines, including a VAE-latent multi-view denoiser.The VAE-based baseline uses the same denoising architecture as GARD but operates in a conventional latent space.
- Pose estimation: GARD produces more accurate and geometrically consistent camera pose trajectories than baselines on degraded multi-view inputs.The evaluation reports camera pose accuracy using AUC5 and AUC30 on the DA3 benchmark.
- 3D reconstruction: GARD achieves the best 3D reconstruction performance across five DA3 benchmarks under severe motion blur.The evaluation uses Overall↓ for DTU and F-score↑ for the remaining benchmarks, alongside qualitative point-cloud comparisons.
- Image restoration: GARD achieves higher PSNR and lower LPIPS than single-view and VAE-based multi-view restoration baselines.The results indicate that denoising in a high-dimensional geometry-aware feature space preserves visual fidelity and structural consistency.
- Ablation experiments: Attention alignment yields consistent gains when combined with interpolated flow matching, but not with the standard flow-matching objective.Interpolated flow matching introduces partial structural information from the degraded input, facilitating correspondence learning.
- Ablation experiments: Increasing the number of input views consistently improves camera pose estimation and 3D reconstruction quality.The study attributes these improvements to complementary geometric and visual information across views; HiRoom results are reported only up to 10 views.
5 Conclusion
GARD restores degraded multi-view inputs in the geometry-aware feature space of feed-forward 3D reconstruction models, jointly recovering scene geometry and RGB imagery. Its diffusion-based design improves robustness, but iterative denoising limits efficiency in latency-sensitive settings.
- GARD denoises degraded representations in the geometry-aware feature space of feed-forward 3D reconstruction models.
- Refined representations are passed to geometry and RGB decoders to jointly recover accurate 3D scene geometry and high-quality multi-view RGB images.
- GARD consistently outperforms existing single-view and multi-view restoration methods in extensive experiments.
- Iterative diffusion denoising limits GARD's efficiency in latency-sensitive settings.
A.1 Feature Similarity Analysis
The analysis measures how closely restored features match clean representations across the reconstructor’s layers and compares correspondence quality across feature spaces. GARD improves feature fidelity when applied early, while DA3 features retain the strongest cross-view geometric correspondences under degradation.
- Feature Similarity Analysis: Feature similarity measures how effectively GARD recovers degraded representations toward their corresponding clean representations across transformer layers.
- Feature Similarity Analysis: Without GARD, similarity to clean representations progressively decreases through deeper layers, indicating gradual loss of geometric information.
- Feature Similarity Analysis: Applying GARD at an early layer initially improves similarity between restored and clean representations and supports reconstruction and pose estimation performance.
- Feature Cost Volume Visualization: DA3 features produce sharper and more geometrically consistent cross-view correspondences than DINOv2 and VAE features under clean and degraded conditions.
- Feature Cost Volume Visualization: VAE features yield scattered and ambiguous correspondences, reflecting limited geometry-aware representation quality caused by heavy latent compression.
A.3 Multi-View Depth Estimation
GARD improves multi-view depth estimation under degraded conditions by restoring observations in a geometry-aware representation space. The evaluation also documents benchmark protocols and comparisons with restoration baselines, including SIR-Diff and different encoder capacities.
- Multi-View Depth Estimation: GARD outperforms all baseline methods on four DA3 benchmark datasets and has only a marginal performance gap on DTU for multi-view depth estimation.
- Multi-View Depth Estimation: Depth evaluation reports AbsRel↓ and δ1↑, measuring absolute relative depth error and the fraction of predictions within a 1.25 threshold ratio.
- Multi-View Depth Estimation: The reported results support restoring degraded observations in geometry-aware representation space to preserve cross-view consistency and fine-grained structural details for depth prediction.
- Multi-View Depth Estimation: SIR-Diff is compared with GARD using pose estimation and 3D reconstruction results on the DA3 benchmark.
- Multi-View Depth Estimation: The evaluation includes pose AUC30 and task-dependent 3D reconstruction metrics, with Overall↓ for DTU and F-score↑ for other benchmarks.
B.1 Evaluation Metrics
The evaluation follows the DA3 protocol for pose estimation and 3D reconstruction, using thresholded pose accuracy and geometric reconstruction metrics. It also describes RGB reconstruction through an adapted decoder that receives restored representations.
- Evaluation Metrics: Pose estimation uses AUC for Relative Rotation Accuracy and Relative Translation Accuracy at thresholds 5 and 30.
- Evaluation Metrics: 3D reconstruction evaluation reports accuracy, completeness, their average Chamfer Distance, and F1-score under a distance threshold.
- Evaluation Metrics: The DA3-BASE pose evaluation reports AUC5↑ and AUC30↑, while reconstruction evaluation reports Overall↓ and F-Score↑.
- Evaluation Metrics: Restored representations are propagated to the geometry decoder and an adapted ViT-based RGB decoder to reconstruct scene geometry and RGB images.
B.3 Training Details
GARD training combines complementary synthetic datasets, degraded multi-view inputs, geometry-aware alignment, and a diffusion-based denoising pipeline for downstream reconstruction and restoration.
- Training dataset: Hypersim and TartanAir provide complementary indoor, outdoor, appearance, motion, and viewpoint diversity for training geometric representations.Hypersim emphasizes realistic indoor appearance, while TartanAir contributes dynamic trajectories and large viewpoint changes.
- Training hyperparameters: The GARD denoiser is trained for 10 epochs with AdamW, a 2 × 10−4 learning rate, global batch size 8, EMA stabilization, and gradient clipping.Training uses fp32 parameters, decay 0.9995, and a maximum gradient norm of 1.0.
- Attention alignment: Attention alignment constructs 3D correspondence targets from clean-view depth and camera poses, then aligns denoiser attention with pooled ViT patch tokens.The target correspondence map is formed after unprojecting pixels into shared world coordinates and pooling to patch resolution.
- Attention alignment: Attention alignment improves geometric correspondence quality in denoiser attention maps, addressing unreliable correspondence in a decoder layer trained only with flow matching.The visualizations compare attention before and after adding the alignment loss.
- Inference pipeline: At inference, DA3 encoder features from layer 18 are denoised in feature space, then supplied to geometry and RGB decoders for joint reconstruction and image restoration.The framework selects four feature levels and performs both outputs in a single forward pass without separate restoration and reconstruction stages.
C Additional Qualitative Results
Additional qualitative evaluations visualize camera pose, 3D reconstruction, image restoration, and depth estimation on the DA3 benchmark.
- Pose estimation: Camera pose estimation results show top-down trajectories for ten input views, with black dots marking the starting camera points.The qualitative comparison is presented in Figure 13.
- 3D reconstruction: 3D reconstruction results visualize point clouds produced from ten input views on the DA3 benchmark.The qualitative comparison is presented in Figure 14.
- Image restoration: Image restoration results show three selected views from each ten-view input set on the DA3 benchmark.The qualitative comparison is presented in Figure 15.
- Depth estimation: Depth estimation results show three selected views from each ten-view input set on the DA3 benchmark.The qualitative comparison is presented in Figure 16.