PaperlayerView exact revision on arXiv

Source-linked AI summary

SurGe: Improved Surface Geometry in Point Maps

Karim Knaebel, Gonzalo Martin Garcia, Christian Schmidt, Ilya Fradlin, Lucas Nunes, Daan de Geus, Bastian Leibe

arXiv:2605.31577v1cs.CV

TL;DR

Feedforward point-map models recover global 3D geometry well but retain local surface artifacts that standard evaluations capture weakly. SurGe introduces normal-based evaluation, gradient matching, and Neighborhood Attention decoding, improving local geometry across eight zero-shot benchmarks while preserving strong global performance.

  • Problem

    Point-map models estimate plausible global geometry but retain visible local surface artifacts that standard point-position metrics capture only indirectly.

  • Method

    SurGe combines a point-map normal metric, scale-invariant point gradient matching loss, and a Neighborhood Attention Decoder for local surface geometry.

  • Results

    SurGe improves local point-map and point-map normal performance across eight zero-shot benchmarks while preserving strong global geometry.

  • Takeaways & Limitations

    The point-map normal metric makes local surface quality a more explicit evaluation target for geometry models.

  • Takeaways & Limitations

    The gradient-matching loss omits pairs near occlusion boundaries and high-frequency terms when annotations are noisy or sparse.

Abstract

from arXiv · show

Recent feedforward 3D reconstruction methods predict point maps and estimate global 3D geometry remarkably well. However, their predictions still exhibit inaccurate local surface geometry, which is clearly visible qualitatively but only weakly reflected in common metrics. To make these errors more explicit in evaluation, we introduce a point map normal metric that evaluates the local surface orientation induced by neighboring 3D predictions. To reduce these errors, we propose two complementary components: a point gradient matching loss that supervises depth-normalized 3D finite differences, and a Neighborhood Attention Decoder (NAD) that progressively upsamples features and uses Neighborhood Attention for local feature mixing. Across eight zero-shot monocular geometry benchmarks, our model, SurGe, achieves the best average rank for global point map AbsRel and consistently improves local point map and point map normal evaluations.

1 Introduction

The introduction identifies inaccurate local surface geometry as a weakness of point-map reconstruction, especially on thin structures, and attributes its under-detection to pointwise metrics. It proposes point-map normal evaluation, point gradient matching loss, and a Neighborhood Attention Decoder to improve local geometry while preserving global performance.

  • Thin structures can appear bent or oscillatory in 3D even when their depth maps look plausible, because neighboring predicted points form inconsistent surface patches.The distortion worsens as structures become thinner and backgrounds farther away.
  • Standard point-map metrics weakly reflect local surface geometry because they average aligned 3D point-position errors and can overlook incoherent neighboring surfaces.Low- and high-frequency perturbations can receive nearly identical AbsRelglob and AbsRelloc scores despite different surface coherence, whereas MAEnormal reflects the degradation.
  • The Neighborhood Attention Decoder replaces convolutional mixing while retaining progressive multiscale decoding, selectively aggregating local evidence to improve thin structures and local surface geometry.It avoids the cost of full-resolution self-attention and patch-aligned artifacts associated with pure ViT decoding.
  • Combined with point gradient matching, the model achieves state-of-the-art local point map and point map normal evaluations and best global point map results on several zero-shot benchmarks.The introduction reports these gains relative to recent point-map models.
  • The proposed MAEnormal metric evaluates local surface quality through normals induced by neighboring point differences, improving sensitivity to surface-geometry errors.It is introduced as an evaluation metric that better reflects local surface quality than standard point-map metrics.
  • The scale-invariant point gradient matching loss Lpgm supervises local surface structure in point maps.Its formulation is inspired by log-depth gradient matching.

2 Related Work

Recent geometry-estimation methods largely combine strong ViT backbones with convolutional or attention-based decoders, including monocular and multi-view architectures for point-map prediction. Despite strong overall performance, existing pointwise metrics and methods leave local surface coherence insufficiently measured and modeled.

  • Monocular geometry estimation: Most monocular geometry architectures pair a DINOv2-initialized plain ViT with a convolutional multi-resolution decoder, such as DPT or ConvStack.These decoders are commonly used for geometry estimation across several recent architectures.
  • Multi-view geometry estimation: Multi-view methods adapt ViT-based architectures with inter-frame feature fusion and per-frame decoders, predicting point maps in shared or image-specific coordinates.DUSt3R predicts point maps from two input views in a shared coordinate system, while later methods use global attention over all frames for large image sets.
  • Local surface geometry: Existing methods achieve impressive geometry-estimation performance through stronger backbones, feature fusion, large-scale training, and improved supervision, yet local surface predictions remain poor around thin structures.The paper reports that its Neighborhood Attention Decoder visibly improves this local geometry compared with DPT heads and ConvStack decoders.
  • Point-map evaluation: After alignment, global affine-invariant AbsRel and local point-map metrics remain pointwise and do not test whether neighboring predictions form coherent surfaces.The local metric evaluates instance-mask regions after separate local alignment, motivating a point-map normal metric.
  • Neighborhood Attention: Neighborhood Attention restricts each query to a local neighborhood, providing an inductive bias or computational savings through sparse attention.Prior work applies it in hierarchical encoders, isometric Transformers, encoder-decoder systems, medical imaging, and image restoration.

3 Method

SurGe combines a DINOv2-initialized ViT encoder with a Neighborhood Attention Decoder and trains on global, local, and point gradient matching losses. The proposed loss matches depth-normalized 3D finite differences to supervise local direction and displacement magnitude while remaining pairwise scale-invariant.

  • Architecture: SurGe predicts a dense per-pixel 3D point map using a DINOv2-initialized ViT encoder, Neighborhood Attention Decoder, and global, local, and point gradient matching losses.The decoder progressively upsamples encoder features to produce point predictions.
  • Neighborhood Attention Decoder: NAD replaces global self-attention with Neighborhood Attention and uses window-matched RoPE, omitting pre-attention and pre-FFN LayerNorm.The attention window is k = 9, head dimension is d_h = 64, and LayerNorm removal empirically improves accuracy.
  • Neighborhood Attention Decoder: NAD uses five stages, with stages 1–4 upsampling by 2×, and predicts (ξeρ, ηeρ, eρ) from output variables (ξ, η, ρ).Each stage contains three NAD blocks; channel dimensions are (1024, 512, 256, 128, 64).
  • Surface supervision: Lpgm matches full 3D, depth-normalized finite differences between neighboring predicted and ground-truth points, supervising both local direction and magnitude with pairwise scale invariance.Normalization uses the z coordinate and the nearer endpoint depth as a conservative local scale.
  • Surface supervision: Lpgm is evaluated only on annotated neighboring pairs and excludes pairs near occlusion boundaries, which slightly improves training by removing largely irreducible residuals.For noisier or sparse annotations, high-frequency loss terms are omitted according to label quality.
  • Training objective: The full dense-label objective is L = Lglob + Lloc,4 + Lloc,16 + Lloc,64 + 10Lpgm.The coefficient 10 was selected empirically to balance Lpgm with the global point loss.

4 Experiments

SurGe is evaluated zero-shot on eight monocular geometry benchmarks using global and local point-map metrics, including the proposed normal error. It achieves the strongest local results while maintaining highly competitive global accuracy, with ablations attributing improvements to NAD and Lpgm.

  • Evaluation: SurGe is evaluated zero-shot on eight monocular geometry benchmarks using global and instance-wise local AbsRel metrics, plus MAEnormal where dense ground truth supports reliable normal estimation.MAEnormal is computed from local normals formed by cross products of adjacent point-map differences; lower is better for all metrics.
  • Main results: SurGe achieves the lowest AbsRelloc on every evaluated dataset and follows the same pattern for MAEnormal, indicating more accurate local surfaces rather than only lower pointwise error.AbsRelloc measures pointwise accuracy after instance-level alignment, whereas MAEnormal measures local surface orientation induced by neighboring predictions.
  • Main results: SurGe ranks first on four of eight datasets and obtains the best average rank for global point-map accuracy, without sacrificing global scene geometry.Global accuracy remains highly competitive with MoGe, MoGe-2, and π3.
  • Main results: Qualitatively, SurGe preserves thin structures and produces smoother surfaces with sharper geometric detail, whereas MoGe-2 and InfiniDepth can recover coarse shape but exhibit irregular local surfaces.The differences are visible in rendered 3D geometry and point-map normals.
  • Decoder design: Decoder ablations attach NAD, DPT, ConvStack, ConvStack-L, and ViT decoders to the same DINOv2 ViT-Large backbone and shared losses, isolating decoder architecture.Compared with ConvStack-L, NAD produces less warped thin structures, including chair legs and the wall region.
  • Surface loss: With NAD fixed, Lpgm improves local surface evaluations over prior surface losses and generally outperforms Lgm globally, providing a practical replacement without sacrificing global point-map accuracy.Both gradient-based losses improve local evaluations over Lnormal, while Lgm generally performs worse than Lnormal globally; Lpgm is attributed to remaining in the 3D point-displacement space.

5 Conclusion

SurGe targets local surface geometry in monocular point maps, improving local point map and point map normal performance across eight zero-shot benchmarks while preserving strong global geometry. The gains stem from both the Neighborhood Attention Decoder (NAD) head and point gradient matching loss (Lpgm), alongside a point map normal metric that exposes surface artifacts weakly captured by existing evaluation.

  • Conclusion: SurGe improves local point map and point map normal performance across eight zero-shot benchmarks while preserving strong global geometry.The model is designed to improve local surface geometry rather than only average point accuracy.
  • Conclusion: Ablations show that both the NAD head and Lpgm contribute to SurGe’s gains.NAD denotes the Neighborhood Attention Decoder, while Lpgm is the point gradient matching loss.
  • Conclusion: The point map normal metric exposes surface artifacts that existing metrics capture only weakly.This extends the paper’s contribution beyond the model to improved geometric evaluation.

Appendix · A Details on the Point Gradient Matching Loss

This appendix specifies the point gradient matching loss Lpgm, which matches depth-normalized 3D displacements to optimize local gradient orientation and magnitude. The loss is scale-invariant and applies only to pairs whose involved points are valid.

  • A Details on the Point Gradient Matching Loss: Lpgm matches predicted and ground-truth 3D displacements to supervise the orientation and magnitude of local point gradients.Depth normalization makes the loss scale-invariant.
  • A Details on the Point Gradient Matching Loss: The loss is computed only for point pairs where all involved points are valid.Validity is supplied through the valid mask for predicted and ground-truth pair endpoints.
  • A Details on the Point Gradient Matching Loss: The pseudocode defines point_gradient_loss over predicted and ground-truth point maps with shape [B, H, W, 3] and validity masks shaped [B, H, W].These inputs provide batched spatial point maps and per-pixel validity indicators.
  • A Details on the Point Gradient Matching Loss: The appendix presents this procedure as Pseudocode A: Point gradient matching loss Lpgm.The specification exposes the loss inputs, pairwise gradient construction, directional terms, and final aggregation.
  • A Details on the Point Gradient Matching Loss: For each pair, predicted and ground-truth gradients are formed by subtracting endpoint coordinates and dividing by the minimum endpoint depth.The normalization uses the third coordinate of both endpoints.
  • A Details on the Point Gradient Matching Loss: The implementation computes horizontal gradients from adjacent columns of the predicted and ground-truth point maps.Validity masks for the corresponding column pairs are passed to pair_loss.
  • A Details on the Point Gradient Matching Loss: The implementation computes vertical gradients from adjacent rows of the predicted and ground-truth point maps.Validity masks for the corresponding row pairs are passed to pair_loss.
  • A Details on the Point Gradient Matching Loss: Lpgm returns the average of the horizontal and vertical pair losses.The pseudocode computes (loss_dx + loss_dy) / 2.

B Architectural Details

The NAD decoder combines stage-wise UV embeddings with window-matched RoPE to encode absolute positions and local relative offsets. Training uses a reciprocal square root learning-rate schedule with warmup and cooldown, while normalization removal improves accuracy but can cause instability.

  • Positional Encoding: Window-matched RoPE supplies relative offset representations inside local attention windows, with its base frequency adapted to the attention kernel size.UV coordinates provide absolute positions, while RoPE helps attention patterns transfer across neighborhoods.
  • Normalization: Removing LayerNorm before NAD attention and FFN blocks improves accuracy but occasionally causes irrecoverable training instabilities.The authors also tested RMSNorm, DyT, and LayerScale as alternatives.
  • Optimization: The reciprocal square root learning-rate schedule uses a 2400-step timescale, 1000-step linear warmup, and a 10% cooldown to 10^-2 of the peak learning rate.The schedule supports a larger initial learning rate than standard cosine decay because early training remains stable enough for it.
  • Positional Encoding: Stage-wise UV embeddings provide aspect-ratio-aware absolute position information by linearly projecting normalized image-coordinate grids at each NAD stage.The coordinates are normalized by the image diagonal and matched to each stage’s feature-map resolution.

C Training Details

SurGe models and ablations use a 120,000-step AdamW training setup with staged learning-rate warmup and backbone freezing. Training progressively increases image resolution and encoder-token budgets, with a final cooldown restricted to high-quality synthetic data.

  • Optimization: All models and ablations train for 120,000 optimizer steps with total batch size 128 using AdamW, gradient clipping at 1.0, and a reciprocal square root schedule.Peak learning rates are 3×10−4 for the decoder and 3×10−5 for the DINOv2 backbone.
  • Optimization: The decoder learning rate warms up for 1,000 steps while the backbone remains frozen, followed by a 1,000-step backbone warmup after unfreezing.The backbone warmup ends at 0.1 times the decoder learning rate.
  • Data and Compute Scheduling: During the first 80% of training, images use a fixed area of 512² pixels, target aspect ratios in [0.5, 2.0], and an encoder-token budget of 1024.In the final 20%, image areas are sampled uniformly from [512², 960²] pixels and token budgets from {1024, . . . , 280²}.
  • Data and Compute Scheduling: The final 10% cooldown uses only high-quality synthetic data, excluding the Ken Burns and G-Objaverse datasets.All other synthetic training datasets are included during cooldown.

D Training Data

SurGe is trained on a publicly available subset of MoGe-2’s datasets, using a similar mix and weighting. The training collection spans outdoor, indoor, in-the-wild, object, driving, SfM, LiDAR, and synthetic data.

  • Training dataset: SurGe trains on a subset of MoGe-2’s datasets, all publicly available for academic use.The full dataset list is provided in Table A.
  • Training dataset: The training mix includes diverse domains and sources, including Argoverse2 LiDAR, ARKitScenes and BlendedMVS SfM, and multiple synthetic datasets.Listed domains include outdoor/driving, indoor, in-the-wild, and object scenes.
  • Runtime: DINOv2-giant NAD (ours) reaches 27.15 versus 20.89 for ConvStack-L, with 1.30× relative runtime and 5.23 peak memory.Runtime uses NVIDIA H100 median latency over 50 iterations at 512×512 resolution and batch size 1.
  • Runtime: DINOv2-Large NAD (ours) reaches 21.15 versus 14.53 for ConvStack-L, with 1.46× relative runtime and 2.02 peak memory.The comparison uses compiled models, bf16 encoder autocast, fp32 decoders, and PyTorch peak-memory measurement.
  • Runtime: NAD (ours) reaches 13.56 versus 5.94 for ConvStack-L without an encoder, with 2.28× relative runtime and 0.88 peak memory.All reported runtime measurements use the stated H100 evaluation protocol.

E Decoder Runtime Tradeoff

NAD improves local surface quality over convolutional decoder baselines but increases inference latency and peak memory. At 512×512 resolution, its runtime cost is larger for the decoder alone and smaller in the full model because the shared encoder contributes to total inference time.

  • Runtime tradeoff: 2.28× slower than ConvStack-L is the decoder-only inference cost of NAD at 512×512 resolution.This runtime tradeoff accompanies improved local surface quality over convolutional decoder baselines.
  • Runtime tradeoff: 1.30× with DINOv2-giant and 1.46× with DINOv2-Large are the full-model slowdowns introduced by NAD.The slowdown is smaller than for the decoder alone because the shared encoder accounts for part of total inference time.
  • Runtime tradeoff: Peak memory increases modestly with NAD, making it an accuracy-oriented decoder design with higher computational cost.The reported tradeoff is improved local surface quality versus increased inference latency and memory.

F Additional Qualitative Examples

SurGe is qualitatively compared with recent methods on in-the-wild scenes featuring large depth ranges and high dynamic range. The comparisons show cleaner local surface geometry and additional qualitative predictions from SurGe.

  • In-the-wild comparisons: In large-depth-range scenes, PPD and InfiniDepth compress distant structure, while their normal maps reveal surface noise and artifacts.VGGT is less affected but can still break near-field geometry or slightly compress distant structure.
  • Additional SurGe examples: Additional qualitative results from SurGe include RGB inputs, predicted depth, point map normals, and rendered point maps.Figures D, E, and F present these additional examples.
  • In-the-wild comparisons: In a high-dynamic-range scene, VGGT, InfiniDepth, and PPD collapse distant geometry, while VGGT oversmooths surfaces.InfiniDepth, MoGe-2, and MoGe also introduce oscillatory or bending artifacts on table and bench legs.
  • In-the-wild comparisons: SurGe produces cleaner local surface geometry than MoGe in a high-dynamic-range scene and avoids the distant-structure and near-camera failures shown by competing methods.The comparison specifically reports that InfiniDepth and PPD struggle with high dynamic range, while VGGT fails to reconstruct geometry close to the camera.
Loading 2605.31577v1…