PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyGS2Mesh: Feed-Forward Mesh Reconstruction from 3D Gaussian Splatting with Arbitrary-Resolution Views

Yuxuan Song, Fan Gao, Yibo Zhao, Jiarui Wen, Youcheng Cai, Ligang Liu

arXiv:2609.03304v1cs.CG

TL;DR

Existing mesh reconstruction from Gaussian scene representations often depends on iterative, scene-specific optimization, while post-hoc stereo-depth fusion can remain heuristic and underuse Gaussian geometry. AnyGS2Mesh combines Gaussian-guided reasoning, variable-resolution streaming, and hybrid depth refinement for feed-forward mesh reconstruction; it reports improvements to existing Gaussian pipelines across DTU and Tanks and Temples.

  • Problem

    Direct high-quality mesh extraction from 3DGS is challenging because Gaussian primitives need not form coherent surfaces, and existing methods rely on scene-specific optimization.

    Source: Introduction · p. 1

  • Method

    AnyGS2Mesh uses Gaussian-guided spatial reasoning, streaming patchwise encoding, and hybrid depth refinement to convert 3DGS into meshes for variable-count, variable-resolution views.

    Source: Introduction · p. 2, Method · p. 3

  • Results

    Across DTU and Tanks and Temples, AnyGS2Mesh improves existing Gaussian pipelines; on DTU, it reduces 2DGS mean Chamfer distance from 0.799 to 0.653.

    Source: Results and Evaluation · p. 6, Conclusion · p. 7

  • Takeaways & Limitations

    AnyGS2Mesh can serve as a plug-in reconstruction module for different Gaussian pipelines, with qualitative results also showing stable reconstruction on unbounded real-world scenes.

    Source: Conclusion · p. 7

    The prior GS2Mesh approach still relies on heuristic stereo-depth fusion and does not fully exploit explicit Gaussian geometry.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Existing 3D mesh reconstruction methods from Gaussian scene representations predominantly rely on iterative optimization, resulting in slow inference and limited scalability to high-resolution inputs. In this paper, we present AnyGS2Mesh, the first feed-forward framework for directly reconstructing 3D meshes from 3D Gaussian Splatting representations with support for arbitrary input image resolutions. Our approach incorporates a Gaussian-Guided Transformer architecture that exploits explicit 3D geometric priors for efficient mesh generation. We introduce three key components: (1) a Gaussian-Guided Spatial Reasoning Transformer represents Gaussian primitives as structured 3D tokens and jointly reasons over Gaussian and image features; (2) a Streaming and Patchwise Geometry Encoder processes native-resolution views sequentially and aggregates information across variable-length view sets; (3) a Scale-Aligned Hybrid Depth Refiner uses a PatchFusion-style encoder--decoder to fuse RGB-conditioned predicted depth with Gaussian-rendered metric depth, combining fine local structures with globally consistent metric scale. The refined depth maps are integrated through TSDF fusion, followed by Marching Cubes for deterministic mesh extraction. Extensive experiments show that AnyGS2Mesh achieves state-of-the-art reconstruction quality while significantly reducing inference time compared with optimization-based baselines, enabling near-real-time, high-quality mesh reconstruction. Our results demonstrate the potential of combining Gaussian representations and feed-forward Transformer architectures for scalable 3D geometry reconstruction. The code will be made publicly available upon acceptance.

Introduction

Direct mesh extraction from 3D Gaussian Splatting remains challenging because Gaussian primitives need not form coherent surfaces, and existing high-quality approaches rely on per-scene optimization. AnyGS2Mesh proposes a feed-forward alternative that reasons over Gaussian geometry and images while supporting variable numbers and resolutions of views.

Introduction

  • Existing surface-reconstruction methods can produce high-quality meshes, but remain dependent on scene-specific optimization.Examples include SuGaR, 2DGS, and QGS.Source: Introduction · p. 1
  • GS2Mesh uses rendered stereo views and a pretrained depth model, but its fusion is heuristic and it underuses Gaussian geometry.This motivates integrating explicit Gaussian information into the reconstruction process.Source: Introduction · pp. 1–2
  • Variable view counts and resolutions challenge conventional vision transformers, which typically require fixed-length token sequences.The paper notes that synthesized 3DGS views vary in both number and resolution.Source: Introduction · p. 2
  • AnyGS2Mesh directly converts 3DGS into meshes with feed-forward inference, supporting arbitrary numbers and resolutions of input images.Its three components are GSRT, SPGE, and SHDR.Source: Introduction · p. 2
  • Across DTU, Mip-NeRF 360, and Tanks and Temples, AnyGS2Mesh reports reconstruction quality comparable to optimization-based approaches with substantially reduced inference time.The paper frames its contributions as Gaussian-and-image reasoning, streaming patchwise encoding, and scale-aligned hybrid depth refinement.Source: Introduction · p. 2

Related Work

Prior work improves Gaussian-based surface geometry through representation changes or added geometric supervision, while transformer methods amortize multi-view reasoning. AnyGS2Mesh targets post-hoc mesh conversion by combining primitive-level Gaussian guidance with native-resolution streaming and metric depth alignment.

Gaussian Splatting-based Surface Reconstruction

  • Surface-oriented Gaussian methods improve geometry through surface-aligned primitives or explicit depth and geometry guidance, but remain tied to scene-specific fitting.Examples include SuGaR, 2DGS, QGS, RaDe-GS, PGSR, and GausSurf.Source: Gaussian Splatting-based Surface… · p. 2
  • GS2Mesh instead converts existing 3DGS through rendered stereo views, while AnyGS2Mesh exposes primitive attributes to a learned geometric reasoner.This preserves the post-hoc conversion setting while adding direct primitive-level reasoning.Source: Gaussian Splatting-based Surface… · p. 2

Feed-forward 3D Reconstruction

  • Transformer approaches amortize multi-view geometry reasoning, including VGGT's feed-forward prediction of cameras, depth maps, and point maps.Other work extends this paradigm with depth and camera priors or Gaussian scene generation.Source: Feed-forward 3D Reconstruction · p. 2
  • Existing approaches address image scale or sequence length separately, but do not combine native-resolution streaming with primitive-level 3DGS guidance for post-hoc mesh conversion.AnyGS2Mesh targets this combination alongside metric depth alignment.Source: Feed-forward 3D Reconstruction · p. 2

Preliminaries

The preliminaries describe 3DGS as a scene representation made of anisotropic Gaussian primitives, then introduce VGGT as a feed-forward template for multi-view geometric prediction. AnyGS2Mesh builds on that template while additionally using the metric structure encoded in its Gaussian input.

3DGS

  • 3DGS represents a scene with anisotropic Gaussian primitives carrying position, covariance, opacity, and color.The covariance is typically parameterized by learnable scale and rotation.Source: Preliminaries · pp. 2–3
  • For a camera view, 3DGS projects primitives into 2D Gaussian footprints and composites their colors by depth-sorted alpha blending.Source: Preliminaries · p. 3

VGGT

  • VGGT encodes each input frame, exchanges information across views, and decodes geometry-aware tokens into camera, depth, and point predictions.This feed-forward template avoids scene-specific optimization; AnyGS2Mesh additionally exploits metric 3D structure from Gaussian primitives.Source: Preliminaries · p. 3

Method

AnyGS2Mesh converts a 3D Gaussian Splatting scene and calibrated views into a mesh through three learned modules. Its design supports variable-resolution views and variable-length view sets.

Method

  • Predicted per-view depth maps are integrated with TSDF fusion to extract a mesh from the input 3DGS scene.The pipeline then uses Marching Cubes for mesh extraction.Source: Method · p. 3, Figure 2 · p. 4

Gaussian-Guided Spatial Reasoning Transformer

  • GSRT couples multi-view image tokens with projected Gaussian tokens through view-aligned spatial attention, then decodes visual depth and Gaussian-derived metric depth.The visual depth preserves fine local detail but may lack metric alignment; the Gaussian-derived depth is metric, and surface-aware parameterization aligns decoded Gaussians with the underlying surface.Source: Gaussian-Guided Spatial Reasoning… · pp. 3–4

Streaming and Patchwise Geometry Encoder

  • SPGE encodes each native-resolution view as spatial patches and processes views sequentially, reusing cached key–value representations to aggregate variable-length inputs.Intra-patch and cross-patch attention preserve local detail while propagating global layout information.Source: Streaming and Patchwise Geometry Encoder · pp. 4–5

Scale-Aligned Hybrid Depth Refiner

  • SHDR aligns visual depth to the Gaussian metric frame, then predicts per-pixel fusion weights and residual corrections to combine the two depth sources.The Gaussian depth anchors metric scale, while aligned visual depth preserves image-aligned details; the result has globally consistent depth with sharp boundaries.Source: Scale-Aligned Hybrid Depth Refiner · p. 5

Loss Function

  • The loss supervises refined depth and induced surface normals and adds multi-view consistency, with separate weights for depth, normal, and consistency terms.The multi-view consistency loss is adopted from Chen et al. 2024.Source: Loss Function · p. 5

Experiments

Experiments assess AnyGS2Mesh on DTU and Tanks and Temples, with qualitative Mip-NeRF 360 results and component ablations. The method improves existing Gaussian pipelines and reconstructs meshes without scene-specific mesh optimization.

Experiment Setup

  • Training mixes nine public datasets; evaluation uses DTU Chamfer distance, Tanks and Temples F-score, and qualitative Mip-NeRF 360 results.DTU evaluation covers 15 scans; Tanks and Temples evaluation covers four intermediate scenes, with all methods using the same splits and metrics.Source: Experiment Setup · p. 5

Results and Evaluation

  • AnyGS2Mesh lowers 2DGS mean Chamfer distance from 0.799 to 0.653 and GGGS from 0.475 to 0.468, yielding the best Table 1 average.For rendering-oriented 3DGS, reconstruction accuracy is comparable to GS2Mesh while post-processing falls from 83 seconds to 30 seconds.Source: Table 1 · p. 6, Results and Evaluation · p. 6
  • 3DGS mean F-score rises from 0.110 to 0.417, while GGGS+Ours reaches 0.703 versus 0.693 for GGGS on Tanks and Temples.The standalone model remains behind strong surface-optimized methods under severe occlusion; GGGS+Ours achieves the best mean result.Source: Table 2 · p. 6, Results and Evaluation · p. 6
  • Mip-NeRF 360 results show coherent meshes on unbounded scenes without fixed-resolution resizing or scene-specific mesh optimization.Qualitative results attribute local-structure preservation to streaming patchwise encoding and broader metric priors to Gaussian-Guided reasoning; no quantitative mesh metrics are reported.Source: Results and Evaluation · pp. 6–7, Figure 3 · p. 7

Ablation Study

  • Removing the 2D visual stream lowers Tanks and Temples F1 to 0.108 and raises DTU Chamfer distance to 1.830, the largest ablation degradation.The authors attribute this result to the importance of high-resolution visual evidence.Source: Ablation Study · p. 7
  • Removing the 3D Gaussian stream yields TnT F1 0.282 and DTU Chamfer 0.882; removing spatial reasoning yields 0.346 and 0.791, respectively.The authors identify explicit metric Gaussian priors and view-aligned cross-stream interaction as important to effective feature fusion.Source: Ablation Study · p. 7
  • Removing SHDR yields TnT F1 0.315 and DTU Chamfer distance 0.921, supporting the effectiveness of scale-aligned depth fusion.Across both datasets, the paper reports that all components contribute to reconstruction quality.Source: Ablation Study · p. 7

Conclusion

AnyGS2Mesh converts existing 3D Gaussian Splatting representations into explicit meshes without scene-specific mesh optimization. It combines spatial reasoning, native-resolution streaming visual encoding, and hybrid depth refinement, and improves existing Gaussian representations on DTU and Tanks and Temples.

  • AnyGS2Mesh converts existing 3D Gaussian Splatting representations into explicit meshes without scene-specific mesh optimization.Source: Conclusion · p. 7
  • Combining Gaussian-Guided spatial reasoning, native-resolution streaming visual encoding, and scale-aligned hybrid depth refinement makes AnyGS2Mesh a plug-in module for different Gaussian pipelines.Source: Conclusion · p. 7
  • Experiments show improvements to 2DGS and GGGS on DTU and Tanks and Temples, with stable reconstruction on unbounded real-world Mip-NeRF 360 scenes shown qualitatively.Source: Conclusion · p. 7

Supplementary Material

The supplementary methods describe how GSRT aligns Gaussian tokens with image patches and how SPGE encodes native-resolution views and aggregates them sequentially.

A.1 View-Aligned Spatial Reasoning in GSRT

  • GSRT assigns Gaussian tokens to image patches and lets each visual patch interact only with Gaussian tokens projected to its image support.It uses view-aligned spatial interaction rather than dense attention between all 2D and 3D tokens.Source: §A.1 · p. 8
  • GSRT rejects geometrically inconsistent Gaussians using rendered depth, retaining tokens in front of the camera and within 10% relative depth of the visible surface.Tokens must also project inside the valid image region; at most 50 candidates are retained per patch, ranked by depth residual.Source: §A.1 · p. 8
  • Each visual patch attends to its assigned Gaussian features, producing a Gaussian-conditioned token that is fused with the original patch tokens for patch-level refinement.The fusion operation is applied once after the final attention group.Source: §A.1 · p. 8

A.2 SPGE for Native-Resolution and Streaming Inputs

  • SPGE preserves each view’s native aspect ratio and valid extent, padding only bottom and right boundaries instead of resizing views to a shared spatial shape.Patch coordinates remain in the global image grid across local windows.Source: §A.2 · p. 9
  • SPGE conditions visual patch tokens on RGB, depth, and normal inputs, using separate depth and normal adapters while leaving the pretrained RGB encoder unchanged.The depth adapter takes normalized depth and mask; the normal adapter takes normal and mask.Source: §A.2 · p. 9
  • Alternating local-window and cross-patch attention preserves fine structures within windows while exchanging information between spatially separated regions.Global positional encoding keeps interactions consistent across window boundaries and reduces patch-seam artifacts without resizing.Source: §A.2 · p. 9
  • SPGE aggregates views in input order, with each current view querying its own tokens and a cache of preceding views.The streaming and cache updates are described as causal view aggregation.Source: §A.2 · p. 9

B Training Data and Implementation Details

This section provides additional details about the training protocol, model initialization, and experiments.

B Training Data and Implementation Details

B.1 Training datasets.

Training uses nine public datasets spanning varied scene types, with metric-depth supervision available across the collection.

B.1 Training datasets.

  • Training uses nine public datasets spanning synthetic and real data, indoor and outdoor environments, and object-centric and scene-level captures.The mixture includes ARKitScenes, Kubric, BlendedMVS, ETH3D, CO3D-v2, Google Scanned Objects, WildRGBD, Objaverse, and the DTU training split.Source: §B.1 · p. 10
  • All nine datasets provide usable metric-depth ground truth, while surface normals come from dataset annotations or are computed from depth or 3D geometry.The calibrated RGB images and depth maps form multi-view training samples and provide observations for constructing input 3DGS representations.Source: §B.1 · p. 10
  • At each training iteration, one dataset is sampled according to a dataset-level probability distribution.Table 4 summarizes dataset types and depth-supervision sources; sampling probabilities are to be reported with the released training configuration.Source: §B.1 · p. 10

B.2 Construction of Training Inputs

Training inputs are constructed from scene-level 3DGS and view-wise data; the model initialization details describe which components are jointly optimized and how datasets contribute scenes.

B.2 Construction of Training Inputs

  • Scenes with more than 300 images are randomly limited to 300 views; otherwise, all available views are retained.Source: §B.2 · p. 10

B.3 Model Initialization

  • The input scene-level 3DGS and camera parameters remain fixed during training, while the visual backbone, SPGE, Gaussian stream, reasoning layers, prediction heads, and SHDR are jointly optimized.The Gaussian renderer, TSDF fusion, and Marching Cubes have no learnable parameters.Source: §B.3 · p. 10
  • Approximately 100–200 scenes, sequences, or objects are selected from each dataset when available; datasets with fewer scenes contribute all available training scenes.Source: Table 4 · p. 11

C More Results

The additional results assess SPGE scalability: runtime grows gradually as view count increases, while native-resolution processing supports variable-length inputs with different resolutions.

C.1 View-Count Scalability and Native-Resolution Processing

  • Runtime rises gradually from approximately 25 seconds for 10 views to 45 seconds for 350 views as added views require more patch tokens and causal aggregation.The reported increase is gradual despite the much larger view set.Source: §C.1 · p. 11, Figure 4 · p. 11
  • SPGE accepts variable-length views at their native resolutions and aspect ratios, subject to available computational resources.Patchwise encoding and cached causal aggregation support view sequences without resizing all images to a shared spatial shape.Source: §C.1 · p. 11
  • Unlike standard VGGT's bounded-view, prescribed-resolution setup, native-resolution patches avoid accuracy degradation caused specifically by mandatory downsampling.Runtime still increases with the number of input images, trading computational cost for input flexibility and reconstruction quality.Source: §C.1 · p. 11

C.2 Post-processing Compatibility on DTU

Additional comparisons show AnyGS2Mesh producing more complete, coherent geometry on DTU and generalizing to unbounded Mip-NeRF 360 scenes without scene-specific mesh optimization. Component analyses show how image detail and Gaussian-derived metric structure complement one another in depth refinement.

C.2 Post-processing Compatibility on DTU

  • The same trained model improves standard 3DGS and EDGS scenes on DTU without updating Gaussian primitives or scene-specific fine-tuning.It removes floating geometry, fills incomplete surfaces, and improves continuity while preserving appearance and fine structures.Source: §C.2 · p. 11, Figure 5 · p. 12

C.3 Qualitative Comparison with Gaussian-to-Mesh Methods

  • On DTU, AnyGS2Mesh recovers more complete surfaces and preserves thin structures and depth discontinuities; on Mip-NeRF 360, it generalizes to unbounded scenes without scene-specific mesh optimization.Comparisons include 3DGS, 2DGS, GS2Mesh, and RaDe-GS; DTU ground-truth geometry is shown in the last column.Source: §C.3 · p. 12, Figure 7 · p. 13, Figure 8 · p. 14

C.4 Analysis of Intermediate Depth and Geometry Predictions

  • GSRT associates Gaussian features with visual patches, enabling image tokens to reason jointly about native-resolution appearance and explicit 3D geometry.The visual decoder predicts image-conditioned depth, while the Gaussian decoder predicts surface-aware primitives used to render metric depth.Source: §C.4 · pp. 12–13
  • Image-conditioned depth preserves fine structures and sharp boundaries, whereas Gaussian-rendered depth supplies metric structure and stronger cross-view consistency.The Gaussian depth shares the input scene's coordinate system but can contain floating structures, noisy backgrounds, incomplete surfaces, and inaccurate boundaries, especially around thin objects, occlusions, and sparse regions.Source: §C.4 · pp. 12–13
  • The full model's Gaussian stream and scale-aligned refinement suppress floating artifacts and improve foreground-background separation and spatial coherence over visual-only depth.Examples include clearer table-to-background separation, preserved loader silhouettes, and more coherent furniture boundaries.Source: §C.4 · pp. 12–13, Figure 9 · p. 14

Related papers

Loading 2609.03304v1…