PaperlayerView exact revision on arXiv

Source-linked AI summary

TriSplat: Simulation-Ready Feed-Forward 3D Scene Reconstruction

Weijie Wang, Zimu Li, Jinchuan Shi, Zeyu Zhang, Botao Ye, Marc Pollefeys, Donny Y. Chen, Bohan Zhuang

arXiv:2605.26115v1cs.CV

TL;DR

Sparse-view, pose-free reconstruction methods often represent scenes with implicit-surface Gaussians, requiring costly post-hoc mesh extraction for simulation and physics applications. TriSplat instead predicts oriented triangle primitives and camera parameters in one feed-forward pass, producing more accurate surfaces and stronger mesh-rendering quality than Gaussian feed-forward baselines across multiple datasets.

  • Problem

    Existing feed-forward Gaussian and point-map models expose surfaces only implicitly, so usable meshes require costly post-hoc extraction, especially with sparse or unknown camera poses.

  • Method

    TriSplat jointly predicts point-map geometry, oriented triangle attributes, appearance, and camera parameters, anchoring triangle orientation to refined predicted normals.

  • Results

    TriSplat surpasses Gaussian feed-forward baselines in mesh-rendering quality and consistently outperforms them on surface-accuracy metrics across RealEstate10K and DL3DV, with zero-shot ScanNet generalization.

  • Takeaways & Limitations

    Because its rendering primitives are exported surface triangles, TriSplat produces meshes directly consumable by physics engines and standard rendering pipelines.

  • Takeaways & Limitations

    The direct export produces a non-manifold triangle soup, making it unsuitable for applications requiring watertight meshes such as finite-element analysis.

Abstract

from arXiv · show

Sparse-view 3D reconstruction is increasingly addressed with feed-forward splatting networks that predict explicit primitives directly from images. Yet most existing methods remain centered on Gaussian primitives and expose surfaces only indirectly: extracting a usable mesh for downstream simulation, physics reasoning, or embodied interaction still requires expensive post-hoc steps that break the feed-forward promise. This limitation is especially pronounced in pose-free settings, where scene structure and camera parameters must be estimated jointly from sparse observations. We present TriSplat, a feed-forward reconstruction network that represents scenes with oriented triangle primitives and directly exports simulation-ready mesh scenes from a single forward pass. Given input images, the network predicts local 3D point maps, triangle attributes, camera poses, and optional intrinsics. Rather than regressing triangle orientation as an unconstrained latent variable, our approach constructs geometry normals from the predicted point maps, refines them with an image-conditioned normal head, and converts them into stable local frames for triangle parameterization. A mono-normal bootstrap schedule further stabilizes early training, while opacity and blur scheduling progressively sharpens the learned surface representation for direct mesh extraction. Experiments on RealEstate10K and DL3DV show that this representation produces more geometry-faithful reconstructions than Gaussian feed-forward baselines while maintaining competitive novel-view rendering quality. Because the rendering primitives are themselves surface triangles, the output can be directly ingested by physics engines, collision detectors, and standard rendering pipelines without any conversion, making it a practical simulation-ready solution for feed-forward 3D scene reconstruction.

1 Introduction

TriSplat is a feed-forward reconstruction model that predicts oriented triangle primitives, geometry, appearance, and camera parameters from sparse unposed images in one pass. Because triangles are native surface elements, its outputs can be exported directly as meshes for simulation and standard rendering without post-hoc extraction.

  • Motivation: Simulation-ready reconstruction requires explicit triangle meshes that physics engines, finite-element solvers, path tracers, and rendering systems can ingest directly.The motivation includes collision checking, contact-rich planning, and physics simulation for robotics, augmented reality, and embodied perception.
  • Limitations of Existing Methods: Gaussian and point-map feed-forward methods provide implicit or surface-less geometry, so usable mesh extraction requires costly post-hoc processing.The passage specifically identifies Gaussian primitives and point maps as lacking explicit surface structure and mentions TSDF-based extraction.
  • TriSplat: The model stabilizes triangle construction and training through geometry-anchored orientation, normal bootstrapping, and validity-aware masking.These mechanisms address the greater sensitivity of triangles to orientation errors than Gaussian splats.
  • TriSplat: TriSplat predicts local 3D point maps, per-pixel triangle attributes, camera poses, and optional focal lengths from unposed images in a single forward pass.Its triangle orientation is anchored to predicted local geometry, with geometry normals refined by an image-conditioned head.
  • Results: TriSplat surpasses Gaussian feed-forward baselines in mesh-rendering quality and surface accuracy, while avoiding the quality drop caused by lossy TSDF fusion.Its rendering primitives are already meshes, so outputs are directly consumable by physics engines and standard rendering pipelines.

2 Related Work

Prior feed-forward scene reconstruction methods favor efficient Gaussian or point-map predictions, but their volumetric or implicit surfaces limit direct mesh use. Surface-aware approaches improve geometry through normals, surfel adaptation, or related regularization, yet the supplied passage indicates remaining limitations.

  • Splatting-Based Scene Representations: 3DGS and its extensions enable real-time, high-quality novel-view synthesis, but volumetric Gaussians yield view-inconsistent depth and poorly defined surfaces.2DGS addresses this limitation by collapsing each Gaussian into a planar disk.
  • Feed-Forward Sparse-View Reconstruction: Feed-forward methods predict 3D representations in one pass from sparse images, while Gaussian-based approaches provide efficient, high-quality novel-view synthesis.Earlier image-based and NeRF-based methods regress radiance fields but retain costly volumetric rendering.
  • Feed-Forward Sparse-View Reconstruction: Pose-free methods extend direct prediction to Gaussian primitives, but their Gaussian or point-map outputs leave surface topology implicit.The listed methods include NoPoSplat, InstantSplat, Splatt3R, FreeSplatter, RegGS, UFV-Splatter, FLARE, and YoNoSplat.
  • Surface-Aware Feed-Forward Reconstruction: Surface-aware feed-forward methods strengthen geometry using normal supervision, weighted Chamfer regularization, or Nyquist-guided surfel adaptation.MeshSplat substantially improves mesh quality over baselines, while SurfelSplat introduces Nyquist-guided surfel adaptation.

3 Method

TriSplat reconstructs sparse-view scenes in one forward pass as oriented surface triangles while jointly predicting geometry, camera parameters, and primitive attributes. Geometry-anchored orientation, scheduled training softness, and direct mesh extraction make the representation simulation-ready without post-processing reconstruction.

  • Feed-forward scene representation: TriSplat jointly predicts dense local 3D point maps, per-pixel triangle attributes, camera poses, and optional intrinsics, producing oriented triangle primitives in a single forward pass.Because the rendering primitives are explicit surface triangles, the output can be exported directly as a mesh.
  • Network architecture: A DINOv2 encoder and transformer decoder use intra-view self-attention, cross-view joint attention, rotary position embeddings, and ray-direction embeddings for spatial and multi-view reasoning.Parallel heads predict point maps, camera parameters, and primitive attributes; camera rotations are projected onto SO(3) via SVD orthogonalization.
  • Geometry-conditioned orientation: Triangle orientation is anchored to predicted point-map geometry: finite-difference normals are refined with an image- and depth-conditioned U-Net, then converted into orthonormal tangent frames.The refinement head is zero-initialized so it begins as an identity mapping, avoiding early orientation perturbations.
  • Training stabilization: A mono-normal bootstrap blends pretrained monocular teacher normals with model normals through takeover, cosine-decay blending, and release phases before relying entirely on predicted geometry.The teacher normal enters triangle construction directly, shaping rendered outputs and downstream gradients rather than serving only as a matching-loss target.
  • Training stabilization: Opacity and blur schedules transition triangles from forgiving blurred primitives to sharp mesh-ready surfaces, mitigating zero-gradient failures caused by coarse early predictions.This progression addresses the greater sensitivity of hard-edged triangles to position and orientation errors compared with Gaussian primitives.
  • Mesh extraction: Mesh extraction discards low-opacity triangles, corrects winding using face and per-pixel normals, and merges nearby duplicate vertices through quantized position hashing.The triangle-native world-space output requires no auxiliary reconstruction before producing a standard triangulated mesh.

4 Experiments

TriSplat is evaluated for surface geometry, mesh-based novel-view rendering, depth and normal accuracy, runtime, and ablations across RE10K, DL3DV, and zero-shot ScanNet. It consistently improves geometric fidelity and mesh rendering while producing meshes substantially faster than Gaussian baselines.

  • Datasets and protocol: Experiments use RE10K and DL3DV, with zero-shot ScanNet evaluation using RE10K-trained models without fine-tuning.DL3DV is evaluated with 6, 12, and 24 input views; RE10K uses 6 views, while ScanNet uses 6 views.
  • Surface geometry: TriSplat attains a Chamfer Distance of 0.190 and an F1 score of 0.622 on RE10K, improving over YoNoSplat by 0.077 in CD and 0.179 in F1.Its Recall improvement is +0.227, indicating better coverage of thin structures than TSDF-fused Gaussian meshes.
  • Mesh rendering: TriSplat reaches 24.69 dB PSNR under mesh rendering on RE10K, compared to 21.94 dB for the strongest Gaussian baseline, a margin of +2.75 dB.The comparison directly evaluates exported meshes rasterized by a standard triangle pipeline.
  • Depth and normals: On zero-shot ScanNet, TriSplat achieves an AbsRel of 0.188 and an AbsDiff of 0.341, with a mean angular error of 27.9◦ and <30◦ accuracy of 71.7%.The strongest baseline records 54.1◦ mean error and 41.0% at <30◦; qualitative results show smoother, surface-aligned normals and sharper depth boundaries.
  • Runtime: TriSplat’s end-to-end time-to-mesh is 0.57 s, 0.62 s, and 1.23 s for 6, 12, and 24 views, respectively.It is 33× faster than AnySplat at 6 views and up to 249× faster than the slowest baseline at 24 views because no TSDF-fusion stage is required.
  • Ablations: Ablations show that replacing normal anchoring with an unconstrained quaternion drops F1 by 0.057 and PSNR by 1.11 dB, while removing mono-normal bootstrap yields F1 −0.065.The evaluated components target distinct failure modes in surface geometry and mesh-rendering quality.

5 Conclusion · A Implementation Details

TriSplat is a feed-forward reconstruction model that natively represents scenes with oriented triangle primitives and jointly predicts geometry, appearance, and camera parameters from sparse unposed images in one pass. Its geometry-oriented design and progressive training improve surface accuracy over Gaussian feed-forward baselines, while direct export remains limited by non-manifold topology and resolution-tied triangle density.

  • 5 Conclusion: TriSplat represents scenes natively as oriented triangle primitives.The representation is designed for feed-forward reconstruction from sparse observations.
  • 5 Conclusion: The model jointly predicts geometry, appearance, and camera parameters from sparse unposed images in a single forward pass.This combines scene and camera estimation within the feed-forward reconstruction process.
  • 5 Conclusion: Triangle orientation is anchored to predicted point-map geometry rather than learned as an unconstrained field.This geometry-based anchoring is paired with a mono-normal bootstrap to warm-start orientation learning.
  • 5 Conclusion: A mono-normal bootstrap and progressive sharpening curriculum bridge warm-started orientation learning with soft-to-crisp optimization.The curriculum progressively sharpens the learned representation during training.
  • 5 Conclusion: TriSplat attains substantially more accurate surface geometry than Gaussian feed-forward baselines.The comparison concerns surface geometry accuracy, while the model jointly handles geometry, appearance, and camera parameters.
  • 5 Conclusion: Direct export produces a non-manifold triangle soup adequate for rendering and physics but unsuitable for applications requiring watertight meshes.Finite-element analysis is given as an example of an application requiring watertight meshes.
  • 5 Conclusion: Per-pixel prediction ties triangle density to input resolution, motivating topology-aware export and adaptive tessellation as future directions.These directions address limitations in topology and resolution-dependent triangle density.

A.1 Network Architecture … B Baseline Mesh Extraction: Quantitative Details

TriSplat combines pose-free multi-view prediction with oriented triangle primitives, geometry-anchored normal refinement, and scheduled training procedures. Its supplementary implementation details specify the network heads, loss terms, training schedules, monocular normal teacher, and direct mesh-export parameters.

  • A.1 Network Architecture: The backbone uses DINOv2 ViT-L/14 with 2D rotary position embeddings, alternating intra-view and cross-view attention, 1024-dimensional tokens, and fourth-degree pixel-level intrinsic encoding.The intrinsic encoding supports pose-free operation by embedding per-pixel camera information.
  • A.1 Network Architecture: The point head predicts three-channel point maps, while the primitive head fuses RGB tokens and outputs 11 triangle attributes comprising scale, quaternion, spherical-harmonic, and blur parameters.Both dense heads use transformer decoding and token upsampling; the point output is pixel-shuffled to 14× token resolution.
  • A.2 Loss Formulation: The camera loss combines pairwise relative-translation Huber terms with relative-rotation angular terms, providing global-frame invariance and independent constraints from every view pair.The formulation supervises ordered view pairs rather than only per-view absolute camera predictions.
  • A.2 Loss Formulation: The normal loss compares refined normals with monocular teacher normals using cosine similarity over pixels passing geometry, finite-value, and optional object-mask checks.The valid-pixel set determines where the normal supervision is applied.
  • A.3 Training Protocol: Training uses 150K steps at 224×224 for RE10K, followed by 100K steps at 224×224 and 100K at 224×448 for DL3DV, with context views sampled uniformly from 2 to 8.Frame-gap ranges warm up separately for RE10K and DL3DV, and novel-view comparisons use the 224×224 model for protocol alignment.
  • A.3 Training Protocol: Progressive sharpening ramps opacity exponent from 1 to 2 and temperature from 1.0 to 5.0, while the blur multiplier decays from 1.0 to 0.5 over 16,000 steps.After 40K warm-up steps, samples exceeding total loss 0.2, MSE 0.06, or pose loss 1.0 are nearly filtered out.
  • A.4 Mono-Normal Teacher: The offline Omnidata DPT teacher uses the vitb_rn50_384 variant, with normals resized by bilinear interpolation and bootstrap timings of 6,000 and 20,000 steps.Teacher normals are computed for every input view before training.
  • B Baseline Mesh Extraction: Quantitative Details: Direct mesh export prunes triangles below opacity 0.10 at temperature 5.0, deduplicates vertices at precision 10^-5 with normal-octant keys, and completes in less than 0.1 s on one GPU.Triangle colors derive from zeroth-order spherical-harmonic coefficients using C0 ≈0.282; Gaussian baselines instead use TSDF fusion with voxel size 0.005 and alpha masking below 0.3.

C Primitive Rendering Comparison

This section compares native primitive rendering with exported-mesh rendering, showing that Gaussian baselines can score or appear stronger before mesh conversion, while TriSplat preserves rendering quality because its native primitives are the exported triangles. The comparison supports TriSplat’s simulation-ready objective by avoiding quality loss from post-hoc TSDF-based surface extraction.

  • Evaluation protocol: Primitive rendering evaluates each method with its native rasterizer, using Gaussian splatting for baselines and triangle splatting for TriSplat.The protocol reports native primitive-rendering metrics on DL3DV and RE10K before mesh export.
  • Primitive rendering: Gaussian baselines attain their strongest numerical scores under primitive rendering because smooth radial falloff provides locally forgiving gradient coverage.Their native-rendering advantage does not directly represent the quality consumed by standard mesh pipelines.
  • Primitive-to-mesh transfer: TSDF fusion discards Gaussian smooth primitives, producing substantially lower mesh-rendering PSNR than their native primitive-rendering quality.This degradation is exposed when Gaussian representations are converted for downstream mesh-based simulation and graphics pipelines.
  • Primitive-to-mesh transfer: TriSplat has markedly smaller primitive-to-mesh degradation because its rendering primitives are already the exported triangles.The same representation is used during training and inference and can be consumed directly as a mesh with minimal quality loss.
  • Primitive-to-mesh transfer: A smaller |∆| indicates that an exported mesh faithfully preserves the native primitives’ rendering quality.Table C reports PSNR for primitive rendering, mesh rendering, and their difference ∆ on RE10K with 6 views.

D Opacity Mapping Analysis · E Triangle Adapter Details

The opacity mapping preserves boundary values, starts as identity, and progressively binarizes intermediate opacities through exponent and temperature-based sharpening. Triangle construction uses a canonical equilateral template with depth- and intrinsics-aware scaling, confidence-based coverage boosting, scheduled blur, and geometry-valid frame overrides with fallback quaternions.

  • D Opacity Mapping Analysis: The opacity mapping preserves 0 and 1, reduces to identity at e = 1, and binarizes intermediate values as e increases.In the limit e →∞, o(p; e) approaches 1 for p > 0.5.
  • D Opacity Mapping Analysis: Temperature sharpens opacity from τinit = 1.0 to τfinal = 5.0 over 16,000 steps, while an alpha floor of 0.02 prevents premature pruning.The exponent nonlinearity and temperature sharpening form a richer curriculum than either mechanism alone.
  • E Triangle Adapter Details: The triangle adapter supplements the formula-level construction with code-level reproduction details using a canonical equilateral template.The template vertices are (0, 0.577, 0), (−0.5, −0.289, 0), and (0.5, −0.289, 0).
  • E Triangle Adapter Details: The template is pre-scaled by 4, with sigmoid-bounded scale logits converted to world-space sizes using depth and an inverse-intrinsic pixel-footprint multiplier.Scale ranges are [0.5, 18.0] in Stage 1 on RE10K and [1.2, 15.0] in Stage 2.
  • E Triangle Adapter Details: Triangles with opacity below 0.20 receive proportional scale boosts, encouraging uncertain primitives to cover wider areas and obtain more photometric gradients.The boost depends on the gap between the 0.20 threshold and the triangle opacity.
  • E Triangle Adapter Details: The blur value is constrained positive as σ = sigmoid(ˆσ) · β(t) + ϵ, with β(t) decaying linearly from 1.0 to 0.5 over 16,000 steps.This schedule progressively reduces the blur scale during training.
  • E Triangle Adapter Details: Geometry-derived quaternions override network predictions where tangent-frame rotations are valid, while boundary or degenerate pixels retain the network quaternion.Invalid geometry conditions therefore use the learned orientation as a fallback.

F Additional Results on ScanNet

On ScanNet, TriSplat is evaluated zero-shot using models trained on RE10K without fine-tuning, with mesh rendering as the primary metric. Despite the domain gap between real-estate walkthrough videos and indoor scans, it maintains competitive mesh-rendering performance.

  • Evaluation Protocol: Mesh rendering is the primary novel-view synthesis metric, while primitive rendering is included for reference.Primitive rendering uses each method’s native rasterization, whereas mesh rendering rasterizes the exported mesh.
  • Results: Despite the significant domain gap between real-estate walkthrough videos and indoor scans, TriSplat maintains competitive performance under mesh rendering.The reported comparison also considers the primitive-to-mesh degradation observed on the training domain.
  • Evaluation Setting: TriSplat is evaluated on ScanNet in a zero-shot setting using models trained on RE10K without fine-tuning.The evaluation uses 6 input views.

G Additional Ablation Studies … G.3 Opacity Temperature

The additional ablations evaluate hyperparameters and architectural variants using surface geometry and mesh-rendering metrics. They show that triangle scale, blur scheduling, and opacity temperature each require balanced settings for simulation-ready reconstruction.

  • G Additional Ablation Studies: All additional ablation tables report CD, F1, PSNR, and LPIPS, evaluating design choices on the simulation-ready metrics used in the main experiments.The metrics jointly measure surface geometry and mesh-rendering quality.
  • G.1 Triangle Scale Range: Triangle scale ranges that are too narrow limit coverage of large surface regions and reduce recall.The ablation varies the triangle scale range [smin, smax] on RE10K with 6 views.
  • G.1 Triangle Scale Range: Excessively wide triangle scale ranges permit large triangles that introduce rendering artifacts.The study concerns the triangle scale range [smin, smax].
  • G.2 Blur Schedule: Fixed low blur causes poor early-training gradient coverage, whereas fixed high blur stabilizes training but produces soft surfaces.The blur-scheduling ablation is conducted on RE10K with 6 views.
  • G.2 Blur Schedule: The default blur schedule decaying from 1.0 to 0.5 over 16K steps achieves the best balance.This balance addresses early-training stability and surface sharpness.
  • G.3 Opacity Temperature: Without opacity temperature scaling, τ = 1.0 throughout leaves opacities soft, producing semi-transparent surfaces that degrade mesh quality.The opacity-temperature ablation is reported on RE10K with 6 views.
  • G.3 Opacity Temperature: A very high final temperature of τ = 25.0 produces near-binary opacities that cause gradient instability.The result comes from the opacity temperature schedule ablation.

H Additional Simulation Experiments … I More Visual Comparisons

TriSplat’s directly exported triangle meshes support dynamic simulation and embodied interaction without manual cleanup, conversion, or mesh repair. Additional comparisons across RE10K, DL3DV, and ScanNet further examine mesh- and primitive-rendering quality.

  • H Additional Simulation Experiments: Four-frame dynamic sequences use directly exported triangle meshes in Unity and NVIDIA Isaac Sim without manual cleanup or format conversion.The demonstrations cover robotic grasping, ball dynamics, and multi-platform locomotion.
  • H.1 Rigid-Body Dynamics: In NVIDIA Isaac Sim, ball drops produce intricate bounce trajectories, while stacked rigid objects remain stable on reconstructed surfaces.These responses indicate faithful collision geometry and surface flatness.
  • H.2 Legged Locomotion: A simulated quadruped uses TriSplat’s mesh surface normals and collision geometry to plan footholds and successfully navigate scenes containing stairs and chairs.The experiment evaluates outdoor-scene traversal with reconstructed geometry.
  • H.2 Legged Locomotion: Unity imports the exported TriSplat mesh as static scene geometry, enabling character navigation and collision handling across four temporal frames without mesh repair.The sequence progresses from t = 1 to t = 4.
  • I More Visual Comparisons: Additional qualitative comparisons cover RE10K, DL3DV, and ScanNet scenes excluded from the main-paper visual pages.The figures include mesh-rendering, primitive-rendering, textured-mesh, depth/normal, and zero-shot ScanNet results.
  • I More Visual Comparisons: Unity object interaction and Isaac Sim humanoid and quadruped sequences show stable contact-rich behavior on the same exported TriSplat mesh without intermediate reconstruction or repair.The sequences span t = 1 to t = 4 and use standard collision, physics, and contact-solver components.
  • I More Visual Comparisons: RE10K comparisons group mesh-rendered baselines, TriSplat, and ground truth across six input views, with artifacts reflecting exported-mesh representation quality.The listed methods include MVSplat, DepthSplat, AnySplat, YoNoSplat, MeshSplat, and SurfelSplat.
  • I More Visual Comparisons: DL3DV examples isolate mesh-rendering quality after export, so sharper results indicate better preservation of the original image evidence.The comparison includes MVSplat, DepthSplat, AnySplat, YoNoSplat, and TriSplat against ground truth.

J Mesh Evaluation Protocol

The mesh evaluation follows MeshSplat and ScanNet protocols, using uniformly sampled, voxel-downsampled meshes and world-coordinate metrics. Additional comparisons emphasize that native primitive renderings can diverge from exported-mesh quality, while TriSplat preserves direct triangle geometry in textured mesh outputs.

  • Evaluation protocol: Meshes are uniformly evaluated after point-cloud sampling and voxel downsampling at resolution 0.02, following MeshSplat [22] and ScanNet [33].Predicted and ground-truth meshes are processed identically to ensure uniform density.
  • Evaluation protocol: Chamfer Distance sums the two one-sided nearest-neighbor distances, while Precision and Recall use δ = 0.05 and F1 is their harmonic mean.KD-trees from Open3D accelerate nearest-neighbor queries, and all metrics are computed in the world coordinate frame.
  • Rendering comparisons: Native Gaussian primitive renderings do not necessarily predict exported-mesh quality, motivating mesh rendering as the relevant protocol for simulation-ready reconstruction.The comparisons are presented on RE10K and DL3DV under each method’s native rasterizer before mesh export.
  • Textured mesh comparisons: On RE10K textured mesh comparisons, TSDF-fused Gaussian meshes show over-smoothed surfaces, missing thin structures, and fragmented regions, whereas TriSplat preserves direct triangle geometry and appearance.The figure visualizes exported textured meshes rather than target-view renders.
  • Zero-shot ScanNet evaluation: Zero-shot ScanNet mesh comparisons render every exported mesh through the same triangle pipeline without border cropping, preserving each method’s full fixed-resolution frame.This retains large empty regions or incomplete surfaces present in some exported baseline meshes.
Loading 2605.26115v1…