PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Views

Yongzhen Hu, Hao Yu, Jiawei Wang, Xianming Zeng, Jianyun Xu, Xingliang Liu, Ruizhen Hu, Xiaowei Zhou, Hujun Bao, Sida Peng

arXiv:2610.20341v1cs.CV

TL;DR

InfiniSplat v2 addresses the challenge of learning Gaussian attributes from imperfect predicted geometry for high-quality novel-view rendering. It freezes geometry, aligns Gaussians with ground-truth cameras, and uses large-angle target views; the same trained model achieves the lowest mean LPIPS and DISTS among evaluated baselines across six datasets in both pose settings.

  • Problem

    Feed-forward methods must learn Gaussian attributes from imperfect predicted geometry to produce high-quality novel-view renderings, despite outliers and cross-view inconsistencies in predicted point clouds.

    Source: §1 · p. 2

  • Method

    InfiniSplat v2 freezes the geometry branch, aligns predicted Gaussians using ground-truth cameras during training, and applies large-angle target-view supervision.

    Source: §1 · p. 2

  • Results

    Across six datasets, the same trained model achieves the lowest mean LPIPS and DISTS among evaluated baselines in both pose settings; with camera poses, LPIPS and DISTS decrease 25.8% and 32.9% versus the strongest evaluated baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With image-reconstruction supervision alone, the attribute branch assigns lower opacity to cross-view-inconsistent outliers, reducing visible artifacts in novel views.

    Source: §1 · p. 2

    Fixed centers cannot correct severe geometry errors; pixel-aligned redundancy can survive voxel merging, and limited input coverage can leave holes.

    Source: C LIMITATIONS AND FUTURE WORK · pp. 20–21

Abstract

from arXiv · show

Feed-forward Gaussian reconstruction builds 3D scenes directly from input images for novel view rendering, but learning Gaussian attributes from predicted geometry for high-quality rendering under large viewpoint changes remains challenging. We analyze limitations in Gaussian attribute learning and introduce InfiniSplat v2, which improves rendering quality through architectural and training changes. Our method decouples geometry and attribute prediction with a frozen geometry branch, aligns predicted Gaussians using ground-truth camera poses during training, and adopts large-angle target view supervision. Across six datasets, the feed-forward outputs of the same trained model achieve state-of-the-art mean LPIPS and DISTS among the evaluated methods in both pose-conditioned and pose-free settings. In the pose-conditioned setting, mean LPIPS and DISTS decrease by 25.8% and 32.9%, respectively, compared with the strongest evaluated baselines. We further find that opacity learned with only image reconstruction supervision suppresses geometrically inconsistent points. Project page: https://zju3dv.github.io/InfiniSplat2/

1 INTRODUCTION

Feed-forward Gaussian reconstruction must turn imperfect predicted geometry into scenes that render well under novel views. InfiniSplat v2 changes geometry and attribute learning, and its same trained model achieves the lowest mean LPIPS and DISTS among evaluated baselines in both pose settings across six datasets.

1 INTRODUCTION

  • Existing feed-forward methods still show blur, ghosting, and missing detail, while aggregated predicted point clouds can contain spatially isolated or cross-view-inconsistent clusters.These imperfections motivate learning attributes that support high-quality novel-view rendering from predicted geometry.Source: §1 · p. 2
  • Across six datasets, the same trained model achieves the lowest mean LPIPS and DISTS among evaluated baselines in both pose settings; with camera pose inputs, LPIPS and DISTS decrease 25.8% and 32.9% versus the strongest evaluated baselines.The evaluation covers pose-conditioned and pose-free settings.Source: §1 · p. 2
  • InfiniSplat v2 freezes geometry prediction while learning Gaussian attributes, aligns predicted Gaussians with ground-truth cameras during training, and uses large-angle target-view supervision.The method is designed for arbitrary input views and high-quality rendering at extreme viewpoints.Source: §1 · p. 2
  • With only image-reconstruction supervision, learned opacity selectively suppresses geometrically inconsistent points, reducing their rendering contribution across views.The paper reports that this reduces visible artifacts in novel views.Source: §1 · p. 2

2 RELATED WORK

Related work develops learned geometry as a basis for feed-forward scene reconstruction and explores ways to predict Gaussian scenes and refine them through rendering feedback.

2.1 FEED-FORWARD 3D GEOMETRY ESTIMATION

  • Geometry foundation models predict scene structure, providing a geometric basis for feed-forward Gaussian reconstruction, which also requires learning attributes such as opacity, scale, and appearance.The cited work includes models for varying input counts, poses, longer sequences, and larger scenes.Source: §2.1 · p. 3

2.2 FEED-FORWARD GAUSSIAN SPLATTING

  • Feed-forward Gaussian methods vary in input views and pose requirements, sequence length, and primitive organization, including single-image, multi-view, pose-free, and token- or voxel-based approaches.Examples include SHARP and InfiniSplat for single images, NoPoSplat and FLARE for pose-free reconstruction, and C3G, TokenGS, and VolSplat for primitive organization.Source: §2.2 · p. 3
  • Geometry estimation for Gaussian prediction includes cross-view correspondence, multi-view cost volumes, pretrained monocular depth features, and learned geometric priors; InfiniSplat v2 focuses on attributes from imperfect predicted point clouds.QuerySplat separately decodes geometry and appearance from frozen geometric features.Source: §2.2 · p. 3
  • Learned refiners use rendering gradients or feature-space rendering errors; InfiniSplat v2 uses input-view rendering gradients to update attributes while keeping centers fixed.G3R and Flux4D use explicit rendering gradients, whereas ReSplat uses feature-space errors without explicit gradient feedback.Source: §2.2 · p. 4

3 METHOD

InfiniSplat v2 builds Gaussian scenes from predicted multi-view geometry while separating geometry prediction from attribute learning. Its training aligns points and supervision with scale-aligned ground-truth cameras, samples large-angle target views, and refines attributes using rendering feedback.

3.1 BACKGROUND: DA3-BASED GAUSSIAN RECONSTRUCTION

  • Scenes are represented by Gaussian centers and attributes; DA3-GS uses a frozen backbone and combines predicted geometry and attributes into a renderable scene.The attributes include opacity, scale, rotation, and color; scale and rotation determine Gaussian shape.Source: §3.1 · p. 4

3.2 FEED-FORWARD GAUSSIAN RECONSTRUCTION

  • A frozen geometry branch predicts depths and cameras, while a separate attribute branch predicts features and learns Gaussian attributes without positional offsets.The geometry branch remains frozen, and Gaussian centers stay at back-projected point positions while scales and rotations remain learnable.Source: §3.2 · p. 4
  • Training uses scale-aligned ground-truth cameras for point back-projection and target rendering, then favors target views with larger angular differences from input views.Target-view losses are restricted to regions with valid, depth-consistent support from at least one input view; target data serves sampling and supervision, not scene construction or scale estimation.Source: §3.2 · p. 5

3.3 LEARNED RENDERING-FEEDBACK REFINEMENT

  • A recurrent KNN Transformer combines current-scene features, fixed projected DPT context, and input-view rendering-gradient feedback to update attributes while centers and counts stay fixed.Rendering gradients are inputs to learned updates rather than direct gradient-descent steps, and the spatial KNN graph can be reused across steps.Source: §3.3 · pp. 5–6

3.4 TRAINING AND INFERENCE

  • Training first updates the attribute branch and Gaussian decoder, then freezes the reconstructor while training feature projection and the refiner; the same model supports pose-conditioned and pose-free inference.At inference, refinement weights remain fixed and feedback uses the input views.Source: §3.4 · p. 6

4 EXPERIMENTS

InfiniSplat v2 is evaluated against feed-forward reconstruction baselines on six datasets in pose-conditioned and pose-free settings, using shared view splits and a common scoring protocol. It achieves the best mean LPIPS and DISTS in both settings, while ablations assess reconstruction choices on ETH3D and DTU.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses six datasets, N = 4, 8, 16 input views, and four held-out targets, with the same trained model evaluated both with and without camera poses.Per-dataset scores average equally over input-view counts, and the overall mean weights the six datasets equally.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 COMPARISON WITH STATE-OF-THE-ART METHODS

  • 25.8% lower LPIPS than ReSplat and 32.9% lower DISTS than WorldMirror 2.0 with pose conditioning; without poses, gains over WorldMirror 2.0 are 19.5% and 30.7%, respectively.Across six datasets, the feed-forward model achieves the best mean LPIPS and DISTS among evaluated baselines in both settings.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 ABLATIONS AND ANALYSIS

  • Geometry fine-tuning and learned positional offsets worsen both metrics on ETH3D and DTU; freezing geometry reduces LPIPS by 16.6% and 23.2%, respectively.With frozen geometry, learned offsets produce local blur and depth distortions in the NRGBD kitchen example.Source: §4.3 · p. 8, Figure 5 · p. 9
  • Fine-tuning the last K = 10 attribute-backbone blocks reduces LPIPS by 1.1% on ETH3D and 20.8% on DTU versus K = 0, with lower DISTS on both.GT-camera backprojection reduces DTU LPIPS by 36.1% versus predicted cameras, and large-angle supervision improves both metrics on ETH3D and DTU versus local interpolation.Source: §4.3 · p. 8
  • Qualitative ablations show more tabletop cracks without attribute fine-tuning or large-angle supervision, and blurred door frames without geometry freezing or GT-camera backprojection.The full model retains more distinct door frames.Source: §4.3 · p. 9, Figure 6 · p. 9
  • At opacity threshold 0.01, pruning removes 51.8% of geometrically inconsistent points but only 7.5% of all points, retaining 93.3% of surface-consistent points.The inconsistent subset is labeled independently using held-out Hypersim GT depth; points violate observed free space in at least two views and agree with no observed surface depth.Source: §4.3 · p. 9, Figure 7 · p. 9

5 CONCLUSION

The conclusion highlights InfiniSplat v2’s improved mean LPIPS and DISTS across six datasets in both pose settings. It also points to fixed centers, attribute adaptation, and learned opacity as supported findings.

5 CONCLUSION

  • InfiniSplat v2 improves mean LPIPS and DISTS across six datasets in both pose settings.Source: §5 · p. 9
  • ETH3D and DTU ablations support fixed centers and attribute adaptation under reconstruction supervision.Source: §5 · p. 9
  • Learned opacity preferentially suppresses geometrically inconsistent points.Source: §5 · p. 9

A.1 DATA AND VIEW SAMPLING

The training and evaluation use multiple input-view counts, while target-view sampling favors larger angular separations. Target supervision is restricted by depth- and visibility-based masks.

A.1 DATA AND VIEW SAMPLING

  • Training samples Hypersim, Raw3D, and ScanNet++ with probabilities of 60%, 10%, and 30%, respectively, using four target views and 4–12 input views.Image resolution varies with input-view count; evaluation uses 4, 8, or 16 input views.Source: §A.1 · p. 14
  • Target views are drawn uniformly without replacement from a large- or small-angle pool, selected with probabilities of 0.7 and 0.3, respectively.Views are ranked by minimum optical-axis angle to input views, and a selected pool must contain at least four candidates.Source: §A.1 · p. 14
  • Target pixels require valid finite GT depth greater than 10^-6 and must pass the supplied validity mask and input-view visibility checks.The mask uses absolute and relative depth tolerances of 0.03 and 0.01; the local-view control retains the full model’s masks and target supervision.Source: §A.1 · p. 14

A.2 ARCHITECTURE, RENDERING, AND COORDINATE CONVENTIONS

The method constructs Gaussian centers from predicted depth and camera information, then predicts and refines attributes separately. Training combines image reconstruction supervision with a learned, multi-step attribute refiner.

A.2 ARCHITECTURE, RENDERING, AND COORDINATE CONVENTIONS

  • Gaussian centers are built from normalized predicted depth, GT intrinsics, and scale-aligned GT poses, while Gaussian attributes are predicted by a separate branch.The branches start from the same initialization with independent parameters, and Gaussians below opacity 0.01 are pruned after voxel merging.Source: §A.2 · p. 14

A.3 OPTIMIZATION SETTINGS

  • Stage I trains the last ten attribute-backbone Transformer blocks and Gaussian head for 25,000 steps; Stage II freezes the reconstructor and trains the feature projection and refiner for 5,000 steps.Both stages use GT pose conditions and AdamW; training takes about 48 hours for Stage I and 13 hours for Stage II.Source: §A.3 · p. 14

A.4 STAGE-I RECONSTRUCTION LOSS

  • The initial scene is rendered into input and target views, and valid pixels receive equal-weighted RGB absolute-error and LPIPS losses.LPIPS uses VGG features, with invalid regions zeroed before input normalization.Source: §A.4 · p. 15

A.5 STAGE-II LEARNED REFINEMENT

  • A shared KNN Transformer refiner updates Gaussian attributes from input-view rendering feedback while keeping centers, counts, and correspondences fixed.Training samples one to three updates and inference uses three; attribute paths are detached while hidden-state residuals remain differentiable, and the objective penalizes opacity-logit changes.Source: §A.5 · pp. 15–16

A.6 CAMERA PROTOCOL AND EVALUATION METRICS

Pose-conditioned and pose-free evaluations are reported separately, using a shared trained model and a common multi-level aggregation scheme. Novel-view rendering is assessed with four metrics.

A.6 CAMERA PROTOCOL AND EVALUATION METRICS

  • Pose-conditioned and pose-free results are ranked separately; in pose-free evaluation, reference cameras establish the frame and translation scale but do not enter reconstruction.Target poses are rendering queries, and target RGB is never used for reconstruction or refinement feedback.Source: §A.6 · p. 16
  • Novel-view rendering is evaluated with PSNR, SSIM, LPIPS, and DISTS, with equal averaging across target views, examples, scenes, input configurations, and datasets.The aggregation is applied successively at each level.Source: §A.6 · p. 16

B.1 FULL QUANTITATIVE RESULTS

The appendix reports six-dataset mean comparisons, feed-forward ablations, refinement comparisons, geometric-consistency analysis, and latency measurement. The reported refinement advantage over Adam is quantified on ETH3D.

B.1 FULL QUANTITATIVE RESULTS

  • Refinement ranks first on all four six-dataset mean metrics with poses; without poses, it leads PSNR, SSIM, and LPIPS, while feed-forward output leads DISTS.Tables 3 and 4 summarize the pose-free and pose-conditioned results, respectively.Source: §B.1 · p. 16, Table 3 · p. 17, Table 4 · p. 17

B.2 FEED-FORWARD RECONSTRUCTOR ABLATIONS

  • The feed-forward ablations vary attribute-backbone training, geometry-backbone training, center offsets, camera inputs, and target-view sampling.The offset variant shows stronger surface artifacts and depth deviations than the fixed-center model in the NRGBD comparison.Source: §B.2 · p. 17, Figure 9 · p. 17

B.3 LEARNED REFINEMENT VERSUS DIRECT GRADIENT DESCENT

  • On ETH3D, learned refinement improves PSNR by 0.64 dB versus 0.02 dB for Adam, and performs better on the other three metrics.Both methods start from the same scene, perform three updates, and keep centers fixed.Source: §B.3 · pp. 17, 19, Table 8 · p. 19

B.4 OPACITY AND GEOMETRIC CONSISTENCY

  • The opacity analysis labels 36,125 points as geometrically inconsistent using GT depth and multi-view free-space and surface checks.Opacity distributions and pruning statistics at threshold 0.01 are shown in Figure 7.Source: §B.4 · p. 19

B.5 INPUT RESOLUTION AND FEED-FORWARD NETWORK LATENCY

  • Feed-forward network latency is measured at four resolutions on one H20 using one warmup and five synchronized CUDA-event measurements.The table reports mean and min–max over five runs.Source: §B.5 · p. 19, Table 9 · p. 19

B.6 ADDITIONAL QUALITATIVE RESULTS

The additional results illustrate pose-free rendering quality, reconstruction across diverse scene types, and large-scale scene construction. They also show reconstructions with varying numbers of input views.

B.6 ADDITIONAL QUALITATIVE RESULTS

  • On ETH3D pose-free examples, selected crops show fewer repetitive stripe artifacts and better-preserved relief structures; Figure 11 highlights grass textures and relief details.Source: §B.6 · p. 19, Figure 11 · p. 20
  • Reconstructions span seven datasets covering streets, buildings, interiors, and objects.Source: §B.6 · p. 19
  • A KITTI street sequence of over 1,000 frames is reconstructed into a large-scale Gaussian scene using sliding windows.Source: §B.6 · p. 19
  • The examples include feed-forward reconstructions and renderings with 12–64 input views.Source: Figure 12 · p. 20

C LIMITATIONS AND FUTURE WORK

Opacity learning and pruning can suppress unreliable Gaussians, but fixed centers leave severe predicted-geometry errors uncorrected. Pixel-aligned prediction can also leave redundancy, and limited input coverage can leave holes in unobserved regions.

C LIMITATIONS AND FUTURE WORK

Related papers

Loading 2610.20341v1…