PaperlayerView exact revision on arXiv

Source-linked AI summary

FineSplat: Fine Patches, Local Groups, Detailed Splats

Hyebin Kim, Hwasik Jeong, Joonhyuk Park, Gyeongjin Kang, Jungwoo Kim, Younghyun Kim, Eunbyung Park

arXiv:2610.25011v1cs.CV

TL;DR

Feed-forward 3DGS still struggles to recover fine-grained geometry and appearance, while global attention over fine tokens is prohibitively expensive. FineSplat combines fine patches, view-wise and 3D spatial grouping, rendering-error feedback, and compact Gaussian generation. It reports state-of-the-art quality across posed and pose-free settings, with strong PSNR gains over 2Xplat using fewer Gaussians.

  • Problem

    Feed-forward 3DGS must resolve multi-view ambiguity and recover fine details in one pass, while coarse patches can discard cues needed for detailed geometry and appearance.

    Source: §1 · p. 2

  • Method

    FineSplat uses fine-grained patches with complementary view-wise and 3D spatial grouping, rendering-error feedback, and compact Gaussian generation.

    Source: §1 · pp. 2–3

  • Results

    Up to +2.88 dB posed and +2.75 dB pose-free PSNR improvement over 2Xplat, FineSplat reports consistent state-of-the-art performance across input-view configurations using about 4 times fewer Gaussians.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    FineSplat achieves strong novel-view synthesis quality with a small primitive budget.

    Source: §6 · p. 11

    The authors do not claim fine patches are universally preferable, noting sufficiently large foundation models trained at scale may compensate for coarse tokenization.

    Source: §6 · p. 11

Abstract

from arXiv · show

Feed-forward 3D Gaussian Splatting (3DGS) promises instant scene reconstruction with high-quality novel-view synthesis. Yet, fine-grained geometry and appearance remain difficult to recover, motivating us to revisit how feed-forward models represent, aggregate, and refine detailed visual information. We introduce FineSplat, a feed-forward 3DGS framework built around fine-grained image patches. Since global multi-view attention becomes prohibitively expensive at such high token resolutions, we develop a dual-grouping strategy that combines two complementary forms of locality: view-wise grouping enables efficient interaction among neighboring views, while 3D spatial grouping reconnects geometrically related tokens that may never have interacted under view-wise grouping by lifting them into the 3D scene space. We further leverage reconstruction error as direct feedback, providing an explicit rendering-based signal to recover fine-grained structures and appearance, thereby overcoming the limitations of one-pass feed-forward prediction. Finally, a compact Gaussian generation scheme preserves the benefits of fine-grained features without unnecessarily increasing the number of primitives. Together, these designs enable FineSplat to recover richer geometry and appearance while remaining efficient and compact, achieving state-of-the-art results by a substantial margin, under the same or smaller Gaussian budgets than existing methods.

1 INTRODUCTION

FineSplat targets the detail lost by coarse tokens and the computational cost and blind spots of global or view-restricted attention. It combines fine-grained tokens, dual grouping, rendering-error feedback, and compact Gaussian generation to improve reconstruction while keeping the representation compact.

1 INTRODUCTION

  • Coarse patches can discard high-frequency cues needed for thin structures, sharp boundaries, and textures, and make precise cross-view localization harder.Broad patches retain spatial context that can help geometric reasoning and disambiguate correspondences.Source: §1 · p. 2
  • Global multi-view attention becomes prohibitively expensive with finer tokens, while fixed view groups can separate same-surface correspondences across group boundaries.View-wise grouping works especially well for sequential views with overlapping scene regions, but tokens from different groups may never directly interact.Source: §1 · p. 2
  • FineSplat combines attention within neighboring-view groups with 3D spatial regrouping, allowing geometrically nearby tokens from different view groups to interact.Tokens are lifted into 3D using predicted scene locations before regrouping by spatial proximity.Source: §1 · p. 2
  • Rendering input views provides reconstruction-error feedback that gives the feed-forward model a second chance to recover details missed in its first stage.This uses a signal repeatedly exploited by per-scene optimization without resorting to costly iterative optimization.Source: §1 · pp. 2–3
  • FineSplat achieves state-of-the-art rendering quality by a substantial margin in posed and pose-free settings while using the same or fewer Gaussians than existing methods.Its compact Gaussian scheme predicts only a small number of Gaussians per patch token rather than densely generating pixel-aligned Gaussians.Source: §1 · p. 3

2 RELATED WORKS

Related work spans feed-forward Gaussian representations, pose-free reconstruction, image-to-3D locality, and iterative refinement. FineSplat combines fine-grained patch features with cross-view 3D interaction and rendering-error feedback.

2 RELATED WORKS

  • Pixel-aligned methods preserve pixel-ray correspondence but can create redundant primitives, while compact token- or voxel-based methods lose that correspondence and can hurt fine detail.Patch-aligned methods occupy an intermediate resolution; FineSplat predicts a small number of Gaussians along each sub-patch's rays and prunes by context-view rendering contribution.Source: §2 · p. 3
  • Pose-free methods often predict Gaussians atop coarse-token geometry backbones; FineSplat instead uses a geometry foundation model for camera poses and predicts Gaussians with fine-grained tokens.The passage notes that coarse tokenization may discard appearance cues and that end-to-end backbone adaptation is computationally expensive.Source: §2 · p. 3
  • Prior methods bring image-space features into 3D neighborhoods, and FineSplat similarly lifts tokens from all views so tokens separated by view grouping can interact.The cited examples include ODIN, DITR, MoGe-3, and ATSplat.Source: §2 · pp. 3–4
  • Learned iterative refinement repeatedly updates initial estimates across vision tasks, while Gaussian-splatting work has used network updates, rendering gradients or errors, and scene-specific token tuning.FineSplat uses rendering errors as feedback in its second stage.Source: §2 · p. 4

3 METHOD

FineSplat encodes fine-grained patches with affordable view-wise attention, then refines tokens through 3D spatial grouping and rendering-error feedback. Its patch-aligned decoder predicts sub-pixel Gaussian rays while keeping primitives compact.

3 METHOD

  • The method encodes fine-grained 4 × 4 patches, predicts initial Gaussians, renders them into context views, and updates tokens using error maps and 3D spatial grouping.The grouping links tokens in different view groups after lifting them into 3D.Source: §3 · p. 4

3.1 STAGE 1: VIEW-WISE TOKEN GROUPING

  • Global attention is too costly for fine tokens, so Stage 1 applies fine-grained attention within small view groups while coarse global tokens exchange information across views.Fixed view groups still leave blind spots because tokens in different groups cannot directly interact.Source: §3.1 · pp. 4–5

3.2 STAGE 2: 3D SPATIAL TOKEN GROUPING

  • Stage 2 decodes initial Gaussians, anchors tokens at their predicted Gaussian centers in 3D, and regroups them by spatial proximity for attention across view groups.The process repeats Gaussian prediction, grouping, error-guided interaction, and token updating.Source: §3.2 · p. 5
  • Rendered-image and ResNet-18 feature differences form error maps, and each token gathers error features from its own view and other views that observe it.The error features are combined with finest-scale first-stage features; 3D serialized attention exchanges information among nearby tokens at lower cost than explicit k-nearest-neighbor search.Source: §3.2 · pp. 5–6

3.3 PATCH-ALIGNED SUB-PIXEL GAUSSIAN RECONSTRUCTION

  • The decoder predicts four Gaussians per 4 × 4 token, assigning one to each non-overlapping 2 × 2 sub-patch and learning bounded image-space offsets for sub-pixel localization.Each displaced coordinate is unprojected using predicted camera intrinsics.Source: §3.3 · p. 6

4 EXPERIMENTS

FineSplat is evaluated against posed and pose-free baselines across multiple datasets and view counts. It reports strong benchmark and cross-dataset results, including larger gains as context views increase on Mip-NeRF 360.

4.1 IMPLEMENTATION DETAILS

  • The model is trained only on DL3DV and evaluated on DL3DV, RE10K, and Mip-NeRF 360, with held-out target views and native-resolution encoding and rendering.FineSplat uses no depth supervision, unlike several baselines, and scene scale is normalized from context camera poses.Source: §4.1 · pp. 6–7

4.2 NOVEL VIEW SYNTHESIS

  • Up to +2.88 dB posed and +2.75 dB pose-free PSNR improvement over 2Xplat, FineSplat consistently outperforms prior state-of-the-art methods across input-view configurations using about 4 times fewer Gaussians.The comparison uses the 2Xplat variant without view-dependent opacity.Source: §4.2 · p. 7
  • On RE10K, FineSplat is best across PSNR, SSIM, and LPIPS in the posed setting and best among zero-shot pose-free methods, despite training only on DL3DV.Comparisons are less controlled because some methods were trained on RE10K.Source: §4.2 · p. 7
  • On Mip-NeRF 360, FineSplat has the highest PSNR at every tested view count, with gains of 2.49 to 3.33 dB posed and 0.88 to 1.49 dB pose-free.From 32 to 128 context views, its PSNR rises by 1.32 dB posed and 1.00 dB pose-free; baselines gain at most 0.5 dB or lose ground.Source: §4.2 · p. 7

5 ANALYSIS

In posed DL3DV analysis, Stage 2 uses 3D-local interactions to refine Stage 1 predictions, with gains that grow with more context views. FineSplat also offers favorable reconstruction quality relative to time across the tested view counts.

Cross-group interactions in 3D space

  • Tokens exchange self and aggregated reconstruction-error signals with nearby 3D tokens from different frames and view groups.The attention window is illustrated in Figure 6; the analysis describes four blocks using different space-filling curves.Source: §5 · p. 8, Figure 6 · p. 10

More Stage 2 steps and context views

  • Stage 2’s gain increases from 1.22 dB with 6 views to 1.46 dB with 24 views as more views provide richer regional observations.The analysis attributes this trend to 3D interactions combining observations and error information across view groups.Source: §5 · pp. 9–10
  • A single Stage 2 step improves PSNR by 0.85 dB with 6 views and 1.04 dB with 24 views.Quality improves gradually through the four trained Stage 2 steps, with no further gain beyond training.Source: Table 4 · p. 10, §5 · p. 10

Efficiency

  • FineSplat provides the largest quality gain per unit time across view counts, leading the stronger baseline by 2.6 dB with 6 views and 4.6 dB with 24 views.It remains fastest with 6 and 12 views, and is faster than ZipSplat with 24 views; Figure 7 plots PSNR against reconstruction time.Source: §5 · p. 10, Figure 7 · p. 11

6 CONCLUSION AND DISCUSSION

FineSplat combines fine-grained patches, view-wise and 3D spatial grouping, reconstruction feedback, and compact Gaussian generation. The discussion cautions that fine patches are not universally preferable, while the appendix describes the two-stage architecture.

6 CONCLUSION AND DISCUSSION

  • FineSplat pairs high-resolution patches and dual grouping with reconstruction feedback and compact Gaussian generation to achieve strong novel-view synthesis with a small primitive budget.The dual grouping combines view-wise and 3D spatial locality to avoid the cost of global multi-view attention.Source: §6 · p. 11
  • The authors do not claim fine patches are universally preferable, since sufficiently large foundation models trained at scale may compensate for coarse tokenization.Preliminary combinations of foundation-model features with fine-grained tokens showed no noticeable improvement; integration that could improve generalization remains future work.Source: §6 · p. 11

A.1 ARCHITECTURE

  • Stage 1 uses a 14-layer hierarchical transformer, moving from 4 × 4 to 8 × 8 and 16 × 16 patches as feature and attention scope increase.At the 8 × 8 level, view-wise groups contain two adjacent frames.Source: §A.1 · p. 16
  • Stage 2 fuses source-view and visibility-averaged error features, then applies four serialized 3D-local attention blocks across four shared-weight steps.The appendix reports 13.2M trainable parameters for Stage 2.Source: §A.1 · p. 16

A.2 TRAINING

FineSplat is trained in three phases, with Stage 2 learning corrections on top of frozen Stage 1 before end-to-end fine-tuning. Its scene normalization derives scale from context camera poses in a frame centered on those cameras.

A.2 TRAINING

  • Training uses three phases, with 120K pixel-aligned iterations, 80K patch-aligned fine-tuning iterations, and 20K Stage-2 iterations before final fine-tuning.The end-to-end phase runs for 10K iterations; all stages use AdamW with cosine learning-rate decay.Source: §A.2 · pp. 16–17
  • Stage 2 trains on frozen Stage 1 using ground-truth cameras, then the full model is fine-tuned end-to-end in a pose-free setting.The final phase predicts context and target cameras without camera loss and also supervises the Stage-1 Gaussians.Source: §A.2 · pp. 16–17

A.3 SCENE SCALE NORMALIZATION

  • Scene scale is derived from context camera poses in a frame centered on the context cameras, rather than on the first view.The context-derived transform is applied unchanged to target views in both posed and pose-free settings.Source: §A.3 · p. 17
  • The context-camera frame uses mean viewing and image-down directions, and rescales camera centers so they lie within [−1, 1]3.This differs from YoNoSplat’s first-camera origin and maximum pairwise-distance scale convention.Source: §A.3 · p. 17

A.4 OCCLUSION TEST FOR ERROR FEATURE AGGREGATION

Error-feature aggregation selects only views that observe a token, using frustum and depth tests to screen out invalid or occluded projections. The appendices also specify pose alignment for pose-free evaluation and contribution-based Gaussian pruning.

A.4 OCCLUSION TEST FOR ERROR FEATURE AGGREGATION

  • Views contribute error features only when the token passes frustum and depth tests, preventing errors from occluding surfaces from being assigned to it.The depth tolerance allows a token to be at most 15% behind the rendered surface; uncovered pixels have no occluder and remain eligible.Source: §A.4 · pp. 17–18, Figure 8 · p. 18

B.1 EVALUATION-TIME POSE ALIGNMENT

  • Pose-free evaluation freezes the reconstructed Gaussians and optimizes each target camera pose for 200 iterations using L2 loss.Ground-truth poses are used directly when available, without additional alignment.Source: §B.1 · p. 18

B.2 CONTRIBUTION-BASED PRUNING

  • Contribution-based pruning ranks Gaussians by their rendered blending weights across context views and keeps the top K = k · N, with k = 10,368 per view.Scores are computed once per scene without additional optimization, using context images and evaluation cameras.Source: §B.2 · pp. 18–19

B.3 EVALUATION RESOLUTION AND RESIZE FILTER

The resize protocol standardizes rendered and target images at 224 × 224 while allowing a separate Bilinear+AA evaluation setting. The view-group ablation finds group size 2 best across tested view counts and stages.

B.3 EVALUATION RESOLUTION AND RESIZE FILTER

  • The main evaluation resizes final rendered and ground-truth target images to 224 × 224, while reconstruction and rendering use each method’s native training resolution.FineSplat operates natively at 224 × 224, the smallest input resolution among compared methods.Source: §B.3 · p. 19
  • A separate FineSplat evaluation uses 252 × 252 Bilinear+AA inputs and targets, with baselines taken from ZipSplat’s matching evaluation protocol.FineSplat and other methods were otherwise trained with Lanczos, while ZipSplat alone also used Bilinear+AA training images.Source: §B.3 · p. 19

C.1 VIEW GROUPING AND THE CHOICE OF GROUP SIZE

  • Group size 2 is best at every tested view count and both stages; a single 24-view group is 2.53 dB below the default at Stage 1.Removing cross-view interaction with group size 1 drops Stage-1 PSNR by 4.18 dB at 6 views and 1.93 dB at 24 views; Stage 2 narrows these gaps.Source: §C.1 · pp. 19–20, Table 6 · p. 20

C.2 ERROR FEATURE COMPOSITION

The two error features provide complementary information: self-error has the larger measured effect, while aggregated error’s benefit grows with view count. Observations outside a token’s view group carry information unavailable within that group.

C.2 ERROR FEATURE COMPOSITION

  • Removing either error feature degrades quality, with removing eself costing 0.44 dB at 6 views and 0.62 dB at 24 views in PSNR.Removing eagg causes smaller drops that grow from 0.09 dB to 0.17 dB as view count increases.Source: §C.2 · p. 21

C.3 OBSERVATIONS BEYOND THE VIEW GROUP

  • At 6 to 24 views, 55.5% to 84.0% of visible views lie outside the token’s group, and removing these views causes a larger drop than random removal.The difference grows from 0.04 dB to 0.17 dB, indicating that other view groups provide observations unavailable within the group.Source: §C.3 · p. 21

C.4 ADDITIONAL CROSS-DATASET GENERALIZATION

On ScanNet++, FineSplat generalizes from DL3DV training to unseen scenes and view counts, leading posed-setting PSNR comparisons and improving its standing as input views increase. In the pose-free setting, it leads at 128 views despite using photometric supervision alone.

C.4 ADDITIONAL CROSS-DATASET GENERALIZATION

  • +1.19 dB at 32 views to +2.07 dB at 128: FineSplat leads DL3DV-trained methods in posed PSNR and surpasses ScanNet++-trained methods at 64 and 128 views with fewer Gaussians.The results are reported for 32, 64, and 128 input views.Source: §C.4 · pp. 21–22, Table 9 · p. 22
  • Figure 10 provides qualitative ScanNet++ comparisons across varying numbers of input views.Source: Figure 10 · p. 22
  • In pose-free evaluation, FineSplat trails some methods at 32 and 64 views but leads at 128, exceeding methods trained directly on ScanNet++.The comparison notes that FineSplat uses photometric supervision alone, while the early leaders rely on depth supervision.Source: §C.4 · p. 22

D ADDITIONAL QUALITATIVE RESULTS

The additional qualitative results compare reconstructions across datasets and input-view counts, including a zero-shot RE10K evaluation. The evaluation uses 224×224 resolution with PIL Lanczos resizing.

D ADDITIONAL QUALITATIVE RESULTS

  • The qualitative evaluation uses 224×224 resolution and the PIL Lanczos resizing filter.Source: D ADDITIONAL QUALITATIVE RESULTS · p. 23
  • Figure 11 presents DL3DV qualitative comparisons using 6, 12, and 24 context views.Source: Figure 11 · p. 23
  • Figure 12 compares RE10K renderings from a FineSplat model evaluated zero-shot with ZipSplat trained on RE10K, using six input views.Source: Figure 12 · p. 24
  • Figure 13 shows DL3DV-to-ScanNet++ and Mip-NeRF 360 generalization at 32, 64, and 128 context views.The first six columns show ScanNet++; the last three show Mip-NeRF 360.Source: Figure 13 · p. 25

Related papers

Loading 2610.25011v1…