PaperlayerView exact revision on arXiv

Source-linked AI summary

Anchor3DGS: Feed-Forward 3D Gaussian Splatting with Compact Anchor-Based Representation

Sheng Ye, Jiangke Lin, Fudong Wang, Yi Yuan, Zhen-Hui Dong, Yong-Jin Liu

arXiv:2610.21634v1cs.CV

TL;DR

Feed-forward 3DGS methods commonly tie Gaussian count to input pixels, producing redundancy and leaving coverage gaps. Anchor3DGS replaces this with content-guided 3D anchors, and reports state-of-the-art sparse-view quality and comparable dense-view quality with an order of magnitude fewer Gaussians.

  • Problem

    Pixel-aligned feed-forward 3DGS ties Gaussian count to input resolution and view count, with redundant primitives in simple regions and insufficient coverage in poorly observed areas.

    Source: §1 · pp. 1–2

  • Method

    Anchor3DGS uses a fixed-budget 3D anchor representation, sampled with broad coverage and greater allocation to visually complex regions, with occlusion-aware multi-view aggregation and local Gaussian decoding.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Anchor3DGS achieves state-of-the-art rendering quality in sparse-view settings and comparable dense-view quality with an order of magnitude fewer Gaussians.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    The results support a compact anchor-based representation that decouples Gaussian allocation from input resolution and view count while retaining rendering quality.

    Source: §1 · p. 2, §6 · p. 10

    Performance depends on geometry-backbone quality, and inaccurate poses or noisy depth can degrade anchor placement and final rendering; very large or unbounded scenes may need an adaptive anchor budget.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent feed-forward 3D Gaussian Splatting (3DGS) methods have enabled effective multi-view 3D reconstruction by predicting Gaussian primitives in a single forward pass. However, existing approaches predominantly adopt a pixel-aligned paradigm that predicts one Gaussian per input pixel, coupling the Gaussian count to the input resolution and number of views. This also leads to redundant primitives in textureless regions and insufficient coverage in geometrically complex areas. We present Anchor3DGS, a pose-free feed-forward 3DGS framework that replaces pixel-aligned prediction with a compact anchor-based representation. Guided by information entropy, our method places a set of 3D anchors in the scene, concentrating them in regions of higher visual complexity while maintaining broad spatial coverage. Each anchor performs occlusion-aware aggregation of multi-view image features, exchanges information with other anchors, and is decoded into a set of local Gaussians. A lightweight feature enhancement module further refines the coarse RGB output from Gaussian splatting and improves rendering fidelity. Extensive experiments on diverse benchmark datasets demonstrate that our Anchor3DGS achieves state-of-the-art novel view synthesis quality while only using an order of magnitude fewer Gaussian primitives.

1 Introduction

Existing feed-forward 3DGS methods predict one Gaussian per input pixel, tying representation size to image resolution and view count while leaving redundancy and coverage challenges. Anchor3DGS instead uses content-guided 3D anchors and reports strong novel-view synthesis results with fewer Gaussians.

1 Introduction

  • The framework targets pose-free reconstruction from uncalibrated multi-view images, extending feed-forward 3DGS beyond approaches that require precomputed camera poses.Earlier feed-forward methods predict Gaussians in a single forward pass, and recent methods eliminate the dependency on precomputed camera poses.Source: §1 · pp. 1–2
  • Pixel-aligned methods allocate one Gaussian per input pixel, causing redundancy in homogeneous regions, insufficient coverage in poorly observed areas, and Gaussian counts tied to resolution and view count.Independent single-pixel predictions do not use context from well-observed neighbors to reconstruct poorly observed regions.Source: §1 · pp. 1–2
  • Anchor3DGS replaces pixel-aligned prediction with a fixed-size set of 3D anchors, decoupling Gaussian allocation from input resolution and view count.The anchor count is a hyperparameter, with representation capacity allocated according to scene content.Source: §1 · p. 2
  • Entropy-guided sampling provides broad scene coverage while allocating more anchors to visually complex regions and fewer to simple regions.Anchors aggregate features across views with occlusion-aware weighting, exchange information, and decode into local Gaussians; feature vectors and a lightweight network refine coarse splatted RGB.Source: §1 · p. 2
  • Anchor3DGS achieves state-of-the-art rendering quality in sparse-view settings and comparable quality in dense-view settings with an order of magnitude fewer Gaussians.The conclusion also reports lower storage cost in dense-view settings.Source: §1 · p. 2

2 Related Work

Prior work moves from per-scene NeRF and 3DGS optimization toward feed-forward and pose-free reconstruction. Anchor3DGS addresses the remaining reliance on dense pixel-aligned Gaussian allocation by representing scenes with compact 3D anchors.

Per-Scene Novel View Synthesis

  • NeRF and 3DGS per-scene methods require camera poses and tedious optimization, whereas Anchor3DGS predicts poses and Gaussians in a feed-forward pass.3DGS uses explicit anisotropic Gaussians and tile-based rasterization for real-time rendering; NeRF uses density and radiance fields with volume rendering.Source: §2 · p. 3

Feed-Forward Novel View Synthesis

  • Feed-forward Gaussian methods commonly use dense pixel-aligned predictions, leaving massive redundancy; Anchor3DGS instead allocates a compact set of 3D anchors using scene structure and visual complexity.AnySplat [11] applies post-hoc voxelization, but the passage says the remaining Gaussians are still far from compact.Source: §2 · p. 3
  • Pose-free methods such as FLARE [44] and AnySplat [11] infer camera poses and geometry-aligned Gaussians using geometric priors from 3D foundation models.Other feed-forward approaches include pixelSplat [4], which uses epipolar attention, and MVSplat [6], which constructs explicit cost volumes.Source: §2 · p. 3

3 Method

Anchor3DGS builds a fixed-budget 3D anchor representation from uncalibrated views, then encodes anchors with multi-view context and decodes them into local Gaussians.

3.1 Pose-Free Geometry Initialization

  • A VGGT-based pose-free geometry backbone predicts camera parameters and depth maps, which are unprojected into 3D points for anchor placement and later view projection.Source: §3.1 · pp. 3–4

3.2 Entropy-Guided Anchor Sampling

  • Anchors combine coarse grid subsampling for scene coverage with entropy-proportional sampling that concentrates budget in visually complex regions; the fixed count is view- and resolution-independent.Source: §3.2 · p. 4

3.3 Occlusion-Aware Multi-View Anchor Aggregation

  • Each anchor gathers projected per-view features, with depth-based visibility weights suppressing likely occluded views; anchor self-attention lets weakly observed regions draw context from better-observed neighbors.Source: §3.3 · p. 5

3.4 Anchor-to-Gaussian Decoding and Rendering

  • Each anchor decodes K local Gaussians whose offsets scale with local anchor spacing, while shared-weight RGB and feature rasterization feeds a UNet residual refinement.Source: §3.4 · p. 6

4 Training

Training uses varied multi-view data and context-view sampling, with photometric and geometric losses supervising refined renderings and Gaussian depth consistency.

4 Training

  • Five datasets spanning synthetic and real-world, indoor and outdoor, object-scale to street-scale scenes supply diverse training captures.Context-view sampling uses index gaps for sequential captures and camera-pose-distance rankings otherwise, with coverage extent randomized across iterations.Source: §4 · p. 6
  • Training minimizes pixel-wise MSE and perceptual LPIPS losses between refined renderings and ground-truth images.A depth-consistency term also encourages Gaussian-rasterized depth to agree with geometry-backbone depth maps.Source: §4 · pp. 6–7
  • The sparse-view comparison evaluates PSNR, SSIM, LPIPS, and predicted Gaussian count across three benchmarks.The comparison covers 3- and 6-input-view settings on VR-NeRF, MipNeRF360, and DL3DV.Source: Table 1 · p. 7

5 Experiments

Experiments assess Anchor3DGS across sparse- and dense-view benchmarks, reporting strong rendering quality with a compact Gaussian representation. Qualitative comparisons and ablations examine visual outcomes and the contribution of design choices.

5.1 Experimental Setup

  • Benchmarks cover object-centric, indoor, and outdoor scenes, with comparisons against pose-free feed-forward methods and a dense-view optimization reference.NoPoSplat and FLARE are included only for 3- and 6-view comparisons; dense-view evaluation uses 32 views.Source: §5.1 · p. 7

5.2 Quantitative Results

  • Over 2 dB PSNR gains over the best baseline on MipNeRF360 and DL3DV accompany 0.26M Gaussians per scene in sparse-view settings.Across all three benchmarks, Anchor3DGS achieves the best PSNR, SSIM, and LPIPS; dense-view results are comparable to pixel-aligned baselines with an order of magnitude fewer Gaussians.Source: §5.2 · pp. 7–8, Table 2 · p. 8

5.3 Qualitative Results

  • Qualitative comparisons show coherent structure, sharp detail, and fewer poorly observed-region artifacts than sparse-view baselines.The comparisons note blur in NoPoSplat and holes in AnySplat; dense-view renderings remain on par while using an order of magnitude fewer primitives.Source: §5.3 · pp. 8–9, Figure 4 · p. 8

5.4 Ablation Study

  • Removing any of the four key designs degrades quality; removing entropy-guided sampling causes a 1.53 dB PSNR drop at 3 views.The default N = 32,768 anchors and K = 8 decoded Gaussians balance rendering quality and compactness.Source: §5.4 · p. 9, Table 3 · p. 9

6 Conclusion

Anchor3DGS replaces pixel-aligned prediction with a compact anchor-based representation whose capacity is allocated according to scene content. The conclusion reports strong sparse-view quality and comparable dense-view quality with fewer Gaussians, while noting backbone and scene-scale limitations.

6 Conclusion

  • Entropy-guided sampling, occlusion-aware multi-view aggregation, and feature-based refinement support high-fidelity novel-view synthesis with a compact representation.The representation decouples Gaussian count from input resolution and view number.Source: §6 · p. 9
  • Sparse-view quality is state of the art, while dense-view quality is comparable with an order of magnitude fewer Gaussians and lower storage cost.The compact representation is also proposed as a potential global geometric memory for future video world models.Source: §6 · p. 10
  • Rendering may degrade when the geometry backbone predicts inaccurate poses or noisy depth, and very large or unbounded scenes may benefit from an adaptive anchor budget.Scene-adaptive budgeting is left as future work.Source: §6 · p. 10

A More Implementation Details

Implementation details specify the model architecture, evaluation protocol, and benchmark dataset composition. Evaluation aligns predicted target poses to the Gaussian coordinate frame before measuring rendered-image metrics.

A More Implementation Details

  • A VGGT-based ViT backbone fuses intermediate layer features into per-view maps, while four self-attention blocks refine aggregated anchor features.A three-level UNet outputs a three-channel RGB residual; its final convolution is zero-initialized so training begins as an identity mapping.Source: A More Implementation Details · p. 14
  • Training uses five datasets, while evaluation reports on VR-NeRF, MipNeRF360, and held-out DL3DV scenes.The evaluation set includes four VR-NeRF scenes, four MipNeRF360 scenes, and ten held-out DL3DV scenes.Source: A More Implementation Details · p. 14
  • Evaluation aligns poses from a context-only pass and a context-plus-target pass before rendering Gaussians at target views.For NoPoSplat, target poses are recovered through 200-step test-time optimization minimizing rendered-image MSE against ground truth.Source: A More Implementation Details · p. 14

B Additional Experimental Results

Additional evaluations report pose-estimation performance and test novel-view synthesis with 64 context views. Anchor3DGS outperforms the compared pose baselines across all reported thresholds and remains competitive in dense-view rendering with far fewer Gaussians.

B.1 Camera Pose Estimation

  • Anchor3DGS outperforms AnySplat and VGGT across the 10-, 20-, and 30-degree pose-estimation AUC thresholds on Co3Dv2.The authors report that joint training with the anchor-based 3DGS pipeline does not compromise the geometry backbone's pose-estimation ability.Source: §B.1 · p. 14, Table A1 · p. 15

B.2 Novel-View Synthesis with Denser Views

  • With 64 context views, Anchor3DGS remains competitive with or better than DA3 and AnySplat in rendering metrics while using roughly one to two orders of magnitude fewer Gaussians.The evaluation covers VR-NeRF and DL3DV and reports Gaussian count and storage size alongside rendering metrics.Source: §B.2 · p. 14, Table A2 · p. 15

B.3 Rendering Speed Analysis

Rendering-speed tests show that Anchor3DGS maintains stable speed as context views increase, unlike pixel-aligned baselines. At 64 views, it is faster than both compared baselines.

B.3 Rendering Speed Analysis

  • The speed test measures per-frame 3DGS rendering over 100 passes on one H20 GPU after generating Gaussians for each view configuration.The comparison uses VR-NeRF and two pixel-aligned feed-forward baselines.Source: §B.3 · p. 15, Table A3 · p. 15
  • At 64 views, Anchor3DGS is 1.6× faster than AnySplat and 2.2× faster than DA3; its rendering speed stays stable with about 0.26M Gaussians across view counts.The authors attribute the stable speed to a constant Gaussian count, while pixel-aligned baselines' FPS drops as view count increases.Source: §B.3 · p. 15, Table A3 · p. 15

B.4 Multi-View Consistency Analysis

The consistency analysis compares UNet-refined renders with renders from the same geometry backbone and anchor decoder without the UNet. The authors report improved rendering quality with only a marginal increase in adjacent-view warping error.

B.4 Multi-View Consistency Analysis

  • The UNet improves rendering quality while increasing adjacent-view warping RMSE only marginally over the version without the UNet.RMSE is measured along an interpolated virtual-camera trajectory and masked to co-visible, geometrically consistent pixels.Source: §B.4 · p. 15, Table A4 · p. 16
  • The UNet's residual is predicted from a feature map rasterized from the same 3D Gaussians and alpha-compositing weights as the base image.The authors state that this lets the residual inherit the underlying 3D geometric structure rather than being hallucinated in 2D image space; the lightweight UNet also limits its capacity to introduce artifacts.Source: §B.4 · pp. 15–16

B.5 More Ablation Results

The additional ablations attribute the main rendering gains to the anchor-based representation, while reporting that removing individual components degrades quality. The supplementary material also discusses social impacts and provides further visual comparisons.

B.5 More Ablation Results

  • Removing any one of the UNet enhancement, anchor attention, occlusion-aware aggregation, or entropy sampling components degrades rendering quality; the default N=32,768 and K=8 balances quality and Gaussian compactness.The ablation covers DL3DV at 3, 6, and 32 input views, while the anchor-count and Gaussians-per-anchor study covers DL3DV and MipNeRF360.Source: §B.5 · p. 16, Table A5 · p. 16, Figure A1 · p. 16
  • At 3 views, Anchor3DGS gains +2.10 dB on VR-NeRF and +3.33 dB on DL3DV over its backbone with a pixel-aligned decoder.With the fine-tuned backbone held constant, the authors report that the backbone contribution alone is small and that the rendering gain primarily comes from the anchor-based representation.Source: §B.5 · pp. 16–17, Table A6 · p. 17

C Broader Social Impacts

  • The authors describe reduced computational and memory footprints as benefits, while noting possible misuse for misleading imagery and privacy concerns when reconstructing private spaces from public images.They identify responsible data curation, watermarking or provenance tracking, and consent as possible mitigations.Source: C Broader Social Impacts · p. 17

D More Visualizations

Loading 2610.21634v1…