PaperlayerView exact revision on arXiv

Source-linked AI summary

HierGF: Hierarchical Gaussian Fields via Geometry-perception Message Passing for Sparse-view 3D Reconstruction

Bi'an Du, Zhimin Zhang, Daizong Liu, Baoquan Chen, Wei Hu

arXiv:2610.01056v1cs.CVeess.IV

TL;DR

Sparse-view reconstruction must recover coherent geometry and appearance from few incomplete views, while existing approaches leave reliability and structural guidance unresolved in under-sampled regions. HierGF uses coarse-to-fine Gaussian fields, enhanced pseudo-views, confidence weighting, and geometry-consistent densification. The paper reports improved reconstruction quality and cross-view consistency across three primary sparse-view benchmarks, with additional comparisons on LLFF and DTU.

  • Problem

    Few sparse views provide weak multi-view constraints and leave regions under-sampled, while prior methods still rely on fixed priors or heuristic density control.

    Source: §I · pp. 1–2

  • Method

    HierGF refines a separately optimized fine Gaussian field using enhanced pseudo-views, confidence-weighted supervision, and geometry-consistent densification initialized from a coarse field.

    Source: §I · p. 2

  • Results

    Across three primary sparse-view benchmarks, HierGF improves reconstruction quality and cross-view consistency from few inputs, with additional quantitative comparisons on LLFF and DTU.

    Source: §V · p. 14

  • Takeaways & Limitations

    Normal-guided insertion and interpolation refine geometry and appearance in under-sampled regions while maintaining surface continuity and recovering fine details.

    Source: §I · p. 2

Abstract

from arXiv · show

Sparse view 3D reconstruction is an important and common scenario in multimedia applications, such as augmented reality/virtual reality (AR/VR) content creation, cultural heritage digitization, and certain robotic applications, where only a limited number of randomly captured views may be available. However, sparse views contain only limited 3D information, posing two major challenges:1) too few images are available for matching, making it difficult to build multi-view consistency; 2) insufficient view coverage leads to a lack of information in under-sampled regions, resulting in missing parts of object structure. Existing methods mostly still rely on limited reprojection errors and regularization terms, which are prone to overfitting to a single view and inconsistent appearances across views. In geometrically under-sampled regions, they often rely on heuristic density control, lacking reliable guidance and often resulting in blurring and structural holes.To address these issues, this paper proposes Hierarchical Gaussian Fields (HierGF), which revisits sparse-view reconstruction from a hierarchical geometry-perception perspective and converts limited observations into reliable self-generated supervision beyond fixed priors and heuristic density control. In particular, we transform coarse 3D geometric information and additional 2D generative priors into structured pseudo-supervision through a two-stage geometry-perception backbone network, thereby enhancing multi-view consistency with very few input views. In addition, we introduce a learnable confidence network to guide gradients toward cross-view consistent content, and a geometrically consistent densification module to improve the reconstruction of multi-view alignment and under-sampled regions.

I. INTRODUCTION

Sparse-view reconstruction must recover coherent geometry and appearance from few uneven observations, but prior per-scene methods struggle in under-constrained regions. HierGF combines coarse-to-fine Gaussian refinement, perceptual pseudo-supervision, and geometry-consistent densification; the introduction also reports reconstruction-time comparisons on one evaluated setting.

I. INTRODUCTION

  • Sparse captures provide limited and uneven constraints, while dependence on dense, calibrated acquisition limits reconstruction in mobile, outdoor, and time-constrained scenarios.Few images hinder multi-view matching, and poor coverage leaves regions under-sampled; heuristic density control may blur or omit missing content.Source: §I · pp. 1–2

A. Optimization-based Per-scene Sparse-view Reconstruction

  • Prior optimization-based methods use depth or unseen-view regularization, visual-hull priors, or density-control refinements, but may remain inconsistent or depend on heuristic priors and densification.The introduction identifies reliable enhanced views and structure-aware Gaussian optimization in under-sampled regions as open problems.Source: §I · p. 2, A. Optimization-based Per-scene… · p. 2

I. INTRODUCTION

  • HierGF first estimates a coarse field, then selectively initializes a separately optimized fine field while pseudo-view supervision and geometry-consistent densification refine appearance and under-sampled structure.Only spatial means and RGB initialization transfer; the fine field re-estimates opacity, anisotropic scale, rotation, and higher-order appearance attributes.Source: §I · p. 2
  • 632 s per scene on one NVIDIA A100 GPU in the MipNeRF360 kitchen 4-view setting corresponds to speedups of 2.07×, 11.12×, and 2.85× over ZeroRF, GaussianObject, and WaveletGS, respectively.The introduction also reports the lowest measured post-optimization rendering time among those methods.Source: §I · p. 2

II. RELATED WORKS

  • The introduction positions HierGF within optimization-based reconstruction and contrasts it with feed-forward and 3D-prior work reviewed for conceptual context.The paper says its quantitative comparisons are restricted to per-scene baselines.Source: §II · p. 2, A. Optimization-based Per-scene… · p. 3

A. Optimization-based Per-scene Sparse-view Reconstruction

  • HierGF’s related-work distinction is an explicit two-field refinement that combines selective parameter inheritance, confidence-weighted enhanced supervision, and geometry-consistent sparse-region completion.The cited comparison table is identified as covering diffusion-based sparse-view methods, but its caption does not establish table outcomes.Source: A. Optimization-based Per-scene… · p. 3, Table I · p. 3

B. Feed-forward Sparse-view Reconstruction and 3D Priors

Feed-forward reconstruction and 3D priors offer fast, scalable view synthesis, but do not directly target high-quality per-scene 3D Gaussian reconstruction in this setting. HierGF instead couples a coarse geometry field to perceptual pseudo-view supervision, confidence weighting, and geometry-consistent densification.

B. Feed-forward Sparse-view Reconstruction and 3D Priors

  • Feed-forward models can synthesize views quickly, but may lack explicit 3D representation or iterative scene-specific optimization and are not the paper’s direct quantitative baselines.The paper presents them as conceptual context and potential prior sources, and restricts quantitative comparisons to per-scene methods.Source: B. Feed-forward Sparse-view… · p. 3

C. Reliability-aware and Structural Supervision

  • Synthesized views can contain local artifacts, while prior work does not explicitly learn per-pixel or per-view reliability and tie Gaussian densification to evolving geometry.HierGF combines perceptual enhancement, learned confidence, and normal-guided densification.Source: C. Reliability-aware and Structural… · p. 3

III. METHOD

  • The method couples coarse- and fine-scale Gaussian fields through geometry–perception message passing, with enhanced and filtered pseudo-views supplying training signals.The method overview assigns structure-aware, normal-weighted densification to the fine-scale field in under-sampled regions.Source: §III · pp. 3–4

A. The Motivation

  • Sparse views create weak matching constraints and uncovered regions, where limited gradients and heuristic density control can yield single-view artifacts, blur, holes, floaters, or broken thin structures.HierGF renders pseudo-views from a coarse field for diffusion-prior refinement and confidence-based pixel and view selection, while normal-weighted interpolation guides new Gaussians along estimated surface directions.Source: A. The Motivation · p. 4

B. Problem Formulation and Representation

HierGF uses an explicit Gaussian representation and a two-stage process to refine sparse-view reconstructions. A coarse field supplies geometry and pseudo-views; confidence-weighted supervision and geometry-consistent densification refine a separately optimized fine field.

B. Problem Formulation and Representation

  • 3D Gaussian Splatting represents each primitive with a center, rotation, anisotropic scale, opacity, and spherical-harmonic color coefficients for rendering images and optionally depth.Its explicit parameterization supports real-time rendering and fine-grained density control.Source: B. Problem Formulation and… · p. 4
  • The coarse Gaussian field renders additional views that are enhanced with 2D-assisted perceptual supervision and confidence weighting before they guide fine-field refinement.The framework combines this feedback with geometry-consistent densification in under-sampled regions.Source: B. Problem Formulation and… · pp. 4–5

C. Hierarchical Geometry–perception Message Passing

  • Weakly constrained regions give photometric losses little guidance and leave density growth directionless, motivating a coarse-to-fine scheme with perceptual supervision and geometry-consistent densification.The coarse stage targets stable visible-region alignment, reasonable global scale, and basic cross-view coherence rather than early photorealistic detail.Source: C. Hierarchical Geometry–perception… · p. 5
  • The fine stage periodically enhances coarse-field pseudo-views and predicts per-pixel and per-view confidence weights to reweight color and perceptual losses.New Gaussians are inserted along estimated surface normals via normal-weighted interpolation; the diffusion backbone is not jointly optimized with the final Gaussian representation.Source: C. Hierarchical Geometry–perception… · p. 5
  • Only spatial means and RGB initialization transfer to the fine field; opacity, anisotropic scale, rotation, and higher-order appearance attributes are reinitialized.This selective inheritance avoids locking in suboptimal appearance from the coarse field.Source: C. Hierarchical Geometry–perception… · p. 5
  • Alternating pseudo-view generation, enhancement, confidence estimation, optimization, and densification lets pseudo-observations track evolving geometry and progressively strengthen multi-view supervision.The passage describes this feedback as gradually completing previously under-constrained areas.Source: C. Hierarchical Geometry–perception… · p. 5
  • Confidence weighting adjusts enhanced pseudo-view losses, whereas geometry-consistent densification changes the Gaussian representation in sparse regions; confidence weights do not control densification.Source: C. Hierarchical Geometry–perception… · p. 5

D. 2D-assisted Perceptual Enhancement

The section creates pseudo-supervision by rendering coarse-field views, enhancing them with an adapted 2D prior, and using the results to train a fine field. A confidence network weights pixel- and view-level supervision so unreliable enhanced content has less influence.

D. 2D-assisted Perceptual Enhancement

  • Only ControlNet LoRA parameters are adapted for enhancement; the Stable Diffusion v1.5 backbone remains frozen.Adaptation uses degraded/reference pairs rendered from the current scene after injecting noise into Gaussian attributes.Source: D. 2D-assisted Perceptual Enhancement · p. 5
  • Coarse-field views are enhanced by a 2D prior and used as self-supervision to optimize the fine Gaussian field.Rendered views are passed through the fine-tuned prior to obtain enhanced references for fine-field training.Source: D. 2D-assisted Perceptual Enhancement · p. 5
  • The refinement is described as relying solely on the provided dataset, without adding extraneous visual information.The authors characterize this independent refinement as supporting accurate and consistent 3D models.Source: D. 2D-assisted Perceptual Enhancement · p. 6
  • Both Gaussian fields use color and monocular-depth losses, with the color loss combining L1 and D-SSIM terms.The depth loss is translation- and scale-invariant and guides geometry; rendered and monocular-estimated depths are computed per frame.Source: D. 2D-assisted Perceptual Enhancement · p. 6
  • A lightweight confidence network predicts pixelwise and view-wise weights from coarse renderings and their enhanced counterparts.The weights reweight reprojection and perceptual losses, focusing gradients on reliable pixels and views while reducing unstable regions' influence.Source: D. 2D-assisted Perceptual Enhancement · p. 6
  • Per-image mean normalization makes pixel weights redistribute mass within each image rather than alter its global weight scale.The confidence objective also regularizes score distributions to guard against collapse to all-zero or all-one weights.Source: D. 2D-assisted Perceptual Enhancement · pp. 6–7

E. Geometry-Consistent Densification

Geometry-consistent densification targets sparse regions where weak image gradients make conventional density control unreliable. It identifies sparse voxels and interpolates new Gaussians using normal and distance cues to preserve local structure.

E. Geometry-Consistent Densification

  • Normal-weighted interpolation places new Gaussians along the underlying surface to improve geometric continuity where conventional density control can leave gaps or distortions.The method is designed for sparse regions with weak gradients, where interpolation from a few neighbors can fail.Source: E. Geometry-Consistent Densification · p. 7
  • Voxels containing fewer than threshold τ Gaussians are marked sparse, while robust scene extent uses the 95th percentile of pairwise center distances after outlier removal.Source: E. Geometry-Consistent Densification · p. 7
  • Normal-guided insertion requires at least 8 neighbors, retains at most the nearest 32, and falls back to the original ADC update when a seed has fewer than 8.Normals are estimated by PCA over a radius-r neighborhood.Source: E. Geometry-Consistent Densification · p. 8
  • Interpolation weights combine normal similarity, controlled by α, and spatial distance, controlled by β, through their product.Source: E. Geometry-Consistent Densification · p. 8
  • Weighted averages of neighboring positions and attributes are applied throughout each sparse region, doubling its original density while preserving geometry and mitigating arbitrary-interpolation distortions.Source: E. Geometry-Consistent Densification · p. 8

A. Implementation Detail

HierGF uses a 10,000-iteration coarse-to-fine optimization schedule, with perceptual enhancement between the coarse and fine stages. The reported per-scene runtime is about 10.5 minutes on one NVIDIA A100 GPU.

A. Implementation Detail

  • The coarse field is optimized for 4,000 iterations and the fine field for 6,000, for a total Gaussian optimization budget of 10,000 iterations per scene.Source: A. Implementation Detail · p. 8
  • Coarse-stage adaptive density control runs every 100 iterations from iteration 500 to 3,500, with opacity resets at iterations 1,500 and 3,000.The coarse stage uses ZoeDepth for translation- and scale-invariant monocular depth supervision.Source: A. Implementation Detail · p. 8
  • Between stages, degraded/reference image pairs are constructed and only the enhancement module’s LoRA parameters are adapted for 500 steps; the diffusion backbone remains frozen.Source: A. Implementation Detail · p. 8
  • The fine field inherits only spatial means and RGB initialization, refreshes enhanced pseudo-views every 500 iterations, and uses geometry-consistent densification.Source: A. Implementation Detail · p. 8
  • At 779 × 520 resolution, per-scene reconstruction takes approximately 10.5 minutes (632 s) on a single NVIDIA A100 GPU.Source: A. Implementation Detail · p. 8

B. Dataset

The evaluation uses MipNeRF360, OmniObject3D, and OpenIllumination for sparse-view reconstruction. The datasets span unbounded scenes, scanned objects, and real-world LightStage captures.

B. Dataset

  • The evaluation covers MipNeRF360, OmniObject3D, and OpenIllumination across varying input-view counts for sparse-view 360° object reconstruction.The OpenIllumination sparse dataset is the version introduced in ZeroRF.Source: B. Dataset · pp. 8–9
  • OmniObject3D, one of three datasets evaluated for sparse-view 360° object reconstruction with varying input views, features a comprehensive collection of over 6,000 high-quality, real-scanned 3D content.Source: B. Dataset · p. 8
  • Figure 4 shows qualitative examples from OpenIllumination using four input views.Source: Figure 4 · p. 9
  • OpenIllumination is a real-world dataset captured using LightStage.Source: B. Dataset · p. 9

C. Evaluation Results & Analysis

HierGF is reported to achieve the best or very competitive PSNR and SSIM across three benchmarks and to lead on LPIPS in most scenes. Qualitative comparisons and ablations examine appearance, geometry, and sparse-region completion.

C. Evaluation Results & Analysis

  • The main evaluation compares HierGF with a broad set of sparse-view reconstruction methods across three benchmarks; competitors include Gaussian-based approaches, vanilla 3DGS, few-view NeRF variants, and FSGS.Table VI covers OpenIllumination; † marks metrics taken from ZeroRF, with dataset sampling and processing following ZeroRF.Source: C. Evaluation Results & Analysis · p. 9, Table VI · p. 9
  • Additional LLFF and DTU quantitative comparisons cover 3-, 6-, and 9-view settings, with qualitative comparisons in the 3-view setting against ground truth and three competing methods.The displayed scenes and scans include fern, orchids, room, fortress, and scans 34, 40, 103, and 8.Source: Table VIII · p. 10, Table IX · p. 10, C. Evaluation Results & Analysis · p. 10
  • For directly related methods, the paper distinguishes reported from re-run results and discloses prior or initialization assumptions, aligning evaluation settings when re-running methods.Table VII lists representation type, assumptions, aligned benchmarks, and result source.Source: C. Evaluation Results & Analysis · p. 10, Table VII · p. 10
  • Across MipNeRF360, OmniObject3D, and OpenIllumination, HierGF achieves the best or very competitive PSNR and SSIM at all tested view counts, often improving 4-view PSNR by more than 10% over the strongest baselines.It surpasses the strongest baselines on LPIPS in most scenes, leading in most four- and six-view cases.Source: C. Evaluation Results & Analysis · p. 10
  • Four-view examples show sharper distant structures, cleaner backgrounds, fewer floaters, and finer object details; OpenIllumination examples show more consistent shading and fewer cross-view intensity jumps.Source: C. Evaluation Results & Analysis · pp. 10, 12
  • In the densification ablation, distance-only interpolation reduces some holes but causes point drift and softened edges, while full geometry-consistent completion best preserves fine details and stable geometry.Figure 8 highlights improved thin-structure preservation and gap closure for the full model.Source: Figure 8 · p. 12

D. Ablation Studies

The ablations compare HierGF’s hierarchical stages and supervision components, finding advantages for the full two-stage design, confidence-guided perceptual enhancement, and geometry-consistent densification. Additional controlled tests examine depth supervision, transfer strategies, densification radius, and weighting.

Hierarchical backbone

  • Coarse-only recovers global shape but leaves under-sampled regions blurry and thin structures broken; fine-only sharpens textures yet destabilizes geometry and lowers PSNR and SSIM.Single-field joint training reduces some artifacts relative to fine-only but still trails HierGF in perceptual quality and multi-view consistency.Source: D. Ablation Studies · p. 12, Table X · p. 12

Coarse-to-fine transfer

2D-assisted perceptual enhancement

  • Full 2D-assisted enhancement with diffusion and both confidence terms achieves the best perceptual quality and multi-view consistency on all three datasets.Removing 2D assistance degrades realism in occluded and low-light regions and worsens cross-view shading; dropping either confidence term reduces LPIPS and cross-view SSIM.Source: Table XI · p. 12, D. Ablation Studies · p. 13

ZoeDepth supervision

Pseudo-view and diffusion components

  • Enhanced pseudo-supervision is most effective when combined with confidence-guided filtering.The controlled ablation separates pseudo-views, the frozen diffusion prior, LoRA adaptation, and pixel- and view-level confidence weighting.Source: D. Ablation Studies · p. 13, Table XV · p. 13

Geometry-consistent densification

  • Without geometry-consistent densification, under-sampled areas develop gaps, floaters, and broken thin structures, with SSIM and LPIPS drops.Distance-only interpolation reduces some holes but causes point drift across surface layers and softened edges; the full method best preserves fine details and stable geometry.Source: Table XII · p. 12, D. Ablation Studies · p. 13

Densification radius

Densification weighting

  • The study compares distance-only, additive normal-distance, and multiplicative normal-distance weighting, identifying the multiplicative formulation as HierGF’s choice.Source: D. Ablation Studies · p. 13

E. Efficiency Analysis

On the MipNeRF360 kitchen scene with four input views, HierGF reports 632 s for reconstruction and 70 s for post-optimization rendering. The conclusion reports improved reconstruction quality and cross-view consistency on primary sparse-view benchmarks, while identifying unposed and dynamic scenes as future extensions.

E. Efficiency Analysis

  • 632 s for reconstruction and 70 s for post-optimization rendering; HierGF’s reconstruction is 2.07×, 11.12×, and 2.85× faster than ZeroRF, GaussianObject, and WaveletGS, respectively.On the same scene and setting, rendering time is reduced by 10.3%, 21.3%, and 26.3% relative to those methods; measurements use 779 × 520 resolution on one NVIDIA A100 GPU.Source: E. Efficiency Analysis · pp. 13–14, Table XVIII · p. 14
  • Diffusion processing is used only during reconstruction for pseudo-view enhancement; final rendering uses the optimized Gaussian representation alone.The stage-wise breakdown identifies Gaussian optimization as the dominant runtime cost.Source: E. Efficiency Analysis · p. 14, Figure 9 · p. 14

V. CONCLUSION

  • On three primary sparse-view benchmarks, with additional quantitative comparisons on LLFF and DTU, HierGF improves reconstruction quality and cross-view consistency from few inputs.Source: §V · p. 14
  • Replacing interpolation with learnable geometric priors and extending to unposed or dynamic scenes are future-work directions.Source: §V · p. 14

Related papers

Loading 2610.01056v1…