PaperlayerView exact revision on arXiv

Source-linked AI summary

HyRF: Hybrid Radiance Fields for Memory-efficient and High-quality Novel View Synthesis

Zipeng Wang, Dan Xu

arXiv:2509.17083v2cs.CV

TL;DR

3DGS offers real-time, high-quality novel views but incurs memory overhead, and neural-field compression can miss high-frequency Gaussian details. HyRF combines grid-based neural fields with compact explicit Gaussians and a neural background; it reports superior quality, a 20× smaller model than 3DGS, and real-time performance.

  • Problem

    3DGS has substantial memory overhead from per-Gaussian color and anisotropic-shape parameters, while fixed-resolution neural fields can miss high-frequency Gaussian variations and fine details.

    Source: §1 · pp. 1–2

  • Method

    HyRF uses grid-based fields for low-frequency Gaussian properties and compact explicit Gaussians for high-frequency details, with separate geometry and appearance fields and a neural background map.

    Source: §1 · p. 2

  • Results

    HyRF achieves superior rendering quality, reduces model size by 20× compared to 3DGS, and maintains real-time performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    HyRF combines high-frequency detail preservation with reduced memory overhead while retaining real-time rendering performance.

    Source: §1 · pp. 2–3

    HyRF does not address aliasing, sometimes produces inaccurate surface reconstruction, and currently benefits from high-end GPUs for high rendering speed.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recently, 3D Gaussian Splatting (3DGS) has emerged as a powerful alternative to NeRF-based approaches, enabling real-time, high-quality novel view synthesis through explicit, optimizable 3D Gaussians. However, 3DGS suffers from significant memory overhead due to its reliance on per-Gaussian parameters to model view-dependent effects and anisotropic shapes. While recent works propose compressing 3DGS with neural fields, these methods struggle to capture high-frequency spatial variations in Gaussian properties, leading to degraded reconstruction of fine details. We present Hybrid Radiance Fields (HyRF), a novel scene representation that combines the strengths of explicit Gaussians and neural fields. HyRF decomposes the scene into (1) a compact set of explicit Gaussians storing only critical high-frequency parameters and (2) grid-based neural fields that predict remaining properties. To enhance representational capacity, we introduce a decoupled neural field architecture, separately modeling geometry (scale, opacity, rotation) and view-dependent color. Additionally, we propose a hybrid rendering scheme that composites Gaussian splatting with a neural field-predicted background, addressing limitations in distant scene representation. Experiments demonstrate that HyRF achieves state-of-the-art rendering quality while reducing model size by over 20 times compared to 3DGS and maintaining real-time performance. Our project page is available at https://wzpscott.github.io/hyrf/.

1 Introduction

3DGS enables real-time, high-quality novel views but uses many per-Gaussian parameters, while grid-based neural fields can miss high-frequency detail. HyRF combines compact explicit Gaussians with decoupled neural fields and a neural background to target quality, compactness, and real-time rendering.

1 Introduction

  • 3DGS uses 59 parameters per Gaussian, including 48 for view-dependent color and 7 for anisotropic scale and rotation.NeRF-based methods model view-dependent effects through neural-network conditioning with minimal parameter growth.Source: §1 · pp. 1–2
  • Fixed-resolution grid-based neural fields struggle with high-frequency Gaussian-property changes, limiting reconstruction of thin structures and fine color variations.The problem is pronounced at object boundaries with rapid opacity and scale changes and for high-frequency view-dependent effects.Source: §1 · p. 2
  • HyRF assigns low-frequency variations to grid-based neural fields and preserves high-frequency details with sparse compact Gaussians storing positions, isotropic scales, opacity, and diffuse colors.Separate geometry and appearance fields predict scale, opacity, and rotation versus view-dependent color.Source: §1 · p. 2
  • HyRF culls out-of-view Gaussians, predicts remaining properties with neural fields, and alpha-composites a neural background map with foreground splats.The background map is projected onto a sphere to represent foreground and remote background objects.Source: §1 · p. 2
  • HyRF achieves superior rendering quality, reduces model size by 20× compared to 3DGS, and maintains real-time performance.Source: §1 · p. 3

2 Related Work

Related work contrasts implicit neural radiance fields with explicit Gaussian representations and reviews efforts to make 3DGS more compact. HyRF differs from hybrid compression methods by using global grid-based predictions with explicit residual Gaussians.

Neural Radiance Fields

  • NeRF methods use neural networks and volume rendering, but extensive querying slows rendering; structured feature arrays accelerate them, though grid resolution and hash collisions can limit quality.TensoRF uses low-rank tensor components, while Instant-NGP combines a multi-resolution hash table with a fully-fused MLP.Source: §2 · p. 3

Explicit Radiance Fields

  • 3DGS uses point-based alpha blending for state-of-the-art quality and real-time performance, but its models are considerably larger than NeRF-based methods.Source: §2 · p. 3

Compressed 3D Gaussian Splatting

  • 3DGS compression includes vector quantization and neural prediction; unlike Scaffold-GS and LocoGS, HyRF globally predicts properties with grid-based fields and adds explicit residual Gaussians.Its smaller explicit representation is also compatible with vector quantization.Source: §2 · p. 3

3 Methodology

HyRF combines compact explicit Gaussians with grid-based neural fields, separating geometry from view-dependent color. Its rendering pipeline culls invisible Gaussians and uses a radiance field for the background; optimization uses L1 and SSIM losses with opacity resets and pruning.

3.1 Preliminary: 3DGS

  • 3DGS represents scenes with optimizable Gaussians defined by position, opacity, rotation, scale, and color, which are depth-ordered, projected, and alpha-blended.Color uses spherical harmonics to model view-dependent effects.Source: §3.1 · pp. 3–4

3.2 Hybrid Radiance Fields

  • HyRF stores compact explicit Gaussian parameters alongside grid-based neural fields, then aggregates predicted and stored properties for 3DGS rasterization.The explicit Gaussians store position, diffuse color, isotropic scale, and opacity; neural fields predict properties that complement them.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • Separate neural fields predict geometry properties and view-dependent color, addressing the weak correlation that makes joint prediction unsatisfactory.The geometry field models scale, opacity, and rotation; the appearance field models view-dependent color.Source: §3.2 · p. 4
  • Scene coordinates are normalized and contracted to (0, 1), encoded into features, and decoded into Gaussian properties.The contracted range meets the input requirements of the multi-resolution hash encoding.Source: §3.2 · pp. 4–5
  • View-dependent Gaussian color prediction adds an encoded view direction to the MLP input.The view-direction encoding uses positional encoding techniques similar to NeRF-based methods [27] [3].Source: §3.2 · p. 5

3.3 Hybrid Rendering

  • Visibility pre-culling removes points outside the projected image before neural-field queries, retaining a tolerance band for partially intersecting Gaussians.Gaussians too close to the image plane are also discarded because they may cause optimization instability.Source: §3.3 · pp. 5–6
  • A radiance field predicts background color at each ray’s intersection with a large background sphere, which is composited with foreground Gaussian color.The background prediction addresses 3DGS’s difficulty densifying and optimizing extremely distant objects, which can yield blurry backgrounds.Source: Figure 3 · p. 5, §3.3 · p. 6

3.4 Optimization

  • Optimization uses the original 3DGS L1 and SSIM losses, periodically resets explicit opacity during densification, and prunes low-opacity Gaussians.The SSIM loss is weighted by λ.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate HyRF across 25 real-world, synthetic, and urban scenes, finding strong rendering quality with compact models and competitive speed. Ablations show how decoupled fields, background rendering, explicit Gaussian properties, and visibility pre-culling contribute to the results.

4.1 Experimental Setup

  • Experiments cover 25 scenes across three standard real-world datasets, eight NeRF Synthetic scenes, and four scenes from two large-scale urban datasets.Baselines include NeRF-based, grid-based, 3DGS, and Scaffold-GS methods, plus five recent 3DGS compression methods.Source: §4.1 · p. 6

4.2 Results and Evaluation

  • Over 12 times smaller than 3DGS with superior rendering quality and comparable speed; versus Scaffold-GS, 1.5 to 5 times smaller and faster.Table 1 reports the standard real-world scene evaluation.Source: Table 1 · p. 7, §4.2 · p. 7
  • HyRF achieves the best results on NeRF Synthetic, is slightly larger than Instant-NGP, and is over 4 times smaller than 3DGS; on urban scenes, it has the best rendering quality and is 4 to 7 times smaller than 3DGS-based methods.On urban scenes, it is also over 7000 times faster than NeRF-based methods; the speed gap with 3DGS narrows as rendered points increase.Source: §4.2 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 8
  • After compression, HyRF outperforms five state-of-the-art 3DGS compression methods in both model size and rendering quality.The authors report that it maintains superior visual fidelity even after aggressive compression.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Model Analysis

  • A single field predicting all Gaussian parameters degrades image-quality metrics relative to decoupled fields and produces inaccurate geometry, including gaps and holes.The comparison uses a single-field hash size of 2^18, giving it a slightly larger parameter count than the decoupled architecture.Source: §4.3 · p. 8, Table 5 · p. 8, Figure 5 · p. 9
  • Disabling background rendering significantly degrades visual quality for only marginal rendering-speed improvement.The authors identify the background rendering contribution as particularly important for distant scene elements such as clouds and sky.Source: §4.3 · p. 8, Figure 6 · p. 9
  • Removing neural components reduces novel-view quality, while removing explicit color, scale, or opacity respectively harms illumination modeling, thin structures and training stability, or causes floaters.Without neural components, the method falls back to isotropic Gaussians with diffuse shading; positions remain for neural-field queries in the explicit-property ablation.Source: §4.3 · pp. 8–9, Figure 7 · p. 9
  • Visibility pre-culling improves rendering speed by 3.9× without reducing visual quality on real-world 360° scenes, and training analysis reports faster convergence with smaller models than baselines.Figure 8 compares PSNR and model size during training.Source: §4.3 · p. 9, Figure 8 · p. 9

5 Conclusion

HyRF combines explicit compact Gaussians, decoupled neural fields, and hybrid rendering to address the memory bottleneck of explicit Gaussian representations. The authors report that it retains rendering quality and speed advantages, while noting unresolved aliasing, occasional inaccurate surfaces, and hardware constraints on high rendering speed.

5 Conclusion

  • HyRF combines explicit-implicit representation, decoupled geometry and appearance fields, and hybrid rendering to address the memory bottleneck of explicit Gaussians.The supplied ablation-table caption identifies detailed studies of explicit Gaussian properties.Source: §5 · p. 9, Table 6 · p. 9
  • The authors present HyRF as a step toward practical, high-quality real-time neural rendering.They describe novel view synthesis as relevant to applications including virtual production and autonomous systems.Source: §5 · p. 10
  • HyRF does not address the aliasing issue and sometimes produces inaccurate surface reconstruction; high rendering speed currently benefits from high-end GPUs.Comparable efficiency on web platforms or integrated graphics remains an open challenge.Source: §5 · p. 10

A.1 Scene Contraction

HyRF normalizes scene coordinates using an axis-aligned bounding box, then contracts them to (0, 1) for the multi-resolution hash input.

A.1 Scene Contraction

  • Scene coordinates are normalized using the scene’s axis-aligned bounding box before contraction.The AABB is set using dataset-dependent bounds.Source: §A.1 · p. 13
  • For the NeRF synthetic dataset, the AABB minimum and maximum are set to -1.3 and 1.3.For standard datasets, bounds use camera positions; for large-scale datasets, they use point-cloud positions between the 1st and 99th percentiles.Source: §A.1 · p. 13
  • The normalized points are contracted to the range (0, 1) for the multi-resolution hash input.The stated target range meets the hash encoding’s input requirements.Source: §A.1 · p. 13

A.2 Derivation of Ray-Sphere Intersection

The derivation substitutes a ray into a sphere equation to find their intersection, then selects the forward intersection because the ray starts inside the sphere.

A.2 Derivation of Ray-Sphere Intersection

  • Substituting ray r(t) = o + td into a sphere centered at the origin yields a quadratic intersection equation.The sphere has radius r.Source: §A.2 · p. 13
  • The quadratic coefficients are A = d · d, B = 2(o · d), and C = o · o − r^2.These coefficients express the substituted ray-sphere equation as a quadratic in t.Source: §A.2 · p. 13
  • Because the ray originates inside the sphere, both solutions are real, and the positive solution gives the forward intersection.Source: §A.2 · p. 13

A.3 Ablation for View-dependent Appearance Modeling

For view-dependent appearance, HyRF’s hybrid approach reduces model size and slightly improves visual quality compared with spherical-harmonic coefficients.

A.3 Ablation for View-dependent Appearance Modeling

  • HyRF’s hybrid appearance modeling achieves a significant model-size reduction and slightly better visual quality than SH coefficients.The comparison is between SH coefficients and an MLP integrated with the neural field and explicit Gaussians.Source: §A.3 · p. 13, Table 7 · p. 13
  • The ablation compares SH coefficients with a hybrid MLP and neural-field/explicit-Gaussian approach for view-dependent appearance.Source: §A.3 · p. 13, Table 7 · p. 13
  • The authors characterize the hybrid approach as a more compact and powerful way to model view-dependent appearance.Source: §A.3 · p. 13

A.4 Evaluation in Street Scenes

On the KITTI street-scene sequence, HyRF maintains visual quality while using a model over 10 times smaller than 3DGS; background rendering further improves quality, particularly for distant objects and sky.

A.4 Evaluation in Street Scenes

  • On the KITTI 2011_09_26_drive_0002 sequence, HyRF has similar visual quality to 3DGS while being over 10 times smaller.Source: §A.4 · p. 14, Table 8 · p. 14
  • Adding background rendering produces consistent quality improvements, particularly for distant objects and sky regions.Source: §A.4 · p. 14
  • The street-scene evaluation uses the KITTI dataset.Source: §A.4 · p. 14, Table 8 · p. 14

A.5 Number of Explicit Gaussians

The appendix reports fewer explicit Gaussians for HyRF, additional comparisons with recent methods and specular scenes, qualitative comparisons, and per-scene metrics across datasets.

A.5 Number of Explicit Gaussians

  • HyRF uses 24–45% fewer explicit Gaussians than 3DGS across MipNeRF360, Tanks&Temples, and DeepBlending, without additional pruning.The table compares the number of explicit Gaussians.Source: §A.5 · p. 14, Table 9 · p. 14

A.6 Additional Comparison with Recent 3DGS-based Methods

  • On DeepBlending, the additional comparison covers GOF, Spec-GS, Mini-Splatting2, and DashGaussian across rendering speed, training time, peak GPU memory, and model size.The comparison is described as an expanded evaluation of recent 3DGS-based methods.Source: §A.6 · p. 14, Table 10 · p. 14

A.7 Additional Comparison on Specular Scenes

  • On Spec-GS’s eight-scene anisotropic synthetic dataset, HyRF exceeds 3DGS by 1.58 dB PSNR while using 82% less memory.The passage attributes the improved performance to the benefits of MLPs over SH coefficients for modeling high-frequency view-dependent effects.Source: §A.7 · p. 14, Table 11 · p. 15

A.8 Additional Qualitative Comparisons

  • Figure 9 presents additional qualitative comparisons of HyRF with previous approaches on standard real-world datasets.The supplied figure caption identifies the visual as qualitative comparisons.Source: Figure 9 · p. 15, §A.8 · p. 15

A.9 Per-scene Metrics

Loading 2509.17083v2…