PaperlayerView exact revision on arXiv

Source-linked AI summary

BRF-GS: Hyperspectral Bidirectional Reflectance Factor Modeling and Image Generation Based on 3D Gaussian Splatting

Yiling Yao, Wenjuan Zhang, Bowen Wang, Bocheng Li, Wentao Song, Bing Zhang

arXiv:2608.31159v1cs.CV

TL;DR

Existing BRF simulation requires complex scene construction and costly radiative-transfer solvers, while standard 3DGS struggles with complex directional reflectance and hyperspectral dimensionality. BRF-GS combines geometry-reliable band selection, a hybrid BRDF-driven kernel, and two-stage geometry–spectral training, and experiments report superior spatial and spectral fidelity over existing methods.

  • Problem

    BRF modeling lacks an efficient way to generate multi-angle hyperspectral reflectance imagery while representing complex directional reflectance and preserving hyperspectral fidelity.

  • Method

    BRF-GS uses geometry-reliable spectral-band selection, a hybrid BRDF-driven kernel, and geometry–spectral decoupled two-stage training within 3DGS.

  • Results

    BRF-GS significantly outperforms existing state-of-the-art methods in image-generation visual quality and spectral fidelity.

  • Takeaways & Limitations

    BRF-GS provides an efficient framework for hyperspectral BRF modeling and directional-reflectance image generation in remote-sensing scenes.

  • Takeaways & Limitations

    A single microfacet-based BRDF model cannot adequately capture complex scene-level directional reflectance in remote-sensing scenes.

Abstract

from arXiv · show

The bidirectional reflectance factor (BRF) characterizes the directional radiative properties of terrestrial surfaces. However, existing three-dimensional (3D) radiative transfer models require complex scene construction and computationally intensive radiative transfer solvers, limiting efficient generation of multi-angle hyperspectral reflectance imagery. 3D Gaussian Splatting (3DGS) offers an efficient framework for neural scene representation and novel view synthesis, but its low-order spherical harmonics representation is insufficient for complex directional reflectance, while the high dimensionality and inter-band quality differences of hyperspectral data introduce additional challenges. To address these challenges, we propose BRF-GS, a 3DGS-based framework for BRF modeling and hyperspectral reflectance image generation. BRF-GS introduces a hybrid BRDF-driven kernel to represent complex directional reflectance, selects geometry-reliable spectral bands for robust 3D scene initialization, and adopts a two-stage training strategy that decouples geometry optimization from spectral modeling. We further construct the AIR-BRF dataset, a multi-angle hyperspectral directional reflectance dataset comprising three scenes with diverse natural and artificial targets. Experiments demonstrate that BRF-GS achieves superior spatial and spectral fidelity and accurately reproduces characteristic view-dependent BRF responses. The proposed framework provides an efficient data-driven approach for BRF modeling and multi-angle hyperspectral reflectance image generation in remote sensing scenes.

1. INTRODUCTION

Remote-sensing BRF modeling needs efficient reconstruction of continuous, wavelength-dependent directional reflectance from sparse multi-angle hyperspectral observations. BRF-GS addresses these challenges with 3DGS-based geometry initialization, BRDF-driven appearance modeling, decoupled training, and the AIR-BRF benchmark.

  • Motivation: Sparse angular sampling limits characterization of continuous directional reflectance, especially for structurally complex targets such as forest canopies and urban buildings.Directional reflectance can vary nonlinearly with viewing geometry.
  • Motivation: 3D radiative transfer models provide physically meaningful BRF simulation but require complex scene construction, parameter specification, and computationally costly radiative-transfer solutions.These requirements can involve substantial prior knowledge and manual intervention.
  • Motivation: 3DGS learns scene representations directly from multi-view observations and uses differentiable, rasterization-based rendering, avoiding manually constructed scenes and iterative radiative-transfer solutions.The original 3DGS reports rendering rates exceeding 100 frames per second at 1080p resolution.
  • Motivation: Original 3DGS remains unsuitable for hyperspectral BRF because low-order spherical harmonics poorly represent strongly directional nonlinear responses, while hundreds of bands increase parameters and computational burden.Hyperspectral reconstruction must preserve spectral shape and magnitude across the full spectral range.
  • Contributions: BRF-GS introduces geometry-reliable band selection, a hybrid BRDF-driven kernel, and geometry–spectral decoupled two-stage training for hyperspectral BRF modeling.The framework is designed for hyperspectral directional-reflectance image generation in remote-sensing scenes.
  • Contributions: AIR-BRF is a multi-angle hyperspectral BRF dataset covering three remote-sensing scenes with diverse natural land-cover types and artificial targets.It supports benchmarking BRF image generation and systematic evaluation of hyperspectral 3DGS methods.

2. RELATED WORK

Prior 3DGS extensions improve directional appearance or hyperspectral representation, but existing methods remain limited in remote-sensing scale, physical reflectance modeling, and spectral fidelity. BRF-GS is motivated by these gaps while addressing the trade-offs identified across related approaches.

  • 3DGS foundations: Original 3DGS represents view-dependent color with spherical harmonics, whose coefficient count grows quadratically with order and whose low orders miss high-frequency directional variation.Extending this representation to hundreds of hyperspectral channels creates substantial computational and memory overhead.
  • Directional reflectance modeling: GaussianShader replaces SH appearance with diffuse, specular, and residual physically based shading, improving reconstruction for scenes containing reflective surfaces.Its representation also encodes local surface orientation through learnable surface normals.
  • Directional reflectance modeling: Spec-Gaussian uses anisotropic spherical Gaussians to capture narrow-lobe high-frequency anisotropic specular-reflection details.Other related methods use deferred shading, illumination factorization, roughness-aware rendering, or view-consistent physical attributes.
  • Remote-sensing limitations: These directional-reflectance methods mainly target close-range RGB images of individual objects and rely on microfacet BRDF models.Such models are insufficient for heterogeneous remote-sensing scenes involving component mixing, occlusion, shadowing, and multiple scattering.
  • Hyperspectral extensions: Existing image-domain approaches reconstruct view-dependent appearance but do not explicitly model wavelength-dependent spectral characteristics of directional reflectance.This limits their direct applicability to hyperspectral remote-sensing scenes.
  • Hyperspectral extensions: SpectralGaussians replaces RGB attributes with multispectral representations and introduces spectral shading, but its applicability to complex remote-sensing scenes remains insufficiently investigated.Its spectral channels are modeled independently, and evaluations emphasized small-scale objects and synthetic multispectral datasets.
  • Hyperspectral extensions: HyperGS compresses hyperspectral information into a latent space and reconstructs full spectra with spectral decoding, while using grayscale geometry initialization and depth-aware densification.Latent compression may discard subtle spectral variations, narrow absorption features, and high-frequency spectral details.

3. METHOD

BRF-GS uses geometry-reliable spectral bands for initialization and a geometry–reflectance decoupled optimization strategy. It represents each Gaussian with hybrid BRDF-driven components for arbitrary-view hyperspectral reflectance rendering and analysis.

  • The optimized Gaussian representation supports arbitrary-view rendering and directional reflectance analysis, including BRF and BRDF modeling.
  • BRF-GS takes multi-angle hyperspectral reflectance observations as input and generates hyperspectral images, directional reflectance responses, depth maps, and normal maps.
  • The framework comprises geometry-reliable spectral-band selection, hybrid BRDF-driven Gaussian kernels, and geometry–reflectance decoupled two-stage training.
  • Geometry-reliable spectral-band selection: Band suitability varies with sensor response, signal quality, and spatial feature quality, making exhaustive geometric evaluation costly for hyperspectral multi-view data.
  • Geometry-reliable spectral-band selection: In the illustrated example, the band near 800 nm provides stable spatial features, whereas 370 nm and 690 nm lack detectable features and 1000 nm is noise-prone.
  • Geometry-reliable spectral-band selection: The method ranks bands by equivalent feature count and selects the three bands with the highest geometric reliability for feature matching, pose estimation, and sparse 3D initialization.
  • Hybrid BRDF-driven kernel representation: The hybrid kernel replaces conventional low-order spherical harmonics with isotropic diffuse, volumetric, geometric-optical, and microfacet specular components.
  • Hybrid BRDF-driven kernel representation: Each Gaussian jointly encodes spatial geometry, opacity, compressed spectral features, and hybrid BRDF parameters, whose decoded spectrum models wavelength-dependent directional reflectance.

Volumetric scattering component

The volumetric scattering component models directional reflection using the RossThick kernel and the phase angle between illumination and observation directions.

  • The volumetric scattering contribution is modeled with the RossThick kernel.
  • The phase angle ξ and relative azimuth angle ϕ describe the illumination–observation geometry used by the directional scattering model.

Geometric-optical component

The geometric-optical component uses the Li-Sparse kernel to represent directional reflection geometry.

  • The geometric-optical scattering component adopts the Li-Sparse kernel.

Microfacet specular component

The microfacet specular component models smooth and metallic-surface reflection with Fresnel-based and Cook–Torrance formulations.

  • Smooth-surface reflection is modeled with a Fresnel term approximated by the Schlick model using fixed normal-incidence reflectance F0=0.04.
  • Metallic-object specular reflection uses the Cook–Torrance microfacet BRDF with the GGX normal distribution function.
  • The half-vector, Gaussian-derived surface normal, and learnable roughness parameter define the microfacet specular formulation.
  • Material-dependent spectral variation in specular reflection is implicitly captured by the learned latent specular component z_k.

Strategy

BRF-GS uses a geometry–reflectance decoupled two-stage strategy: reliable spectral bands first stabilize geometry, then all hyperspectral bands model directional reflectance.

  • Two-stage optimization: BRF-GS separates geometry reconstruction from directional spectral reflectance learning to reduce ambiguity between geometric and reflectance parameters.The ambiguity arises because geometry is wavelength-independent, whereas directional reflectance varies with wavelength and viewing direction.
  • Two-stage optimization: Stage one jointly optimizes Gaussian geometric parameters and reflectance coefficients using geometry-reliable spectral bands and grayscale structural similarity.Its objective combines pixel-wise reflectance loss with structural similarity loss, balanced by α.
  • Two-stage optimization: After geometry stabilizes, stage two fixes geometry and spectral feature vectors while optimizing bidirectional reflectance functions across all hyperspectral bands.This stage reconstructs continuous spectral reflectance responses under different viewing directions.
  • Two-stage optimization: The second-stage objective combines reflectance reconstruction and spectral cosine similarity losses to constrain absolute accuracy and spectral-shape consistency.The reconstruction compares ground-truth and reconstructed hyperspectral reflectance vectors over optimized pixels.

4. EXPERIMENTS AND ANALYSIS

Experiments evaluate BRF-GS on AIR-BRF’s three diverse remote-sensing scenes and show stronger directional-reflectance, spatial, textural, and spectral reconstruction than comparison methods.

  • Dataset and acquisition: AIR-BRF contains three representative Hebei scenes spanning natural and artificial surfaces with diverse directional reflectance characteristics.The scenes are the HuaiLai Remote Sensing Test Site, SaiHanBa Mechanical Forest Farm, and GuanTing Reservoir.
  • Dataset and acquisition: Multi-angle hyperspectral observations were acquired with a DJI M300 RTK UAV equipped with a Cubert Ultris X20P imaging system.The dataset records multiple viewing directions for evaluating directional reflectance across varied land-cover types.
  • Qualitative comparison: BRF-GS captures water and vegetation directional responses while reducing artifacts caused by specular effects, canopy structure, volumetric scattering, and shadowing.It reproduces water glints, complex canopy reflectance, and fine structures such as field ridges, roads, and water ripples.
  • Quantitative comparison: BRF-GS consistently achieves the best performance across all three scenes and evaluated metrics, indicating stronger reconstruction fidelity and perceptual consistency.The reported gains cover directional reflectance, geometric structures, and fine-scale textures.
  • Quantitative comparison: SAM values are 0.112, 0.119, and 0.128 for BRF-GS versus 0.278, 0.291, and 0.304 for MS-Splatting across the three scenes.The corresponding spectral angular-error reductions are approximately 59.7%, 59.1%, and 57.9%.
  • Spectral reconstruction: BRF-GS reconstructs continuous directional reflectance distributions and preserves characteristic spectral trends, absorption features, specular peaks, and anisotropic responses.The reconstructed curves closely match ground truth for natural and artificial targets, including water, forest, grass, soil, roofs, and vehicles.

Ablation of Geometry-reliable Band Selection

The ablation study shows that geometry reliability, rather than band count, SNR, or feature quantity alone, is central to robust hyperspectral BRF reconstruction.

  • Experimental design: The study compares eight spectral-band selection strategies for geometric initialization on the Forest Farm scene using PSNR, SSIM, and SAM.Subsequent Gaussian optimization and rendering settings remain identical across configurations.
  • Findings: Adding more spectral bands does not necessarily improve reconstruction because noisy or low-information bands can weaken features and cross-view correspondence.A single informative band can provide a comparable number of effective features to a three-band combination.
  • Findings: Highest SNR and largest feature count do not reliably identify the best geometric bands or reconstruction performance.The results show that SNR and feature quantity alone are insufficient measures of geometric utility.
  • Findings: The central 675 nm band produces substantially fewer features and degraded reconstruction because it provides insufficient spatial information.This contrasts with the limited benefit observed when increasing the number of informative bands.
  • Conclusion: Overall, the ablations support selecting initialization bands by geometric reliability rather than maximizing band count, SNR, or detected features.The criterion targets the availability and consistency of local geometric features across views before SfM reconstruction.

Ablation of Directional Reflectance Kernels

Kernel ablations show that BRF-GS’s physically motivated BRDF components contribute complementarily, outperforming conventional spherical harmonics for complex directional-reflectance modeling.

  • Experimental design: The ablation compares four physically motivated kernels, their removals, the full BRF-GS model, and a conventional spherical-harmonics baseline using PSNR and SAM.Only the directional-reflectance representation changes across variants; other training and modeling settings remain fixed.
  • Ablation results: Removing any individual kernel consistently degrades reconstruction, while the full BRF-GS model achieves the highest PSNR and lowest SAM.The result indicates complementary contributions among the directional-reflectance components.
  • Ablation results: Removing the isotropic component causes the largest performance drop, indicating its central role in modeling the overall reflectance response.Volumetric-scattering and geometric-optical components also provide substantial improvements, whereas removing the specular component causes a smaller degradation.
  • Baseline comparison: BRF-GS substantially outperforms the spherical-harmonics baseline by leveraging BRDF kernels derived from empirical and semi-empirical land-surface reflectance knowledge.The comparison tests whether physically motivated kernels represent complex directional reflectance more effectively than generic basis functions.

Ablation of Geometry–Reflectance Decoupled Training

The ablation compares geometry–reflectance decoupled training with joint optimization and evaluates BRF-GS against a ray-tracing radiative transfer method. Decoupling improves spectral fidelity while preserving substantially faster hyperspectral rendering.

  • The ablation compares the proposed two-stage training strategy with joint optimization under otherwise identical settings.
  • Decoupled training improves PSNR from 24.07 to 24.44 dB relative to joint optimization.
  • SAM decreases from 0.188 to 0.120 with decoupled training, indicating a substantially larger spectral-fidelity improvement than the PSNR change.
  • BRF-GS first establishes scene geometry and then optimizes spectral directional reflectance with geometry fixed, reducing geometry–reflectance coupling.
  • The ray-tracing radiative transfer baseline uses a manually constructed 3D model with 1.4 million facets, 12 material classes, and 2,000 rays per pixel.
  • BRF-GS renders a single 164-band BRF image in 42 ms, compared with 235 s for the ray-tracing method.

5. CONCLUSIONS AND FUTURE WORK

The conclusions present BRF-GS and AIR-BRF as a framework and dataset for scene-level hyperspectral directional-reflectance modeling. Results show strong image and spectral fidelity, while validation remains limited across land-cover, seasonal, and atmospheric conditions.

  • Conclusions: BRF-GS reconstructs and generates hyperspectral BRF imagery using 3D Gaussian splatting, alongside the AIR-BRF multi-angle dataset.
  • Conclusions: BRF-GS significantly outperforms existing state-of-the-art methods in image-generation visual quality and spectral fidelity.
  • Conclusions: The hybrid BRDF-driven directional-reflectance kernel characterizes diverse BRF variation patterns within a scene-level 3D representation.
  • Conclusions: The geometry–reflectance decoupled two-stage strategy minimizes spectral distortion, with superior performance reported for PSNR, SSIM, and especially SAM.
  • Conclusions: BRF-GS captures angular responses including vegetation hotspot effects and water-surface specular reflection.
  • Conclusions: AIR-BRF supports hyperspectral directional-reflectance modeling, algorithm evaluation, multi-angle data generation, and subsequent remote-sensing analyses.
  • Future Work: The method models calibrated apparent reflectance without explicitly correcting atmospheric absorption and scattering along different viewing paths.
  • Future Work: The dataset excludes distinctive directional-reflectance surfaces such as deserts and snow/ice because of limited spatial and seasonal coverage.
Loading 2608.31159v1…