PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral World Renderer: Unified 3D Hyperspectral Unmixing

Chia-Ming Lee, Yu-Jou Hsiao, Ming-Ching Chang, Xin Li, Yu-Lun Liu, Chih-Chung Hsu

arXiv:2610.19520v1cs.CVcs.GR

TL;DR

RGB-based 3D systems recognize scene contents but have difficulty identifying materials, while existing 3D unmixing approaches face compositing, speed and ground-truth-data constraints. SWR combines executable worlds with known composition and unmixing during rendering. It identifies materials best across eight worlds and achieves the best mean PSNR on real captures, with faster training than unmixing baselines.

  • Problem

    3D material unmixing is difficult because spectra are mixed within pixels and along rays, illumination and similar materials make decomposition ambiguous, and multi-view data with known composition are lacking.

    Source: §1 · p. 2

  • Method

    SWR renders executable worlds with measured spectra and shared-sample material ground truth, then decodes each Gaussian into material abundances before compositing.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    SWR identifies materials best in all eight worlds, has the best mean PSNR on real captures, and trains 3.7–12.9× faster than unmixing baselines.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    SWR provides queryable per-Gaussian material abundances, including for same-coloured materials, without material labels during training.

    Source: §3 · p. 3, §5 · p. 8

Abstract

from arXiv · show

Cameras and the foundation models built on them recognize what a 3D scene contains, but not what it is made of: surfaces of the same colour can be different materials. Hyperspectral unmixing explains each spectrum as a mixture of material signatures, yet in 3D it is applied to rendered images after the views are composited, or tied to slow implicit fields, and no multi-view data with known composition exist to check the result. We present the Spectral World Renderer (SWR), which unifies world generation and unmixing. A coding agent builds executable 3D worlds that are rendered with measured spectra, so every hyperspectral image comes with exact labels and sub-pixel proportions; the resulting SWR-Bench provides eight indoor and outdoor worlds in which decompositions can be verified. SWR then unmixes inside rendering: each Gaussian of a compact 3D representation is decoded into material abundances before compositing, so one render yields spectra and material maps, and any surface can be queried by material name, a clicked pixel or a reference spectrum. Without material labels, SWR identifies materials best in all eight worlds, has the best mean PSNR on real captures, trains 3.7-12.9x faster than unmixing baselines, renders 34.9 labelled views/s and separates same-coloured materials.

1. Introduction

RGB-based 3D methods recognize objects but struggle to identify materials, especially when different materials share a colour or mix within a pixel. SWR pairs worlds with known composition and unmixes materials during rendering, addressing limitations in post-compositing and implicit-field approaches.

1. Introduction

  • RGB foundation models and material segmenters struggle to distinguish similarly coloured materials and assign one category per pixel despite within-pixel mixtures.A single object can also combine materials such as wood, laminate and metal.Source: §1 · p. 1
  • 3D unmixing is challenged by band-dependent storage, mixing both within pixels and along rays, and ambiguity from illumination and similar materials.Posthoc unmixing cannot undo mixing introduced by compositing; implicit-field unmixing is slow and lacks queryable primitives.Source: §1 · p. 2
  • SWR-Bench provides eight worlds with measured spectra, material labels and proportions, rendered from shared sub-pixel samples for verifiable decompositions.The SWR renderer decodes Gaussian features into abundances before compositing, producing spectra and queryable material maps in one render.Source: §1 · p. 2, Table 1 · p. 2
  • SWR identifies materials best in all eight worlds and has the best mean PSNR on real captures, while training 3.7–12.9× faster than unmixing baselines.It also provides real-time material queries and separates same-coloured materials.Source: §1 · p. 2

2. Related Work

Prior work addresses 3D reconstruction, spectral unmixing, and procedural world generation largely as separate capabilities. SWR combines measured spectra, inspectable geometry and sampled composition to support material identification and verifiable multi-view data.

Hyperspectral 3D reconstruction and spectral unmixing

  • Hyperspectral fields reconstruct spectra, while 2D unmixing estimates abundances and UnMix-NeRF brings unmixing into a radiance field; SWR uses explicit Gaussians and unmixes before compositing.Existing spectral methods model appearance or spectra, but do not also reveal materials.Source: §2 · pp. 2–3

Programmatic world generation and synthetic benchmarks

  • Figure 2 depicts SWR’s coding-agent world generation, in which shared rendering produces hyperspectral and RGB images alongside exact material ground truth.Checks on layout and rendering return flawed worlds for revision.Source: Figure 2 · p. 3
  • Procedural and generative world methods build scenes or render video, but do not model spectra or proportions; SWR binds measured spectra to inspectable geometry.SWR-Bench samples area proportions with each image, varies specimens and illumination, and stores coefficients for regeneration.Source: §2 · p. 3

3. Spectral World Renderer

SWR-Bench supplies generated worlds with composition known from their shared rendering samples, and SWR performs unmixing at each ray intersection before compositing. Its band-independent representation and multi-view appearance regularization support material decomposition without material labels.

3.1. World Generator and SWR-Bench

  • SWR-Bench uses executable worlds and shared camera samples to render aligned images, material labels, and area proportions, with measured spectra grounding the materials.The benchmark contains indoor and outdoor worlds with composition known by construction.Source: §3.1 · pp. 3–4

3.2. Unmixing at Every Ray Intersection

  • At each ray intersection, a shared decoder maps band-independent Gaussian features to abundances and intensity before compositing, so one render yields spectra and abundances.The same visibility weights render both outputs, and the abundances mix a scene-wide endmember dictionary.Source: §3.2 · pp. 4–5

3.3. Multi-View Appearance Regularization

  • Multi-view appearance clusters initialize endmembers and weakly regularize pixel assignments, helping prevent similar materials from merging without using material labels.Clusters are derived from spectral shape and brightness, and their assignments are lifted across views to constrain which endmember explains each pixel.Source: §3.3 · pp. 5–6, Figure 5 · p. 5

3. Spectral World Renderer

  • SWR has the highest mIoU in every SWR-Bench world, the best or second-best PSNR and SAM, and endmembers closest to the true spectra.The comparison is made without material labels.Source: Table 2 · p. 6

3.4. Optimization

  • Optimization first fits and refines the RGB anchor on RGB images, then freezes geometry while updating features, the decoder, and endmembers on hyperspectral data.The second-stage objective also includes spectral-angle and decaying assignment-regularization terms.Source: §3.4 · p. 6

4. Experiments

Across eight known-composition worlds, experiments assess material identification, spectral fidelity, efficiency, and material queries. SWR unmixes before compositing, and its performance shows why spectral fidelity alone does not establish accurate material labels.

4.1. Material Identification and Abundances

  • Across all eight worlds, SWR recovers material layers closest to ground truth, while posthoc 3DGS+VCA and UnMix-NeRF spread materials across unrelated surfaces.On Market and Study, 3DGS has lower SAM than SWR but far lower mIoU; NNLS with true spectra reaches 0.84–0.94 mIoU on clean radiance and 0.32–0.77 under noise.Source: §4.1 · p. 6, Figure 6 · p. 7

4.2. Ablation and Analysis

  • 36.5 s of the 397 s City pipeline goes to fixed-anchor refinement, which improves Market by 2.9 dB and 0.12 mIoU and the real apple by 2.6 dB.One feature per Gaussian lowers Market mIoU from 0.47 to 0.34; fixed intensity lowers it to 0.22, and decoding after compositing raises PSNR from 33.72 to 34.37 dB but lowers mIoU from 0.47 to 0.37.Source: §4.2 · p. 7

4.3. Real Captures and Efficiency

  • SWR has the highest PSNR on every real dataset, while 3DGS has lower mean and p95 spectral angles than SWR on HS-NeRF.On the 204-band HSI-SC apple, PSNR, mean SAM, and p95 all improve over HyperGS.Source: §4.3 · p. 7, Table 5 · p. 8, Figure 8 · p. 8
  • 6 min 37 s trains City with SWR, versus 24 min for UnMix-NeRF and 85 min for 3DGS+VCA/NNLS; its fused decoder renders 34.9 labelled views/s.Table 4 reports views/s with spectra and label maps.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Application: Material Queries

  • SWR separates foliage-coloured mineral leaves from ordinary trees and changed crowns from healthy foliage, while RGB queries mix them.The mineral leaves are 2° apart from foliage in RGB but 42° apart in spectrum; SWR’s mineral-leaf query retrieves no ordinary tree, whereas the RGB query retrieves eleven.Source: §4.4 · p. 8, Figure 9 · p. 8
  • A shadow-click query retrieves 77% of the sunlit part of its material, against 10% for RGB, averaged over four outdoor worlds.The query takes 3 ms.Source: §4.4 · p. 8, Figure 9 · p. 8

5. Conclusion

SWR pairs known-composition world generation with unmixing inside rendering. It identifies materials best across eight worlds, achieves the best mean PSNR on real captures, and supports fast, label-free material queries.

5. Conclusion

  • SWR unifies world generation and 3D unmixing in a verifiable framework, with exact material composition and per-Gaussian abundances that support material queries.The renderer provides spectra and material maps in one render; queries can use a material name, clicked pixel, reference spectrum, or band.Source: §5 · p. 8

Supplementary Material

SWR-Bench provides controlled indoor and outdoor worlds, with shared-sample images, labels, and material proportions. Its protocol documents noise, evaluation, spectral sourcing, and scope boundaries.

A. SWR-Bench Worlds and Protocol

  • Eight SWR-Bench worlds use shared 16 sub-pixel samples per pixel for spectra, RGB, and ground truth, with separate stored contributions for transmitted and interreflected light.Outdoor worlds use ASTM G173 sunlight and skylight; the Study and Office are path traced, and all worlds use 64 bands over 420–980 nm.Source: A. SWR-Bench Worlds and Protocol · p. 14
  • Noise is released as clean data and as Poisson shot plus Gaussian read noise scaled from training views, and all methods share splits, classes, and the stereo point cloud.PSNR and spectral angle are measured against clean radiance on ground-truth support; the true-spectrum reference is undefined indoors and for NVIDIA HQ.Source: A. SWR-Bench Worlds and Protocol · p. 14

B. Spectra of SWR-Bench

  • 37 of 40 natural-material types with enough references lie within 3° of independent library measurements, while all 21 vegetation spectra pass red-edge and NDVI checks.Proxies include pigment powders for paints, cotton for paper, and granite for an inkstone; glazing is opaque, most outdoor transport lacks spectral interreflection, and bands end at 980 nm.Source: B. Spectra of SWR-Bench · p. 14

C. Implementation Details

Implementation details cover the RGB anchor, feature refinement and decoder training, and validation-based endmember selection. The supplementary tables describe benchmark views and additional evaluation metrics.

C. Implementation Details

  • The RGB anchor uses multi-view density control, then discards colours and folds screen-space variance into Gaussian scales.Harmonic features are refined for 5k steps without further densification; the decoder trains for 30k Adam steps, with endmembers frozen for 3k updates before learning.Source: C. Implementation Details · pp. 14–15
  • Table S1 reports training, validation, and test views, noise levels, and noisy-test PSNR against clean views; Table S2 covers test metrics at K equal to the class count.Table S2 includes training PSNR, p95 angle, pixel accuracy, clean-radiance mIoU, and true-dictionary NNLS on noisy and clean radiance.Source: Table S1 · p. 15, Table S2 · p. 15
  • Dictionary size K is selected from the class count, 16, 24, and 32 using validation dominant-endmember mIoU; the validation choice is also the best test candidate in seven worlds.At K=24, 3DGS+VCA/NNLS remains below SWR in every world except Study, where they reach 0.51–0.52.Source: C. Implementation Details · p. 15

D. Decoder Analysis

The decoder analysis separates spectral fidelity from material identification and examines how illumination, Gaussian features, and primitives represent materials. The real-capture supplement also describes data filtering and the scope of spectral-reference checks.

D. Decoder Analysis

  • Replacing unmixing with a free spectral decoder gains at most 0.07 dB, while one feature per Gaussian costs 0.4 to 1.6 dB.The reported controls indicate that fidelity comes from the feature field rather than relaxing the mixing model.Source: D. Decoder Analysis · p. 15
  • SWR-Bench class names describe assigned material roles that may use measured proxies, and some proxies lack independent spectral references.Reference-check counts concern source measurements, not surveyed historical materials.Source: Table S3 · p. 16, Table S4 · p. 17
  • Within a Gaussian, intensity varies with shading while mixture coefficients change mainly at material boundaries, although some mixture variation also follows shadows.This lets a primitive span an edge; Figure S1 compares per-hit features and within-Gaussian mixture and intensity variation against mean features.Source: D. Decoder Analysis · p. 16, Figure S1 · p. 16
  • Sun and shadow explain 84% and 67% of within-class intensity variance on Market and City, while dominating endmembers for one Market class overlap by 5%.On Office, illumination explains 56% of within-class class-mass variance; Lambertian-world abundances vary by at most 0.04 TV across 45° views, versus 0.18 for NVIDIA HQ.Source: D. Decoder Analysis · p. 16, Figure S2 · p. 17
  • Tetrahedra for one surface share a decoded class, and rendered-map class boundaries coincide with primitive boundaries, supporting material queries over individual Gaussians.Figure S3 reports that the median visible Gaussian places more than 98% of its centre abundance on one class.Source: D. Decoder Analysis · p. 16

E. Real Captures

  • For HS-NeRF real captures, only uniquely aligned views are kept; errors concentrate at edges and thin parts.Fakeplant2 and Pinecone are excluded because their processed RGB has a white background absent from the raw cubes.Source: E. Real Captures · p. 17

F. Efficiency

SWR keeps storage independent of spectral band count and reports fast rendering of spectra and labels using a fused kernel.

F. Efficiency

  • 59 values per Gaussian make SWR models 6 to 32 times smaller on real captures, including 4.2 MB versus 127 MB on the 204-band apple.The comparison is with 78 values for HyperGS and 62+L for per-band fields.Source: F. Efficiency · p. 17
  • 34.9 labelled views per second are rendered by the fused fp16 kernel, about ten times 3DGS with VCA.The City pipeline takes 397 s; changing K repeats only clustering and decoder tuning because the anchor and features do not depend on K.Source: F. Efficiency · p. 17

G. Rare Materials and Altered Leaves

SWR separates altered leaves from healthy foliage better than RGB clustering, but changed leaves are generally assigned to another class rather than identified as their own. Rare-class errors are associated with endmember allocation.

G. Rare Materials and Altered Leaves

  • Class IoU falls with class frequency, while true spectra still separate most rare classes, pointing to endmember allocation rather than the data as the error source.Figure S7 plots per-class IoU against pixel share across the eight worlds and shows NNLS with true class spectra separating most rare classes that SWR merges.Source: G. Rare Materials and Altered Leaves · p. 17
  • Altered foliage has label-distribution total variation of .85 and .83 with SWR, versus .10 and .40 for RGB k-means and .95 and .97 for true spectra.Despite this separation, most changed leaves map to another class rather than a class of their own.Source: G. Rare Materials and Altered Leaves · p. 17

H. Object Queries

The object-query procedure builds tree instances from crown-material Gaussians and then compares them with a spectral query. In the City variants, spectral querying avoids wrong-tree retrievals for metamer leaves, while some targets are lost before querying.

H. Object Queries

  • Tree instances are formed by clustering above-ground crown-material Gaussians with DBSCAN, with large clusters treated as instances.A measured leaf spectrum then selects instances by comparing each crown with the query and with ordinary leaves under sunlight and skylight.Source: H. Object Queries · pp. 18–19
  • For the substituted mineral, 42° spectral separation from ordinary leaves contrasts with only 2° in RGB; SWR returns no wrong trees, while RGB retrieves wrong trees.The table notes that targets may be lost before the spectral query or found, and that SWR retrieves no ordinary tree in the metamer variant.Source: H. Object Queries · p. 19, Table S11 · p. 22
  • Most missed targets are lost earlier, chiefly because crowns are too sparse to form instances, rather than at spectral matching.Parameters were tuned on the base City.Source: H. Object Queries · p. 19

I. Query by Spectrum

Spectral queries compensate for illumination when matching clicked or library spectra to Gaussian primitives, enabling material retrieval across views. The section evaluates query accuracy, metamer separation, threshold selection, speed, and supplementary SWR-Bench views.

I. Query by Spectrum

  • Shadowed-click spectral queries raise AP from .45 to .61 and sunlit-part recall from 10% to 77%, while class-name queries reach .82 AP.The AP values are means over Market, City, Botanical and Karst; the class-name query remains more accurate when the material name is known.Source: Table S12 · p. 22, §I · pp. 22–23, Figure S9 · p. 23
  • Library spectra use an illumination cone, while clicked spectra use whitening to reduce shading effects when matching Gaussian spectra.The cone spans sunlit, penumbral and shadowed mixtures; whitening projects out two principal illumination directions that account for 99.999% of variation.Source: §I · p. 22
  • Otsu thresholding yields IoU about .14 below each query’s best threshold and is unreliable for rare materials, so a slider sets the final selection.A single spectrum also cannot represent classes assembled from differently coloured specimens, such as vehicles and painted facades.Source: §I · p. 23
  • For foliage-coloured mineral leaves 2° apart in RGB but 42° in spectrum, living-leaf queries achieve AUC .99 versus .26 for RGB and retrieve 2% versus 99% of mineral-leaf pixels.The mineral spectrum retrieves 92% of mineral-leaf pixels and 2% of living-leaf pixels.Source: §I · p. 23, Figure S8 · p. 23
  • With 61k–84k Gaussians, queries take 1.6–2.9 ms, or 3–4 ms including rasterization at 960 × 540.Source: §I · p. 23

J. Additional Views of SWR-Bench

  • Additional test-view panels show three separated cameras per world, with rendered input, material ground truth and SWR predictions.Views are selected by camera separation from a fixed reference, not by prediction scores.Source: J. Additional Views of SWR-Bench · p. 24, Figure S15 · p. 30, Figure S16 · p. 31
  • Simulated hyperspectral views show outdoor spectra reproducing library reflectance, while indoor path-traced spectra deviate through interreflection and residual Monte Carlo variance.The Office’s near-infrared bands come only from daylight through blinds because its lights emit no near-infrared.Source: Figure S10 · p. 25, Figure S11 · p. 26
  • Released RGB renders show 15 cameras along each world’s trajectory, spanning Forbidden City, Market, Botanical, NVIDIA HQ, Study, Karst, City and Office.Source: Figure S12 · p. 27, Figure S13 · p. 28, Figure S14 · p. 29

Related papers

Loading 2610.19520v1…