PaperlayerView exact revision on arXiv

Source-linked AI summary

Luce: Relightable Gaussians for 3D Asset Generation

Mayank Singh, Michele Stoppa, Alvise Memo, Rui Yu, Harsha Kalli, Srimanth Gunturi, Muhammad Ahmed Riaz, Behrooz Shahsavari, Waleed Abdulla, David E. Jacobs

arXiv:2608.23943v1cs.CVcs.AIcs.GR

TL;DR

High-fidelity image-to-3D generation needs a representation that jointly captures geometry and PBR materials for relighting and standard rendering. Luce uses multimodal Gaussian primitives, a unified latent, and image-conditioned rectified flow to generate relightable assets, achieving strong benchmark results while preserving fine details. Its finite-resolution voxelization limits detail that spans only a few voxels and excludes several complex appearance effects.

  • Problem

    High-fidelity image-to-3D generation lacks a representation that jointly captures fine geometry and physically based materials for relighting and standard rendering.

  • Method

    Luce uses dedicated Gaussian sets for PBR modalities, compresses them into a unified latent, and generates that latent from a single image with rectified flow and multi-layer image features.

  • Results

    Luce achieves state-of-the-art generation quality, including 20.99 FID on Toys4K and higher AI-generated-image CLIP alignment than the best baseline, 0.8519 versus 0.8299.

  • Takeaways & Limitations

    Luce produces relightable PBR Gaussians and textured meshes while preserving fine details such as text, logos, inscriptions, and high-frequency surface detail.

  • Takeaways & Limitations

    Finite-resolution voxelization can underresolve details spanning only a few voxels, and the material model does not explicitly represent several complex appearance effects.

Abstract

from arXiv · show

High-fidelity image-to-3D generation requires a 3D representation that captures both geometry and appearance. To support relighting and integration into standard rendering pipelines, the representation should include physically based rendering (PBR) modalities such as albedo, metallic-roughness, and surface normals. We propose Luce, a 3D representation that unifies geometry and PBR materials within a voxelized multimodal Gaussian cloud, using dedicated Gaussian primitives for each modality. A variational autoencoder compresses this representation into a unified material-aware latent space. A rectified-flow transformer generates this latent from a single image, conditioned on multi-layer features from a pretrained image encoder that preserve both semantic context and fine spatial detail. The latent then decodes into relightable PBR Gaussians and an optional textured mesh with a tangent-space normal map. On Toys4K, Luce achieves state-of-the-art single-image-to-3D generation, improving FID by 28% over the strongest baseline. We further introduce a benchmark of AI-generated images, on which Luce improves the CLIP image-alignment score over the best baseline (0.8519 vs. 0.8299). Luce generates relightable, geometrically accurate, and materially faithful assets that preserve fine details such as text, logos, and inscriptions.

1 INTRODUCTION

Luce addresses the need for a 3D representation that jointly captures fine geometry and physically based materials for standard rendering and relighting. Its multimodal Gaussian representation, unified latent generation, multi-layer image conditioning, and tangent-space normal transfer target high-fidelity, detail-preserving assets.

  • Representation: Luce represents each surface-intersecting voxel with dedicated Gaussian sets for albedo, metallic-roughness, and surface normals.This unifies fine geometry and PBR materials in a relightable, 3D-native format compatible with standard rendering pipelines.
  • Evaluation: 20.99 FID is Luce’s Toys4K result, improving on TRELLIS 2 (29.22) and LiTo (29.76) by more than 8 FID.The reported result is the lowest FID among the compared systems.
  • Latent generation: A compact unified latent encodes geometry and all material modalities, enabling rectified-flow generation of complete relightable assets from one image.The latent decodes directly into PBR Gaussians and optionally into textured meshes with baked PBR texture maps.
  • Image conditioning: Multi-layer DINOv2 features combine shallow spatial detail with deep semantic information for high-fidelity 3D generation.The conditioning preserves logos, labels, and inscriptions that are critical to reproducing fine asset details.
  • Detail transfer: Tangent-space normal maps transfer authored fine surface detail, including engravings, fabric weave, and embossed text, onto extracted meshes without increasing polygon count.Luce decodes normal Gaussians and bakes them into mesh normal maps at inference.

2 RELATED WORK

Prior 3D generation methods span implicit fields and explicit primitives, while relightable methods must separate appearance into material properties. Luce’s overview combines multimodal Gaussian splats, structured latent compression, image-conditioned generation, and dual decoding.

  • 3D representations: Implicit 3D representations can model arbitrary-genus surfaces but generally target closed, watertight geometry, whereas newer Gaussian methods render appearance explicitly.The related-work discussion contrasts implicit level sets and sparse fields with 3D Gaussian Splatting.
  • 3D generation: Latent-space 3D generation increasingly uses diffusion transformers and flow matching, including structure-then-latent systems such as TRELLIS and TRELLIS 2.Other related systems condition on image features or generate implicit-function and primitive-based latents.
  • PBR and relighting: Relighting requires decomposing appearance into material properties, but prior per-scene inverse-rendering methods do not generalize to novel objects.Luce instead predicts normals as a separate modality, allowing detail finer than the geometry.
  • Luce overview: Figure 3 organizes Luce into representation and generation stages, linking per-modality Gaussian splats and SLatVAE compression to single-image SLatFlow generation.The output decodes into relightable PBR Gaussians and optionally textured meshes with tangent-space normal maps.

3 METHOD

Luce represents geometry and PBR materials as a multimodal Gaussian cloud, compresses them into a compact latent, and generates relightable assets from a single image. Multi-layer image conditioning preserves fine spatial detail, while dual decoding supports direct Gaussian rendering or textured meshes with transferred normal detail.

  • 3.1 MULTIMODAL PBR GAUSSIAN REPRESENTATION: Luce assigns dedicated Gaussian sets to albedo, metallic-roughness, and surface normals within each occupied voxel.The modalities have independent Gaussian layouts, allowing each to concentrate primitives where its own detail is richest.
  • 3.1 MULTIMODAL PBR GAUSSIAN REPRESENTATION: The representation renders PBR buffers by independently splatting modalities and applying deferred Cook–Torrance image-based lighting.The renderer produces albedo, metallic, roughness, normal, and opacity buffers without mesh extraction or UV unwrapping.
  • 3.2 VARIATIONAL AUTOENCODER: SLatVAE compresses the high-dimensional Gaussian cloud into a compact per-voxel latent by trading spatial resolution for increased Gaussian density during decoding.The decoder reconstructs fine material and geometric detail from the coarser latent resolution.
  • 3.3 FLOW-BASED GENERATION: Rectified-flow generation first predicts the sparse voxel structure, then generates joint geometry-and-material latent features at occupied voxels.SLatFlow is a sparse DiT-style transformer whose image features enter through cross-attention.
  • 3.3 FLOW-BASED GENERATION: Multi-layer DINOv2 conditioning combines shallow spatial detail with deep semantics, improving Toys4K FID from 25.21 to 20.99 and AI-image CLIP from 0.8081 to 0.8519.The conditioning uses layers 6, 12, 18, and 24 rather than layer 24 alone, preserving details such as text and logos.
  • 3.4 TANGENT-SPACE NORMAL MAP TRANSFER: The decoded latent supports direct relightable PBR Gaussian rendering or optional textured-mesh extraction with tangent-space normal maps.Normal-map transfer restores subvoxel surface detail without increasing polygon count.

4 EXPERIMENTS

Luce is evaluated for generation and reconstruction on Toys4K and an AI-generated-image benchmark, with comparisons spanning image alignment, PBR modalities, and qualitative detail preservation.

  • 4.1 IMPLEMENTATION DETAILS: The evaluation uses Toys4K for reconstruction and generation, plus an AI-generated-image benchmark, comparing Luce with TRELLIS, TRELLIS 2, LiTo, and 3DTopia-XL.Metrics include FID, KID, CLIP, SigLIP2, ULIP, Uni3D-L, PSNR, SSIM, and LPIPS.
  • 4.1 IMPLEMENTATION DETAILS: Luce preserves legible text and fine surface details while providing per-modality decompositions of albedo, metallic-roughness, and normals.Qualitative comparisons show shaded renders alongside modality-specific outputs when available.
  • 4.2 GENERATION: 20.99 FID is achieved by Luce GS on Toys4K, improving on TRELLIS 2 at 29.22 by more than 8 FID.The result is reported for generation quality on Toys4K.
  • 4.2 GENERATION: 0.8519 CLIP is achieved by Luce GS on the AI-generated-image benchmark, exceeding TRELLIS GS at 0.8299.Luce GS also leads SigLIP2, while mesh variants lead the mesh-only baselines.
  • 4.3 RECONSTRUCTION: 36.1 dB color-rendering PSNR and 34.6 dB normal-reconstruction PSNR are achieved by Luce GS, while TRELLIS 2 leads albedo and metallic-roughness.Luce’s albedo LPIPS is close to TRELLIS 2 at 0.055 versus 0.051.
  • 4.3 RECONSTRUCTION: Baking decoded normal Gaussians onto the extracted mesh improves normal fidelity across all metrics and enhances color reconstruction.The transfer recovers rough textures, engravings, and dents that are otherwise smoothed.

5 CONCLUSION

Luce presents a multimodal PBR Gaussian representation for relightable image-to-3D generation, with outputs spanning PBR Gaussians and textured meshes. Its reported results improve generation quality while supporting production-compatible asset creation, within stated resolution and material-model limits.

  • 5 CONCLUSION: Luce assigns dedicated Gaussian sets for albedo, metallic-roughness, and normals, producing a compact latent that decodes into relightable Gaussians and textured meshes.Mesh outputs can include tangent-space normal maps.
  • 5 CONCLUSION: Luce achieves an FID of 20.99 on Toys4K, improving on TRELLIS 2 by more than 8 FID, and leads CLIP and SigLIP2 alignment on the AI-generated-image benchmark.These results are presented as state-of-the-art generation quality across the reported benchmarks.
  • 5 CONCLUSION: The representation is intended to bridge generative 3D modeling and production-compatible asset creation through explicit materials and relightable outputs.The conclusion states that outputs are compatible with standard rendering workflows.
  • Limitations and future work: Finite-resolution voxels can underresolve details spanning only a few voxels, and the current material model omits several complex appearance effects.Unmodeled effects include subsurface scattering, anisotropy, translucency, thin-film interference, and strongly view-dependent reflectance.

A MODEL ARCHITECTURE AND IMPLEMENTATION DETAILS

The implementation uses SLatVAE and SLatFlow to compress, generate, and decode sparse multimodal PBR Gaussian clouds. The architecture preserves voxel-level detail while enabling fixed-resolution flow processing, multimodal rendering supervision, and optional mesh and PBR rendering outputs.

  • SLatVAE: SLatVAE compresses the sparse multimodal Gaussian cloud into a compact latent grid and decodes denser Gaussians per voxel and modality.The decoder emits 32 Gaussians per voxel per modality from a latent grid, versus 8 at the input.
  • Mesh and texture export: The same latent supports textured mesh extraction, while texture baking uses UV parameterization and multi-view fitting for each PBR modality.Normal baking adds tangent-space detail to the extracted mesh.
  • PBR rendering: Each occupied voxel contains dedicated Gaussian sets for albedo, metallic-roughness, and normals, matching the PBR modality decomposition.The representation supports deferred PBR shading and relighting under novel environment maps.
  • SLatFlow: SLatFlow is a 30-block transformer that keeps active-voxel tokens at one fixed resolution and uses variable-length attention across samples.Image conditioning combines DINOv2 features from layers 6, 12, 18, and 24 through projected cross-attention.
  • Training objective: The SLatVAE is trained through differentiable rendering against intrinsic images, with separate reconstruction losses for each modality.The objective includes pixel, perceptual, structural-similarity, and latent-prior terms, plus Gaussian volume and opacity regularizers.

B MULTI-LAYER DINOV2 MOTIVATION

The ablation isolates multi-layer DINOv2 conditioning by comparing layer-24-only features with fused features from layers 6, 12, 18, and 24 while holding other choices fixed.

  • B MULTI-LAYER DINOV2 MOTIVATION: The ablation compares single-layer layer-24 DINOv2 conditioning against the default fusion of layers 6, 12, 18, and 24.Both variants use the same remaining architecture and training choices and are evaluated on the GS render path.
  • B MULTI-LAYER DINOV2 MOTIVATION: Multi-layer conditioning improves all reported metrics relative to single-layer conditioning.The single-layer variant omits the fusion projection.

C INFERENCE DETAILS

At test time, Luce generates sparse structure and structured PBR Gaussian latents before decoding to Gaussians or optionally a textured mesh. The section also specifies model sizes, sampling-step accounting, and ablation-table conventions.

  • Inference pipeline: Luce first generates a sparse voxel structure, fills it with PBR Gaussian latents via SLatFlow, and decodes to PBR Gaussians or an optional textured mesh.SLatFlow uses 10 sampling steps.
  • Model size: The GS path has 4.5B parameters, while adding the FlexiCubes mesh decoder increases the mesh-path total to 4.6B.
  • Conditioning ablation: Table 4 compares single-layer layer-24 conditioning with multi-layer layers 6, 12, 18, and 24 conditioning for SLatFlow, with all other settings held equal.Both rows use the GS render path; KID is reported ×100, and FIDdino and KIDdino use a DINOv2 backbone.
  • Sampling steps: Inference times reported in Table 1 are based on each method’s recommended sampling-step counts across the sparse-structure and structured-latent stages.

D EVALUATION RENDERING AND ILLUMINATION

Evaluation renders use CC0 HDRI environment maps, with assets evaluated under ninomaru teien. Different conditioning and evaluation illumination affects whether methods relight or retain baked appearance.

  • Environment maps: All evaluation renders use CC0 HDRI environment maps from Poly Haven, and both benchmarks evaluate generated assets under ninomaru teien.On Toys4K, the conditioning view uses studio small 01.
  • Illumination mismatch: Because Toys4K conditioning illumination differs from evaluation illumination, material-decomposition methods relight while baked-appearance methods carry the conditioning illumination.Luce, TRELLIS 2, and 3DTopia-XL relight; TRELLIS and LiTo retain baked appearance.

E ADDITIONAL QUALITATIVE RESULTS

Additional results span diverse asset categories and display each input alongside Luce’s per-modality PBR Gaussians and shaded render.

  • Asset diversity: Figure 11 presents additional Luce generations across furniture, vehicles, characters, and household objects.
  • Visualization: Each result shows the input condition image, albedo, metallic-roughness, and normal PBR Gaussians, followed by a shaded render.

F BASELINE COMPARISONS ON THE AI-GENERATED-IMAGE BENCHMARK

The benchmark comparisons use per-sample mosaics across methods and four evaluation views, with row metrics directly comparable to dataset-level averages. Across representative inputs, Luce shows sharper material decomposition, more accurate normals, and stronger fine-detail preservation than the closest baseline.

  • Mosaic layout: Per-sample mosaics compare methods in rows across four evaluation views at yaws 300°, 30°, 120°, and 210°.The condition image appears in the top-left, and Luce rows are identified as ours.
  • Pose caveat: LiTo’s views are not pose-matched to other methods because it generates assets in the input view’s frame rather than a canonical orientation.The four yaws therefore place LiTo’s views closer to the conditioning viewpoint.
  • Qualitative comparison: Across eight representative mosaics, Luce consistently produces sharper material decomposition, more accurate surface normals, and better preservation of fine spatial detail than the closest baseline.The samples include heavy texture detail, fine-grained geometry, mixed-material surfaces, and legible text or logos.
  • Metrics: Per-row CLIP and SigLIP2 scores, plus ULIP and Uni3D-L for mesh rows, are computed on the depicted renders.These scores are directly comparable to the dataset-level averages reported in the main results table.
  • Material views: For methods with explicit material decomposition, thumbnails show albedo, metallic-roughness, and normals beside shaded renders.
Loading 2608.23943v1…