Luce: Relightable Gaussians for 3D Asset Generation

• 20.99 FID on Toys4K places Luce ahead of TRELLIS 2 (Xiang et al., 2025) at 29.22 and LiTo at 29.76.Luce achieves the lowest reported FID, improving over both baselines by more than 8 FID.Source: §1 · p. 2
• Luce attaches modality-specific Gaussians to sparse surface-intersecting voxels, jointly encoding geometry, albedo, metallic-roughness, and surface normals.Per-modality encoding helps preserve high-frequency details, including legible surface text. Relighting uses deferred shading under image-based illumination with a standard PBR reflectance model.Source: §1 · p. 2
• A rectified-flow transformer generates a compact unified geometry-and-material latent from one image, decoding into relightable PBR Gaussians.An optional mesh extraction and texture-baking step converts the decoded Gaussians into a textured mesh with PBR maps.Source: §1 · p. 3
• Multi-layer conditioning combines shallow and deep DINOv2 (Oquab et al., 2024) features to preserve fine spatial details such as logos, labels, and inscriptions.Source: §1 · p. 3
• Normal Gaussians learn surface details from authored normal maps and transfer them onto extracted meshes as tangent-space normal maps without increasing polygon count.This preserves details beyond mesh geometry, including engravings, fabric weave, and embossed text.Source: §1 · p. 3

3D representations for generation

• Luce combines Gaussian rendering efficiency with per-modality PBR decomposition in a structured latent, avoiding mesh dependence and baked appearance.Implicit representations, including TripoSG (Li et al., 2024) and Direct3D-S2, target geometry with watertight-surface requirements. Existing Gaussian generation approaches lack material modeling and bake lighting into appearance, like NeRF (Mildenhall et al., 2020).Source: §2 · p. 3
• Structured-latent predecessors differ in what their sparse representations encode: learned image features, native geometry with PBR attributes, or surface light fields.TRELLIS (Xiang et al., 2024) stores DINOv2 features (Oquab et al., 2024) and decodes multiple formats, whereas TRELLIS 2 (Xiang et al., 2025) uses native O-Voxel geometry and per-voxel PBR attributes. LiTo (Chang et al., 2026) tokenizes surface light fields using Perceiver IO (Jaegle et al., 2022).Source: §2 · p. 3

3D generative models

• Luce uses rectified-flow transformers in a unified PBR Gaussian latent to generate geometry and materials jointly.Within the dominant latent-space paradigm, TRELLIS (Xiang et al., 2024) and TRELLIS 2 (Xiang et al., 2025) generate structure before latents. LiTo (Chang et al., 2026) instead conditions a single DiT on DINOv2 (Oquab et al., 2024).Source: §2 · p. 3

PBR and relightable 3D

• Luce predicts normals independently of geometry, preserving finer detail and baking rendered normal Gaussians directly into tangent-space maps.Per-scene inverse rendering decomposes materials for individual assets but does not generalize to novel objects; Gaussian-based methods tie normals to reconstructed surfaces. Luce instead learns a compact latent from large-scale PBR data, jointly generating material modalities and decoding to PBR Gaussians and textured meshes.Source: §2 · p. 3
• Related relightable methods impose mesh, conditioning, staging, optimization, or appearance-representation constraints that distinguish them from Luce.TexGaussian (Xiong et al., 2025) requires a mesh; RelitLRM (Zhang et al., 2024b) uses spherical harmonics rather than explicit PBR; MGM (Ye et al., 2025) is multistage and text-conditioned; MatSpray (Tu et al., 2025) optimizes each scene from multiple views. TRELLIS 2 (Xiang et al., 2025) is mesh-centric, while LiTo (Chang et al., 2026) uses spherical harmonics that cannot be decomposed into materials.Source: §2 · p. 3

3.1 MULTIMODAL PBR GAUSSIAN REPRESENTATION

• Luce assigns geometrically independent albedo, metallic-roughness, and normal Gaussian sets to each occupied voxel, allowing modality-specific detail allocation.Each Gaussian stores geometric parameters and a view-independent modality value, without spherical harmonics; explicit normals can differ from underlying geometry. Preprocessing independently fits each modality against multiview PBR renders to provide SLatVAE inputs.Source: §3.1 · pp. 4–5
• Independent modality splatting produces PBR buffers for deferred environment-map shading, requiring neither mesh extraction nor UV unwrapping.Rendering uses 3DGS alpha-compositing (Kerbl et al., 2023), renormalizes composited normals, and applies split-sum image-based lighting with the Cook–Torrance microfacet BRDF. The metallic workflow uses base reflectance 0.04 for dielectrics and albedo-colored specular reflectance for metals.Source: §3.1 · pp. 4–5

3.2 VARIATIONAL AUTOENCODER

• SLatVAE trades voxel-grid resolution for more decoded Gaussians per voxel, keeping latent generation compact while reconstructing fine geometric and material detail.A sparse transformer with shifted-window attention (Liu et al., 2021) encodes concatenated modality sets into a coarser grid; decoding remains at that resolution. Training uses differentiable per-modality rendering reconstruction, following TRELLIS (Xiang et al., 2024), alongside a latent KL prior and scale/opacity regularization.Source: §3.2 · p. 5

3.3 FLOW-BASED GENERATION

• Image-conditioned generation first constructs a sparse voxel scaffold, then uses SLatFlow to generate latent features jointly describing geometry and materials.The structure-then-latent approach follows TRELLIS (Xiang et al., 2024), using the unmodified pretrained sparse-structure VAE and flow from TRELLIS 2 (Xiang et al., 2025). SLatFlow adapts a DiT-style transformer (Peebles & Xie, 2023) to variable-length voxel sequences, with timestep-modulated adaptive layer normalization and image cross-attention.Source: §3.3 · pp. 5–6
• 20.99 versus 25.21 Toys4K FID and 0.8519 versus 0.8081 AI-generated-image CLIP favor multi-layer over single-layer DINOv2 conditioning.Features from layers 6, 12, 18, and 24 are concatenated and projected into the conditioning space. Combining early spatial detail with deep semantics preserves features such as legible text and logos on generated surfaces.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4 TANGENT-SPACE NORMAL MAP TRANSFER

• 33.0 dB versus 29.5 dB mesh normal PSNR results from tangent-space normal map transfer, recovering high-frequency surface detail without increasing polygon count.Normal Gaussians learn effective rendering normals, including authored normal-map perturbations, and are expressed in the extracted mesh’s local tangent frame. The final mesh carries albedo, metallic, roughness, and tangent-space normal textures for production renderers.Source: §3.3 · p. 6, §3.4 · pp. 6–7

3.3 FLOW-BASED GENERATION

• Dual decoding produces either the primary relightable PBR Gaussian cloud or a textured mesh from the same learned latent.For comparison with mesh-output baselines, Luce retrains an architecturally unchanged FlexiCubes mesh decoder on its latent space, following TRELLIS (Xiang et al., 2024).Source: §3.3 · p. 6

4 EXPERIMENTS

• ∼500K PBR-filtered assets plus 158K TexVerse assets support training, with evaluation covering Toys4K reconstruction, Toys4K generation, and diverse AI-generated images.Training combines Objaverse (Deitke et al., 2023), Objaverse-XL (Deitke et al., 2024), and TexVerse (Zhang et al., 2025). Evaluation uses 338 PBR assets for reconstruction and 412 assets for generation from Toys4K (Stojanov et al., 2021), plus 130 diverse, detail-rich images from Gemini 3.1 Flash Image (Google DeepMind, 2026).Source: §4 · p. 7
• Baselines retain their released conditioning models and resolutions, while Luce uses higher-resolution image conditioning and evaluates methods only on supported output modalities.Luce uses DINOv2 ViT-L/14 at 1036 × 1036 and inherits TRELLIS 2’s sparse-structure stage unchanged; baseline conditioning resolutions range from 512 × 512 to 1024 × 1024. TRELLIS 2 (Xiang et al., 2025) and 3DTopia-XL (Chen et al., 2025) receive full PBR evaluation, whereas TRELLIS (Xiang et al., 2024) and LiTo (Chang et al., 2026) receive shaded-color and surface-normal evaluation.Source: §4 · p. 7
• Generation evaluation measures distribution quality and input-output alignment, while reconstruction evaluation measures fidelity separately for shaded renders and PBR modalities.Toys4K generation uses Inception- and DINO-based FID and KID; both benchmarks use CLIP, SigLIP2 (Tschannen et al., 2025), ULIP (Xue et al., 2024), and Uni3D-L (Zhou et al., 2024). Reconstruction reports PSNR, SSIM, and LPIPS for albedo, metallic-roughness, normal maps, and shaded renders.Source: §4 · p. 7

4.1 IMPLEMENTATION DETAILS

• 150 spherical camera views per normalized asset provide 1024 × 1024 intrinsic images for independently fitting and voxelizing each modality’s Gaussians.Assets are centered and scaled to a unit bounding box; authored normal maps remain active during rendering to preserve detail finer than base geometry. Differentiable fitting uses a 128^3 sparse grid with eight Gaussians per voxel per modality and tanh-bounded center offsets.Source: §4.1 · pp. 7–8

4.3 RECONSTRUCTION

• 36.1 dB color-rendering PSNR and 34.6 dB normal-reconstruction PSNR make Luce GS best on these metrics; TRELLIS 2 leads albedo and metallic-roughness.Luce’s albedo LPIPS is close to TRELLIS 2’s (0.055 vs. 0.051). Reconstruction supports the representation choices, while generation remains the primary contribution.Source: Table 2 · p. 9, §4.3 · p. 9

4.2 GENERATION

• 20.99 FID places Luce GS ahead of the strongest baseline, TRELLIS 2 at 29.22, on Toys4K generation.On AI-generated images, Luce GS leads CLIP (0.8519) and SigLIP2 (0.8508) over TRELLIS GS (0.8299 and 0.8339), while Luce’s mesh variants lead mesh-only ULIP and Uni3D-L. Its native material decomposition supports Gaussian and textured-mesh decoding, relighting, and legible text and fine details that baselines often blur or distort.Source: §4.2 · pp. 8–9, Figure 5 · p. 8

4.4 ABLATION STUDIES

• Multi-layer DINOv2 conditioning improves all evaluated metrics, while normal baking improves normal fidelity across all metrics and enhances color reconstruction.Baking decoded normal Gaussians onto meshes recovers rough textures, engravings, and dents lost by bare meshes, without increasing polygon count.Source: §4.4 · p. 9, Figure 8 · p. 10

5 CONCLUSION

• 20.99 FID on Toys4K improves on TRELLIS 2 by more than 8 FID, while Luce leads CLIP and SigLIP2 alignment on the AI-generated-image benchmark.TRELLIS 2 is identified as the strongest Toys4K baseline.Source: §5 · p. 10
• Each occupied voxel contains dedicated Gaussian sets for albedo, metallic-roughness, and normals, making materials explicit and compatible with flow-based generation and production rendering.A compact, diffusible latent decodes into relightable PBR Gaussians and textured meshes with tangent-space normal maps.Source: §5 · p. 10
• Finite voxel resolution can underresolve features that are small relative to an asset’s overall extent.Proposed cascaded or adaptive-resolution decoding would capture global structure coarsely, then refine local geometry, textures, and normals at higher resolution.Source: §5 · p. 10
• The Cook–Torrance material model does not explicitly capture subsurface scattering, anisotropy, translucency, thin-film interference, or strongly view-dependent reflectance.Additional material channels or learned view-dependent residuals are proposed extensions beyond the current albedo, metallic-roughness, and normal attributes.Source: §5 · p. 10
• Mesh export uses the FlexiCubes decoder architecture from TRELLIS (Xiang et al., 2024), with higher-resolution or learned UV-aware extraction proposed as improvements.Source: §5 · p. 10
• Luce targets object-centric assets; scene-level generation with coherent lighting, material consistency, and object interactions remains an open direction.Source: §5 · p. 10

Appendix

• The appendices document implementation, conditioning ablations, inference costs, evaluation conditions, and additional qualitative comparisons.Appendix A covers architecture, training, and texture baking; B isolates multi-layer conditioning; C reports inference cost and sampling steps. D specifies evaluation renders and illumination, while E–F provide qualitative results and per-sample comparisons against every baseline.Source: Appendix · p. 15

SLatVAE

• 16-block sparse transformers delay SLatVAE downsampling from 128^3 to 64^3 until the encoder’s end, while the decoder stays at 64^3.The decoder emits 32 Gaussians per voxel per modality, versus eight at input, trading resolution for density. The 336-channel input packs eight 14-parameter Gaussians per modality, padding metallic-roughness values to three channels.Source: A MODEL ARCHITECTURE AND… · pp. 15, 17, Figure 9 · p. 16

Texture baking

• The shared latent also feeds a mesh decoder, producing textured meshes with baked PBR modalities and a standard tangent-space normal map.After FlexiCubes extraction, xatlas supplies UVs; textures are fitted to Gaussian renders from 150 viewpoints with total-variation regularization. Duplicate vertices are merged for smooth shading across seams, and rasterized per-vertex tangent frames convert baked object-space normals into tangent space.Source: Figure 9 · p. 16, A MODEL ARCHITECTURE AND… · p. 17

PBR shaded rendering

• Decoded albedo, metallic-roughness, and normals enable PBR shading under novel environment maps without mesh extraction or UV unwrapping.Figure 10 uses environment maps from Poly Haven (2026) and compares deferred rendering of the fitted Gaussians against a textured-mesh reference rendered with Blender EEVEE (Blender Online Community, 2024).Source: Figure 10 · p. 16

SLatFlow

• 30-block SLatFlow maintains fixed-resolution voxel tokens throughout, using rotary embeddings on integer 3D coordinates and QK-RMSNorm.Concatenated samples use boundary tracking and variable-length flash attention (Xiang et al., 2024), avoiding padding and cross-sample attention. Both models use Muon for hidden-layer matrices and AdamW for embeddings, biases, and normalization parameters, with learning rate and weight decay both 10^-3.Source: A MODEL ARCHITECTURE AND… · pp. 16–17
• Image conditioning concatenates DINOv2 ViT-L/14 features from layers 6, 12, 18, and 24, then projects them for cross-attention at fixed token resolution.The projection maps 4×1024 features to 1024 channels, with separate learned normalization for cross-attention. Each transformer block also uses adaptive layer normalization with six-way modulation conditioned on the diffusion timestep.Source: A MODEL ARCHITECTURE AND… · pp. 16–17

Training objective

• SLatVAE learns through differentiable rendering against intrinsic images, with separate albedo, normal, and metallic-roughness reconstruction losses to prevent gradient entanglement.Training uses randomized constant-color backgrounds, pixel, perceptual, and structural-similarity losses, and a VAE latent prior. Gaussian-volume regularization discourages oversized primitives, while opacity regularization pushes opacities toward one to prevent transparent primitives from wasting capacity.Source: A MODEL ARCHITECTURE AND… · p. 17

• The default conditioning fuses DINOv2 features from layers 6, 12, 18, and 24, providing the multi-layer reference for the ablation.Source: B MULTI-LAYER DINOV2 MOTIVATION · p. 17
• The single-layer SLatFlow variant conditions only on layer-24 DINOv2 features and carries no fusion projection.Source: B MULTI-LAYER DINOV2 MOTIVATION · p. 17
• Both variants are evaluated on the GS render path, with all other architecture and training choices held fixed.Source: B MULTI-LAYER DINOV2 MOTIVATION · p. 17

C INFERENCE DETAILS

• ∼42s for GS, ∼148s for mesh without a tangent normal map, and ∼159s for mesh with one are average inference times on a single H100.Times are averaged across both evaluation benchmarks.Source: C INFERENCE DETAILS · p. 17
• 10 SLatFlow sampling steps populate a generated sparse voxel structure with PBR Gaussian latents, which decode into PBR Gaussians or an optional textured mesh.Sparse voxel structure generation precedes SLatFlow sampling.Source: C INFERENCE DETAILS · p. 17
• 4.5B parameters support the GS path; adding the FlexiCubes mesh decoder increases the mesh-path total to 4.6B.The GS total includes the sparse-structure flow, SLatFlow, and shared SLatVAE decoder.Source: C INFERENCE DETAILS · p. 17
• Each method uses sampling-step counts recommended with its public release; Table 5 summarizes the structure-stage and structured-latent-stage counts underlying reported inference times.Single-stage methods have no separate structure flow.Source: C INFERENCE DETAILS · p. 18, Table 5 · p. 18

D EVALUATION RENDERING AND ILLUMINATION

• Generated assets are evaluated under ninomaru teien on both the Toys4K and AI-generated-image benchmarks, using CC0 Poly Haven HDRI maps (Poly Haven, 2026).These are the evaluation conditions for Table 1.Source: D EVALUATION RENDERING AND ILLUMINATION · p. 18
• On Toys4K, conditioning uses studio small 01; Luce, TRELLIS 2, and 3DTopia-XL relight to ninomaru teien, while TRELLIS and LiTo retain conditioning illumination.Luce, TRELLIS 2, and 3DTopia-XL decompose materials, whereas TRELLIS and LiTo have baked appearance.Source: D EVALUATION RENDERING AND ILLUMINATION · p. 18
• On the AI-generated-image benchmark, conditioning images have no controllable illumination.Source: D EVALUATION RENDERING AND ILLUMINATION · p. 18
• Qualitative relighting figures also use Poly Haven maps, with Fig. 10 cycling through six environments.Fig. 1 uses spruit sunrise, studio small 01, and courtyard; Fig. 2 uses studio small 01; Figs. 6 and 11 use spruit sunrise. Fig. 10 cycles through venice sunset, spruit sunrise, moonless golf, studio small 01, hotel room, and courtyard.Source: D EVALUATION RENDERING AND ILLUMINATION · p. 18

• Additional generation results span furniture, vehicles, characters, and household objects.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 18
• Each generated asset is shown through per-modality PBR Gaussians for albedo, metallic-roughness, and normals.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 18
• Each result includes the input condition image and a shaded render of the generated asset.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 18

F BASELINE COMPARISONS ON THE AI-GENERATED-IMAGE BENCHMARK

• Four evaluation views connect the per-sample comparisons directly to the dataset-level alignment metrics, following the TRELLIS (Xiang et al., 2024) evaluation setup.The yaws are 300°, 30°, 120°, and 210°; row scores use these same renders. CLIP and SigLIP2 appear for each method, with ULIP and Uni3D-L additionally shown for mesh rows.Source: F BASELINE COMPARISONS ON THE… · pp. 18–19, Figure 12 · p. 20
• LiTo’s input-view coordinate frame places its evaluation views closer to the conditioning viewpoint, preventing pose-matched comparisons with other methods.The same four yaw angles are used despite this orientation difference.Source: F BASELINE COMPARISONS ON THE… · p. 19
• Luce consistently produces sharper material decomposition, more accurate surface normals, and better fine-detail preservation than the closest baseline across eight representative inputs.The samples cover heavy texture detail, fine-grained geometry, mixed materials, and legible text or logos.Source: F BASELINE COMPARISONS ON THE… · p. 19, Figure 12 · p. 20
• The mosaics expose material outputs alongside shaded renders, while the Luce mesh row includes its tangent-space normal map.Material-decomposing methods show albedo, metallic-roughness, and normal thumbnails; LiTo shows only normals, and TRELLIS GS only shaded renders. Table 1 also reports Luce meshes without the normal map.Source: F BASELINE COMPARISONS ON THE… · p. 19, Figure 12 · p. 20