PaperlayerView exact revision on arXiv

Source-linked AI summary

MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR

Xudong Xu, Zhaoyang Lyu, Xingang Pan, Bo Dai

arXiv:2308.09278v1cs.CV

TL;DR

Existing text-to-3D methods struggle to recover high-fidelity materials, either using Lambertian reflectance or entangling BRDFs with environment lights. MATLABER predicts latent BRDF codes from a smooth auto-encoder trained on real-world material collections; the authors report realistic, coherent materials and downstream support for relighting and editing.

  • Problem

    Existing text-to-3D methods use Lambertian reflectance or produce BRDF materials entangled with environment lights, limiting high-fidelity material recovery.

    Source: §3.2 · p. 4

  • Method

    MATLABER trains a smooth latent BRDF auto-encoder on real-world BRDF collections and predicts latent codes rather than BRDF parameters during appearance modeling.

    Source: §3.3 · pp. 4–5

  • Results

    The authors report that MATLABER generates high-quality, coherent materials with geometry-appearance disentanglement and more natural textures and richer details than Fantasia3D.

    Source: §4 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    The generated BRDF materials support relighting, material editing, and scene manipulation.

    Source: §5 · pp. 8–9

    Some environment lights expose artifacts caused by imperfect geometry, and current generated 3D assets still lack diversity.

    Source: §5 · p. 9

Abstract

from arXiv · show

Based on powerful text-to-image diffusion models, text-to-3D generation has made significant progress in generating compelling geometry and appearance. However, existing methods still struggle to recover high-fidelity object materials, either only considering Lambertian reflectance, or failing to disentangle BRDF materials from the environment lights. In this work, we propose Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR (\textbf{MATLABER}) that leverages a novel latent BRDF auto-encoder for material generation. We train this auto-encoder with large-scale real-world BRDF collections and ensure the smoothness of its latent space, which implicitly acts as a natural distribution of materials. During appearance modeling in text-to-3D generation, the latent BRDF embeddings, rather than BRDF parameters, are predicted via a material network. Through exhaustive experiments, our approach demonstrates the superiority over existing ones in generating realistic and coherent object materials. Moreover, high-quality materials naturally enable multiple downstream tasks such as relighting and material editing. Code and model will be publicly available at \url{https://sheldontsui.github.io/projects/Matlaber}.

1 Introduction

Text-to-3D can reduce the time and specialized skill required for 3D asset creation, but existing methods struggle to recover high-fidelity materials, limiting applications such as relighting. MATLABER uses a latent BRDF auto-encoder trained on real-world material data and is reported to generate coherent, realistic materials that support downstream editing and relighting.

1 Introduction

  • Text-to-3D generation has potential to reduce the time and specialized skill required by traditional asset creation pipelines.The paper notes applications including gaming, film, and AR/VR.Source: §1 · p. 1
  • Existing text-to-3D methods struggle to recover high-fidelity materials, limiting real-world applications such as relighting.Prior approaches model Lambertian reflectance or predict BRDF materials entangled with environment lights.Source: §1 · p. 1
  • MATLABER uses a latent BRDF auto-encoder trained on real-world BRDF data to predict latent codes rather than BRDF values during material generation.The smooth latent space embeds TwoShotBRDF priors, supporting selection of suitable materials while reducing concern about predicted BRDF validity.Source: §1 · pp. 1–2
  • The authors report that MATLABER produces realistic, coherent materials and disentangles geometry from appearance, supporting relighting, material editing, and scene manipulation.They describe these operations as enabled by effective material estimation.Source: §1 · p. 2

2 Related Work

Prior work has adapted text-to-image models and 2D supervision to generate 3D content, while material-estimation research has studied BRDF recovery and geometry-material acquisition. MATLABER differs by targeting text-aligned materials suitable for downstream use in new lighting environments.

2.1 Text-to-Image Generation

  • Text-to-image diffusion models generate complex images, but these approaches remain constrained to the 2D domain and do not address demand in 3D.Stable Diffusion uses latent diffusion to support quality and flexibility under limited computational resources.Source: §2.1 · p. 2

2.2 Text-to-3D Generation with 2D Supervision

  • Text-to-3D methods use CLIP or diffusion-model supervision to optimize 3D representations, with later work improving resolution, geometry, or generation diversity.DreamFusion introduced SDS guidance; Magic3D raised rendered resolution from 64 to 512, while ProlificDreamer proposed VSD.Source: §2.2 · p. 3

2.3 Material Estimation

  • Material-estimation research includes BRDF recovery and joint geometry-material acquisition, while MATLABER targets high-quality surface materials aligned with text prompts.Fantasia3D also creates geometry and materials from language but obtains materials entangled with lights, making them intractable in new environments.Source: §2.3 · p. 3

3 Method

MATLABER separates text-to-3D generation into geometry and appearance stages, using SDS to guide geometry and a learned latent BRDF prior to model materials. Its material network predicts latent codes that are decoded into BRDF parameters for rendering.

3.1 Score Distillation Sampling (SDS)

  • SDS uses a pretrained diffusion model as guidance for optimizing scene parameters, and prior work applies it to normal maps to encourage detailed geometry.Stable-dreamfusion and Fantasia3D [7] use publicly available latent diffusion models as guidance; w(t) is a weighting function.Source: §3.1 · pp. 3–4

3.2 Appearance Modeling in Text-to-3D Generation

  • Prior text-to-3D methods either omit specular reflectance or entangle predicted materials with fixed environment lighting, motivating MATLABER’s use of both diffuse and specular terms and real-world BRDF priors.Fantasia3D [7] uses a fixed HDR environment map and typically sets metalness to zero; DreamFusion [4] and Magic3D [8] use Lambertian shading.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 Latent BRDF Auto-Encoder

  • TwoShotBRDF provides 11,250 high-quality SVBRDF maps for training an encoder-decoder that maps 7-dimensional BRDFs to 4-dimensional codes, with four losses shaping reconstruction and latent-space smoothness.The losses include reconstruction, KL divergence, interpolation smoothness, and cyclic consistency.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

3.4 Material-Aware Text-to-3D

  • Geometry generation uses DMTET and SDS on rendered normal maps to update a geometry MLP until the geometry aligns with the text prompt.The normal map is rendered from the DMTET representation at a sampled camera pose.Source: §3.4 · p. 5
  • For appearance generation, Γ predicts a latent BRDF code at each surface point, which the pretrained decoder converts to BRDF parameters; rotating multiple environment maps with rendered-image SDS encourages material-light disentanglement.A separate regularizer enforces smooth diffuse materials.Source: §3.4 · pp. 5–6
  • Decoded diffuse, specular, and roughness parameters are rendered under incoming illumination, with the specular intensity approximated using precomputed lookup textures.The rendering description identifies GGX distribution, geometric attenuation, and Fresnel terms, and uses split-sum integration.Source: §3.4 · pp. 5–6

4 Experiments

User ratings favor MATLABER on realism, details, and disentanglement, while qualitative comparisons find its geometry competitive and appearance more natural. Relighting under varied illumination and smooth material interpolation demonstrate uses for its generated BRDF materials.

Qualitative results

  • Against DreamFusion, Magic3D, and Fantasia3D on shared prompts, MATLABER produces geometry competitive with Fantasia3D and more natural textures with richer details.DreamFusion and Magic3D results were taken from their project pages because their model weights were inaccessible.Source: §4 · pp. 6–7, Figure 4 · p. 8

User study

  • In a user study with 80 volunteers and 800 responses, MATLABER achieved the best performance on realism, details, and disentanglement, while DreamFusion and Magic3D showed better text alignment.MATLABER outperformed three baseline methods by a large margin on disentanglement.Source: §4 · pp. 7–8

Relighting and material editing

  • Rotating an HDR environment map yields natural, photorealistic relighting, and interpolating latent embeddings smoothly morphs goblet and tulip materials.The goblet example interpolates between embeddings optimized for “A golden goblet” and “A silver goblet.”Source: §4 · p. 8

5 Conclusion

MATLABER uses a latent BRDF auto-encoder trained on real-world collections to generate coherent materials for text-to-3D assets, which support relighting, editing, and scene manipulation.

5 Conclusion

  • MATLABER’s real-world BRDF prior supports high-quality, coherent materials and operations including relighting, material editing, and scene manipulation.The conclusion describes the approach as achieving disentanglement of geometry and appearance.Source: §5 · pp. 8–9
  • Figure 5 presents generated diffuse, specular, and roughness materials alongside relit images under a rotating environment light.The figure shows the material descriptions on the left and the relit images on the right.Source: Figure 5 · p. 9
  • Linear interpolation in the smooth BRDF latent space changes a goblet from gold to silver and a tulip from blue to red.Figure 6 illustrates these gradual material changes.Source: Figure 6 · p. 9
  • Generated objects can show artifacts under some environment lights because of imperfect geometry, and the current method still lacks diversity.Reported examples include weird bread textures and hollows in a relit strawberry.Source: §5 · p. 9, Figure 7 · p. 9

Related papers

Loading 2308.09278v1…