PaperlayerView exact revision on arXiv

Source-linked AI summary

CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets

Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, Jingyi Yu

arXiv:2406.13897v1cs.CV

TL;DR

Existing 3D creation tools and generative methods remain limited by expertise requirements, scarce high-quality 3D data, and entangled geometry and appearance. CLAY combines a large-scale 3D-native latent generative framework with multimodal controls and a material diffusion model, achieving controllable asset generation and strong user-study preferences. The model still requires separate geometry and material stages and depends on training data whose quantity and quality can improve.

  • Problem

    3D creation is hindered by demanding existing tools, limited quality 3D datasets, and the entanglement of geometry and appearance.

  • Method

    CLAY uses a multi-resolution VAE and latent DiT trained progressively on processed 3D data, with multimodal adaptations and multi-view PBR material diffusion.

  • Results

    CLAY supports controllable high-quality 3D asset creation, and user studies report 67.4% appearance and 78.9% geometry votes in text-to-3D comparisons.

  • Takeaways & Limitations

    CLAY provides a 3D-native foundation for converting text, images, and diverse 3D primitives into detailed geometry and realistic materials.

  • Takeaways & Limitations

    CLAY is not end-to-end, requiring separate geometry and material stages plus remeshing and UV unwrapping, while its training data still has quantity and quality limitations.

Abstract

from arXiv · show

In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.

1 INTRODUCTION

CLAY addresses limitations in 3D creation by combining a large-scale 3D-native generative model with material generation and multimodal controls. Its architecture, training data, and adaptation schemes support detailed, realistic, and controllable asset creation.

  • Motivation: Limited quality 3D datasets and entangled geometry and appearance make it difficult to transfer recent 2D generation progress to 3D asset creation.These challenges constrain efforts to bridge human imagination and digital 3D creation.
  • Geometry generation: CLAY learns rich 3D priors from diverse geometries using neural fields, a multi-resolution VAE, and a latent DiT with adaptive latent size.A progressive training scheme scales the model to 1.5 billion parameters.
  • Training data: CLAY standardizes diverse 3D data by remeshing surfaces into occupancy fields while preserving sharp edges and flat surfaces, then adds GPT-4V annotations.The pipeline is designed to improve training-data quality for fine-grained geometry generation.
  • Material generation: Its material system generates multi-view PBR diffuse, roughness, and metallic textures, super-resolves them, and maps them onto generated geometry at 2K UV resolution.The approach avoids tedious distillation and is reported to be faster than traditional optimization methods.
  • Controllable creation: CLAY supports text, images, and 3D primitives including multi-view images, voxels, bounding boxes, point clouds, and implicit representations.LoRA-like fine-tuning and cross-attention conditioning enable creations ranging from sketches to detailed assets.

2 RELATED WORK

Prior 3D generation methods either lift 2D priors into 3D or learn directly from 3D data, but each route faces limitations in geometric fidelity, efficiency, diversity, or detail. CLAY follows the 3D-native route while using latent-space generation and large-scale training to address these challenges.

  • Overview: 3D generation research spans direct 3D-dataset learning, 2D-image priors, and 3D priors imposed on image-generation systems.These routes differ in how they represent and generate geometry.
  • Imposing 2D Images as Priors: 2D-prior methods use image generators and optimization, but repeated parameter adjustment and long optimization times can be required for satisfactory results.Multi-view extensions improve consistency but still begin from 2D images.
  • Imposing 2D Images as Priors: Because 2D-based methods prioritize generated-image quality rather than geometric fidelity, their geometry often becomes incomplete and lacks detail.This limitation is stated for approaches that extract 3D shapes from generated multi-view results.
  • Imposing 3D Geometry as Priors: 3D-prior methods use sparse- or single-view reconstructors and deep transformers to reconstruct NeRFs, but volume-rendering objectives can produce coarse or noisy geometry.These approaches are commonly referred to as Large Reconstruction Models.
  • Direct 3D Generation: Direct 3D methods represent geometry with grids, point clouds, signed distance fields, or occupancy fields, offering explicit surface mechanisms but imposing representation and dataset constraints.Some VAE-based methods are primarily trained or tested on ShapeNet, limiting shape diversity and variety.

3 LARGE-SCALE 3D GENERATIVE MODEL

CLAY separates geometry and texture generation, using a scalable 3D-native latent architecture and standardized training data to generate controllable, detailed 3D assets.

  • Model Design: CLAY separates geometry and texture generation rather than relying on 2D generation to assist geometry creation.
  • Model Design: 1.5 billion parameters support CLAY’s large-scale 3D-native generative model, pretrained on high-quality 3D data.
  • Representation and Model Architecture: CLAY encodes sampled mesh surface points into dynamic latent codes, denoises them with DiT, and decodes them into neural-field occupancy values.
  • Representation and Model Architecture: The multi-resolution VAE varies point-cloud sampling among 2048, 4096, and 8192 points to capture geometric detail across resolutions.The decoder uses 24 self-attention layers and a cross-attention layer to output occupancy logits.
  • Representation and Model Architecture: The 24-layer DiT uses cross-attention for text conditioning and progressively increases latent length from 512 to 1024 to 2048 during training.Learning rates are reduced at each increase in latent length.
  • Data Standardization for Pretraining: CLAY standardizes diverse 3D datasets through remeshing that preserves geometric features while addressing non-watertight meshes and volumetric accuracy.Cross-sectional analysis describes positive-volume maximization and preservation of sharp geometric features.

4 ASSET ENHANCEMENT

CLAY enhances generated geometry through mesh optimization and material synthesis, producing assets compatible with existing computer-graphics pipelines and realistic rendering. Its multi-view material workflow supports diverse PBR modalities, style adaptation, and 2K textures.

  • Asset Enhancement: Two-stage enhancement combines post-generation geometry optimization with material synthesis to transform coarse meshes into usable digital assets.Geometry optimization improves structural integrity, compatibility, and form, while material synthesis adds realistic textures.
  • Asset Enhancement: Mesh quadrification and atlasing replace uneven triangle meshes and simplify editing, export, and UV-based texture mapping.Marching Cubes can produce millions of uneven triangles that complicate mesh editing and automatic UV unwrapping.
  • Material Synthesis: Over 40,000 Objaverse objects provide high-quality PBR supervision for diffuse, roughness, and metallic texture synthesis.The textures are generated from multiple views and mapped onto the geometries’ UV spaces.
  • Material Synthesis: A multi-view Material Diffusion model concurrently denoises texture modalities with view consistency using additional UNet branches and skip connections.The model is adapted from MVDream to generate texture attributes across multiple channels and modalities.
  • Asset Enhancement: LoRA fine-tuning on specific datasets enables LEGO-duck variants in stone and pocket-monster styles.The figure illustrates style-specific generation after fine-tuning on rock and pocket-monster datasets.
  • Material Synthesis: 2K texture resolution, targeted inpainting, Real-ESRGAN, and MultiDiffusion produce high-quality textures for realistic, production-quality rendering.The authors report higher quality and visual pleasantness than previous 3D generation schemes.

5 MODEL ADAPTATION

CLAY adapts its pretrained 3D generator for controllable generation by adding parallel conditioning modules and spatial embeddings. These mechanisms support text, visual, voxel, geometric, and multi-view inputs while preserving modality-specific spatial information.

  • Model Adaptation: CLAY supports efficient style adaptation through LoRA on DiT attention layers while also enabling multiple conditional modalities.Supported user-provided conditions include text, image or sketch, voxel, and other 3D representations.
  • Conditioning Scheme: Additional conditions are added as parallel residuals alongside text conditioning through independently trainable cross-attention modules.Each additional module receives its own condition and can be scaled by a learned or controlled coefficient.
  • Conditioning Scheme: Scalar α_i directly controls the influence of each additional condition in the residual conditioning framework.CrossAttn denotes text conditioning, while CrossAttn_i and c_i denote the i-th additional module and condition.
  • Spatial Control: Learned positional embeddings associate spatial features with 3D points so attention can distinguish coordinates from feature content.This design addresses spatial-control challenges caused by dynamically generated latent codes interwoven with spatial coordinates.
  • Condition Implementations: CLAY separately trains conditioning modules for images, sketches, voxels, bounding boxes, sparse point clouds, multi-view images, and partial point clouds.The conditions provide visual, spatial, shape, placement, and completion controls through modality-specific feature processing.
  • Voxel: Voxel conditioning converts a 16^3 occupancy grid into an 8^3 feature volume before cross-attention integration, enabling detailed geometry from voxel designs.The volume features are augmented with positional embeddings of voxel centers and then flattened.

6 RESULTS

CLAY generates diverse, high-quality 3D assets from text, images, and varied geometric conditions, with strong reconstruction, alignment, diversity, and user-preference results. It also produces detailed geometry and PBR textures faster than several compared methods.

  • Generation Results: CLAY generates varied, detailed 3D models spanning historical objects, cultural artifacts, vehicles, and futuristic designs.These outputs are described as suitable for gaming, film, and virtual simulations.
  • Multi-modal Control: CLAY supports image, bounding-box, multi-view, point-cloud, voxel, and partial-geometry conditioning across diverse reconstruction and generation tasks.It can resemble image inputs, create scenes from bounding boxes, reconstruct from multiple views or sparse point clouds, and complete missing geometry.
  • Quantitative Evaluation: Larger CLAY models achieve higher scores and more accurate text-shape alignment on text-to-shape evaluation.The evaluation uses render-FID, render-KID, P-FID, P-KID, CLIP, and ULIP-T over a 16K text-shape validation set.
  • Quantitative Evaluation: Additional conditions improve geometric details while preserving alignment, with multi-view normal conditioning among the strongest settings.Conditioned generation is evaluated using CD, EMD, Voxel-IoU, F-score, ULIP-T, and ULIP-I.
  • Geometry Diversity: CLAY produces novel shapes from text and novel structural combinations from images while retaining strong resemblance to image content.Geometry diversity is assessed by comparing normalized ULIP features with retrieved dataset geometries.
  • Comparisons with SOTA: 45 seconds is CLAY’s reported total generation time, including shape latent generation, latent decoding, mesh processing, and PBR generation.The timing breakdown is 4 seconds, 1 second, 8 seconds, and 32 seconds respectively on one Nvidia A100 GPU.
  • Comparisons with SOTA: CLAY outperforms compared methods in qualitative geometry, appearance, and all reported CLIP-based text-to-3D and image-to-3D metrics.Compared methods show artifacts, roughness, missing details, or long optimization times, whereas CLAY generates smooth, detailed geometry with PBR textures.
  • User Studies: 150 volunteers preferred CLAY for both appearance and geometry in text-to-3D and image-to-3D user studies.For text-to-3D, CLAY received 67.4% of appearance votes and 78.9% of geometry votes; for image-to-3D, it received 85.4% and 91.2%.

7 DISCUSSIONS AND CONCLUSIONS

The paper presents CLAY as a large-scale, multi-modal 3D generator combining geometry and material generation for high-quality asset creation. The authors report broad efficacy and adaptability, while identifying end-to-end integration, data limitations, composed-object robustness, and misuse risks as boundaries.

  • Conclusions: CLAY combines a multi-resolution VAE, DiT, progressive training, standardized remeshing, and GPT-4V annotation to generate high-quality 3D assets.The framework supports multi-modal controls for geometry and textures.
  • Conclusions: CLAY’s reported geometry quality, diversity, and material richness position it as a leading 3D generator.This conclusion follows the paper’s comprehensive experiments and user studies.
  • Ethics Statement: CLAY’s generalization capability creates potential misuse risks, including deceptive virtual assets or scenes that violate regulations or propagate false information.The authors note that pretrained CLIP and DINO feature encoders contribute to this generalization capability.
  • Limitations and Future Work: CLAY is not end-to-end because geometry and material generation remain separate and require remeshing and UV unwrapping.The authors identify integrated geometry-and-PBR architectures as future work.
  • Limitations and Future Work: CLAY remains vulnerable on complex composed objects with text-only inputs, partly because training data and textual descriptions are insufficient.The authors contrast this with greater robustness on single-object assets.
Loading 2406.13897v1…