PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on 3D Content Generation

Jian Liu, Xiaoshui Huang, Tianyu Huang, Lu Chen, Yuenan Hou, Shixiang Tang, Ziwei Liu, Wanli Ouyang, Wangmeng Zuo, Junjun Jiang, Xianming Liu

arXiv:2402.01166v2cs.CVcs.AI

TL;DR

3D content generation has practical applications but presents substantial technical challenges. This survey organizes major techniques into three categories and reviews about 60 papers. It summarizes unresolved challenges and future research directions.

  • Problem

    3D content generation has practical value across applications, while current techniques face substantial technical challenges.

    Source: §1 · p. 1

  • Method

    The survey reviews 60 papers and organizes 3D-native, 2D-prior-based, and hybrid methods in a taxonomy.

    Source: §1 · p. 2, §7 · p. 7

  • Results

    The survey synthesizes the three method categories and identifies unresolved challenges and promising research directions.

    Source: §7 · p. 7

  • Takeaways & Limitations

    The synthesis is intended to guide further advances as researchers address open problems in 3D content generation.

    Source: §7 · p. 7

    Production-level quality, scale, and precision in 3D content generation remain unresolved.

    Source: §6.1 · p. 7

Abstract

from arXiv · show

Recent years have witnessed remarkable advances in artificial intelligence generated content(AIGC), with diverse input modalities, e.g., text, image, video, audio and 3D. The 3D is the most close visual modality to real-world 3D environment and carries enormous knowledge. The 3D content generation shows both academic and practical values while also presenting formidable technical challenges. This review aims to consolidate developments within the burgeoning domain of 3D content generation. Specifically, a new taxonomy is proposed that categorizes existing approaches into three types: 3D native generative methods, 2D prior-based 3D generative methods, and hybrid 3D generative methods. The survey covers approximately 60 papers spanning the major techniques. Besides, we discuss limitations of current 3D content generation techniques, and point out open challenges as well as promising directions for future work. Accompanied with this survey, we have established a project website where the resources on 3D content generation research are provided. The project page is available at https://github.com/hitcslj/Awesome-AIGC-3D.

1 Introduction

The survey addresses a fast-growing 3D content-generation field whose applications include game, construction, and industrial design. It proposes a three-part taxonomy, reviews major techniques, and identifies open challenges and future directions.

1 Introduction

  • 3D generation could reduce labor and time in game design and support virtual assembly and low-cost revision in industrial design.Construction designers can also use concept models to communicate with customers.Source: §1 · p. 1
  • 3D-native methods train on 3D datasets, while 2D-prior-based methods build on image-text diffusion models and hybrid methods combine both approaches.The survey notes that 3D data are scarce compared with image-text pairs; DreamFusion [Poole et al., 2023] optimizes a NeRF using score distillation sampling, while one2345++ [Liu et al., 2023a] trains a 3D diffusion model on 2D-prior-based multi-view images.Source: §1 · p. 1
  • Earlier surveys cover only portions of the field, leaving recent 3D-native and hybrid methods largely unaddressed.The authors identify a need for a comprehensive survey to help practitioners navigate the expanding research frontier.Source: §1 · p. 2
  • The survey proposes a taxonomy, reviews 60 papers spanning major techniques, and discusses open challenges and promising future directions.It is intended for both industrial and academic communities.Source: §1 · p. 2

2 Preliminaries

The preliminaries distinguish explicit from implicit 3D representations and introduce diffusion models as a generative framework. They outline the structures and properties relevant to later 3D-generation methods.

2.1 3D Representation

  • 3D representations are broadly classified as explicit or implicit, with this section surveying both categories.The section also introduces diffusion models.Source: §2.1 · p. 2

Explicit Representations

  • Explicit representations directly describe an object's geometry or structure, including point clouds, voxel grids, and meshes.Meshes encode surface topology and connectivity, while voxels use a regular volumetric grid and work well with convolutional neural networks.Source: Explicit Representations · p. 2

Implicit Representations

  • Implicit representations define shapes through a function or level set; examples include NeRF, 3D Gaussian Splatting, and signed distance functions.NeRF can be slow to render, while 3D Gaussian Splatting produces results quickly but unstably; DMTet combines SDFs and meshes to refine generated geometry.Source: Implicit Representations · p. 2

2.2 2D Diffusion Models

  • Diffusion models learn iterative denoising from noisy signals and generate samples by reversing that process from random noise.Figure 1 presents a chronological overview of relevant 3D-native generative methods.Source: §2.2 · p. 3, Figure 1 · p. 3

3 3D Native Generative Methods

3D-native methods generate 3D representations under 3D-data supervision, with representation and supervision identified as key components. The survey reviews object, scene, and human-generation approaches.

3 3D Native Generative Methods

  • 3D-native methods generate 3D representations under 3D-data supervision and are grouped into object, scene, and human methods.The survey identifies representation and supervision as crucial to generation quality.Source: §3 · p. 3

3.1 Object

  • Object-generation methods expanded from limited-category datasets toward larger vocabularies, but the collected assets and captions for Point-E and Shap-E were not publicly released.Text2Shape used 75K descriptions for 15K chairs and tables; ShapeCraft expanded this to 369K shape-text pairs, while later work explored image-to-triplane, diffusion, and open-vocabulary approaches.Source: §3.1 · p. 3

3.2 Scene

  • Scene-generation methods moved from GAN-based radiance fields to latent diffusion approaches, but single-vector scene representations limit complex distributions.NeuralField-LDM is described as time-consuming and relatively low-resolution; X3 uses hierarchical voxel latent diffusion to generate higher-resolution representations coarse to fine.Source: §3.2 · p. 3

3.3 Human Avatar

  • Human-avatar methods progressed from parametric models to learned methods using 3D scans or multi-view images.Parametric models struggle with complex geometries such as hair and loose clothing; later approaches include implicit-function clothed-human models and tri-plane diffusion for detailed head avatars.Source: §3.3 · pp. 3–4

4 2D Prior-based 3D Generative Methods

2D priors offer a route around scarce 3D training data, while 2D-prior methods still face quality, view-consistency, and speed challenges. The section surveys object, scene, and human-avatar generation approaches.

4.1 Object

  • DreamFusion [Poole et al., 2023] pioneered per-input 3D optimization guided by pretrained 2D diffusion models, but view consistency and synthesis speed remained challenges.Stable Diffusion’s front-view bias could cause generated objects to repeat front views across angles; DreamGaussian [Tang et al., 2024] used 3D Gaussian splatting and reported around 10x acceleration over NeRF-based approaches.Source: §4.1 · p. 4
  • Magic3D [Lin et al., 2023] used coarse-to-fine optimization, Fantasia3D [Chen et al., 2023b] disentangled geometry and appearance, and ProlificDreamer [Wang et al., 2023b] introduced variational score distillation.TEXTure [Richardson et al., 2023] and TexFusion [Cao et al., 2023] instead paint textures onto predefined geometry from multiple viewpoints.Source: §4.1 · pp. 4–5

4.2 Scene

  • Scene methods use pretrained 2D models and inpainting or depth lifting to build larger scenes, while SceneTex [Chen et al., 2023a] optimizes a multiresolution texture field.SceneDreamer [Chen et al., 2023c] uses a Bird’s Eye View representation, and CityDreamer [Xie et al., 2024] decouples building and background generation.Source: §4.2 · p. 5

4.3 Human Avatar

  • Human-avatar methods use parametric models as geometric priors, with later approaches extending score distillation and diffusion-based generation to heads and full-body avatars.AvatarCLIP [Hong et al., 2022a] uses vision-language models with implicit representations; DreamWaltz [Huang et al., 2023d] adds occlusion-aware score distillation and skeleton conditioning.Source: §4.3 · p. 5

5 Hybrid 3D Generative Methods

Hybrid methods combine abundant 2D visual resources with targeted 3D supervision to address limitations of 3D-native and 2D-prior approaches. The section reviews object, scene, human-avatar, and dynamic generation.

5.1 Object

  • One-2-3-45 [Liu et al., 2023b] generates a 3D mesh from one image in 45 seconds by pairing Zero123-generated views with a reconstruction model.Zero123 [Liu et al., 2023c] fine-tunes a pretrained 2D diffusion model with 3D data to generate novel views conditioned on one input view.Source: §5.1 · p. 5
  • Subsequent object methods generate multiple views simultaneously or exchange information across views to improve consistency, and some use 3D-prior initialization for more coherent outputs.Examples include SyncDreamer [Liu et al., 2024], MVDream [Shi et al., 2024], Wonder3D [Long et al., 2023], and Dreamcraft3d [Sun et al., 2023].Source: §5.1 · p. 5

5.2 Scene

  • Hybrid scene methods address cross-view error accumulation with global-aware parallel generation, or combine room layouts with panorama generation and geometry alignment.MVDiffusion uses correspondence-aware attention; ControlRoom3D [Schult et al., 2023] generates room meshes from a text description and user-defined layout.Source: §5.2 · p. 6

5.3 Human Avatar

  • Hybrid human generation combines 2D appearance data with 3D geometry priors, including unpaired learning and diffusion pipelines for geometry and texture.HumanNorm [Huang et al., 2023c] uses two diffusion models fine-tuned on 2.9K 3D human models.Source: §5.3 · p. 6

5.4 Dynamic

  • Dynamic methods extend static-to-dynamic generation with temporal representations, or reconstruct 3D scenes from generated or supplied videos using 4D and diffusion priors.Examples include hex-plane in MAV3D [Singer et al., 2023], 4D grid encoding in Animate124 [Zhao et al., 2023], and Gaussian-splatting reconstruction in 4DGen [Yin et al., 2023].Source: §5.4 · pp. 6–7

6 Future Direction

The survey identifies unresolved quality, controllability, and speed problems in 3D content generation. It points to data, model, and evaluation needs as future directions.

6.1 Challenges

  • Current methods still struggle with compact meshes, reasonable wiring, detailed textures, lighting and shadow effects, material properties, and precise conditional output.Feed-forward and score-distillation methods based on Gaussian splatting are faster but lower quality than NeRF optimization; production-level quality, scale, and precision remain unresolved.Source: §6.1 · p. 7

6.2 Data

  • A key data challenge is collecting billions of 3D objects, scenes, and humans, alongside extracting implicit 3D knowledge from multi-view images and videos.The survey suggests open-world 3D gaming platforms and diverse unlabeled data as possible routes toward larger datasets.Source: §6.2 · p. 7

6.3 Model

  • Future work should explore 3D representations and architectures that scale with growing datasets, potentially leading to foundation models specialized for 3D content generation.The survey also notes the possibility of multimodal language models operating 3D modeling software, while emphasizing the need for research into beneficial development.Source: §6.3 · p. 7

6.4 Benchmark

  • Evaluation relies mainly on human ratings, and holistic assessment remains difficult because it must account for physical 3D properties and intended designs.The survey calls for robust metrics that assess geometric and textural fidelity against photorealism standards.Source: §6.4 · p. 7

7 Conclusion

The conclusion presents a comprehensive analysis of 3D-native, 2D-prior-based, and hybrid 3D generation, along with a taxonomy of recent methods. It also summarizes unresolved challenges and promising research directions.

7 Conclusion

  • The survey analyzes 3D-native, 2D-prior-based, and hybrid 3D content generation techniques.Source: §7 · p. 7
  • It introduces a taxonomy to summarize advances in recent 3D content generation methods.Source: §7 · p. 7
  • The survey identifies unresolved challenges and proposes promising research directions for the field.Source: §7 · p. 7

Related papers

Loading 2402.01166v2…