PaperlayerView exact revision on arXiv

Source-linked AI summary

LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis

Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, Xiaohui Zeng

arXiv:2403.15385v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Per-prompt text-to-3D optimization can take up to an hour, and amortized methods have struggled to scale while capturing high-frequency detail. LATTE3D amortizes neural-field and surface generation while using 3D data to support larger-scale training. It generates high-quality shapes in 400ms, with quality and generalization further improvable through test-time optimization.

  • Problem

    Text-to-3D methods can require up to an hour of optimization per prompt, while amortized approaches struggle with high-frequency details and scaling to large prompt sets.

    Source: §1 · p. 2

  • Method

    LATTE3D uses a scalable architecture, 3D-aware diffusion priors, regularization, and reconstruction pretraining to amortize neural-field and textured-surface generation.

    Source: §1 · p. 2, §3.2 · pp. 5–6

  • Results

    400ms: LATTE3D generates high-quality shapes, and its performance on seen and unseen prompts is competitive with baselines while inference is at least one order of magnitude faster.

    Source: §4.2 · p. 11, §5 · p. 14

  • Takeaways & Limitations

    Fast test-time optimization can further improve quality and generalization beyond LATTE3D’s single-pass generation.

    Source: §5 · p. 14

    LATTE3D relies on text-to-image models that can fail to respect fine-grained details, and frozen stage-1 geometry cannot be fixed in stage 2 or test-time optimization.

    Source: §5 · p. 14

Abstract

from arXiv · show

Recent text-to-3D generation approaches produce impressive 3D results but require time-consuming optimization that can take up to an hour per prompt. Amortized methods like ATT3D optimize multiple prompts simultaneously to improve efficiency, enabling fast text-to-3D synthesis. However, they cannot capture high-frequency geometry and texture details and struggle to scale to large prompt sets, so they generalize poorly. We introduce LATTE3D, addressing these limitations to achieve fast, high-quality generation on a significantly larger prompt set. Key to our method is 1) building a scalable architecture and 2) leveraging 3D data during optimization through 3D-aware diffusion priors, shape regularization, and model initialization to achieve robustness to diverse and complex training prompts. LATTE3D amortizes both neural field and textured surface generation to produce highly detailed textured meshes in a single forward pass. LATTE3D generates 3D objects in 400ms, and can be further enhanced with fast test-time optimization.

1 Introduction

Per-prompt text-to-3D optimization can take up to an hour, while amortized approaches remain limited in scale and detail. LATTE3D targets fast, high-quality generation by scaling amortization and incorporating 3D data during training.

1 Introduction

  • The goal is real-time text-to-3D synthesis for diverse prompts, with rapid iteration and optional fast test-time optimization for further quality gains.The motivation is to empower user creativity in 3D content creation.Source: §1 · p. 2
  • Prior text-to-3D methods can take up to an hour per prompt and may lack high-frequency detail or fail to generate the desired object [39].Surface-based refinement adds high-frequency detail, but the two-stage pipeline remains expensive and prone to prompt failure [57].Source: §1 · p. 2
  • ATT3D amortizes prompt optimization but is limited to hundreds or thousands of prompts, low-fidelity textures, and neural-field output without high-frequency details.Its simple architecture struggles to scale with dataset size and rendering resolution, and both ATT3D and underlying per-prompt methods can fail on general prompts.Source: §1 · p. 2
  • LATTE3D introduces a scalable amortization architecture intended to handle prompt sets orders of magnitude larger than prior approaches.The method also uses 3D-aware diffusion priors, shape regularization, and 3D-reconstruction pretraining, and amortizes surface refinement to boost quality.Source: §1 · p. 2

2 Related Work

Related work moves from class-specific synthesis to text-conditioned 3D generation using diffusion, while addressing the cost of per-prompt optimization. Faster alternatives trade off detail or scale, motivating larger-scale amortized synthesis.

2 Related Work

  • Early 3D object synthesis typically generated objects from a given class [27], while captioned 3D shapes enabled text-to-3D models to generate diverse categories but required 3D supervision [16] [35].This restricted them to synthetic datasets limited in size, diversity, and visual fidelity.Source: §2 · p. 3
  • Differentiable rendering enabled text-to-3D methods to use text-to-image diffusion; DreamFusion [39] introduced SDS for neural-field optimization, followed by surface refinement for high-frequency detail.This two-stage optimization pipeline defines most state-of-the-art text-to-3D approaches [31] [55] [57].Source: §2 · p. 3
  • Text-to-image-to-3D lifting methods improve speed but offer limited quality detail and take 5–20 seconds per prompt [19] [52], whereas amortized methods train one model across prompts.LATTE3D extends amortization to about 100,000 prompts using 3D-aware techniques; concurrent alternatives include AToM [41], ET3D, and LGM [52].Source: §2 · p. 3

3 Methodology

LATTE3D amortizes both volumetric geometry generation and surface-based texture refinement, using 3D reconstruction pretraining and 3D-informed supervision to support prompt-conditioned textured meshes. Its inference pipeline produces a mesh from a text prompt in 400ms, with optional test-time refinement and a separate stylization application.

3 Methodology

  • LATTE3D amortizes a volume-based geometry stage and a surface-based texture stage, using 3D knowledge to scale to larger, more diverse prompt sets.The model is initialized through reconstruction pretraining, anneals from real point clouds to a dummy input during stage 1, and uses a refined texture network at inference.Source: §3 · p. 4

3.1 Pretraining to reconstruct shapes.

  • Reconstruction pretraining teaches the model to encode and decode 3D shapes by matching rendered opacity and RGB images, providing an initialization for subsequent amortized SDS training.The model takes a sampled point cloud and predicts a 3D shape, whose multiple rendered views are compared with renderings of the input shape.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 Model Architecture

  • Geometry network G and texture network T share encoder weights in stage 1; stage 2 freezes G and refines T, whose upsampled triplane features support high-frequency texture details.Stage 2 upsamples the latent triplane from resolution 128 to 480 and adds a text-conditioned residual MLP; stage 1 uses volume rendering, while stage 2 extracts and rasterizes a mesh.Source: §3.2 · pp. 5–6, Figure 4 · p. 5

3.3 Amortized Learning

  • Stage 1 trains volumetric geometry with a 3D-aware diffusion prior and regularization against retrieved 3D shapes, while stage 2 freezes geometry and trains texture with depth-conditional SDS guidance.MVDream supplies multiview supervision, but its four views can miss full geometry or produce enlarged geometry and floaters; mask regularization compares the generated shape with a retrieved shape.Source: Amortizing Stage-1 Generation · p. 7, Amortizing Stage-2 Generation · p. 7

3.4 Inference

  • 400ms on an A6000 GPU is enough to generate a textured mesh from a text prompt, with optional test-time refinement to improve geometry and textures.The geometry network is frozen after stage 1; the refined texture network supplies the inference texture.Source: §3.4 · p. 8

3.5 Test Time Optimization

  • For prompts that deviate significantly from seen prompts, test-time optimization freezes geometry and fine-tunes texture with depth-conditional SDS guidance.The authors report faster convergence than MVDream or other optimization-based text-to-3D methods because optimization starts from the amortized network rather than a randomly initialized network.Source: §3.5 · p. 8

3.6 3D Stylization

  • In 3D stylization, the model keeps point-cloud input and amortizes across multiple style prompts per shape, producing variations that remain structurally similar to the source shape.At inference, training-set point clouds can be paired with novel prompts; the paper reports lower per-pair computational cost than optimizing each shape-prompt pair separately.Source: §3.6 · p. 8

4 Experiments

The experiments evaluate LATTE3D’s scale, quality, generalization, and speed against baselines, then test applications and ablate design choices.

4.1 Experimental Setup

  • The experiments compare qualitative outputs across a training prompt and two unseen prompts, and LATTE3D is also evaluated for user preference over optimization time.The user-preference study compares MVDream at different optimization times with Latte3D on seen prompts.Source: Figure 6 · p. 9, Figure 7 · p. 11

Datasets

  • The gpt-101k training set contains 101k prompts and 34k filtered shapes; evaluation uses 50 seen prompts and 67 unseen DreamFusion prompts.Three augmented prompts are gathered per shape.Source: Datasets · p. 9

Baselines and Evaluation Schemes

  • Baselines include ATT3D, MVDream, Instant3D, and LGM; evaluation covers Render-FID, CLIP Score, user preference, and GPU time per prompt.User preference averages three users’ comparisons across prompts.Source: Baselines and Evaluation Schemes · pp. 9–10

4.2 Experimental Results

  • At least one order of magnitude faster inference, Latte3D remains competitive with baselines on seen and unseen prompts.The comparison includes quantitative metrics and average user preference for the gpt-101k model.Source: Table 2 · p. 10, §4.2 · p. 11
  • 10× lower optimization cost: MVDream requires 36 GPU minutes per prompt to match Latte3D’s preference, versus Latte3D’s 215 GPU seconds.The comparison is for gpt-101k and uses user preference as the match criterion.Source: Total optimization cost · p. 11

4.3 Application

  • Test-time optimization improves quality, particularly on unseen prompts, while animal-style training supports stylized variations and generalizes to held-out combinations.For test-time optimization, unseen-prompt FID drops by 11.6 versus 8.8 for seen prompts, and CLIP gains are 0.04 versus 0.02; animal-style Latte3D is competitive with MVDream at 10× lower optimization cost.Source: Per Prompt Test-time Optimization · p. 11, Stylizing 3D content · p. 12

4.4 Ablation Studies

  • MVDream guidance, the LATTE3D architecture, reconstruction pretraining, and shape regularization each improve ablation performance; stage-2 refinement substantially improves texture details.Shape regularization improves geometry adherence, evidenced by a drop in Mask-FID.Source: §4.4 · p. 13, Figure 9 · p. 13
  • Higher regularization blend factors improve consistency with the input shape, while point-cloud annealing reduces sensitivity to dummy inputs at inference.Without annealing, dummy input causes a small performance drop versus retrieved point clouds; annealing reduces that gap.Source: §4.4 · p. 13, Figure 10 · p. 13, Table 4 · p. 13

5 Conclusion and Limitation

LATTE3D scales amortized text-to-3D generation using 3D data and reports 400 ms generation, with further quality gains possible through test-time optimization.

5 Conclusion and Limitation

  • The approach combines 3D-aware 2D SDS, pretraining, 3D regularization, and a more scalable architecture for amortized generation.The conclusion presents these as the means used to scale amortization to larger datasets.Source: §5 · p. 14
  • LATTE3D generates high-quality shapes within 400 ms, and test-time optimization can further improve quality and generalization.Source: §5 · p. 14
  • Stage-2 and test-time optimization freeze geometry, so neither can fix geometry flaws from stage-1; SDS also relies on text-to-image model understanding of fine-grained details.The paper identifies stable geometry changes during stage-2 as future work.Source: §5 · p. 14

Disclosure of Funding

The funding disclosure identifies NVIDIA funding and scholarship support for five authors.

Disclosure of Funding

Supplementary Material

The supplement outlines additional methodology, experiments, limitations, and controllability work. It explains amortized optimization and how LATTE3D applies it to text-to-3D generation.

Supplementary Material

A.1 Background on Amortized Learning

  • Amortized optimization trains a network to map sampled problem contexts to solutions, sharing optimization effort instead of solving each instance independently.The approach can be more efficient than generating a dataset of solved instances and training a regressor.Source: §A.1 · p. 20
  • LATTE3D amortizes text-to-3D optimization by using the prompt embedding as context and the generated 3D object as the solution.The repeated per-prompt problem uses the SDS loss to generate an object consistent with the prompt.Source: §A.1 · p. 20

A.2 Model Architecture

The architecture combines point-cloud and text-conditioned triplane processing with separate geometry and texture prediction. The supplement also describes training, test-time optimization, dataset construction, and baseline settings.

A.2 Model Architecture

  • The geometry network encodes reference point clouds into triplanes, combines them with text embeddings in U-Nets, and predicts an SDF for rendering or surface extraction.The texture network shares the U-Net architecture and upsamples its triplanes in stage 2; geometry is frozen because its upsampling destabilized training.Source: Geometry Network · pp. 20–21, Texture Network · p. 22

A.3 Details on Amortized Learning

  • Training uses reconstruction pretraining, then stage 1 with four views per shape and stage 2 with ten views per shape; late stage-1 annealing replaces input points with dummy points.For reported results, shape-regularization strength varies by dataset: α = 0.2 for animal-real, 0.1 for animal-style, and 0.05 for gpt-101k.Source: §A.3 · pp. 22–23

A.4 Test Time Optimization

  • Test-time optimization lowers the learning rate to 0.0001 and anneals the sampled timestep range from [0.5, 0.9] to [0.1, 0.5] over 400 iterations.It samples eight training views per iteration and otherwise uses stage-2 architecture and guidance settings.Source: §A.4 · p. 23

B Additional Experiments

  • The supplement provides additional experimental details and results, including comparisons with baselines and ablations.The cited passage introduces this additional-experiments section without reporting its outcomes.Source: B Additional Experiments · p. 23

Details on Baselines

  • Baseline settings include 10,000-iteration SDS fitting for MVDream and ATT3D comparisons using either its original hypernetwork or the triplane architecture.3DTopia's reported 20 seconds covers diffusion sampling, while LGM's six seconds covers MVDream sampling and Gaussian reconstruction.Source: Details on Baselines · pp. 23–24

Dataset

  • Prompt sets derive from object captions, while the curated shape set contains 380,000 filtered Objaverse shapes and 100 manually selected animal shapes per animal dataset.For gpt-101k, three target prompts are gathered per caption; shapes are normalized and rendered from 32 cameras to produce images and colored point clouds.Source: Dataset · pp. 24–25

Evaluation Metrics

The evaluation uses image-fidelity, text-alignment, geometry-mask, and user-preference measures. Qualitative comparisons report that stage-2 amortized refinement improves reproduced ATT3D results.

Evaluation Metrics

  • Render-FID compares generated-shape renderings with Stable Diffusion images, while CLIP scores measure alignment between prompt text and rendered views.Mask-FID compares binary masks of generated shapes with masks from the 3D dataset.Source: Evaluation Metrics · pp. 25–26
  • The user study presents paired method videos with prompts and asks users to judge appearance, naturalness, detail, appeal, and textural faithfulness.Each comparison uses 200 prompts per method and is shown to three users, with video order randomized.Source: Evaluation Metrics · p. 26

B.2 Validating ATT3D Reproduction

  • The ATT3D reproduction with triplane-plus-U-Net architecture and stage-1 training alone achieves quality similar to the source ATT3D on DF27.DF27 contains 27 prompts from DreamFusion reused in ATT3D.Source: §B.2 · p. 26
  • Stage-2 amortized refinement improves stage-1 quality by an amount described as comparable to Magic3D's improvement over single-stage methods.The refinement is called MagicATT3D or ATT3D+S2 and does not use 3D data.Source: §B.2 · p. 26

B.3 Additional Results

Additional results include qualitative generalization examples, comparisons with Instant3D, and test-time optimization visualizations. Animal-style results report virtually no degradation on unseen prompts.

B.3 Additional Results

  • Animal-style generations show virtually no performance degradation on unseen prompts, corroborating quantitative results and indicating combinatorial generalization.For gpt-101k, DreamFusion prompts are described as generally further out of distribution than additional prompts suggested from training prompts.Source: §B.3 · p. 27
  • The supplement adds qualitative side-by-side comparisons with Instant3D on 65 DreamFusion prompts selected as closer to the gpt-101k training distribution.Instant3D's reported generations for DreamFusion prompts are used in the comparison.Source: §B.3 · p. 27
  • Test-time optimization examples compare outputs after 0, 60, 120, 300, and 600 steps.The zero-step column is the unoptimized output.Source: §B.3 · p. 27

B.4 Additional Ablation Studies

The ablations examine stage-1 guidance, stage-2 guidance, and stage-2 upsampling. They report trade-offs in prompt alignment, geometry, and texture fidelity.

Additional results for ablations on stage-1

  • The appendix reports that the ATT3D ablation outperforms adding MVDream on CLIP score, while the authors associate SDS’s higher Janus-face rate with worse geometry.They cite higher Mask-FID as evidence of geometric errors.Source: §B.4 · p. 27

Ablation on the upsampling layer in stage-2

  • Upsampling the stage-2 latent triplane significantly improves Render-FID while retaining a strong CLIP-Score.The ablation uses α = .9 on realistic animals.Source: §B.4 · p. 28, Table B.4 · p. 28

Ablation on stage-1 guidance choices

  • Stage-1 MVDream avoids Janus-face artifacts that can arise with Stable Diffusion and propagate into stage 2.The authors note that reference-geometry regularization is especially important with Stable Diffusion.Source: §B.4 · p. 28

Ablation on stage-2 guidance choices

  • Depth-SD and Stable Diffusion perform better than MVDream and VSD for stage-2 guidance; Depth-SD is preferred for more faithful textures.Depth-SD and Stable Diffusion are close on Render-FID and CLIP score; the user study favors Depth-SD.Source: §B.4 · p. 28, Table B.5 · p. 28

C Failure Cases and Limitations

The appendix shows failure cases and supplementary comparisons across refinement, guidance, generalization, controllability, and test-time optimization. The authors identify missed details in composed prompts, multi-object prompts, and thin geometry as limitations.

Failure cases and limitations

  • GPT-101k-trained models can miss details in composed prompts and generate only one object for multi-object prompts; thin features may be lost when stage-1 geometry is frozen.The authors expect greater prompt diversity to reduce these failures and suggest stabilizing stage-2 geometry training to recover some thin details.Source: C Failure Cases and Limitations · p. 29, Figure C.6 · p. 33

Additional comparisons

  • Stage-2 refinement improves the ATT3D reimplementation’s quality without requiring 3D data, while the reimplementation is similar in quality to source ATT3D.The comparison shows source ATT3D, a triplane proxy after surface extraction, and the refined version.Source: Figure B.2 · p. 30

Guidance comparisons

  • For prompts where stage-1 SDS produces Janus-face artifacts, MVDream’s 3D-aware guidance avoids them, while depth conditioning addresses Janus faces in stage-2 textures.The stage-1 artifacts persist through stage-2 refinement.Source: Figure B.4 · p. 31, Figure B.5 · p. 32

Training dynamics and generalization

  • The reported training-to-test gap is small at the end of training; Render-FID improves through training with a large jump from stage 1 to stage 2.Mask-FID improves at the start of stage 2, and stronger shape regularization often lowers Mask-FID.Source: Figure C.11 · p. 38

Additional qualitative results

  • Stage-2 upsampling helps the model learn high-frequency details, with quantitative assessments reported in Table B.4.The appendix also shows additional results on unseen in-distribution prompts, animal prompts, and text-to-3D comparisons with Instant3D.Source: Figure C.7 · p. 34, Figure C.8 · p. 35, Figure C.9 · p. 36
  • The appendix includes examples of blending settings and test-time optimization, with the latter compared against the 400ms result.The figure caption identifies the time budget for each optimized result but gives no outcome comparison.Source: Figure C.18 · p. 45, Figure C.19 · p. 46

D Additional New Capability: User-controllability via Amortizing the Regularization Strength

The method amortizes training across regularization strengths so users can choose α at inference. This enables interpolation between an input 3D object and a text-prompt result, with a small reported quality and compute impact.

Amortizing the regularization strength

  • Amortizing across α allows users to select the loss weighting dynamically at inference with minimal quality loss.The model is trained on multiple uniformly sampled α values, which are also given as network inputs.Source: D Additional New Capability:… · p. 47

User-controllability

  • Blending the input object’s RGB loss with text guidance yields an interpolation from the source 3D object to a text-prompt object across α ∈ [0, 1].The interpolation is a 3D animation, with a separate 3D object for each α.Source: D Additional New Capability:… · p. 47

Amortizing the regularization strength

  • Amortized weighting achieves comparable quality to training separate models at fixed α values, while using one third of the compute in the illustrated comparison.The method makes α a network input and samples it during training.Source: Figure D.20 · p. 47

Related papers

Loading 2403.15385v1…