PaperlayerView exact revision on arXiv

Source-linked AI summary

Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model

Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, Sai Bi

arXiv:2311.06214v2cs.CV

TL;DR

Existing text-to-3D systems are either slow and artifact-prone optimization methods or low-quality feed-forward methods constrained by scarce 3D data. Instant3D generates four consistent views with a fine-tuned 2D diffusion model, then directly reconstructs a NeRF with a transformer-based sparse-view model. It produces diverse, high-quality assets in about 20 seconds, over 200× faster than optimization-based methods, while achieving comparable or better quality.

  • Problem

    Existing methods suffer from slow optimization, low diversity, Janus artifacts, or low quality caused by scarce 3D training data.

  • Method

    Instant3D fine-tunes a 2D text-to-image diffusion model to generate four consistent views, then uses a transformer-based sparse-view reconstructor to regress a NeRF.

  • Results

    About 20 seconds yields high-quality and diverse 3D assets, with more than 200× speedup over optimization-based methods and comparable or better quality.

  • Takeaways & Limitations

    Instant3D supports fast 3D asset creation, prototyping, and iteration for applications including 3D design and modeling.

  • Takeaways & Limitations

    CLIP scores can overlook artifacts from inconsistent views, and the reported SD1.5 versus SDXL comparison is not conclusive because SD1.5 performs poorly under the direct fine-tuning protocol.

Abstract

from arXiv · show

Text-to-3D with diffusion models has achieved remarkable progress in recent years. However, existing methods either rely on score distillation-based optimization which suffer from slow inference, low diversity and Janus problems, or are feed-forward methods that generate low-quality results due to the scarcity of 3D training data. In this paper, we propose Instant3D, a novel method that generates high-quality and diverse 3D assets from text prompts in a feed-forward manner. We adopt a two-stage paradigm, which first generates a sparse set of four structured and consistent views from text in one shot with a fine-tuned 2D text-to-image diffusion model, and then directly regresses the NeRF from the generated images with a novel transformer-based sparse-view reconstructor. Through extensive experiments, we demonstrate that our method can generate diverse 3D assets of high visual quality within 20 seconds, which is two orders of magnitude faster than previous optimization-based methods that can take 1 to 10 hours. Our project webpage: https://jiahao.ai/instant3d/.

1 INTRODUCTION

Instant3D addresses slow, artifact-prone, low-diversity text-to-3D optimization and limited 3D-data feed-forward methods with a two-stage feed-forward approach. It generates four consistent views, reconstructs 3D directly, and produces high-quality diverse assets in about 20 seconds.

  • Existing 2D-to-3D optimization methods improve visual quality and text alignment but require hours per prompt and can produce oversaturated colors, multi-face artifacts, and limited diversity.
  • Instant3D splits generation into sparse-view 2D generation and direct 3D reconstruction using pretrained 2D diffusion models.
  • The first stage fine-tunes a text-to-image diffusion model to generate four-view images in a 2×2 grid through one denoising process, encouraging view consistency.
  • About 20 seconds produces a high-quality, diverse 3D asset, reported as 200× faster than previous optimization-based methods with comparable or better quality.

2 RELATED WORKS

Prior 3D generation methods use varied representations and conditioning strategies but often generalize poorly, lack realism, or require slow optimization. Recent methods leverage 2D diffusion priors, while sparse-view reconstruction remains difficult because generated views can be inconsistent.

  • Earlier 3D generative methods model point clouds, meshes, or implicit representations but are usually trained on limited categories and generalize poorly to novel classes.
  • Methods that optimize meshes or NeRFs with CLIP can generate diverse content but exhibit deficient visual realism.
  • Score-distillation methods use pretrained 2D diffusion models to optimize 3D representations, but their optimization-based formulation remains slow and prone to artifacts.
  • One-2-3-45 generates 32 novel-view images from one input image, but the many generated views can be inconsistent during reconstruction.
  • Instant3D instead combines sparse-view generation with feed-forward NeRF reconstruction.

3 METHOD

Instant3D generates four structured views from text and feeds them to a transformer-based sparse-view reconstructor that predicts a NeRF. The method uses curated multi-view training data, Gaussian initialization, pose-aware image encoding, and triplane decoding.

  • 3 METHOD: Instant3D has two stages: sparse-view generation followed by feed-forward NeRF reconstruction.
  • 3.1 TEXT-CONDITIONED SPARSE VIEW GENERATION: The first stage fine-tunes a pretrained text-to-image diffusion model to produce a 2×2 grid of four mutually consistent views.
  • 3.1 TEXT-CONDITIONED SPARSE VIEW GENERATION: Four views balance reconstruction coverage against view inconsistency and reduced per-view resolution, while limiting the reconstructor’s need to hallucinate unseen parts.
  • 3.1 TEXT-CONDITIONED SPARSE VIEW GENERATION: Training data contains four 512 × 512 renderings of about 750K Objaverse objects, arranged at fixed elevation and four equidistant azimuths.
  • 3.1 TEXT-CONDITIONED SPARSE VIEW GENERATION: A scorer trained on 2,000 manually labeled samples selects the top 10K objects because using all data reduces photorealism and 3D asset quality.
  • 3.1 TEXT-CONDITIONED SPARSE VIEW GENERATION: Gaussian blob initialization guides inference toward clean white-background views, making the second-stage feed-forward reconstruction easier.
  • 3.2 FEED-FORWARD SPARSE-VIEW LARGE RECONSTRUCTION MODEL: The reconstructor encodes four posed images into tokens, decodes a triplane with transformer layers, and converts triplane features into NeRF density and color.
  • 3.2 FEED-FORWARD SPARSE-VIEW LARGE RECONSTRUCTION MODEL: The reconstruction model trains on all Objaverse objects with randomly sampled multi-view renderings rather than the first stage’s curated fixed-camera data.

4 EXPERIMENTS

Experiments compare Instant3D with prior text-to-3D and sparse-view reconstruction methods, then examine the effects of several design choices. Results show strong qualitative and CLIP-based performance, substantially faster inference, and sensitivity to backbone, initialization, data, and fine-tuning settings.

  • Text-to-3D comparisons: Instant3D produces sharper textures, better geometry, and stronger text-3D alignment than Shap-E in qualitative comparisons.Compared with DreamFusion and ProlificDreamer, it has more photorealistic appearance and better geometric details, while those methods show over-saturation, over-smoothing, low-quality geometry, or Janus artifacts.
  • Text-to-3D comparisons: Instant3D achieves higher CLIP scores than Shap-E and DreamFusion, with competitive scores to ProlificDreamer.The evaluation uses 400 DreamFusion text prompts, 10 random rendered views per model, and multiple CLIP variants.
  • Inference time: 20 seconds: Instant3D generates one asset on an A100, versus 1.5 hours for DreamFusion and 10 hours for ProlificDreamer.These correspond to 270× and 1800× speedups, respectively; reducing DDIM steps can further reduce inference time without obvious quality loss.
  • Ablation study: SDXL yields superior text comprehension and visual quality to SD1.5, while SD1.5 achieves consistently lower CLIP scores.The SDXL model is three times larger than SD1.5, and the comparison uses separately tuned hyper-parameters.
  • Ablation study: Gaussian blob initialization produces clean white backgrounds that better match the second-stage reconstructor, unlike the cluttered backgrounds often produced without it.The fine-tuned model can generate multi-view images without this initialization, but the resulting backgrounds challenge feed-forward reconstruction.
  • Ablation study: Curated data improves visual quality, while dataset size shows double descent: 1K harms view consistency and 100K harms compositionality, photo-realism, and text alignment.Curated versus randomly selected data produces a slight CLIP increase but a substantial qualitative improvement; the size comparison uses 1K, 10K, and 100K examples.
  • Ablation study: 10K fine-tuning steps maximize the final SDXL setup's CLIP score, while both 4K and 20K steps visibly degrade result quality.Scores do not increase monotonically with training steps, and the peak may move earlier for larger models, although that comparison is not conclusive.

5 CONCLUSIONS

Instant3D is a feed-forward two-stage method that generates high-quality, diverse 3D assets from text prompts within 20 seconds. It uses consistent four-view generation followed by transformer-based reconstruction and outperforms or matches prior methods across speed and quality.

  • 20 seconds: Instant3D generates high-quality and diverse 3D assets from text prompts in a feed-forward pipeline.
  • Instant3D fine-tunes a 2D text-to-image diffusion model to generate consistent four-view images, then reconstructs 3D with a transformer-based large reconstruction model.
  • Instant3D outperforms previous feed-forward methods in quality while remaining equally fast, and achieves comparable or better performance than optimization-based methods with more than 200× speed-up.
  • The model inherits generation capability from SDXL, while lightweight fine-tuning may preserve related ethical and legal considerations and author-preference data bias.

A.1 DIVERSITY OF GENERATION

Instant3D generates diverse 3D assets from one text prompt by varying random seeds. Its outputs can differ substantially in texture and geometry, unlike the similar results often produced by SDS-optimization methods.

  • Different random seeds produce diverse 3D assets from the same text prompt during Instant3D’s feed-forward pass.
  • The generated assets can have strikingly distinct textures and geometries despite sharing one prompt.
  • SDS-optimization methods are prone to generating similar results even with different initializations.

A.2 DATA CURATION DETIALS

Instant3D curates Objaverse data with a quality scorer before training. Curated data improves visual quality over random data, while the scorer uses limited manual labels and multi-view CLIP features.

  • A quality scorer filters Objaverse objects using 2,000 manually labeled assets categorized by realistic textures and complex geometry.
  • The scorer trains a binary SVM on averaged pretrained CLIP features extracted from five random camera viewpoints per object.
  • Curated objects are rendered in a structured four-view setup with a white background using Blender’s Cycles path tracer.
  • Curated-data models generate more realistic 3D assets, whereas random-data models tend toward cartoon-like assets with simple, flat textures.

A.3 SDXL FINE-TUNING DETAILS

The first-stage model uses SDXL with standard denoising diffusion fine-tuning and fixed image conditioning. Training uses fp16 on 32 A100 GPUs with a 10K curated dataset.

  • SDXL fine-tuning uses AdamW with learning rate 10^-5, β1 = 0.9, β2 = 0.999, and weight decay 10^-2.
  • Training uses fp16 on 32 NVIDIA A100 GPUs with total batch size 192 and no gradient accumulation.
  • The model is trained on 10K curated data for 40K steps, requiring around 3 hours.
  • The model is trained with the standard denoising diffusion loss.
  • Four 512×512 views are assembled into a 1024×1024 image, with resolution and aspect-ratio conditioning fixed to (1024, 1024).

A.4 SD1.5 FINE-TUNING DETAILS

SD1.5 is fine-tuned on 100K data using 8 A100 GPUs, gradient accumulation, and the AdamW optimizer.

  • SD1.5 is fine-tuned on 100K data using 8 A100 GPUs and a total batch size of 64.
  • Gradient accumulation of 3 steps produces an effective batch size of 192.
  • 120K training steps correspond to 40K parameter updates and require roughly 33 hours.

A.5 GAUSSIAN BLOBS INITIALIZATION

Gaussian blob initialization guides diffusion inference toward the 2×2 view-grid and white-background distribution used during fine-tuning. The initialization blends structured image latents with Gaussian noise before denoising.

  • Naively starting from random Gaussian noise produces outputs far from the fine-tuning distribution.
  • Fine-tuning images place four views in quadrant centers against white backgrounds, encouraging view-consistent denoising.
  • A grayscale image with a white background and central black Gaussian blob is constructed as the structured initialization image.
  • Four Gaussian-blob images are assembled into a 2×2 grid, with sigma controlling blob width.
  • The blob-image latent and an independent Gaussian-noise latent are blended using a weighted sum for initialization.
  • The blended latent is used as the initial denoising noise, with tN set to 980 for a 50-step process.

A.6 SPARSE-VIEW RECONSTRUCTION DETAILS

The sparse-view reconstructor encodes four images with a vision transformer and decodes their features into triplanes for NeRF rendering. Ablations show that LPIPS, rendering resolution, and novel-view supervision materially affect reconstruction quality.

  • Architecture: DINO-ViT-B/16 encodes each 512-resolution image into 1024 spatial tokens plus a [CLS] token.
  • Architecture: Camera information is integrated through identity-initialized modulation layers inserted into transformer attention and MLP layers.
  • Architecture: Tokens from four images form 4100 condition features used as keys and values in decoder cross-attention.
  • Architecture: The transformer decoder maps image features into three 64×64 triplanes, which represent XY, YZ, and XZ planes.
  • Architecture: Triplane features are concatenated into 240-dimensional point features and mapped by a 10-layer MLP to RGB colors and density.
  • Ablations: Without LPIPS loss, LPIPS decreases substantially while PSNR and SSIM improve slightly, and rendered images become blurry.
  • Ablations: Novel-view supervision is critical: supervising only the four input views significantly reduces all three metrics and causes floaters in uncovered regions.
  • Extensions: The method can accept text and an object image together for more fine-grained 3D-model control.
Loading 2311.06214v2…