EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior

1. Introduction

• Text-to-3D generation is difficult because it requires 3D consistency across a larger output space, while paired text–3D training data are scarce.Diffusion models have advanced text-to-image generation, but those conditions handicap effective 3D generation.Source: §1 · p. 1
• Pretrained 2D diffusion models may misread view instructions, producing inaccurate guidance that can destabilize 3D creation and contribute to the Janus problem [1] [16].The paper identifies this view-instruction failure as the primary cause of Janus.Source: §1 · pp. 1–2
• EfficientDreamer fine-tunes a 2D diffusion model on Objaverse renderings to generate mutually consistent composite images from four orthogonal viewpoints.These images provide geometric structure priors while retaining the pretrained model’s capacity for detailed and diverse objects.Source: §1 · p. 2
• The 3D synthesis fusion network combines orthogonal-view and pretrained 2D diffusion priors, gradually shifting guidance from geometry toward local texture and appearance.The orthogonal-view prior receives greater weight early, then its weight is reduced as the object becomes more complete.Source: §1 · p. 2

2.1. 3D Reconstruction with Neural Fields

• NeuS transforms an SDF field for volume rendering, with rendering weights peaking at the first outside-to-inside intersection; the paper adopts NeuS for mesh extraction.The section contrasts neural-field methods with supervised multi-view stereo, which can struggle in less constrained scenarios.Source: §2.1 · pp. 2–3

2.2. Text-to-image Generation

• Text-to-image diffusion advances through large image–text datasets, latent-space sampling in Stable Diffusion, cascaded super-resolution, and condition-controlled architectures.The section notes that pretrained diffusion models’ generative ability makes them convenient priors for text-to-3D mesh synthesis.Source: §2.2 · p. 3

2.3. Text-to-3D Generation

• Early text-to-3D methods use CLIP supervision but often produce poor geometry; later methods use diffusion priors, coarse-to-fine optimization, disentangled geometry and appearance, or VSD.DreamFusion introduces SDS with a pretrained text-to-image prior; subsequent methods include Magic3D, Fantasia3D, TextMesh, and ProlificDreamer.Source: §2.3 · p. 3

3.1. Orthogonal-view Diffusion Model

• The orthogonal-view model is fine-tuned to generate a 2×2 composite of four orthogonal views, using Objaverse renders arranged at 90-degree azimuth intervals.The training objective fine-tunes the denoiser to predict noise from the prompt-conditioned noisy latent of each composite image.Source: §3.1 · pp. 3–4
• The orthogonal-view model is trained on approximately 420K filtered Objaverse assets, with 48 rendered 512×512 images per asset.The assets are normalized to a unit cube; renders use four listed elevation angles and uniformly sampled azimuths.Source: §3.1 · p. 3
• The trained model aligns corresponding parts across viewpoints, including for objects absent from its training data.Figure 3 contrasts its composite orthogonal-view outputs with Stable Diffusion, which struggles with specific viewpoint instructions.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2. Text-to-3D viathe3DSynthesisFusionNetwork

• Four camera views are tiled and passed to orthogonal-view SDS for scene updates; reducing the noise timestep during training is used to enhance stability.The paper states that supervision from four orthogonal views provides semantic structure guidance from different viewpoints.Source: §3.2 · pp. 4–5
• The fusion strategy weights orthogonal-view and pretrained diffusion priors, emphasizing the former initially and shifting toward the latter as generation progresses.The orthogonal-view prior targets consistent geometry; the pretrained 2D prior is incorporated to address potential local-appearance artifacts from relying solely on orthogonal views.Source: §3.2 · p. 5

3.3. Implementation Details

• Text-to-3D generation uses NeuS for a 5,000-step coarse stage, followed by DMTet and another 5,000 steps for textured-mesh refinement.The refinement stage uses SDS and VSD guidance.Source: §3.3 · p. 5

4.1. Comparison with State-of-the-arts

• Our method produces higher-fidelity 3D models without Janus and performs better than state-of-the-art methods on CLIP and FID.Table 1 compares methods using CLIP score and FID.Source: Figure 4 · p. 6, §4.1 · p. 6, Table 1 · p. 6

4.2. Comparison with Perp-Neg

• Perp-Neg can reduce multi-face errors but may leave details such as three ears or five legs, whereas orthogonal-view guidance avoids these issues.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3. Coarse-to-fine Two-stage Optimization Results

• Coarse optimization uses orthogonal-view structure priors to address Janus; fine optimization initializes from that stage and applies SDS to geometry and VSD to texture.Source: §4.3 · p. 7, Figure 6 · p. 7

4.4. User Study

• In a 50-participant study of 22 prompts, our method received the highest average rating, and 84.27% preferred it over all competing methods.Source: §4.4 · p. 7, Table 2 · p. 7

4.5. Ablation Study

• The fusion-network ablation found that orthogonal-view-only guidance can leave local distortions and roughness, while the fused guidance produces complete, smooth meshes and eliminates possible Janus problems.Source: §4.5 · p. 7
• Four-view supervision mitigates the residual multi-face problems and local distortions seen with reduced view counts; reducing supervision also decreases quantitative performance.Source: §4.5 · p. 8, Figure 8 · p. 8, Table 3 · p. 8

• The orthogonal-view diffusion model creates composite images of sub-images from different orthogonal viewpoints, providing geometric structure priors for 3D generation.Source: §5 · p. 8
• A 3D synthesis fusion network combines orthogonal-view and pretrained diffusion priors, while a dynamic strategy balances their guidance during SDS optimization.Source: §5 · p. 8
• The method effectively addresses the Janus problem while ensuring fidelity of generated 3D models.Source: §5 · p. 8