PaperlayerView exact revision on arXiv

Source-linked AI summary

Single Image to Textured 3D Object Generation in Frequency Domain: From Theory to Pipeline

Qisen Wang, Yifan Zhao, Jia Li

arXiv:2609.07085v1cs.CV

TL;DR

Single-view 3D reconstruction must infer missing 3D information, but existing 2D and 3D diffusion priors respectively suffer from view/color inconsistency and missing high-frequency detail. The paper introduces frequency-domain hybrid optimization and Morpheus3D, whose high-pass 2D guidance improves generation quality while suppressing these problems. The authors report that high-pass guidance outperforms full-pass and no-2D-guidance alternatives in ablations, while fixed prior weights and filtering rates remain heuristic limitations.

  • Problem

    2D diffusion priors suffer from view inconsistency and color deviation, while 3D priors lack high-frequency details; spatial-domain combination introduces erroneous low-frequency guidance.

  • Method

    The paper proposes frequency-domain hybrid optimization and a two-stage Morpheus3D pipeline that enhances full-pass 3D guidance with high-pass image-prompt 2D guidance.

  • Results

    High-pass 2D prior guidance takes a clear margin over no-pass and full-pass guidance on novel-view quality and semantic consistency in ablations.

  • Takeaways & Limitations

    Frequency-selective combination of 3D and image-prompt 2D priors can address complementary consistency, color, and high-frequency-detail problems in single-image 3D generation.

  • Takeaways & Limitations

    The method uses handcrafted, fixed prior weights and frequency filtering rates that require heuristic design.

Abstract

from arXiv · show

Single-view 3D reconstruction, also known as image-to-3D, is a persistently challenging task due to the extreme lack of information. Recently, diffusion models pre-trained on large-scale datasets served as 2D priors are used to solve the ill-posed task but suffer from color deviation and view inconsistency, which can be curbed by using diffusion models fine-tuned with 3D annotated data served as 3D priors. However, 3D priors lack high-frequency details, which cannot be solved by direct complementation with 2D priors in spatial domain for introducing erroneous low-frequency 2D prior guidance. In this paper, we revisit the characteristics of different diffusion priors from the frequency perspective. Based on our observations, we theoretically present a unified framework of hybrid optimization using multiple diffusion priors in frequency domain. Under this framework, we further propose Morpheus3D, a pipeline of 3D object generation from any single unposed image in the wild. Morpheus3D enhances 3D prior with high-pass image-prompt 2D prior guidance to reconstruct high-quality 3D objects while effectively suppressing view inconsistency, low-frequency color deviation, and high-frequency lacking problems. Both quantitative and qualitative experiments on the public and our collected datasets with complex textures show that our method exhibits significant improvements in generation quality.

1 Introduction

Single-view 3D reconstruction is ill-posed because one image lacks sufficient 3D information, while existing diffusion priors trade off object consistency, color fidelity, and high-frequency detail. The paper introduces a frequency-domain hybrid framework and Morpheus3D, which combines 3D and high-pass image-prompt 2D guidance in a two-stage pipeline.

  • Single-view reconstruction is difficult because a single image provides insufficient 3D information.
  • 2D priors can produce color deviation and view inconsistency, whereas 3D priors improve structural consistency but lack texture details.
  • Spatial-domain blending of 2D and 3D priors improves structures and textures but still leaves color deviation and the Janus Problem.
  • The paper asks whether score-distillation gradients and multiple diffusion priors can be integrated in a decoupled frequency-domain optimization framework.
  • Morpheus3D uses 3D guidance for shape and coarse textures, then high-pass image-prompt 2D guidance to enhance high-frequency details.
  • The paper analyzes diffusion priors from a frequency perspective and presents a unified hybrid optimization framework for controlling their guidance.

2 Related Work

Single-view 3D reconstruction methods face limited geometric and texture information, while the scarcity of 3D data constrains generalization to out-of-domain, in-the-wild images. Related work increasingly uses large pretrained visual and diffusion models as priors.

  • Single-view 3D reconstruction is challenging because learning from one image provides imprecise geometric structures and textures.
  • Early methods focused on specific object classes or object-centric datasets, while newer feed-forward methods benefit from expanding 3D data.
  • 3D data remains much scarcer than 2D data, limiting feed-forward models on out-of-domain and in-the-wild single images.
  • Large pretrained visual models are used to extract implicit 3D information when 3D annotation data is limited.
  • Pretrained diffusion models have become priors for zero-shot downstream tasks, but text-prompt priors can provide unrealistic guidance because textual semantics are ambiguous.

3 Preliminaries

Diffusion models learn to denoise noisy data toward a learned data distribution, and score distillation transfers diffusion-prior information to 3D representation optimization. Variational Score Distillation formulates this transfer through distribution matching and gradient flow.

  • 3.1 Diffusion Models: Diffusion models add noise in a forward process and learn a reverse process that progressively denoises Gaussian noise into clean data.
  • 3.1 Diffusion Models: The diffusion objective trains a noise-prediction model by minimizing a weighted or simple squared error between predicted and actual noise.
  • 3.1 Diffusion Models: The diffusion score is related to the predicted noise, enabling pretrained diffusion models to serve as priors for single-view 3D reconstruction.
  • 3.2 Variational Score Distillation: Score distillation extracts 3D information from diffusion priors to guide the optimization of 3D representations.
  • 3.2 Variational Score Distillation: Variational Score Distillation models distributions of 3D representations and rendered images, then expresses optimization through a weighted KL-divergence objective.
  • 3.2 Variational Score Distillation: VSD solves the distribution-matching objective through gradient flow using predicted noise from the prior and fine-tuned model.

4 Method

The method analyzes diffusion priors in the frequency domain and combines their complementary properties through a unified hybrid optimization framework. Morpheus3D uses full-pass 3D-prior guidance for low-frequency characteristics and high-pass image-prompt 2D guidance for high-frequency details.

  • Analysis of Different Diffusion Priors: Text-prompt 2D priors provide high-frequency details but can introduce Janus-style view inconsistency because text has high information entropy.The method therefore avoids text-prompt priors for reconstruction.
  • Analysis of Different Diffusion Priors: Image-prompt 3D priors provide strong 3D consistency but lack high-frequency details and tend to be over-smoothed.The paper identifies this limitation through power spectral density analysis and reports that VSD does not fully overcome it.
  • Hybrid Optimization using Multiple Diffusion Priors with VSD in Frequency Domain: The unified framework represents rendered and reference images in the frequency domain and applies weighted guidance from multiple diffusion priors.It extends VSD from spatial to frequency-domain optimization, with spatial-domain multi-prior optimization appearing as a special case.
  • Analysis of Different Diffusion Priors: Image-prompt 2D priors use Canny edges from coarse renderings as implicit camera-pose conditions, preserving object consistency while introducing low-frequency color deviation.PFD is used as the image-prompt 2D prior through ControlNet conditioning.

5 Experiments

Experiments evaluate Morpheus3D on Realfusion15 and the more challenging MorpheusObj30, compare reconstruction quality and efficiency, and ablate frequency-domain design choices. Results support high-pass 2D guidance and fine-tuning as important for semantic consistency, texture detail, and novel-view quality.

  • Datasets and metrics: Morpheus3D is evaluated on Realfusion15 and MorpheusObj30 using reference-view and novel-view reconstruction metrics.The evaluation includes PSNR and LPIPS for reference views, and MANIQA and CLIPIQA for novel views.
  • Quantitative comparison: +0.195 CLIP-Similarity and +0.157 CLIPIQA are reported over SyncDreamer and TRELLIS, respectively, on Realfusion15.The reported gains correspond to 30.33% semantic-consistency improvement and 26.57% generation-quality improvement.
  • Ablation study: High-pass 2D prior guidance outperforms null-pass and full-pass guidance on novel-view generation quality and semantic consistency.The ablation evaluates MANIQA, CLIPIQA, and CLIP-Similarity, while full-pass guidance introduces heavier color deviation.
  • Ablation study: Fine-tuning the 2D prior overcomes SDS oversmoothing, boosting high-frequency details and improving novel-view generation quality and semantic consistency.Without fine-tuning, high-pass guidance remains close to the Base condition; training-process PSD measurements show increased high-frequency content with fine-tuning.

5.5 Study of fine-tuning 3D prior

The supplied passage introduces a question about why VSD cannot be used directly, but does not provide the study's analysis or conclusion.

  • Study motivation: The section raises why VSD cannot be used directly after the preceding experiments, without stating an answer in the supplied passage.No further evidence is provided for the study's findings.

5.6 Computational Costs

Morpheus3D is reported as efficient among optimization-based methods and can refine feed-forward assets, improving their texture details. The supplied passages also contrast optimization-based and feed-forward generation times.

  • Computational comparison: Morpheus3D achieves the highest reported performance among optimization-based methods while consuming the least training time in that comparison.The method avoids textual inversion and uses a coarse-to-fine two-stage pipeline.
  • Computational comparison: Feed-forward methods often generate results in minutes, whereas optimization-based methods often require hours.Despite this broader speed difference, Morpheus3D is reported as the most efficient among the optimization-based methods.
  • Feed-forward refinement: Applying Morpheus3D to TRELLIS-generated assets improves generation quality and visibly enhances texture details.The pre-generated textured mesh replaces the standard Instant-NGP NeRF initialization before subsequent optimization.

6 Conclusion and Limitation

The paper unifies multiple diffusion priors through frequency-domain hybrid optimization and proposes a two-stage pipeline for single-image 3D generation. It reports suppression of view inconsistency, low-frequency deviation, and high-frequency deficiency, while acknowledging heuristic parameter design as a limitation.

  • Conclusion: The framework addresses text-prompt 2D prior entropy, image-prompt 2D color deviation, and image-prompt 3D high-frequency deficiency.It uses frequency-domain hybrid optimization to extract complementary advantages from multiple priors.
  • Conclusion: Morpheus3D is a two-stage pipeline for generating 3D objects from a single unposed image in the wild.The reported scope includes suppression of view inconsistency, low-frequency deviation, and high-frequency lack.
  • Limitation: The prior weights and frequency-bound filtering rate are handcrafted and fixed, requiring heuristic design.The authors state that these parameters are nevertheless quite robust for generation quality.

A More Results

The section recommends reviewing additional visual results for Morpheus3D through the project webpage or supplementary video.

  • The authors strongly recommend watching more visual results.
  • The Morpheus3D project webpage is provided as one viewing option.
  • The supplementary-material video is provided as another viewing option.

B Proof

This section introduces Parseval’s Theorem and a global-optimum result for VSD as proof background.

  • Parseval’s Theorem states that total signal energy is identical in spatial and frequency domains.
  • The theorem is expressed for a signal x(t) and its spectral representation u(f).
  • The section also states a global-optimum result for VSD for each t > 0.

B.2 Proof of Proposition 1

The proof develops Proposition 1 by rewriting the optimization objective, applying ELBO-related transformations, and deriving its gradient flow in Wasserstein space.

  • The proof begins from the optimization object and rewrites it into an equivalent form.
  • The objective is transformed through an upper-bound optimization aligned with ELBO and the VSD construction.
  • The weighting function ω(t) is time-dependent, while αt and σt are degradation coefficients.
  • The resulting distribution µ(θ|y) is optimized in the 2-Wasserstein space W2(Θ), yielding a gradient flow.
  • The derivation uses the Fokker–Planck formulation to obtain an intermediate equation before the final gradient-flow expression.

B.3 Proof of Proposition 2

The proof establishes frequency-domain counterparts of the VSD optimum and diffusion score, then derives the frequency-domain optimization and connects full frequency passage to the spatial-domain case.

  • The section states a global optimum of VSD in the frequency domain for each t > 0.
  • It introduces a frequency-domain conversion of the diffusion score function from ∇xt log pt(xt).
  • The proof analyzes each frequency component ui using diagonal matrices Λi whose entries consist of 0 and 1.
  • The frequency-domain optimization objective is transformed through a gradient-flow derivation based on the frequency-domain score relation.
  • The proof reparameterizes x0 as y + σϵ with ϵ ∼ N(0, I) and derives the corresponding frequency variables.
  • When Λi = I, the resulting gradient flow is equivalent to the spatial-domain expression.

B.4 Proof of Proposition 3

The section derives the frequency-component guidance and gradient flow for the weighted optimization objective, then gives implementation details for the theoretical framework and two-stage pipeline. It also specifies camera, training, and reproduction settings.

  • B.4 Proof of Proposition 3: The proof decomposes the weighted optimization objective and its guidance into individual frequency components, following the earlier Proposition 2 derivation.The resulting component-wise gradient flow is assembled into the full objective.
  • Implementation assumptions: The implementation assumes shared diffusion hyper-parameters across priors, while allowing personalized objectives when priors use different settings.The shared-parameter assumption is motivated by current priors inheriting settings from Stable Diffusion.
  • Theoretical framework implementation: The framework’s implementation combines diffusion-prior gradients with frequency-domain transforms and weighted reconstruction losses before updating generator parameters.The pseudocode applies DCT and inverse DCT operations, filters frequency components, and accumulates weighted gradients.
  • Pipeline implementation: The two-stage pipeline first uses a 3D prior for reconstruction, then combines 2D and 3D priors while applying high-pass 2D guidance and updating a LoRA model.The second stage computes DCT-domain high-pass components for the 2D prior and retains the 3D prior gradient.
  • Experimental settings: Experiments use fixed camera and training settings, including unposed-image camera ranges, specified diffusion priors, regularization losses, and reproduction adjustments for competing methods.The reported setup includes elevation and azimuth ranges, fixed field of view, staged time schedules, and method-specific evaluation resolutions or views.
Loading 2609.07085v1…