PaperlayerView exact revision on arXiv

Source-linked AI summary

Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridging

Chongjie Ye, Yushuang Wu, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, Xiaoguang Han

arXiv:2503.22236v2cs.GRcs.CV

TL;DR

Existing image-to-3D methods have difficulty reproducing fine-grained geometry because RGB images are ambiguous and training data has domain and quality limitations. Hi3DGen bridges images and geometry through normal maps, combining specialized normal estimation, normal-regularized latent diffusion, and synthesized data. The paper reports rich geometric details and higher generation fidelity than state-of-the-art approaches.

  • Problem

    Existing methods struggle to reproduce fine-grained geometric details because of limited high-quality 3D data, domain gaps, and ambiguities in RGB images.

  • Method

    Hi3DGen uses normal maps as an intermediate representation, with NiRNE for sharp and stable normal estimation, NoRLD for normal-regularized latent diffusion, and DetailVerse for synthesized training data.

  • Results

    Hi3DGen generates rich, fine-grained geometric details and surpasses state-of-the-art approaches in generation fidelity.

  • Takeaways & Limitations

    Normal maps provide a supported intermediate representation for improving image-to-3D geometry generation fidelity and consistency with input images.

Abstract

from arXiv · show

With the growing demand for high-fidelity 3D models from 2D images, existing methods still face significant challenges in accurately reproducing fine-grained geometric details due to limitations in domain gaps and inherent ambiguities in RGB images. To address these issues, we propose Hi3DGen, a novel framework for generating high-fidelity 3D geometry from images via normal bridging. Hi3DGen consists of three key components: (1) an image-to-normal estimator that decouples the low-high frequency image pattern with noise injection and dual-stream training to achieve generalizable, stable, and sharp estimation; (2) a normal-to-geometry learning approach that uses normal-regularized latent diffusion learning to enhance 3D geometry generation fidelity; and (3) a 3D data synthesis pipeline that constructs a high-quality dataset to support training. Extensive experiments demonstrate the effectiveness and superiority of our framework in generating rich geometric details, outperforming state-of-the-art methods in terms of fidelity. Our work provides a new direction for high-fidelity 3D geometry generation from images by leveraging normal maps as an intermediate representation.

1. Introduction

Existing image-to-3D methods struggle to reproduce fine-grained geometry because of limited high-quality data, synthetic-to-real domain gaps, and RGB ambiguity. Hi3DGen addresses these challenges by bridging images and geometry with normal maps, specialized estimation and diffusion components, and synthesized training data.

  • Existing methods inadequately reproduce geometric details in real-world images, limiting realism, precision, and practical applicability.
  • Direct RGB-to-geometry learning is constrained by scarce high-quality 3D data, a synthetic-to-real domain gap, and ambiguities from lighting, shading, and textures.
  • Normal maps bridge RGB images and 3D geometry by providing surface-orientation information and clearer geometric cues than RGB images.
  • Hi3DGen combines NiRNE for noise-injected dual-stream normal estimation, NoRLD for normal-regularized latent diffusion, and DetailVerse data synthesis.
  • The framework targets robust, stable, and sharp normal estimation while generating rich, fine-grained geometry details.

2. Related Work

Prior work spans 3D dataset construction and normal-guided generation, but faces limitations in asset quality, coverage, licensing, texture availability, and normal-estimation sharpness or stability. Hi3DGen contributes a normal-bridging framework with noise-injected regression and normal-regularized diffusion learning.

  • Datasets for 3D Generation: Early 3D datasets cover limited object categories, while later repositories still contain quality, licensing, texture, and geometric-detail imbalances.
  • Normal Estimation: Regression-based normal estimation provides stable deterministic predictions but struggles with fine-grained sharp details.
  • Normal Estimation: Hi3DGen introduces a noise-injected regressive method with dual-stream training to support sharp and stable normal estimation across domains.
  • Normal Maps in 3D Generation: Normal maps are used in reconstruction, optimization, and multi-view generation pipelines to provide geometric cues and improve 3D accuracy or consistency.
  • Normal Maps in 3D Generation: Hi3DGen integrates normal supervision directly into diffusion learning of 3D latent codes to address limitations of prior methods.

3. Method

Hi3DGen bridges image-to-geometry generation through normal maps, combining dual-stream normal estimation, normal-regularized latent diffusion, and synthesized high-quality 3D data. Its method targets sharper normal predictions, more detailed geometry supervision, and richer training assets.

  • Framework overview: Hi3DGen divides image-to-geometry generation into image-to-normal estimation and normal-to-geometry mapping using normal maps as a 2.5D bridge.The framework includes a dual-stream normal estimator and an online normal regularizer for diffusion training.
  • 3.1. Noise-Injected Regressive Normal Estimation: Noise injection in the regressive estimator targets high-frequency information, where diffusion analysis indicates faster signal-to-noise degradation during noising.The method is motivated by sharper normal details at edges and cavities.
  • 3.1. Noise-Injected Regressive Normal Estimation: The dual-stream architecture uses a clean stream for low-frequency structure and a noise-injected stream for high-frequency features, decoupling representation learning.The design combines structural information associated with low frequencies and sharp-detail sensitivity associated with high frequencies.
  • 3.2. Normal-Regularized Latent Diffusion: Latent-only supervision can compress fine-grained geometry, motivating explicit geometry-space regularization for more precise surface-detail supervision.The method addresses detail loss and detail-level inconsistency in 3D latent diffusion.
  • 3.2. Normal-Regularized Latent Diffusion: Normal-regularized latent diffusion adds explicit normal-map supervision to the usual latent diffusion objective during training.The predicted clean latent is decoded into 3D geometry, rendered into normal maps, and compared with corresponding ground-truth normals online.
  • 3.3. DetailVerse Dataset: The DetailVerse construction pipeline synthesizes and cleans 3D assets, preserving 700k high-quality meshes with richer surface details.The pipeline uses text-to-image-to-3D generation followed by expert and automated assessment.

4. Experiments

Experiments evaluate Hi3DGen’s normal estimation, normal bridging, geometry generation, and training data across quantitative, qualitative, and user-study settings. Results support improved accuracy, sharpness, fidelity, and detail generation.

  • Experiment Setup: Hi3DGen is evaluated against regression-, diffusion-, and state-of-the-art 3D-generation methods using complementary datasets and metrics.Normal estimation uses NE and SNE; geometry evaluation renders normal maps from 22 viewpoints.
  • Image-to-Normal Estimation: NiRNE achieves significantly superior overall and sharp-region normal estimation performance on LUCES-MV.The comparison includes both regression-based and diffusion-based estimators.
  • Image-to-Normal Estimation: Qualitative comparisons show NiRNE combines generalizability, stability, and sharpness, with fewer wrong details than diffusion methods and sharper predictions than regression methods.Results cover human and object inputs.
  • Ablation Study: Normal bridging outperforms direct image-to-geometry generation, while smoother or incorrect normal conditions reduce final 3D quality.Using identical normal regularization and training data, direct generation produces fake details.
  • Ablation Study: DetailVerse improves NiRNE by 0.4 NE and 1.7 SNE, while additional normal-geometry pairs improve NoRLD’s generated details.The improvements are reported in Table 3 and Figure 10.
  • Ablation Study: Online normal regularization greatly improves generation fidelity with or without DetailVerse training data.The ablation visualizes the effect on roof details.

5. Conclusion

Hi3DGen bridges images and 3D geometry through normal maps, combining normal estimation, normal-regularized synthesis, and detail-rich synthetic data. The framework generates detailed results, but some details may remain inconsistent with the input.

  • Conclusion: Hi3DGen uses normal maps as a 2.5D intermediate representation for generating rich details consistent with input images.The framework divides image-to-geometry generation into image-to-normal estimation and normal-to-3D synthesis.
  • Conclusion: The framework combines robust, stable, and sharp normal estimation, normal-regularized geometry synthesis, and a detail-rich 3D data pipeline.These components support training and geometry consistency.
  • Limitations: Some generated details remain potentially inconsistent or nonaligned with the input because of the generative nature of 3D latent diffusion learning.The paper identifies reconstruction-level 3D generation as future work.

6. More Details for the Method

Additional method details describe noise-injected dual-stream normal estimation, inference preprocessing, sharp-region evaluation, and comparisons with photometric stereo methods. These details operationalize the framework’s stability and detail-preservation goals.

  • Image-to-Normal Estimation: The fine-grained encoder receives EDM-style noise during training, while the coarse encoder remains noise-free.Noise is injected into encoder-output features before decoder layers rather than back into the encoder.
  • Inference: During inference, Hi3DGen isolates, crops, pads, and resizes the foreground to 768×768 before NiRNE estimation.Inference omits feature noise injection to preserve captured detail information.
  • Comparisons: NiRNE is compared with regression-based, diffusion-based, and photometric-stereo normal estimators.The supplied comparison caption identifies LUCES-MV SNE evaluation and best-result marking.
  • Evaluation Metrics: Sharp Normal Error measures normal angle error within dilated Canny-detected salient edge regions, complementing whole-map Normal Error.SNE emphasizes locations where geometric details are most salient.

7. More Details for the DetailVerse

DetailVerse is constructed through prompt curation, image generation, orientation filtering, Trellis synthesis, and expert-guided mesh-quality assessment. The pipeline prioritizes semantic diversity, geometric fidelity, and clean training assets.

  • Prompt Curation: Text prompts are curated and standardized to control semantic diversity and enforce explicit geometric cues.Near-isometric viewpoints and CGI-rendered aesthetics are used to enhance synthesis fidelity.
  • Image Generation: Flux.1-Dev generates images selected for visual fidelity, rich detail, suitable viewpoints, and reconstruction-friendly styles.Generated images are ranked by sharp-pixel counts from Canny edge detection, retaining the top 50%.
  • Image Filtering: OrientAnything rejects images whose camera-to-canonical orientation deviation exceeds 60° to reduce geometry distortion.The filtering process preserves 1 million images for subsequent 3D synthesis.
  • 3D Synthesis: Trellis converts the prepared images into preliminary high-fidelity meshes for the dataset.The generator is selected for its performance with high-quality inputs.
  • Quality Filtering: Ten trained experts triple-blind-assess 10K sampled meshes for surface holes and noise artifacts.Their annotations train a DINOv2-feature quality classifier applied across the dataset.
  • Quality Filtering: Meshes classified positively across all four rendered views are selected for NoRLD training.The quality network uses four equiangular normal-map views per mesh.

8. More Ablation Studies

Qualitative ablations show that each component of NiRNE contributes positively to Hi3DGen’s final performance.

  • 8. More Ablation Studies: Each NiRNE component makes a positive contribution to the final performance.The qualitative ablation results are shown in Fig. S11.

9. More Results

Additional qualitative comparisons show that normal-bridged Hi3DGen produces detailed 3D geometries that are more consistent with input images than existing methods.

  • 9. More Results: Hi3DGen generates 3D detailed geometries that are more consistent with input images than existing methods.Additional qualitative comparisons are presented in Fig. S14.
  • 9. More Results: The supplementary material provides more DetailVerse data exhibitions.Figure S13 presents additional DetailVerse data.
  • 9. More Results: The supplementary material provides additional 3D generation result comparisons.Figure S14 presents more generation comparisons.
Loading 2503.22236v2…