PaperlayerView exact revision on arXiv

Source-linked AI summary

Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridging

Chongjie Ye, Yushuang Wu, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, Xiaoguang Han

arXiv:2503.22236v2cs.GRcs.CV

TL;DR

Existing image-to-3D methods have difficulty reproducing fine-grained geometry because RGB images are ambiguous and training data has domain and quality limitations. Hi3DGen bridges images and geometry through normal maps, combining specialized normal estimation, normal-regularized latent diffusion, and synthesized data. The paper reports rich geometric details and higher generation fidelity than state-of-the-art approaches.

  • Problem

    Existing methods struggle to reproduce fine-grained geometric details because of limited high-quality 3D data, domain gaps, and ambiguities in RGB images.

    Source: §1 · pp. 1–2

  • Method

    Hi3DGen uses normal maps as an intermediate representation, with NiRNE for sharp and stable normal estimation, NoRLD for normal-regularized latent diffusion, and DetailVerse for synthesized training data.

    Source: §1 · p. 2

  • Results

    Hi3DGen generates rich, fine-grained geometric details and surpasses state-of-the-art approaches in generation fidelity.

    Source: §1 · p. 2, §9 · p. 15

  • Takeaways & Limitations

    Normal maps provide a supported intermediate representation for improving image-to-3D geometry generation fidelity and consistency with input images.

    Source: §1 · p. 2, §9 · p. 15

Abstract

from arXiv · show

With the growing demand for high-fidelity 3D models from 2D images, existing methods still face significant challenges in accurately reproducing fine-grained geometric details due to limitations in domain gaps and inherent ambiguities in RGB images. To address these issues, we propose Hi3DGen, a novel framework for generating high-fidelity 3D geometry from images via normal bridging. Hi3DGen consists of three key components: (1) an image-to-normal estimator that decouples the low-high frequency image pattern with noise injection and dual-stream training to achieve generalizable, stable, and sharp estimation; (2) a normal-to-geometry learning approach that uses normal-regularized latent diffusion learning to enhance 3D geometry generation fidelity; and (3) a 3D data synthesis pipeline that constructs a high-quality dataset to support training. Extensive experiments demonstrate the effectiveness and superiority of our framework in generating rich geometric details, outperforming state-of-the-art methods in terms of fidelity. Our work provides a new direction for high-fidelity 3D geometry generation from images by leveraging normal maps as an intermediate representation.

1. Introduction

Existing image-to-3D methods struggle to reproduce fine-grained geometry because of limited high-quality data, synthetic-to-real domain gaps, and RGB ambiguity. Hi3DGen addresses these challenges by bridging images and geometry with normal maps, specialized estimation and diffusion components, and synthesized training data.

  • Existing methods inadequately reproduce geometric details in real-world images, limiting realism, precision, and practical applicability.Source: §1 · p. 1
  • Direct RGB-to-geometry learning is constrained by scarce high-quality 3D data, a synthetic-to-real domain gap, and ambiguities from lighting, shading, and textures.Source: §1 · pp. 1–2
  • Normal maps bridge RGB images and 3D geometry by providing surface-orientation information and clearer geometric cues than RGB images.Source: §1 · p. 2
  • Hi3DGen combines NiRNE for noise-injected dual-stream normal estimation, NoRLD for normal-regularized latent diffusion, and DetailVerse data synthesis.Source: §1 · p. 2
  • The framework targets robust, stable, and sharp normal estimation while generating rich, fine-grained geometry details.Source: §1 · p. 2

2. Related Work

Prior work spans 3D dataset construction and normal-guided generation, but faces limitations in asset quality, coverage, licensing, texture availability, and normal-estimation sharpness or stability. Hi3DGen contributes a normal-bridging framework with noise-injected regression and normal-regularized diffusion learning.

Datasets for 3D Generation

  • Early 3D datasets cover limited object categories, while later repositories still contain quality, licensing, texture, and geometric-detail imbalances.Source: §2 · p. 2

Normal Estimation

  • Regression-based normal estimation provides stable deterministic predictions but struggles with fine-grained sharp details.Source: §2 · p. 2
  • Hi3DGen introduces a noise-injected regressive method with dual-stream training to support sharp and stable normal estimation across domains.Source: §2 · p. 3

Normal Maps in 3D Generation

  • Normal maps are used in reconstruction, optimization, and multi-view generation pipelines to provide geometric cues and improve 3D accuracy or consistency.Source: §2 · p. 3
  • Hi3DGen integrates normal supervision directly into diffusion learning of 3D latent codes to address limitations of prior methods.Source: §2 · p. 3

3. Method

Hi3DGen bridges image-to-geometry generation through normal maps, combining dual-stream normal estimation, normal-regularized latent diffusion, and synthesized high-quality 3D data. Its method targets sharper normal predictions, more detailed geometry supervision, and richer training assets.

Framework overview

  • Hi3DGen divides image-to-geometry generation into image-to-normal estimation and normal-to-geometry mapping using normal maps as a 2.5D bridge.The framework includes a dual-stream normal estimator and an online normal regularizer for diffusion training.Source: §3 · p. 3

3.1. Noise-Injected Regressive Normal Estimation

  • Noise injection in the regressive estimator targets high-frequency information, where diffusion analysis indicates faster signal-to-noise degradation during noising.The method is motivated by sharper normal details at edges and cavities.Source: §3.1 · p. 4
  • The dual-stream architecture uses a clean stream for low-frequency structure and a noise-injected stream for high-frequency features, decoupling representation learning.The design combines structural information associated with low frequencies and sharp-detail sensitivity associated with high frequencies.Source: §3.1 · pp. 3–4

3.2. Normal-Regularized Latent Diffusion

  • Latent-only supervision can compress fine-grained geometry, motivating explicit geometry-space regularization for more precise surface-detail supervision.The method addresses detail loss and detail-level inconsistency in 3D latent diffusion.Source: §3.2 · pp. 4–5
  • Normal-regularized latent diffusion adds explicit normal-map supervision to the usual latent diffusion objective during training.The predicted clean latent is decoded into 3D geometry, rendered into normal maps, and compared with corresponding ground-truth normals online.Source: §3.2 · p. 5

3.3. DetailVerse Dataset

  • The DetailVerse construction pipeline synthesizes and cleans 3D assets, preserving 700k high-quality meshes with richer surface details.The pipeline uses text-to-image-to-3D generation followed by expert and automated assessment.Source: §3.3 · pp. 5–6, Figure 4 · p. 5

4. Experiments

Experiments evaluate Hi3DGen’s normal estimation, normal bridging, geometry generation, and training data across quantitative, qualitative, and user-study settings. Results support improved accuracy, sharpness, fidelity, and detail generation.

Experiment Setup

  • Hi3DGen is evaluated against regression-, diffusion-, and state-of-the-art 3D-generation methods using complementary datasets and metrics.Normal estimation uses NE and SNE; geometry evaluation renders normal maps from 22 viewpoints.Source: §4.1 · p. 6

Image-to-Normal Estimation

  • NiRNE achieves significantly superior overall and sharp-region normal estimation performance on LUCES-MV.The comparison includes both regression-based and diffusion-based estimators.Source: §4.2 · p. 6
  • Qualitative comparisons show NiRNE combines generalizability, stability, and sharpness, with fewer wrong details than diffusion methods and sharper predictions than regression methods.Results cover human and object inputs.Source: §4.2 · p. 6

Ablation Study

  • Normal bridging outperforms direct image-to-geometry generation, while smoother or incorrect normal conditions reduce final 3D quality.Using identical normal regularization and training data, direct generation produces fake details.Source: §4.4 · p. 7
  • DetailVerse improves NiRNE by 0.4 NE and 1.7 SNE, while additional normal-geometry pairs improve NoRLD’s generated details.The improvements are reported in Table 3 and Figure 10.Source: §4.4 · p. 7, Table 3 · p. 8
  • Online normal regularization greatly improves generation fidelity with or without DetailVerse training data.The ablation visualizes the effect on roof details.Source: §4.4 · p. 8

5. Conclusion

Hi3DGen bridges images and 3D geometry through normal maps, combining normal estimation, normal-regularized synthesis, and detail-rich synthetic data. The framework generates detailed results, but some details may remain inconsistent with the input.

Conclusion

  • Hi3DGen uses normal maps as a 2.5D intermediate representation for generating rich details consistent with input images.The framework divides image-to-geometry generation into image-to-normal estimation and normal-to-3D synthesis.Source: §5 · p. 8
  • The framework combines robust, stable, and sharp normal estimation, normal-regularized geometry synthesis, and a detail-rich 3D data pipeline.These components support training and geometry consistency.Source: §5 · p. 8

Limitations

  • Some generated details remain potentially inconsistent or nonaligned with the input because of the generative nature of 3D latent diffusion learning.The paper identifies reconstruction-level 3D generation as future work.Source: §5 · p. 8

6. More Details for the Method

Additional method details describe noise-injected dual-stream normal estimation, inference preprocessing, sharp-region evaluation, and comparisons with photometric stereo methods. These details operationalize the framework’s stability and detail-preservation goals.

Image-to-Normal Estimation

  • The fine-grained encoder receives EDM-style noise during training, while the coarse encoder remains noise-free.Noise is injected into encoder-output features before decoder layers rather than back into the encoder.Source: §6 · p. 13

Inference

  • During inference, Hi3DGen isolates, crops, pads, and resizes the foreground to 768×768 before NiRNE estimation.Inference omits feature noise injection to preserve captured detail information.Source: §6 · p. 13

Comparisons

  • NiRNE is compared with regression-based, diffusion-based, and photometric-stereo normal estimators.The supplied comparison caption identifies LUCES-MV SNE evaluation and best-result marking.Source: Figure S12 · p. 13, Table S4 · p. 14

Evaluation Metrics

  • Sharp Normal Error measures normal angle error within dilated Canny-detected salient edge regions, complementing whole-map Normal Error.SNE emphasizes locations where geometric details are most salient.Source: §6 · p. 14

7. More Details for the DetailVerse

DetailVerse is constructed through prompt curation, image generation, orientation filtering, Trellis synthesis, and expert-guided mesh-quality assessment. The pipeline prioritizes semantic diversity, geometric fidelity, and clean training assets.

Prompt Curation

  • Text prompts are curated and standardized to control semantic diversity and enforce explicit geometric cues.Near-isometric viewpoints and CGI-rendered aesthetics are used to enhance synthesis fidelity.Source: §7 · p. 14

Image Generation

  • Flux.1-Dev generates images selected for visual fidelity, rich detail, suitable viewpoints, and reconstruction-friendly styles.Generated images are ranked by sharp-pixel counts from Canny edge detection, retaining the top 50%.Source: §7 · p. 14

Image Filtering

  • OrientAnything rejects images whose camera-to-canonical orientation deviation exceeds 60° to reduce geometry distortion.The filtering process preserves 1 million images for subsequent 3D synthesis.Source: §7 · p. 14

3D Synthesis

  • Trellis converts the prepared images into preliminary high-fidelity meshes for the dataset.The generator is selected for its performance with high-quality inputs.Source: §7 · p. 14

Quality Filtering

  • Ten trained experts triple-blind-assess 10K sampled meshes for surface holes and noise artifacts.Their annotations train a DINOv2-feature quality classifier applied across the dataset.Source: §7 · p. 14
  • Meshes classified positively across all four rendered views are selected for NoRLD training.The quality network uses four equiangular normal-map views per mesh.Source: §7 · p. 14

8. More Ablation Studies

Qualitative ablations show that each component of NiRNE contributes positively to Hi3DGen’s final performance.

8. More Ablation Studies

  • Each NiRNE component makes a positive contribution to the final performance.The qualitative ablation results are shown in Fig. S11.Source: §8 · p. 15

9. More Results

Additional qualitative comparisons show that normal-bridged Hi3DGen produces detailed 3D geometries that are more consistent with input images than existing methods.

9. More Results

  • Hi3DGen generates 3D detailed geometries that are more consistent with input images than existing methods.Additional qualitative comparisons are presented in Fig. S14.Source: §9 · p. 15
  • The supplementary material provides more DetailVerse data exhibitions.Figure S13 presents additional DetailVerse data.Source: Figure S13 · p. 16
  • The supplementary material provides additional 3D generation result comparisons.Figure S14 presents more generation comparisons.Source: Figure S14 · p. 17

Related papers

Loading 2503.22236v2…