Source-linked AI summary
StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal
Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, Xiaoguang Han
TL;DR
StableNormal addresses the conflict between stochastic diffusion inference and the deterministic requirements of monocular surface-normal estimation. It uses reliable one-step initialization with semantic-guided refinement, achieving stable and sharp estimates across indoor benchmarks and challenging real-world conditions.
Problem
Diffusion-based normal estimators can produce sharp-looking details but suffer high variance and directional deviation, conflicting with deterministic geometric estimation and motivating costly post-ensembling.
Method
StableNormal uses a coarse-to-fine strategy combining reliable YOSO one-step initialization with semantic-guided SG-DRN diffusion refinement.
Results
StableNormal demonstrates strong quantitative and qualitative performance on DIODE-indoor, iBims, ScannetV2, and NYUv2, including robustness under challenging imaging conditions and improvements in reconstruction scenarios.
Takeaways & Limitations
Reducing diffusion stochasticity can retain both stability and sharpness for accurate monocular normal estimation without additional ensembling.
Takeaways & Limitations
The one-step approach oversimplifies the diffusion Markov chain, while regression-based estimators remain limited by constrained training data and fine-grained detail capture.
Abstract
from arXiv · showhide
This work addresses the challenge of high-quality surface normal estimation from monocular colored inputs (i.e., images and videos), a field which has recently been revolutionized by repurposing diffusion priors. However, previous attempts still struggle with stochastic inference, conflicting with the deterministic nature of the Image2Normal task, and costly ensembling step, which slows down the estimation process. Our method, StableNormal, mitigates the stochasticity of the diffusion process by reducing inference variance, thus producing "Stable-and-Sharp" normal estimates without any additional ensembling process. StableNormal works robustly under challenging imaging conditions, such as extreme lighting, blurring, and low quality. It is also robust against transparent and reflective surfaces, as well as cluttered scenes with numerous objects. Specifically, StableNormal employs a coarse-to-fine strategy, which starts with a one-step normal estimator (YOSO) to derive an initial normal guess, that is relatively coarse but reliable, then followed by a semantic-guided refinement process (SG-DRN) that refines the normals to recover geometric details. The effectiveness of StableNormal is demonstrated through competitive performance in standard datasets such as DIODE-indoor, iBims, ScannetV2 and NYUv2, and also in various downstream tasks, such as surface reconstruction and normal enhancement. These results evidence that StableNormal retains both the "stability" and "sharpness" for accurate normal estimation. StableNormal represents a baby attempt to repurpose diffusion priors for deterministic estimation. To democratize this, code and models have been publicly available in hf.co/Stable-X
1 INTRODUCTION
Monocular normal estimation must reconcile diffusion models’ sharp detail with the deterministic, stable, and accurate geometric cues required by the task. StableNormal addresses this trade-off through reliable initialization and semantic-guided refinement, with strong benchmark and challenging-condition results.
- Accurate monocular normal maps support surface realism, relighting, intrinsic decomposition, and other applications bridging 2D images and 3D surfaces.
- Diffusion-based estimators produce sharp-looking normals, but their outputs can be temporally inconsistent, incorrect, and unstable relative to ground truth.
- Diffusion stochasticity conflicts with the deterministic nature of normal estimation, especially under extreme lighting, camera motion, blur, and low-quality imaging.
- StableNormal uses a coarse-to-fine scheme combining one-step normal estimation for reliable initialization with semantic-guided diffusion refinement for progressively sharper maps.
- YOSO reduces training variance with a Shrinkage Regularizer, while SG-DRN integrates DINO semantic priors to reduce sampling variance and enhance local details.
- StableNormal is evaluated on DIODE-indoor, iBims, ScannetV2, and NYUv2, with reported robustness under extreme lighting, blur, transparency, reflections, and clustered scenes.
2 RELATED WORKS
Prior normal-estimation methods progress from regression-based models to diffusion-based geometric estimators, but both families retain important limitations. Regression methods struggle with generalization and fine details, while diffusion methods face variance, computational, and stability–sharpness trade-offs.
- Regression-based normal estimators typically use a feature extractor followed by a prediction head and have been improved through large-scale, diverse, and prompting-based training.
- Despite these advances, regression-based methods trained on limited or constrained data continue to face generalization problems and miss fine-grained geometric details.
- Diffusion-based approaches capture sharp-looking surface details, but Gaussian initialization and multi-step propagation amplify variance and cause deviations from ground-truth normals.
- Prior responses include post-processing ensembles or one-step estimation, but ensembling is computationally intensive and one-step methods can smooth intrinsic local geometry.
- Diffusion-based normal estimators can show sharper results without notably improved quantitative performance, while omitting ensembling leaves outputs with large variance.
- Repurposing diffusion models for deterministic normal estimation therefore requires balancing stability against sharpness.
3 METHOD
StableNormal repurposes conditional diffusion for monocular normal estimation while addressing the mismatch between stochastic sampling and the deterministic geometric cues required by the task. Its method combines a reliable one-step initialization with restricted, semantically guided refinement.
- Diffusion preliminaries: Diffusion models transform Gaussian noise into data through forward and backward processes, with DDIM offering a more efficient alternative sampler whose noise parameter can make sampling deterministic.The forward process progressively adds noise, while the backward model predicts the denoising trajectory; setting τ to 0 makes DDIM independent of noise.
- Diffusion-based Normal Estimator: For normal estimation, an RGB image is encoded into a latent control signal that conditions a U-Net diffusion model to predict the corresponding normal-map latent.The input image is processed by a pretrained VAE encoder and an additional encoder, while the noisy ground-truth normal latent supplies the diffusion target.
- Diffusion-based Normal Estimator: Diffusion-based normal predictions can look sharp but remain stochastic, misaligned with the input, and expensive to stabilize through ensembling.The reported variance mainly comes from initial and intermediate Gaussian noise, motivating a dual-phase inference procedure with limited refinement steps.
- You-Only-Sample-Once Normal Initialization: YOSO maps a Gaussian sample to an intermediate noisy normal latent in one step, using x_t+-parameterization and a Shrinkage Regularizer to address the resulting many-to-one learning problem.Unlike fully noise-free one-step estimation, YOSO retains a Gaussian noise input to balance sharpness and stability; the regularizer reduces variance through a regularized loss.
4 EXPERIMENTS
StableNormal is evaluated against state-of-the-art normal estimators on four indoor benchmarks, downstream surface reconstruction, and component ablations. The experiments show reduced inference variance without ensembling, while YOSO initialization, SG-DRN refinement, DINO features, and the Shrinkage Regularizer each contribute under supported conditions.
- Benchmark comparison: StableNormal is compared with DSINE, Marigold†, GenPercept, and GeoWizard on DIODE-indoor, iBims, ScanNetV2, and NYUv2.The evaluation uses challenging indoor benchmarks and includes both regression- and diffusion-based baselines.
- Benchmark comparison: StableNormal achieves superior performance across iBims, ScanNetV2, and DIODE-indoor, but is slightly inferior to DSINE on NYUv2.The authors attribute differences on ScanNetV2 and NYUv2 partly to inaccurate ground-truth normals from low-quality sensors.
- Inference variance and speed: 0.410 output variance is achieved within 3 seconds, compared with GeoWizard’s 1.370 variance after approximately 10 seconds of five-sample ensembling.The comparison is reported on DIODE-indoor using a single A100 GPU; StableNormal does not use ensemble strategies.
- Downstream reconstruction: StableNormal achieves the lowest mean Chamfer distance among compared normal-estimation methods on the DTU surface-reconstruction evaluation.The normal maps are used to regularize reconstruction, including challenging non-Lambertian surfaces.
- Ablation study: Removing SG-DRN degrades performance on iBims-1 and DIODE-indoor, while smooth-ground-truth datasets can favor the smoother YOSO-only predictions quantitatively.Qualitative results still show sharper predictions with refinement, and SG-DRN+DSINE also supports multi-step refinement on DIODE-indoor and iBims-1.
- Ablation study: Replacing YOSO with DSINE raises DIODE-indoor mean angle error from 13.701° to 18.453°, while replacing DINO with ResNet-50 raises it from 13.701° to 15.611°.These ablations support the roles of YOSO initialization and DINO semantic guidance in the refinement pipeline.
- Ablation study: The Shrinkage Regularizer improves overall metrics on both DIODE-indoor and iBims-1.It is evaluated as a training component intended to mitigate the difficulty of learning many-to-one mappings.
5 APPLICATIONS
StableNormal improves monocular surface reconstruction by providing high-fidelity normal estimates for object-level, indoor-scene, and normal-integration settings. It also enhances generated multi-view normals and resulting 3D shapes.
- StableNormal achieves the lowest mean Chamfer distance among compared techniques on DTU surface reconstruction.Its normal maps regularize reconstruction following 2DGS, including for non-Lambertian surfaces.
- On 80 DiLiGenT samples, StableNormal improves Normal RMSE, Mean Angle Error by 20%, and Depth Mean Angle Error over previous methods.The evaluation uses ground-truth normals across varying lighting conditions.
- StableNormal faithfully recovers intricate geometric structures in monocular reconstruction compared with ground truth and GeoWizard.The comparison uses extracted meshes produced through normal-field integration with BiNI.
- Applying StableNormal to Wonder3D improves the detail of generated multi-view normal maps and the resulting 3D shapes.
6 CONCLUSION
StableNormal adapts diffusion priors for monocular normal estimation through a coarse-to-fine design that balances stability and sharpness. The conclusion reports validation on indoor benchmarks and real-world applications, while acknowledging failure cases.
- StableNormal combines reliable initialization with semantic-guided refinement to balance the stability–sharpness trade-off in monocular normal estimation.
- DTU comparisons visualize input images, estimated normal maps, and rendered world-space normal maps after reconstruction.
- The paper includes qualitative comparisons on the DiLiGenT dataset and reconstructed-mesh normal comparisons across different scenes.
- The method is validated through multiple indoor benchmarks and various real-world applications.
- The authors identify failure cases in the supplementary appendix and state that code and models will be publicly available for research.
- Qualitative comparisons on NYUv2, ScanNet, iBims-1, and DIODE-indoor report that StableNormal outperforms related works in accuracy and sharpness.
A MORE DETAILS ABOUT IMPLEMENTATION
The implementation fine-tunes Stable Diffusion V2.1 with robustness-oriented augmentation and standardized input preprocessing. The appendix also provides a detailed U-Net architecture figure.
- StableNormal fine-tunes Stable Diffusion V2.1 with AdamW using a fixed learning rate of 3e-5.
- Exposure augmentation is used to improve robustness against exposure variations.
- Input maps are transformed to [-1, 1] to match the VAE input range.
- Training uses random crops with varying aspect ratios and black padding to reach a fixed box resolution.
- Figure R.1 details the architecture of the U-Net.
B THE ARCHITECTURE OF U-NET IN BOTH STAGES
The second-stage U-Net retains most ControlNet building blocks while adding a DINO-based semantic guider and YOSO output as refinement inputs. The appendix also presents typical StableNormal failure cases.
- The SG-DRN architecture retains most ControlNet building blocks with modifications for normal estimation.The described figure shows the second stage.
- A fixed prompt, “The Normal Map,” is used during both training and testing.
- The semantic guider encodes DINO features and combines them with the YOSO output as SG-DRN input.
- Featup and bilinear interpolation upsample semantic features to the YOSO output resolution.
- Figure R.2 shows typical bad cases generated by StableNormal.
C FAILURE CASE
StableNormal can fail in extreme cases, including transparent objects and complex plant surfaces, because its training data lack sufficient outdoor and plant examples.
- StableNormal may partially predict normals belonging to objects behind transparent surfaces.The failure is illustrated on the left side of Figure R.2.
- For plants, StableNormal may assign a similar green normal color despite complex surface normal directions.The authors attribute this to limited plant representation in the training data.
- The authors identify missing outdoor scenes and plants in the training dataset as the source of these extreme-case failures.They suggest adding more simulated renderings as a future remedy.
D MORE QUALITATIVE ANALYSIS OF YOSO
StableNormal produces sharper and more accurate normals than YOSO Only, although its qualitative appearance can seem worse when benchmark ground truths are smoother and less detailed.
- StableNormal predicts sharper and more accurate normals than YOSO Only.
- StableNormal can appear qualitatively worse because NYUv2 and ScanNet ground-truth normal maps are smoother and less detailed.
- Qualitative comparisons between YOSO Only and StableNormal are presented on the NYUv2 and ScanNet datasets.
E MORE QUALITATIVE COMPARISONS
The paper provides additional qualitative comparisons of StableNormal with several diffusion- and geometry-based normal estimators across multiple figures.
- The additional qualitative comparisons span Figures R.4 through R.7.
- Figures R.4 and R.5 compare DSINE, GenPercept, GeoWizard, Marigold, and StableNormal.In both figures, the methods are arranged from left to right in that order.
- Figures R.6 and R.7 compare GeoWizard, DSINE, and StableNormal.The methods are arranged from left to right in both figures.