PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Gwangbin Bae, Ignas Budvytis, Roberto Cipolla

arXiv:2109.09881v1cs.CV

TL;DR

Single-image surface normal estimation lacks aleatoric uncertainty estimates and often loses detail because training favors large planar surfaces. The paper predicts per-pixel normal distributions and uses uncertainty-guided pixel-wise refinement. It reports state-of-the-art performance on ScanNet and NYUv2, with uncertainty correlating with prediction error and improved detail near boundaries and small structures.

  • Problem

    Existing surface normal methods cannot estimate aleatoric uncertainty and produce insufficient detail because training is biased toward large planar surfaces.

  • Method

    The method predicts per-pixel surface normal distributions with angular-loss negative log-likelihood and trains pixel-wise MLP refiners using uncertainty-guided sampling.

  • Results

    The method achieves state-of-the-art performance on ScanNet and NYUv2, while estimated uncertainty correlates with prediction error.

  • Takeaways & Limitations

    Uncertainty-guided sampling improves prediction detail, especially near object boundaries and on small structures, while quantifying aleatoric uncertainty through expected angular error.

Abstract

from arXiv · show

Surface normal estimation from a single image is an important task in 3D scene understanding. In this paper, we address two limitations shared by the existing methods: the inability to estimate the aleatoric uncertainty and lack of detail in the prediction. The proposed network estimates the per-pixel surface normal probability distribution. We introduce a new parameterization for the distribution, such that its negative log-likelihood is the angular loss with learned attenuation. The expected value of the angular error is then used as a measure of the aleatoric uncertainty. We also present a novel decoder framework where pixel-wise multi-layer perceptrons are trained on a subset of pixels sampled based on the estimated uncertainty. The proposed uncertainty-guided sampling prevents the bias in training towards large planar surfaces and improves the quality of prediction, especially near object boundaries and on small structures. Experimental results show that the proposed method outperforms the state-of-the-art in ScanNet and NYUv2, and that the estimated uncertainty correlates well with the prediction error. Code is available at https://github.com/baegwangbin/surface_normal_uncertainty.

1. Introduction

The paper targets two limitations in single-image surface normal estimation: missing aleatoric uncertainty estimates and oversmoothed predictions lacking detail. It introduces distribution-based uncertainty estimation and uncertainty-guided pixel refinement, reporting state-of-the-art performance and improved detail on ScanNet and NYUv2.

  • Motivation: Existing methods cannot estimate aleatoric uncertainty from noisy, depth-derived ground-truth normals.Ground-truth normals are sensitive to depth noise and the normal-computation algorithm, motivating uncertainty modeling for real-world deployment.
  • Motivation: Training on all pixels biases learning toward large planar surfaces, producing oversmoothed predictions with less detail.The resulting weakness is especially relevant near object boundaries and on small structures.
  • Method: The network predicts a per-pixel surface normal probability distribution whose negative log-likelihood is angular loss with learned attenuation.The expected angular error from the predicted distribution measures aleatoric uncertainty.
  • Method: Uncertainty-guided sampling trains pixel-wise MLP refiners on high-uncertainty pixels complemented by uniformly sampled pixels.This strategy counters the training bias toward large planar surfaces and improves predictions near boundaries and on small structures.
  • Results: The paper reports that estimated uncertainty correlates well with prediction error.This is presented as a contribution of the proposed surface normal probability parameterization.
  • Results: The method achieves state-of-the-art performance on ScanNet and NYUv2 while producing higher-detail predictions with clearer object boundaries.The qualitative comparison is reported against TiltedSN [6].

2. Related Work

Prior work on single-image surface normal estimation evolved from hand-crafted features and boosting to CNN-based architectures with multi-scale, geometric-consistency, spherical-regression, and spatial-rectification strategies. This paper addresses aleatoric uncertainty, which prior CNN-based surface normal methods had not studied.

  • Surface normal estimation: Early surface normal methods used hand-crafted features with multi-class AdaBoost to regress normals as combinations of discrete normals.Later approaches replaced feature extraction and prediction components with CNNs.
  • Surface normal estimation: CNN-based methods incorporated global and local cues, multi-scale prediction, depth-normal consistency, spherical regression, and spatial rectification.These approaches expanded the architectural and geometric treatment of surface normal estimation.
  • Uncertainty modeling: Aleatoric uncertainty captures noise inherent in the data and can be modeled through per-pixel output distributions under heteroscedastic uncertainty.The paper identifies this task-specific formulation as previously unstudied for CNN-based surface normal estimation.
  • Uncertainty modeling: The proposed distribution variant makes negative log-likelihood equivalent to minimizing angular error, while the decoder extends uncertainty-based point-wise refinement to surface normal estimation.The related framework addresses undersampling near object boundaries on regular grids.

3. Method

The method estimates per-pixel surface-normal distributions and uses a parameterization whose NLL corresponds to angular loss with learned attenuation. An uncertainty-guided refinement decoder then emphasizes challenging pixels while progressively increasing resolution.

  • Aleatoric Uncertainty: The network learns a per-pixel surface-normal probability distribution from image features and trains it by minimizing negative log-likelihood.
  • Aleatoric Uncertainty: The von Mises-Fisher baseline uses mean direction μ and concentration κ, where larger κ indicates lower uncertainty.
  • Aleatoric Uncertainty: The proposed Angular vonMF distribution makes NLL equivalent to angular error with learned attenuation for high-uncertainty pixels.
  • Aleatoric Uncertainty: The expected angular error is used as the pixel-wise aleatoric uncertainty measure, while angular loss is more robust to asymmetric normal noise.
  • Uncertainty-Guided Sampling: The pipeline begins at 1/8 resolution, applies the loss to all coarse pixels, and progressively produces full-resolution predictions through refinement modules.
  • Uncertainty-Guided Sampling: The decoder upsamples coarse predictions through three refinement modules whose pixel-wise MLPs train on uncertainty-selected subsets to reduce bias toward large planar surfaces.

4. Experimental Setup

The method is evaluated on ScanNet and NYUv2 using established data splits and ground-truth surface normals. Accuracy is measured with angular-error metrics on pixels having valid ground truth.

  • Datasets: Evaluation uses ScanNet and NYUv2, two RGB-D datasets containing indoor scenes.
  • Datasets: ScanNet evaluation uses the ground truth surface normals and data split provided by FrameNet.
  • Datasets: NYUv2 evaluation uses its official test set and the same training set as GeoNet++ for fair comparison.
  • Metrics: Angular error is evaluated on pixels with valid ground truth using mean, median, and root-mean-squared error, with lower values preferred.

5. Experiments

Experiments evaluate the loss, decoder, sampling strategy, benchmark accuracy, and uncertainty quality. The proposed components improve prediction detail and accuracy while producing uncertainty that correlates better with prediction error.

  • Ablation Study: The ablations compare NLL-AngMF, NLL-vonMF, angular loss, and decoder components on a ScanNet subset.NLL-AngMF is angular loss with learned attenuation, whereas NLL-vonMF is L2 loss with learned attenuation.
  • Ablation Study: NLL-AngMF achieves higher accuracy than NLL-vonMF except for RMSE.
  • Ablation Study: Both pixel-wise MLPs and uncertainty-guided sampling improve all metrics, with βUG = 0.7 giving the best performance.Uncertainty-guided sampling reduces bias toward large planar surfaces and improves predictions near object boundaries and on small structures.
  • Comparison with the State-of-the-Art: The method achieves state-of-the-art performance on NYUv2 and outperforms GeoNet++ decisively despite noisier ground truth.Its predictions also contain a higher level of detail.
  • Comparison with the State-of-the-Art: In cross-dataset evaluation, the method outperforms other methods except for median error and 11.25° accuracy.The evaluation trains on ScanNet and tests on NYUv2 without fine-tuning.
  • Comparison with the State-of-the-Art: On ScanNet, the method outperforms state-of-the-art methods across all metrics.
  • Quality of Uncertainty: As high-uncertainty pixels are removed, the method becomes significantly more accurate than alternative uncertainty estimators.All methods perform similarly when evaluated on all pixels, but the proposed uncertainty correlates better with prediction error.

6. Conclusion

The paper estimates aleatoric uncertainty through per-pixel normal distributions and uses uncertainty-guided refinement to improve detail. Experiments report state-of-the-art performance on ScanNet and NYUv2 and uncertainty that correlates with prediction error.

  • The method estimates aleatoric uncertainty from per-pixel surface normal probability distributions.Its expected angular error quantifies the uncertainty.
  • The AngMF parameterization makes negative log-likelihood equal to angular loss with learned attenuation.
  • Uncertainty-guided sampling trains pixel-wise MLPs on selected pixels, reducing bias toward large planar surfaces and improving detail.
  • The method achieves state-of-the-art performance on ScanNet and NYUv2, while estimated uncertainty correlates well with prediction error.

A. Derivations for the proposed AngMF distribution

The appendix derives the Angular von Mises-Fisher distribution, its training loss, and the expected angular error used as aleatoric uncertainty. The concentration parameter controls both distribution sharpness and expected error.

  • The AngMF distribution modifies von Mises-Fisher so its NLL uses angular error with learned attenuation.The angular error is the angle between predicted mean direction µ_i and ground-truth normal n_i^gt, weighted by concentration κ_i.
  • The resulting PDF is proportional to exp(-κ_i cos^-1(µ_i^T n_i)) with normalization D(κ_i).The normalization is derived by integrating over angular error and azimuth.
  • Training minimizes the NLL, whose nonconstant terms are -log(κ_i^2 + 1), log(1 + exp(-κ_iπ)), and κ_i cos^-1(µ_i^T n_i^gt).The constant log 2π is omitted.
  • The expected angular error measures aleatoric uncertainty and decreases as κ increases.For κ = 0, the distribution is uniform and the expected error is π/2.
  • The cumulative angular-error probability is derived and differentiated to obtain the angular-error density.

B. Quantitative evaluation with additional metrics

The supplementary evaluation extends the paper’s quantitative accuracy and uncertainty analyses with additional metrics and figures.

  • Tables 7–9 extend Tables 4–6 with additional quantitative metrics, while Figures 11–12 extend the sparsification analysis in Figure 8.

B.1. Comparison against TiltedSN

The proposed uncertainty estimator outperforms task-independent alternatives across uncertainty metrics, while sparsification curves show that its uncertainty better tracks prediction error.

  • Comparison against TiltedSN: Accuracy differences against TiltedSN increase at lower error thresholds on ScanNet.The comparison is reported using the percentage of pixels whose error is below each threshold.
  • Quality of the estimated uncertainty: The proposed NLL-based pipeline significantly outperforms dropout, augmentation, and combined approaches across all uncertainty metrics.These task-independent methods use multiple stochastic or flipped-image inferences and do not require distributional outputs.
  • Sparsification curves: As high-uncertainty pixels are removed, the proposed method becomes significantly more accurate than competing methods on NYUv2 and ScanNet.When evaluated on all pixels, the methods perform similarly; the difference emerges during sparsification.
  • Sparsification curves: The sparsification results suggest that the proposed uncertainty correlates better with prediction error than competing estimates.The method is also compared with an oracle that sorts pixels by their actual error.

C. Cross-dataset evaluation on KITTI and DAVIS

The supplementary evaluation tests a ScanNet-trained network on the more challenging KITTI and DAVIS datasets without fine-tuning, including unseen object categories and severe image tilts as failure cases.

  • Cross-dataset evaluation on KITTI and DAVIS: A network trained only on ScanNet is evaluated without fine-tuning on KITTI and DAVIS to assess generalization beyond similar indoor-scene datasets.Predictions are compared with TiltedSN in the corresponding figures.
  • Cross-dataset evaluation on KITTI and DAVIS: The network generalizes to unseen objects such as humans, cars, and buildings despite ScanNet’s indoor-scene coverage and missing dynamic-object ground truth.The authors attribute this generalization to low-level visual features such as edges and shades.
  • Failure modes: The network is robust to mild image rotations around 20° but suffers under severe tilts from approximately 30° onward.The expected error also increases for severely tilted images, indicating useful uncertainty estimates in this failure case.
  • Failure modes: A spatial rectifier is proposed as a possible way to handle tilted images, but its investigation is left for future work.The rectifier would warp images so their surface-normal distribution matches that of the training images.

D.2. Inherent ambiguity of the problem

Artificial edge-and-shade images expose ambiguity in mapping visual structures to surface geometry: the network predicts concave corners in some cases but remains uncertain when training examples support multiple interpretations.

  • Artificial visual cues: Artificial images containing only edges and shades can represent concave corners, convex corners, or drawings on flat walls because pixel depths are arbitrary.The experiments use Y-shaped structures and vertically flipped versions to probe the visual cues used by the network.
  • Inherent ambiguity: The network predicts a concave corner for the final three images because similar structures mostly appeared in lower corners of cuboid-shaped rooms.This reflects a learned association between the visual structure and its usual training-scene location.
  • Inherent ambiguity: Predictions remain unclear for Y-shaped structures because the training data included them in both concave and convex corner contexts.The ambiguity arises even though the same visual structure can correspond to different surface geometries.
  • Inherent ambiguity: The authors identify multimodal surface-normal distributions as a future direction for handling such ambiguous visual structures.The proposed distribution would combine multiple unimodal distributions with mixing coefficients.
Loading 2109.09881v1…