Source-linked AI summary
Normal Integration: A Survey
Yvain Quéau, Jean-Denis Durou, Jean-François Aujol
TL;DR
Normal integration reconstructs 3D shape from surface normals, but methods must meet several demands beyond accuracy. This survey studies two pioneering approaches, derives criteria for reviewing existing methods, and concludes that none satisfies all the required properties.
Problem
Normal integration reconstructs 3D shape from normals, but methods face several simultaneous requirements beyond accuracy.
Source: §1 · p. 1Method
The survey studies Horn and Brooks’ variational approach and Frankot and Chellappa’s Fourier approach, then uses them to select criteria for reviewing normal integration methods.
Source: §1 · p. 1Results
None of the reviewed methods satisfies all the required properties, and numerical accuracy comparisons across them were not feasible in practice.
Source: §4.4 · p. 15Takeaways & Limitations
The review identifies a need for new integration methods, with the companion paper focusing on depth discontinuities associated with occlusions.
Source: §5 · p. 16Recovered surfaces can vary substantially with boundary-condition choice because the governing equation permits harmonic-function alternatives.
Source: §2.4 · p. 4
Abstract
from arXiv · showhide
The need for efficient normal integration methods is driven by several computer vision tasks such as shape-from-shading, photometric stereo, deflectometry, etc. In the first part of this survey, we select the most important properties that one may expect from a normal integration method, based on a thorough study of two pioneering works by Horn and Brooks [28] and by Frankot and Chellappa [19]. Apart from accuracy, an integration method should at least be fast and robust to a noisy normal field. In addition, it should be able to handle several types of boundary condition, including the case of a free boundary, and a reconstruction domain of any shape i.e., which is not necessarily rectangular. It is also much appreciated that a minimum number of parameters have to be tuned, or even no parameter at all. Finally, it should preserve the depth discontinuities. In the second part of this survey, we review most of the existing methods in view of this analysis, and conclude that none of them satisfies all of the required properties. This work is complemented by a companion paper entitled Variational Methods for Normal Integration, in which we focus on the problem of normal integration in the presence of depth discontinuities, a problem which occurs as soon as there are occlusions.
1 Introduction
The survey uses a study of two pioneering approaches to identify criteria beyond accuracy for normal integration. It reviews existing methods against these properties and concludes that no method is completely satisfactory.
1 Introduction
- Normal integration is expected to be accurate, fast, and robust to noisy data or outliers.The survey notes that the problem's apparent simplicity has led to many papers because it involves several requirements.Source: §1 · p. 1
- A study of Horn and Brooks’ variational method and Frankot and Chellappa’s Fourier method informs six evaluation criteria beyond accuracy.The survey summarizes its evaluation in Table 1.Source: §1 · p. 1
- The survey reviews normal integration methods against the selected properties and concludes that none satisfies them all.Its planned structure derives basic equations, reviews the two pioneering methods, then selects properties and discusses related work.Source: §1 · p. 1
2 Basic Equations of Normal Integration
The section formulates normal integration as recovering a surface from its normals, derives the equations for three projection models, and then treats non-integrable gradient fields through quadratic regularization and boundary conditions.
2 Basic Equations of Normal Integration
- Normal integration recovers a surface from its known normal field; under orthographic, weak-perspective, and perspective projection, it reduces to integrating a corresponding gradient field.The equations for all three projection models can be handled by a generic pair of linear PDEs when camera intrinsics are known.Source: §2 · p. 2, §2.3 · p. 4
2.1 Orthographic Projection
- Under orthographic projection, integrating the normal field yields depth up to an additive constant, provided the derived gradient field is integrable; depth cannot be recovered at occluding-contour points.When no depth value is known, the constant may be chosen to minimize RMSE if ground truth is available.Source: §2.1 · p. 2
2.2 Weak-perspective Projection
- Weak-perspective projection scales the orthographic gradient equations by the magnification, with the reference plane placed at the surface’s mean location.The magnification is defined as f/d, where the reference plane has equation z = d.Source: §2.2 · pp. 2–3
2.3 Perspective Projection
- Under perspective projection, normal integration uses a transformed gradient field, and recovering depth requires the focal length and principal-point location to be known.The resulting depth is determined up to a multiplicative constant.Source: §2.3 · p. 4
2.4 Integration Using Quadratic Regularization
- Because practical normal fields are not exactly integrable, quadratic regularization estimates depth by optimization, but the boundary condition affects the recovered surface.For a free or Neumann boundary the solution remains undetermined up to an additive constant; a bounded domain with Dirichlet conditions has a unique solution.Source: §2.4 · pp. 4–5
3 Two Pioneering Normal Integration Methods
This section examines two pioneering normal-integration methods and their adaptations, focusing on their computational properties and handling of boundary conditions, non-rectangular domains, and depth discontinuities.
3.1 Horn and Brooks’ Method
- MHB solves a discrete Poisson equation with Jacobi iteration, which converges slowly from distant initializations and requires careful discretization of natural boundary conditions.The natural-boundary discretization is especially challenging because many cases must be handled.Source: §3.1 · p. 5
3.2 Improvement of Horn and Brooks’ Method
- MDC discretizes the quadratic functional before deriving optimality conditions, yielding a simpler natural-boundary discretization that approximates the natural boundary condition.For interior pixels, its optimality condition matches MHB’s discrete equation.Source: §3.2 · pp. 5–6
- MDC’s quadratic regularization fails to preserve depth discontinuities; removing the flat ground improves reconstruction but requires segmentation and a non-rectangular domain.The vase reconstruction over the whole grid has a much higher RMSE than when the domain excludes the flat ground.Source: §3.2 · p. 6, Figure 5 · p. 7
- Gradient-field non-integrability is neither necessary nor sufficient to detect depth discontinuities: shadows can cause it, while a scale viewed from above can remain integrable.The cited examples connect shadow-induced outliers to photometric-stereo estimates.Source: §3.2 · p. 6
3.3 Frankot and Chellappa’s Method
- MFC projects the gradient field onto integrable fields using a Fourier basis and is fast via FFT, but periodic boundary assumptions distort non-periodic surfaces and exclude non-rectangular domains.The face example forces matching depths at the cheek and nose; MFC also leaves the solution of the Poisson equation ambiguous up to a harmonic function.Source: §3.3 · pp. 6–7, Figure 6 · p. 8
3.4 Improvements of Frankot and Chellappa’s Method
- Discrete Fourier transforms solve the discrete Poisson equation only for periodic boundary conditions, whereas discrete sine and cosine transforms suit Dirichlet and Neumann conditions, respectively.The Neumann solution is determined only up to an additive constant.Source: §3.4 · pp. 8, 10
- MSCS works well for non-periodic surfaces and is as fast as MFC, substantially improving Frankot and Chellappa’s original method.The passage illustrates the result with Figure 7.Source: §3.4 · p. 10
- The homogeneous Dirichlet or Neumann boundary-condition approach works only on rectangular domains, and MSCS cannot reproduce the Figure 5-c vase result.The claimed extension to non-rectangular domains using embedding techniques is neither detailed nor really proved; applying MSCS to the vase gradient field yields the Figure 5-d result instead.Source: §3.4 · p. 10
4 Main Normal Integration Methods
This section reviews and assesses main normal integration methods across six selected properties, including how they address depth discontinuities. It cautions that binary scores limit nuance and that counting positive scores is not a global measure of quality.
4.1 A List of Expected Properties
- Methods should be fast and robust to noisy normals, handle free boundaries and non-rectangular domains, preserve depth discontinuities, and require few or no tunable parameters.The survey also regards accuracy as a basic requirement, while noting that accuracy comparisons are difficult and it therefore reviews method features instead.Source: §4.1 · p. 10
4.2 Integration and Integrability
- The survey treats noise and depth discontinuities as the relevant sources of non-integrability, but warns that finite-difference test gradients can falsely appear integrable.It assumes calibrated photometric stereo with at least three images, and does not expect integration to compensate for other defects such as shadow outliers; a numerically approximated gradient can yield Eint = 0 even with discontinuities.Source: §4.2 · p. 11
4.3 Most Representative Normal Integration Methods
- The review divides methods into those that address discontinuities and those that do not, alongside earlier path-integration heuristics for non-integrable fields.The heuristics average values obtained along different integration paths.Source: §4.3 · p. 12
4.3.1 Methods which do not care about Discontinuities
- Methods that do not address discontinuities trade off the other properties: Poisson solvers vary in speed and boundary support, while Fourier-based methods impose domain or boundary constraints.MHB is robust, parameter-free, and supports free boundaries and non-rectangular domains, but is slow and does not preserve discontinuities; MSCS offers fast Poisson solving and can support natural boundaries, but does not practically support non-rectangular domains or preserve discontinuities. MFC is fast, robust, and parameter-free, but assumes a rectangular periodic domain.Source: §4.3.1 · p. 12
4.3.2 Methods which care about Discontinuities
- Badri et al.’s Lp method handles noise and outliers (PRobust, PDisc), but loses PFast and PNoPar and does not satisfy PFreeB or PNoRect.It uses p ∈ ]0,1[; its non-convex half-quad resolution is iterative, requires good initialization, and uses FFT, implying a rectangular domain and periodic boundary condition.Source: §4.3.2 · p. 15
4.4 Summary of the Review
- None of the reviewed methods satisfies all required properties, and numerical accuracy comparisons across them were not practical.The survey instead summarizes their main features; accuracy is identified as a basic requirement, but the methods are not numerically compared.Source: §4.4 · p. 15
- Almost every method differs from the others across the six selected properties.Table 1 uses binary scores; more levels for each criterion could have led to more arbitrary ratings.Source: §4.4 · p. 15
- A method’s number of positive scores is not a global score, since it may satisfy some properties while neglecting others.Source: §4.4 · p. 15
5 Conclusion and Perspectives
The review discusses specialized normal-integration problems and concludes that no reviewed method satisfies all selected criteria. It points to unresolved method choice and to a companion paper focused on integration across depth discontinuities.
5 Conclusion and Perspectives
- L1 is reported as the globally best option among the tested estimators for outliers, but the survey does not treat outlier robustness as a required integration-method property.The authors argue that outliers need not be compensated by the integration method and note that photometric stereo can be robust to them.Source: §5 · p. 15
- Fixing depth at selected points is considered useful but not required, and Horovitz and Kiryati and Kimmel and Yavneh address it specifically.Source: §5 · p. 15
- Balzer’s method addresses depth-dependent deflectometry normals, which Ettl et al. note are usually not noisy; the proposed iterative method is described as limited.Balzer also suggests extending the approach to photometric-stereo normals without assuming distant light sources.Source: §5 · pp. 15–16
- Chang et al.’s multiview normal integration yields complete 3D models, but its reported results are limited to synthetic multiview photometric-stereo images.The authors say multiview inputs are an intuitive way to satisfy PDisc.Source: §5 · p. 16
- Choosing an algorithm for a particular application remains open, and the authors hope their review helps readers assess the many existing approaches.Source: §5 · p. 16
- No reviewed method satisfies all selected criteria, motivating a companion paper on normal integration with depth discontinuities caused by occlusions.Source: §5 · p. 16