Source-linked AI summary
Surface Normal Estimation of Tilted Images via Spatial Rectifier
Tien Do, Khiem Vuong, Stergios I. Roumeliotis, Hyun Soo Park
TL;DR
Surface normal estimators trained on gravity-aligned images perform less reliably on tilted imagery from arbitrarily oriented sensors. The paper learns a spatial rectifier to align tilted-image normal distributions with training data, adds a truncated angular loss and efficient network design, and reports state-of-the-art results across ScanNet, NYUv2, and Tilt-RGBD.
Problem
Existing surface normal estimators have limited capability on arbitrarily oriented images because training data is predominantly gravity-aligned and does not represent all surfaces equally.
Method
The method jointly learns a gravity- and principal-direction-parameterized spatial rectifier, truncated angular loss, and efficient network using synthesized tilted images.
Results
The resulting estimator outperforms state-of-the-art methods and data-augmentation baselines on ScanNet, NYUv2, and Tilt-RGBD.
Takeaways & Limitations
Spatial rectification enables accurate surface normal estimation for tilted images without external sensors, while the design maintains computational efficiency.
Takeaways & Limitations
The rectifier relies on predicting gravity and a principal direction from visual semantics, and the L2 baseline has vanishing gradients near small angular errors.
Abstract
from arXiv · showhide
In this paper, we present a spatial rectifier to estimate surface normals of tilted images. Tilted images are of particular interest as more visual data are captured by arbitrarily oriented sensors such as body-/robot-mounted cameras. Existing approaches exhibit bounded performance on predicting surface normals because they were trained using gravity-aligned images. Our two main hypotheses are: (1) visual scene layout is indicative of the gravity direction; and (2) not all surfaces are equally represented by a learned estimator due to the structured distribution of the training data, thus, there exists a transformation for each tilted image that is more responsive to the learned estimator than others. We design a spatial rectifier that is learned to transform the surface normal distribution of a tilted image to the rectified one that matches the gravity-aligned training data distribution. Along with the spatial rectifier, we propose a novel truncated angular loss that offers a stronger gradient at smaller angular errors and robustness to outliers. The resulting estimator outperforms the state-of-the-art methods including data augmentation baselines not only on ScanNet and NYUv2 but also on a new dataset called Tilt-RGBD that includes considerable roll and pitch camera motion.
1 Introduction
Tilted images from body- and robot-mounted cameras challenge surface normal estimators trained mainly on gravity-aligned data. The paper addresses this gap with a learned spatial rectifier, truncated angular loss, and efficient network design, achieving strong performance across standard and tilted-image benchmarks.
- The paper targets surface normal estimation from arbitrarily oriented cameras, motivated by applications including augmented reality, 3D reconstruction, and robot navigation.
- Training on predominantly gravity-aligned images creates a domain gap that causes state-of-the-art estimators to degrade substantially on tilted images.FrameNet is accurate on upright images but highly erroneous on tilted regions because surface orientations are unevenly represented.
- The spatial rectifier learns a global transformation, parameterized by gravity and a principal direction, to match tilted-image surface-normal distributions to upright training data.It synthesizes tilted images with homographies and jointly learns rectification with surface normal estimation.
- The resulting method outperforms state-of-the-art methods and data-augmentation baselines on ScanNet, NYUv2, and Tilt-RGBD, including images with substantial roll and pitch.The contribution also includes a new Tilt-RGBD dataset collected with body-mounted cameras.
- The truncated angular loss addresses the mismatch between angular evaluation and conventional vector losses by improving sensitivity to small errors and robustness to outliers.Its stated accurate small-error region is below 7.5°.
- The method combines spatial rectification with an efficient architecture that accesses global and local visual features while reducing computational burden.The paper reports real-time inference and lower FLOPs and memory than DORN+TAL in its efficiency comparison.
2 Related Work
Prior work estimates gravity from visual cues and uses it for scene understanding, while surface-normal methods differ in input modalities, losses, and network designs. This paper situates spatial rectification among approaches addressing tilted imagery without relying on external sensors or offline gravity estimation.
- Visual cues such as vanishing points and learned visual semantics have been used to estimate gravity without external sensors.Gravity estimates have supported tasks including single-view depth prediction and semantic regularization.
- Prior scene-understanding systems use gravity or in-plane rotation correction to improve downstream depth prediction, but the paper proposes spatial rectification for tilted-image surface normals.
- Classification-based surface normal learning is limited by the resolution of its discrete classes.
- Surface-normal training losses have progressed from vector-based measures toward angular losses because angular evaluation exposes limitations of L2-style objectives.Related approaches include L1 measures on unit vectors, spherical regression, and angular loss.
3 Method
The method jointly learns a spatial rectifier and surface normal estimator for tilted images, warping each image toward a rectified surface-normal distribution that matches gravity-aligned training data. It combines learnable geometric transformation, truncated angular loss, and a compact multi-scale network.
- 3.1 Spatial Rectifier: The spatial rectifier learns a rotation-based warp that transforms tilted-image surface normals from underrepresented regions toward densely represented regions in the training distribution.The warp is parameterized by gravity direction g and principle direction e, using a homography induced by a pure 3D rotation.
- 3.1 Spatial Rectifier: The rectified view is selected by minimizing KL divergence between its surface-normal distribution and the discretized distribution of all training data, while penalizing invisible pixels.The visibility term encourages a minimal transformation; λ_e balances distribution matching against visibility.
- 3.2 Joint Training: The rectifier predicts gravity and principle directions from tilted-image visual semantics, and training synthesizes random tilts before rectifying them and warping predicted normals back to tilted coordinates.The estimator operates on rectified images, reducing the need to memorize all possible tilted orientations.
- 3.3 Truncated Angular Loss: Truncated angular loss addresses the mismatch between angular evaluation and conventional vector losses by clamping near-zero errors and reducing sensitivity to large-error outliers.For n^Tbn < 0, the loss is linear and assigns constant weight, making training less sensitive to outliers than angular loss.
- 3.4 Network Architecture: The surface normal estimator uses an asymmetric encoder-decoder with multi-scale features, ASPP, and combined upsampling to predict quarter-resolution surface normals.Features span spatial resolutions from 1/4 to 1/32 and channel dimensions from 256 to 2048.
4 Results
The evaluation spans ScanNet, NYUv2, and Tilt-RGBD, comparing baselines, network architectures, spatial rectification, and training losses for surface-normal estimation. The proposed approach improves tilted-image performance, while DFPN also offers substantial efficiency gains and TAL improves tight-threshold accuracy.
- 4.1 Evaluation Dataset: Tilt-RGBD evaluation uses body-mounted-camera data with tilted images, alongside ScanNet and NYUv2, without domain adaptation.All networks are trained and validated on ScanNet and tested on NYUv2 and Tilt-RGBD.
- 4.3 Surface Normal Estimation on Tilt-RGBD: On Tilt-RGBD, DFPN+TAL+IMU achieves the best median and tight-threshold performance, while the proposed method reaches on-par performance despite estimating gravity less accurately.The comparison covers both gravity-aligned and tilted images.
- 4.3 Surface Normal Estimation on Tilt-RGBD: DFPN+TAL+SR significantly outperforms DFPN+TAL+AUG and performs on par with DFPN+TAL+IMU on tilted Tilt-RGBD images without requiring an external sensor.All tilted-image methods show degradation, but DFPN+TAL+SR has less degradation and improves over augmentation.
- 4.4 Network Efficiency: DFPN+TAL with ResNeXt-101 outperforms DORN+TAL in accuracy on ScanNet and NYUv2 while using smaller FLOPS, memory consumption, and realtime inference at 30 FPS.With ResNet-101, DFPN+TAL uses less than one-third of DORN+TAL’s FLOPs, memory, and inference time while achieving comparable accuracy.
- 4.5 Surface normal training loss: TAL converges faster than L2 and AL and improves tight angular thresholds of 5° and 7.5° on both ScanNet and NYUv2 while matching AL at looser thresholds.The comparisons are summarized in Figure 9 and Table 5, whose lower section also reports improvements over FrameNet and VPLNet across metrics.
5 Summary
The paper introduces a spatial rectifier for surface normal estimation in tilted images, jointly trained with an estimator and supported by an efficient network and truncated angular loss. It outperforms state-of-the-art baselines on ScanNet, NYUv2, and Tilt-RGBD, including images with large roll and pitch motions.
- The spatial rectifier warps tilted images so their surface normal distributions match those of the training data.It is jointly trained with the surface normal estimator by synthesizing tilted images from ScanNet.
- The method includes an efficient network that maintains state-of-the-art accuracy with lower computational burden.
- The truncated angular loss addresses L2-loss limitations and improves estimation in regions with small angular error.
- The method outperforms state-of-the-art baselines on ScanNet, NYUv2, and Tilt-RGBD, including large roll and pitch camera motions.