Source-linked AI summary
Rethinking Inductive Biases for Surface Normal Estimation
Gwangbin Bae, Andrew J. Davison
TL;DR
Surface normal estimation methods often reuse general-purpose dense-prediction biases, motivating task-specific treatment of camera geometry and neighboring normals. The paper supplies per-pixel ray directions and estimates neighboring-normal relative rotations, yielding detailed, crisp, piece-wise smooth predictions with stronger generalization than a recent transformer-based state-of-the-art model despite much less training data. The approach supports arbitrary image resolutions and aspect ratios, but requires prior camera knowledge.
Problem
Existing surface normal estimators use general-purpose dense prediction models without explicitly encoding ray direction or common relationships between neighboring normals.
Method
The method inputs dense per-pixel ray directions and recasts surface normal estimation as axis-angle relative-rotation estimation between neighboring pixels.
Results
The method shows stronger generalization and higher prediction detail than a recent transformer-based state-of-the-art model despite training on an orders-of-magnitude smaller dataset, while handling arbitrary resolution and aspect ratio.
Takeaways & Limitations
Explicit camera and inter-pixel geometry produces piece-wise smooth predictions that remain crisp near object boundaries and supports domain- and camera-agnostic downstream 3D perception.
Takeaways & Limitations
The model requires prior knowledge of camera intrinsics, although the paper discusses calibration, metadata-based approximation, or single-image estimation as possible sources.
Abstract
from arXiv · showhide
Despite the growing demand for accurate surface normal estimation models, existing methods use general-purpose dense prediction models, adopting the same inductive biases as other tasks. In this paper, we discuss the inductive biases needed for surface normal estimation and propose to (1) utilize the per-pixel ray direction and (2) encode the relationship between neighboring surface normals by learning their relative rotation. The proposed method can generate crisp - yet, piecewise smooth - predictions for challenging in-the-wild images of arbitrary resolution and aspect ratio. Compared to a recent ViT-based state-of-the-art model, our method shows a stronger generalization ability, despite being trained on an orders of magnitude smaller dataset. The code is available at https://github.com/baegwangbin/DSINE.
1. Introduction
Surface normal estimation lacks task-specific inductive biases despite its importance for scene geometry and downstream vision tasks. The paper introduces ray- and rotation-aware architectural changes and reports stronger generalization than a recent ViT-based method despite much less training data.
- Motivation: The paper addresses limited discussion of task-specific inductive biases for surface normal estimation, although normals support many downstream 3D vision tasks.Existing methods commonly reuse dense prediction biases from depth estimation and semantic segmentation.
- Problem: Existing CNN methods share weights across image locations, overlooking pixel ray direction and limiting accuracy and generalization to out-of-distribution cameras.The paper identifies translational weight sharing as suboptimal because ray direction constrains the surface normal.
- Contributions: The method supplies dense ray directions, uses a ray-aware activation, and estimates neighboring-normal relative rotations in axis-angle form.These changes target camera-intrinsics-aware inference, visible predictions, and piece-wise smooth geometry.
- Contributions: The proposed model produces detailed, piece-wise smooth predictions that remain crisp at surface intersections.The architectural design explicitly targets smooth regions without blurring boundaries.
- Results: The method outperforms a recent ViT-based state-of-the-art model quantitatively and qualitatively despite training on an orders-of-magnitude smaller dataset.The reported advantage is framed as stronger generalization on challenging in-the-wild images.
2. Related work
Related work progresses from handcrafted and geometric approaches to deep models for monocular surface normal estimation. The paper positions a transformer-based model trained on more than 12 million diverse images as its principal state-of-the-art comparison.
- Earlier approaches: Early methods discretized normals, used handcrafted features, detected geometric primitives, or imposed Manhattan-world assumptions.These approaches sought dense predictions through classification, patch alignment, or geometric adjustment.
- Deep learning methods: Deep learning methods later incorporated Manhattan lines, spatial rectification, and aleatoric uncertainty to improve surface normal estimation.The cited techniques address geometric structure, tilted images, and difficult regions such as small structures and object boundaries.
- State of the art: Eftekhar et al. trained a U-Net on more than 12 million images spanning diverse scenes and camera intrinsics.This large-scale model established an important data-diversity baseline for comparison.
- State of the art: An updated transformer-based model using sophisticated 3D augmentation and cross-task consistency is identified as the current state of the art and main comparison.The paper evaluates its proposed method against this transformer-based reference.
3. Inductive bias for surface normal estimation
The paper argues that surface normal estimation benefits from camera-ray geometry and explicit relationships between neighboring normals. It encodes ray direction for visibility and camera-aware inference, and models inter-pixel changes as rotations to support smooth yet boundary-preserving predictions.
- Ray direction: Each pixel has a camera ray determined by perspective projection and, under a pinhole model, by focal lengths and the principal point.The ray representation uses pixel coordinates together with camera intrinsics.
- Ray direction: Ray direction supplies geometric cues at vanishing points and occluding boundaries, where surface normals can be constrained by line orientation or ray perpendicularity.These cues connect image geometry to 3D surface orientation.
- Ray direction: Ray direction also restricts the range of visible normals at each pixel, effectively halving the output space.The paper treats this visibility constraint as a task-specific output bias.
- Encoding per-pixel ray direction: Existing CNNs omit ray direction, while transformer positional embeddings are difficult to inter- or extrapolate across out-of-distribution camera intrinsics.The proposed design instead provides dense ray direction explicitly for camera-aware inference.
- Modeling inter-pixel constraints: Neighboring unit normals share a camera-centered origin and can therefore be related by a 3D rotation represented with an axis and angle.The exponential map converts the axis-angle representation back into a rotation matrix.
- Modeling inter-pixel constraints: Relative rotations are near zero on flat surfaces, capture perpendicular or parallel surface relationships, and can transfer evidence to regions with limited visual cues.The paper also motivates rotation learning as less view-dependent than directly predicting normals and useful for preserving piece-wise smooth structure.
4. Our approach
The approach incorporates surface-normal-specific inductive biases through ray-direction encoding, visibility-constrained activation, and learned relative rotations between neighboring normals. A lightweight recurrent architecture implements these components efficiently on a small, diverse meta-dataset.
- 4.1. Ray direction encoding: Focal length-normalized image coordinates are supplied to intermediate network layers, enabling camera-intrinsics-aware inference without cropping or padding images to fixed intrinsics.The alternative would waste computation, lose high-frequency details through downsampling, and fail on wider fields of view.
- 4.2. Ray ReLU activation: Ray ReLU removes the normal component pointing along the viewing ray and re-normalizes the result, restricting predictions to the visible range.The activation enforces n · r ≤ 0 for normalized surface normal n and ray direction r.
- 4.3. Recasting surface normal estimation as rotation estimation: Neighboring surface normals are modeled through pairwise rotations whose learned angles and axes capture smooth regions and surface intersections.The rotation angle is independent of viewing direction, while the axis can be inferred from its 2D image-plane projection and the surface normal.
- 4.4. Network architecture: A lightweight CNN produces initial normals and recurrent state features, then a ConvGRU iteratively estimates rotation angles, axes, and fusion weights for neighborhood updates.Updates occur at H/8 × W/8 resolution with a 5 × 5 neighborhood and five iterations, balancing long-range modeling with computational efficiency.
- 4.3. Recasting surface normal estimation as rotation estimation: Learned fusion weights down-weight neighboring pairs on disconnected or non-smooth surfaces, while ray ReLU keeps rotated normals visible for the target pixel.This supports predictions that are piecewise smooth without forcing unreliable relationships across difficult boundaries.
- 4.5. Training dataset: The training data comprise a small meta-dataset drawn from 10 RGB-D datasets, with 1,655 scenes and 160K images compared with Omnidata’s 1,905 scenes and 12M images.The model is designed for sample efficiency through fully convolutional weight sharing and rotation decomposition into angles and axes.
5. Experiments
Experiments evaluate generalization against state-of-the-art methods across varied datasets and ablate the proposed ray-direction and rotation-estimation biases. The method generalizes strongly to out-of-distribution cameras, produces more detailed in-the-wild predictions, and improves piece-wise consistency near boundaries.
- 5.2. Comparison to the state-of-the-art: The method significantly outperforms other methods across all metrics on Sintel and Virtual KITTI, which contain out-of-distribution fields of view and aspect ratios.The authors attribute this mainly to explicit ray-direction encoding.
- 5.2. Comparison to the state-of-the-art: On OASIS, the method produces significantly more detailed predictions than Omnidata v2, although Omnidata v2 achieves better quantitative accuracy.OASIS annotations cover small, generally flat patches, limiting how faithfully the metrics represent performance.
- 5.2. Comparison to the state-of-the-art: The model trains in 12 hours on one NVIDIA 4090 GPU and uses 40% fewer parameters than Omnidata v2, at 72M versus 123M.It also does not require geometry-aware 3D augmentations or additional supervisory signals.
- 5.3. Ablation study: Adding per-pixel ray direction and iterative rotation estimation both improves overall metrics, with ray encoding especially beneficial for out-of-distribution camera intrinsics.The quantitative ablation is summarized in Table 3.
- 5.3. Ablation study: Rotation estimation yields modest metric gains but improves piece-wise consistency and prediction sharpness near surface boundaries.Large planar regions dominate the metrics, while qualitative results expose improvements that the metrics underrepresent.
6. Conclusion
The paper encodes camera-aware ray directions and neighboring-normal relationships through relative rotation estimation. This combination improves generalization, detail, and boundary-aware piece-wise smoothness while supporting arbitrary image resolutions and aspect ratios.
- 6. Conclusion: Per-pixel ray direction enables camera-intrinsics-aware inference and improves generalization, especially for images from out-of-distribution cameras.The model also generates detailed predictions for arbitrary-resolution and arbitrary-aspect-ratio images.
- 6. Conclusion: Explicitly modeling inter-pixel constraints through rotation estimation produces piece-wise smooth predictions that remain crisp near object boundaries.This models relative rotational relationships between neighboring pixels in the output.
- 6. Conclusion: Compared with a recent transformer-based state-of-the-art method, the approach shows stronger generalization and higher prediction detail despite training on an orders-of-magnitude smaller dataset.Its fully convolutional architecture avoids image resizing and position-encoding interpolation or extrapolation.
7. Limitation and future work
The method relies on camera intrinsics encoded as dense per-pixel ray directions, which introduces a requirement for prior camera knowledge.
- Unknown camera intrinsics make single-image surface normal estimation inherently ambiguous, motivating dense per-pixel ray directions.
- Encoding camera intrinsics helped improve single-image surface normal estimation but requires prior knowledge about the camera.
- Most RGB-D datasets provide calibrated parameters, while monocular cameras can be calibrated using patterns with known relative coordinates.
- For in-the-wild images, camera intrinsics can be approximated from metadata or potentially estimated from vanishing points.
9. Network architecture
The CNN extracts initial surface normals, an initial hidden state, and context features; ConvGRU and convex upsampling use architectures from prior work.
- The CNN extracts initial surface normals, an initial hidden state, and a context feature for the network.
- The ConvGRU cell and convex upsampling layer follow the architectures of prior methods.
10. Data preprocessing
Training applies several probabilistic image degradations, including resampling, compression, and blur, to the input images.
- Each listed augmentation is applied with probability p = 0.1 during training.
- Downsample-and-upsample resizes images to a random 0.2–1.0 fraction before returning them to the original resolution.
- JPEG compression uses a randomly sampled quality q ∼U(10, 90).
- Gaussian blur uses an 11 × 11 kernel with σ ∼U(0.1, 5.0).
- Motion blur is included as a separate training augmentation simulating image motion.
11. Additional figures and video
Additional material includes a qualitative comparison with Omnidata v2 on OASIS in-the-wild images and a tabulated network architecture.
- Table 4 documents the network architecture, including convolution parameters, feature-map resolutions, ray-direction maps, concatenation, and bilinear upsampling.
- Figure 8 provides an additional qualitative comparison with Omnidata v2 on in-the-wild images from the OASIS dataset.