Source-linked AI summary
Designing Deep Networks for Surface Normal Estimation
Xiaolong Wang, David F. Fouhey, Abhinav Gupta
TL;DR
The paper asks how to design CNNs for single-image surface-normal estimation using lessons from 3D scene understanding. It builds a network that combines top-down and bottom-up predictions with structural constraints and intermediate representations, achieving state-of-the-art performance and a reported 7–8% improvement over a standard feed-forward network.
Problem
The paper investigates how CNN architecture for single-image surface-normal estimation can benefit from decades of 3D scene-understanding research.
Method
The model combines top-down global and bottom-up local networks through fusion, adding room-layout, vanishing-point, and edge-labeling constraints.
Results
7–8% improvement over a standard feed-forward network is reported, alongside state-of-the-art surface-normal performance and coherent predictions of room layout and edge labels.
Takeaways & Limitations
Injecting 3D representations and reasoning into CNN design produces a surface-normal estimator that captures fine details such as table legs and curved couch surfaces.
Abstract
from arXiv · showhide
In the past few years, convolutional neural nets (CNN) have shown incredible promise for learning visual representations. In this paper, we use CNNs for the task of predicting surface normals from a single image. But what is the right architecture we should use? We propose to build upon the decades of hard work in 3D scene understanding, to design new CNN architecture for the task of surface normal estimation. We show by incorporating several constraints (man-made, manhattan world) and meaningful intermediate representations (room layout, edge labels) in the architecture leads to state of the art performance on surface normal estimation. We also show that our network is quite robust and show state of the art results on other datasets as well without any fine-tuning.
1. Introduction
The paper asks how lessons from 3D scene understanding can improve CNNs for single-image surface-normal estimation. It combines top-down and bottom-up reasoning with structural constraints and reports state-of-the-art performance and a 7–8% gain over a standard feed-forward network.
- CNN success on semantic tasks has outpaced its study in 3D scene understanding and correspondence.
- The central question is whether decades of 3D scene-understanding research provide design lessons for surface-normal estimation.
- 7–8% improvement over a standard feed-forward network is reported after injecting 3D representations and reasoning into the deep-learning framework.
- The resulting network supports coherent understanding of surface normals, room layout, and edge labels.
- The architecture separately learns top-down and bottom-up processes, then fuses their contradictory beliefs into a final interpretation.
2. Related Work
Related work frames single-view 3D understanding around choosing useful primitives and reconciling local evidence with global scene structure. This paper combines learned pixel-derived representations with reasoning inspired by prior 3D research.
- Single-view 3D understanding asks which primitives support understanding and how local evidence yields global scene interpretation.
- Geons, edges, superpixels, and segments have been explored as primitives, but edge- and segment-based representations remain locally ambiguous.
- Prior work resolves ambiguities through global reasoning or better primitives and likelihood estimates.
- The paper uses CNNs to learn representations and primitives directly from pixels rather than relying on manually designed features such as HoG.
- Its approach joins representation learning and reasoning within a deep-network framework for 3D scene understanding.
3. Overview
The overview presents complementary global and local predictions for single-image surface normals, connected by a fusion network. Additional layout, vanishing-point, and edge tasks inject global and local scene constraints into the final model.
- The motivation is to combine single-view 3D knowledge with CNN representation learning so known scene structure is captured from visual data.
- A top-down whole-image network predicts coarse global structure, while a bottom-up sliding-window network maps local patches to local orientations.
- A fusion network learns to combine the complementary errors of the top-down and bottom-up predictions.
- Additional tasks predict room layout and edge labeling, while fusion incorporates room layout and vanishing-point estimates for a more robust final network.
4. Method
The method uses complementary top-down global and bottom-up local CNNs, then fuses their outputs to estimate surface normals while incorporating room layout and edge information.
- Output: Regression as Classification: Surface normals and room layouts are represented as classification problems using codebooks rather than directly regressed continuous outputs.Normals use triangular coding over k-means codewords, while room layouts use k-medoids codewords learned from 6000 layouts.
- Top-down Global Network: The top-down network processes a 55 × 55 whole-image input through four convolutional layers and predicts global surface normals and room layouts.The shared convolutional representation feeds a 20 × 20 × 20 surface-normal output and 300 room-layout categories.
- Bottom-up Local Network: The bottom-up network uses sliding 55 × 55 windows to predict local normals and convex, concave, occlusion, and no-edge labels for central 13 × 13 patches.Sliding windows recover predictions across the whole image, with 40 normal codewords used for finer local detail.
- Visualization: The top-down and bottom-up networks learn distinct representations: global units capture high-level structures, whereas local units respond to texture and edges.Both networks share the same convolutional structure and have 31 × 31 receptive fields, but their activations reflect different scene evidence.
- Fusion Network: The architecture comprises top-down and bottom-up networks whose outputs are fused because their complementary failure modes can improve results over either network alone.The fusion network combines coarse normals, room layout, local normals, edge labels, and vanishing-point-aligned coarse features.
5. Experiments
Experiments on NYU v2 evaluate the architecture quantitatively and qualitatively, including ablations and transfer to B3DO without fine-tuning. The complete model captures coarse room structure and fine details while outperforming the baselines across reported metrics.
- Dataset and Settings: The evaluation uses 249 NYU v2 training scenes, 200K extracted training frames, and 654 standard test images.The official split contains 215 scenes for testing, with normals, room layouts, and edge labels prepared for training and evaluation.
- Baselines: The method is compared with state-of-the-art surface-normal baselines and a feed-forward coarse CNN adapted from Eigen et al.The adapted baseline regresses surface normals using a negative dot-product loss and has no intermediate representations or designed structure.
- Qualitative Results: The complete architecture captures room layout and fine details such as table tops, chair legs, couch surfaces, nightstands, and counters.Figures 6 and 7 compare input images, Kinect ground truth, and predictions.
- Quantitative Results: The complete approach significantly outperforms all baselines across metrics, with improvements reaching 15% over previous state-of-the-art results.It also performs significantly better than the implemented coarse network of Eigen et al.
- Berkeley B3DO Dataset: On B3DO, the NYU-trained model achieves state-of-the-art performance without further fine-tuning despite substantial differences in scene capture.B3DO includes many close-up and downward-facing views; the evaluation compensates for the latter using detected vanishing points.
- Ablative Analysis: The fusion network combines top-down and bottom-up predictions, whose complementary failure modes are further improved by layout, edge, and vanishing-point inputs.The full fusion network performs especially better on median error and the 11.25° error metric.
6. Conclusion
The paper concludes that injecting prior 3D scene-understanding insights into CNN design yields state-of-the-art surface-normal estimation. Qualitative predictions preserve fine structures including table legs and curved couch surfaces.
- Conclusion: The proposed CNN architecture injects 3D insights from prior scene-understanding research into surface-normal estimation.The authors characterize it as a novel architecture for predicting surface normals.
- Conclusion: The model reaches state-of-the-art performance and qualitatively captures fine details such as table legs and curved couch surfaces.These examples are presented as evidence of the model's qualitative performance.