Source-linked AI summary
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
David Eigen, Rob Fergus
TL;DR
The paper asks whether depth, surface normals, and semantic labels can be predicted with one adaptable architecture. It uses a multiscale convolutional network that makes coarse global predictions and progressively refines them without superpixels. The model achieves state-of-the-art results on all three investigated tasks.
Problem
Scene understanding includes multiple pixel-level tasks, including semantic labeling, surface normals, and depth estimation.
Method
A common multiscale convolutional architecture directly predicts pixel-maps, progressively refining coarse global outputs with finer-scale networks.
Results
The model achieves state-of-the-art results on depth prediction, surface normal estimation, and semantic segmentation.
Takeaways & Limitations
The shared architecture provides a convenient baseline for three scene-understanding tasks while sharing computation across modalities.
Takeaways & Limitations
A compatibility constraint between directly predicted normals and normals derived from predicted depth did not improve either task individually, so the paper does not use it.
Abstract
from arXiv · showhide
In this paper we address three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling. We use a multiscale convolutional network that is able to adapt easily to each task using only small modifications, regressing from the input image to the output map directly. Our method progressively refines predictions using a sequence of scales, and captures many image details without any superpixels or low-level segmentation. We achieve state-of-the-art performance on benchmarks for all three tasks.
1 Dept. of Computer Science, Courant Institute, New York University 2 Facebook AI Research
The section lists the authors' contact email addresses.
- The listed contact addresses are deigen@cs.nyu.edu and fergus@cs.nyu.edu.
1. Introduction
The paper treats scene understanding as encompassing semantic, geometric, and interaction-related pixel-level descriptions, then develops one multiscale architecture for three such tasks. Its shared design targets rapid adaptation, efficient multimodal systems, and strong performance across depth, surface normals, and semantic segmentation.
- The paper addresses depth prediction, surface normal estimation, and semantic segmentation with one common architecture.
- The multiscale network directly generates pixel-maps and refines them through convolutional stacks applied at increasing resolution.It avoids low-level superpixels or contours while aligning predictions to image details.
- At test time, the architecture generates all three outputs in real time at approximately 30Hz.
- A general pixel-map regressor can speed adaptation to new tasks by concentrating task-specific work on the training set and loss function.
- Sharing one architecture can simplify systems requiring multiple modalities, including robotics or augmented reality.
- Shared computation between depth and normals makes the system more efficient.
2. Related Work
The related-work discussion contrasts approaches that aggregate local or parallel multiscale predictions with this model's global-first, sequential-refinement strategy. The architecture uses a large full-image field of view and avoids superpixels or post-processing while adapting across tasks.
- Unlike methods using local features or proposal regions, this network combines local and global views across scales.
- For depth prediction, the paper extends a two-stage model into a three-scale network that also handles surface normals and per-pixel semantic labeling.
- Compared with prior normal-estimation systems, the authors report equally good or superior results using a more general multiscale architecture.
- For semantic segmentation, the method predicts globally first and then performs iterative local refinements informed by the scene context.
- The model uses a large full-image field of view at the coarsest scale and does not require superpixels or post-process smoothing.
- The architecture applies networks for different scales in series, enabling more complex edits and refinements than averaging parallel predictions.
3. Model Architecture
The model first predicts a coarse global output and then refines it through finer-scale networks using increasingly detailed image information. Architectural changes include greater depth, a higher-resolution third scale, and feature-map transmission between stages.
- The multiscale network first predicts a coarse global output from the entire image, then refines it with finer-scale local networks.
- Architectural improvements include a deeper network, a third higher-resolution scale, and multichannel feature maps passed between the first two scales.The first two scales are trained jointly from the start.
- Scale 1: Full-Image View: The first scale uses a large full-image field of view to produce spatially varying coarse features.Two fully connected layers connect each output location to all image features.
- The model is trained in AlexNet-based and VGG-based sizes to measure the impact of model size across tasks.
- Scale 2: Predictions: The second scale combines a narrower, more detailed image view with full-image information to produce mid-level predictions.
- Scale 3: Higher Resolution: The third scale concatenates Scale-2 outputs with finer-stride features from the original input to refine higher-resolution details.The final output resolution is half the network input.
4. Tasks
The same multiscale architecture supports depth, surface-normal, and semantic-label prediction, with task-specific targets and losses. Depth uses log-depth differences and gradient matching, normals use normalized three-component vectors, and labels use pixelwise softmax classification.
- All three tasks use the same architecture structure but different loss functions and target data.
- Depth: Depth prediction compares predicted and ground-truth log depth maps over valid pixels.The loss also includes first-order gradient matching, encouraging local structure to follow depth gradients without degrading measured l2 performance.
- Depth: The depth ground-truth mask excludes pixels where ground-truth depth is missing.
- Surface Normals: Surface-normal prediction outputs three components per pixel, normalizes each vector to unit l2 norm, and compares it with ground truth using a dot-product loss.Ground-truth normals are estimated by fitting least-squares planes to neighboring point sets in the point cloud.
- Semantic Labels: Semantic labeling uses a pixelwise softmax classifier with one output channel per class and a pixelwise cross-entropy loss.For NYUDepth RGB-D labeling, ground-truth depth and normals are supplied as additional input channels.
5. Training
Training proceeds in two phases, with Scales 1 and 2 trained jointly before Scale 3 is trained after their parameters are fixed. Random crops and shared settings reduce the cost of high-resolution training while preserving task-specific adjustments.
- Training first jointly optimizes Scales 1 and 2, then fixes them while training Scale 3.
- Scale 3 training uses random 74x55 crops because its four-times-larger pixel count makes full-image gradient steps expensive.
- Most initialization and learning-rate settings are shared across tasks after tuning on depth and checking them on 50 validation scenes.
- Normals use a learning rate 10 times larger than depth or labels, while labeling changes relative layer rates and increases layer-1.6 dropout to 0.8.
- Training uses ImageNet weights for Scale 1 convolutional layers, random initialization elsewhere, batch sizes of 32 or 16, and a tenfold learning-rate drop after about 2M steps.The smaller batch size is used for the VGG-initialized model because of memory constraints; training continues for another 0.5M steps.
- Random scaling, rotation, translation, color, flips, and contrast augment the data with corresponding transformations applied to RGB, depth, normals, and labels.
- Depth and normals share the Scale 1 stack but use separate Scale 2 and 3 stacks, producing both outputs from RGB with a 1.6x speedup.
6. Performance Experiments
The multiscale architecture performs strongly across depth prediction, surface-normal estimation, and semantic labeling on indoor and outdoor benchmarks. Results also show adaptability across datasets and tasks, with larger models generally improving performance.
- 6.1. Depth: Our depth model achieves the best performance across all reported metrics, while the larger VGG model outperforms the smaller AlexNet version.Qualitative results also show sharper depth details than [8].
- 6.2. Surface Normals: The larger surface-normal model substantially outperforms all comparison methods, whereas the smaller model performs similarly to or slightly better than Wang et al.Examples retain details including blanket curvature, sofas, and objects.
- 6.3.1 NYU Depth: Our semantic-labeling model decisively outperforms comparison methods on the 4- and 14-class NYUDepth tasks and exceeds selected baselines on the 40-class task.It produces large constant regions despite using neither superpixels nor piecewise-constant assumptions.
- Experimental settings: The experiments evaluate the architecture across NYUDepth v2, Sift Flow, and Pascal VOC using task- and dataset-specific training and evaluation settings.Sift Flow reuses NYUDepth kernel sizes and learning rates, while adjusting output sizes for different image dimensions.
- 6.3.2 Sift Flow: On Sift Flow, the smaller model substantially outperforms all methods except Long et al., while the larger model performs similarly to Long et al.The experiment uses regular and class-reweighted model variants.
7. Probe Experiments
The probe experiments examine how scale choices and geometric inputs affect depth, normal, and semantic-labeling performance. Adding scales improves all tasks, while the most influential scale depends on the task and input configuration.
- Contributions of Scales: The experiments compare depth, normals, and semantic labeling using individual scales, scale combinations, and all three scales.For Scale 1 alone, the coarse stack is modified to predict target-size pixel maps directly.
- Contributions of Scales: Coarse Scale 1 contributes most for depth and normals, whereas local Scale 2 contributes most for semantic tasks under the tested architecture.For semantic labeling, this difference reflects that depth and normals channels are introduced at Scale 2.
- Effect of Depth and Normals Inputs: The input study evaluates 13-class semantic labeling with RGB only, RGB plus predicted depth and normals, or RGB plus true depth and normals.
8. Discussion
The discussion presents the multiscale convolutional architecture as a simple, fast approach for jointly modeling depth, surface normals, and semantic labels. Its broad benchmark performance and shared design make it a convenient baseline for these three modalities.
- The models beat existing methods on the vast majority of explored benchmarks for depth, surface normals, and semantic labels.
- The architecture provides a simple and fast common system for producing a rich account of a scene across three modalities.
- The authors position the model as a convenient new baseline for the three tasks.Code and trained models are reported as available online.