Source-linked AI summary
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction
Aayush Bansal, Bryan Russell, Abhinav Gupta
TL;DR
The paper develops a computational framework for Marr’s theory by predicting detailed surface normals from single images and using them with appearance cues for CAD retrieval. It achieves strong surface-normal and pose results, while retrieving models matching depicted style and pose.
Problem
Marr’s computational realization was limited by difficult surface-normal estimation, unreliable 2.5D-to-3D reconstruction, and insufficient need for intermediate 3D representations.
Method
A VGG-based skip network predicts detailed surface normals, followed by a two-stream network combining images and normals to learn pose and style for CAD retrieval.
Results
The approach recovers fine object detail, achieves state-of-the-art surface-normal accuracy, matches RGB-D-based pose approaches with predicted normals, surpasses state of the art with RGB-D, and retrieves style- and pose-matching CAD models.
Takeaways & Limitations
Accurate predicted surface normals support feed-forward 3D object recognition and CAD retrieval from 2D images while retaining object detail.
Takeaways & Limitations
The evaluation cannot directly compare against approaches and [33] on the NYUD2 test set.
Abstract
from arXiv · showhide
We introduce an approach that leverages surface normal predictions, along with appearance cues, to retrieve 3D models for objects depicted in 2D still images from a large CAD object library. Critical to the success of our approach is the ability to recover accurate surface normals for objects in the depicted scene. We introduce a skip-network model built on the pre-trained Oxford VGG convolutional neural network (CNN) for surface normal prediction. Our model achieves state-of-the-art accuracy on the NYUv2 RGB-D dataset for surface normal prediction, and recovers fine object detail compared to previous methods. Furthermore, we develop a two-stream network over the input image and predicted surface normals that jointly learns pose and style for CAD model retrieval. When using the predicted surface normals, our two-stream network matches prior work using surface normals computed from RGB-D images on the task of pose prediction, and achieves state of the art when using RGB-D input. Finally, our two-stream network allows us to retrieve CAD models that better match the style and pose of a depicted object compared with baseline approaches.
1. Introduction
The paper revisits Marr’s feed-forward idea by predicting 2.5D surface normals from images and using them with appearance to estimate 3D structure and retrieve CAD models. Its framework addresses missing object detail and bridges 2.5D estimation with 3D model alignment.
- Motivation: Single-image 3D understanding remains difficult because surface-normal estimation, 2.5D-to-3D inference, and reliable 3D training data each pose challenges.Recent 2.5D estimation and expanding CAD repositories motivate revisiting this sequential framework.
- Contribution: The work reports state-of-the-art surface-normal estimation, RGB-D pose prediction, and CAD retrieval that better matches depicted style and pose.Its RGB-only pose model is reported as comparable to prior RGB-D-based approaches.
- Contribution: The skip-network uses features from all ConvNet levels to preserve fine details while estimating surface layout.The approach is designed to bridge 2.5D surface estimation and 3D volumetric model retrieval.
- Framework: The system predicts detailed surface normals from a single image, then combines the image and normals to retrieve CAD models matching object style and pose.Training uses NYU Depth v2 for normal prediction and rendered ModelNet views for CAD retrieval.
2. Predicting Detailed Surface Normals
The paper predicts per-pixel surface normals from a single RGB image using a fine-detail skip network built from concatenated CNN responses. A multi-layer regression network is trained end-to-end, improving nonlinear modeling while retaining information across feature scales.
- Problem: The task is to predict a surface-normal map from one image despite variation caused by texture, lighting, and viewpoint.The model predicts normals independently at each pixel location.
- Architecture: The skip network concatenates CNN responses from coarse, mid, and fine levels into per-pixel hypercolumn features.This feature combination is intended to retain fine object details that prior architectures often miss.
- Architecture: A three-layer multilayer perceptron maps each hypercolumn feature to the predicted surface normal.The final fully connected output is normalized, yielding a cosine-loss objective.
- Training: The convolutional layers forming the hypercolumns are fine-tuned jointly with the regression network in an end-to-end pipeline.This differs from approaches that learn only a single regression layer over fixed hypercolumn features.
- Training: The implementation initializes convolutional layers from VGG-16 converted to a fully convolutional network and uses six convolutional layers for hypercolumns.Training uses stochastic gradient descent with sampled pixels from resized images.
3. Learning Pose and Style for CAD Retrieval
The retrieval system combines image appearance with predicted surface normals to learn object pose and style for matching CAD models. It trains pose prediction on rendered CAD views and extends the pose network with Siamese learning for style similarity.
- Style learning: The method uses appearance and predicted-normal information because image-only retrieval is not optimized for matching CAD-model pose and style.Training uses rendered ModelNet views for CAD retrieval and NYUv2 data for surface-normal prediction.
- Pose learning: A two-stream network combines image appearance I with predicted surface normals n(I) to estimate object pose.The streams use CaffeNet-like architectures initialized from pretrained ImageNet networks.
- Pose learning: Registered CAD models provide canonical viewpoints and surface normals for generating rendered training views.Surface normals are generated per pixel by ray casting to model faces.
- Pose learning: The pose model discretizes viewing angles and classifies concatenated pool5 features through fully connected and softmax layers.The loss is backpropagated through all CaffeNet layers during training.
- Pose learning: At test time, the network accepts an image and predicted normals, while also supporting RGB-D input with normals derived from depth.The paper compares pose prediction for predicted-normal and RGB-D inputs.
- Style learning: A Siamese two-stream network extends pose learning to style using hand-aligned similar and dissimilar CAD model-image pairs.The network optimizes a contrastive loss that pulls similar pairs together and separates dissimilar pairs with margin m = 1.
4. Experiments
Experiments evaluate the pipeline’s surface-normal prediction, pose estimation, and style-aware CAD retrieval. The approach captures fine object details, improves surface-normal results, and benefits pose prediction from better normals and combined appearance cues.
- The evaluation analyzes surface-normal prediction, pose estimation, and CAD retrieval components of the pipeline.
- Surface Normal Estimation: The skip network uses multiple VGG-16 layers spanning low, mid, and high levels to estimate surface normals.The architecture combines layers {12, 22, 33, 43, 53, 7} after converting VGG-16 fully connected layers to convolutional layers.
- Surface Normal Estimation: The method outperforms previous approaches on all global scene-layout evaluation criteria, including without Manhattan-world rectification.Manhattan-world rectification worsens performance for this network, unlike Fouhey et al.; the authors report 1–3% gains over Eigen-Fergus’s latest results across criteria.
- Surface Normal Estimation: On chair, sofa, and bed regions, the method performs approximately 3–10% better than previous approaches across all reported statistics.The local-object evaluation focuses on pixels belonging to each furniture category rather than the entire scene.
- Pose Estimation: On NYUD2 validation images, the method outperforms RGB-only and RGB-D baselines for chairs and sofas in pose prediction.The comparison plots the fraction of instances below each pose angular-error threshold, using objects with more than 50% valid depth pixels.
- Pose Estimation: Better predicted surface normals produce better object-pose predictions, while appearance cues and geometric scoring affect retrieval performance.Combining appearance with predicted normals improves performance, whereas combining appearance with depth-derived normals hurts it; geometric scoring outperforms dot-product scoring.
- Style Estimation: Style re-ranking consistently promotes CAD chairs with styles matching the depicted object, such as chairs with wheels.The re-ranked results differ from the initial Pose Network retrievals in style consistency.
5. Conclusion
The paper demonstrates feed-forward 3D object recognition from 2D images through 2.5D surface-normal prediction, combining detailed geometry with image information for pose and CAD retrieval.
- The approach recovers fine object detail and achieves state-of-the-art performance on the NYU depth benchmark.
- A two-stream pose network jointly reasons over the 2D image and predicted surface normals.
- Using predicted normals, pose accuracy is comparable to approaches based on RGB-D images.
- With RGB-D input, the pose network surpasses the state of the art.
- The pose-style network retrieves CAD models matching both the depicted object's style and pose.
- Accurate surface normals could support reliable 2.5D predictions for natural images and contribute to computer graphics and full 3D scene understanding.