PaperlayerView exact revision on arXiv

Source-linked AI summary

Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene

Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A. Efros, Jitendra Malik

arXiv:1712.01812v2cs.CV

TL;DR

Recovering a scene's 3D structure from one image is difficult because existing representations can lose object distinctions or invisible geometry. The paper predicts a factored representation of layout and object shape and pose, and reports qualitative and quantitative benefits over depth maps and voxel grids.

  • Problem

    Depth maps and surface normals omit invisible scene parts, while voxel grids mix shape and pose and erase distinctions between objects.

    Source: §1 · p. 1

  • Method

    Convolutional networks predict an amodal scene layout and object shape and pose from a single image, using scene-level and object-centric components.

    Source: §3 · p. 2, §3.1 · p. 2

  • Results

    The authors report qualitative and quantitative benefits over per-pixel depth maps and single voxel grids.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Factoring scenes into discrete entities keeps pose and shape uncertainty separate and permits reasoning about changes such as moving an object.

    Source: §1 · p. 1

    The system does not reason about scene physics or support relationships, and training on synthetic data limits performance on real data.

    Source: §5 · p. 8

Abstract

from arXiv · show

The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose. We propose a convolutional neural network-based approach to predict this representation and benchmark it on a large dataset of indoor scenes. Our experiments evaluate a number of practical design questions, demonstrate that we can infer this representation, and quantitatively and qualitatively demonstrate its merits compared to alternate representations.

1. Introduction

Existing scene representations either collapse objects into undifferentiated surfaces or voxels, or omit invisible geometry. The paper proposes inferring a scene's layout and individual objects, with shape and pose represented separately, from a single image.

  • Depth and surface-normal maps omit invisible scene portions, while both 2.5D and voxel representations fail to preserve object distinctions.Voxel grids also mix shape and pose beliefs, although the thickness of a table may be easy to infer while its exact depth is difficult.Source: §1 · p. 1
  • The proposed representation factors a scene into its amodal enclosing layout and discrete objects, each represented by 3D shape and pose.The figure illustrates removing objects or the table and considering a moved chair.Source: §1 · p. 1, Figure 1 · p. 1
  • Factoring separates shape from pose uncertainty and allocates voxel resolution automatically, enabling high-resolution output.The organization into discrete entities also permits reasoning about questions such as moving a chair.Source: §1 · p. 1
  • The method uses convolutional neural networks to predict scene layout and object shape and pose from an image and generic object proposals.It is trained using synthetic data and shown on synthetic and natural data.Source: §1 · p. 1
  • The experiments include extensive ablations, assess full-representation inference and its performance limitations, and compare against standard representations.The introduction identifies design decisions as non-obvious and describes the ablations as extensive.Source: §1 · pp. 1–2
  • The paper compares its representation with per-pixel depth maps and single voxel grids, reporting qualitative and quantitative benefits.Figure 6 presents qualitative comparisons, Figure 8 quantitative ones, and the paper also shows qualitative results on NYUv2.Source: §1 · p. 2

2. Related Work

Prior work estimates view-based scene properties, reconstructs isolated objects under relaxed assumptions, or predicts selected 3D object properties. This paper positions its approach as jointly handling scene layout, detailed object shape, and pose.

  • View-based depth and orientation estimates cannot describe nonvisible scene parts, while voxel reconstructions had been demonstrated only for isolated, presegmented objects.Scene reconstruction adds challenges of object delineation, shape-and-pose uncertainty, and resolution scaling.Source: §2 · p. 2
  • RGB-D completion supplies object locations, while CAD retrieval and scene retrieval assume a predetermined shape dictionary; this work jointly infers shape and pose.The authors state that their approach requires neither precise visible-pixel locations nor a predetermined set of objects.Source: §2 · p. 2
  • Earlier work predicts 3D pose, low-dimensional parametric shapes [8, 34], or surface normals, whereas this representation includes volumetric shape, pose, and scene layout.Source: §2 · p. 2

3. Approach

The approach predicts an amodal scene layout and object shape and pose from an image using end-to-end trainable scene and object-centric networks. Layout is represented as an inverse-depth map, while object predictions factor shape from pose.

3. Approach

  • The scene network predicts an amodal layout, while an object-centric network maps bounding boxes to object shape and pose; the components can be trained end-to-end.The paper sketches the architecture and refers to the supplement for full specifications.Source: §3 · p. 2

3.1. Layout

  • The layout is an inverse-depth map of enclosing surfaces as if the scene contained no objects, a more general alternative to fitting a vanishing-point-aligned 3D box.A skip-connected network predicts the layout with an end-to-end L1 objective.Source: §3.1 · p. 2

3.2. Object Predictions

  • Objects use canonical, centered shape coordinates to specify pose, and an autoencoder-mimicking stage bootstraps shape-code prediction before joint voxel-loss optimization.The canonical shape is normalized and centered; the decoder is initialized from the autoencoder.Source: §3.2 · pp. 2–3
  • For each box, fine-resolution ROI features, coarse full-image context, and box location predict a shape code, scale, translation, and rotation; the shape code decodes to voxels.The object network combines all three feature sources before predicting the factors.Source: Figure 2 · p. 3, §3.2 · p. 3
  • Rotation is classified over 24 quaternion bins using negative log-likelihood, while anisotropic scale and translation are regressed with squared Euclidean loss, in log-space for scale.The final rotation prediction is the most likely bin.Source: §3.2 · p. 3

3.3. Training to Predict A Full Scene

  • With annotated boxes and 3D structure, full-scene training uses 1K Edge Boxes per image; IoU >0.7 marks foreground, <0.3 background, followed by 8 mimicking and 1 volumetric-loss epoch.Proposals are matched to ground-truth objects using modal 2D bounding-box IoU.Source: §3.3 · pp. 3–4

4. Experiments

The experiments validate the proposed predictor across synthetic SUNCG and natural NYU images, first isolating object prediction, then evaluating full detection and comparing scene representations.

4.1. Datasets

  • SUNCG provides over 400,000 rendered training images, with evaluation restricted to beds, chairs, desks, sofas, tables, and televisions; NYU is used for qualitative generalization checks.SUNCG houses are split 70%-10%-20% into train, validation, and test sets.Source: §4.1 · p. 4

4.2. Metrics

  • Evaluation measures shape, rotation, scale, and translation separately, then defines detection average precision using thresholds for all five metrics, including the 2D box.The component metrics use medians and threshold-based fractions for isolated shape prediction.Source: §4.2 · p. 4

4.3. Analyzing 3D Object Prediction

  • With ground-truth boxes, decoder fine-tuning improves shape prediction; retrieval is less robust, rotation classification beats regression, and context helps each error metric, especially translation and scale.The study isolates 3D object prediction from detection errors and compares decoder, retrieval, feature, and loss choices.Source: §4.3 · pp. 4–5, Table 1 · p. 5

4.4. Placing Objects in Scenes

  • 3D detections can be duplicated even when the underlying boxes were not classified as duplicates by 2D non-maximum suppression.The paper identifies this as an error mode in full-scene predictions.Source: §4.4 · p. 6, Figure 5 · p. 6
  • 40.3% average precision is reported on SUNCG for the full 3D prediction task.Figure 7 also plots variants that relax one condition at a time.Source: §4.4 · p. 6, Figure 7 · p. 7

4.5. Comparing Scene Representations

  • Across visible depth, voxel occupancy, objects, and visible or amodal layout, the factored representation generalizes better to other tasks and predicts visible layout better than full-image depth.Each representation performs best on the task it was trained for.Source: §4.5 · pp. 7–8, Figure 6 · p. 7, Figure 8 · p. 8

4.6. Results on NYU

  • On NYU test images, the SUNCG-trained model produces qualitative scene interpretations using RGB input and 2D box proposals from.No additional training on NYU is described.Source: §4.6 · p. 8, Figure 9 · p. 8

5. Discussion

The discussion argues for representing a scene as layout plus objects factored into shape and pose, and presents a learning-based system that infers this representation from one image. It also identifies unmodeled physical relationships and reliance on synthetic training data as remaining limitations.

5. Discussion

  • The proposed scene representation factors structure into layout and individual objects, each described by shape and pose.The authors present this as the representation to infer for understanding 3D scene structure.Source: §5 · p. 8
  • A learning-based system infers this 3D representation from a single image.Source: §5 · p. 8
  • The system does not reason about scene physics or support relationships, and synthetic training data limits performance on real data.The authors describe these as remaining challenges.Source: §5 · p. 8

Related papers

Loading 1712.01812v2…