Im2Struct: Recovering 3D Shape Structure from a Single RGB Image

1. Introduction

• Deep image-to-volume models primarily output voxel representations, which lose shape topology and part structure.Once converted to a volume, topology can be difficult to recover, especially when reconstruction contains topological defects.Source: §1 · p. 1
• Part decomposition and relations are implicit in 2D, making recovery ill-posed amid fine-grained structures and varied appearance or clutter.The paper identifies diverse substructures, textures, lighting, and background clutter as challenges.Source: §1 · p. 2
• A structure-masking network accentuates object structures at multiple scales, then fused mask and image features feed recursive cuboid-hierarchy decoding.The recovery network explicitly models part relations and decodes a tree of cuboids with plausible spatial configuration.Source: §1 · p. 2
• The two networks are jointly trained on image-mask and cuboid-structure pairs generated from part-segmented 3D CAD models.The paper also reports using mechanisms to avoid overfitting.Source: §1 · p. 2
• The paper proposes direct recovery of 3D shape structures, including part connectivity and symmetry, from single RGB images.The authors describe this detailed recovery as unprecedented to their knowledge.Source: §1 · p. 2
• Recovered structures are proposed for refining volumetric reconstructions and assisting structure-aware editing of 2D images.The cuboid structure serves as an editing proxy in the illustrated application.Source: Figure 1 · p. 1, §1 · p. 2

Discriminative vs. Generative

• For image-to-3D mapping, the authors choose a discriminative model because it is easier to train, using moderate-sized training data.The passage contrasts this choice with generative models, which it describes as hard to train.Source: §2 · p. 3

Geometry reconstruction vs. Structure recovery

• Prior deep models generated volumes, point clouds [4], cuboids [19], or manifold surfaces [14], but these representations lacked part structure and relations.Source: §2 · p. 3
• Li et al.'s recursive neural networks encode and decode arbitrary part counts and relation types [13], and this paper integrates that approach into RGB-to-structure mapping.Source: §2 · p. 3

3.2. Structure Masking Network.

• A two-scale network predicts a 56 × 56 binary contour mask from a 224 × 224 RGB image, with jump connections fusing features for detailed structure prediction.The first scale captures whole-image information; the second refines the mask.Source: §3.2 · pp. 3–4

3.1. Network architecture

• Masking-network features and original-image CNN features are fused into an 80D code, which the RvNN recursively unfolds into a hierarchy of boxes.The original-image features complement imperfect mask predictions by retaining object information.Source: §3.1 · p. 3, §3.3 · p. 4

3.3. Structure Recovery Network

• Feature fusion combines masking-network and original-image features, which the authors report improves structure recovery accuracy and domain adaptation from rendered to real images.The two feature sources are complementary: masking features retain shape detail amid background, texture, and lighting variation, while image features retain object information.Source: §3.3 · p. 4
• The RvNN expands a root code into leaf, adjacency, and symmetry nodes, using a jointly trained classifier to select the decoder at each node.The box decoder maps a leaf code to 12 parameters for a 3D oriented box; symmetry decoding outputs a generator code and eight symmetry parameters.Source: §3.3 · p. 4

3.4. Training details

• Training first fits the masking network, then jointly refines it with structure recovery, whose loss sums box-parameter reconstruction error and node-classification cross-entropy.Shapes are resized to a unit bounding box before reconstruction errors are computed.Source: §3.4 · p. 5
• Figure 4 reports lower reconstruction loss with the structure masking network than without it.The top plots show training and testing losses for box reconstruction, symmetry recovery, and node classification.Source: Figure 4 · p. 5, §3.4 · p. 5

4. Experiments

• The method is evaluated on 800 ShapeNet shapes—500 chairs, 200 tables, and 100 aeroplanes—with 70% used for training and 30% for testing.The authors also conduct a qualitative Google image search challenge.Source: §4 · p. 5

4.1. Training data generation

• Each 3D shape yields 60 rendered RGB views, and its segmented parts are organized into a hierarchy using connectivity and symmetry; structure-aware deformation expands training to 16K shapes.Views combine 36 rotations and three elevations with 24 random views; each shape receives 20 deformations that preserve part connectivity and symmetry.Source: §4.1 · pp. 5–6

4.2. Results and evaluation

• The masking network captures object details while filtering background clutter, and the Google image challenge reports detailed recovery of part connections and symmetries from single views.Examples include reflectional symmetry in chair legs and airplane wings, and rotational symmetry in swivel-chair or table legs.Source: §4.2 · p. 6, Figure 6 · p. 7
• Failures include a chair not composed of multiple parts and a table whose structure is unseen in the training dataset.The authors state that the unseen table does not receive a reasonable recovered structure.Source: §4.2 · p. 6, Figure 6 · p. 7
• Structure recovery is evaluated with Hausdorff error and thresholded accuracy; comparisons report that structure masking significantly helps decoding and VGG-19 boosts performance to a certain degree.The comparison includes a vanilla VGG-16 baseline and a VGG-19 replacement.Source: §4.2 · pp. 6–7, Table 2 · p. 7
• Against Huang et al. and Tulsiani et al. [19], the method produces structures described as more faithful to the input and more structurally plausible.The paper attributes faithfulness to structure masking and plausibility to part-relation recovery.Source: §4.2 · p. 7

• The applications use recovered shape structure for image editing and processing, demonstrated through structure-aware editing and 3D volume refinement.Source: §5 · pp. 7–8

Structure-aware image editing

• For image editing, the method recovers cuboids and part relations, estimates camera view, and uses cuboid projections to guide image segmentation before deformation.Segments are assigned to cuboids by image-space overlap; the editing method in is then applied.Source: §5 · p. 8

Structure-assisted 3D volume refinement

• Limited volumetric resolution can leave reconstructed shapes with missing parts and broken structure, motivating structure-assisted refinement.Source: §5 · p. 8
• For volume refinement, recovered cuboids segment a 3D-GAN volume, and recovered part symmetry is used to complete missing voxels.The volume and cuboids are aligned using camera-view estimation.Source: §5 · p. 8, Figure 9 · p. 8

• The framework combines structure masking to discern object structure with structure recovery to infer 3D cuboids, reporting fidelity to the image and plausible 3D structures.The authors describe it as the first method, to their knowledge, to recover detailed 3D shape structures from single 2D images.Source: §6 · p. 8
• The method fails on object categories unseen during training and currently recovers cuboids, not underlying part geometry.Source: §6 · p. 8
• Future directions include incremental part assembly, synthesizing detailed part geometry, and studying correlations between 2D features and 3D structure for more explainable decoding.Source: §6 · p. 8