Source-linked AI summary
IM2CAD
Hamid Izadinia, Qi Shan, Steven M. Seitz
TL;DR
The paper addresses reconstructing a photographically similar indoor 3D scene from one room image when exact furniture models are unavailable. It presents an automatic system that recognizes objects, estimates room geometry, and jointly optimizes CAD object configurations against rendered images using CNN features. The resulting full-scene CAD reconstructions resemble real scenes and improve performance on multiple scene-understanding tasks, while remaining limited by simplified room geometry, occlusion, object-placement, and object-type assumptions.
Problem
IM2CAD asks how to reconstruct realistic indoor scenes from a single photograph using available database models despite missing exact object models and difficult photo-to-CAD matching.
Method
The system recognizes objects, infers room geometry, and jointly optimizes furniture placement and scale by minimizing CNN-feature distance between rendered scenes and the input image.
Results
The system produces full-scene CAD models from single RGB images, faithfully resembling photographed indoor scenes and significantly improving 2D and 3D room-layout and 3D object-location estimation.
Takeaways & Limitations
Available furniture-model databases can support automatic reconstruction of detailed indoor scene CAD models across varied living-room and bedroom imagery.
Takeaways & Limitations
The method assumes cubic room geometry and ground-plane or wall-attached objects, and heavily occluded objects remain challenging.
Abstract
from arXiv · showhide
Given a single photo of a room and a large database of furniture CAD models, our goal is to reconstruct a scene that is as similar as possible to the scene depicted in the photograph, and composed of objects drawn from the database. We present a completely automatic system to address this IM2CAD problem that produces high quality results on challenging imagery from interior home design and remodeling websites. Our approach iteratively optimizes the placement and scale of objects in the room to best match scene renderings to the input photo, using image comparison metrics trained via deep convolutional neural nets. By operating jointly on the full scene at once, we account for inter-object occlusions. We also show the applicability of our method in standard scene understanding benchmarks where we obtain significant improvement.
1. Introduction
IM2CAD extends single-image 3D reconstruction to realistic indoor scenes by assembling approximate furniture CAD models into a full scene. Its automatic pipeline combines object recognition, room modeling, deep-feature matching, and joint scene optimization.
- Prior vision: Roberts’s earlier system inferred complete 3D scenes, including occluded surfaces and recognized objects, from a single photograph using known 3D models.The paper frames IM2CAD as a practical extension of this vision to realistic imagery.
- Motivation: Reliable reconstruction remains difficult because exact object models are unavailable and photograph-to-CAD feature matching is challenging when models differ from photographed objects.These constraints motivate reconstructing scenes from available rather than exact furniture models.
- IM2CAD problem: IM2CAD reconstructs a photographically similar scene from database furniture models, accepting that retrieved objects may resemble rather than exactly match photographed objects.The formulation targets scene similarity while restricting reconstructions to available 3D object models.
- Approach: The system combines ShapeNet’s large object database, object recognition, CNN features for photo-to-render matching, and room reconstruction.These components provide the database, detections, learned visual comparison, and geometric estimation used by the pipeline.
- Contributions: The main contribution is a fully automatic system that produces compact full-scene CAD models from one photo and jointly optimizes object placement and scale while accounting for occlusions.The system includes room and furniture geometry, semantic object segmentation, and transformations specifying position, orientation, and scale.
2. Related Work
Related work covers room-shape estimation, single-object pose recognition, nonphotographic CAD scene synthesis, and image-based scene retrieval. IM2CAD differs by reconstructing scenes from individual furniture models using RGB images and joint occlusion-aware optimization.
- Room reconstruction: Room-shape research estimates floors, walls, and ceilings from single images, while this paper reports competitive room-shape results.The cited approaches represent a progression toward stronger indoor room geometry estimation.
- Object recognition: Single-object recognition methods can align chairs and other objects accurately, but IM2CAD combines object recognition with room estimation to solve for all room objects jointly.The joint formulation accounts for inter-object occlusions.
- CAD scene synthesis: Nonphotographic CAD scene methods use examples, text, or furniture-arrangement optimization, often relying on correlations learned from databases of scene models.The paper positions its image-based reconstruction as a way to generate comprehensive indoor models automatically.
- Scene retrieval: Scene retrieval methods require a complete matching scene, whereas IM2CAD reconstructs each scene from scratch using a database of furniture models.This permits a broader range of reconstructable scenes than complete-scene retrieval.
- Differences: Compared with prior systems, IM2CAD uses RGB rather than RGBD input, learned CNN features rather than handcrafted features, and detailed ShapeNet models rather than low-fidelity synthesized rooms.The related systems also impose assumptions such as axis alignment, absent walls, easy segmentation, or a smaller model database.
3. Algorithm
The algorithm reconstructs a room from one image by estimating room geometry, detecting furniture, retrieving aligned CAD models, and jointly optimizing object configurations against the photo.
- Overview: The pipeline estimates room geometry, detects furniture, aligns CAD models, and optimizes 3D object poses and sizes to match the input image.It combines recognition, room-shape estimation, and rendering-based optimization.
- Room Geometry Estimation: Room geometry is estimated by labeling pixels as ceiling, floor, or walls and fitting a 3D box hypothesis.The method uses a deep convolutional feature approach to rank room hypotheses.
- Room Geometry Estimation: An end-to-end Fully Convolutional Network labels surface orientations while using contextual information from the entire image, including furniture.Furniture can provide cues about adjacent walls rather than being treated only as a distractor.
- Object Detection: Faster-RCNN detects chairs, tables, sofas, bookshelves, beds, night tables, chests, and windows using region proposals and category likelihoods.The implementation uses VGG16 and is fine-tuned on three publicly available detection datasets.
- Object Alignment: CAD retrieval searches 9,193 ShapeNet models rendered from 32 viewpoints, ranking them by cosine similarity between deep features of renderings and detected image regions.The learned comparison is used because simple L2 norms are unreliable under shape, appearance, shading, and occlusion differences.
- Scene Optimization via Render and Match: Joint scene optimization minimizes cosine distance between convolutional features of the rendered scene and the input image across all objects.Higher-layer features support coarse alignment, while lower-layer features refine object placement during later iterations.
4. Coloring CAD models
The system assigns representative colors to furniture and room surfaces for scene optimization and visualization, while Figure 7 illustrates joint refinement toward image similarity.
- Coloring CAD models: Each object receives a medoid color computed from the median color channels within its image mask.The closest color appearing within the projected object mask is selected.
- Coloring CAD models: The same medoid-color procedure is applied to each room wall using masks derived from the estimated room geometry.This supplies representative wall colors for rendered scene visualization.
- Scene Optimization via Render and Match: Figure 7 compares initial placement with intermediate and final renderings, alongside objective-function error and optimization convergence.The objective minimizes dis-similarity between the real and rendered images, with red dots marking displayed iterations.
5. Experimental Results
IM2CAD produces reasonable single-image indoor reconstructions and improves room-layout and 3D scene-understanding benchmarks, while failures arise from pose, detection, occlusion, and room-shape ambiguities.
- Qualitative Evaluation: 100 real-world Zillow Digs living-room and bedroom images cover varied styles, furniture arrangements, complexity, and clutter.
- Qualitative Evaluation: Most test images receive reasonable reconstructions, with the strongest examples detecting, matching, and posing most furniture correctly.Figure 8 presents representative results from the top 30% of outputs.
- Qualitative Evaluation: Typical failures involve omitted or misplaced furniture, incorrect chair poses, chair or table misdetections, non-cubic rooms, and confusion between windows and paintings.Heavy occlusion challenges detection, while local optima can trap pose estimation and non-cubic geometry can defeat room-layout recovery.
- 2D Room Layout Estimation: 12.4% pixel misclassification error improves over 28.9% from on the leading room-layout benchmark.With box fitting, IM2CAD reaches 10.15% error and improves reported results from,, and [33] by 2.7%, 3.1%, and 4.2%, respectively.
- 2D Room Layout Estimation: IM2CAD significantly outperforms previous approaches on LSUN room-layout pixel misclassification and obtains competitive results with [33].
- 3D Room Estimation and Scene Understanding: On SUN RGB-D, IM2CAD outperforms by 13.2% for 3D room estimation and improves scene understanding by 33.5% mean AP and 11.7% scene free-space prediction.Scene optimization further improves scene-understanding accuracy.
- 3D Room Estimation and Scene Understanding: On 3DGP, IM2CAD outperforms by 15.9% for 3D room estimation and performs significantly better for free-space prediction under voxel IoU and mean AP.Evaluation uses 372 test images without training on the 3DGP training set.
6. Conclusion
The paper presents a fully automatic system for reconstructing indoor CAD scenes from single photographs and reports improvements on room-layout and 3D scene-understanding benchmarks, while identifying important scope limitations.
- IM2CAD reconstructs indoor 3D CAD scenes from single photographs using a large database of furniture models, estimating room geometry and object poses.
- Novel room-modeling and scene-optimization methods support reconstructions that faithfully resemble varied living-room and bedroom scenes.
- The approach obtains significant improvement on 2D room-layout estimation and 3D scene-understanding benchmarks.
- The system assumes cubic room geometry and restricts objects to the ground plane or walls, while heavy occlusion and limited object categories constrain its scope.Future extensions include complicated room geometries, objects such as lamps on tables, and additional room and object types.