PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Scene Inference for 3D Object Compositing

Kevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig

arXiv:1912.12297v1cs.GReess.IV

TL;DR

The paper addresses the difficulty of making physically consistent edits to photographs without manually reconstructing their scenes. It automatically infers a 3D scene from a single LDR image, including geometry, reflectance, camera parameters, and lighting, then uses that model for insertion, relighting, and focus editing. User studies found realistic results, especially for indoor or weakly directional lighting, while the inferred scene components remain coarse and error-prone.

  • Problem

    Existing physically grounded photo editing requires substantial artistry or manual scene reconstruction to determine object lighting, shadows, and perspective.

  • Method

    The system combines single-image depth estimation with geometric reasoning and data-driven illumination inference to recover a complete scene model, including out-of-view light sources.

  • Results

    Users can insert and relight objects, change illumination, and modify depth of field; in many cases, people confused insertion results with real pictures over one third of the time.

  • Takeaways & Limitations

    A coarse scene model inferred from one legacy LDR photograph can support physically grounded editing operations with perceptually convincing results.

  • Takeaways & Limitations

    The method works best with diffuse, generally indoor lighting, and its inverse-rendering estimates can contain errors in depth, reflectance, and lighting direction.

Abstract

from arXiv · show

We present a user-friendly image editing system that supports a drag-and-drop object insertion (where the user merely drags objects into the image, and the system automatically places them in 3D and relights them appropriately), post-process illumination editing, and depth-of-field manipulation. Underlying our system is a fully automatic technique for recovering a comprehensive 3D scene model (geometry, illumination, diffuse albedo and camera parameters) from a single, low dynamic range photograph. This is made possible by two novel contributions: an illumination inference algorithm that recovers a full lighting model of the scene (including light sources that are not directly visible in the photograph), and a depth estimation algorithm that combines data-driven depth transfer with geometric reasoning about the scene layout. A user study shows that our system produces perceptually convincing results, and achieves the same level of realism as techniques that require significant user interaction.

1. INTRODUCTION

The paper presents a fully automatic system that infers a physical 3D scene from one LDR photograph, enabling physically grounded image edits with minimal user interaction. Its key technical contributions are data-driven illumination inference, including out-of-view sources, and depth estimation combining learned transfer with geometric reasoning.

  • Existing image editors require artists to infer lighting, shadows, and perspective when inserting synthetic objects.
  • The system automatically infers geometry, illumination, and related scene properties from a single LDR photograph without special hardware.It builds a full 3D scene model including camera parameters and reflectance estimates.
  • Its two technical contributions are illumination inference and depth estimation for recovering a full 3D scene model.Depth inference combines data-driven estimation with geometric reasoning, while illumination inference models out-of-view light sources.
  • The interface supports seamless 3D-object insertion, relighting, post-process depth-of-field changes, and other physically grounded edits.These operations are intended to replace painstaking editing that requires substantial artistry and expertise.
  • A user study found that subjects had difficulty distinguishing edited photographs from ground-truth photos in side-by-side comparisons.
  • The method is limited by errors in inferred geometry, illumination, or materials, and it does not automatically insert objects behind existing scene elements.Camera-pose and depth estimation assume a Manhattan World, although the method is also shown to work when that assumption does not hold.

2. RELATED WORK

Prior work addressed scene geometry, reflectance, and illumination using user guidance, hardware, restricted scene models, or limited lighting representations. This paper instead targets automatic, scene-level inference suitable for physically grounded object insertion from a single image.

  • Physically based image editing requires modeling geometry, illumination, and surface reflectance, but existing approaches often require manual reconstruction or hardware aids.
  • Related single-image inverse-rendering methods may require coarse input depth or estimate illumination only near surfaces, limiting their direct suitability for object insertion.
  • Single-image depth methods learn relationships between image features and geometry, while this work augments appearance-based inference with geometric constraints.
  • Some scene-reconstruction methods predict semantic room and furniture models, but their models may not align well with image edges and boundaries for editing.
  • Prior illumination methods recover point lights, sky-based environment maps, or wrapped-image lighting, depending on the scene and representation.
  • This paper predicts illumination both inside and outside the photograph’s view frustum using a data-driven matching approach.

3. METHOD OVERVIEW

The method first reconstructs scene space and diffuse reflectance, then estimates illumination conditioned on those estimates, and finally enables physically grounded editing operations. Its scene representation uses a depth-derived mesh and area or image-based lights, with indoor/outdoor classification selecting appropriate models and classifiers.

  • The pipeline has three stages: estimate camera, geometry, and diffuse reflectance; infer illumination; then composite objects or edit lighting and depth of field.
  • Scene parameterization: Geometry is represented as a depth map triangulated into a polygonal mesh using an estimated pinhole camera.
  • Scene parameterization: Illumination is represented with polygonal area sources and one or more spherical image-based lights.
  • Scene parameterization: The unconventional scene models are compatible with off-the-shelf rendering software and produce better-looking estimates than simpler planar-geometry models with infinitely distant lighting.
  • Automatic indoor/outdoor scene classification: The system uses indoor/outdoor classification based on k-nearest-neighbor matching of GIST features against indoor and outdoor datasets.It chooses k = 7 and uses majority voting to classify the input image.
  • Automatic indoor/outdoor scene classification: Different training images and classifiers are used according to whether the input is classified as indoor or outdoor.

4. SINGLE IMAGE RECONSTRUCTION

The method reconstructs a single photograph’s physical scene by combining RGBD-based depth transfer with geometric reasoning, producing a dense, physically suitable scene model. It also estimates camera parameters and diffuse materials for downstream rendering and editing.

  • Scene representation: The first reconstruction step estimates a depth map, camera parameters, and spatially varying diffuse materials from one image.The camera model uses focal length, camera center, and extrinsic parameters; diffuse albedo is estimated per pixel.
  • Depth estimation: The depth estimator transfers per-pixel depth from an RGBD database while incorporating geometric information extracted from vanishing points and lines.The formulation begins with data-driven depth transfer and is reformulated to use scene geometry.
  • Depth estimation: The pipeline combines geometric reasoning, sparse surface orientations, and non-parametric depth sampling to produce a dense scene reconstruction suitable for physically grounded editing.The estimated geometric cues enforce orientation constraints, piecewise planarity, and surface smoothness during inference.
  • Depth estimation: Geometric priors enforce Manhattan-world structure, planar-surface orientations, and smoothness in 3D rather than image space.The corresponding terms replace image-based smoothness and dataset-depth prior terms in the optimization.
  • Depth estimation: Compared with Karsch et al.’s depth-transfer method, similar-looking depth maps can conceal substantially different surface orientations.The paper reports that noisy orientations from Karsch et al. can produce unrealistic insertion and relighting effects.
  • Evaluation: The authors report state-of-the-art results on two benchmark datasets using the new depth estimator.

5. ESTIMATING ILLUMINATION

The system estimates visible and out-of-view illumination from a single image, then optimizes a compact lighting model whose rendering matches the input while supporting efficient relighting.

  • Lighting model: Light sources are categorized as visible or out-of-view, with visible sources detected by a trained light classifier and hidden sources estimated from SUN360 panoramas.The resulting model combines area and spherical emitting sources.
  • Visible illumination: The light classifier segments images into superpixels, extracts multi-scale appearance and neighboring features, and predicts whether each superpixel emits or reflects substantial light.Its feature representation contains 340 features per superpixel.
  • Visible illumination: The classifier significantly outperforms thresholding baselines on light-source detection.The passage reports qualitative and quantitative supplemental results but gives no numerical score.
  • Visible illumination: Detected source pixels are back-projected using estimated depth and camera intrinsics, then represented by fitted 3D quadrilaterals.This procedure estimates positions for visible sources; intensities are handled separately.
  • Out-of-view illumination: Out-of-view illumination is predicted by matching sampled rectilinear projections of annotated panoramas to the input image and ranking candidate image-based lights.The method samples ten projections per panorama and uses the top-ranked IBLs for relighting.
  • Intensity estimation: The IBL matching metric compares renderings of canonical objects with varied materials using mean L2 error, while optimization prunes inefficient light sources and improves rendering speed.The optimization selects lighting configurations that match the image while favoring faster rendering.

6. PHYSICALLY GROUNDED IMAGE EDITING

The prototype uses automatically inferred scene models to support physically grounded object insertion, lighting adjustments, and post-process depth-of-field editing. These operations combine interactive controls with physically based rendering and compositing.

  • Drag-and-drop insertion: Users can insert 3D objects by dragging them into photographs, with the system automatically computing scene geometry and illumination.The application adds local depth constraints around the object base and estimates the scene model automatically for each input image.
  • Drag-and-drop insertion: The estimated perspective camera scales inserted objects with depth, while the clicked surface normal determines their automatic orientation.Users can additionally scale and rotate objects through mouse and keyboard controls.
  • Drag-and-drop insertion: Differential rendering composites rendered objects and their lighting contributions into the input photograph using object and non-object renderings.The compositing equation combines an object mask, the rendered object image, the no-object rendering, and the input image.
  • Implementation boundary: Rendering took between 5 minutes and several hours in the reported results, depending on image size, scene structure, materials, and other factors.The implementation uses an unbiased ray-tracer, which contributes to the rendering time.
  • Lighting adjustments: Users can modify light-source intensities to change scene brightness physically, including changing inserted-object illumination and interreflections without altering the rest of the scene.When source intensities are changed equally in object and no-object renderings, only the inserted object's intensity and interreflections change.
  • Lighting adjustments: Tracking each light source's contribution enables real-time lighting adjustments without re-rendering, while predicted depth supports adaptive post-focus blur controlled by depth of field and aperture.The interface also exposes slider controls for softening or hardening shadows and interreflections and for applying synthetic depth of field.

7. EVALUATION

Two-alternative forced-choice studies evaluated whether inserted objects looked realistic against real or fully rendered ground-truth images. The method was often confused with ground truth, especially indoors and under diffuse lighting, despite imperfect scene estimates.

  • 7.1 Real image user study: 30 subjects evaluated real-image pairs containing small real objects or corresponding synthetic insertions, using two-alternative forced-choice realism judgments.The study used 10 synthetic objects across three real scenes, with each subject viewing seven to nine side-by-side trials.
  • 7.1 Real image user study: 34.1% of 232 real-image pairs selected the automatic insertion as the real image, versus 35.8% for manually refined lighting; the difference was not significant.An optimal result would be 50% confusion.
  • 7.1 Real image user study: The real-image study was restricted to small tabletop accessories because obtaining corresponding real and synthetic objects with high-quality geometry and reflectance was difficult.This limits how broadly the real-image findings represent larger or more varied objects.
  • 7.2 Synthetic image user study: 35.5% confusion across 1776 synthetic-study image pairs exceeded the 30.7% and 26.9% baselines by statistically significant margins.The optimal confusion rate was 50%, and the comparison isolated illumination estimation while keeping other pipeline components constant.
  • 7.2 Synthetic image user study: The method performed best indoors and under nondirectional lighting, remained above 31% confusion otherwise, and benefited from inserted objects having reasonable perspective.Subjects’ expertise did not significantly affect performance, with p value = 0.323.
  • 7.3 Ground truth comparison: Estimated depth, reflectance, and illumination often differed from ground truth, yet insertion results could remain convincing because relative scene cues mattered more than absolute physical accuracy.Typical errors included texture-confused depth, reflectance failures under hard shadows or non-Lambertian materials, and inaccurate illumination directions.

8. RESULTS AND CONCLUSION

The system produces physically grounded edits from inferred scene models and works across varied indoor and outdoor images, although quality depends on scene conditions. The authors conclude that automatic insertions often appear realistic and may support faster rendering workflows.

  • 8. RESULTS AND CONCLUSION: Typical results demonstrate applicability indoors and outdoors, including several reasonable results despite the algorithm’s Manhattan World assumption.Failure cases are shown separately, and additional results vary in quality and scene type.
  • 8. RESULTS AND CONCLUSION: The method is better suited to indoor or nondirectional-light scenes; people confused its insertion results with real pictures over one third of the time.For outdoor scenes, simpler illumination methods may suffice, while the estimated geometry remains useful for light intensity and shadow catching.
  • 8. RESULTS AND CONCLUSION: The authors suggest applications in gaming, videos, object deletion or movement, physically based animation, and depth-order or occlusion-boundary inference.These are presented as directions for extending the scene models and editing operations.
  • 8. RESULTS AND CONCLUSION: The estimated scene may enable faster incremental rendering than rerendering a full scene because it can contain fewer light sources and polygons.The proposed use case is quickly inserting a new object into an already created and rendered scene.
  • 8. RESULTS AND CONCLUSION: The interface supports seamless 3D object insertion, on-the-fly lighting changes, and post-process depth-of-field modification for legacy LDR images.These operations are enabled by automatic single-image depth estimation and data-driven illumination inference.

corridor

The figures compare estimated scene components and insertion outcomes with reference imagery, while additional examples and failure cases illustrate the method’s practical range and errors.

  • corridor: Figure 9 compares ground-truth depth, diffuse reflectance, and illumination with the method’s corresponding estimates.Illumination maps are tonemapped for display, and the scenes correspond to insertion results in Figure 8.
  • corridor: Figure 10 shows convincing indoor and outdoor results, including non-Manhattan scenes, without explicit assumptions about scene geometry.Virtual staging is identified as a natural application.
  • corridor: Figure 11 shows failures from inaccurate geometry, including missing shadows, flat depth on slanted surfaces, and errors amplified by large insertions or reflective scenes.Examples include a dragon, chest, statue, and curved mirrors.
Loading 1912.12297v1…