PaperlayerView exact revision on arXiv

Source-linked AI summary

OBJECT 3DIT: Language-guided 3D-aware Image Editing

Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, Tanmay Gupta

arXiv:2307.11073v1cs.CVcs.AIcs.GR

TL;DR

Existing generative editing methods lack the 3D awareness needed to preserve geometry and lighting when editing objects from a single image. The paper introduces OBJECT, a procedurally generated editing dataset, and 3DIT, a language-guided model trained on it. 3DIT outperforms baselines across four tasks, is preferred by human evaluators, and generalizes to real-world images despite synthetic training data.

  • Problem

    Generative models struggle to translate or rotate objects from a single image while preserving 3D geometry, scene context, and lighting.

    Source: §1 · p. 2

  • Method

    The paper introduces OBJECT, a dataset of procedurally generated object-editing examples, and fine-tunes the language-conditioned 3DIT model on it.

    Source: §1 · p. 2

  • Results

    3DIT outperforms baselines across all four OBJECT tasks, receives human preference scores of 70% or more across the tasks, and generalizes to real-world images.

    Source: §1 · p. 2, §5.3 · p. 9

  • Takeaways & Limitations

    3DIT’s edits tend to preserve scene structure, including object scale and viewpoint, appropriate shadows, and previously occluded image regions.

    Source: §1 · p. 2

    The authors suggest that more photorealistic simulators or fine-tuning on a small set of handcrafted real-world edits may improve real-world transfer.

    Source: §6 · p. 10

Abstract

from arXiv · show

Existing image editing tools, while powerful, typically disregard the underlying 3D geometry from which the image is projected. As a result, edits made using these tools may become detached from the geometry and lighting conditions that are at the foundation of the image formation process. In this work, we formulate the newt ask of language-guided 3D-aware editing, where objects in an image should be edited according to a language instruction in context of the underlying 3D scene. To promote progress towards this goal, we release OBJECT: a dataset consisting of 400K editing examples created from procedurally generated 3D scenes. Each example consists of an input image, editing instruction in language, and the edited image. We also introduce 3DIT : single and multi-task models for four editing tasks. Our models show impressive abilities to understand the 3D composition of entire scenes, factoring in surrounding objects, surfaces, lighting conditions, shadows, and physically-plausible object configurations. Surprisingly, training on only synthetic scenes from OBJECT, editing capabilities of 3DIT generalize to real-world images.

1 Introduction

The paper targets object edits that preserve 3D geometry and scene lighting, which existing generative models struggle to handle from a single image. It introduces OBJECT and 3DIT, reporting strong performance across four tasks and transfer to real-world images.

1 Introduction

  • Generative models promise object-level edits beyond the pixel-level work of professional editing tools, which can be daunting for novice users.The introduction situates the work in the growing demand for image editing by amateur photographers.Source: §1 · p. 2
  • Existing generative models lack reliable object-centric translation or rotation that preserves 3D geometry, scene context, and original lighting from a single image.Some methods operate on isolated objects, while others require multiple viewpoints; the paper identifies a need for single-image editing with re-rendered shadows under original lighting.Source: §1 · p. 2
  • OBJECT provides 400K procedurally generated editing examples with language instructions and ground-truth edits for four object manipulations.The rendering process aims to capture physically plausible changes in geometry, illumination, and shading.Source: §1 · p. 2
  • 3DIT supports the four manipulation tasks using language conditioning and is fine-tuned on OBJECT from the Zero-1-to-3 diffusion model.The training curriculum proceeds from web-scale image-text pretraining to Zero-1-to-3 pretraining and OBJECT fine-tuning.Source: §1 · p. 2
  • 3DIT outperforms baselines on scene-edit faithfulness across all four OBJECT tasks and receives human preference over baselines more than 70% of the time.Its edits tend to preserve scene structure, object scale and viewpoint, and appropriate shadows; the model also generalizes to CLEVR and real-world images.Source: §1 · p. 2

2 Related work

Related work spans language-guided generative editing, 3D-aware image methods, scene rearrangement, and synthetic datasets. The paper positions OBJECT and 3DIT around object-level scene editing that accounts for geometry and lighting, alongside a shortage of public 3D-scene editing benchmarks.

Image editing with generative models

  • Despite progress in generative image editing, existing models lack the 3D awareness needed to maintain geometric and lighting consistency.OBJECT is presented as a way to bridge this gap and evaluate future methods.Source: §2 · p. 3
  • Generative editing developed from manual image manipulation through GAN-based methods to diffusion models paired with large image-text datasets.The paper cites Photoshop and prior generative editing research as part of this progression.Source: §2 · p. 3

3D-aware image editing

  • 3DIT considers the entire scene—including geometry, lighting, and background attributes—rather than only the object of interest.The related-work discussion contrasts this with research lifting 2D images to 3D, including StyleNeRF [23], EG3D [5], SJC [70], DreamFusion [52], Zero-1-to-3 [45], and Make-It-3D [68].Source: §2 · p. 3

Scene rearrangement

  • Unlike room rearrangement and embodied-agent work, this paper focuses on language-instructed, object-level 3D-aware image editing.The paper says 3DIT is trained with OBJECT to edit scenes with realism and 3D coherence.Source: §2 · p. 3

Synthetic datasets for vision models

  • OBJECT uses procedurally generated scenes with natural objects from Objaverse [12] and associated language instructions.The related-work discussion distinguishes these from synthetic visual-reasoning datasets such as CLEVR [31] and CLEV-ERER [78], which use simple primitives and questions about scene content.Source: §2 · p. 3

Benchmarks for image editing

  • Public benchmarks for evaluating generative image editing are scarce, especially for 3D scene editing, and several existing benchmarks are not publicly available.The paper states that OBJECT will be made publicly available.Source: §2 · p. 3

3 OBJECT: A benchmark for Object Editing in Context

OBJECT is a benchmark for language-guided object edits in multi-object scenes, where models must infer scene properties from pixels rather than receive the underlying 3D scene. It combines four edit tasks with procedurally generated scenes, assets, layouts, and lighting designed to produce physically plausible examples.

3.1 Object editing tasks

  • OBJECT contains four object-editing tasks: translation, rotation, insertion, and removal, each targeting one object in a potentially multi-object scene.Models receive language descriptions and task-specific values, while the underlying scene geometry, object locations, and lighting are not provided.Source: §3.1 · p. 4, §3 · p. 4
  • Translation requires moving an object while preserving its pose and surface contact, with newly visible regions, perspective scale, and changed illumination handled.The camera remains fixed relative to the scene, so moving the object can alter its apparent size and lighting.Source: §3.1 · p. 4
  • Rotation requires inferring object shape from one viewpoint and generating the changed ground shadow after turning it around the vertical axis.The specified rotation is counter-clockwise around the object’s center of mass and perpendicular to the ground.Source: §3.1 · p. 4
  • Insertion requires generating a described object at a designated location in a stable pose with surface contact and a realistic shadow.Removal requires locating and segmenting the object, then inpainting its region from scene context, including previously occluded areas.Source: §3.1 · p. 4

3.2 Benchmark curation

  • OBJECT addresses the scarcity of paired editing data by synthesizing large volumes of examples from photorealistic rendering and 3D asset libraries.Each example pairs an initial image and edit description with the resulting edited image.Source: §3.2 · p. 4
  • The benchmark uses 3D objects selected from Objaverse and filters its more than 800K assets to 59K usable objects.Crowdsourced category labels and object descriptions are collected, with categories cleaned to remove spelling errors and unusual or rare labels.Source: §3.2 · p. 5
  • Scene construction uses rejection sampling, simulated resting poses, object-placement constraints, and camera-angle constraints to produce plausible layouts.The camera is placed in the upper hemisphere and its elevation is constrained to 40°–80°; the edited object is kept visible before and after manipulation.Source: §3.2 · p. 5
  • OBJECT combines environment maps, camera-adaptive three-point lighting, and directional light to illuminate generated scenes.The dataset contains 100K training examples per task and 1,024 validation and test scenes, rendered in Blender with Cycles.Source: §3.2 · p. 5

4 3DIT: a scene-aware editing model

3DIT learns to produce an edited image directly from a source image, language instruction, and task-specific values, without explicit access to scene geometry, lighting, or camera parameters. It adapts Zero-1-to-3 with edit conditioning and is trained through a curriculum that culminates in OBJECT fine-tuning.

4 3DIT: a scene-aware editing model

  • 3DIT aims to generate the edited image from the source image, edit text, and task values without explicit access to the 3D scene, lighting, or camera parameters.Those scene properties must be inferred implicitly from the single source image.Source: §4 · p. 6
  • 3DIT builds on Zero-1-to-3 and changes its conditioning module to combine source-image and task-value representations with edit-text tokens.The source image and task values are encoded together through an MLP, then appended with text tokens from CLIP’s text encoder.Source: §4 · p. 6
  • The model is fine-tuned using the standard diffusion loss to estimate a denoising score from a noised target-image latent, timestep, and conditioning information.Fine-tuning starts from the 16,500-step Zero-1-to-3 checkpoint.Source: §4 · p. 6
  • 3DIT’s three-stage curriculum combines web-scale image-text pretraining, Zero-1-to-3 viewpoint-pair pretraining, and OBJECT training for physically plausible object edits.The stages are described as developing rich text-conditioned imagery, 3D shape and camera-projection understanding, and object manipulation in scenes, respectively.Source: §4 · p. 6

5 Experiments

Experiments compare single-task and multi-task 3DIT models with task-specific baselines, using quantitative metrics, human preferences, and qualitative real-world transfer tests. Across the four tasks, 3DIT outperforms baselines on the reported image metrics and is preferred by people, while synthetic-only training yields encouraging qualitative real-world results.

5 Experiments

  • The experiments evaluate four single-task models on novel scenes with seen and unseen training objects, alongside a multi-task model trained on all four tasks.The tasks are object rotation, translation, insertion, and removal.Source: §5 · p. 6

5.1 Baselines

  • Task-specific baselines combine foundation-model operations such as localization, segmentation, compositing, and inpainting, with Zero-1-to-3 used for rotation.The removal baseline uses SAM masks and Stable Diffusion inpainting; insertion uses a fixed-size mask and a scene-generation prompt.Source: §5.1 · pp. 7–8

5.2 Quantitative evaluation

  • 3DIT outperforms baselines on PSNR, SSIM, and LPIP across all four tasks; Table 1 reports the quantitative evaluation.The multi-task model also performs well against the single-task variant despite sharing the same number of learnable parameters across four tasks.Source: §5.2 · p. 8, Table 1 · p. 8
  • Baseline FID scores tend to be higher, while 3DIT’s diffusion rendering is more prone to poorer fidelity despite handling challenging 3D scene changes.The authors attribute baseline FID results to cut-and-paste editing that can retain image fidelity even when object scale is incorrect; they note that automatic metrics miss some editing nuances.Source: §5.2 · p. 8, Figure 4 · p. 9

5.3 Human evaluation studies

  • Human evaluators prefer single-task 3DIT at 70% or more across all four tasks for geometric and lighting consistency; Table 2 reports these preferences.Evaluations compare two images and allow a tie; each pairwise evaluation uses 30 test samples.Source: §5.3 · p. 9, Table 2 · p. 10

5.4 Real-world transfer

  • Qualitative tests find that synthetic-trained 3DIT follows varied prompts on real-world images and produces reasonably photorealistic outputs with appropriate shadows.The tests cover all four editing tasks, but the authors describe the transfer results as preliminary.Source: §5.4 · p. 10, Figure 5 · p. 10

6 Limitations and Broader Impact

The work explores synthetic data for physically plausible, scene-aware image editing, while noting possible misuse of generative models for misinformation.

6 Limitations and Broader Impact

  • Training scenes have limited realism and complexity; the authors suggest more photorealistic simulators or handcrafted real-world edits may improve real-world transfer.The authors also note that generative models could potentially be misused to propagate misinformation.Source: §6 · p. 10

7 Conclusion

The conclusion presents 3DIT as a language-instructed object editor trained on OBJECT, a 400K-example procedurally generated 3D-scene dataset, with promising generalization beyond OBJECT.

7 Conclusion

  • 3DIT edits individual objects from language instructions, is trained on OBJECT’s 400K procedurally generated 3D scenes, and shows promising generalization to CLEVR and real-world images.Source: §7 · p. 11

8 Appendix

The appendix documents training and inference settings, tests robustness across scene complexity and rotation severity, presents multitask examples, and analyzes OBJECT’s composition.

8.1 Training and Inference Details

  • Training uses 20,000 steps with a frozen CLIP text encoder, 256 × 256 images, and DDIM sampling for inference.The effective batch size is 1024, and the selected checkpoint is chosen using metrics on an unseen-object validation set.Source: §8.1 · p. 15

8.2 Robustness to Severity of Transformation

  • Mask PSNR drops slightly as scenes grow from one to four objects, while the rotation model performs better for smaller angle deviations.Figures 6 and 7 show performance by object count and rotation-angle range, respectively.Source: §8.2 · p. 15, Figure 6 · p. 16, Figure 7 · p. 16

8.3 Additional Results

  • Figure 8 presents qualitative results from the multitask model across all four editing tasks.The passage identifies the examples as qualitative results, without reporting a comparative outcome.Source: §8.3 · p. 15, Figure 8 · p. 17

8.4 Dataset Analysis

  • Dataset analysis summarizes OBJECT’s object and category counts, per-category statistics, category distribution, and category-name frequencies.These are presented in Table 3 and Figures 9 and 10; the supplied passages do not give the table’s specific values.Source: §8.4 · p. 15, Table 3 · p. 16, Figure 9 · p. 18
Loading 2307.11073v1…