PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct 3D-Aware Object Insertion via Decomposed Visual Proxies

Jingbo Gong, Yikai Wang, Yushi Lan, Yuhao Wan, Ziheng Ouyang, Rui Zhao, Ming-Ming Cheng, Qibin Hou, Chen Change Loy

arXiv:2606.06601v1cs.CVcs.AIcs.LG

TL;DR

Existing object-insertion methods lack explicit 3D-pose control despite strong 2D visual quality. DIRECT combines a rendered 3D proxy with separate geometry, appearance, and context guidance, outperforming baselines in reconstruction quality, identity preservation, and complex pose handling.

  • Problem

    Existing object-insertion methods lack explicit 3D-pose control, limiting precise spatial alignment in practical scenarios.

  • Method

    DIRECT renders a reference-derived 3D proxy at a specified 6-DoF pose and separately injects geometry, appearance, and context guidance.

  • Results

    DIRECT consistently outperforms baselines in reconstruction quality and identity preservation while accurately handling complex pose transformations.

  • Takeaways & Limitations

    DIRECT reconciles explicit 3D spatial control with high-fidelity 2D object insertion across different generative backbones.

  • Takeaways & Limitations

    DIRECT depends on a reasonably accurate upstream 3D proxy because severe geometry-reconstruction errors can propagate into the final output.

Abstract

from arXiv · show

Object insertion aims to seamlessly composite a reference object into a specified region of a background image. Recent diffusion-based methods achieve high visual quality but formulate insertion as a simple 2D inpainting task, providing no explicit control over the object's 3D pose and limiting their practical applicability. We propose DIRECT (Decomposed Injection for Reference Composition and Target-integration), a novel framework that integrates interactive pose manipulation with high-fidelity 2D image synthesis to enable pose-controllable object insertion. Our method decomposes the insertion conditions into three complementary components: appearance guidance capturing visual details from the reference object, geometry guidance derived from the user-adjusted 3D proxy, and context guidance from the target background. By injecting them through separate pathways, DIRECT avoids feature entanglement and simultaneously preserves reference appearance, follows the user-specified pose, and adapts the object to the target scene. We also introduce an automated data construction pipeline to improve the diversity and quality of training data. Experiments show that DIRECT outperforms previous methods in both geometric controllability and visual quality.

1. Introduction

DIRECT addresses the lack of explicit 3D-pose control in reference-guided object insertion by combining a rendered 3D proxy with decomposed appearance, geometry, and context guidance. It also introduces an automated data pipeline and reports stronger geometric controllability and visual quality than prior methods.

  • Motivation: Existing reference-guided methods achieve strong identity preservation and environmental harmonization but remain confined to the 2D image plane.Their synthesis is guided by 2D appearance context rather than an explicitly specified 3D pose.
  • Motivation: Text-guided and other current control mechanisms struggle with precise spatial constraints because language is spatially ambiguous.Descriptions such as “leaning against” may fail to specify exact contact geometry, causing plausible but incorrect outcomes.
  • DIRECT framework: DIRECT lifts the reference image into a coarse 3D representation and renders it under a specified 6-DoF pose to produce a dense geometric condition image.This explicit proxy-based condition converts pose requirements into a form usable for high-fidelity image synthesis.
  • DIRECT framework: DIRECT decomposes conditioning into complementary appearance, geometry, and context components, injecting them through independent pathways to avoid proxy-induced feature entanglement.The decomposition addresses texture degradation and visual artifacts in rendered 3D proxies that can otherwise introduce noise or confuse generation.
  • Data and evaluation: Over 160k paired samples are curated by combining synthesized SA-1B data with a high-quality MVImgNet subset, using VLM filtering and generative novel-view editing.The pipeline targets greater object diversity, visual quality, and natural background interactions than simplified existing 3D datasets.
  • Data and evaluation: Evaluations on the curated hybrid dataset show that DIRECT consistently outperforms baselines in reconstruction quality and identity preservation while handling artifacts and complex pose transformations robustly.The reported robustness addresses geometric distortion and texture degradation from upstream 3D priors.

2. Related Work

Related work has progressed from semantics-driven object synthesis toward identity-preserving, feature-injected, and unified-inpainting approaches. 3D-aware editing and image-to-3D methods address geometric control and efficiency, while DIRECT uses an explicit visual 3D proxy with decomposed conditioning pathways.

  • Object Insertion: Object insertion has evolved from semantics-driven synthesis to identity preservation through feature injection, copy-paste-harmonize workflows, and unified inpainting.IMPRINT and AnyDoor use feature injection; SEELE uses copy-paste-harmonize; InsertAnything uses FLUX and a diptych design.
  • 3D-Aware Editing: 3D-aware editing methods use abstract geometric controls or training-free feature manipulation, but these approaches create alignment gaps or incur high test-time optimization costs.Encoded camera parameters and bounding boxes can struggle with fine-grained geometry, while inversion-based methods have high test-time optimization costs.
  • Our Approach: DIRECT lifts a single image into an explicit visual 3D proxy and injects the proxy, reference image, and target scene context through decomposed pathways.This design provides precise pose control, high-fidelity identity preservation, and realistic scene integration without high-quality 3D assets or test-time optimization.
  • Image-to-3D Generation: Image-to-3D generation has shifted from computationally intensive per-scene optimization toward efficient feed-forward inference from a single image.DreamFusion and Magic3D optimize NeRF representations per object, whereas LRM and LGM directly regress 3D representations in seconds.

3. Method

DIRECT formulates object insertion as conditional generation under an explicit user-specified 6-DoF pose, combining reference appearance, rendered geometry, and background context through decomposed guidance pathways. It also constructs a hybrid training dataset from curated in-the-wild images and filtered multi-view data to improve pose-control training.

  • Problem Formulation: Object insertion is formulated as conditional image generation from a reference image, background image, and binary insertion mask under a user-specified 6-DoF pose.The pose constraint distinguishes DIRECT from standard subject-driven inpainting based only on semantic compatibility.
  • 3D Visual Proxy Lifting: A 2D reference object is lifted into a manipulable 3D proxy whose user-specified pose is rendered as dense RGB geometry guidance.RGB geometry guidance resolves the semantic ambiguity of depth and normal maps for symmetric objects, such as paintings.
  • Decomposed Generative Objective: DIRECT decomposes conditioning into appearance, geometry, and context guidance so outputs preserve reference identity, follow the proxy pose, and harmonize with the background scene.The geometry signal supplies spatial arrangement, the reference restores high-fidelity texture, and context preserves scene-level semantics across local and global processing.
  • Decomposed Triplet Injection: Separate injection pathways address condition interference that can make naive concatenation over-rely on the degraded geometry proxy while ignoring the reference image.Appearance and geometry use distinct positional embeddings and modality-specific adapters, while global context is encoded separately.
  • Training Data Construction: The automated data pipeline filters complete, precisely segmented objects and synthesizes novel-view references from single-view images for pose-diverse training pairs.It combines 65k SA-1B pairs with 93k filtered MVImgNet samples, yielding approximately 160k hybrid pairs.

4. Experiments

Experiments show that DIRECT consistently delivers strong fidelity, identity preservation, and pose accuracy across generative backbones and pose variations. Ablations and visual analyses attribute these gains to decomposed guidance, hybrid data, context awareness, and robustness to degraded 3D proxies, while revealing dependence on coarse proxy geometry.

  • Evaluation Setup: The evaluation benchmark contains 200 non-overlapping image pairs: 100 from MVImgNet and 100 synthesized from SA-1B with manual verification.The benchmark represents both real-world observations and automatically constructed samples.
  • Quantitative Comparison: DIRECT consistently outperforms competing approaches across Stable Diffusion and FLUX-based categories on all evaluated metrics.The authors attribute reconstruction improvements to context guidance, which models scene context for realistic integration.
  • Pose Robustness: DIRECT maintains identity preservation and pose accuracy without a clear degradation trend as pose-change magnitude increases.Visual examples further cover substantial rotations, unseen-view synthesis, and counterfactual pose changes while preserving pose consistency and appearance fidelity.
  • Ablation Studies: RGB geometry guidance distinguishes semantic orientation that surface normals miss, particularly for symmetric objects such as circular road signs.Normal maps capture physical silhouette but fail to identify semantic orientation.
  • Ablation Studies: Decomposed injection prevents feature entanglement, avoiding blurry geometry-derived textures while isolating structural guidance from identity preservation.Naive concatenation causes the model to over-rely on geometry-guidance appearance.
  • Ablation Studies: CLIP-I rises from 0.904 to 0.943 and Matching Error falls from 26.9 to 22.7 with hybrid-data training; context guidance raises PSNR by 0.18.Shape-Decomposed Mask Augmentation reduces LPIPS from 0.190 to 0.155 and Matching Error from 20.7 to 19.0.
  • Robustness and Limitations: DIRECT preserves complex visual semantics despite degraded 3D reconstruction by using proxies primarily for geometry and retrieving texture from the reference.However, severe coarse-geometry errors, such as reconstructing a rectangular plaque as square, can propagate into the final output.

5. Conclusions

DIRECT enables pose-controllable object insertion by combining decomposed geometry, appearance, and context conditioning with independent injection pathways. The framework reconciles 3D spatial control with high-fidelity 2D synthesis, while future work targets end-to-end geometry refinement to address proxy topology errors.

  • DIRECT is a framework for pose-controllable object insertion.
  • DIRECT decomposes conditioning signals into geometry, appearance, and context, then injects them through independent pathways.This visual triplet reconciles 3D spatial control with high-fidelity 2D synthesis.
  • DIRECT achieves state-of-the-art performance by reconciling 3D spatial control with high-fidelity 2D synthesis.
  • Future work will explore end-to-end geometry refinement during generation to reduce severe proxy topology errors.The goal is to further advance 3D-aware image editing.

Impact Statement · Appendix Overview

The paper presents DIRECT as a high-fidelity, controllable object-insertion framework with benefits for generative media and augmented reality, while noting misuse risks. The appendix documents prompts, data construction, implementation, analyses, baselines, and visual demonstrations.

  • Impact Statement: DIRECT targets high-fidelity, controllable object insertion for generative media and augmented-reality applications.The framework is positioned as a contribution to both fields.
  • Impact Statement: The method lowers the technical barrier to complex image composition.The stated application benefits include virtual staging, e-commerce photography, and creative design.
  • Impact Statement: Potential misuse includes creating misleading visual content through photorealistic manipulations.The paper identifies this as a risk associated with technologies that generate photorealistic edits.
  • Appendix Overview: The appendix provides prompts for competing methods, data-construction details, implementation details, additional baselines, extended analyses, and visual demonstrations.These materials support reproduction and broader evaluation of the proposed framework.
  • Appendix Overview: Appendix sections cover competing-method prompts, training-data construction, interactive inference, and explicit geometric-condition derivation.These topics are organized in Sections A–C.
  • Appendix Overview: Additional appendix sections analyze an intrinsic-guided compositing baseline, inference latency, memory overhead, and sensitivity to 3D proxy-scene misalignment.These analyses are presented in Sections D–F.
  • Appendix Overview: The appendix evaluates performance in complex environments involving occlusion, lighting, and reflections.This evaluation is identified as Section G, followed by additional visual material in Section H.

A. Prompts for Competing Methods in Fig. 1

Figure 1 compares competing-method prompts for inserting a reference book into a bookshelf and rotating it by a specified angle.

  • Nano Banana Pro: Nano Banana Pro is prompted to insert the book from the first image into the second image’s bookshelf, placing it on the second shelf from the top and leaning right.The prompt specifies the book’s target location and orientation relative to neighboring books.
  • Object3DIT: Object3DIT is prompted to rotate the book by 320◦.The competing-method prompt specifies an explicit rotation angle.

B. Training Data Construction Details · C. Interactive Inference Pipeline

The paper constructs hybrid training data from curated SA-1B and filtered MVImgNet pairs, using explicit quality controls and novel-view synthesis while retaining real-image supervision. Its interactive inference pipeline lets users manipulate a 3D proxy to determine a 6-DoF pose and automatically generate pixel-aligned conditions for high-fidelity insertion.

  • B. Training Data Construction Details: Training data combines automatically curated SA-1B pairs with filtered multi-view data from MVImgNet to improve diversity and quality.The two sources are handled with separate quality-assurance strategies.
  • B. Training Data Construction Details: SA-1B pair construction applies staged quality control to retain fully visible, unoccluded objects with precise masks.The pipeline proposes salient categories, segments instances, filters boundary-touching or tiny masks, and verifies completeness and boundary precision.
  • B. Training Data Construction Details: Verified masks enable background removal and novel-view reference synthesis, while the original real image remains the supervision target.This allows the editing model to focus on generating the object while anchoring training to real-world captures.
  • B. Training Data Construction Details: Hybrid training data significantly improves both identity preservation and pose accuracy.The benefit is empirically verified in Table 3.
  • B. Training Data Construction Details: The constructed dataset may retain category imbalance and source-specific bias, alongside residual errors from imperfect masks, synthesis artifacts, and automatic curation failures.Explicit filtering and verification minimize avoidable errors but cannot eliminate all inherited or generated defects.
  • C. Interactive Inference Pipeline: The interactive alignment interface lets users manipulate a coarse 3D proxy over the background instead of specifying transformation matrices or manually annotating masks.The finalized alignment deterministically produces the 6-DoF pose ξ and insertion region M.
  • C. Interactive Inference Pipeline: The system lifts the segmented reference object into a TRELLIS 3D Gaussian proxy, then renders pose-conditioned images, masks, and background composites for generation.Rendering yields Irender and binary alpha mask m; the composite background uses a dilated mask region with inpainting mask M, while recentering produces Igeo.

D. Additional Intrinsic-Guided Compositing Baseline

The intrinsic-guided compositing baseline reconstructs a 3D asset from the reference object and uses intrinsic maps from the asset and target scene for insertion. It follows the rendered proxy well but sacrifices image fidelity and identity preservation, whereas DIRECT targets both pose control and appearance preservation.

  • Method: The baseline reconstructs a 3D asset from the reference object with TRELLIS, then inserts it using intrinsic maps from the asset and target scene.This adapts intrinsic-guided compositing methods to the pose-controllable insertion setting.
  • Results: Intrinsic guidance yields strong adherence to the rendered proxy, reflected by a very low Matching Error.The compositing process directly leverages intrinsic maps as conditions.
  • Results: The baseline performs substantially worse in image fidelity and identity preservation because its conditions do not explicitly preserve the reference object’s appearance.DIRECT instead aims to combine pose control with strong appearance preservation and performs better overall under this task requirement.

E. Inference Latency and Memory Overhead

The section evaluates DIRECT’s inference latency and peak GPU memory in the SD-based setting, including runtime breakdowns across processing stages. Despite slower upfront 3D proxy generation, DIRECT achieves comparable end-to-end latency and peak memory to SD-based baselines because its 2D generation is faster.

  • Evaluation scope: The evaluation reports 3D generation, 2D generation, other processing steps, end-to-end latency, and peak allocated GPU memory.These measurements are compared with corresponding SD-based baselines.
  • Overall overhead: DIRECT’s end-to-end latency and peak memory usage are comparable to those of the SD-based baselines.The comparison is summarized in Table 5.
  • Runtime breakdown: DIRECT’s 3D generation stage is slower than Object3DIT, but it occurs upfront before user interaction begins.Users specify the pose and insertion region only after the 3D proxy has been generated.
  • Runtime breakdown: DIRECT’s 2D generation stage is faster than the other compared methods, contributing to competitive end-to-end latency.The faster 2D stage offsets the additional upfront 3D generation cost.

F. Sensitivity to 3D Proxy-Scene Misalignment

DIRECT remains robust to mild inaccuracies in the user-specified 3D proxy placement relative to the target scene. Representative examples show natural insertions despite slight ground elevation and imperfect alignment with a supporting surface.

  • Robustness to Misalignment: Proxy placement accuracy is important in practice because user manipulation may leave the proxy imperfectly aligned with surrounding scene geometry.The paper attributes such inaccuracies to imperfect manipulation or ambiguity in the desired placement.
  • Robustness to Misalignment: Mild proxy-scene placement errors still produce natural object insertion results, indicating robustness to small inaccuracies in user-specified proxy placement.The evaluated cases include a proxy placed slightly above the ground and another not perfectly aligned with its supporting surface.

G. Performance in Complex Environments · H. Visual Demonstrations

DIRECT produces visually plausible object insertions in complex environments involving occlusion, directional lighting, and reflections, despite not explicitly modeling physical interactions, illumination, or view-dependent materials. Additional demonstrations show robustness, generalization, explicit pose control, and preservation of reference-object identity and texture details.

  • G. Performance in Complex Environments: DIRECT produces visually plausible insertions under occlusion, directional lighting, and reflections without explicitly modeling physical interactions, illumination, or view-dependent materials.Its learned context guidance helps infer plausible object-scene interactions in challenging scenarios.
  • G. Performance in Complex Environments: For occlusion, inserting a pen into a pen holder yields a plausible depth relationship between the pen and holder structure.This example is presented in Figure 15 as representative evidence of performance in complex environments.
  • G. Performance in Complex Environments: Under strong directional illumination, DIRECT generates a car shadow that is plausible and consistent with the surrounding scene.The lighting example appears among the representative complex-environment results in Figure 15.
  • G. Performance in Complex Environments: For reflections, Figure 15 includes a boat insertion demonstrating DIRECT’s handling of reflective scene effects.The figure presents reflections alongside occlusion and lighting examples.
  • H. Visual Demonstrations: Figure 16 provides additional visual examples intended to demonstrate DIRECT’s robustness and generalization.The examples insert various objects into complex real-world backgrounds.
  • H. Visual Demonstrations: The demonstrations support explicit pose control through varying object angles and orientations while preserving reference-object identity and texture details.Figure 16 describes these results as having high visual fidelity.
Loading 2606.06601v1…