PhysInsert: Physically Plausible Object Insertion with Geometric Priors and Bidirectional Object-Scene Interactions

• Geometry-Guided Adaptation evolves a background-initialized geometry stream with image and text tokens, supervising its readouts against with-object scene geometry to guide object perspective and pose.Source: §1 · pp. 2–3
• 2D supervision jointly targets object occupancy and external effects, while image–text attention grounds spatial instructions in object regions for placement control.Source: §1 · p. 2
• InsertWorld pairs aligned counterfactual scenes with cross-view, cross-lighting object references and spatial instructions to support object adaptation and complex scene interactions.Source: §1 · pp. 2–3
• PhysInsert-Bench tests illumination and pose adaptation beyond appearance copying with dedicated cross-environment, cross-view, and in-the-wild cases.Source: §1 · pp. 2–3

• Prior methods use real counterfactual pairs, learned-editing backgrounds, or rendered pairs, while their control approaches range from masks and boxes to text guidance and 3D proxies.Source: §2 · p. 3
• Viewpoint support varies across prior methods, whereas PhysInsert combines controlled counterfactual rendering and diverse references with supervised geometry states, object–effect supervision, and spatial-text grounding.Source: §2 · p. 3

3.1 Comparison with Existing Datasets

• Unlike AnyInsertion and DIRECT, InsertWorld combines strictly aligned counterfactual pairs with multiview and multilight references for adaptation across environments and viewpoints.Source: §3.2 · p. 4, Table 1 · p. 4

3.2 Dataset Construction

• InsertWorld pairs scenes rendered with and without an object under fixed camera, lighting, and remaining settings, recomputing shadows, mirror images, and reflections.Quality filtering and object deduplication retain suitable scene–object combinations.Source: §3.2 · p. 4
• Each example includes an object-free background, with-object target, and references rendered across viewpoints and lighting, with instructions for placement and target pose.Source: §3.2 · p. 4

3.3 Dataset Overview

• The training set contains 12,951 counterfactual scene pairs and 275,310 reference images of 1,710 objects; PhysInsert-Bench has 400 cases from 103 non-overlapping scenes, including Optical and Structural cases.The cases include 180 Optical examples—154 aligned-view and 26 cross-view—and 220 Structural examples with independent reference viewpoints.Source: §3.3 · pp. 4–5

4 PhysInsert Model

• Given an object-free background, reference images, and an instruction, PhysInsert aims to generate a with-object image that preserves identity while adapting the object and environmental effects.The model overview combines geometry-guided adaptation and 2D spatial supervision.Source: §4 · p. 5, §4.1 · p. 5

4.1 Geometry-Conditioned Object Insertion

• PhysInsert combines Geometry-Guided Adaptation for scene-consistent object adaptation with 2D Object–Scene Spatial Supervision for object regions, environmental changes, and spatial-text alignment.Source: §4.1 · p. 5

4.1.1 Geometry-Guided Adaptation

• GGA initializes a dynamic geometry stream from background VGGT features, evolves it through interaction with image and text streams, and supervises its readouts toward VGGT features of the with-object scene.Target features serve as stop-gradient supervision and are not injected into the generator.Source: §4.1.1 · pp. 5–6

4.1.2 2D Object–Scene Spatial Supervision

• COSS uses aligned counterfactual pairs to supervise object occupancy and external effects, with difference-map labels serving as proxies for visible changes such as shadows and reflections.Object and scene-effect losses combine balanced binary cross-entropy and Dice losses.Source: §4.1.2 · p. 6
• SSG aligns spatial-text attention with the soft object map across supervised layers, connecting placement instructions to the inserted object’s image region.The overall 2D supervision combines COSS and SSG.Source: §4.1.2 · p. 7

4.2 Benchmark Construction

• PhysInsert-Bench uses aligned counterfactual pairs to evaluate optical and spatial–pose adaptation, including cases with independent reference viewpoints that test pose adaptation.Cases annotate bidirectional lighting, spatial relationships, and pose, with physical-effect labels and evaluation-focus tags.Source: §4.2 · p. 7

5.1 Experimental Setup

• The experiments use PhysInsert-Bench, DreamBooth, and PhysInsert-Bench-Wild, with reported metrics covering image fidelity, identity, placement, depth consistency, and physical plausibility.Text-guided results use one reference image per case; the reported table captions specify text-guided and box-guided evaluation scopes.Source: §5.1 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8

5.2 Quantitative Results

• PhysInsert leads reported placement, fidelity, identity, depth-consistency, and Phy-VLM metrics in text-guided evaluation, and leads DINO-Obj. and PSNR while minimizing LPIPS and FID in box-guided evaluation.InsertWorld fine-tuning also improves all reported metrics for Insert-Anything and OmniPaint under the same training budget.Source: Table 2 · p. 8, Table 3 · p. 8, §5.2 · p. 9
• PhysInsert achieves the highest PSNR and lowest FID across reported PhysInsert-Bench-Wild reference-view settings and DreamBooth, while its DINO-Obj. scores lead open-source methods throughout and all compared DreamBooth methods.In a 27-participant study of 25 cases, it receives the highest lighting-integration, perspective, and pose ratings and the highest overall score, 4.04.Source: §5.2 · p. 9, Table 4 · p. 9, Figure 4 · p. 10

5.3 Qualitative Results

• Qualitative examples show adaptation of object lighting, reflections, orientation, cast shadows, occlusion, perspective, and scale across benchmark and user-study cases.The examples include a can reflected in adjacent glass, orientation changes for a tray and toy, and scale adaptation on a chair.Source: §5.3 · p. 10, Figure 5 · p. 10

5.4 Ablation Studies

• The full model improves PSNR, DIoU, and LPIPS over PhysW., while removing GGA, SSG, or COSS degrades all reported metrics.Qualitative ablations show SSG correcting vase placement, GGA adapting a poster to oblique perspective, and COSS strengthening its floor reflection.Source: §5.4 · pp. 10–11, Table 5 · p. 11
• Mechanism probes show more object-focused spatial-text attention, clearer affected-region responses, geometry-stream changes that retain coarse scene structure, and text-guided control of pose and placement.The attention and effect-response comparisons use identical GT-noised inputs; the geometry probe compares features at blocks 14, 29, and 59.Source: §5.4 · p. 11, Figure 6 · p. 11

6 Conclusion

• PhysInsert combines InsertWorld counterfactual scene pairs and references across viewpoints and lighting with a VGGT geometry stream and 2D object–scene supervision for joint adaptation.PhysInsert-Bench and PhysInsert-Bench-Wild evaluate controlled adaptation and real-world generalization; experiments demonstrate improved image fidelity and placement.Source: §6 · p. 11

A Limitations

• Identity preservation remains limited when reference and target viewpoints differ substantially, and the method does not explicitly optimize for it.The authors suggest exploiting complementary information across multiple reference views as future work.Source: A Limitations · p. 15

B Geometric Foundation Models

• Geometry alone does not determine insertion-induced shadows and reflections, motivating the combination of geometric priors and counterfactual supervision.Source: B Geometric Foundation Models · p. 15

C More Detailed Quantitative Results

• PhysInsert achieves the best PSNR, LPIPS, and FID on Optical Adaptation, and the best FID, DINO-Obj., and PSNR in both Pose Adaptation reference settings.The evaluation comprises 180 Optical Adaptation cases and 220 Pose Adaptation cases; the three-reference setting covers the same 220 Pose Adaptation cases.Source: C More Detailed Quantitative Results · p. 15, Table S1 · p. 15

D Evaluation Metric Definitions

• The metric suite covers full-image fidelity, placement, identity, object-region depth consistency, and VLM-judged physical plausibility.Metrics are averaged per case unless otherwise specified; dataset-level FID is computed over each evaluation set.Source: D Evaluation Metric Definitions · pp. 15–16, Table S2 · p. 16

D.1 Full-Image Fidelity

• PSNR, SSIM, and LPIPS assess full-image fidelity, while FID compares generated and ground-truth full-image feature distributions.FID is computed once over all generated and ground-truth images in each evaluation set, not averaged across per-image scores.Source: §D.1 · pp. 16–17

D.2 Text-Guided Placement Accuracy

• Text-guided placement is evaluated on 400 cases using DIoU for box alignment and Mask IoU for visible-object overlap.The same automatic localization and segmentation evaluator is applied to all methods; it can omit parts or select background regions.Source: §D.2 · pp. 17–18

D.3 Object Identity Preservation

• DINO-Obj. measures identity preservation by comparing generated-image and ground-truth object crops using frozen DINOv2 features.The crops retain context inside expanded object boxes, and missed detections remain in the dataset average.Source: §D.3 · p. 18

D.4 Object-Region Depth Consistency

• Object-region depth consistency compares DA3-estimated generated and ground-truth depths directly within valid ground-truth object pixels.AbsRel and δ1.25 summarize the per-case differences; scores remain sensitive to depth-estimator errors and paired-image context.Source: §D.4 · pp. 18–19

D.5 VLM-Based Physical Plausibility

• Phy-VLM uses GPT-5.5 to rate physical plausibility across 400 cases, averaging three valid ratings per case before averaging cases equally.The assessment focuses on gravitational, optical, and geometric relationships relative to the ground truth.Source: §D.5 · pp. 19–20

E.1 Latent Encoding and Flow Matching

• The latent-encoding and flow-matching setup encodes the target, background, and references, then trains image and text branches with LoRA while keeping the VAE and multimodal encoder frozen.The image stream combines noisy target, background, and reference latents; the flow-matching objective uses sampled noise and timestep-dependent weighting.Source: §E.1 · p. 21

E.2 Geometry Features and Alignment

• Geometry alignment initializes a geometry stream from frozen VGGT background features and supervises its selected-layer readouts against stop-gradient VGGT features of the with-object target.The stream evolves jointly with image and text states; target features serve only as training targets and are not injected into the generator.Source: §E.2 · p. 21

E.3 Counterfactual Targets and Spatial Prediction

• Counterfactual spatial prediction uses aligned image pairs and object masks to supervise object occupancy and external-effect maps on the target-token grid.The external-effect map proxies visible changes outside the object, including cast shadows and reflections, rather than separating individual optical effects.Source: §E.3 · pp. 21–22

E.4 Spatial-Text Attention and Mask Alignment

• Spatial-text grounding aligns the image relevance distribution for spatial-instruction tokens with a normalized soft object mask, while the mask is used for training supervision rather than required at inference.The spatial-text token subset is identified from a dedicated spatial-description field, and attention is normalized over target-image positions.Source: §E.4 · p. 22

F Training and Inference Hyperparameters

• The base model trains for approximately 16,000 optimizer updates in one stage, while the box-guided model receives 4,000 additional fine-tuning steps.The +PhysW. baselines use the same training budget in each setting, including the extra box-guided adaptation.Source: F Training and Inference Hyperparameters · p. 24
• The appendix identifies the base-model training and inference settings and separately lists the auxiliary-loss weights and module hyperparameters.The additional box-guided training budget is listed separately.Source: Table S3 · p. 24, Table S4 · p. 25

G Inference Efficiency

• The pipeline has approximately 30.143B parameters, a 4.48% increase over the backbone, under the stated accounting that excludes training-only readout and spatial heads.The parameter count includes the multimodal encoder, VAE, and frozen VGGT aggregator; shared weights are counted once.Source: G Inference Efficiency · p. 26
• Inference efficiency is measured on one NVIDIA L20X GPU with matched model precision, software, inputs, prompts, and seeds, including online VGGT extraction and unfused LoRA execution.Timing brackets the complete pipeline call, including encoding, denoising, and decoding, while excluding file I/O and model loading.Source: G Inference Efficiency · p. 26, Table S5 · p. 26

H More Qualitative Results

• Selected PhysInsert-Bench cases illustrate relative advantages in placement, viewpoint adaptation, and scene integration across box-guided and text-guided settings.The figures show ten single-reference cases per guidance setting; box-guided Ours uses additional adaptation, whereas text-guided Ours uses its text-guided checkpoint.Source: H More Qualitative Results · p. 27
• The additional DreamBooth comparisons cover 20 one-reference cases, and Figure S5 shows 11 cases from the separate in-the-wild user-study set.The DreamBooth figures include several methods with distinct conditioning, including a GT-pose RGB proxy for DIRECT and text guidance for GPT-Image2 and Nano Banana Pro.Source: H More Qualitative Results · p. 30, Figure S3 · p. 30, Figure S4 · p. 31

I Dataset Construction Details

• InsertWorld construction filters rendered counterfactual pairs for image changes and quality, then deduplicates retained objects using DINO feature similarity.Gemini reviews object completeness, background plausibility, image quality, realism, unsuitable scene structures, residual semantic parts, and rendering artifacts.Source: I Dataset Construction Details · p. 33
• Scene augmentation varies camera position while keeping each retained object–scene configuration unchanged, and object references span seven lighting conditions and 23 viewpoints per condition.This yields 161 reference images per object, associated with corresponding scene pairs.Source: I Dataset Construction Details · p. 33
• The dataset comparison defines controlled counterfactual pairs as object-presence/absence pairs captured or rerendered under otherwise fixed settings, and distinguishes multiview, multilighting, and control-input coverage.The passage describes the table’s check, cross, and partial-coverage conventions but does not provide cell values.Source: I Dataset Construction Details · p. 33

I.1 Controlled Benchmark Construction

• Each controlled benchmark case includes an object-free background, reference images, an instruction, a with-object target, and the target object region.Box guidance additionally supplies a reference image annotated with the Blender-rendered target bounding box; text guidance uses spatial instructions without an inference-time mask.Source: §I.1 · p. 33
• Physical-effect labels and evaluation-focus tags cover scene-to-object illumination, visibility, and placement requirements, plus object-to-scene shadows, reflections, mirror images, and occlusion changes.Optical cases combine substantial reference–target illumination differences with visible environmental effects.Source: §I.1 · p. 33
• Spatial & Mechanical cases and 26 Optical challenges use independent reference viewpoints to test scene-dependent orientation or pose adaptation.Source: §I.1 · p. 33

I.2 Real-World Test Set Construction

• The DreamBooth test set contains 40 cases, each evaluated with one randomly selected target and the remaining images of that object as its reference pool.Objects are removed from target images using Nano Banana Pro, and the resulting backgrounds are manually reviewed.Source: §I.2 · p. 34
• PhysInsert-Bench-Wild contains 50 cases drawn from ABO, NAVI v1.5, ORIDa, and real target images paired with references rendered from SAM 3D Objects reconstructions.Its reference pools include multiview product images, multiview photographs of the same object, and cross-scene observations.Source: §I.2 · p. 34

J Dataset and Benchmark Samples

• Five InsertWorld examples show counterfactual scene pairs and object references, while twenty PhysInsert-Bench examples cover distinct interaction tags, one per selected tag.These are illustrative samples, not model predictions or a performance-ranked selection.Source: J Dataset and Benchmark Samples · p. 35, Figure S6 · p. 36, Figure S7 · p. 37

J.1 Gemini Prompts for Dataset Curation

• Both curation stages produce an insertion caption from ordered visual inputs; classification also receives an environment-difference visualization, and the caption is distinct from the generation model’s full editing prompt.The documented templates cover six-image quality scoring and seven-image interaction classification; older variants and batch overrides may differ, and the templates specify no universal retention threshold.Source: §J.1 · p. 41

J.1.1 Stage 1: Six-Dimension Quality Scoring

• Stage 1 assigns six 0–5 scores for object validity, three-view quality, insertion-result validity, image realism, insertion difficulty, and overall quality.Difficulty rewards useful complexity but cannot compensate for severe failures in the basic quality dimensions.Source: §J.1 · p. 41, §J.1.1 · pp. 42–43

J.1.2 Stage 2: Object Properties and Scene Interactions

• Stage 2 classifies object properties and scene interactions using visible evidence from the scene pair and isolated-object views, with the difference map serving only to locate candidate environmental changes.Mechanical interaction and lighting/shadow complexity use 0/1/2 levels; five other interaction attributes are Boolean, and map-indicated changes must be verified against the original scene pair.Source: §J.1 · p. 41, §J.1.2 · p. 44
• Target visibility, integrity, and substantial unrendered gray regions trigger explicit realism score caps, while coherent dark materials, shadows, and ordinary cropping are not failures by themselves.The rubric also reserves score 5 for fully coherent images with plausible placement, contact, lighting, and no meaningful quality concerns.Source: §J.1.2 · pp. 45–46
• Annotations classify object scale and rigidity, occlusion, perspective depth, and reflective, mirror, or emissive lighting interactions using visible evidence from aligned images and object views.Reflective-surface changes must be target-caused and visibly alter a reflective receiver; mirror interactions require recognizable reflected content, while emissive interactions require broad environmental change.Source: §J.1.2 · pp. 46–48
• Mechanical interaction ranges from no force relation to simple support or mechanically constrained balance, while shadow complexity depends on distinct target-related projections or spatially varied lighting.Level 2 requires visible mechanical evidence; multiple shadow directions count even when one projection is faint or soft, but contact-only ambient occlusion does not.Source: §J.1.2 · pp. 48–49

K User Study Protocol

• The user study collected complete responses from 27 participants on 25 in-the-wild cases, comparing PhysInsert, Insert-Anything, DIRECT, and Nano Banana Pro with identities hidden and orders randomized.Participants rated lighting integration, perspective, pose, identity preservation, and background preservation from 1 to 5; the overall score equally weighted these criteria, and Figure 4 intervals used 50,000 participant-level bootstrap resamples.Source: K User Study Protocol · p. 51