Source-linked AI summary
SpatialDiff: 3D-Aware Object Movement via Implicit Spatial Modeling
Zheng Liu, Zijian He, Huiguo He, Weizhi Zhong, Yejun Tang, Huan Yang, Kun Gai, Guanbin Li
TL;DR
Precise object movement remains difficult for image-editing methods in complex scenes with depth variation and occlusion. SpatialDiff integrates implicit 3D spatial modeling with global spatial supervision in a 2D diffusion framework, and experiments report improved spatial reasoning and perceptual fidelity. The method nevertheless retains limitations in physical reasoning, distance control, and inference efficiency.
Problem
Existing image-editing methods struggle to maintain accurate spatial relationships during object movement in complex scenes involving different depth layers or occlusion.
Method
SpatialDiff uses 3D-aware features aligned to the DiT latent space and latent-level global spatial supervision to model and constrain 3D spatial structure during editing.
Results
SpatialDiff achieves superior spatial reasoning and perceptual fidelity, producing geometrically coherent object movements, including edits involving occlusion and depth inconsistencies.
Takeaways & Limitations
The method enriches diffusion-based image editing with implicit spatial awareness for precise and consistent object movement in complex scenes.
Takeaways & Limitations
SpatialDiff still exhibits incorrect perspective scaling, incomplete shadow removal, ambiguous distance control, and increased inference time from 29s to 59s with 1024 spatial tokens.
Abstract
from arXiv · showhide
Recent advances in image editing allow impressive manipulation of objects, existing methods still struggle to handle spatial movement in complex scenes, such as objects span different depth layers or are partially occluded. Most image editing methods focus solely on prior information from 2D datasets, emphasizing planar features while lacking support for spatial structures. Even approaches that incorporate explicit positional information fail to capture true 3D spatial relationships, thus limiting accurate object movement in complex scenes. In this paper, we present SpatialDiff, a method that effectively captures 3D spatial structures, enabling precise and consistent object movements in complex scenes. Our core innovations are twofold: (1) Implicit 3D Spatial Modeling, which introduces 3D prior knowledge and enables the model to internally build a comprehensive understanding of the three-dimensional spatial structure; and (2) Global Spatial Supervision, which constrains the latent spatial features to enable the model to perceive changes in object spatial positions caused by editing operations. Experimental results demonstrate that our method significantly improves the accuracy and fidelity of spatial movement in complex scenes.
1. Introduction
Instruction-driven image editing can modify appearance effectively, but precise and natural object relocation remains difficult in complex scenes. SpatialDiff addresses this gap by implicitly incorporating 3D spatial priors into a 2D diffusion framework.
- Object movement within a single image remains challenging when edits must be precise and natural.
- Purely 2D diffusion methods are flexible and efficient but lack depth and spatial-layout knowledge needed for consistent 3D motion.
- Explicit 3D-aware methods reason about geometry, viewpoint, depth, occlusion, and spatial relationships, but can falter on single-image inputs and complex scenes.
- The paper asks whether 3D spatial priors can be brought implicitly into 2D diffusion without full 3D reconstruction.
- SpatialDiff combines a 3D visual geometry encoder and Connector with Global Spatial Supervision to support flexible, precise movement in complex scenes.
2. Related Work
Related work spans instruction-driven diffusion editing, explicit 3D reasoning, and flow-based diffusion formulations. These approaches provide the conceptual and optimization background for SpatialDiff.
- Instruction-based image editing: Instruction-based editing uses natural-language commands to enable targeted image modification with improved controllability and a simplified workflow.
- 3D-aware image editing: Explicit 3D-aware diffusion methods introduce geometry through reconstruction or spatial lifting to reason about object structure, depth, and viewpoint.
- Diffusion-based editing: Diffusion models progressively denoise Gaussian-corrupted samples, supporting controllable generation for instruction-driven image editing.
- Flow-based formulation: SpatialDiff adopts a flow-based formulation that linearly interpolates between a clean image and Gaussian noise to generate intermediate samples.
- Flow-based formulation: Its training objective predicts target velocity from noisy samples conditioned on auxiliary information such as text prompts, reference images, or control signals.
4. Method
SpatialDiff integrates implicit 3D geometry into a 2D diffusion editing pipeline, then uses global spatial supervision and staged training to support consistent object movement in complex scenes.
- Implicit 3D Spatial Modeling: The 3D Visual Geometry Encoder extracts camera, point-map, depth-map, and point-track information from a single image using a 3D foundational-model backbone.These features provide spatial-aware representations for the editing process.
- Implicit 3D Spatial Modeling: SpatialDiff incorporates 3D spatial information from the input image into image editing without explicitly reconstructing depth or other 3D attributes.A 3D Visual Geometry Encoder supplies implicit geometric priors while avoiding task-specific prediction heads.
- Implicit 3D Spatial Modeling: A Connector uses learnable query tokens and cross-attention to align encoder features with the diffusion transformer’s latent feature space.The aligned spatial tokens are integrated with image-latent and instruction tokens before multimodal processing.
- Global Spatial Supervision: Global Spatial Supervision aligns original 3D spatial tokens with target-image positions during training using depth-guided supervision.The method obtains target depth with a pretrained monocular estimator and applies an MSE loss in VAE latent space.
- Global Spatial Supervision: Latent Depth Supervision is favored over explicit pixel-level depth alignment because explicit supervision can produce unnatural edits and content loss in non-edited regions.The supervision is designed to update spatial information dynamically while maintaining globally consistent geometry and suppressing residual artifacts.
- Training Strategy: SpatialDiff uses two-stage training: first optimizing the Connector, then jointly optimizing the DiT backbone, Connector, and spatial decoder with GSS.The second-stage objective combines flow matching with GSS using λ = 0.01.
5. Experiments
Experiments evaluate SpatialDiff through quantitative, qualitative, ablation, and human studies. Across these evaluations, the method achieves strong instruction following, perceptual quality, spatial consistency, and human-rated performance.
- Quantitative Comparison: SpatialDiff achieves the best overall performance across all quantitative metrics for spatial movement on SpatialBench.GPT-SC and Qwen-SC are 0.803 and 0.778; GPT-PQ and Qwen-PQ are 0.886 and 0.838; GPT-O and Qwen-O are 0.843 and 0.807.
- Qualitative Comparison: SpatialDiff preserves object appearance, scene consistency, and spatial relationships while relocating objects according to instructions.It places a table relative to a sofa, moves a cat between other animals while preserving background textures, and maintains front–back depth ordering.
- Ablations: Explicit Depth Supervision raises GPT-O from 0.620 to 0.673 and Qwen-O from 0.592 to 0.627, but can damage non-edited regions through over-constrained learning.Latent Depth Supervision instead achieves GPT-SC 0.804, GPT-PQ 0.871, and GPT-O 0.837 while preserving global scene consistency.
- User Study: A user study with 35 participants and 4200 votes finds that SpatialDiff receives higher human-evaluated H-SC, H-PQ, and H-O scores across methods.The human results are reported as generally consistent with the quantitative comparisons.
6. Conclusion
The conclusion presents SpatialDiff as an instruction-based approach for precise, consistent object movement in complex scenes. It combines implicit 3D spatial modeling with global spatial supervision to improve spatial reasoning and perceptual fidelity.
- 6. Conclusion: SpatialDiff enables precise and consistent instruction-based object movement in complex scenes.The method is designed to produce geometrically coherent results with occlusions and multi-depth structures.
- 6. Conclusion: Implicit 3D Spatial Modeling integrates 3D knowledge into the latent space, while Global Spatial Supervision provides latent-level constraints during training.Together, these designs guide consistent object movement and improve understanding of spatial information.
- 6. Conclusion: Experiments demonstrate superior spatial reasoning and perceptual fidelity for geometrically coherent editing results.The conclusion specifically includes scenes with occlusions and multi-depth structures.
Supplementary Material
The supplementary material extends the evaluation and documentation of SpatialDiff beyond the main paper. It includes additional dataset comparisons, visual results, baseline comparisons, ablations, user-study details, and limitations discussion.
- Supplementary Material: The supplement reports quantitative and qualitative results on OBJect-3DIT against various baseline methods.
- Supplementary Material: Additional visual results, commercial-baseline comparisons, and ablation studies are provided.
- Supplementary Material: The supplement details the user study and discusses SpatialDiff’s limitations and potential future directions.
1. Results on OBJect-3DIT
On the OBJect-3DIT test dataset, SpatialDiff achieves the best overall performance, although the dataset contains relatively simpler scenes than SpatialBench.
- 1. Results on OBJect-3DIT: SpatialDiff consistently achieves the best overall performance on the OBJect-3DIT test dataset.
- 1. Results on OBJect-3DIT: OBJect-3DIT contains relatively simpler scenes than SpatialBench.Most existing methods perform better in instruction following on OBJect-3DIT under GPT-SC and Qwen-SC evaluations than on SpatialBench.
2. Additional Qualitative Results
SpatialDiff handles complex object movements involving physical interactions, front–back occlusion, depth inconsistencies, and occlusions while preserving instruction following and object consistency.
- SpatialDiff moves a clock onto clothes while generating a natural indentation consistent with realistic 3D physical behavior.
- An apple is accurately placed between watermelon and banana despite substantial front–back occlusion.
- Across examples with depth inconsistencies or occlusions, SpatialDiff achieves strong instruction following and object consistency.
3. Comparison with Commercial Baseline
SpatialDiff outperforms the commercial baseline Nano Banana 2 overall, although Nano Banana 2 remains competitive on some generation-quality metrics.
- SpatialDiff outperforms Nano Banana 2 in the comparison reported in Table 5.
- Nano Banana 2 remains competitive on generation-quality metrics such as GPT-PQ.
- SpatialDiff achieves significant margins over Nano Banana 2 in spatial instruction adherence measured by Qwen-SC.
- SpatialDiff achieves significant margins over Nano Banana 2 in overall accuracy measured by Qwen-O.
4. Additional Ablation
The ablation study identifies Implicit Spatial Modeling as critical for instruction following, while LDS links depth supervision with VGGT priors to enforce spatial consistency; a user study evaluates semantic and perceptual quality.
- Replacing VGGT features with learnable queries while retaining LDS defines the Full model w/o ISM variant.
- Removing ISM causes a large drop in instruction following, validating its critical role.
- LDS acts as a linker coupling depth supervision with VGGT priors and enforcing spatial consistency.
- The user study evaluates Semantic Consistency and Perceptual Quality through separate questions for each case.
6. Limitations and Discussion
SpatialDiff remains limited by imperfect physical reasoning, ambiguous natural-language distance control, and increased inference cost from additional 3D conditioning tokens.
- Incorrect perspective scaling and incomplete shadow removal remain, reflecting limitations in physical reasoning for complex object interactions.
- Precise distance control remains challenging because natural-language instructions can be ambiguous.
- Additional conditioning tokens increase training and inference computation, reducing efficiency.
- With 1024 spatial tokens, inference time increases from 29s to 59s.
- The proposed future direction is to integrate 3D priors into model parameters and omit extra prior tokens during inference.