PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-Fixup: Advancing Photo Editing with 3D Priors

Yen-Chi Cheng, Krishna Kumar Singh, Jae Shin Yoon, Alex Schwing, Liangyan Gui, Matheus Gadelha, Paul Guerrero, Nanxuan Zhao

arXiv:2505.10566v1cs.CV

TL;DR

Most generative editing methods focus on appearance or 2D image patches, leaving out-of-plane rotations and 3D translations unsupported despite their relevance to image editing. 3D-Fixup is a feed-forward model trained on real-world video transformations combined with image-to-3D priors for 3D-aware editing. 3D-Fixup handles large out-of-plane rotations and translations, and quantitative LPIPS and FID evaluations report that it outperforms state-of-the-art baselines such as Magic Fixup.

  • Problem

    Most generative editing methods focus on appearance or 2D image patches, leaving out-of-plane rotations and 3D translations unsupported despite their relevance to image editing.

    Source: §1 · p. 2

  • Method

    3D-Fixup is a feed-forward model trained on real-world video transformations combined with image-to-3D priors for 3D-aware editing.

    Source: §1 · p. 2

  • Results

    3D-Fixup handles large out-of-plane rotations and translations, and quantitative LPIPS and FID evaluations report that it outperforms state-of-the-art baselines such as Magic Fixup.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The method supports realistic edits that preserve content identity while maintaining fidelity across different perspectives.

    Source: §5 · p. 8

    Intricate details may not be preserved well, and results can be suboptimal when the image-to-3D guidance is low quality due to occlusion, incompleteness, or mask detection.

    Source: §5 · p. 8

Abstract

from arXiv · show

Despite significant advances in modeling image priors via diffusion models, 3D-aware image editing remains challenging, in part because the object is only specified via a single image. To tackle this challenge, we propose 3D-Fixup, a new framework for editing 2D images guided by learned 3D priors. The framework supports difficult editing situations such as object translation and 3D rotation. To achieve this, we leverage a training-based approach that harnesses the generative power of diffusion models. As video data naturally encodes real-world physical dynamics, we turn to video data for generating training data pairs, i.e., a source and a target frame. Rather than relying solely on a single trained model to infer transformations between source and target frames, we incorporate 3D guidance from an Image-to-3D model, which bridges this challenging task by explicitly projecting 2D information into 3D space. We design a data generation pipeline to ensure high-quality 3D guidance throughout training. Results show that by integrating these 3D priors, 3D-Fixup effectively supports complex, identity coherent 3D-aware edits, achieving high-quality results and advancing the application of diffusion models in realistic image manipulation. The code is provided at https://3dfixup.github.io/

1 Introduction

Editing an object’s 3D pose from a single image requires plausible handling of unseen scene information, while most current methods focus on appearance or 2D edits. 3D-Fixup combines real-world video data and image-to-3D priors to support controlled 3D edits of natural images.

1 Introduction

  • 3D editing matters for showing e-commerce objects from multiple angles and reconfiguring or relighting image scenes without reconstructing them fully.Out-of-plane rotations and 3D translations are examples of edits that most current approaches do not support.Source: §1 · p. 2
  • Existing methods trade speed, training-data quality, and user control: optimization is slow, while feed-forward approaches rely on limited 2D or synthetic data and ambiguous text prompts.These limitations can reduce generality, spatial understanding, and precision of edits.Source: §1 · p. 2
  • 3D-Fixup generates training pairs from real-world video transformations and combines them with image-to-3D priors, avoiding the need for explicit 3D annotations on every frame.The video dynamics and structural priors provide guidance for learning controlled edits in natural scenes while preserving object identity.Source: §1 · p. 2
  • The paper contributes a data pipeline, an efficient feed-forward model using 3D guidance, and evaluations reporting realistic edits that outperform state-of-the-art approaches.The stated goal includes precise 3D editing on natural images.Source: §1 · p. 2

2 Related Work

Prior work spans 3D-aware editing, broader generative editing, and image-to-video motion control. The paper distinguishes its approach by targeting 3D controls without inference-time optimization and by using 3D-aware priors.

2.1 3D-Aware Generative Image Editing

  • 3D-aware editing methods range from synthetic-data feed-forward models to optimization-based approaches; the latter can be slow, while prior methods may lack reasoning about unseen object parts.Object3DIT uses 3D transformation parameters but is limited by its small synthetic training set; Diffusion Handles and GeoDiffuser [Sajnani et al. 2024] use inference-time optimization.Source: §2.1 · p. 2

2.2 Generative Image Editing

  • Generative editing covers drag-based interactions and alternatives such as insertion, removal, and repositioning, but the cited methods do not provide 3D-aware priors or controls for out-of-plane rotations and 3D translations.Examples include Magic Fixup [Alzayer et al. 2024], ObjectStitch, and IMPRINT.Source: §2.2 · p. 3

2.3 Image-to-video with motion control

  • Image-to-video methods with motion control can produce frames usable as edits, but their edits are limited to plausible video motions and, to the authors’ knowledge, lack 3D control.Source: §2.3 · p. 3

3 Approach

3D-Fixup turns a user’s object edit into 3D guidance by reconstructing and transforming the object, then uses video-derived examples to train a diffusion model conditioned on that guidance and the source image. Its data pipeline estimates transformations between video frames and creates training pairs, while the model combines guided generation with source-detail features.

3 Approach

  • The workflow reconstructs the selected object from a mask, applies the requested 3D edit, and renders guidance for the edited image.The method targets out-of-plane rotations and translations that change the object’s perspective.Source: §3 · p. 3

3.1 Constructing the Dataset from Videos

  • Video pairs provide source and target frames; tracked correspondences, depth maps, and reconstructed meshes support estimating a transformation between them.Optical flow filters out videos with too little motion, and the transformation is optimized using a reprojection loss.Source: §3.1 · pp. 3–5
  • The estimated transformation is rigid, so non-rigid video motion is represented by a close rigid approximation in the guidance while the model trains on the observed motion.The authors describe this discrepancy as desirable because it encourages plausible non-rigid adjustments, such as changes in an animal’s foot placement.Source: §3.1 · p. 5
  • Training examples use either Transform Source, pasting a transformed source mesh onto the target, or Transform Target, pasting the target mesh onto the source.The guidance mask marks holes as 0.0, rendered regions as 0.5, and original background as 1.0; the model inpaints, enhances edited regions, and preserves intact content accordingly.Source: Figure 5 · p. 4, §3.1 · p. 5

3.2 3D Editing with a Diffusion Model

  • The editing model uses a diffusion generator conditioned on 3D guidance and its mask, plus a detail extractor whose source-image features enter the generator through cross-attention.The mask distinguishes missing regions to inpaint, ambiguous rendered regions to modify, and confident regions where content and identity are preserved; at inference, the edited mesh supplies the guidance.Source: §3.2 · pp. 5–6

3.3 Implementation details

  • Training starts from pretrained Stable Diffusion 1.4 weights and mixes Transform Target, Transform Source, and Magic Fixup samples at probabilities 0.35, 0.35, and 0.3.The model is trained for about two days on eight A100 GPUs; inference uses 100-step DDIM sampling.Source: §3.3 · p. 6

4 Experiments

The experiments compare 3D-Fixup with editing baselines, test large 3D rotations, and evaluate data and conditioning choices. The reported results favor 3D-Fixup on realism, detail preservation, and challenging 3D edits.

4 Experiments

  • The evaluation includes curated user edits and a test set of large 3D transformations, with training and testing based on stock video and about 50k training data points.Source: §4 · p. 6
  • The baseline comparison spans seven methods and uses LPIPS for perceptual similarity to ground truth and FID for generated-image realism.Source: §4 · p. 6

4.1 Comparison with Baselines

  • Across conditions, none of the baselines accurately follows the target 3D transform while preserving identity; 3D-Fixup generates high-quality edits for large transformations while preserving identity.Source: Figure 7 · p. 7, §4.1 · p. 7

4.3 Quantitative Comparison of 3D Editing

  • Table 1 reports that 3D-Fixup’s edits are closer to ground truth and the real distribution; it also achieves better LPIPS detail preservation and runs in ∼2s for 50 DDIM steps versus ∼877s per sample for Image-sculpting.Source: Table 1 · p. 7, §4.3 · p. 7

4.1 Comparison with Baselines

  • Magic Fixup struggles with pose changes, while 3D-Fixup produces more realistic images using 3D-transformation-based guidance.Source: §4.1 · p. 7

4.2 3D Editing with Continuous Rotations

  • 3D-Fixup handles out-of-plane rotations from minor adjustments to rotations as large as 180 degrees about the y-axis.Source: §4.2 · p. 7

4.4 Ablation Study of Data Setting and Conditioning

5 Conclusion

3D-Fixup combines a feed-forward editing method with an automatic training-data framework built from real-world videos. Experiments report realistic edits across large 3D transformations, alongside limitations in fine-detail preservation and sensitivity to low-quality guidance.

5 Conclusion

  • 3D-Fixup pairs a feed-forward method with automatic collection of suitable training data from real-world videos to address the time and expense of obtaining corresponding 3D edits.Source: §5 · p. 8
  • Experiments report handling large out-of-plane rotations, translations, and pose changes, with LPIPS and FID evaluations outperforming state-of-the-art baselines such as Magic Fixup.Source: §5 · p. 8
  • Fine details may be lost, and results can be suboptimal when the image-to-3D guidance is low quality because of occlusion, incompleteness, or mask detection.Source: §5 · p. 8
Loading 2505.10566v1…