PaperlayerView exact revision on arXiv

Source-linked AI summary

Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects

Amir Barda, Matheus Gadelha, Vladimir G. Kim, Noam Aigerman, Amit H. Bermano, Thibault Groueix

arXiv:2412.00518v1cs.CVcs.GR

TL;DR

Text prompts are easy to use for 3D editing but provide limited control over where an edit appears, while SDS-based methods are slow and can produce low-quality fills. Instant3dit uses masked multiview inpainting and reconstruction, reporting higher quality and orders-of-magnitude faster editing than previous approaches.

  • Problem

    Text prompts lack fine control over edits to existing 3D models, and SDS-based 3D inpainting is described as slow and low-quality.

    Source: §1 · pp. 1–2

  • Method

    Instant3dit inpaints masked canonical views with a multiview-consistent diffusion model, then reconstructs the edited 3D object.

    Source: §1 · p. 2

  • Results

    The authors report higher quality than previous techniques and orders-of-magnitude faster editing than prior work.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The method supports localized editing across meshes, Gaussian Splats, and radiance fields using 3D masks and multiview reconstruction.

    Source: §1 · p. 2, §6 · p. 8

    Thin masks may be ignored, and with white-background training renders the model may generate a white background instead of following the prompt; changing the random seed can usually resolve these cases.

    Source: §4 · p. 7

Abstract

from arXiv · show

We propose a generative technique to edit 3D shapes, represented as meshes, NeRFs, or Gaussian Splats, in approximately 3 seconds, without the need for running an SDS type of optimization. Our key insight is to cast 3D editing as a multiview image inpainting problem, as this representation is generic and can be mapped back to any 3D representation using the bank of available Large Reconstruction Models. We explore different fine-tuning strategies to obtain both multiview generation and inpainting capabilities within the same diffusion model. In particular, the design of the inpainting mask is an important factor of training an inpainting model, and we propose several masking strategies to mimic the types of edits a user would perform on a 3D shape. Our approach takes 3D generative editing from hours to seconds and produces higher-quality results compared to previous works.

1. Introduction

Text prompts offer easy control over 3D generation but lack precise spatial control for edits to existing objects. Instant3dit replaces slow SDS-style 3D optimization with multiview inpainting and reconstruction, using masks and training strategies designed for localized, view-consistent edits.

Motivation

  • Existing 3D inpainting methods are described as too slow and low-quality for production pipelines, with SDS-style optimization identified as the source of both problems.The authors attribute poor quality to inaccurate, fuzzy results and limited sample diversity under SDS-like optimization.Source: §1 · p. 2

Approach

  • Instant3dit inpaints canonical-view images and reconstructs the edited 3D object, avoiding direct optimization of the object with SDS.Reconstruction uses either feed-forward prediction or lightweight optimization.Source: §1 · p. 2

Technical challenge

  • The method targets multiview-consistent fills that respect the original 3D mask, so the generated views can represent a coherent, reconstructable object.The authors identify consistency across viewpoints and adherence to the 3D mask as the central technical challenge.Source: §1 · p. 2

Training data

  • A custom training strategy and novel 3D-mask dataset address occlusion and support three editing modes with different levels of granularity.The authors report that avoiding occlusion problems in multiview masks is critical for performance.Source: §1 · p. 2

Training strategy

  • The authors train from a pretrained image inpainter to add multiview consistency and compare this strategy with fine-tuning a multiview diffuser for inpainting.Their contributions also include localized editing across NeRFs, Gaussian Splats, and meshes.Source: §1 · p. 2

2. Related Work

Prior work uses text-to-image priors for 3D editing through approaches such as CLIP optimization, SDS, and iterative dataset update, while multiview generation methods seek view consistency. Instant3dit brings multiview generation and inpainting together, then reconstructs the edited views into different 3D representations.

3D generative editing

  • Prior 3D generative editing is organized by how it distills 2D priors—CLIP optimization, SDS, or iterative dataset update—or by the targeted representation.The passage lists meshes, NeRFs, and SDFs among the targeted representations.Source: §2 · p. 2
  • SDS is described as noisy and tuning-sensitive, with optimization typically taking around one hour; Gaussian Splatting reduces rendering time but remains non-interactive at about 15 minutes.The passage also notes that high classifier-free guidance can produce saturated, low-quality textures.Source: §2 · p. 3

Positioning

  • Instant3dit represents an asset with a multiview image grid and uses off-the-shelf reconstructors to produce a NeRF, Gaussian Splat, or mesh.The overview describes four canonical views and reconstruction into the chosen 3D representation.Source: §2 · p. 3, Figure 2 · p. 3

Multiview generation

  • Multiview generation methods promote cross-view information exchange or generate image grids, and Instant3dit extends grid-based generation to editing with inpainting.The authors compare fine-tuning the inpainting model with NeRFiller’s single-view inpainting baseline, which lacks multiview fine-tuning and uses about 30k IDU steps.Source: §2 · p. 3

Image inpainting

  • Image inpainting benefits from pretrained priors, but single-view models such as Stable Diffusion do not ensure 3D consistency across frames.The authors propose teaching multiview consistency to a single-view inpainter and use three masking strategies to resemble user-drawn 3D edit masks.Source: §2 · p. 3

3. Method

Instant3dit represents a masked 3D shape through multiple rendered views, inpaints those views with a diffusion model, and reconstructs the edited shape. Its training combines multiview generation with inpainting and uses 3D-consistent masks designed to reflect different kinds of edits.

3. Method

  • Ic(S, M) and Ib(M, S) arrange visible shape pixels and the corresponding binary mask across four canonical views in a 2 × 2 grid.Color and binary renderings use the same view configuration, keeping the mask aligned with the shape across views.Source: §3 · p. 4
  • The editing pipeline inpaints multiview renderings of a masked shape, then reconstructs the edited 3D shape with a selectable reconstruction method.Learning-based reconstruction can produce NeRFs, meshes, or Gaussian Splats; lightweight optimization can also preserve asset attributes such as color, connectivity, and UVs.Source: §3 · p. 4
  • One diffusion inference takes about 3 seconds on an A100, while the reconstruction methods range from milliseconds to a few seconds.The reconstruction choice trades speed against options such as geometric regularization and preservation of original asset attributes.Source: §3 · p. 4

3.1. Background on Diffusion Models.

  • Fine-tuning a pretrained text-to-image model on 2 × 2 object renderings supplies multiview generation, using 5K curated Objaverse objects and generated captions.The paper notes that training from scratch would perform poorly given the scarcity of high-quality 3D data.Source: §3.1 · p. 5
  • Inpainting conditions combine the prompt, masked multiview image, and binary mask; training sometimes drops the mask to retain multiview diffusion training.For latent models, the encoded image, downsampled mask, and noisy latents form a nine-channel denoiser input.Source: §3.1 · p. 5

3.2. Multi-View Inpainting Masks

  • The training set pairs clean and masked multiview images, masks, and shape captions; about 30 masks per shape across 5K shapes yield roughly 150K examples.Masks are rendered from 3D shapes to preserve view consistency and account for occlusion.Source: §3.2 · p. 5
  • The mask distribution is designed to resemble test-time edits, and the authors report that 3D-aware masks and multiview-consistent images are crucial for better performance.The paper contrasts this design with naively using random 2D masks.Source: §3.2 · p. 5
  • Three mask strategies target coarse edits, precise mesh sculpting, and local surface-texture changes, matching different kinds of user edits.Coarse masks envelop a selected portion of the shape; sculpting masks fit the selected mesh region more tightly and require more expertise; surface masks use cylinders around a sampled vertex.Source: §3.2 · p. 5

4. Experiments

Experiments compare multiview inpainting with diffusion and reconstruction baselines, test training-mask and backbone choices, and assess user-mask and camera-angle generalization. The results favor the proposed method and multi-mask training across the reported comparisons, with thin- and large-mask failures remaining.

Comparison to Baselines

  • The method outperforms Instant3D coupled with blended diffusion on prompt adherence, visual quality, and other reported metrics, while SDXL baselines perform poorly on multiview consistency.Table 1 covers prompt adherence, multiview consistency, and visual quality; Figure 4 shows example baseline comparisons.Source: Table 1 · p. 6, §4 · p. 6, Figure 4 · p. 6

Ablating Diffusion Backbones

  • SDXL-inpainting fine-tuning generally improves prompt adherence and visual fidelity, while Instant3D gives the best multiview consistency among the evaluated backbones.The authors suggest inpainting is harder to learn than multiview consistency, making an inpainting-trained starting model useful.Source: §4 · p. 6

Mask Ablation

  • Multi-mask fine-tuning performs best on arbitrary test masks, while training on one mask type favors that type and random rectangular masks perform substantially worse.Table 2 compares training-mask choices across benchmark subsets containing different mask types.Source: §4 · p. 7, Table 2 · p. 7

Mask Generalization

  • On 15 user-generated masks, multi-mask training yields the highest prompt similarity; FID is not meaningful for this small sample.Qualitative results also show the highest inpainting quality for multi-mask training.Source: §4 · p. 7

Generalization to Novel Camera Angles

  • FID, ClipL, and ClipG remain consistent across azimuth offsets, and the fine-tuned model generalizes to unseen camera orientations and FOV angles.The camera-offset evaluation samples 16 azimuth offsets and generates 500 images per offset.Source: §4 · p. 7

Limitations

  • Thin masks may be ignored, while white-background training can lead to off-prompt results when masks provide little inductive bias; changing the random seed usually resolves these cases.The benchmark holds out 500 multiview images and masks, uses BLIP captions, and evaluates prompt adherence, consistency, and visual quality.Source: §4 · pp. 6–7

5. Applications

Instant3dit applies multiview inpainting to localized edits across NeRFs, Gaussian Splats, and meshes, with reconstruction steps adapted to each representation. In the NeRF comparison, users preferred its results to NeRFiller’s in most paired cases.

5. Applications

  • The multiview inpainting method can be paired with a large reconstruction model to infer a full 3D shape from completed views and support different asset representations.The applications include an interactive tool for loading shapes, creating masks with basic primitives, and visualizing results.Source: §5 · p. 7
  • For NeRF editing, Instant3dit generates edited views and passes them to an LRM; Vox-E and MVEdit do not accept user masks, while Progressive3d and NeRFiller struggle with multiview consistency.The cited comparison reports Instant3dit is significantly faster than existing alternatives, with results in seconds rather than minutes.Source: §5 · pp. 7–8
  • Users preferred Instant3dit in 180 of 208 comparisons (86%), versus NeRFiller in 28 (14%).Figure 6 compares front-facing edited assets and reports average timing for the methods.Source: Figure 6 · p. 8, §5 · p. 8
  • Gaussian Splat editing uses the multiview inpainting output with an appropriate large reconstruction model [56].Source: §5 · p. 8
  • For mesh editing, MeshLRM guidance and adaptive remeshing preserve original attributes and triangulation in unedited regions [4].The ROAR [3] geometry optimization used for locality adds 20 seconds per edit.Source: §5 · p. 8, Figure 7 · p. 8
  • Texture edits reconstruct a 3D representation with NeRF-LRM and back-project colors to the selected mesh region, including to correct left-right inconsistencies from prior texturing [37].Source: §5 · p. 8

6. Conclusion

Instant3dit is a multiview inpainting diffusion model for fast, localized editing of 3D assets. The authors report higher quality and orders-of-magnitude faster performance than contemporary 3D editing pipelines, while identifying possible directions for further improvement.

6. Conclusion

  • Instant3dit reports superior quality to contemporary 3D editing pipelines and orders-of-magnitude faster performance, running as fast as a single diffusion image generation.Source: §6 · pp. 8–9
  • The authors propose three inpainting-mask types corresponding to user edits, and plan to release the dataset.They assess the training strategy and masks using prompt adherence, 3D consistency, and generation quality.Source: §6 · p. 8
  • The authors identify one-/few-step diffusion distillation as a route to fraction-of-a-second editing, while video diffusion may improve quality and 3D consistency but is currently much slower than text-to-image models.Source: §6 · p. 9

Related papers

Loading 2412.00518v1…