PaperlayerView exact revision on arXiv

Source-linked AI summary

SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field

Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, Zhaopeng Cui

arXiv:2303.13277v2cs.CV

TL;DR

Semantic 3D editing remains limited by reliance on modeling skills or narrow category coverage. SINE edits a NeRF from one image with a prior-guided editing field, and experiments demonstrate editing quality across real-world and synthetic scenes.

  • Problem

    Semantic-driven 3D editing often requires 3D modeling skills or is limited to a few categories.

    Source: §1 · p. 1

  • Method

    SINE edits a pretrained NeRF from one edited image using a prior-guided field for fine-grained geometry and texture changes, with proxy-mesh cycles, color compositing, and feature-cluster regularization.

    Source: §1 · p. 2

  • Results

    Experiments and editing examples across real-world and synthetic, object-centric and unbounded 360° scenes demonstrate superior editing capabilities and quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SINE produces high-fidelity, multi-view-consistent novel views from a photo-realistic template NeRF edited with one user-edited image.

    Source: §5 · p. 8

    SINE does not support edits that change topology.

    Source: §5 · p. 8

Abstract

from arXiv · show

Despite the great success in 2D editing using user-friendly tools, such as Photoshop, semantic strokes, or even text prompts, similar capabilities in 3D areas are still limited, either relying on 3D modeling skills or allowing editing within only a few categories. In this paper, we present a novel semantic-driven NeRF editing approach, which enables users to edit a neural radiance field with a single image, and faithfully delivers edited novel views with high fidelity and multi-view consistency. To achieve this goal, we propose a prior-guided editing field to encode fine-grained geometric and texture editing in 3D space, and develop a series of techniques to aid the editing process, including cyclic constraints with a proxy mesh to facilitate geometric supervision, a color compositing mechanism to stabilize semantic-driven texture editing, and a feature-cluster-based regularization to preserve the irrelevant content unchanged. Extensive experiments and editing examples on both real-world and synthetic data demonstrate that our method achieves photo-realistic 3D editing using only a single edited image, pushing the bound of semantic-driven editing in 3D real-world scenes. Our project webpage: https://zju3dv.github.io/sine/.

1. Introduction

SINE addresses the limited scope and cumbersome workflows of 3D semantic editing with single-image NeRF editing. Its prior-guided editing field and supporting techniques target fine-grained geometry and texture changes while preserving consistent, high-fidelity novel views.

1. Introduction

  • Existing 3D editing can require laborious annotations, apply global styles without part-level semantics, or struggle with photo-realistic textures and out-of-distribution objects.SINE aims to make editing convenient from one 2D image and applicable to real-world scenes with vivid appearances.Source: §1 · p. 1
  • SINE edits a pre-trained NeRF from a single image through a prior-guided field that encodes fine-grained geometric and texture changes for consistent, high-fidelity novel views.Users can modify a rendered image with off-the-shelf tools or provide an image for texture transfer.Source: §1 · pp. 1–2
  • Cyclic proxy-mesh constraints support geometric editing, color compositing enhances texture editing, and feature-cluster regularization controls the affected area while preserving irrelevant content.These techniques address geometric supervision, semantic texture editing, and drift in regions outside the intended edit.Source: §1 · p. 2
  • Experiments and editing examples on real-world and synthetic data, including object-centric and unbounded 360° scenes, demonstrate editing quality and effortless operation.The introduction reports these results without supplying quantitative comparisons.Source: §1 · p. 2

2. Related Works

Prior neural rendering and editing methods use external priors, annotations, constrained edit types, or category-specific latent spaces. SINE instead learns fine-grained edits directly on a given photo-realistic NeRF from one image.

2. Related Works

  • Neural rendering research spans novel-view rendering, surface reconstruction, scene editing, generation, inverse rendering, and SLAM, with external priors also used for few-shot learning and 3D inpainting.The cited prior-guided approaches can produce visually plausible results but may lack accurate geometry under insufficient 3D supervision.Source: §2 · p. 2
  • 3D scene editing has relied on laborious annotations, limited deformation or translation, or global style transfer without strong semantic meaning.Semantic editing with category latent spaces depends on datasets and may not generalize to rich appearances or out-of-distribution features.Source: §2 · p. 2
  • SINE edits a given photo-realistic NeRF directly, learning fine-grained changes from a single image rather than relying on a category-level latent space.Source: §2 · p. 2

3. Method

SINE represents geometry and texture edits with a prior-guided field over a template NeRF, then renders and blends the two edited components. Shape and semantic texture priors guide editing, while feature-based masks regularize changes outside the intended region.

3.1. SINE Rendering Pipeline

  • The editing field deforms queries into template space and separately encodes modification colors; the two rendered images are composited into the edited view.The geometry field maps edited-space queries to template-space points, while the texture field produces modification colors.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Prior-Guided Geometric Editing

  • A deformed proxy mesh represents edited NeRF geometry for shape-prior supervision, while cycle constraints connect the forward and inverse editing fields.For objects without a category-level prior, a proxy mesh can be deformed using 2D correspondence and monocular depth, then supervised with a Chamfer loss; the proxy may omit fine-grained identity details.Source: §3.2 · pp. 3–4
  • Geometric editing combines prior losses with edited-image color and silhouette supervision, plus deformation regularization for spatial smoothness.The color and opacity targets come from the user's edited image; the geometric prior can preserve semantic shape properties such as symmetry or physical conformity.Source: §3.2 · p. 4

3.3. Prior-Guided Texture Editing

  • A pretrained ViT supplies multi-view semantic texture supervision, while decoupled rendering and a 2D CNN compositing layer blend modification colors into the deformed template image.The texture loss matches edited-image cues and structural similarity; the compositing layer is used instead of directly adding modification colors to template radiance.Source: §3.3 · pp. 4–5

3.4. Editing Regularization

  • DINO-ViT feature clusters derived from the user's editing silhouette define semantic masks that regularize geometry and texture changes outside the edited region.The regularization constrains background query deformation and rendered colors to remain close to the template.Source: §3.4 · p. 5

4. Experiments

Experiments span real and synthetic objects and scenes, comparing SINE with manual editing and prior NeRF methods. The results show fine-grained, multi-view-consistent edits, while ablations highlight the roles of geometric priors, color compositing, and regularization.

4.1. Datasets

  • SINE is evaluated on real-world cars, synthetic chairs and objects, and unbounded 360° scenes.The datasets also include birds and airplanes.Source: §4.1 · p. 5

4.2. Semantic-driven vs. Manual Editing

  • SINE enables geometric editing from a single-view image without the modeling skills or multi-view pose setup required by manual approaches.Single-view photometric fine-tuning can leave unseen regions unpainted, whereas SINE uses semantic priors for multi-view-consistent edits.Source: §4.2 · p. 5, Figure 4 · p. 5

4.3. Semantic-driven Geometric Editing

  • SINE produces high-fidelity geometric edits, while EG3D and EditNeRF show limitations in fine detail or single-view consistency.The comparisons cover real-world cars and PhotoShape chairs; SINE uses geometric priors for reliable editing.Source: §4.3 · p. 6, Figure 5 · p. 6

4.4. Semantic-driven Texture Editing

  • 43 cases with 30 users favored SINE over ARF and CLIP-NeRF in texture-editing user studies.SINE also produces finer details and texture edits with semantic meaning matching the target images on cars and 360° scenes.Source: Figure 7 · p. 7, §4.4 · p. 8

4.5. Ablation Studies

  • Geometric priors mitigate single-image geometric ambiguity, color compositing supports vivid local patterns and global appearance, and editing regularization preserves irrelevant content.Without geometric priors, novel views can show distorted shapes; without regularization, unrelated object parts or background may change.Source: §4.5 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

5. Conclusion

SINE edits a photo-realistic template NeRF from one user-edited image and produces high-fidelity, multi-view-consistent novel views. It does not support topology changes and assumes edits are semantically meaningful.

5. Conclusion

  • SINE edits a photo-realistic template NeRF from a single user-edited image and delivers high-fidelity, multi-view-consistent novel views.The approach is semantic-driven.Source: §5 · p. 8
  • SINE does not support editing that changes topology, which the authors identify as future work.Source: §5 · p. 8
  • The method assumes semantically meaningful user edits and cannot use target images with meaningless random paintings.Source: §5 · p. 8

Supplementary Material

The supplementary material covers model architecture, dataset preparation, implementation details, and additional experiments.

Supplementary Material

A. Model Architecture

The architecture decouples the template NeRF from geometric and texture editing fields, then uses dual volume rendering and color compositing to form edited views.

A. Model Architecture

  • The template NeRF and editing field are built separately, using a 16-level voxel-hashing encoder with two-dimensional features per level.The template NeRF also encodes ray directions with degree-4 spherical harmonics.Source: A. Model Architecture · p. 13
  • The editing field comprises geometric and texture modification fields, with voxel hashing used for the texture field and positional encoding applied to geometric inputs.Both geometric fields use one-hidden-layer, 128-unit ReLU MLPs; the texture field uses a matching voxel-hashing encoder followed by a one-hidden-layer, 128-unit ReLU MLP.Source: A. Model Architecture · p. 13
  • Dual volume rendering produces deformed-template and color-modification images with shared density values, then a compact UNet-like compositing layer blends them into the edited view.Rendering uses 64 coarse and 128 fine samples per ray.Source: A. Model Architecture · p. 13
  • A temporal attribute from 0 to 1 can condition geometric editing on edited transitions, and examples combine geometric and texture editing in hybrid objects.The hybrid-editing examples are identified in Figure J.Source: A. Model Architecture · p. 13

B. Dataset Preparation

The evaluation uses real-world and synthetic data spanning object and scene datasets, with dataset-specific preparation and target-image editing workflows.

B. Dataset Preparation

  • Real-world car sequences contain 72 turntable images, with Colmap used to recover camera poses relative to each car’s center.Source: B. Dataset Preparation · p. 13
  • For Photoshape, the authors regenerate testing cases in Blender because EditNeRF provides no edited ground-truth images, using more challenging edits such as stretching chairs or enlarging holes.Source: B. Dataset Preparation · p. 13
  • Blenderswap scenes are rendered with Blender’s Cycle engine and realistic environment HDR maps.Source: B. Dataset Preparation · p. 13
  • Target edits use After Effects or Photoshop for deformation and painting, EditGAN or Text2LIVE for semantic strokes or text prompts, and background-removed internet images for texture transfer.Internet targets such as cars and chairs have their backgrounds removed with remove.bg.Source: B. Dataset Preparation · p. 13

C. Implementation Details

Implementation uses a template NeRF followed by geometric and texture editing from a single perspective, with auxiliary mechanisms for proxy-mesh supervision and feature-based masking.

C. Implementation Details

  • Figure L compares our method with EG3D on synthetic cars using PSNR, SSIM, and LPIPS, while Figure M compares with EditGAN on real-world cars.The supplied passages identify the comparisons and metrics but do not report their values.Source: Figure L · p. 14, Figure M · p. 15
  • Texture editing uses deferred color compositing and patch-wise back-propagation, while coarse-to-fine features and pose perturbations support training.The compositing layer blends the texture modification into the template image; training the template NeRF and editing field takes about 12 hours on one Nvidia RTX 3090.Source: C. Implementation Details · pp. 13–14
  • The initial proxy mesh is obtained with NeuS and bound to a latent code before editing optimizes the code and deforms the mesh.The initial latent code is obtained through auto-decoding.Source: C. Implementation Details · p. 14
  • Feature-cluster masks are formed from 1,000 painted-region samples clustered into 15 groups, with normalized feature distances below 0.5 marking foreground pixels.The feature field uses DINO-ViT feature maps.Source: C. Implementation Details · p. 14
  • Our texture transferring results outperform SPLICE-ViT when source and target views differ in perspective or the objects differ substantially in shape.The comparisons cover real-world cars and toys.Source: C. Implementation Details · p. 15
  • The user study presents 17 cases—eight image-target editing cases and nine text-prompt cases—for participants to select the result best matching the target.Participants see a source image, target image or text prompt, and results from different methods.Source: C. Implementation Details · p. 15

D. More Experiments

The additional experiments examine combined geometry-and-texture edits, comparisons with alternative editing methods, and robustness to proxy-mesh noise. They also identify a boundary: topology-changing deformation is unsupported.

D. More Experiments

  • Our method combines geometry editing with target-image texture transfer, including examples of a plush toy raising its hands and an airplane extending its wings.The examples also apply new textures to both objects.Source: D. More Experiments · p. 15
  • Compared with EG3D on synthetic cars, our method achieves better visual quality and all reported metrics: PSNR, SSIM, and LPIPS.The comparison notes preserved specular effects, while EG3D struggles to produce photorealistic results.Source: D. More Experiments · p. 15
  • Against EditGAN on real-world cars, our method produces high-quality edits with multi-view consistency, whereas EditGAN’s results are inconsistent across views.EditGAN’s geometry-agnostic edits cannot be precisely applied across views.Source: D. More Experiments · p. 16
  • Our method robustly learns geometric edits with noisy proxy meshes, including noise drawn from N(0, 0.0042).Figure R examines proxy meshes jittered across noise variances from 0.0012 to 0.012.Source: D. More Experiments · pp. 16–17
  • Against ARF, CLIP-NeRF, and DFF, our method produces more realistic texture edits; the alternatives struggle with fine-grained placement or vivid effects.The comparison covers real-world cars and a 360° scene dataset.Source: D. More Experiments · p. 17
  • The method does not support topology-changing deformation, such as breaking a plate, though it can render the broken part white as a visually plausible result.The paper proposes integrating more flexible representations in future work.Source: D. More Experiments · p. 18

Related papers

Loading 2303.13277v2…