PaperlayerView exact revision on arXiv

Source-linked AI summary

Editing Conditional Radiance Fields

Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan Russell

arXiv:2105.06466v2cs.CVcs.GRcs.LG

TL;DR

The paper addresses user editing of category-level implicit 3D object representations from coarse 2D scribbles. It introduces a modular conditional radiance field and hybrid updates, demonstrating appearance and shape edits across datasets and on real photographs with novel-view propagation.

  • Problem

    The paper studies how to let users edit and control implicit continuous volumetric representations of 3D objects using local appearance and shape edits.

  • Method

    The method learns a category-level conditional radiance field with a shared shape branch and uses targeted hybrid network updates to propagate sparse scribbles into 3D edits.

  • Results

    The approach demonstrates view synthesis, color and shape editing, and real-photograph edits whose appearance and shape changes propagate to extrapolated novel views across three shape datasets.

  • Takeaways & Limitations

    Sparse 2D scribbles can support consistent category-level 3D appearance and shape editing across viewpoints without explicit spatial or boundary constraints.

  • Takeaways & Limitations

    Shape editing is not yet interactive, because users currently wait over a minute for feedback.

Abstract

from arXiv · show

A neural radiance field (NeRF) is a scene model supporting high-quality view synthesis, optimized per scene. In this paper, we explore enabling user editing of a category-level NeRF - also known as a conditional radiance field - trained on a shape category. Specifically, we introduce a method for propagating coarse 2D user scribbles to the 3D space, to modify the color or shape of a local region. First, we propose a conditional radiance field that incorporates new modular network components, including a shape branch that is shared across object instances. Observing multiple instances of the same category, our model learns underlying part semantics without any supervision, thereby allowing the propagation of coarse 2D user scribbles to the entire 3D region (e.g., chair seat). Next, we propose a hybrid network update strategy that targets specific network components, which balances efficiency and accuracy. During user interaction, we formulate an optimization problem that both satisfies the user's constraints and preserves the original object structure. We demonstrate our approach on various editing tasks over three shape datasets and show that it outperforms prior neural editing approaches. Finally, we edit the appearance and shape of a real photograph and show that the edit propagates to extrapolated novel views.

1. Introduction

The paper enables local editing of category-level implicit radiance fields by propagating sparse 2D scribbles into consistent 3D color and shape changes. A modular conditional radiance field and targeted network updates support these edits across object instances and views.

  • Motivation: Editing implicit volumetric representations requires propagating sparse 2D edits through 3D regions and identifying which network parameters control shape and appearance.Prior explicit-representation editing tools do not directly apply to neural representations.
  • User Edits: Users can change a local part’s color or shape, or transfer color or shape from another object instance, using 2D scribbles and target selections.Examples include recoloring a chair seat, removing a wheel, or swapping in arms from another chair.
  • Method: The conditional radiance field learns a class-level prior and a shared shape branch that propagates sparse scribbles across selected 3D regions without explicit spatial or boundary constraints.The shared branch biases the network toward representations common across the shape class.
  • Method: The method uses color and shape editing losses to satisfy user targets while preserving the original object structure.The edits are optimized to align the radiance field with the specified local target.
  • Results: Results on three shape datasets cover view synthesis, color editing, and shape editing, with comparisons against prior neural editing methods.The paper also edits a real photograph and propagates the result to extrapolated novel views.

2. Related Work

The paper situates its contribution between neural novel-view synthesis and interactive editing, targeting local edits in learned implicit continuous volumetric representations. It differs from related class-level radiance fields through an instance-agnostic branch, photometric training, and direct user editing.

  • Novel view synthesis: Novel-view synthesis methods infer scene structure and view-dependent appearance from input views using explicit, discrete volumetric, or implicit continuous representations.Implicit neural representations encode shape or shape and appearance in network weights and support complex real-world renderings.
  • Novel view synthesis: Unlike closely related generative radiance fields, this approach adds an instance-agnostic branch that biases the network toward common class features.The paper also uses photometric loss rather than adversarial loss and can align to a single view of a novel instance.
  • Interactive editing: Prior interactive editing methods operate on light fields or explicit and discrete volumetric geometry, whereas this work edits learned implicit continuous volumetric representations.The paper focuses on propagating user edits to volumetric data for rendering under different viewpoints.
  • Interactive editing: The work adapts the idea of edit propagation from sparse image edits across photo collections or videos to edits of 3D volumetric data.This extends the editing target from image collections or video to viewpoint-dependent volumetric rendering.

3. Editing a Conditional Radiance Field

The paper introduces a conditional radiance field for category-level editing, with modular shape and appearance representations that support targeted updates. Users provide coarse 2D scribbles, and optimization propagates edits to local 3D regions while balancing edit accuracy and efficiency.

  • Conditional radiance field: The conditional radiance field extends NeRF with latent shape and appearance vectors for instances within an object category.Its network maps 3D location, viewing direction, and instance-specific codes to radiance and density.
  • Network architecture: Rendered pixel colors are computed by sampling points along a ray, predicting radiance and density, and applying alpha compositing.The sampled points are ordered from near to far, and the compositing operation produces the output pixel color.
  • Network architecture: A shared shape branch and instance-specific shape components provide modular shape–appearance disentanglement for category-level editing.The shared geometric representation operates on positional encoding, while instance-specific features are fused before predicting density and radiance.
  • Modular network updates: The proposed hybrid update strategy finetunes only selected later layers, improving the accuracy–efficiency tradeoff over code-only or full-network updates.Color edits update Frad and z(c), while shape edits update Ffuse and Fdens.
  • Modular network updates: 3.7× color-edit speedup reduces optimization time from 972 to 260 seconds, while shape-edit updates reduce time by 3.2× from 1,081 to 342 seconds.Additional subsampling and feature caching further reduce computation while preserving edit quality according to the reported method description.
  • Color editing: Color editing uses coarse foreground and optional background scribbles to optimize target colors while preserving unchanged regions.The editing loss is optimized over the latent color vector and radiance network component.
  • Shape editing: Shape addition composites a copied region from another instance with the original instance before optimizing the modified and unmodified regions.The copied shape code is combined with the original color code at the paste location.

4. Experiments

Experiments evaluate conditional radiance-field reconstruction, color and shape editing, and comparisons with neural editing baselines across shape datasets. The method combines strong view synthesis with edits that propagate to unseen views while preserving unedited structure.

  • Experimental setup: The experiments cover PhotoShape chairs, Aubry chairs, and GRAF CARLA cars, spanning appearance variation, shape variation, and single-view training.The datasets are selected to test the method under varying appearance, shape, and training-view complexity.
  • Conditional radiance field training: Separating shape and color codes and using shared and instance network components improves view synthesis quality over the reported ablations and single-instance NeRF models.Table 1 evaluates these architectural choices through view reconstruction.
  • Conditional radiance field training: When trained on all 626 PhotoShape instances, the model reaches PSNR 35.79, compared with PSNR 33.91 without the shared shape branch.The shared shape branch helps the model scale to more instances.
  • Color edits: Finetuning only the color code cannot fit the desired color edit, whereas updating the entire network changes instance shape through effects on downstream density.The experiments motivate selectively updating network components.
  • Color edits: For color editing, the method propagates scribble-based edits to desired regions and unseen views, while single-instance NeRF and GAN baselines produce artifacts, inconsistent colors, or unrealistic outputs.The single-instance baseline lacks a category-specific appearance prior, while 2D GAN editing lacks a 3D representation.
  • Shape edits: For shape editing, optimizing only the shape code or Fdens leaves the chair mostly unchanged, while whole-network optimization introduces artifacts; the proposed method removes arms, fills holes, and generalizes edits to unseen views.The averaged shape-editing results are summarized in Table 3.
  • Shape/color code swapping: Changing the color code while holding the shape code fixed preserves shape, and changing the shape code while holding color fixed preserves color, enabling cross-instance shape and appearance transfer.The density output is independent of the color code, while the model learns to disentangle color from shape when predicting radiance.

5. Discussion

The method enables intuitive conditional radiance-field editing, but shape edits remain slow and reconstruction can fail for novel instances unlike the training class.

  • Limitations: Shape editing currently takes over a minute to provide user feedback, with most computation spent rendering views.The authors expect NeRF rendering improvements to reduce this delay.
  • Limitations: The method fails to reconstruct novel object instances that are very different from other class instances.
  • Contributions: The learned disentangled representation supports intuitive editing operations over conditional radiance fields.

A. Additional Experimental Details

Additional details describe dataset rendering, conditional radiance-field training, view-dependence regularization, and computationally targeted editing updates.

  • Dataset rendering: PhotoShape uses 40 Blender-rendered views per instance with clean backgrounds, while Aubry images are resized and center-cropped.CARLA uses the same dataset as GRAF.
  • Conditional radiance-field training: Training uses coarse and fine networks, with stratified and hierarchical sampling followed by photometric optimization over sampled rays.The coarse network estimates density, and the fine network renders rays at test time.
  • Editing optimization: Editing keeps the coarse network fixed and updates the fine network, network components, and codes for 100 iterations at learning rate 10^-2.Training uses Adam with learning rate 10^-4 and typically converges in around 1M iterations.
  • Architecture: The architecture caches an 8-dimensional color-branch input to make feature caching computationally feasible.
  • View dependence: View-dependence regularization makes synthesized colors consistent across views, unlike the unregularized model’s green-blue and purple-green hallucinations.The comparison is shown on CARLA models trained with one view per instance.

B. Additional Evaluations

Additional evaluations report SSIM analyses, subsampling considerations, a PSNR result, and comparisons against a naive generator fine-tuning baseline.

  • Evaluation metrics: The appendix reports SSIM evaluations for model ablation, color editing, and shape editing in Tables 4–6.
  • User constraints: Subsampling user constraints can reduce edit quality because edits may propagate into unwanted areas not represented by sampled rays.Background regions excluded from sampled rays can potentially change.
  • Color editing: 34.49 is the average PSNR over three color edits after adding the subsampling optimization.
  • GAN baseline: The naive generator fine-tuning baseline freely optimizes all generator weights for 10,000 steps at learning rate 10^-3.This is contrasted with model rewriting, which performs a low-rank update of a particular layer.

C. Additional Shape Editing Methods

Additional shape-editing procedures address occluded removal and shape addition, while quantitative tables evaluate ablations, color edits, and computationally balanced hybrid updates.

  • Shape removal: Occluded shape removal zeros the first density mode along each foreground ray, but the procedure can be slow and leave incomplete-removal artifacts.
  • Quantitative evaluations: Table 4 evaluates conditional radiance-field ablations, emphasizing separated shape and color codes plus shared and instance networks.
  • Quantitative evaluations: Table 5 reports color-editing results, while Table 6 evaluates shape editing and the hybrid update’s visual quality and computational cost.
  • Shape addition: Shape addition uses composite-image reconstruction for effective but slow edits, alongside a faster density-based loss that is less effective.
  • Density optimization: The cross-entropy loss encourages predicted densities to match copied target densities in edited regions and remain unchanged elsewhere.Foreground and background densities are normalized to sum to one.
  • User interface: The interface supports color edits, shape removal, shape addition, and color or shape transfer through rendered-view scribbles and instance selection.After the user executes an edit, latent variables and network weights are fine-tuned and renderings are updated.

E. Additional Color Edits

Additional evaluations show that the method propagates color edits across views while avoiding artifacts and unwanted edits seen with alternative approaches.

  • Color editing evaluation: The method accurately propagates target-instance colors from scribbles on one view, unlike GAN rewriting and single-instance NeRF editing.GAN-based editing fails to generalize across unseen views, while single-instance NeRF editing produces floating artifacts or non-transferring colors.
  • Quantitative evaluation: The method outperforms baselines for all color edits on all evaluation criteria.
  • Color editing evaluation: The evaluation quantifies four color edits across the first two main-paper edits and two additional edits.The additional edit changes a chair seat from brown to green and its back from beige to grey.
  • Single-instance comparison: Single-instance NeRF editing creates red background artifacts, whereas the proposed model confines the edit to the chair seat.
  • Additional results: Additional results color Aubry chair seats red and change car body colors to red and pink.

F. Additional Shape Edits

Additional shape-editing evaluations reinforce that the proposed update strategy fits local edits more effectively than partial or whole-network alternatives.

  • Quantitative shape evaluation: Finetuning only the shape code or shape branch cannot fit both edits, while finetuning the whole network causes unwanted instance changes.
  • Single-instance comparison: Single-instance NeRFs cannot learn shape edits that generalize to unseen views, likely because they lack a category-level prior.
  • Additional shape results: Additional experiments evaluate shape edits on PhotoShapes and CARLA cars.
  • Shape and color swapping: Swapping the color code preserves instance shape, while swapping the shape code preserves instance color.

H. View Reconstruction

The conditional radiance field produces view-consistent reconstructions across instances and can infer unseen or occluded scene content.

  • View consistency: Rendered views remain consistent in shape and color across fixed instances and datasets.On CARLA, the model infers occluded car regions despite training on only one image per instance.
  • Additional reconstruction: The model almost perfectly reconstructs unseen PhotoShapes viewpoints and learns convincing depth estimates.Each instance is rendered from four unseen viewpoints and compared with ground-truth views.

I. Changelog

The changelog records an updated Figure 8 and added appendix details, while supplementary figures document editing, transfer, and reconstruction results.

  • Changelog: Version 2 updated Figure 8 and added additional appendix details.
  • Shape editing: Shape-editing figures show that partial updates fail to fit both edits, whereas whole-network optimization is slow and changes the instance unnecessarily.
  • Shape editing: The method removes chair backs, arms, and holes, and removes car mirrors in qualitative shape-editing results.
  • Quantitative results: Shape-editing quantitative results report that the method outperforms baselines for all listed criteria.
  • Transfer results: Shape and color transfer experiments show that swapping one code leaves the other attribute unchanged.
  • View reconstruction: View-reconstruction figures show realistic, consistent renders and near-perfect reconstruction of ground-truth views from four unseen viewpoints.
Loading 2105.06466v2…