PaperlayerView exact revision on arXiv

Source-linked AI summary

Consolidating Attention Features for Multi-view Image Editing

Or Patashnik, Rinon Gal, Daniel Cohen-Or, Jun-Yan Zhu, Fernando De la Torre

arXiv:2402.14792v1cs.CVcs.GRcs.LG

TL;DR

Editing multiple views of one scene can yield inconsistent results, and existing 3D consolidation methods struggle with complex geometric changes. The paper trains QNeRF on self-attention queries and progressively injects rendered, consolidated queries during denoising. It reports improved multi-view consistency and fidelity, with NeRFs better aligned to target controls.

  • Problem

    Editing separate views of a shared scene can produce inconsistent results, while 3D-based consolidation methods assume edits are small and struggle with complex geometric changes.

    Source: §1 · pp. 1–2

  • Method

    QNeRF learns a 3D-consistent representation from self-attention queries and progressively renders and softly injects those queries during denoising to guide multi-view generation.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    The approach achieves greater visual quality than alternative consistency-preserving approaches and produces NeRF geometry better aligned with target controls.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method is presented as generic for diffusion-based editing techniques that modify image layouts and is demonstrated with articulation and rough bounding-box controls.

    Source: §6 · p. 8

    Fine details and hallucinated background regions may remain inconsistent, causing blurry areas when NeRFs are trained on edited views.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large-scale text-to-image models enable a wide range of image editing techniques, using text prompts or even spatial controls. However, applying these editing methods to multi-view images depicting a single scene leads to 3D-inconsistent results. In this work, we focus on spatial control-based geometric manipulations and introduce a method to consolidate the editing process across various views. We build on two insights: (1) maintaining consistent features throughout the generative process helps attain consistency in multi-view editing, and (2) the queries in self-attention layers significantly influence the image structure. Hence, we propose to improve the geometric consistency of the edited images by enforcing the consistency of the queries. To do so, we introduce QNeRF, a neural radiance field trained on the internal query features of the edited images. Once trained, QNeRF can render 3D-consistent queries, which are then softly injected back into the self-attention layers during generation, greatly improving multi-view consistency. We refine the process through a progressive, iterative method that better consolidates queries across the diffusion timesteps. We compare our method to a range of existing techniques and demonstrate that it can achieve better multi-view consistency and higher fidelity to the input scene. These advantages allow us to train NeRFs with fewer visual artifacts, that are better aligned with the target geometry.

1. Introduction

Editing separate views of one scene can produce inconsistent results, and prior 3D-based consolidation struggles with complex geometric changes. The paper consolidates self-attention queries across views during generation to improve edited-image consistency.

1. Introduction

  • Individually applying image-editing methods to views of a shared scene can produce inconsistent edits, including changes in object shape or relative position.The figure illustrates this with ControlNet and MasaCtrl edits guided by controls projected from a shared 3D model.Source: §1 · p. 1, Figure 3 · p. 2
  • Prior 3D-representation methods assume edits are small enough to average inconsistencies, an assumption that fails for complex geometric changes such as raising a person’s hands.The approach focuses on articulations and shape changes using spatial controls such as skeletons or loose depth maps.Source: §1 · pp. 1–2
  • QNeRF consolidates self-attention queries into a 3D-consistent representation and progressively renders and injects them to guide generation across views.The process is interleaved with denoising, using queries extracted from the generated images to update QNeRF.Source: §1 · p. 2, Figure 4 · p. 2
  • Self-attention queries significantly influence output structure, motivating their consolidation as a way to improve multi-view consistency.The paper builds on the idea that consistent internal features can improve consistency in generated frames.Source: §1 · p. 2
  • The approach enables articulation and shape edits with greater visual quality than alternative consistency-preserving approaches, while producing NeRF geometry better aligned with target controls.The authors validate results with qualitative evaluations, automated metrics, and user preference scores.Source: §1 · p. 2

2. Related Work

Prior work edits single images, consolidates multi-view edits through 3D representations, or edits 3D assets directly. This work instead consolidates multi-view geometric edits in attention-feature space.

Image Editing with Diffusion Models

  • Most cited diffusion-editing approaches manipulate internal representations or self-attention components for single-image editing, whereas this work uses self-attention for consistent multi-view editing.Prior work leverages queries, keys, and values to obtain different edits.Source: §2 · p. 2

Multi-view Data Synthesis and Editing

  • Earlier multi-view methods use 3D representations to consolidate edits, including iterative NeRF dataset updates, overlapping-view texturing, and 3D-consistent generated views.These approaches provide context for consolidating information across views.Source: §2 · p. 3
  • The approach consolidates 2D edits across views while changing underlying geometry, using attention features rather than pixel-space operations and thereby reducing pixel-space averaging artifacts.The paper positions this as more aligned with the real-image manifold.Source: §2 · p. 3
  • Text-guided 3D editing methods generally cannot easily handle control-based guidance and typically target appearance or small structural changes rather than large geometric changes.Other shape-editing approaches extract and deform NeRF meshes but require expert knowledge.Source: §2 · p. 3

Feature NeRFs

  • Feature-NeRF work distills semantic or generation features into 3D representations; this work distills diffusion attention features and renders them during denoising for multi-view editing.Earlier work also used rendered NeRF features for consistent multi-view image generation.Source: §2 · p. 3

3. Preliminaries

The preliminaries describe how self-attention features relate to image structure and appearance, then introduce NeRFs as renderable 3D representations that can also carry latent features.

3.1. Self-Attention in Diffusion Models

  • Self-attention queries are associated with pixel semantics and image structure, while keys and values determine appearance and can transfer it between images.MasaCtrl preserves original appearance by injecting the original image’s keys and values into the edited image.Source: §3.1 · p. 3

3.2. Neural Radiance Fields

  • A NeRF maps spatial locations and viewing directions to density and RGB color, which volume rendering uses to produce an image along a camera ray.The expected ray color is computed from density-weighted color along the ray.Source: §3.2 · p. 3
  • This work trains a NeRF on latent representations rather than RGB values and renders those latents with the same volumetric approach.The paper follows prior latent-feature NeRF work.Source: §3.2 · p. 3

4. Method

The method edits posed views using spatial controls, then consolidates their self-attention queries in a shared 3D representation. Rendered queries guide denoising through interleaved intervals, while original-image keys and values preserve appearance.

4. Method

  • The method aligns object shapes across views by training a NeRF on self-attention queries and softly injecting its consolidated features into the denoising networks.Original-image keys and values are also injected to preserve appearance.Source: §4 · p. 4

4.1. QNeRF

  • QNeRF is trained on queries from nine UNet decoder layers, with nine output heads, shared density, and no viewing-direction dependence.Its training objective combines query loss with depth loss, and each interval's QNeRF is initialized from the prior interval's model.Source: §4.1 · p. 4

4.2. Query Guidance

  • Soft guidance updates the noisy latents to bring generated queries closer to QNeRF-rendered queries, avoiding direct replacement that produced visual artifacts.A DDIM denoising step follows the latent update.Source: §4.2 · pp. 4–5

4.3. Multi-view Image Denoising Interval

  • Each overlapping interval uses τ guided steps followed by τ unguided steps, then trains a new QNeRF from the updated queries for the next interval.The unguided steps let queries evolve; the first interval instead runs unguided for 2τ steps before training its first QNeRF.Source: §4.3 · p. 5

4.4. Progressive Consolidation

  • Progressive consolidation trains each interval's QNeRF on queries already affected by the prior interval, limiting drift before reconsolidation.The zipper analogy depicts successive query trajectories being brought closer together.Source: §4.4 · pp. 5–6, Figure 7 · p. 6

5. Experiments

Across the reported fidelity and user-preference evaluations, the method outperforms the baselines; ablations and qualitative comparisons show how its consistency and scene preservation differ from alternatives.

5. Experiments

  • The method was evaluated on seven multi-view image sets of 200–500 images each, with three used for comparisons; edited-image quality was measured using KID and final 3D representation quality through a user study.Technical users ranked depth-map videos from NeRFs trained for each method by target-pose alignment and quality; the study collected 120 responses from 20 unique users.Source: §5 · p. 6, §5.2 · p. 7

5.1. Ablation Studies

  • The full method keeps legs consistently positioned and their shape stable while preserving the deck shape; independent edits, direct query injection, and non-progressive training show drawbacks.Independent edits vary leg positions and deck height, direct injection diverges from the original, and non-progressive training can add artifacts.Source: §5.1 · p. 6, Figure 8 · p. 6

5.2. Comparisons

  • Table 1 reports that our method outperforms the baselines in fidelity and user preference.Image quality improves across all scenarios, and users preferred our method in a majority of cases.Source: Table 1 · p. 7, §5.2 · pp. 7–8
  • IN2N-CN and CSD-CN produce ghostly or noisy limbs, while TokenFlow shows noisy geometry and weaker source-frame fidelity; our method avoids these artifacts.The paper attributes dataset-update failures on shape changes to partial edits being averaged in a NeRF, while TokenFlow's flow assumptions are violated by geometric edits.Source: §5.2 · p. 7, Figure 9 · p. 8

6. Conclusions, Limitations, and Future work

The paper presents progressive QNeRF consolidation for multi-view edits, demonstrated with articulations and rough bounding boxes. It also notes limitations from text-to-image generation and black-box optimization.

6. Conclusions, Limitations, and Future work

  • QNeRF progressively consolidates image attention features during editing and is demonstrated with articulations and rough bounding-box controls.The controls are intentionally lenient to make them easier to use.Source: §6 · p. 8
  • Inherited text-to-image weaknesses include difficult hand generation and hallucinated fine details that remain inconsistent across highly detailed objects despite shared shape.Hallucinations in detailed disoccluded backgrounds can cause blurry regions when training a NeRF on edited views.Source: §6 · p. 8
  • Black-box QNeRF optimization may average over outliers that robust statistics could instead filter.The authors also envision exploring other consolidation representations, including Gaussian Splats.Source: §6 · p. 8

A. Implementation Details

The implementation constructs spatial controls and preserves original appearance through self-attention injection and fine-tuning, with depth supervision and stated diffusion hyperparameters.

A. Implementation Details

  • Skeleton editing extracts 2D poses with OpenPose, triangulates them into a 3D skeleton for editing, then projects the edited skeleton into original camera views.Source: A. Implementation Details · p. 14
  • For loose-depth edits, scenes are manually modeled with Blender boxes and planes, edited, then rendered from the original camera viewpoints.The modeling takes a couple of minutes per scene and could be automated with LooseControl's data-processing pipeline.Source: A. Implementation Details · p. 14
  • Original appearance is preserved by injecting self-attention keys and values from denoising step four in decoder resolutions 32 and 64, alongside LoRA fine-tuning.The fine-tuning uses batch size 2 for 300 steps, and the same fine-tuned model is used for all baselines.Source: A. Implementation Details · p. 14
  • QNeRF depth supervision assumes query depth matches image depth, using masked depth rendered from a NeRF trained on the original views and the DS-NeRF loss.The rough mask covers the union of the object before and after editing.Source: A. Implementation Details · p. 14
  • The implementation uses Stable Diffusion v1.5 with 50 denoising steps and trains QNeRF for 10,000 steps per interval with depth-loss coefficient 1 and query-guidance α = 60.Source: A. Implementation Details · p. 14

B. Datasets Details

The section identifies the dataset sources and scenes, then reports that vanilla Instruct-NeRF2NeRF leaves the scene NeRF unchanged for the tested text prompt.

B. Datasets Details

  • The person dataset comes from Instruct-NeRF2NeRF, while the other datasets were collected using a phone camera.The listed scenes are Person, Spiderman, Statue, Alligator-toy, Sofa, Lamp, and Boot.Source: B. Datasets Details · p. 14

C.1. Comparisons

  • For “raise his hands,” vanilla Instruct-NeRF2NeRF leaves the NeRF unchanged across prompts and hyperparameters.Figure 12 shows the comparison results for that prompt.Source: §C.1 · p. 14, Figure 12 · p. 14

C.2. Dataset Size Analysis

The approach is more consistent than isolated editing even with 50 images, improves with larger input sets, and can use rendered novel views to expand a set that already supports NeRF training.

C.2. Dataset Size Analysis

  • The method requires enough images to train a NeRF, and a trained NeRF can render novel views to enlarge the input set.The method requires the same dataset size as NeRF training.Source: §C.2 · p. 14
  • With 50 images, the method achieves much more consistent results than MasaCtrl, which edits each image in isolation.Results improve as the input set grows and are highly consistent with 200 images.Source: §C.2 · p. 14
  • The tested lamp-scene input sets ranged from 50 to 305 randomly sampled images, with consistency improving as dataset size increased.The study tested set sizes of 50, 100, 150, 200, 250, and 305 images.Source: §C.2 · p. 14

D. Limitations

The reported limitations include low-quality hand anatomy and inconsistent fine details across views, particularly for the detailed alligator toy.

D. Limitations

  • The edited person’s hand lacks fingers, and the palm has low visual quality.These defects are shown in the two edited images of the person.Source: D. Limitations · p. 14
  • For the high-detail alligator toy, fine details such as the number of fingers are not consistently preserved.The original right foot has four fingers, but the edited result differs across views.Source: D. Limitations · p. 14
  • The toy’s edited foot has four fingers in view 2 but five in view 1; the authors suggest query resolution may contribute, and the deck also looks different after editing.The queries are lower resolution than the generated image.Source: D. Limitations · p. 15

E. Running Time

For a 300-image dataset, the pipeline takes roughly 8.5 hours on an RTX A5000; the section also gives runtimes for IN2N-CN and CSD-CN.

E. Running Time

  • 8.5 hours is the typical full-pipeline runtime for 300 images on an NVIDIA RTX A5000 GPU, versus roughly 5.5 hours for IN2N-CN on the same dataset and GPU.The full pipeline includes image inversion, denoising, and QNeRF training; IN2N-CN is slower than vanilla IN2N because MasaCtrl requires DDIM inversion at each dataset update.Source: E. Running Time · p. 15
  • The method’s running time scales linearly with the number of images in the multi-view set.Source: E. Running Time · p. 15
  • CSD-CN takes about three days on an NVIDIA H100 GPU for the same 300-image dataset.Source: E. Running Time · p. 15
Loading 2402.14792v1…