PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-aware Blending with Generative NeRFs

Hyunsu Kim, Gayoung Lee, Yunjey Choi, Jin-Hwa Kim, Jun-Yan Zhu

arXiv:2302.06608v3cs.CVcs.GRcs.LG

TL;DR

Image blending remains challenging when 2D methods encounter misaligned camera poses and object shapes. This paper uses generative NeRFs for 3D-aware alignment and blending in latent space. It reports better photorealism and faithfulness than existing methods, along with color-geometry disentanglement and multi-view-consistent results.

  • Problem

    2D blending does not account for object 3D structure, while misaligned poses and shapes make seamless composition difficult.

    Source: §1 · p. 1

  • Method

    The method uses generative NeRFs to align images through pose estimation and novel-view synthesis, then blends them in NeRF latent space.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    The method outperforms classic and learning-based methods in photorealism and faithfulness, and produces color-geometry disentanglement and multi-view-consistent results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that 3D-aware alignment improves existing 2D baselines and that latent-space blending exceeds those improved baselines.

    Source: §5 · p. 9

    The method depends on GAN inversion, a quality and speed bottleneck that can fail for inputs far from the mean face latent code.

    Source: §5 · p. 9

Abstract

from arXiv · show

Image blending aims to combine multiple images seamlessly. It remains challenging for existing 2D-based methods, especially when input images are misaligned due to differences in 3D camera poses and object shapes. To tackle these issues, we propose a 3D-aware blending method using generative Neural Radiance Fields (NeRF), including two key components: 3D-aware alignment and 3D-aware blending. For 3D-aware alignment, we first estimate the camera pose of the reference image with respect to generative NeRFs and then perform 3D local alignment for each part. To further leverage 3D information of the generative NeRF, we propose 3D-aware blending that directly blends images on the NeRF's latent representation space, rather than raw pixel space. Collectively, our method outperforms existing 2D baselines, as validated by extensive quantitative and qualitative evaluations with FFHQ and AFHQ-Cat.

1. Introduction

Image blending remains difficult when foreground and background differ in 3D pose or shape, because 2D alignment and blending do not account for object geometry. The paper uses generative NeRFs for 3D-aware alignment and latent-space blending, reporting stronger results than existing methods.

1. Introduction

  • 2D methods struggle with unaligned images: even slight pose mismatch can look unnatural, while affine warps miss out-of-plane rotation and 3D shape differences.Inferring 3D structure from a single view makes alignment difficult; blending in 2D also ignores object geometry.Source: §1 · p. 1
  • Generative NeRFs support alignment by projecting input images into latent space and synthesizing novel views; blending then optimizes a latent code to match the reference foreground while preserving the original background.The method uses NeRFs trained from single-view image collections.Source: §1 · pp. 1–2
  • Figure 2 contrasts unaligned 2D blending, 2D blending with the proposed 3D-aware alignment, and the final method, which adds blending in NeRF latent space.The compared methods use neither 3D labels nor 3D morphable models.Source: Figure 2 · p. 2, §1 · p. 2
  • The method outperforms classic and learning-based methods in photorealism and faithfulness, while enabling color-geometry disentanglement and multi-view-consistent results.The authors describe it as a general-purpose method for diverse unaligned images.Source: §1 · p. 2

2. Related Work

Prior blending work ranges from low-level visual cues to learned compositing, while 3D-aware approaches include domain-specific methods and generative NeRFs. This paper distinguishes its general-purpose blending task from other NeRF-based editing tasks.

2. Related Work

  • Earlier blending methods use low-level cues such as gradients, frequency bands, color and noise transfer, or segmentation, followed by data-driven composition under similar lighting, poses, and scene contexts..Source: §2 · p. 2
  • Learning-based blending mixes deep features or uses deep-feature losses; GAN inversion, spatial latent blending, and diffusion are also used, but large pose differences can still harm identity preservation or realism.The passage characterizes learning-based methods as generally more robust than pixel-based methods.Source: §2 · p. 2
  • Methods for faces or hair can blend unaligned images, but are domain-limited and often rely on 3D face morphable models or multiview images.The paper presents its approach as handling diverse objects without 3D data.Source: §2 · p. 2
  • Generative NeRFs combine implicit scene representations, volumetric rendering, and GANs; Figure 3 shows alignment using encoder-inferred camera poses, latent codes, and novel-view synthesis.Examples include EG3D's tri-planes and StyleSDF's SDF-based volume renderer.Source: §2 · p. 2, Figure 3 · p. 3
  • NeRF-based editing work addresses shape, color, geometry, attributes, or semantic decomposition, whereas this paper focuses on blending, which requires both alignment and harmonization..Source: §2 · p. 3

3. Method

The method aligns unaligned inputs using generative-NeRF camera and latent representations, then blends by optimizing a latent code with image and volume-density constraints. An optional Poisson stage further improves preservation of original-image details.

3.1. 3D-aware alignment

  • A frozen CNN estimates camera poses; PTI inversion projects both images into NeRF latent codes, novel-view rendering aligns the reference pose, and ICP adjusts target-region scale and translation in AFHQv2.The reference is rendered at the original image’s estimated camera pose; local alignment is available for regions such as faces, eyes, and ears.Source: §3.1 · p. 3

3.2. 3D-aware blending

  • The image-blending loss matches original-background color and perceptual similarity while matching reference-foreground details with LPIPS.The reference region uses LPIPS alone to avoid pixel-space overfitting; λ2 controls whether reference color is reflected.Source: §3.2 · p. 4
  • Density blending matches output NeRF density to the reference inside masked rays and to the original outside, complementing 2D image constraints.The paper identifies density blending as key because image-only optimization can produce blurry results and fail to represent structured objects correctly.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. Combining with Poisson blending

  • Poisson post-blending improves preservation of original-image details; its variant uses reference-latent initialization and boundary-only reconstruction, reducing iterations from 200 to 100.Hair and fur are more pronounced with Poisson blending, while the method alone can omit details such as an earring.Source: §3.3 · p. 5, Figure 5 · p. 5

4. Experiments

Experiments compare the method with classic and learning-based baselines on facial datasets, and examine alignment, density blending, Poisson blending, and additional capabilities. The results report improvements over baselines and demonstrate color-geometry disentanglement and multi-view consistent blending.

4.1. Experimental setup

  • Experiments use FFHQ and AFHQv2-Cat models, test on CelebA-HQ and AFHQv2-Cat, and evaluate masked L2, masked LPIPS, and KID.The study also includes ShapeNet-Car results; 250 image pairs per test dataset yield 1,250 CelebA-HQ and 750 AFHQv2-Cat blended images.Source: §4.1 · p. 5

4.2. Comparison with baselines

  • Our method outperforms baselines in all metrics on CelebA-HQ and AFHQv2-Cat, including baselines using our alignment, and produces realistic results while preserving original identity and reflecting the reference.Poisson blending further improves the method’s realism score in the user studies.Source: Table 1 · p. 6, §4.2 · p. 6, Table 2 · p. 7
  • 3D-aware alignment consistently improves all baselines across KID, LPIPSm, and masked L2; without alignment, reference pose differences degrade blending quality.Figure 6 illustrates that aligned references produce more realistic results that better reflect the reference.Source: §4.2 · p. 6, Figure 6 · p. 7, §4.3 · p. 8
  • MTurk participants prefer our method, with and without Poisson blending, over baselines for photorealism on CelebA-HQ and AFHQv2-Cat.The user studies use pairwise comparisons; percentages above 50% indicate preference for our method over a baseline.Source: Table 3 · p. 7, Table 4 · p. 7, §4.2 · p. 7

4.3. Ablation study

  • The density-blending loss helps preserve structured geometry: without it, the blended hair is blurry and the resulting mesh has shorter hair than the reference.The ablation attributes capturing highly structured objects in blending to the density-blending loss.Source: §4.3 · p. 8, Figure 8 · p. 8
  • Poisson blending further improves our method’s automatic metrics and realism scores, while adding it to baselines does not meaningfully help them.The baselines continue to show artifacts or fail to reflect the reference identity.Source: Table 3 · p. 7, Table 4 · p. 7, §4.3 · p. 8

4.4. Additional advantages of NeRF-based blending

  • Adjusting λ2 in the image-blending loss trades off reference color against reference geometry, while the original-image loss remains active to preserve the original.A lower or removed reference image-blending loss favors geometry; a higher weight favors color.Source: Figure 9 · p. 8, §4.4 · p. 9
  • An optimized latent code enables novel-view blended images in CelebA-HQ, AFHQv2-Cat, and ShapeNet-Car.Figure 10 shows multi-view blending results across these datasets.Source: §4.4 · p. 9, Figure 10 · p. 9

5. Discussion and Limitations

The method uses NeRFs to align and blend images in a 3D-aware manner, and the authors suggest extending it to unaligned multi-category datasets. Its reliance on GAN inversion limits overall quality and speed, with failures for inputs far from the mean face or with inaccurate estimated poses.

5. Discussion and Limitations

  • The approach uses NeRFs to align and blend images from collections of 2D images, with alignment improving 2D baselines and blending surpassing aligned baselines.The authors state it can be extended to unaligned multi-category datasets such as ImageNet because it relies on a pretrained generator.Source: §5 · p. 9
  • The method relies on GAN inversion, a bottleneck for overall quality and speed.Inversion may fail for inputs far from the mean face generated from the mean latent code or when the encoder-estimated camera pose is overly inaccurate.Source: §5 · p. 9

A. Experimental details

Experiments compare blending methods on FFHQ and AFHQv2-Cat, with dataset-specific resolution and training adjustments. Evaluation measures faithfulness to aligned reference images, while the appendix specifies implementation settings.

A. Experimental details

  • Baseline settings include OpenCV normal-clone Poisson blending and 1,000 StyleGAN3 projection iterations.StyleMapGAN uses spatial stylemaps at 8×8 and 16×16 for AFHQ, and 32×32 for FFHQ; SDEdit uses 300 iterations for small parts and 500 for face and hair.Source: A. Experimental details · p. 14
  • Experiments use FFHQ at 1024×1024 and AFHQv2-Cat at 512×512, with bilinear upsampling for SDEdit and EG3D in the FFHQ comparison.StyleMapGAN is fine-tuned on AFHQv2-Cat, EG3D on the original FFHQ crop, and ShapeNet-Car is used for additional demonstrations.Source: A. Experimental details · p. 14
  • Masked LPIPS evaluates faithfulness to aligned reference images used as pseudo-ground truth, and the method is always evaluated with alignment.The metric is used in Tables 1 and 2 for experiments both with and without 3D-aware alignment.Source: A. Experimental details · p. 14
  • The 3D-aware blending optimization uses a W+ latent initialized from the original image and runs for 200 iterations with Adam at a 0.02 learning rate.The latent code has dimensions 14×512; β1 = 0.9 and β2 = 0.999.Source: A. Experimental details · p. 14

B. Inversion details

Inversion first estimates an input image’s camera pose and latent code, then fine-tunes the generative NeRF. Mesh-based local alignment follows, and the reported single-image inversion runtime is 44 seconds on an A100 GPU.

B. Inversion details

  • The pose encoder predicts Euler angles rather than 4×4 camera extrinsics, then converts them back to extrinsics for inference.Training images are pre-aligned in scale and translation, so the camera pose can be simplified to a rotation matrix.Source: B. Inversion details · p. 15
  • PTI inversion first optimizes a 512-dimensional latent code for 300 iterations, then fine-tunes the generator for 100 iterations using reconstruction and regularization losses.The regularization compares the fine-tuned generator with its frozen version on randomly sampled, interpolated latent codes; λreg = 0.1.Source: B. Inversion details · p. 15
  • Local alignment extracts mesh-intersecting points through the target mask and uses ICP to estimate a scale-and-translation transform for rendering the aligned reference.The meshes are generated from density fields using Marching Cubes.Source: Figure 12 · p. 15
  • Single-image inversion takes 23.6 seconds for latent-code optimization and 20.4 seconds for NeRF fine-tuning on one A100 GPU.The two stages total 44 seconds.Source: B. Inversion details · p. 15

C. Local alignment

Local alignment refines pose alignment by matching target-region scale and translation in 3D. It improves the naturalness of blending when object parts differ in size, while ICP leaves rotation to pose alignment.

C. Local alignment

  • Local alignment matches target-region scale and translation by extracting mesh-intersecting 3D points through the mask and fitting an ICP transform.The transform uses uniform scaling and translation because pose alignment has already matched rotation.Source: C. Local alignment · pp. 15–16
  • The transform is applied in NeRF space by resampling points through the inverse matrix, changing apparent scale while the field mapping remains fixed.ICP stops at a difference threshold or a maximum of 20 iterations and does not estimate rotation.Source: C. Local alignment · p. 16
  • Users preferred results with local alignment in 60.3% of cases, and the method produced a more natural-looking cat blend with a properly scaled face.Pose alignment alone left a face too large when the original cat’s face was smaller than the reference cat’s.Source: C. Local alignment · p. 16, Figure 13 · p. 16
  • Pose alignment takes 1.7 seconds, while local alignment takes 4.2, 4.6, and 5.9 seconds for ears, eyes, and face, respectively.Larger blending regions take longer to align; the implementation uses CPU-based Trimesh functions.Source: C. Local alignment · p. 16

D. Details of 3D-aware Blending

The method uses W+ optimization to balance realism and reference faithfulness, with masks defining the aligned and blended regions. Tests also show its blending approach applies to StyleSDF, where density-related losses matter for structured objects.

D. Details of 3D-aware Blending

  • W+ produces favorable realism and reference faithfulness, while W is more realistic but less faithful and generator optimization is more faithful but less realistic.The comparison notes that W misses the reference’s green eyes, while generator optimization yields odd boundaries.Source: D. Details of 3D-aware Blending · p. 16, Figure 15 · p. 18
  • The target blending region uses the union of original and reference masks, and both masks are used for AFHQ local alignment.The reference mask is modified after local alignment.Source: D. Details of 3D-aware Blending · p. 16

E. 3D-aware blending in StyleSDF

F. User study details

The user studies compare perceived realism of blending results on Amazon Mechanical Turk. They evaluate comparisons with baselines and an ablation of local alignment.

F. User study details

  • MTurk workers selected the more realistic result in pairwise comparisons against baselines and in an ablation with versus without local alignment.The ablation used pose alignment in both conditions.Source: F. User study details · p. 19
  • CelebA-HQ used 60 evaluation sets and 1,500 pairwise comparisons, while AFHQv2-Cat used 40 sets and 1,000 comparisons.Workers needed a HIT Approval Rate above 98%; each set had 25 comparison questions and five deception checks.Source: F. User study details · p. 19
  • The local-alignment ablation used 20 AFHQv2-Cat evaluation sets, totaling 500 pairwise comparisons.Each set took approximately 2 to 3 minutes and offered a $0.5 reward.Source: F. User study details · p. 19

G. Failure cases

The appendix describes blending failures involving mask changes and local alignment, then identifies potential misuse of image blending and a need for forensics for 3D-aware content.

G. Failure cases

  • For large-to-small masks, undesirable effects from the reference can enter the blend; inpainting the original image is proposed as a possible remedy.Figure 22 shows a long-hair-to-short-hair failure case.Source: G. Failure cases · p. 19, Figure 22 · p. 20
  • ICP can fail when masks are inappropriate or target geometry is indistinct, and its point-cloud alignment can fall into a local extremum.The method restricts the scaling factor to [0.75, 1.25] and suggests pairwise registration techniques as future alternatives.Source: Figure 23 · p. 20, G. Failure cases · p. 20

H. Societal impact

  • Image blending could be misused to manipulate a real person’s expression or identity, or create a scene that does not exist.The passage notes that most existing watermarking and visual-forensics methods focus on 2D image content.Source: H. Societal impact · p. 20
  • Visual-forensics methods for 3D-aware content are identified as important future work.The stated context is that existing watermarking and visual-forensics methods mostly focus on 2D image content.Source: H. Societal impact · p. 20

I. Additional results

Additional comparisons describe identity and alignment differences from blending baselines, as well as failures of SDEdit to preserve references. Further examples cover structured objects, multiple views, and additional datasets and generators.

I. Additional results

  • Ours preserves identity while maintaining 3D consistency, whereas Latent Composition and StyleFusion alter identities and spatial-latent methods trade off performance on misaligned blends.The approach combines 3D-aware alignment with constraints from RGB pixels and volume density of the aligned reference.Source: §I · p. 20, Figure 24 · p. 21
  • SDEdit blending often fails to match references in CelebA-HQ and AFHQv2-Cat, with face or ear identities changed in both datasets.The AFHQv2-Cat comparison uses an LSUN-Cat diffusion model because pretrained AFHQ diffusion models are not publicly available.Source: §I · p. 20, Figure 25 · p. 21
  • Eyeglasses blending reflects high-fidelity 3D shape and produces multi-view-consistent results.Additional multi-view examples use EG3D on CelebA-HQ and AFHQv2-Cat, and StyleSDF on generated images.Source: §I · p. 20, Figure 26 · p. 21, Figure 29 · p. 24
  • Figures 27 and 28 provide supplementary baseline comparisons on CelebA-HQ and AFHQv2-Cat.The figures supplement the comparisons in Tables 1 and 2, respectively.Source: Figure 27 · p. 22, Figure 28 · p. 23
  • On ShapeNet-Car, local alignment performs well, and blending remains natural when region sizes differ.The examples use EG3D.Source: Figure 30 · p. 24
Loading 2302.06608v3…