PaperlayerView exact revision on arXiv

Source-linked AI summary

Customizing Text-to-Image Diffusion with Object Viewpoint Control

Nupur Kumari, Grace Su, Richard Zhang, Taesung Park, Eli Shechtman, Jun-Yan Zhu

arXiv:2404.12333v2cs.CV

TL;DR

Customizing text-to-image models does not provide precise control over a new object’s camera viewpoint, leaving users to rely on coarse prompt instructions. The paper conditions diffusion generation on view-dependent 3D object features, and reports stronger performance than image-editing and model-customization baselines while preserving identity, viewpoint, and text alignment.

  • Problem

    Text-to-image customization lacks precise control over a new object’s viewpoint, so users often rely on prompts such as “front-facing” or “side-facing.”

    Source: §1 · p. 2

  • Method

    The method learns view-dependent object features in intermediate diffusion features and conditions generation on features rendered from the target viewpoint.

    Source: §3.2 · p. 4, §5 · p. 11

  • Results

    The method achieves high alignment with custom-object identity and target viewpoint while following text prompts, and enhances performance over 2D and 3D editing baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Custom objects can be synthesized at varying viewpoints while text prompts control other aspects of generation.

    Source: §5 · p. 11

    The method can struggle with extreme viewpoints unseen during training, prompts involving multiple objects, and per-object fine-tuning time of about 40 minutes.

    Source: §5 · p. 11

Abstract

from arXiv · show

Model customization introduces new concepts to existing text-to-image models, enabling the generation of these new concepts/objects in novel contexts. However, such methods lack accurate camera view control with respect to the new object, and users must resort to prompt engineering (e.g., adding ``top-view'') to achieve coarse view control. In this work, we introduce a new task -- enabling explicit control of the object viewpoint in the customization of text-to-image diffusion models. This allows us to modify the custom object's properties and generate it in various background scenes via text prompts, all while incorporating the object viewpoint as an additional control. This new task presents significant challenges, as one must harmoniously merge a 3D representation from the multi-view images with the 2D pre-trained model. To bridge this gap, we propose to condition the diffusion process on the 3D object features rendered from the target viewpoint. During training, we fine-tune the 3D feature prediction modules to reconstruct the object's appearance and geometry, while reducing overfitting to the input multi-view images. Our method outperforms existing image editing and model customization baselines in preserving the custom object's identity while following the target object viewpoint and the text prompt.

1. Introduction

Customizing text-to-image models can place personal objects in new contexts, but offers little precise control over their viewpoint. This work adds target-viewpoint control by conditioning generation on 3D object features, and reports stronger performance than image-editing and model-customization baselines.

1. Introduction

  • Existing customization methods can place personal objects in new environments and attributes, but lack precise viewpoint control, leaving users to rely on tedious text prompts such as “front-facing.”Source: §1 · p. 2
  • The task is to customize a text-to-image model from multi-view object images while allowing inference to use both a target viewpoint and a text prompt.Source: §1 · p. 2
  • The central challenge is bridging 3D neural object representations and diffusion models whose internal features derive from purely 2D representations.Source: §1 · p. 2
  • CustomDiffusion360 inserts FeatureNeRF blocks into intermediate U-Net features, renders view-dependent features at the target viewpoint, and merges them with diffusion features through linear projections.Source: §1 · p. 2
  • The method achieves high alignment with custom-object identity and target viewpoint while following text prompts, and enhances performance over 2D and 3D editing baselines.It is built on Stable Diffusion-XL and demonstrated across cars, chairs, motorcycles, teddy bears, and toys.Source: §1 · p. 2

2. Related Works

Related work spans text-guided image editing, model customization, novel-view synthesis, and 3D editing. The paper distinguishes its goal as viewpoint control for customized objects in text-to-image generation rather than appearance-only edits, scene reconstruction, or editing a single photograph.

Image editing with text-to-image diffusion

  • Text-to-image editing methods use denoising, latent embeddings, attention, or conditional diffusion, but primarily target appearance rather than joint viewpoint and appearance control.Source: §2 · p. 2

Model customization

  • Customization methods embed new concepts into text-to-image models for unseen scenarios, but the authors know of no prior method that controls viewpoint during customization.The paper uses multi-view images of a new concept to add this control.Source: §2 · p. 3

View synthesis

  • Novel-view synthesis targets unseen camera poses for scenes, whereas this work targets object viewpoint control in personalization; ReconFusion instead learns latent-space PixelNeRF for 3D reconstruction.Other camera-conditioning works and CustomNet also differ in their focus or inference conditions.Source: §2 · p. 3

3D editing

  • Unlike 3D editing of objects in photographs or methods generating multi-view-consistent scenes, this work edits pretrained diffusion-model weights to control object viewpoints in new backgrounds or compositions.For this task, the authors report greater photorealism than a 3D editing method.Source: §2 · p. 3

3. Method

The method adds explicit object-viewpoint conditioning to text-to-image customization by joining a 3D representation learned from reference views with the diffusion model’s feature space. It uses FeatureNeRF-rendered features, training losses, and inference guidance to support generation under both target-view and text conditions.

3.1. Diffusion Models

  • The underlying diffusion model learns to predict added noise, then generates images by iteratively denoising Gaussian noise; the method uses Stable Diffusion-XL in a latent autoencoder space.The forward process gradually perturbs real data over timesteps, and sampling strategies can reduce the number of inference steps.Source: §3.1 · p. 3

3.2. Customization with Object Viewpoint Control

  • The method adds pose-conditioned transformer layers to the diffusion U-Net, rendering features from multi-view references at a target viewpoint and merging them with diffusion features.Only the added linear projection layers and FeatureNeRF blocks are fine-tuned; the pre-trained model remains frozen.Source: Figure 2 · p. 4, §3.2 · p. 4
  • FeatureNeRF aggregates projected reference-view features with learnable weighted averaging, updates them with text cross-attention, and renders the resulting volume using predicted densities.Predicted RGB is used only to calculate the training reconstruction loss.Source: Figure 3 · p. 4, §3.2 · p. 5
  • Training combines diffusion reconstruction with FeatureNeRF losses, including RGB reconstruction and mask-based silhouette and background-suppression terms.The losses assume object masks are available and are averaged across pose-conditioned transformer layers.Source: §3.2 · p. 5
  • Inference combines image and text guidance to balance similarity to reference views against consistency with the prompt.Training also uses category-matched generated images 25% of the time and drops prompts with 10% probability.Source: §3.2 · pp. 5, 8
  • Qualitative samples vary the target viewpoint while using prompts to change scene context and object attributes, including color and shape.Images in each row share a seed but are independently generated.Source: Figure 4 · p. 6, Figure 5 · p. 7
  • Our method is preferred over almost all baselines for text alignment, target identity, and photorealism; LoRA + Camera pose overfits and misses new-prompt viewpoints.Table 1 reports human preferences, while Table 2 describes the baseline’s viewpoint failures with new text prompts.Source: Table 1 · p. 8, Table 2 · p. 8

4. Experiments

The experiments assess customization quality and viewpoint adherence against editing and customization baselines, then examine extrapolation, applications, and component ablations.

4. Experiments

  • Experiments use 14 CO3Dv2/NAVI objects, 16 prompts per category, human preferences plus CLIP/DINO alignment, and RayDiffusion pose errors.The pose-error evaluation uses validation camera poses; prompts cover scene, color, composition, and shape changes.Source: §4 · p. 8

4.1. Results

  • Our learned radiance field extrapolates at inference to unseen viewpoints by varying focal length, camera distance, or horizontal and vertical camera position.The reported examples vary focal length from 0.8× to 1.4× the original.Source: §4.1 · pp. 9–10
  • Our method is preferred over all baselines except LoRA + Camera pose, with on-par or better text alignment and better image alignment.Image-editing baselines require careful hyperparameter tuning; the selected best settings were fixed across experiments.Source: Figure 6 · p. 8, §4.1 · p. 9
  • RayDiffusion pose-error evaluation is limited to CO3Dv2 validation poses; LoRA + Camera pose often overfits training images and misses viewpoints under new prompts.RayDiffusion struggles with the unique objects outside CO3Dv2.Source: §4.1 · p. 9
  • On three car objects, Image Sculpting scores 0.26 CLIP and 0.24 DINO, Object3DIT 0.27 CLIP and 0.40 DINO, versus our 0.25 CLIP and 0.48 DINO.Both editing methods produced lower-fidelity results in the qualitative comparison.Source: §4.1 · p. 10, Figure 8 · p. 10
  • Our method combines with SDEdit for inpainting, MultiDiffusion for panoramas, and FeatureNeRF for multi-instance composition.The examples control the viewpoint independently across panorama grids and object instances.Source: §4.1 · p. 10, Figure 9 · p. 10

4.2. Ablation

  • Failures occur for views far from training, such as extreme focal-length reduction or off-center placement, and sometimes in multi-object scenes.The pre-trained model is often biased toward centering the object.Source: Figure 10 · p. 11
  • Removing mask-based losses decreases text alignment and encourages training-image overfitting, while removing text cross-attention reduces target-concept image alignment.The final method balances target-concept, text-prompt, and camera-pose conditions.Source: Table 3 · p. 10, §4.2 · p. 11

5. Discussion and Limitations

The paper introduces viewpoint-controlled customization through view-dependent object features in a diffusion model, while noting limits in viewpoint range, scene composition, compute, and object type.

5. Discussion and Limitations

  • The method learns view-dependent object features in the diffusion model’s intermediate feature space and conditions generation on them.This supports varying object viewpoints while text prompts control other aspects of the synthesis.Source: §5 · p. 11
  • At extreme unseen viewpoints, the method can change object identity or revert to a seen view; the authors expect more viewpoint variation in training to help.Source: §5 · p. 11
  • Multi-object prompts can disrupt text or viewpoint adherence; per-object fine-tuning takes about 40 minutes, and the method currently targets rigid objects.Dynamic objects are left for future work.Source: §5 · p. 11

Appendix

  • The appendices provide additional results and ablations, evaluation and human-preference details, and implementation details.Source: Appendix · p. 17

A. Ablation

Ablations find comparable performance with predicted masks and cameras, while reducing the number of training views preserves text alignment but weakens camera-pose accuracy and image alignment.

A. Ablation

  • With 35 or 20 views, text alignment remains similar, while camera-pose accuracy and image alignment gradually decrease.The main experiments use approximately 50 multiview images with ground-truth poses.Source: A. Ablation · p. 17
  • Replacing ground-truth masks with predicted masks produces similar performance for background losses.Masks are predicted using Detic with Segment Anything, conditioned on the object category.Source: A. Ablation · p. 17
  • 17.12 mean angular error with COLMAP-predicted cameras compares with 16.14 for the final method on 10 objects.CLIP and DINO scores remain similar; COLMAP fails on two car objects, and training takes 2000 rather than 1600 steps.Source: A. Ablation · p. 17

B. More Results

Across categories, the method generally aligns better with text and lies on the Pareto frontier against baselines. Pairwise preferences and comparisons with IP-Adapter plus ControlNet and LoRA provide further evidence across alignment measures.

B. More Results

  • Across categories and guidance scales, our method usually has higher CLIP scores and lies on the Pareto frontier, performing similarly to or better than baselines.The comparison varies guidance scales from 5.0 to 10.0, except LEDITS++ at 10.0 to 15.0.Source: B. More Results · p. 17
  • Predicted masks yield similar performance to ground-truth masks, while fewer views gradually reduce camera-pose accuracy and image alignment.These comparisons are summarized in Table 4.Source: Table 4 · p. 17
  • 60 ± 3% and 75.67 ± 2.6% preferred our method over IP-Adapter plus depth ControlNet for text and image alignment, respectively.Their CLIP and DINO scores were 0.251 and 0.497, compared with 0.253 and 0.481 for our method.Source: B. More Results · p. 17
  • Against LoRA, our method scores 0.253 CLIP and 0.481 DINO, while LoRA scores 0.275 CLIP and 0.468 DINO.The paper describes its CLIP score as marginally lower, while preserving the target concept and adding object-viewpoint control.Source: B. More Results · p. 17

C. Evaluation.

The evaluation uses generated text prompts, validation camera poses, alignment metrics, and pairwise human preferences. Figure 11 reports similar performance with predicted cameras using approximately 50 training images.

C. Evaluation

  • Evaluation prompts include diverse captions for plausible motorcycle images, covering scene changes, inserted objects, motorcycle type, and color.The example instruction requests four captions for each of the four options.Source: C. Evaluation · p. 17
  • Figure 11 shows similar performance with predicted cameras using approximately 50 training images.The camera predictions are obtained using COLMAP.Source: Figure 11 · p. 18
  • For camera-pose accuracy, six validation poses are tested with generated images, and RayDiffusion estimates pose error against each target.Validation camera principal axes point toward the object; text- and image-alignment metrics use perturbed validation poses.Source: C. Evaluation · p. 18
  • Pairwise human studies compare generated images with prompts for text alignment, target images for image alignment, and each other for photorealism.For image alignment, participants see three or four target images before judging the generated pair.Source: C. Evaluation · p. 18
  • DINOv2 image alignment is the mean similarity of each generated image to training images, using masked object crops.The training-image object regions are cropped with masks so similarity measures the target concept.Source: C. Evaluation · p. 18

D.1. Our Method

Training adds FeatureNeRF and projection modules to selected Stable Diffusion-XL layers, with reference views used to learn viewpoint-conditioned features. Inference combines cached reference features with text and image guidance.

D.1. Our Method

  • Training runs for 1600 iterations on four A100 GPUs, using batch size 4 and a learning rate of 1 × 10^-4.The model is Stable Diffusion-XL in 32-bit floating point, and sampling is biased toward later time steps.Source: §D.1 · p. 18
  • Each training step uses five equidistant views, with one target and four references; 12 of 70 U-Net transformer layers receive pose conditioning.FeatureNeRF blocks sample 24 ray points, and importance sampling from previous predicted density is used 90% of the time to improve thin-structure performance.Source: §D.1 · p. 18
  • Inference uses image and text guidance scales of 3.5 and 7.5, respectively, and generates an image in 10 seconds given eight reference views.Generation uses 50 Euler-scheduler sampling steps; reference features are cached for each customized model.Source: §D.1 · p. 18

D.2. Baselines

The appendix compares image-editing, 3D-editing, and model-customization baselines, and describes the evaluation views, concepts, and additional diagnostic comparisons.

D.2. Baselines

  • The image-editing baselines include SDEdit, LEDITS++, and InstructPix2Pix, using their stated inference settings or recommended hyperparameters.For SDEdit with Stable Diffusion-XL, the base model runs 15 steps and the refiner 5 more at strength 0.5; LEDITS++ settings differ for appearance and background edits.Source: §D.2 · pp. 18–20
  • Across object categories, our method achieves higher or the same text alignment as baselines while maintaining on-par image alignment.Figure 12 separates CLIP and DINO scores by category.Source: Figure 12 · p. 19
  • Evaluation camera poses perturb the validation cameras’ locations and focal lengths, and the target concepts cover 14 objects.Figure 13 shows sample training and validation views; Figure 16 shows the target concepts.Source: Figure 13 · p. 19, Figure 16 · p. 20
  • Without silhouette and background losses, FeatureNeRF overfits to background features; fewer training views can also leave back-facing target poses unrespected.Figure 15 reports a bias toward front-facing objects as training views decrease.Source: Figure 14 · p. 19, Figure 15 · p. 19
  • ViCA-NeRF edits key multiview images, reprojects and blends edits, then trains a 3D-edited NeRF.For NAVI concepts, the baseline uses images from one scene; our method uses multiple scenes because it models only the foreground object.Source: §D.2 · p. 20
  • The LoRA baseline fine-tunes Stable Diffusion-XL with rank-64 adapters, while LoRA + Camera pose adds pose conditioning through cross-attention.LoRA trains for 1,000 iterations; LoRA + Camera pose trains for 2,000, with reconstruction loss applied only in the masked region for the latter.Source: §D.2 · p. 20

E. Change log

The change log records additions to the second version, while the supplied figures and table describe further comparisons, samples, and evaluation prompts.

E. Change log

  • Version 2 adds comparisons with 3D-editing methods in Section 4.1 and more ablation experiments in Appendix A.The change log also notes updated citations and writing.Source: E. Change log · p. 20
  • The additional qualitative comparison shows the method against baselines under varied prompts and target-pose conditions.Figure 18 is identified as an additional comparison.Source: Figure 17 · p. 22
  • The supplied supplementary visuals include samples varying the custom object’s camera pose, and the evaluation prompt table lists prompts for five categories.The categories are cars, chairs, teddy bears, motorcycles, and toys.Source: Table 5 · p. 21, Figure 18 · p. 23
Loading 2404.12333v2…