Source-linked AI summary
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao
TL;DR
Editing NeRFs is difficult because their implicit, multi-view representations complicate controlled shape and appearance changes. CLIP-NeRF enables text- or image-driven manipulation through disentangled shape and appearance controls, with experiments demonstrating interactive editing and improved generated-shape quality.
Problem
NeRF editing is challenging because its implicit per-scene representation and multi-view dependencies hinder controlled shape and appearance manipulation.
Method
CLIP-NeRF uses a disentangled conditional NeRF with CLIP-driven code mappers and inversion to control shape and appearance from text prompts or exemplar images.
Results
Experiments show improved generated-shape quality and support user-friendly editing of both shape and appearance from a single text description or exemplar.
Takeaways & Limitations
The unified framework provides flexible interactive control of NeRF content through either language or exemplar images, including editing real images via inversion.
Takeaways & Limitations
The method cannot handle fine-grained and out-of-domain shape and appearance edits because of limitations in the latent space and pretrained CLIP.
Abstract
from arXiv · showhide
We present CLIP-NeRF, a multi-modal 3D object manipulation method for neural radiance fields (NeRF). By leveraging the joint language-image embedding space of the recent Contrastive Language-Image Pre-Training (CLIP) model, we propose a unified framework that allows manipulating NeRF in a user-friendly way, using either a short text prompt or an exemplar image. Specifically, to combine the novel view synthesis capability of NeRF and the controllable manipulation ability of latent representations from generative models, we introduce a disentangled conditional NeRF architecture that allows individual control over both shape and appearance. This is achieved by performing the shape conditioning via applying a learned deformation field to the positional encoding and deferring color conditioning to the volumetric rendering stage. To bridge this disentangled latent representation to the CLIP embedding, we design two code mappers that take a CLIP embedding as input and update the latent codes to reflect the targeted editing. The mappers are trained with a CLIP-based matching loss to ensure the manipulation accuracy. Furthermore, we propose an inverse optimization method that accurately projects an input image to the latent codes for manipulation to enable editing on real images. We evaluate our approach by extensive experiments on a variety of text prompts and exemplar images and also provide an intuitive interface for interactive editing. Our implementation is available at https://cassiepython.github.io/clipnerf/
1. Introduction
CLIP-NeRF introduces a unified framework for manipulating NeRF shape and appearance using either text prompts or exemplar images. It combines a disentangled conditional NeRF with CLIP-based code mappers and inversion for editing real images.
- Motivation: NeRF editing is challenging because its scene-specific implicit function lacks direct shape-editing tools and its multi-view dependencies complicate control.These challenges distinguish NeRF manipulation from editing explicit 3D representations or single-view images.
- Framework: The method provides text-and-image-driven NeRF manipulation through a unified framework using either a text prompt or an exemplar image.CLIP features from the chosen condition are mapped into latent-code updates for editing.
- Disentangled representation: A disentangled conditional NeRF uses a shape code to guide deformation-field warping and an appearance code to control emitted volumetric-rendering colors.This architecture enables individual control over geometry and appearance.
- CLIP-conditioned editing: Feedforward code mappers enable fast inference for editing different objects in the same category compared with optimization-based editing.The mappers translate CLIP features into local displacements in the shape or appearance latent codes.
- Real-image editing: An inversion method infers shape and appearance codes from a real image, enabling edits to existing real-image data.The inferred codes support subsequent manipulation of both geometry and appearance.
2. Related Work
NeRF represents continuous 3D shape and view-dependent appearance in an MLP, enabling high-resolution geometry capture and photorealistic novel-view rendering. However, its millions of parameters make direct editing unintuitive, motivating conditional latent-code approaches such as EditNeRF.
- NeRF and NeRF Editing: NeRF encodes continuous 3D shape and view-dependent appearance in the weights of an MLP.This follows the broader progress in neural implicit representations of 3D models.
- NeRF and NeRF Editing: NeRF captures high-resolution geometry and renders photorealistic novel views.These capabilities have contributed to NeRF’s growing popularity.
- NeRF and NeRF Editing: EditNeRF introduces shape and appearance codes so users can edit NeRF objects by optimizing adjustments to these latent codes.The approach addresses the unintuitive editing of NeRFs represented by millions of network parameters, though the passage notes limited capacity without specifying further details.
3. Method
CLIP-NeRF disentangles shape and appearance control in a conditional NeRF, then uses CLIP-driven mappers to update latent codes for text- or image-guided manipulation. A latent optimization inversion method further enables editing real images.
- Inversion: A novel latent optimization method projects an input image into shape and appearance codes, enabling manipulation of real images.The method is introduced specifically to support editing beyond images generated directly by the conditional NeRF.
- Conditional NeRF: The conditional NeRF models objects with separate shape and appearance codes, mapping position, view direction, and codes to volumetric density and radiance.This formulation provides category-level generative conditioning for shape and appearance.
- Disentangled Conditional NeRF: Shape conditioning deforms positional encoding through a learned displacement field, encouraging smooth deformations that preserve base-shape details.The deformation network maps positions and shape codes to per-frequency displacement vectors, while tanh constrains displacements to [−1, 1].
- Disentangled Conditional NeRF: Appearance conditioning is deferred to radiance prediction by concatenating the appearance code with view direction, allowing appearance edits without changing density.Because density represents shape, this separates appearance manipulation from shape information.
- CLIP-Driven Manipulation: CLIP text or image embeddings drive separate shape and appearance mappers that directly produce code updates for feed-forward NeRF manipulation.The cross-modal CLIP distance compares text and rendered image embeddings, and the same framework supports exemplar-image editing by replacing the text prompt.
- Training Strategy: Training proceeds in two stages: first the disentangled conditional NeRF and deformation network are trained, then the generator is fixed while the CLIP mappers are trained.The generator uses a non-saturating GAN objective with shape codes, appearance codes, and camera poses sampled from their respective distributions.
4. Experiments
Experiments on chair and car datasets show that CLIP-NeRF supports simpler, more natural text-driven editing than EditNeRF while preserving image quality and disentangling shape from appearance. Ablations, inversion tests, real-image experiments, and a user study further assess its controllability, generalization, and matching accuracy.
- Experimental Setup: CLIP-NeRF is evaluated on Photoshapes with 150K chairs and Carla with 10K cars, using randomly viewed images without camera pose parameters.Photoshapes images are rendered at 128×128 and Carla images at 256×256.
- Disentanglement Ablation: The disentangled design keeps appearance unchanged during shape editing by isolating shape deformation from appearance control, while removing it frequently introduces color changes.The deformation network also regularizes generated shapes and further improves resulting quality.
- Comparison with EditNeRF: Compared with EditNeRF, CLIP-NeRF enables simpler text-prompt color editing and produces more natural results, whereas EditNeRF requires target colors and coarse regional scribbles.EditNeRF’s edited colors can show discontinuities and may not faithfully match the target color.
- Image Quality: CLIP-NeRF preserves comparable image quality before and after manipulation, while EditNeRF’s editing notably degrades quality despite better pre-edit chair reconstruction from its 40-view training.The comparison uses FID scores on 2K testing images before and after editing.
- Inversion Ablation: Adding a CLIP constraint to inversion improves over single-image photometric optimization because CLIP supplies robust pose-invariant features and multi-view consistency.The baseline without CLIP is limited by the difficulty of inferring a complete 3D NeRF from one view.
- Generalization and User Study: The method generalizes to real single images and achieves more than 80% matching accuracy in a 23-participant user study covering text and exemplar guidance.Real-image inversion is more challenging for chairs with delicate structures than for cars.
5. Extended Discussions
The method supports continuous, fine-grained, and strength-controlled shape or appearance manipulation, while its latent space enables interpolation. Direct single-NeRF shape editing remains unsatisfactory under the CLIP constraint.
- Continuous Manipulation: Users can continuously edit shape and appearance in either order from one text prompt or exemplar.This provides a user-friendly way to jointly edit shape and appearance.
- Fine-grained appearance manipulation within a same color category: The method supports fine-grained whole-object appearance variations within the same color category, despite limitations on local-part edits.The discussion illustrates such variations using the color blue.
- Scaling along Editing Direction: Increasing scalar s strengthens manipulation along learned shape and appearance directions, including directions learned from exemplars.The scaled scheme produces gradually stronger editing results as s increases.
- Interpolation: The latent space supports interpolation between two codes using z_inter = z_2×r + z_1×(1−r), with r from 0 to 1.0 in steps of 0.1.The interpolated result is obtained from z_inter.
- Necessity of latent space: The CLIP constraint edits a single NeRF’s color without ground truth but fails to produce satisfying text-prompt shape edits.The authors attribute this failure to a CLIP loss insufficiently strong and compact to deform shape without latent-space constraints.
7. Conclusion
CLIP-NeRF provides unified text- and image-driven NeRF manipulation through disentangled shape and appearance control, fast code mappers, and inversion for editing. Its main limitation is difficulty with fine-grained and out-of-domain edits because of limited latent-space and CLIP expressiveness.
- Contributions: CLIP-NeRF enables flexible 3D content manipulation from either a text prompt or an exemplar image.The framework is presented as the first text-and-image driven manipulation method for NeRF.
- Contributions: Its disentangled conditional NeRF separates shape and appearance editing, while two feedforward code mappers support fast inference across objects.The architecture disentangles shape and appearance, and the mappers update codes for editing different objects.
- Contributions: An inversion method infers shape and appearance codes to enable manipulation of input content.The supplied conclusion passage introduces this inversion method but is truncated after “cod”.
- Evaluation: The method is evaluated on varied text prompts and exemplar images and includes an intuitive interface for interactive editing.The conclusion explicitly reports extensive experiments and an interactive editing interface.
- Limitations: Fine-grained and out-of-domain shape and appearance edits remain difficult because the latent space and pre-trained CLIP have limited expressive ability.The authors suggest adding more varied training data as a possible mitigation.