PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRF-Art: Text-Driven Neural Radiance Fields Stylization

Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao

arXiv:2212.08070v1cs.CVcs.GR

TL;DR

Text-guided NeRF stylization remains difficult when both appearance and geometry must change without sufficient deformation or texture detail. NeRF-Art stylizes a pre-trained NeRF using directional and global-local contrastive CLIP constraints, with weight regularization for geometry transformation. Experiments report effective single-view stylization and cross-view consistency across various styles.

  • Problem

    Text-guided NeRF stylization lacks sufficient geometry deformations and texture details, while some methods require mesh guidance or image references.

  • Method

    NeRF-Art combines relative directional CLIP loss with global-local contrastive learning to control style trajectory and strength while transforming a pre-trained NeRF.

  • Results

    The method demonstrates effective stylization quality and cross-view consistency across various styles, including simultaneous appearance and geometry transformation without mesh guidance.

  • Takeaways & Limitations

    Text prompts can guide consistent 3D scene stylization with both appearance and geometry variations without mesh input.

  • Takeaways & Limitations

    The method assumes target and source renderings use the same arbitrarily sampled view during stylization training.

Abstract

from arXiv · show

As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images. Stylizing NeRF, however, remains challenging, especially on simulating a text-guided style with both the appearance and the geometry altered simultaneously. In this paper, we present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. Unlike previous approaches that either lack sufficient geometry deformations and texture details or require meshes to guide the stylization, our method can shift a 3D scene to the target style characterized by desired geometry and appearance variations without any mesh guidance. This is achieved by introducing a novel global-local contrastive learning strategy, combined with the directional constraint to simultaneously control both the trajectory and the strength of the target style. Moreover, we adopt a weight regularization method to effectively suppress cloudy artifacts and geometry noises which arise easily when the density field is transformed during geometry stylization. Through extensive experiments on various styles, we demonstrate that our method is effective and robust regarding both single-view stylization quality and cross-view consistency. The code and more results can be found in our project page: https://cassiepython.github.io/nerfart/.

1 INTRODUCTION

NeRF-Art addresses the challenge of text-guided 3D stylization by jointly transforming appearance and geometry from a pre-trained NeRF. It uses CLIP-based directional and contrastive constraints to improve style control without mesh guidance.

  • 3D stylization aims to transform multi-view content, but explicit representations trade intuitive geometry control for limited complex-scene modeling and rendering capacity.
  • Existing NeRF stylization methods primarily use image references, which may be difficult or impossible to obtain for the desired style and content.
  • Text-guided NeRF stylization remains challenging because existing methods provide insufficient geometry deformations and texture details.
  • NeRF-Art transforms both appearance and geometry from a pre-trained NeRF using a single text prompt, enabling consistent novel-view synthesis.
  • Its CLIP-based global-local contrastive loss strengthens the target style across global structures and local details, while density relaxation supports geometry stylization.

2 RELATED WORK

Prior 3D stylization methods use images, meshes, voxels, point clouds, or NeRFs, but commonly constrain scene complexity, appearance, geometry, or guidance. NeRF-Art instead targets text-guided joint appearance and geometric stylization without mesh input.

  • Neural Style Transfer on Images and Videos: Image and video stylization methods are restricted to given views, so combining them with novel-view synthesis can produce blurriness or view inconsistencies.
  • NeRF-Art: NeRF-Art combines relative directional and global-local contrastive losses to control stylization trajectory and strength.
  • Neural Stylization on Explicit 3D Representations: Explicit 3D methods use meshes, voxels, or point clouds, but their expressiveness is limited for complicated objects or scenes with dedicated structures.
  • Neural Stylization on Implicit 3D Representations: Recent NeRF stylization methods achieve consistent novel-view appearance transfer but do not adjust density parameters, restricting stylization to appearance.
  • Text-Driven Stylization: Text-driven methods such as Text2Mesh and AvatarCLIP support natural-language guidance but require mesh input.

3 OVERVIEW

NeRF-Art separates reconstruction from stylization: it first learns a NeRF from multi-view images, then transforms that model under text guidance using CLIP-based losses.

  • Reconstruction: The pipeline first pre-trains a target-scene NeRF from multi-view input using a supervised reconstruction loss.
  • Stylization: The stylization method introduces directional CLIP loss, global-local contrastive learning, and weight regularization as successive components of the framework.
  • NeRF Representation: NeRF represents a scene as a continuous volumetric field whose network predicts density and view-dependent radiance for sampled spatial coordinates and directions.
  • NeRF Representation: Rendered ray colors are computed by accumulating contributions from sampled points using transmittance and density-dependent weights.

4 TEXT-GUIDED NERF STYLIZATION

This section formulates text-guided NeRF stylization around trajectory and strength control, content preservation, and artifact suppression. The method combines directional and global-local contrastive losses with perceptual and weight regularization objectives.

  • Overview: The method optimizes a stylized NeRF from a reconstructed model while preserving content under a target text prompt.The stylized model is trained from the pre-trained NeRF and must retain its source content.
  • Trajectory Control: Directional CLIP loss guides the stylization trajectory using source-target text directions rather than a single global target direction.The source and stylized renderings share an identical sampled view.
  • Strength Control: Directional loss alone can preserve the trajectory but struggles to produce sufficient stylization strength.It compares normalized embedded directions and therefore does not directly enforce the magnitude of the style change.
  • Strength Control: Global-local contrastive learning uses target-style positives and semantically irrelevant text negatives to control stylization strength across whole views and local patches.The global term can still produce uneven stylization, motivating the combined global-local formulation.
  • Artifact Suppression: Weight regularization suppresses cloud-like artifacts and geometry noise by concentrating ray-sample weights near object surfaces.The loss penalizes scattered large weights across sample pairs, while perceptual loss preserves content during finetuning.
  • Training: Training combines directional and global-local stylization losses with VGG-based perceptual loss and artifact-suppression regularization.The method also addresses memory constraints by avoiding full-ray stylization training, since coarse renderings and patches can lose details and semantic structure.

5 EXPERIMENTS

The experiments include implementation settings and a text-description evaluation using prompts with different levels of detail. The setup uses staged NeRF reconstruction and stylization training with manually collected negative style descriptions.

  • Text Evaluation: Text evaluation compares descriptions at different detail levels for a specific style.The evaluation is motivated by CLIP’s sensitivity to text prompts.
  • Implementation Details: The stylization hyperparameters λ_g, λ_l, λ_p, and λ_r are set to 0.2, 0.1, 2.0, and 0.1, respectively.Negative samples comprise around 200 manually collected Pinterest descriptions of varied styles.

5.2 Data Collection

The evaluation examines prompt design and stylization across several comparison settings, including image-, NeRF-, and mesh-based methods. It also tests generalization and the effects of global-local contrastive losses.

  • Comparisons: The study compares NeRF-Art with StyleGAN-NADA, CLIP-NeRF, DreamField, Text2Mesh, and AvatarCLIP.These methods span text-driven image, NeRF, and mesh-based stylization categories.
  • Text Evaluation: Text descriptions include general styles, named subjects, and augmented prompts such as “Tolkien Elf,” “the back of Tolkien Elf,” and “the face of Tolkien Elf.”Prompt variants are used for detailed refinement and style evaluation.
  • Text Evaluation: General style prompts can be insufficient, with “Fauvism” inducing stylization mainly around the mouth.The text evaluation contrasts broad style descriptions with more specific object or subject references.
  • Generalization Evaluation: Generalization is evaluated on the NeRF-like models VolSDF and NeuS.The evaluation tests whether the approach transfers beyond the original NeRF representation.
  • Ablation Study: Without global-local contrastive losses, stylization becomes insufficient or non-uniform for prompts including “White Walker” and “Tolkien Elf.”This ablation isolates the role of the contrastive losses in text-guided stylization.

5.4 Comparisons

NeRF-Art is compared with image-, NeRF-, and mesh-based text stylization methods using visual, geometry, artifact, and user-preference evaluations. The comparisons emphasize style strength, view consistency, geometry, and fine-grained details.

  • Image Stylization: Compared with StyleGAN-NADA, NeRF-Art better controls style strength and preserves view consistency across stylized views.StyleGAN-NADA stylizes views independently, which can introduce inconsistent shapes or textures and video flickering.
  • NeRF Stylization: NeRF-Art is evaluated against CLIP-NeRF and DreamField as text-guided NeRF stylization baselines.The cited comparison establishes the NeRF-based comparison category, while the detailed outcome is not included in the supplied passage.
  • Geometry Evaluation: NeRF-Art simultaneously modulates geometry and color of a pre-trained NeRF to match a text-described style.The geometry evaluation compares meshes extracted before and after stylization.
  • Weight Regularization: Weight regularization suppresses cloudy artifacts near corners and geometric noise during stylization.The ablation examines the artifacts observed when the regularization loss is removed.
  • Mesh-Based Stylization: Compared with AvatarCLIP, NeRF-Art produces finer beard, garment-wrinkle, and facial details while supporting in-the-wild faces without a 3D mesh.AvatarCLIP requires mesh input and can produce noisy surfaces and background bumps.
  • User Study: The user-preference evaluation compares NeRF-Art with StyleGAN-NADA, CLIP-NeRF, Text2Mesh, DreamField, and AvatarCLIP.Preference rates are reported in Figure 12.

5.5 User Study

The user study evaluates stylization quality through human judgments of content preservation, style faithfulness, and view consistency. Across the compared methods, NeRF-Art receives higher user preference rates.

  • Study Design: 23 questionnaires from 10 male and 13 female participants were collected for the user study.The study used 50 total test cases, with two subjects per comparison category.
  • Study Design: Participants jointly judged content preservation, style faithfulness, and view consistency from two views shown in random order.Each test case included input frames, a textual prompt, and results from different methods.
  • Results: NeRF-Art achieved much higher user preference rates than StyleGAN-NADA, CLIP-NeRF, Text2Mesh, DreamField, and AvatarCLIP.The preference statistics are shown in Figure 12.

5.6 Ablation Study

The ablation study explains how the loss components control stylization direction, magnitude, and spatial uniformity, while weight regularization stabilizes geometry transformations. Together, these components produce more uniform stylization and reduce artifacts.

  • Global-Local Contrastive Learning: Directional CLIP loss controls stylization trajectory but can produce insufficient stylization magnitude.It measures directional similarity between normalized embeddings and ignores their actual distances.
  • Global-Local Contrastive Learning: Global contrastive loss increases stylization magnitude but can leave some regions insufficiently stylized and others excessively stylized.The passage attributes this imbalance to CLIP attending more to distinguishable regions.
  • Global-Local Contrastive Learning: Combining global and local contrastive losses with directional CLIP achieves uniform stylization with correct direction and sufficient magnitude.The combined result is illustrated in the fifth column of Figure 9.
  • Weight Regularization: Weight regularization suppresses cloudy artifacts and geometric noise by encouraging a more concentrated density distribution.The effect is demonstrated in the Figure 11 ablation.

5.7 Generalization Evaluation

NeRF-Art generalizes across NeRF-like models and produces style-matched geometry changes, while linguistic ambiguity can still yield unexpected results.

  • 5.7 Generalization Evaluation: Linguistic ambiguity or semantically meaningless prompts may lead to unexpected stylization results.The limitations figure illustrates these two prompt-related failure modes.
  • 5.7 Generalization Evaluation: NeRF-Art produces similar stylization results on VolSDF and NeuS, demonstrating adaptation across different NeRF-like models.The evaluation uses VolSDF and NeuS, with segmentation applied to NeuS reconstruction for in-the-wild data.
  • 5.7 Generalization Evaluation: Geometry evaluations show style-specific changes, including flattened noses, sharpened ears, and rounded jaws, on both VolSDF and NeuS.Meshes extracted before and after stylization reveal geometry changes matching the requested styles.

6 CONCLUSION

NeRF-Art uses CLIP, directional constraints, global-local contrastive learning, and weight regularization for text-guided joint stylization of NeRF geometry and appearance. Experiments show effective stylization quality and view consistency, although ambiguous prompts remain limiting.

  • 6 CONCLUSION: NeRF-Art jointly modulates NeRF geometry and appearance from text guidance without mesh guidance.The method targets geometry deformations and texture details during stylization.
  • 6 CONCLUSION: Directional constraints control style trajectory, while global-local contrastive loss enforces the desired style strength.These components form the method’s central style-control design.
  • 6 CONCLUSION: Weight regularization alleviates cloudy artifacts and geometry noises during geometry deformation.The regularization addresses artifacts associated with transforming the density field.
  • 6 CONCLUSION: Experiments on real faces and general scenes show effective and robust stylization quality and view consistency.The conclusion reports both visual quality and cross-view consistency across the evaluated scene types.
  • 6 CONCLUSION: Linguistically ambiguous prompts, such as “Digital painting,” remain a limitation because they can describe multiple styles.The conclusion identifies prompt ambiguity as a source of limitation despite success in most cases.
Loading 2212.08070v1…