Source-linked AI summary
ARF: Artistic Radiance Fields
Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, Noah Snavely
TL;DR
ARF addresses the challenge of transferring a 2D artistic style to complex 3D scenes without relying on error-sensitive point-cloud or mesh stylization. It stylizes radiance fields with nearest-neighbor feature matching and deferred back-propagation, and produces detailed, view-consistent renderings that outperform prior methods in visual quality.
Problem
3D stylization methods based on point clouds or meshes are sensitive to geometric reconstruction errors, while Gram-matrix losses can blur style details.
Method
ARF fine-tunes a reconstructed radiance field using NNFM style loss, VGG content loss, color transfer, and deferred back-propagation for full-resolution image losses.
Results
ARF transfers detailed artistic features to complex 3D scenes, producing more visually faithful stylized novel views than previous methods and consistent user-study preference over baselines.
Takeaways & Limitations
Radiance-field stylization with NNFM and view-consistent color transfer supports high-quality novel views that faithfully mimic style color tone and brushstroke details.
Takeaways & Limitations
Optimization remains time-consuming for each style image, and reconstructed artistic radiance fields do not support manual editing.
Abstract
from arXiv · showhide
We present a method for transferring the artistic features of an arbitrary style image to a 3D scene. Previous methods that perform 3D stylization on point clouds or meshes are sensitive to geometric reconstruction errors for complex real-world scenes. Instead, we propose to stylize the more robust radiance field representation. We find that the commonly used Gram matrix-based loss tends to produce blurry results without faithful brushstrokes, and introduce a nearest neighbor-based loss that is highly effective at capturing style details while maintaining multi-view consistency. We also propose a novel deferred back-propagation method to enable optimization of memory-intensive radiance fields using style losses defined on full-resolution rendered images. Our extensive evaluation demonstrates that our method outperforms baselines by generating artistic appearance that more closely resembles the style image. Please check our project page for video results and open-source implementations: https://www.cs.cornell.edu/projects/arf/ .
1 Introduction
ARF stylizes a pre-reconstructed radiance field using a single 2D style image, targeting faithful, view-consistent novel views while avoiding blurry style details and enabling full-resolution style losses.
- Approach: ARF transfers artistic features from a single 2D style image to a real-world 3D scene represented as a radiance field.It converts a photorealistic radiance field into a stylized one for novel-view rendering.
- Approach: The method optimizes differentiably rendered views with content and style losses against captured images and the input style image.This formulation stylizes the radiance field across selected viewpoints.
- Style transfer: NNFM loss focuses on local nearest-neighbor VGG features, preserving distinctive style details better than the commonly used Gram-matrix loss.The method also uses VGG content loss and color transfer to balance stylization, content preservation, and color matching.
- Optimization: Deferred back-propagation enables full-resolution image losses by accumulating cached gradients patch-wise, reducing the memory burden of radiance-field optimization.This addresses the mismatch between memory-intensive volumetric rendering and CNN-based losses requiring full-resolution images.
- Results: ARF transfers detailed artistic features across challenging styles and complex scenes, producing better visual quality and consistent user-study preference over baselines.Previous methods often produce over-smoothed or blurry stylized novel views.
2 Related Work
Prior style-transfer methods address artistic appearance in images, videos, and 3D scenes, but ARF focuses on view-consistent stylization of radiance fields using nearest-neighbor feature matching.
- Image style transfer: Optimization-based 2D style transfer commonly matches content features and style statistics, while nearest-neighbor methods instead match features from content and style patches.ARF draws on nearest-neighbor feature matching and introduces it for 3D stylization.
- Image style transfer: Feed-forward style-transfer methods are fast but often reproduce colors and brushstrokes less faithfully than optimization-based techniques.
- Video style transfer: Video style transfer enforces temporal coherence, whereas 3D stylization must synthesize novel views while maintaining consistency across viewpoints.
- 3D style transfer: Earlier 3D style-transfer approaches represent real-world scenes with point clouds or triangle meshes and generate stylized renderings from those representations.
- 3D style transfer: ARF instead reconstructs a radiance field from multiple photos and stylizes it with an NNFM loss to obtain consistent free-viewpoint renderings.
3 Background of Radiance Fields
Neural radiance fields represent scenes as functions mapping 3D locations and viewing directions to density and color, enabling differentiable rendering and novel-view synthesis.
- A radiance field maps any 3D location x and viewing direction d to volume density σ and RGB color c.
- Differentiable volume rendering lets radiance fields fit multiple captured photos and later synthesize photo-realistic views from new viewpoints.
- ARF extends this representation beyond photo-realism by stylizing the radiance field with an exemplar painting or sketch.
4 Stylizing Radiance Fields
ARF stylizes a reconstructed radiance field with local nearest-neighbor feature matching, content preservation, color transfer, and deferred back-propagation for full-resolution image losses.
- 4.1 Style transfer losses: Prior global-statistics style losses may miss local details and produce averaged-out, degraded stylized renderings across views.
- 4.1 Style transfer losses: NNFM matches each rendered VGG feature to its nearest style-image feature, transferring detailed local structures and complex high-frequency visual details across viewpoints.The loss uses cosine distance between rendered features and their nearest neighbors in the style image’s VGG feature space.
- 4.1 Style transfer losses: The final loss combines NNFM with an ℓ2 content-feature penalty, where larger λ preserves more content and smaller λ produces stronger stylization.
- 4.2 Deferred back-propagation: Deferred back-propagation computes full-resolution image losses first, caches pixel gradients, and accumulates scene-parameter gradients through differentiably rendered patches.This reduces memory usage to that of rendering a small differentiable patch while supporting losses that require full-resolution images.
- 4.3 View-consistent color transfer: Color transfer recolors training views before stylization to address mismatches between rendered colors and the style image.The transformation matches the mean and covariance of the recolored image set to those of the style image.
- 4.4 Implementation details: The framework is agnostic to the radiance-field representation and achieves high visual quality with Plenoxels, NeRF, and TensoRF.During stylization, the density component is fixed and only the appearance component is optimized.
5 Experiments
ARF is evaluated on diverse real-world scenes and style exemplars against prior methods, with experiments examining visual quality, user preference, ablations, representation compatibility, and limitations.
- ARF significantly outperforms baseline methods in producing style-faithful renderings while preserving recognizable semantic and geometric scene features.
- Qualitative comparisons: ARF captures both color tone and brushstrokes more faithfully than Huang et al., whose point-cloud reconstruction can lose complex structures such as plant leaves.
- Qualitative comparisons: Compared with Chiang et al., ARF reproduces intricate style details, whereas the baseline mainly transfers overall color tone and can produce blurry results for unseen styles.
- User study: Users preferred ARF over Huang et al. in 86.8% of comparisons and over Chiang et al. in 94.1%.
- Ablations: Ablations show that NNFM gives more faithful style details than Gram and CNNMRF losses, while color transfer improves color-tone matching.
- Applicability: ARF produces high-quality stylization across Plenoxels, NeRF, and TensoRF, and across varied scene and style-image pairs.
- Limitations: The method remains limited by radiance-field geometric artifacts, optimization time of approximately 3 minutes for forward-facing captures and 20 minutes for 360° captures, and lack of manual editing.
6 Conclusion
The conclusion presents ARF as a method for reconstructing artistic radiance fields from photorealistic ones using user-specified style exemplars. It emphasizes faithful stylized novel views and the combination of NNFM with view-consistent color transfer.
- ARF reconstructs artistic radiance fields from photorealistic radiance fields using user-specified style exemplars.
- The resulting fields render high-quality novel views that mimic style-image color tone and details such as brushstrokes.
- The method couples nearest neighbor feature matching with view-consistent color transfer instead of the commonly used Gram loss.
- Evaluations across varied 3D scenes and 2D styles show superior 3D stylization quality over baselines.
Appendix
The appendix describes covariance-based color-transfer computations and reports that RGB transfer tends to outperform HSV, LAB, and histogram matching alternatives.
- The displayed color-transfer expression uses the style and content eigensystems to transform centered content colors and add the style mean.
- The appendix defines covariance matrices for content and style colors through eigen-decompositions involving U_c, Λ_c, U_s, and Λ_s.
- RGB color transfer tends to generate better results than applying the algorithm in HSV or LAB color spaces, or replacing it with histogram matching.