Source-linked AI summary
StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning
Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, Lin Gao
TL;DR
3D scene stylization must produce style-following images from arbitrary novel views while preserving cross-view consistency, a challenge complicated by the gap between 2D style examples and NeRF’s volumetric representation. StylizedNeRF combines a 2D stylization network with NeRF through mutual learning and conditional latent codes, and outperforms existing methods in visual quality and consistency.
Problem
3D scene stylization seeks stylized images from arbitrary novel views with consistent rendering across views, but 2D style examples and NeRF representations have a significant domain gap.
Method
StylizedNeRF uses mutual learning between a 2D stylization network and NeRF, with consistency and mimic losses plus style-conditioned learnable latent codes.
Results
StylizedNeRF outperforms existing and state-of-the-art methods in visual quality and 3D or long-range consistency.
Takeaways & Limitations
The framework combines the stylization capability of 2D methods with the spatial consistency of NeRF for conditional 3D scene stylization.
Takeaways & Limitations
Voxelized COLMAP Structure from Motion reconstruction lacks fine geometry and precision, which damages stylization results.
Abstract
from arXiv · showhide
3D scene stylization aims at generating stylized images of the scene from arbitrary novel views following a given set of style examples, while ensuring consistency when rendered from different views. Directly applying methods for image or video stylization to 3D scenes cannot achieve such consistency. Thanks to recently proposed neural radiance fields (NeRF), we are able to represent a 3D scene in a consistent way. Consistent 3D scene stylization can be effectively achieved by stylizing the corresponding NeRF. However, there is a significant domain gap between style examples which are 2D images and NeRF which is an implicit volumetric representation. To address this problem, we propose a novel mutual learning framework for 3D scene stylization that combines a 2D image stylization network and NeRF to fuse the stylization ability of 2D stylization network with the 3D consistency of NeRF. We first pre-train a standard NeRF of the 3D scene to be stylized and replace its color prediction module with a style network to obtain a stylized NeRF. It is followed by distilling the prior knowledge of spatial consistency from NeRF to the 2D stylization network through an introduced consistency loss. We also introduce a mimic loss to supervise the mutual learning of the NeRF style module and fine-tune the 2D stylization decoder. In order to further make our model handle ambiguities of 2D stylization results, we introduce learnable latent codes that obey the probability distributions conditioned on the style. They are attached to training samples as conditional inputs to better learn the style module in our novel stylized NeRF. Experimental results demonstrate that our method is superior to existing approaches in both visual quality and long-range consistency.
1. Introduction
StylizedNeRF targets consistent stylization of complex 3D scenes from arbitrary novel views by combining NeRF’s spatial consistency with 2D stylization capability. The method addresses small-patch training limits, inconsistent 2D outputs, and ambiguity in stylization results through mutual learning and conditional latent codes.
- 3D scene stylization seeks arbitrary novel-view images that follow style examples while remaining consistent across views.
- Small-patch training of a stylized NeRF with style and content losses produces poor stylization because rendering requires hundreds of samples per pixel.A single RTX 2080Ti is limited to 32×32 training patches.
- Inconsistent 2D stylized images can cause blurriness when used to train a NeRF.
- The framework mutually trains a stylized NeRF and a 2D stylization method, using NeRF’s geometric consistency to supervise the 2D network.An ordinary NeRF is first trained for the scene’s opacity field, then its color module is replaced by a style module.
- Conditional latent codes model ambiguous 2D stylization results as style-conditioned distributions for robust conditional stylization.Style features are encoded into latent distributions with a pretrained VAE, and a minus log-likelihood constrains the codes.
- StylizedNeRF reportedly outperforms existing methods in visual quality and 3D consistency.
2. Related Work
Related work spans novel-view synthesis, implicit neural scene representations, image and video stylization, and earlier 3D stylization methods. The paper positions its approach against methods that lack spatial consistency or rely on discrete scene representations.
- Novel View Synthesis: Novel-view synthesis includes light fields, multi-plane images, and continual neural representations for rendering views from photographs.
- Novel View Synthesis: NeRF represents scenes as continuous volumetric fields, while extensions address octrees, unbounded scenes, reflectance decomposition, and uncontrolled real-world images.
- Style Transfer Methods: Image stylization evolved from optimization-based methods to feed-forward approaches such as AdaIN, WCT, PhotoWCT, and WCT2.
- Video Stylization: Video stylization commonly uses optical flow, temporal constraints, cross-domain feature alignment, or dynamically adjusted channel distributions to improve frame coherence.
- 3D Scene Stylization: Existing 2D methods lack spatial consistency and 3D scene perception, while prior 3D methods use point clouds or patch-trained NeRFs with representation and complexity limitations.
3. Preliminaries
NeRF models a scene as a continuous volumetric field whose opacity and view-dependent radiance are predicted from spatial coordinates and viewing directions. StylizedNeRF replaces color prediction with a style module while retaining NeRF-based volumetric rendering and uses VGG-derived style features embedded into latent distributions.
- NeRF: NeRF uses MLPs to map 3D position x and viewing direction d to opacity σ(x) and radiance color c(x,d).
- NeRF: A camera ray r(t) = o + td is cast from camera center o along direction d, and pixel color is determined by volumetric integration.
- NeRF: Positional encoding γ(·) maps NeRF inputs to Fourier features containing multiple frequency scales.
- StylizedNeRF Architecture: The stylized architecture renders views with a fixed pretrained NeRF, predicts stylized radiance from coordinates and latent codes, and composites samples using original opacity.
- StylizedNeRF Architecture: VGG style features are embedded into latent distributions through a pretrained VAE before conditioning the style module.
4. Method
The method combines a stylized NeRF with a 2D AdaIN stylization network through mutual learning. NeRF-derived consistency, mimic supervision, and style-conditioned latent codes jointly guide stylized scene rendering.
- 4.1. 2D Stylization Network: AdaIN uses a VGG encoder, adaptive instance normalization, and CNN decoder; only its decoder is learnable during training.The decoder aligns content features with style statistics before generating stylized outputs.
- 4.2. Stylized NeRF: NeRF replaces its original color module with a style module that predicts stylized radiance using 3D coordinates and latent codes.The pretrained opacity field remains part of the stylized NeRF and supplies the scene representation.
- 4.2. Stylized NeRF: Style-conditioned latent codes are initialized from VAE-embedded Gaussian distributions, optimized during learning, and replaced by distribution means at inference.A minus log-likelihood loss constrains each code to its style-conditioned distribution, supporting clustering and generalization.
- 4.3. Mutual Learning: Mutual learning first distills NeRF spatial consistency into the 2D decoder, then jointly trains the NeRF style module, decoder, and latent codes.Training views are rendered by the ordinary NeRF, while stylized NeRF images use the original opacity field and numerical quadrature.
- 4.3. Mutual Learning: The mimic loss minimizes L2 distance between stylized NeRF outputs and 2D stylization outputs while exchanging their complementary knowledge.Content and style losses are computed from decoder outputs, allowing larger patches within limited GPU memory.
5. Experiments
Experiments compare StylizedNeRF with video- and NeRF-based stylization methods using qualitative results, consistency measurements, user preferences, and ablations. The method preserves geometry and stylization quality while achieving stronger consistency, with learnable codes and mutual decoder training improving outputs.
- Qualitative Results: Video stylization methods show long-range color inconsistencies and geometric errors in novel-view videos, whereas the proposed results remain visually consistent.The comparison covers MCCNet and ReReVST on LLFF and Tanks and Temples scenes.
- Qualitative Results: The proposed method preserves fine geometric details better than LSNV and Chiang et al., whose geometry proxies or patch-based losses lose structures and introduce artifacts.Examples include irregular walls, slender poles, thin chains, truck cracks, and other fine-level shapes.
- Quantitative Results: Our method outperforms other methods by a significant margin in both short-range and long-range consistency measured with warping error.Short-range uses view gaps of 5, while long-range uses gaps of 35; lower warping error is better.
- Quantitative Results: User-study preferences favor the proposed method in both stylization quality and consistency quality.Participants evaluated ten series of views from Tanks and Temples scenes against other state-of-the-art methods.
- Ablation Study: Ablations show that learnable latent codes, mutual decoder training, and the distribution loss reduce artifacts, blur, and inconsistency in stylized outputs.The distribution loss clusters latent codes around the pretrained distribution mean, while stylized NeRF outputs retain stricter consistency than 2D outputs.
6. Conclusion
StylizedNeRF combines 2D stylization with NeRF-based spatial consistency through mutual learning and conditional latent codes. It outperforms state-of-the-art methods in visual quality and consistency.
- StylizedNeRF combines a 2D stylization method with NeRF's spatial consistency through mutual learning.
- A consistency loss transfers NeRF's spatial consistency prior to the 2D network, while a mimic loss aligns 2D and stylized-NeRF outputs.
- Learnable latent codes model inconsistent 2D stylization results with style-conditioned distributions for conditional stylization.
- StylizedNeRF outperforms state-of-the-art methods in visual quality and consistency.