Stylizing 3D Scene via Implicit Representation and HyperNetwork

1. Introduction

• The goal is to stylize complex scenes from example scene images and a reference style image, supporting virtual- and augmented-reality applications such as virtual tours.The paper describes virtual tours switching between real-world scenes and artistic styles.Source: §1 · p. 1
• Prior texture-mapping methods target single objects, while point-cloud stylization requires ground-truth or proxy geometry and has limited resolution.The paper turns to implicit NeRF representations to transfer arbitrary styles to complex scenes.Source: §1 · pp. 1–2
• Combining novel-view synthesis with image or video stylization can produce inconsistent views; NeRF stylization also faces appearance controllability and memory challenges.Rendering a 67 × 81 patch for losses and back-propagation can take 17,934 MB.Source: §1 · p. 2, Figure 2 · p. 2
• A NeRF with geometry and appearance branches is reconstructed first; then a hypernetwork predicts appearance parameters from the reference style image.Patch sub-sampling helps train the hypernetwork with content and stylization losses while alleviating GPU memory demands.Source: §1 · p. 2
• The method renders consistent novel views with arbitrary styles, and evaluations favor it over baselines for cross-view stylization consistency.The hypernetwork is optimized for a specific scene, then can render arbitrary and unseen styles.Source: §1 · p. 2

2.1. Novel View Synthesis

• Neural radiance fields encode scenes in an MLP and predict color and density from 3D coordinates and view directions, without proxy-geometry preprocessing.Other approaches use explicit meshes, voxels, or proxy geometry; some require many source images or ground-truth 3D supervision.Source: §2.1 · p. 2

2.2. Image and Video Style Transfer

• Image stylization methods support arbitrary styles, while video methods address temporal consistency; neither category is designed for consistency across scene viewpoints.Video approaches use temporal losses, feature alignment, regularization, multi-channel correlation, or bilateral learning.Source: §2.2 · p. 3

2.3. Texture Transfer

• 3D texture-transfer and representation-based stylization methods rely on explicit geometry and are limited to objects rather than complex scenes.Texture mapping can look unnatural across object boundaries, and prior 3D methods may not support high-quality stylized images.Source: §2.3 · p. 3

3. Proposed Method

• The framework represents a scene with NeRF and uses a hypernetwork to transfer arbitrary style from a reference image for consistent novel views.Geometric training learns the scene representation before stylization training predicts appearance-branch parameters from the style image.Source: §3 · p. 3

3.1. Preliminaries

• NeRF maps sampled 3D points and viewing directions to volume density and view-dependent color; NeRF++ represents unbounded, complex scenes with separate foreground and background volumes.The foreground and background are separated by a unit sphere, with an inverted sphere parameterization for the outer volume.Source: §3.1 · pp. 3–4

3.2. Stylizing Implicit Representations

• The method preserves scene geometry by fixing the geometry branch and using a style-VAE and hypernetwork to modify the appearance branch from a style-image latent vector.The geometry branch comprises F_base and F_geo, while F_app predicts color values.Source: §3.2 · p. 4

3.3. Model Training

• Geometric training optimizes the NeRF scene-function MLPs with rendered-ray color MSE against the corresponding ground-truth pixels.Each iteration samples pixels and their camera rays to obtain 3D points for volume rendering.Source: §3.3 · p. 5
• Stylization training freezes F_base and F_geo, trains the hypernetwork to predict F_app weights, and uses paired-patch content and style losses to limit rendering and back-propagation memory demands.Random image windows are cropped and sparsely sampled; content compares patch features, while style compares feature statistics with the reference image.Source: §3.3 · p. 5

4.1. Qualitative Results

• Image stylization followed by NVS blurs results because independently stylized training images are inconsistent; direct NVS followed by stylization also yields inconsistent views.WCT can fail to preserve scene content, while TPFR can fail to transfer the reference style.Source: §4.1 · p. 6, Figure 5 · p. 6
• The proposed method matches the reference style and remains consistent across novel views, including between short- and long-range viewpoints.Video-stylization baselines can accumulate errors between far-away viewpoints.Source: §4.1 · p. 6

4.2. Quantitative Results

• The proposed method is preferred over image- and video-stylization baselines for style match and consistency in a 73-user study.Video stylization is less flickery than image stylization, but does not maintain consistency between far-away views.Source: §4.2 · p. 6, Figure 6 · p. 7
• Across adjacent and far-away views, the proposed method maintains consistency, while video-stylization baselines lose far-view consistency through error accumulation.Table 1 evaluates adjacent views; Table 2 evaluates far-away views.Source: §4.2 · p. 7, Table 2 · p. 7, Table 1 · p. 7

4.3. Limitations

• Stylization quality is limited by the backbone NeRF: details such as trees can remain blurry when the backbone fails to capture them.The figure also shows clear details in the original and stylized wheels.Source: §4.3 · p. 7, Figure 7 · p. 7

• The proposed NeRF model uses a hypernetwork to predict appearance-related parameters for styling complex 3D scenes from reference images.Source: §5 · p. 7
• A two-stage training strategy and patch sub-sampling algorithm are developed to learn the hypernetwork.Source: §5 · p. 7
• Qualitative comparisons report consistent stylization across view angles, while Figure 9 presents stylized outputs at various novel views.Source: Figure 8 · p. 8, Figure 9 · p. 8