PaperlayerView exact revision on arXiv

Source-linked AI summary

Advances in Neural Rendering

Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, Tomas Simon, Christian Theobalt, Matthias Niessner, Jonathan T. Barron, Gordon Wetzstein, Michael Zollhoefer, Vladislav Golyanik

arXiv:2111.05849v2cs.GRcs.CV

TL;DR

Neural rendering addresses the challenge of synthesizing controllable, photo-realistic imagery from real-world observations when explicit scene parameters are difficult to obtain. This report surveys methods combining classical rendering with learned 3D scene representations, covering static and non-rigid scenes and their resulting applications. It identifies rapid progress across novel-view synthesis, editing, relighting, and avatar generation, while noting unresolved dependencies on viewpoint coverage and substantial computational resources.

  • Problem

    Recovering scene properties for controllable, photo-realistic rendering from real-world observations is challenging, motivating neural methods that learn from images and videos.

  • Method

    The report surveys neural rendering approaches that combine classical rendering principles with learned 3D-consistent scene representations, including neural radiance fields and volumetric rendering.

  • Results

    The reviewed methods enable photo-realistic, parameter-controlled synthesis across rigid and non-rigid scenes, including novel views, shape and material editing, relighting, and human avatar generation.

  • Takeaways & Limitations

    Neural scene representations support applications ranging from free-viewpoint videos to editing, relighting, and human avatar generation.

  • Takeaways & Limitations

    Generalizable volumetric representations remain dependent on large-scale multi-view datasets with sufficient viewpoint coverage, while training neural volumetric representations can consume substantial energy.

Abstract

from arXiv · show

Synthesizing photo-realistic images and videos is at the heart of computer graphics and has been the focus of decades of research. Traditionally, synthetic images of a scene are generated using rendering algorithms such as rasterization or ray tracing, which take specifically defined representations of geometry and material properties as input. Collectively, these inputs define the actual scene and what is rendered, and are referred to as the scene representation (where a scene consists of one or more objects). Example scene representations are triangle meshes with accompanied textures (e.g., created by an artist), point clouds (e.g., from a depth sensor), volumetric grids (e.g., from a CT scan), or implicit surface functions (e.g., truncated signed distance fields). The reconstruction of such a scene representation from observations using differentiable rendering losses is known as inverse graphics or inverse rendering. Neural rendering is closely related, and combines ideas from classical computer graphics and machine learning to create algorithms for synthesizing images from real-world observations. Neural rendering is a leap forward towards the goal of synthesizing photo-realistic image and video content. In recent years, we have seen immense progress in this field through hundreds of publications that show different ways to inject learnable components into the rendering pipeline. This state-of-the-art report on advances in neural rendering focuses on methods that combine classical rendering principles with learned 3D scene representations, often now referred to as neural scene representations. A key advantage of these methods is that they are 3D-consistent by design, enabling applications such as novel viewpoint synthesis of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling non-rigidly deforming objects...

1. Introduction

Neural rendering combines classical computer graphics with learned components to synthesize controllable, photo-realistic imagery from observations. This report emphasizes 3D neural scene representations rendered with analytical graphics processes rather than networks that directly learn the rendering function.

  • Motivation: Traditional rendering generates controllable imagery from explicit scene parameters, but estimating camera, illumination, geometry, and materials from real-world observations is extremely challenging.Physics-based methods can produce photo-realistic images when these parameters are known, while approximate methods reduce parameter requirements at the cost of realism.
  • Motivation: Neural rendering combines classical computer graphics and deep learning to learn compact scene representations and synthesize controllable imagery from existing observations.Supported controls include novel viewpoints, relighting, scene deformation, and compositing.
  • Paradigms: Early neural rendering trained networks to map scene parameters or rendered 2D inputs directly to output images, learning both scene modeling and rendering.This mapping can be viewed as I = M(c), where control parameters c produce an output image I.
  • Paradigms: This report instead focuses on neural networks that represent a particular scene in 3D, which is then rendered by an analytically defined conventional graphics engine.Unlike the earlier paradigm, the network learns to represent a scene rather than to learn the rendering process itself.
  • 3D-consistent rendering: NeRF’s intermediate 3D representation of scene density and radiance enables physics-inspired ray casting and volume integration to render consistent novel views.The report describes this explicit graphics inductive bias as supporting better generalization to novel views.

2. Scope of This STAR

The STAR report surveys advanced neural rendering methods that combine classical rendering with learnable, 3D-consistent scene representations. It concentrates on neural radiance fields and volumetric rendering while excluding mostly screen-space methods and neural super-sampling or denoising.

  • Scope: The report focuses on advanced neural rendering approaches that combine classical rendering with learnable 3D representations and control over scene parameters.The underlying neural representations are 3D-consistent by design.
  • Coverage: It gives a comprehensive overview of scene representations and explains components drawn from classical rendering pipelines and machine learning.The report also details the fundamentals of these components and their design choices.
  • Coverage: The survey concentrates mainly on Neural Radiance Fields and volumetric rendering, covering a broad spectrum of recent techniques.Its selection includes major computer vision and computer graphics conference papers from 2020–2021 and arXiv preprints.
  • Boundaries: The report excludes neural rendering methods that reason mostly in 2D screen space, directing readers to prior work for that literature.It also excludes neural super-sampling and denoising methods for ray-traced imagery.
  • Boundaries: The report aims to provide a panoramic overview and standardized notation rather than claim completeness or replace in-depth study of cited works.The authors recommend consulting the cited papers for detailed treatment.

3. Fundamentals of Neural Rendering

Neural rendering builds on classical camera and rendering concepts while learning 3D scene representations from real-world imagery. These representations distinguish surfaces from volumes and can be implemented with continuous functions, MLPs, or hybrid structures.

  • Neural Rendering: 3D neural rendering learns to represent and render scenes from real-world imagery by mimicking the physical camera-capture process.It separates camera capture from scene representation and rendering.
  • Rendering Foundations: Classical rendering methods range from rasterization to path tracing and volumetric integration, depending on the scene representation.Neural rendering exploits these established rendering methods.
  • Scene Representations: Volumetric representations describe properties throughout space, whereas surface representations store properties relative to object surfaces and cannot directly model volumetric matter.Both types have continuous and discretized forms.
  • Neural Scene Representations: Neural scene representations use neural networks to approximate surface or volumetric representation functions, while optionally storing attributes such as color or view-dependent radiance.Hybrid representations can combine classical points or meshes with MLP-based appearance models.
  • MLPs as Universal Function Approximators: MLPs map spatial coordinates to scene values, and positional encoding partitions the input space to simplify learning higher-frequency signals.The encoding may use fixed or learned basis functions; its frequency range is controlled by p.
  • Surface Representations: Point clouds, meshes, and implicit surfaces provide distinct surface representations, with implicit surfaces defined by a function's zero level-set and supporting unknown or changing topology.Point clouds sample surfaces, meshes use piece-wise linear approximations, and implicit surfaces commonly use signed distance functions.

3.2. Differentiable Image Formation

Differentiable image formation converts 3D scene representations into images through ray casting or rasterization, with camera models defining projection. Representation type determines whether surfaces can be directly projected or must be sampled, while differentiable renderers support optimization.

  • 3.2. Differentiable Image Formation: Rendering converts a 3D scene into a 2D image using ray casting or rasterization, with a defined camera determining the projection.
  • 3.2. Differentiable Image Formation: Explicit surfaces support direct forward rendering, whereas implicit and volumetric representations require sampling 3D space along camera views.Rasterization and point splatting project indexed surfaces; ray marching samples space for representations without explicit surfaces.
  • 3.2. Differentiable Image Formation: Projection is non-injective because of depth ambiguity, but its equations remain suitable for automatic differentiation and end-to-end optimization.
  • 3.2. Differentiable Image Formation: Pinhole projection maps 3D points to image coordinates through camera intrinsics that encode focal lengths, skew, and the principal point.The projection uses homogeneous coordinates and an intrinsic matrix K.
  • 3.2. Differentiable Image Formation: Extrinsic parameters generalize projection to arbitrary camera poses by transforming world coordinates into camera coordinates with rotation and translation.The world-to-camera convention uses R · p_w + t = p_c.
  • 3.2.1. Surface Rendering: Rasterization is faster than ray casting in relevant scaling regimes but represents effects such as lighting, shadows, and reflections less readily.Soft rasterization provides a differentiable implementation.
  • 3.2.1. Surface Rendering: Point-cloud rendering reconstructs a continuous signal from irregular samples, using soft splatting or conventional projection followed by image-space neural reconstruction.Naive point rendering does not provide gradients with respect to point positions, while finite-difference approximations can support geometry and pose optimization.
  • 3.2.1. Surface Rendering: Implicit surface rendering finds ray-surface intersections, typically with sphere tracing, then queries appearance values; volumetric rendering instead models density or occupancy continuously along rays.Continuous volumetric models provide well-behaved gradients for machine-learning optimization.

3.3. Optimization

Neural-network training uses nonlinear, typically gradient-based optimization to fit network weights to training data.

  • Gradient-based optimization, commonly using SGD variants such as Momentum or Adam, adjusts neural-network weights to satisfy training-set constraints.Gradients are obtained through backpropagation.

4. Applications

The report surveys neural-rendering applications spanning static view synthesis, cross-scene generalization, dynamic scenes, editing, relighting, material editing, and engineering frameworks.

  • The applications section covers static novel-view synthesis, cross-object and cross-scene generalization, dynamic scenes, scene editing, relighting, material editing, and engineering frameworks.

4.1. Novel View Synthesis of Static Content

Static novel-view synthesis renders scenes from new camera positions while prioritizing realism and multiview 3D consistency. The surveyed progression moves toward persistent 3D representations, including NeRF-based volumetric fields, while addressing resolution, speed, and aliasing challenges.

  • 4.1. Novel View Synthesis of Static Content: Novel-view synthesis renders a scene from new camera positions using images and camera poses as input.Extensions can also move objects, deform scenes, or change lighting.
  • 4.1. Novel View Synthesis of Static Content: Multiview 3D consistency is a central evaluation criterion alongside image realism, requiring stable content without flickering or warping as the camera moves.
  • 4.1. Novel View Synthesis of Static Content: The field increasingly produces fixed 3D representations that can render new 2D views, improving consistency over heavily black-box 2D image generators.
  • 4.1.1. View Synthesis from a 3D Voxel Grid Representation: Early voxel-grid methods evolved from feed-forward image synthesis toward persistent multiplane or optimized volumetric representations for rendering novel views.The section surveys voxel-grid and volumetric-rendering approaches before neural radiance fields.
  • 4.1.1. View Synthesis from a 3D Voxel Grid Representation: DeepStereo independently predicts each output frame, causing slow runtime and weak multiview consistency, whereas MPI methods retain a 3D representation for view rendering.
  • 4.1.1. View Synthesis from a 3D Voxel Grid Representation: Voxel grids face memory and resolution limits, with cited 3D-CNN approaches unable to exceed 128^3 voxel-grid outputs.
  • 4.1.2. View Synthesis from a Neural Network Representation: NeRF represents each scene with an MLP mapping position and viewing direction to density and color, optimized using pixelwise rendering loss.Volume rendering generates images from the learned field.
  • 4.1.2. View Synthesis from a Neural Network Representation: 5MB of NeRF weights can render 800 × 800 images, compared with close to 2GB for an 800^3 RGBA voxel grid.

4.2. Generalization over Object and Scene Classes

Generalization methods reconstruct novel views across scenes or object instances from limited observations by conditioning neural scene representations locally or globally. The section distinguishes image-based feature aggregation, global scene encodings, and unconditional generative models.

  • Scope: Generalization targets novel-view synthesis from few or single input views across scenes or object instances.The methods are classified by conditioning strategy, generative capability, and underlying 3D representation.
  • Local conditioning: Local-conditioning methods warp or aggregate features from input views, then predict color and density fields for volume rendering.PixelNeRF and related methods concatenate local features with coordinates and learn scene priors from multiple scenes.
  • Local conditioning: PixelNeRF synthesizes DTU novel views without test-time optimization, with training and test sets containing different scenes.This demonstrates generalization beyond memorizing the observed scene.
  • Global conditioning: Global-conditioning methods infer weights or latent codes that encode an entire scene rather than relying on spatially stored input features.Light Field Networks use low-dimensional codes for direct 4D light-field evaluation, while NeRF-VAE supports scene sampling.
  • Generative models: Unconditional generative models sample latent codes and generate scenes through neural scene representations embedded in adversarial frameworks.Later work improves geometric quality and efficiency with surface representations and hybrid image-based refinement.
  • Generative models: CAMPARI jointly learns camera-pose distributions and a generative model, addressing the pose-distribution knowledge required by earlier single-observation methods.Earlier approaches still depended on plausible camera-pose distributions for the target image domain.

4.3. Learning to Represent and Render Non-static Content

Neural radiance fields extend static scene representations to dynamic content through time-conditioned fields or explicit deformation mappings. These approaches support novel-view playback and, in some cases, deformation control, while performance depends on motion complexity, camera trajectories, and supervisory cues.

  • Dynamic representations: Dynamic neural radiance fields support novel-view synthesis of changing scenes, either as playback or with deformation-state control.The report categorizes methods by whether they merely replay time-varying content or permit editing the deformation state.
  • Deformation modeling: Implicit methods condition the radiance field on deformation state, whereas explicit methods warp deformed space into a canonical space.Time inputs and per-time latent codes are common implicit conditioning signals.
  • Playback: Time-varying neural radiance fields enable novel-view video playback without a specific motion model, supporting general objects and scenes.Because these methods forego deformation control, they can handle general content without prescribing motion.
  • Monocular inputs: Monocular non-rigid reconstruction is ill-posed without prior knowledge of object type or 3D shape, motivating geometric regularizers and additional modalities.Methods use scene flow, estimated depth, segmentation, or other cues to stabilize reconstruction.
  • Evaluation and limits: Guo et al.’s method reports the most accurate quantitative and qualitative results on Yoon et al.’s challenging dataset versus Tretschk et al. and Li et al.Its limitations include strong reliance on optical flow and difficulty with arbitrary non-rigid deformations.
  • Monocular deformation: Nerfies uses an as-rigid-as-possible regularizer and coarse-to-fine training to model articulated scenes and small non-rigid deformations from monocular video.The regularizer penalizes deviations from piece-wise rigid configurations, while coarse-to-fine training reduces overfitting to high-frequency details.
  • Controllable dynamics: NR-NeRF uses divergence regularization to stabilize occluded regions and permits novel-view camera trajectories that differ significantly from the inputs.It also demonstrates motion exaggeration and removal of dynamic content, although deformation controllability remains limited.
  • Depth-guided dynamics: TöRF integrates raw time-of-flight measurements, reducing required input views and producing sharper, more accurate dynamic reconstructions than stated baselines.The depth cue improves handling of weakly reflecting regions and restricted sensor depth ranges.

4.4. Compositionality and Editing

Neural scene editing builds on reconstructed 3D representations by separating objects and backgrounds or exposing editable geometry, appearance, and transformations. Methods support object rearrangement, removal, duplication, and temporal edits across image and video settings.

  • Overview: Editing methods extend reconstructed scenes beyond straightforward foreground removal to object rearrangement, affine transformations, structural changes, and appearance edits.The section surveys methods for compositionality and scene editing.
  • Object editing: Conditional NeRF changes rigid-object color and shape from manual 2D edits using a category-trained NeRF whose parameters are adapted to a new instance.The method identifies tunable parameters that propagate edits through the reconstruction.
  • Scene graphs: Neural Scene Graphs decompose monocular driving videos into independently moving objects and background, each represented by separate neural networks.The learned scene graph encodes object transformations and radiances for novel-view rendering and rearrangement.
  • Scene graphs: Neural Scene Graphs support background-foreground decomposition, automotive training-data enrichment, and improved object detection and scene understanding.These applications follow from rendering observed or edited scenes with independently represented objects.
  • Layered editing: ST-NeRF uses layered, space-time-consistent representations to edit independently moving and articulated performers in synchronized multi-view videos.Supported edits include rescaling, shifting, duplication, removal, and temporal rearrangement; the authors identify fewer input views and non-rigid editing as future directions.

4.5. Relighting and Material Editing

Relighting requires scene representations that model light transport rather than only absorption and emission. Neural methods decompose geometry, visibility, lighting, reflectance, and materials to enable relighting or material editing, with different assumptions and computational compromises.

  • Motivation: The absorption-emission volume model supports novel viewpoints but cannot render scenes under different lighting conditions.Relighting requires representations that simulate light transport, including scattering and material effects.
  • Relightable representations: Neural Reflectance Fields represent density, surface normals, and BRDFs, enabling rendering under arbitrary lighting conditions.The approach avoids dense light-ray evaluations by training on objects illuminated by a camera-co-located point light.
  • Visibility: PhySG and NeRD accelerate relighting by assuming full visibility of upper-hemisphere light sources and representing lighting and BRDFs with spherical Gaussians.The full-visibility assumption omits self-occlusion modeling.
  • Visibility: NeRV learns light-source visibility for incoming directions, allowing relightable models to account for significant shadows and self-occlusion effects.Its visibility MLP is queried once per incoming light direction rather than densely along each light ray.
  • Multi-stage methods: NeRFactor converts a pre-trained NeRF into a relightable model by simplifying volumetric geometry into a surface and optimizing visibility, normals, environment lighting, and BRDFs.It uses a multi-stage procedure rather than optimizing the relightable representation from scratch.
  • Multi-stage methods: NeROIC recovers geometry, normals, BRDF properties, and spherical-harmonic lighting from objects captured under multiple unconstrained lighting environments.Latent appearance embeddings explain lighting-driven appearance variation during geometry recovery.
  • Material editing: NeuTex enables intuitive material editing by mapping 3D volume coordinates to 2D texture coordinates, making the recovered texture replaceable or editable.The representation supports texture editing after the scene model has been recovered.
  • Material editing: Ref-NeRF enables material editing through diffuse and specular colors, normals, and roughness, but does not support relighting because incoming light and reflectance remain entangled.It is designed to improve representation of specular surfaces.

4.6. Light Fields

Neural light fields represent scenes by mapping camera rays directly to colors, avoiding ray marching and multi-bounce ray tracing. This efficiency comes with a loss of guaranteed multi-view consistency that requires additional handling.

  • Light-field parameterization: Light Field Networks parameterize a 360-degree light field, while concurrent work models fronto-parallel light fields for forward-facing novel-view synthesis.A light field maps each oriented ray directly to its observed color.
  • Rendering efficiency: Directly sampling the corresponding camera ray yields a pixel color without ray marching, and light fields absorb multi-bounce reflections.This avoids discovering surface intersections before rendering.
  • Limitations: Unlike 3D renderers, neural light fields can assign different colors to rays that intersect the same scene point, losing guaranteed multi-view consistency.Additional means are needed to address this inconsistency.
  • Generalization: Latent-code conditioning supports cross-scene generalization but constrains Light Field Networks to simple scenes, whereas transformer-based fields synthesize views of complex real-world scenes from few observations.The transformer approach parameterizes 360-degree light fields inferred from image observations.

4.7. Engineering Frameworks

Neural rendering requires infrastructure for large-scale, non-sequential data movement, complex differentiation, rendering operations, and long-running experiment management. Existing storage, autodifferentiation, rendering, and configuration frameworks address these needs at different levels.

  • Engineering challenges: Neural rendering processes large image and video datasets non-sequentially and differentiates large, complex computational graphs, creating notable engineering challenges.Efficient operators may require low-level languages that complicate automatic differentiation.
  • Data handling: Pixel-wise random access across entire image or video datasets makes GPU saturation and flexible storage important, especially for temporal reconstruction.A single batch may sample across an entire sequence.
  • Data handling: AIStore provides throughput monitoring and tiered loading, while sharding enables in-memory shuffling with mostly sequential reads within shards.TensorFlow supports sharded storage through TFRecord.
  • Experiment management: Hydra manages complex configurations and supports hyperparameter search through frameworks such as AX, but exhaustive sweeps can remain costly when experiments run until convergence.Bayesian search can select parameters intelligently without eliminating long runtimes.
  • Differentiation: Neural rendering uses both macro and micro autodifferentiation across Python- and CUDA-level workflows, motivating libraries such as STAN, Difftaichi, Enoki, and Jax.The appropriate abstraction depends on whether applications process large vectorized inputs or many small inputs.
  • Rendering primitives: OptiX, Teg, Redner, and Mitsuba 2 provide ray-casting, ray-intersection, differentiable integration, or differentiable rendering primitives.OptiX supports hardware-accelerated ray intersection on NVIDIA RTX hardware.

5. Open Challenges

Open challenges span integration, interpretability, scalability, generalizability, multimodal learning, quality, training time, and sparse-view reconstruction. Progress includes early large-scene, cross-scene, multimodal, and efficiency efforts, but important scope and data dependencies remain.

  • Seamless Integration and Usage: Neural rendering still has a gap between methods operating on classical meshes or point clouds and methods applicable to neural representations.This limits seamless integration with established graphics algorithms and techniques.
  • Interpretability: Neural network weights remain difficult to interpret in terms of target quantities such as 3D point colors and opacities, complicating controllability.Classical graphics pipelines instead rely on analytically derived steps.
  • Scalability: Most volumetric neural-rendering work focuses on single objects or simple composite scenes, while large-scale scenes with partial observations remain challenging.Large environments also make global representations infeasible beyond some scene size, and editing or relighting methods do not extend straightforwardly.
  • Scalability: iMAP demonstrates real-time single-room reconstruction from a handheld RGB-D camera and can fill occluded regions, but large-scale neural scene learning retains many open challenges.The figure marks keyframes in red.
  • Generalizability: Generalizable volumetric methods depend on large-scale multi-view datasets with sufficiently wide baselines and viewpoint coverage for arbitrary novel viewpoints.StereoNeRF uses a dozen spread-out views, while pixelNeRF infers unseen rigid scenes from a single image.
  • Generalizability: Generalization to non-rigid scenes remains an open challenge and may require substantially larger deformable-scene datasets or disentangled deformation and appearance models.Sparse spatiotemporal observations and single-image animation are identified as extreme settings.
  • Multi-Modal Learning: Multimodal learning remains open despite prior work incorporating semantics, text, sound, camera characteristics, IMUs, lidar, and event streams.Telepresence and augmented reality could benefit from jointly rendered views and synthesized sounds.
  • Other Questions: An integral-implicit layer reduced NeRF training computation by approximately 150x FLOPs per epoch, while high-frequency detail, view-dependent appearance, training time, and sparse observations remain challenging.Partial observations tend to produce blurrier regions than fully observed areas.

6. Social Implications

Neural rendering offers highly realistic novel views with potential benefits for research, industry, and content creation, while remaining immature and carrying risks involving misuse, consent, energy consumption, and access.

  • Research and Industry: Neural volumetric representations affect computer vision, computer graphics, and augmented or virtual reality through increased photo-realism and accessible research resources.Well-understood principles, public codebases, and datasets lower barriers to photogrammetry and 3D-reconstruction research.
  • Research and Industry: Rendering photo-realistic novel views from a few images could reduce content-creation effort and reshape established visual-effects workflows.The report identifies this as a potential advantage over existing technology.
  • Trustworthiness: Photo-realism enables malicious actors to present synthetic content as real, particularly for human-face applications, motivating detection methods and further user studies.The discussion also raises labeling synthetic content in different contexts.
  • Trustworthiness: Creating synthetic content from a person’s identity should involve explicit, informed consent for a specified context.This is presented as a collection-side response to potential misuse.
  • Environment: GPU energy use and manufacturing for neural volumetric training can negatively affect the environment and climate, while high hardware demand may limit equal participation.Architectures requiring less training compute are identified as one mitigation direction.

7. Conclusion

The report reviews neural rendering methods that learn 3D scene representations from 2D observations for controllable, photo-realistic synthesis across diverse applications. It also identifies open research challenges and discusses social implications of increasingly accessible photo-realistic content.

  • Neural rendering methods learn 3D scene representations from 2D observations and enable controllable synthesis of photo-realistic imagery.
  • Applications span free-viewpoint videos of rigid and non-rigid scenes, shape and material editing, relighting, and human avatar generation.
  • The field remains emerging, with multiple open challenges identified as directions for future research.
  • The report discusses social implications arising from neural rendering’s democratization and its ability to synthesize photo-realistic image content.
Loading 2111.05849v2…