PaperlayerView exact revision on arXiv

Source-linked AI summary

State of the Art on Neural Rendering

Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer

arXiv:2004.03805v1cs.CVcs.GR

TL;DR

Automatically creating complete scene representations remains difficult, while existing physical models and image-based methods have limitations. This report surveys neural rendering approaches that combine computer graphics with learning-based components, their applications, benefits, limitations, social implications, and open problems.

  • Problem

    Automatically generating scene geometry, materials, lighting, and animations remains difficult, while explicit reconstruction can be hard, error-prone, and artifact-prone.

  • Method

    The report defines and classifies neural rendering approaches that combine classical computer graphics pipelines with learning-based primitives for controllable image generation.

  • Results

    The report covers neural rendering applications including novel view synthesis, photo manipulation, reenactment, relighting, free-viewpoint video, and photo-realistic avatars, while summarizing benefits and limitations.

  • Takeaways & Limitations

    Neural rendering has progressed on view-dependent effects and sparse-observation shape and appearance learning, but reflections, use-case specificity, and training-data dependence remain challenges.

  • Takeaways & Limitations

    Learned generators using pixelwise p-norm losses tend to produce blurry or averaged outputs, including desaturated colorization and missing super-resolution details.

Abstract

from arXiv · show

Efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo-realistic images from hand-crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo-realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state-of-the-art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. This state-of-the-art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.

1. Introduction

Neural rendering addresses the costly manual construction of scene representations by combining learned generative models with computer-graphics knowledge for controllable image synthesis. The report classifies these approaches, examines their controls and pipelines, and surveys applications and social implications.

  • Manual creation of geometry, materials, lighting, and animations remains a major bottleneck in producing photo-realistic virtual worlds.
  • Neural rendering combines deep generative models with physical computer-graphics knowledge to learn mappings from captured images to novel images.The approach can integrate mathematical models such as projection with learned components.
  • The report defines and classifies neural-rendering approaches according to the types and provision of control, learned pipeline components, and synthesis properties.It discusses explicit versus implicit control, generalization, and stochastic versus deterministic synthesis.
  • Applications include novel-view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and photo-realistic avatars.The avatar applications target virtual and augmented-reality telepresence.
  • The report also discusses social implications and the detectability of synthetic content, especially for images depicting humans.

2. Related Surveys and Course Notes

The report situates neural rendering alongside established surveys of deep generative models and prior work on controllable image synthesis using computer graphics and vision.

  • Existing literature includes broad surveys and course notes on deep generative models, including specialized reports on GANs and VAEs.
  • Controllable image synthesis using classical computer-graphics and vision techniques has also been studied extensively.

3. Scope of this STAR

This state-of-the-art report focuses on neural-rendering methods that combine classical graphics pipelines with learnable components to generate controllable photo-realistic imagery.

  • The report examines where and how machine learning can improve classical rendering pipelines and what training data these methods require.
  • It introduces fundamentals of computer graphics and machine learning before surveying the benefits and limitations of current hybrid techniques.
  • The scope centers on machine-learning techniques whose primary goal is controllable photo-realistic image generation.
  • The report excludes work focused primarily on geometric and 3D deep learning for reconstruction or scene understanding.
  • It also excludes machine-learning methods used specifically to denoise ray-traced imagery.

4. Theoretical Fundamentals

The fundamentals section contrasts physical rendering and image-based methods with learned generative approaches, then introduces scene representations, light transport, camera models, and conditional generators.

  • Classical graphics and image-based rendering: Classical rendering models geometry, materials, lights, and cameras, whereas machine learning learns image distributions from real-world examples.
  • Scene representations and cameras: Scene geometry can use explicit primitives such as triangles or points, or implicit functions such as signed distance functions whose zero-crossings define surfaces.
  • Rendering and light transport: Rendering transforms cameras, lights, geometry, and materials into simulated images through rasterization or raytracing.Rasterization transforms geometry into the image domain, while raytracing casts rays from pixels and can simulate recursive reflections and refractions.
  • Classical graphics and image-based rendering: Image-based rendering creates novel images by warping and compositing captured views, commonly using proxy geometry for static-object novel-view synthesis.
  • Classical graphics and image-based rendering: Deep generative models learn image priors from large-scale collections, extending beyond the small image sets historically used in image-based rendering.
  • Learned image synthesis: Conditional GANs learn mappings from an observed input and random vector to an output image, with the discriminator judging whether the input-output pair is real.Inputs may be categorical labels or images such as semantic maps, depending on the application.

D LcGAN(G,D) . (3)

Conditional GAN training pits a generator against a discriminator to produce outputs conditioned on input data, but paired supervision is costly and unpaired mappings require additional constraints.

  • Early conditional GAN implementations omitted noise because the network tended to ignore it, whereas latent vectors later enabled multimodal synthesis.Per-pixel ℓ1 and perceptual-distance losses were also used to stabilize training.
  • The discriminator distinguishes real from synthetic images while the generator learns to fool it during adversarial training.pix2pix uses a U-Net generator and a patch-based fully convolutional discriminator.
  • Perceptual-distance and conditional-GAN objectives both use an auxiliary network to define a learning objective for improving the generator.The auxiliary network is either a feature network or a discriminator.
  • Learning the stated objectives can require hundreds to millions of paired training examples, including expensive pixel-level annotations.Such paired data are difficult to collect for many real-world image-synthesis applications.
  • Without paired examples, source and target sets specify domains but not the correct target for each input, leaving infinitely many possible mappings.Additional constraints such as cycle consistency have been proposed to restrict the mapping.

5. Neural Rendering

Neural rendering combines classical graphics knowledge with learned generative models to produce controllable, high-quality images under specified scene conditions. The survey organizes methods by control, graphics integration, explicitness, modality, and generality.

  • Neural rendering combines classical graphics ideas with learning-based primitives to make image synthesis easier and outputs more controllable.These ideas can appear in input features, scene representations, and network architectures.
  • The survey proposes a taxonomy organized around control, computer-graphics modules, explicit or implicit control, multimodal synthesis, and generality.These axes describe what is controlled, how graphics knowledge is integrated, and how broadly a method applies.
  • Current methods address specific subproblems including novel-view synthesis, relighting, and face or body animation under new expressions and poses.Rendering high-quality images under arbitrary user-specified scene conditions remains an open problem.
  • Graphics knowledge can be integrated through non-differentiable modules that transform rendering into image-to-image translation.Conditional generative models with skip connections can then address the transformed task.
  • Differentiable graphics modules can implement renderers, 3D rotations, or illumination models within the neural-rendering pipeline.This represents a deeper integration of graphics knowledge than directly mapping scene parameters to images.
  • Methods differ in whether users manipulate interpretable scene parameters directly or provide examples that implicitly specify the desired output.Explicit controls include camera viewpoint, illumination, facial pose, and expression.
  • Multimodal systems can generate visibly different output choices from the same controlled input, such as different images from one semantic map.This lets users select among alternatives when only part of the scene is specified.
  • Instance-specific methods often produce higher-quality results but require lengthy retraining for each object, making their training times prohibitive for real-world applications.Improving general models remains an open research direction.

6. Applications of Neural Rendering

The survey covers neural-rendering applications for semantic synthesis and manipulation, and characterizes systems by their data, inputs, outputs, controllable parameters, graphics modules, generality, and multimodal behavior. Semantic manipulation remains difficult because input reconstruction and edit compatibility are challenging.

  • Neural rendering applications include semantic photo manipulation, novel-view synthesis, relighting, free-viewpoint video, and facial and body reenactment.Table 1 summarizes attributes of selected methods discussed in the survey.
  • The survey records required data, network inputs, and network outputs to distinguish what must be acquired, fed into the learned system, and produced by it.Derived data are excluded from the minimal required-data description.
  • Its application taxonomy also records contents, controllable parameters, explicit control, graphics-module type, generality, and multimodal synthesis.The listed parameters include camera, pose, lighting, color, texture, semantics, expression, and speech.
  • Semantic Photo Synthesis: Conditional GANs map user-specified semantic layouts to photorealistic images, while related systems also support color, sketch, and texture inputs.Pix2pix and Karacan et al. were early learning-based methods for semantic image synthesis.
  • Semantic Photo Synthesis: GauGAN uses spatially adaptive normalization to preserve semantic information and a random latent vector to control image style.Its design addresses semantic information being washed away in earlier normalization-based conditional models.
  • Semantic Image Manipulation: Semantic manipulation of an input photograph remains challenging because the generator must reconstruct the image accurately and keep edited content compatible with it.iGAN addresses these issues by optimizing a low-dimensional latent vector to reproduce the input photo.
  • Semantic Image Manipulation: iGAN transfers edits from the generated image to the original photograph using guided image filtering, while Neural Photo Editing blends modified content with original pixels.These systems support semantic editing of user-provided images rather than only synthesis from layouts.
  • Deep generative models create visual content from user controls, whereas classical rendering produces photorealistic images when supplied with high-quality scene specifications.Neural rendering seeks to combine these strengths for controllable synthesis of novel images.

6.2. Novel View Synthesis for Objects and Scenes

Neural rendering approaches synthesize novel views from sparse observations by combining learned models with image-based, geometric, or implicit scene representations. They improve handling of view-dependent effects and sparse inputs, but remain use-case- and data-limited.

  • Problem: Novel view synthesis generates new camera perspectives from fixed images, requiring inference of 3D structure and completion of occluded or unseen regions.Classical image-based rendering can produce ghosting and holes when observations are sparse or viewpoints expose uncovered scene regions.
  • Neural image-based rendering: Neural image-based rendering replaces or augments hand-crafted image-based-rendering components with learned components for novel-view generation.Classical pipelines reproject captured image content using proxy geometry and blend the projections in the target image domain.
  • Neural rerendering: Neural rerendering translates rendered deep buffers into realistic scene views without using input views at runtime, recovering details with a deep network.Neural Rerendering in the Wild reconstructs dense colored point clouds from internet photos and conditions synthesis on depth, color, and appearance information.
  • Multiplane images: Six sparse, wide-baseline views can support synthesis of photo-realistic real-scene views containing high-frequency effects such as shadows and specularities.Xu et al. capture scenes under point illumination and train with synthetic supervision on geometry and appearance.
  • Neural scene representations: Neural scene representations range from unstructured embeddings to 3D-structured voxels, point clouds, multiplane images, and implicit functions.These representations provide different inductive biases for image formation and geometry; GQN models uncertainty from incomplete observations.

6.3. Free Viewpoint Videos

Free-viewpoint video systems capture performers with multiple cameras and reconstruct view-dependent appearance for rendering from arbitrary viewpoints. Neural methods improve realism and efficiency, but capture quality, hardware requirements, and scene materials remain limiting factors.

  • Free-viewpoint capture: Free-viewpoint video, or volumetric performance capture, uses multi-camera setups to acquire performers’ 3D shape and texture.Earlier systems commonly lacked high-frequency and view-dependent effects, causing limited relighting quality and blurred textures.
  • Relightables: The Relightables system combines image-based relighting with high-speed depth sensing to produce relightable volumetric videos with realistic free-viewpoint capture.Its setup uses 58 12.4MP RGB cameras and 32 12.4MP active IR sensors, interleaving two lighting conditions during capture.
  • Limitations: Transparent and translucent objects remain difficult for current free-viewpoint capture systems to handle.Other persistent challenges include temporal consistency for hair and translucent materials.
  • Neural rerendering: LookinGood adds neural rerendering to real-time performance capture, addressing artifacts from missing depth, low-resolution texture, and oversmooth geometry.The framework uses witness cameras alongside a volumetric performance-capture system.
  • Neural Volumes: Neural Volumes encodes multi-view video frames into a latent code decoded as a semi-transparent RGB-opacity volume rendered by differentiable ray marching.The renderer accumulates color and opacity to produce an output image and alpha matte.
  • Limitations and accessibility: Multi-view images at training and test time are a key factor in free-viewpoint-system performance, but the required capture technology is not consumer-accessible.Parallel work seeks to remove infrastructure requirements using deep learning and consumer hardware.

6.4. Learning to Relight

Learning-based relighting combines sparse image inputs with neural models to synthesize scenes under novel illumination. Methods address directional lights, environment maps, facial reflectance, and uncontrolled portrait capture.

  • Sparse-sample relighting: Relighting methods learn to synthesize novel illumination from far fewer captures than traditional image-based relighting.Traditional methods may require tens to hundreds of images, whereas recent neural approaches can operate from only a few inputs.
  • Sparse-sample relighting: Five images captured under learned optimal light directions can produce relit scenes under arbitrary directional illumination.The method can reproduce complex high-frequency effects, including specularities and cast shadows, when applied to real scenes.
  • Outdoor-scene relighting: Outdoor-scene relighting uses multiple naturally illuminated views and coarse reconstructed geometry to render novel sun positions and cloudiness levels.The geometry is used to construct intermediate buffers rather than being directly relit.
  • Facial relighting: Deep Reflectance Fields relight human faces in arbitrary environments from two images recorded under spherical color-gradient illumination.The model learns facial reflectance from 4D reflectance-field data spanning several subjects, expressions, and viewpoints.
  • Portrait relighting: Portrait relighting networks transform a single RGB image captured under uncontrolled lighting into a version illuminated by a user-specified environment map.This targets in-the-wild photographs captured with standard or cellphone cameras.

6.5. Facial Reenactment

Facial reenactment transfers pose, expressions, eye motion, speech, or other facial properties using reconstructed representations and neural rendering. Approaches range from person-specific systems to generalized models with limited explicit 3D control.

  • Overview: Facial reenactment modifies facial properties such as head pose, expressions, and speech beyond viewpoint and lighting changes.Classical approaches commonly reconstruct and edit a 3D face model, while neural methods better handle inaccurate reconstruction and tracking.
  • Video reenactment: Deep Video Portraits transfers head pose, facial expressions, and eye motion from a reference video to a target portrait video.The system represents identity, expression, pose, eye motion, and illumination using a low-dimensional semantic representation derived from 3D face reconstructions.
  • Video reenactment: Text-based Talking-head Video Editing changes a speaker’s transcript and synthesizes a video of the revised speech, including words absent from the input.The pipeline combines a low-fidelity render with a person-specific GAN-based neural renderer that adds details and fills holes.
  • Scene-aware reenactment: Deferred Neural Rendering supports novel-view synthesis and 3D scene editing, including geometry deformation, removal, and copy-move operations.It uses a scene-specific neural texture containing learned feature descriptors for surface points alongside a coarse tracked mesh.
  • Generalized reenactment: Generalized face reenactment controls different identities from sparse 2D keypoints and uses few-shot learning to refine the model for unseen identities.Unlike person-specific systems, it does not provide explicit 3D control of parameters such as pose and expression.
  • Deep Appearance Models: Deep Appearance Models use a viewpoint-conditioned variational autoencoder to model facial geometry and appearance for high-resolution viewpoint-dependent rendering.The learned face model is also animated from cameras mounted inside a virtual-reality headset.

6.6. Body Reenactment

Body reenactment methods control a person’s position, rotation, and pose in generated images or videos, often by transferring motion from a source performance. Neural representations can avoid explicit geometry but remain largely person-specific and imperfect.

  • Pose-guided generation: Neural pose-guided generation controls a person’s position, rotation, and body pose in a target image or video.The cited systems cover both image and video generation.
  • Performance cloning: Full-body performance cloning transfers motion from a source video to a target video, commonly using several minutes of person-specific training footage.Training data often depicts the target person against a static background.
  • Geometry-free representations: A common-texture representation maps multiple 2D views into a global 2D space for re-rendering arbitrary subject viewpoints from desired 3D poses.The system also predicts a body foreground mask and can be driven by new skeleton input images.
  • Geometry-free representations: The body-reenactment pipeline removes the need for explicit geometry at training and test time by learning common texture correspondences.A pretrained DensePose model initializes the texture map, while training improves correspondences and recovers high-frequency details.
  • Limitations: Existing human-performance-cloning methods remain person-specific, do not easily generalize to unseen scales, and produce artifacts such as incoherent detail or misalignment.The survey states that none of the existing methods achieves artifact-free results.

7. Open Challenges

The survey identifies generalization, scalability, editability, and multimodal inputs as open challenges for neural rendering. Progress has been substantial, but broader, more controllable, and more scalable systems remain needed.

  • Generalization: Many neural rendering methods overfit small image sets or single subjects, objects, or scenes, limiting generalization beyond the training distribution.Unseen poses are one reported failure case, and covering all scene-parameter combinations may be infeasible because of dimensionality.
  • Generalization: Additional calibration images or memory banks are proposed as test-time information that could improve generalization.The survey presents these as directions for future work rather than established solutions.
  • Scalability: Scalability remains constrained by methods designed for individual faces, objects, or small environments rather than complex, cluttered, large-scale scenes.Target applications include dynamic crowds and city- or global-scale scenes.
  • Scalability: Compositional scene reasoning could support scalability by segmenting scenes into objects, learning local coordinate systems, and processing observations efficiently.The survey notes that compositionality alone is insufficient; improved architectures and unsupervised learning are also needed.
  • Editability: Neural rendering does not always provide the flexible, intuitive editing available in traditional graphics pipelines, especially for learned feature-based parameters.The survey also emphasizes understanding and reasoning about network outputs and failure cases.
  • Multimodal representations: Using visual and audio inputs together may let neural scene representations exploit information from additional modalities.The report notes audio-based lip synchronization as an existing example.

8. Social Implications

Neural rendering can make image manipulation more accessible, increasing both its legitimate utility and potential for misuse. The report emphasizes synthetic-content detection, clear disclosure, and permission from content owners or performers.

  • Neural rendering lowers the barrier to image and video manipulation for non-experts with limited resources.
  • Synthetic imagery can be used nefariously even when manipulation methods have legitimate applications.
  • The report argues that synthesized images and videos should clearly identify themselves as synthetic.
  • Altered videos should be shared only after obtaining permission from content owners or performers.
  • Automatic passive detection of synthetic or manipulated imagery is increasingly important for verifying digital-content integrity.

9. Conclusion

The report surveys neural rendering across a broad range of applications and concludes that the field is expanding rapidly. It highlights previously intractable applications and the potential to broaden access to complex photo and video editing.

  • Neural rendering spans novel-view synthesis, semantic image editing, free-viewpoint video, relighting, face and body reenactment, and digital avatars.
  • The field has enabled previously intractable applications, including digital-avatar rendering without manual modeling.
  • Neural rendering may make complex photo and video editing tasks accessible to a broader audience.
  • The report aims to introduce neural rendering to a wider research community and support development of future graphics applications.
Loading 2004.03805v1…