Source-linked AI summary
Deep Shading: Convolutional Neural Networks for Screen-Space Shading
Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, Tobias Ritschel
TL;DR
The paper addresses whether CNNs can synthesize screen-space appearance from per-pixel scene attributes rather than infer attributes from appearance. It introduces Deep Shading, which learns shading from example images using a U-shaped CNN. The resulting system models multiple effects at competitive quality and speed, while retaining screen-space limitations and exhibiting artifacts when complex mappings are not fully captured.
Problem
Screen-space shading traditionally relies on manually programmed or physically derived simulations, motivating a learned mapping from per-pixel scene attributes to appearance.
Method
Deep Shading uses a U-shaped CNN trained on example images to convert deferred shading attributes into RGB appearance and combine screen-space effects.
Results
Deep Shading models ambient occlusion, indirect light, scattering, depth-of-field, motion blur, anti-aliasing, and combinations of these at competitive quality and speed.
Takeaways & Limitations
Complex screen-space shading can be learned from data rather than programmed by human experts or derived entirely from first-principles light transport.
Takeaways & Limitations
Deep Shading inherits screen-space shading’s inability to represent shading from objects missing because of occlusion, clipping, or culling.
Abstract
from arXiv · showhide
In computer vision, convolutional neural networks (CNNs) have recently achieved new levels of performance for several inverse problems where RGB pixel appearance is mapped to attributes such as positions, normals or reflectance. In computer graphics, screen-space shading has recently increased the visual quality in interactive image synthesis, where per-pixel attributes such as positions, normals or reflectance of a virtual 3D scene are converted into RGB pixel appearance, enabling effects like ambient occlusion, indirect light, scattering, depth-of-field, motion blur, or anti-aliasing. In this paper we consider the diagonal problem: synthesizing appearance from given per-pixel attributes using a CNN. The resulting Deep Shading simulates various screen-space effects at competitive quality and speed while not being programmed by human experts but learned from example images.
1 Introduction
Deep Shading reverses the usual computer-vision direction by using CNNs to synthesize RGB appearance from per-pixel scene attributes. It learns screen-space shading effects from examples, achieving quality and performance comparable to or better than human-written shaders.
- The approach targets screen-space effects including ambient occlusion, indirect light, scattering, depth-of-field, motion blur, and anti-aliasing.
- Deep Shading maps per-pixel attributes such as positions, normals, and reflectance to RGB appearance using a deep CNN.
- Learned shaders can achieve quality and performance similar to or better than human-written shaders using only example data.
2 Previous Work
Previous work spans graphics methods that convert scene attributes into appearance and vision methods that infer attributes from appearance. Deep Shading learns the full attributes-to-appearance mapping, aiming to replace manually designed screen-space shaders and costly physical simulation with data-driven inference.
- Computer-graphics methods convert scene attributes into appearance, while computer-vision methods generally convert image appearance into attributes.
- Screen-space shading efficiently computes many effects from deferred pixel-attribute buffers but cannot account for surfaces absent from the image.
- Earlier learning-based graphics approaches often remained tied to specific scenes or learned filter parameters rather than complete shading.
- Pyramidal CNNs provide fast, large filters capable of modeling long-range effects such as distant shadows or strong depth-of-field.
- Deep Shading learns entire shading outputs from example data rather than manually combining convolutions or deriving a simulation from physical principles.
- The approach uses one learned CNN to combine previously separate screen-space effects into a single learned shader.
3 Background
The paper frames shading as supervised learning of a complex function from deferred shading buffers to reference appearance. CNNs provide the nonlinear, spatially organized approximation, while downsampling and upsampling support efficient high-resolution per-pixel output.
- Supervised learning fits an approximation ˜f to examples of an unknown function f, with a loss measuring differences between predicted and reference outputs.
- Training examples can be generated in arbitrary quantity using path tracing or another sufficiently powerful image-synthesis algorithm.
- Neural networks represent nonlinear approximations through layered units whose learned weights determine their affine combinations and activations.
- Weights are optimized with loss gradients computed by backpropagation, typically using stochastic gradient descent and mini-batches.
- CNN layers connect spatially local neighborhoods across feature-map slices, with shared weights creating regular spatial processing.
- Downsampling improves efficiency and richer feature extraction, while de-convolutional upsampling restores resolution for fast per-pixel shading.
4 Deep Shading
Deep Shading is trained on paired deferred shading buffers and rendered reference images, then processes rich geometric, material, lighting, and motion inputs with a U-shaped CNN. Its architecture preserves fine detail while adapting capacity across effects.
- 4.1 Data Generation: The dataset contains 61,000 paired deferred shading buffers and reference images, split into 54,000 training, 6,000 validation, and 1,000 testing pairs.
- 4.1 Data Generation: Training and validation use 10 scenes, whereas testing uses 4 different scenes not seen during training or validation.
- 4.1 Data Generation: Training images are augmented through rotations and horizontal and vertical flips, with view-space attributes transformed consistently.
- 4.1 Data Generation: Deferred shading buffers provide per-pixel geometry, material, lighting, and motion information, while reference images provide RGB shaded appearance.
- 4.1 Data Generation: Direct light is supplied as a network input rather than computed by the network itself.
- 4.2 Network: The U-shaped network downsamples through a left branch and upsamples through a right branch across up to 6 levels, from 512×512 to 16×16 resolution.
- 4.2 Network: Skip connections from corresponding down-branch steps preserve fine spatial details during upsampling.
- 4.2 Network: The architecture is shared across effects, while the number of kernels and levels varies according to each effect’s requirements.
5 Results
Deep Shaders reproduce several screen-space effects from per-pixel attributes, with results spanning ambient occlusion, lighting, filtering, depth-of-field, scattering, motion blur, and combined shading. Evaluation reports competitive visual quality, while also exposing effect-specific limitations such as weakened occlusion, blurrier details, and constrained glossiness.
- Evaluation: The results evaluate learned Deep Shaders across different effects using test-set SSIM, network structure, and execution time on 768×512 px inputs.Table 1 reports SSIM against raw network outputs; mono-network timing includes simultaneous execution of three networks.
- Ambient Occlusion: Ambient occlusion reproduces darkening near geometry, with HBAO achieving only marginally higher SSIM in a same-time comparison.The paper describes the learned AO shader as numerically and visually on-par with HBAO.
- Directional Occlusion: Directional occlusion is more difficult than ambient occlusion because indirect shadows vary in color and occur only for particular occlusion directions.The paper also reports weakened occlusion where directional-occlusion shadows create higher-frequency patterns that are harder to encode.
- Diffuse Indirect Light: The learned indirect-light shader brightens shadowed regions with colors resembling nearby lit objects rather than leaving them pitch-black.The method uses monochromatic processing because diffuse light transport is independent across color channels under the stated assumptions.
- Anti-aliasing: The anti-aliasing shader smooths jagged edges while preserving properly sampled interior textures and blurring mainly along depth discontinuities.Its reference is 8× MSAA, which has access to substantially more information.
- Depth-of-field: Depth-of-field increases blur with distance from the focal plane, while textured-object training helps preserve sharp in-focus regions.In the example, blossoms are sharper than background leaves.
- Full Shading: The full shader combines image-based lighting, ambient occlusion, and shallow depth-of-field in one network, producing diffuse and moderately glossy shading with subtle focus effects.Image-based lighting still shows a slight color shift and an upper bound on glossiness, while directional occlusion is harder because colored shadows depend on occlusion direction.
6 Analysis
The analysis examines visual artifacts, resolution and field-of-view changes, network capacity, loss functions, training-data diversity, and regression baselines. It identifies practical settings that balance quality, generalization, and runtime while documenting important limitations.
- Visual Analysis: Deep Shading can produce blur, color shifts, ringing, background darkening, and attribute discontinuities when the learned mapping is incomplete or overly complex.The authors describe high-frequency capture as a key challenge and characterize over-blurring as a possible graceful degradation.
- Resolution and Camera Parameters: Scaling the attributes that determine spatial effect size compensates for resolution changes, preserving similar effects without changing runtime.For a resolution larger by factor N, the effect radius is divided by N through corresponding attribute scaling.
- Resolution and Camera Parameters: FOV has limited influence on quality: the DSSIM curve spans only .12 to .18, and the paper reports only a minimal SSIM drop away from 50◦.The evaluation uses 50◦ as the central reference FOV.
- Network Structure: 3×3 kernels achieve similar test loss to 5×5 kernels while running about twice as fast, making 3×3 the preferred spatial choice for the given training set.The relative timing pattern also holds for networks with u0 = 4 and u0 = 16.
- Network Structure: Reducing the initial kernel count to u0 = 4 loses expressiveness, whereas u0 = 16 is only slightly better than u0 = 8 but costs more than 6× the compute time.The larger network also incurs increased memory use from parameters and intermediate representations; the selected configuration is u0 = 8 with 3×3 kernels.
- Training Data and Regression Comparisons: Training-data diversity matters for generalization: using five scenes reduces DSSIM error by 5% versus one, while ten scenes adds only about another 1% advantage.The comparison uses DO training sets with equal total size and a common test set; random forests lose on both visual quality and runtime relative to Deep Shading.
7 Conclusion
Deep Shading demonstrates CNN-based complex shading learned from data rather than derived from first-principles light transport. It supports multiple screen-space effects, but remains bounded by screen-space visibility and currently matches rather than exceeds human-written performance.
- Deep Shading performs complex shading with CNNs by learning from data instead of using light transport simulation derived from first principles.
- CNNs can model individual screen-space effects and arbitrary combinations at competitive quality and speed.
- Screen-space visibility limits Deep Shading because objects excluded by occlusion, clipping, or culling cannot contribute shading.
- Multiple-effect Deep Shading currently performs on par with, but does not exceed, human-written code.