Source-linked AI summary
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
Wenzheng Chen, Jun Gao, Huan Ling, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, Sanja Fidler
TL;DR
Rendering-based learning is hindered by non-differentiable rasterization, limiting gradient-based access to image formation and informative scene properties. DIB-R replaces this bottleneck with analytical, image-wide gradients from local interpolation and global aggregation, enabling 2D-supervised 3D prediction and textured-shape generation. It achieves numerical and visual state-of-the-art results, while shininess prediction remains inaccurate in the reported experiments.
Problem
Non-differentiable rasterization limits gradient-based learning from 2D images and restricts differentiation with respect to informative properties such as texture and lighting.
Method
DIB-R models rasterization as foreground interpolation and background distance-based aggregation, producing analytical gradients for image pixels and scene attributes.
Results
DIB-R achieves numerical and visual state-of-the-art results in 3D shape and texture recovery and is used to generate 3D textured shapes from 2D supervision.
Takeaways & Limitations
Differentiable rendering allows neural networks to learn shape, texture, and lighting from single images and to learn generators of 3D textured shapes.
Takeaways & Limitations
The model cannot accurately predict shininess constants in the reported experiment, with texture maps compensating for the shininess effect.
Abstract
from arXiv · showhide
Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: https://nv-tlabs.github.io/DIB-R/
1 Introduction
DIB-R addresses the difficulty of differentiating through rasterization by modeling it as local interpolation and global aggregation. The framework supports analytical image-wide gradients and achieves state-of-the-art results in 3D shape and texture recovery applications.
- Motivation: Machine perception receives 2D projections of 3D geometry interacting with light, motivating models that can disentangle geometry from lighting.Such disentanglement is described as important for invariance and robustness.
- Motivation: Existing rasterization methods use approximate gradients and often cannot differentiate through informative properties such as textures and lighting.These limitations can affect performance, rendering fidelity, and the informativeness of learning signals.
- DIB-R: DIB-R computes analytical gradients over the entire image by combining local interpolation for foreground pixels with global aggregation for background pixels.The design supports gradients with respect to vertex attributes and camera positions, alongside several lighting models.
- Applications: Wrapped around a neural network, DIB-R predicts properties of an initial polygon sphere and achieves numerical and visual state-of-the-art results in 3D shape and texture recovery.The framework is showcased across challenging machine learning applications.
2 Related work
Prior differentiable rasterizers either approximate gradients or use soft assignments with computational and fidelity trade-offs. DIB-R follows rasterization-based work while targeting analytical gradients and broader scene-property differentiation for 2D-supervised 3D learning.
- Differentiable Rasterization: OpenDR approximates gradients near mesh-face edges, while other approaches hand-design approximate gradients for face movement across pixels.OpenDR uses first-order Taylor approximation and automatic differentiation.
- Differentiable Rasterization: SoftRas-Mesh analytically differentiates probabilistic pixel assignments to all mesh faces, but incurs higher computational cost.SoftRas-Color extends this formulation theoretically to vertex colors, texture, and lighting.
- Adversarial 3D Object Generation: Earlier adversarial 3D shape-generation methods require full 3D supervision, whereas differentiable rendering enables learning object distributions from only 2D supervision.The paper presents a generator for both shape and texture.
3 DIB-R: Differentiable Interpolation-based Renderer
DIB-R makes rasterization differentiable by interpolating local foreground attributes and aggregating global geometry for background pixels. It supports differentiable rendering across multiple mesh attributes and lighting models, including vertex colors, textures, normals, lighting, and materials.
- Differentiable Rasterization: Foreground pixels are assigned to their closest covering face and rendered by interpolating that face’s vertex attributes.Each covered pixel is influenced exclusively by the closest face after z-buffering.
- Differentiable Rasterization: Background pixels softly assign influence to all faces according to projected distance, allowing gradients to reach occluded geometry and constrain shape learning.The resulting alpha aggregation passes gradients to all faces, including those ignored during foreground rasterization because of occlusion.
- Differentiable Rasterization: DIB-R computes analytical gradients across the entire image by combining local interpolation for foreground rasterization with global aggregation for background rasterization.Foreground pixels use interpolated vertex attributes, while background pixels receive distance-based gradient signals from all faces.
- Rendering Models: The renderer back-propagates through vertex positions, colors, textures, texture coordinates, normals, light directions, camera position, and material properties.These attributes support vertex-color, texture, Lambertian, Spherical Harmonic, and Phong rendering models.
- Lighting Models: DIB-R unifies lighting models by interpolating light-related attributes before applying model-specific illumination in the fragment shader.The framework supports Phong, Lambertian, and Spherical Harmonics models with different combinations of color and lighting factors.
- Optimization Results: Optimization sanity checks cover vertex position, color, texture, texture coordinates, camera position, lighting, and material across the supported rendering models.The experiments optimize L-1 loss between target images and rendered predictions; among rasterization-based renderers, the authors report support for all vertex attributes.
4 Applications of DIB-R
DIB-R is applied to single-image 3D prediction and 3D textured-shape generation using rendered images and 2D supervision. The applications predict geometry, appearance, and lighting, using image-based and adversarial objectives.
- 4.1 Predicting 3D Objects from Single Images: For single-image prediction, a convolutional network predicts mesh vertex positions and colors from an RGBA image, then renders the mesh for comparison with the input.
- 4.1 Predicting 3D Objects from Single Images: Training separates silhouette and color supervision, using Intersection-Over-Union for silhouettes and L-1 loss for colored images.
- 4.1 Predicting 3D Objects from Single Images: A random second-view rendering, alongside the known camera view, prevents the network from focusing only on mesh properties visible from one perspective.
- Geometry, Texture, and Light: The extended model predicts vertex positions, a texture map, and lighting information, while retaining the previous losses and optionally using adversarial, perceptual, and feature-matching losses.
- 4.2 3D GAN of Textured Shapes via 2D supervision: For textured-shape generation, a GAN predicts vertex positions and a texture map from noise and is trained with image and texture discriminators using only 2D supervision.
5 Experiments
Experiments evaluate DIB-R on single-image 3D prediction, texture and lighting recovery, real images, and 3D textured-shape generation. Across these settings, the method generally achieves strong quantitative or qualitative results, while revealing difficulty separating shininess from texture.
- 5.2 Predicting 3D Objects from Single Images: Geometry, Texture and Light: DIB-R achieves about 40% lower rendered-image L-1 texture error and 60% smaller lighting-direction angle difference than N3MR.The comparison evaluates texture, texture plus lighting, and lighting predictions using the metrics defined in Table 2.
- 5.3 Texture and Lighting Results With Adversarial Loss: With adversarial loss, the model disentangles texture and light well, produces accurate 3D shape, and generates crisper textures than the earlier experiment.The experiment includes Phong and Spherical Harmonic lighting models.
- 5.3 Texture and Lighting Results With Adversarial Loss: The separation study finds that the model cannot accurately predict shininess, causing the texture map to compensate for its effect.This limitation may reflect the weak significance of shininess under 2D supervision.
- 5.4 Real Images: On real CUB bird images, DIB-R improves shape and key-point predictions and produces higher-fidelity, more realistic textures than CMR, while both methods show poor artifacts on car images.The authors attribute the texture improvement to predicting a whole image as the texture map rather than using restricted face colors.
- 5.5 3D GAN of Textured Shapes via 2D Supervision: The textured 3D GAN produces high-quality and diverse car shape-and-texture samples, and interpolations between latent codes demonstrate a robust learned distribution.Training uses rendered views and 2D supervision, with four primary views concatenated for the image discriminator.
6 Conclusion
DIB-R is a complete rasterization-based differentiable renderer with analytically computable gradients. Wrapped around neural networks, it predicts shape, texture, and light from single images and learns generators of 3D textured shapes.
- DIB-R computes rasterization gradients analytically.
- Wrapped around a neural network, DIB-R predicts shape, texture, and light from single images.
- DIB-R also learns a generator of 3D textured shapes.