Source-linked AI summary
RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, Yong-Liang Yang
TL;DR
Traditional rendering pipelines are difficult to differentiate because discrete visibility operations impede inverse rendering, while prior differentiable methods remain restricted in occlusion handling and rendering scope. RenderNet introduces a CNN with a learned projection unit that renders voxelized 3D shapes and encodes visibility and shading. Experiments show that it supports multiple shaders and rendering and inverse-rendering applications, with limitations from voxel-focused inputs and blurry outputs under mean squared error training.
Problem
Discrete visibility operations make traditional rendering non-differentiable, while prior differentiable methods remain limited in occlusion handling, rendering styles, input geometries, or output formats.
Method
RenderNet is an end-to-end CNN renderer with a learned projection unit that approximates visibility computation and performs shading from 3D voxel grids.
Results
RenderNet successfully performs rendering and inverse rendering, learns different shaders with one architecture, and supports tasks including shape, pose, lighting, and texture estimation.
Takeaways & Limitations
The architecture provides a common learned renderer for varied appearances and can be integrated into texturing and image-based reconstruction modules.
Takeaways & Limitations
The method focuses on voxel data, while mean squared error training tends to produce blurry results, especially for ambient-occlusion shaders.
Abstract
from arXiv · showhide
Traditional computer graphics rendering pipeline is designed for procedurally generating 2D quality images from 3D shapes with high performance. The non-differentiability due to discrete operations such as visibility computation makes it hard to explicitly correlate rendering parameters and the resulting image, posing a significant challenge for inverse rendering tasks. Recent work on differentiable rendering achieves differentiability either by designing surrogate gradients for non-differentiable operations or via an approximate but differentiable renderer. These methods, however, are still limited when it comes to handling occlusion, and restricted to particular rendering effects. We present RenderNet, a differentiable rendering convolutional network with a novel projection unit that can render 2D images from 3D shapes. Spatial occlusion and shading calculation are automatically encoded in the network. Our experiments show that RenderNet can successfully learn to implement different shaders, and can be used in inverse rendering tasks to estimate shape, pose, lighting and texture from a single image.
1 Introduction
RenderNet addresses the non-differentiability and limited scope of prior differentiable-rendering methods with a CNN architecture that learns visibility, shading, and multiple rendering styles from voxelized 3D shapes.
- Motivation: Traditional rasterization and ray tracing use discrete visibility operations, making it difficult to correlate rendering parameters with images for inverse rendering.Prior differentiable methods use approximate renderers or surrogate gradients but remain limited in occlusion handling, rendering styles, input geometries, or output formats.
- Approach: RenderNet combines CNNs with 3D-world inductive biases through a learned projection unit that approximates visibility computation end to end.The architecture includes object visibility computation and pixel-color calculation, while using deep features rather than low-level primitives.
- Representation: Voxel grids provide the paper’s input representation, enabling high-quality renderings even from low-resolution and noisy inputs.The paper focuses on voxel data and leaves polygon meshes and unstructured point clouds as future extensions.
- Results: RenderNet learns multiple rendering styles with one architecture, spanning Phong, suggestive contour, composite contour-cartoon, and ambient-occlusion shaders.The paper notes that some of these styles are time-consuming and computationally expensive to define explicitly.
- Contributions: The model supports rendering and inverse rendering, generalizes to unseen categories and complex geometry, and can produce textured images from textured voxel grids.The authors also describe integration with texturing and image-based reconstruction modules.
2 Related work
Related work spans image-based rendering, geometry-based differentiable rendering, and image-based shape reconstruction. RenderNet is positioned as a geometry-grounded alternative for rendering effects and inverse-graphics tasks that existing methods handle only partially.
- Image-based rendering: Image-based rendering methods learn images from vectors, attributes, semantic maps, sketches, pixel attributes, or disentangled image representations.Related methods also address novel-view synthesis using category-specific shape priors and optical flow for occlusion and disocclusion.
- Image-based rendering: The paper argues that geometry-based methods may perform better for out-of-plane rotation, image relighting, and shape texturing because they impose stronger 3D assumptions.This is presented as a motivation for using geometry-based rendering rather than as a universal claim about all tasks.
- Geometry-based rendering: OpenDR provides general differentiable rendering but is more strenuous to integrate into neural networks and machine-learning frameworks.The comparison concerns integration difficulty rather than rendering quality.
- Geometry-based rendering: Kato et al. approximate rasterization gradients, but the method is limited to rasterization-based rendering and struggles to represent effects such as global illumination, reflection, and refraction.These effects are typically associated with ray tracing in the paper’s discussion.
3 Model
RenderNet transforms voxelized 3D shapes into 2D images using canonical camera processing, 3D convolutions, a learned projection unit, and 2D convolutions for shading. The design explicitly supplies simpler geometric operations while learning visibility and appearance formation.
- Pipeline: The rendering pipeline transforms world-space voxel data into camera space, samples it trilinearly, and processes it with a CNN and projection unit.The camera is assumed axis-aligned and looks along the negative z-axis of the volumetric grid.
- Architecture: RenderNet uses 3D convolutions, a projection unit for visibility and 3D-to-2D feature projection, and 2D convolutions for shading.Training uses a pixel-space loss between target and generated images.
- Camera transformation: Rigid-body transformation parameterizes azimuth, elevation, and distance R, while embedding the voxel grid prevents rotated objects from being cut off.The transformation keeps the camera in a canonical pose relative to the voxel grid.
- Projection unit: The projection unit reshapes the 4D voxel tensor by collapsing depth with feature channels, then applies an MLP to learn projection and visibility along depth.The resulting squeezed tensor has D · C channels, and the projection unit outputs K channels.
- Conditional rendering: RenderNet can model conditional rendering with extra parameters such as lights or spatially varying texture.The paper represents this as a conditional renderer p(I | V, h).
- Shading: Phong shading combines a light direction, surface normals, and an ambient constant, then multiplies the shading by an albedo map to produce the final image.The paper uses this example to demonstrate extensibility through rendered normal and albedo maps.
4 Experiments
Experiments evaluate RenderNet across rendering, generalization, architecture comparison, and single-image reconstruction tasks. The model learns multiple shaders, handles degraded or unseen inputs, and explicitly reconstructs scene factors for inverse rendering.
- Learning to render: RenderNet learns Phong, contour, cartoon, and ambient-occlusion shaders with one architecture, using separate datasets and unseen test shapes.PSNR scores for these shaders are reported in Figure 5.
- Generalization: RenderNet generalizes from chairs to unseen categories, multiple objects, corrupted inputs, and 50% downsampled volumetric data.It produces plausible or smooth high-resolution renderings despite noise and reduced input resolution.
- Texture rendering: Texture rendering is implemented by mapping a texture vector to a 3D representation and concatenating it with the voxelized shape before rendering.The texture vector contains PCA coefficients for BaselFace albedo generation.
- Architecture comparison: Compared with encoder-decoder baselines, RenderNet renders a normal map first and combines it with lighting, while the alternatives directly predict shaded images.The EC baseline uses fully connected and up-convolution layers; EC-Deep adds residual blocks.
- Architecture comparison: The alternative encoder-decoder produces lower PSNR and misses object details, while RenderNet generalizes better to unseen shape categories and scenes.The reported comparison concerns the Phong-shaded chair dataset and Figure 5’s generalization task.
- Image-based reconstruction: Single-image reconstruction estimates shape, pose, lighting, and texture by minimizing image loss, with pretrained shape and texture decoders regularizing the optimization.The method uses RenderNet as the image-formation model and latent representations for shape and texture.
- Image-based reconstruction: 42.99 ± 0.64 IOU is reported for reconstructed face-dataset voxel grids at a 95% confidence interval.The same experiment is also performed on the chair dataset.
5 Discussion and conclusion
RenderNet is a convolutional differentiable rendering network that supports rendering and inverse rendering in a geometrically grounded framework. Its shared architecture can express multiple shaders, while the authors identify extensions for broader training and synthesis settings.
- RenderNet performs rendering and inverse rendering using an end-to-end convolutional network trained with pixel-space regression.
- The network can share its 3D convolutions and projection unit across different shaders instead of requiring separate networks.
- The paper suggests future extensions including unlabelled training, alternative losses, multi-scale architectures, and style-transfer losses.
- A geometrically grounded approach is intended to improve generated-image performance and fidelity for novel-view synthesis and fine-grained editing tasks.
Appendices
The appendix describes the image-based reconstruction setup, including its loss variables, pose initialization, and iterative optimization procedure. The reconstruction estimates shape, pose, lighting, and texture-related variables from an observed image.
- The reconstruction loss uses an observed image, a latent 3D shape representation, pose, lighting, and texture variables.The latent shape vector z′ is 200-dimensional in this experiment.
- Pose initialization subdivides azimuth and elevation grids over [0–180] degrees before refining around the current best parameters.
- Multiple shape-and-lighting initializations are optimized with gradient descent to reduce the risk of local minima.
- Parameters are reinitialized to the current best solution every 200 steps, and optimization continues until convergence at around 1800 steps.
A.2 Chair reconstruction from a single image
Chair reconstruction estimates 3D shape and pose from a single image using an optimization procedure and a learned shape prior. The task is challenging because chair poses and geometries vary substantially, but the recovered factors support downstream image synthesis and relighting.
- Chair reconstruction compares RenderNet with DC-IGN using grayscale shaded ShapeNet chair images because the original comparison dataset was unavailable.
- The reconstruction optimizes pose and shape variables with a loss containing shape-prior terms, using α = 5 and β = 1.The latent shape vector z′ is 250-dimensional in this experiment.
- Chair reconstruction is harder than face reconstruction because pose spans [0–360] azimuth and [0–180] elevation, while thin parts may exceed the shape prior’s captured variation.
- The reported results recover chair pose and shape, supporting sharper novel-view synthesis and image relighting.
- Removing the learned shape prior causes reconstruction to fail to generate good results.
B Network Architecture
The network architecture uses residual blocks in both 3D and 2D convolutional stages. Each block combines convolutions and nonlinear activation with a shortcut connection.
- The architecture applies PReLU activations throughout except for the final sigmoid layer and uses dropout with probability 0.5 after most convolutional layers.
- 3D residual blocks contain two 3×3×3 convolutions, a PReLU activation, and a shortcut adding the block input to the second convolution’s output.
- 2D residual blocks follow the same structure while replacing 3D convolutions with 2D convolutions.
B.1 RenderNet
RenderNet uses a projection unit to connect 3D convolutional features with 2D rendering, while sharing high-level information such as object visibility across shading modalities.
- Architecture: The projection unit reshapes 3D encoder features before 2D processing, linking volumetric input to image synthesis.It resizes W×H×32×16 features to W×H×(32·16) before a 1×1 convolution.
- Shared representation: Different modalities can share high-level information such as object visibility while differing in pixel appearance through shading.This design supports one model with shared high-level processing and modality-specific low-level layers.
B.2 Alternative architecture
The alternative EC and EC-deep models encode voxel inputs into latent vectors, concatenate lighting and pose information, and decode the result into images and textures.
- Encoder: EC and EC-deep encode voxel inputs with four strided 3D convolutions, followed by a fully connected 200-dimensional latent representation.Both encoders use channels {64, 128, 256, 512}, 4×4 kernels, and strides {2, 2, 2, 2}.
- Latent conditioning: EC concatenates lighting and pose directly with the shape latent, whereas EC-deep first maps each through a fully connected layer.The resulting representation is passed through fully connected layers before reshaping to an 8×8×512 tensor.
- Image decoder: The image decoder uses alternating strided and non-strided 2D convolutions, while EC-deep replaces non-strided convolutions with residual blocks.EC-deep therefore differs from EC primarily in the depth of its non-strided decoding stages.
- Texture decoder: The texture decoder reshapes a fully connected output into a 32×32×32×4 tensor and applies three 3D convolutions to produce 64×64×64×4 output.Its convolutional stages use channels {4, 8, 4} and strides {1, 2, 1}.
C Limitations
RenderNet is limited by blurry outputs from its training loss, input voxel-grid resolution, and loss of thin or sharp geometric features during voxel-based rendering.
- Image quality: Mean squared error, or binary cross-entropy for grayscale images, tends to produce blurry results, especially for ambient occlusion.The authors identify adversarial loss as possible future work to address this issue.
- Input representation: Input voxel-grid resolution constrains the method’s scalability, although cropped training grids and fully convolutional inference allow larger-grid processing.The authors suggest octrees or alternative formats such as unstructured point clouds for further scalability.
- Geometric detail: RenderNet over-smooths sharp diagonal shapes and fails to render extremely thin features that a mesh renderer can handle.The authors attribute the thin-feature failure primarily to voxelization losing those features in the input grid.