Source-linked AI summary
Neural 3D Mesh Renderer
Hiroharu Kato, Yoshitaka Ushiku, Tatsuya Harada
TL;DR
Modeling 3D structure from 2D images with polygon meshes is difficult because rasterization blocks back-propagation. The paper introduces an approximate mesh-rendering gradient, then demonstrates reconstruction and 3D editing with 2D supervision, while noting topology limitations.
Problem
Rasterization is discrete, so mesh rendering cannot straightforwardly be integrated into neural networks despite the usefulness of polygon meshes for 3D modeling.
Method
The paper proposes an approximate gradient for mesh rendering that enables neural-network training and supports gradients to mesh geometry and rendered appearance.
Results
The renderer supports single-image mesh reconstruction with silhouette supervision and gradient-based 3D editing, including style transfer and DeepDream; reconstruction outperforms the voxel-based approach.
Takeaways & Limitations
Integrating a mesh renderer into neural networks enables 2D-supervised reconstruction and image-driven editing of 3D mesh shape and texture.
Takeaways & Limitations
The reconstruction method cannot generate objects with various topologies because the faces-to-vertices relationship is fixed rather than generated dynamically.
Abstract
from arXiv · showhide
For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.
1. Introduction
The paper presents Neural Renderer, an approximate gradient for mesh rasterization that enables neural-network training with rendering. It applies this capability to single-image mesh reconstruction and gradient-based 3D mesh editing using 2D supervision.
- 3D representation: Polygon meshes offer compactness, geometric transformations, and surfaces, making them a promising representation for modeling 3D worlds.Meshes can represent large triangles with few vertices and faces, unlike voxel or point-based sampling.
- Challenge: Rasterization is a discrete rendering operation that prevents back-propagation, making neural integration of mesh rendering challenging.Vertex projection is differentiable, but regular-grid sampling through rasterization blocks gradients.
- Proposed method: Neural Renderer uses an approximate rendering gradient to support end-to-end neural-network training and propagate gradients into shapes, textures, lighting, and cameras.The method is designed specifically for neural networks.
- Applications: Single-image 3D mesh reconstruction uses silhouette image supervision without 3D supervision and outperforms the voxel-based approach.The paper reports advantages over the voxel-based approach as a major contribution.
- Applications: Gradient-based 3D mesh editing enables 2D-to-3D style transfer and 3D DeepDream with 2D supervision.These applications require smooth mesh surfaces and therefore cannot be realized with voxels or point clouds in the stated setting.
2. Related work
The related work contrasts rasterized and geometric 3D representations, reviews mesh integration into neural networks, and situates the paper's reconstruction and image-editing applications.
- 3D representations: Rasterized representations such as voxels are CNN-friendly, but voxel memory efficiency is poor; geometric forms require adaptations for neural processing.Point clouds require permutation invariance, while primitive-based representations model objects with sets of primitives.
- 3D representations: Polygon meshes represent objects with vertices and surfaces, but their graph structure makes generative neural-network tasks difficult.Meshes are memory efficient, support geometric transformations, and provide surfaces for textures and lighting.
- Mesh rendering: Prior mesh reconstruction used OpenGL with REINFORCE-estimated gradients, whereas this work uses geometry-grounded gradients designed for neural networks.OpenDR is described as a general-purpose differentiable renderer, unlike the proposed neural-network-oriented gradients.
- Single-image reconstruction: Single-image reconstruction commonly learns 2D-to-3D mappings with 3D models, while PTN demonstrated reconstruction without 3D supervision using silhouette images.The paper follows the silhouette-supervision setting but generates polygon meshes instead of voxels.
- Image editing: Gradient-based image editing generates images by optimizing image losses, and the paper extends this paradigm to mesh vertices and textures through rendered-image losses.The renderer supplies gradients with respect to mesh vertices and textures, enabling mesh style transfer and DeepDream.
3. Approximate gradient for rendering
Neural Renderer preserves standard rasterized images while supplying approximate, non-zero gradients for mesh parameters, enabling end-to-end neural-network training. The method handles vertex movement, occlusion, textures, and lighting-related optimization.
- Rendering pipeline: Neural Renderer uses differentiable vertex-to-screen transformations followed by rasterization to generate images from mesh vertices and faces.Vertices are transformed into screen space, and the resulting image is generated by sampling.
- The rasterization problem: Rasterization blocks ordinary back-propagation because pixel colors change discretely when a face crosses a pixel center, making vertex derivatives zero almost everywhere.Consequently, loss signals at pixels do not reach the corresponding vertices.
- Approximate gradient: The proposed gradient replaces sudden pixel-color changes with a gradual interpolation between the current vertex position and the position where the face reaches the pixel center.The backward-pass gradient is selected according to whether the error signal indicates that the pixel should become brighter or darker.
- Approximate gradient: When a face cannot overlap a pixel under the considered vertex movement, the method sets the derivative to zero and uses the unmodified rasterizer for the forward pass.This preserves standard rendered images while providing non-zero gradients where appropriate.
- Rendering details: With multiple faces, the rasterizer draws only the frontmost face and suppresses gradients from occluded cross-points.Texture coordinates are obtained with centroid coordinates and sampled by bilinear interpolation; lighting gradients can flow into light parameters and direction.
4. Applications of Neural Renderer
The renderer supports single-image mesh reconstruction from silhouette supervision and gradient-based 3D editing from 2D objectives. Applications include mesh deformation, style transfer, and DeepDream optimization over mesh geometry and textures.
- Applications: The paper applies Neural Renderer to single-image 3D reconstruction with silhouette supervision and to gradient-based 3D mesh editing.The editing applications include 3D style transfer and 3D DeepDream.
- Single-image 3D reconstruction: For reconstruction, a generator deforms a predefined 642-vertex isotropic sphere using local and global bias vectors instead of generating a mesh from scratch.The movable range of each vertex is restricted within the same quadrant on the original mesh.
- Single-image 3D reconstruction: The reconstruction generator combines silhouette loss with smoothness loss, using a weighted objective to match silhouettes while regularizing surface smoothness.Silhouette loss is a negative intersection over union, while smoothness loss encourages adjacent-face angles toward 180 degrees.
- 2D-to-3D style transfer: For 2D-to-3D style transfer, the method initializes a content mesh and optimizes its vertices and textures using content, style, and noise-reduction losses on rendered images.The style loss compares Gram matrices of rendered and style-image features.
- 3D DeepDream: For 3D DeepDream, the method optimizes mesh vertices and textures by minimizing the negative squared norm of feature maps extracted from rendered images.Optimization follows the analogous 2D procedure but operates on the mesh parameters.
5. Experiments
The experiments evaluate the effectiveness of Neural Renderer through the paper’s two proposed applications.
- Evaluation scope: The experimental evaluation focuses on the two applications introduced for Neural Renderer.The passage identifies application-based evaluation rather than a separate experimental objective.
- Evaluation scope: The renderer’s effectiveness is assessed within those application settings.The section frames the experiments as an evaluation of the renderer.
- Evaluation scope: The experiments are organized around the paper’s reconstruction and mesh-editing uses of the renderer.These are the two applications described in the surrounding paper context.
5.1. Single image 3D reconstruction
The method reconstructs polygon meshes from single images using silhouette supervision and compares them with voxel-based reconstruction. Meshes achieve strong visual quality and outperform the voxel-based approach in most evaluated categories, while smoothness regularization improves surfaces but slightly lowers voxel IoU.
- Experimental settings: The evaluation uses 13 ShapeNetCore categories, 24 azimuth viewpoints, 64 × 64 images, and the same train, validation, and test splits as the voxel baseline.Both methods are compared under nearly identical rendering and dataset settings, although the datasets are not exactly the same.
- Experimental settings: Reconstruction accuracy is measured by average voxel IoU after voxelizing both ground-truth and generated meshes at size 32^3.Each test object is reconstructed from images at 24 viewpoints before averaging the IoU scores.
- Qualitative results: The mesh-based method correctly reconstructs meshes from single images and produces visually more appealing shapes than voxel reconstruction.Meshes preserve small parts at high resolution and avoid cubic artifacts.
- Regularization: Smoothness regularization makes reconstructed surfaces smoother and more natural, but slightly lowers voxel IoU across the entire dataset.Without the regularizer, silhouettes can still be matched while surfaces sometimes remain rough.
- Quantitative evaluation: 10 out of 13 categories favor the mesh-based approach in voxel IoU, with especially strong results for airplane, chair, display, loudspeaker, and sofa.The authors suggest that mesh scaling and translation properties contribute to gains for display and loudspeaker objects.
- Limitations: Performance is weaker for car, lamp, and table categories because their relatively complicated shapes are difficult to reconstruct by deforming a sphere.The method also cannot generate objects with various topologies because the faces-to-vertices relationship is fixed rather than dynamic.
5.2. Gradient-based 3D editing via 2D loss
The renderer enables gradient-based 3D editing with 2D supervision, including style transfer and DeepDream. Experiments show that 2D painting styles transfer to 3D textures and shapes, while DeepDream produces object-consistent 3D transformations.
- Applications: The editing experiments apply gradient-based optimization to 3D meshes using 2D-to-3D style transfer and 3D DeepDream.Style transfer uses VGG-16 features, while DeepDream uses the Inception 4c layer from GoogLeNet.
- 2D-to-3D style transfer: Painting styles transfer to both textures and shapes, including straight and wavy patterns appearing on different parts of the edited objects.The transferred style can vary spatially across an object, such as appearing on the side but not the upside of the bunny.
- Applications: The proposed editing method provides an intuitive and quick way to edit 3D models for rapid product-design prototyping and art production.These applications use 2D supervision to modify 3D geometry and appearance.
- 3D DeepDream: DeepDream creates new 3D-consistent forms, such as facial features on the bunny and a bird-like face emerging from the teapot spout.The transformations match the three-dimensional shape of each object.
6. Conclusion
The proposed renderer integrates 3D mesh rendering into neural networks, supporting mesh reconstruction and gradient-based editing from image-based objectives. These applications demonstrate the renderer’s effectiveness and broader potential.
- The method reconstructs 3D meshes from single images using image losses and gradient descent, outperforming an existing voxel-based approach in visual appeal and voxel IoU.The reconstruction uses silhouette image supervision.
- The method edits mesh vertices and textures according to 3D shape using losses defined on rendered 2D images.
- The renderer’s applications are not limited to those presented, suggesting that incorporating the module into other systems can address additional problems.
- 2D-to-3D style transfer applies visual styles to meshes through gradient-based optimization.The initial meshes for style transfer are rendered from six viewpoints.
- 3D DeepDream is another gradient-based mesh-editing application enabled by the renderer.The paper presents DeepDream alongside style transfer as an image-supervised mesh-editing operation.
Appendix A. Additional results
The appendix provides additional visual results for single-image 3D reconstruction and 2D-to-3D style transfer. Reconstruction outputs are shown from multiple viewpoints, while style-transfer examples use varied artwork styles.
- Additional single-image 3D reconstruction results are presented in Figures 9 and 10.The inputs appear at the left, and the reconstructed meshes are rendered from six viewpoints.
- Additional 2D-to-3D style-transfer results are presented in Figures 11–14.The examples use artworks including Self-Portrait, The Starry Night, The Great Wave off Kanagawa, and The Scream as style images.