PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning

Shichen Liu, Tianye Li, Weikai Chen, Hao Li

arXiv:1904.01786v1cs.CV

TL;DR

Standard rasterization blocks differentiable image-to-3D learning, limiting dense supervision from pixels. SoftRas replaces discrete rendering operations with probabilistic, differentiable processes and improves single-view reconstruction while handling image-based shape fitting.

  • Problem

    Standard rasterization makes rendering nondifferentiable, limiting dense pixel-level supervision for general-purpose 3D reasoning from images.

  • Method

    SoftRas renders colorized meshes by assigning probabilistic contributions to all triangles and differentiably aggregating them using screen-space probabilities and relative depths.

  • Results

    SoftRas significantly improves unsupervised single-view mesh reconstruction and handles image-based shape fitting, while providing gradients to occluded and unseen vertices.

  • Takeaways & Limitations

    The framework provides a unified differentiable renderer for optimizing mesh geometry, attributes, camera parameters, and illumination from image-based supervision.

Abstract

from arXiv · show

Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers, which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers.

1. Introduction

Soft Rasterizer addresses the nondifferentiability of standard rasterization by directly rendering meshes with differentiable functions and probabilistically aggregating triangle contributions. Its gradients support dense image-to-3D supervision, including occluded and far-range geometry, enabling unsupervised reconstruction and shape fitting.

  • Prior 2D feature matching and shape priors provide task-specific or weak supervision, whereas rendering can provide dense pixel-level supervision for general-purpose 3D reasoning.
  • Standard rasterization discretely samples meshes, preventing gradients from flowing into mesh vertices; approximate backward gradients can also create forward-backward inconsistency.
  • Soft Rasterizer directly renders colorized meshes with differentiable functions and propagates gradients to geometry, vertex attributes, camera parameters, and illumination.
  • Soft Rasterizer treats rendering as probabilistic aggregation, combining all triangles’ pixel contributions using probability maps and relative depths.
  • The framework provides supervision to occluded and far-range vertices and supports unsupervised single-view mesh reconstruction and image-based shape fitting.

2. Related Work

Prior differentiable renderers approximate rendering derivatives while retaining standard graphics rendering in the forward pass. Soft Rasterizer instead focuses on directly rendering meshes with differentiable functions for image-based 3D reasoning.

  • Recent differentiable renderers generate approximate derivatives while directly using standard graphics renderers in the forward pass.
  • The mismatch between forward rendering and backward propagation may produce uncontrolled optimization behavior and limited generalization to other 3D reasoning tasks.
  • Soft Rasterizer focuses on a general-purpose framework that directly renders meshes using differentiable functions instead of only approximating backward derivatives.
  • Image-based reconstruction methods use multi-view geometry or learned shape priors, but multi-view methods are limited by view coverage and learned methods depend on available shape data.

3. Soft Rasterizer

Soft Rasterizer replaces discrete rasterization and z-buffering with differentiable probability maps and aggregation functions. This formulation supports controllable rendering and gradient flow to occluded, depth-separated, and distant geometry.

  • Differentiable Rendering Pipeline: The framework models rasterization as soft binary masking and z-buffering as differentiable aggregation of triangle colors using probability maps and relative depths.
  • Probability Map Computation: Probability maps encode each triangle’s influence on image pixels from relative position and distance, with σ controlling sharpness.
  • Probability Map Computation: As σ →0, the probability map converges to the exact triangle shape, making the formulation a generalized form of traditional rasterization.
  • Aggregate Function: The color aggregation function weights triangle colors by screen-space probability and relative depth, with γ controlling aggregation sharpness and nearest-triangle behavior emerging as γ →0.
  • Aggregate Function: Dedicated aggregation functions support silhouette rendering, while a neural-network aggregate function is evaluated as an alternative.
  • Comparisons with Prior Works: Compared with prior rasterization-based differentiable renderers, Soft Rasterizer propagates gradients to invisible triangles, triangle depths, and distant vertices.

4. Image-based 3D Reasoning

SoftRas applies differentiable rendering to image-based 3D reasoning, supporting unsupervised single-view reconstruction and image-based shape fitting through direct pixel-to-3D gradients.

  • Single-view Mesh Reconstruction: SoftRas renders generated meshes into silhouette and color images, providing direct gradients to shape and color generators for 3D-unsupervised reconstruction.The mesh and corresponding colors are generated from an input image and passed through the SoftRas layer.
  • Single-view Mesh Reconstruction: The reconstruction objective combines silhouette, color, and geometry losses, with geometry regularizing the Laplacian of shape and color predictions.The color loss compares the rendered image with the input using an l1 norm.
  • Color Reconstruction: Color reconstruction uses a palette-based sampling and selection architecture instead of direct color regression.The method reduces complexity from O(HWNc) to O(Nd(HW + Nc)) and can preserve sharp, accurate color recovery with an appropriate palette size.
  • Single-view Mesh Reconstruction: In qualitative single-view reconstruction, SoftRas produces finer object details and smoother surfaces than the compared Neural Mesh Renderer results.The comparison includes input images, ground truth, SoftRas, Neural Mesh Renderer, and Pixel2mesh from two views, alongside mesh-to-scan distances.
  • Image-based Shape Fitting: The renderer is designed to propagate gradients to occluded regions and distant mesh vertices, addressing two challenges in shape fitting.These properties are important for handling large pose changes and local minima during optimization.
  • Image-based Shape Fitting: Image-based shape fitting uses pixel-level errors to establish dense image-to-3D correspondence for optimizing pose, translation, and non-rigid deformation.The rendering function produces an image from mesh parameters, and the difference from the target image supervises the unknowns.

5. Experiments

Experiments evaluate SoftRas on unsupervised reconstruction, color recovery, controlled ablations, and rigid and non-rigid shape fitting. Across these tasks, the renderer improves reconstruction quality and provides useful gradients for difficult optimization settings.

  • Single-view Mesh Reconstruction: SoftRas reconstructs finer, smoother mesh details than NMR without 3D supervision, avoiding discontinuities and self-intersections.Examples include the airplane tail and rifle barrel.
  • Color Reconstruction: SoftRas recovers sharp mesh colors from a single 64 × 64 image, including subtle airplane color transitions and phone-screen shadows.
  • Quantitative Evaluation: More than 4.5 points in mean 3D IoU separates SoftRas’s best setting from NMR across 13 ShapeNet categories.The comparison uses silhouette-only and silhouette-plus-shading settings against unsupervised methods.
  • Ablation Study: Adding a Laplacian constraint raises performance from 62.0 to 62.4, while adding color loss raises it from 62.4 to 64.6.Euclidean distance generally outperforms barycentric distance, and neural-network aggregation slightly exceeds its non-parametric counterpart at higher computational cost.
  • Rigid Pose Fitting: Without scheduling, SoftRas reduces mean relative rotation error by 10.60° versus NMR; with scheduling, its advantage remains 17.37°.The method also avoids the local minimum observed for NMR by using smoothness controlled by σ and γ.
  • Non-rigid Shape Fitting: SoftRas fits an occluded human hand by accounting for all triangles and optimizing mesh-vertex depth, whereas NMR fails to place the hand correctly.

6. Conclusions

The conclusion presents SoftRas as a fully differentiable renderer that reformulates rasterization and z-buffering probabilistically. Its formulation supplies gradients to mesh properties, including unseen vertices and triangle depths, supporting reconstruction and shape fitting.

  • Conclusions: SoftRas directly renders meshes differentiably while jointly handling extrinsic and intrinsic variables and gradients to vertex attributes.Supported attributes include color and normal.
  • Conclusions: Rasterization and z-buffering are reformulated as differentiable probabilistic processes rather than discrete operations.
  • Gradient Analysis: The probability-map representation factorizes gradients from rendered images toward mesh vertices, including terms absent from conventional pipelines.The framework introduces probability map D to handle gradients involving image coordinates and depth.
  • Probability Representation: Triangle-pixel probability is determined from signed distance, with σ controlling sharpness and convergence toward a binary mask as σ →0.The paper considers signed Euclidean and barycentric distance metrics.

A2.2 Barycentric Metric

The renderer combines triangle color maps using an aggregate function based on screen-space position and relative depth, with normalized depth favoring closer triangles. Its soft aggregation approaches standard z-buffering as γ and σ decrease, while occupancy rendering separately models silhouette coverage.

  • Color aggregation: The aggregate function merges per-triangle color maps using screen-space distances and relative depths to produce the rendered output.
  • Rendering controls: Increasing γ produces more transparent renderings, while increasing σ produces blurrier renderings; decreasing both yields effects closer to standard rendering.
  • Depth handling: Depth is clipped and normalized between near and far frustum cut-offs, with closer triangles receiving larger normalized depth values.
  • Aggregate properties: As γ and σ approach zero, the aggregate weights converge toward one-hot selection of the closest triangle covering each pixel, matching z-buffering.
  • Aggregate properties: The background weight approaches one when no triangle covers the pixel, and the weights are robust to translation along the z-axis.
  • Occupancy aggregation: The occupancy aggregate function describes object silhouette independently of color and illumination.

B. Forward Rendering Results

SoftRas directly renders meshes and supports continuous control over blur and transparency through its probabilistic formulation. These effects also reshape the optimization energy landscape to help avoid local minima.

  • SoftRas directly renders a given mesh, unlike existing rasterization-based differentiable renderers described by the authors.
  • Increasing σ produces blurrier renderings, while increasing γ assigns more weight to far-end triangles and produces greater transparency.
  • Blurring and transparency reshape the energy landscape to help avoid local minima during optimization.

C. Network Structure

The implementation includes neural-network alternatives and an encoder-decoder reconstruction architecture. Shared features feed separate shape and color generators, with the shape generator predicting per-vertex deformations.

  • Aggregate network: AN is an alternative color aggregate function implemented with four convolutional layers followed by a sigmoid nonlinearity.
  • Aggregate network: AN is trained against standard-rendering outputs to produce a parametric differentiable renderer.
  • Reconstruction architecture: The reconstruction model uses an encoder-decoder architecture whose encoder acts as a shared feature extractor for shape and color generators.
  • Shape generator: The shape generator uses three fully connected layers to output per-vertex displacement vectors that deform a template mesh toward a target shape.

D. More Results on Image-based 3D Reasoning

Additional experiments visualize mesh deformation for changing target shapes and evaluate reconstruction on real images. The model generalizes from synthetic training to faithful real-image reconstructions with fine-scale details.

  • Additional results: The experiments provide additional results for single-view mesh reconstruction and image-based shape fitting.
  • Mesh deformation: Intermediate deformations show a sphere template becoming a car and then being further deformed to reconstruct an airplane.
  • Mesh deformation: The mesh deformation quickly converges toward target shapes that match the corresponding input images.
  • Real-image evaluation: Although trained only on synthetic data, the model generalizes to real images and novel views with faithful reconstructions and fine-scale details.
  • Real-image evaluation: Reported fine-scale details include fighter-aircraft tail fins, rifle structures, and table legs.

D1.3 More Reconstruction Results from ShapeNet

Additional results show reconstructed geometry and colored reconstructions, while fitting experiments illustrate how SoftRas handles occlusion and local minima during optimization.

  • Additional reconstruction results present both reconstructed geometry and colored reconstructions for each input image.
  • For a largely occluded color cube, stronger early-stage blurring helps escape local minima and gradually reduce rendering loss during fitting.The intermediate process also visualizes the smoothened rendering and colorized fitting errors.
  • In SMPL body-shape fitting, NMR fails on a completely occluded right hand, whereas SoftRas obtains the correct pose within 320 iterations.The comparison visualizes intermediate fitting processes for both methods and attributes SoftRas’s result to its occlusion-aware technique.
Loading 1904.01786v1…