PaperlayerView exact revision on arXiv

Source-linked AI summary

Appearance-Driven Automatic 3D Model Simplification

Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, Samuli Laine

arXiv:2104.03989v1cs.GR

TL;DR

Matching rendered reference scenes is challenging across complex geometry, appearances, renderers, and scene representations. This paper jointly optimizes triangle meshes and shading parameters from image-space differences using differentiable rendering. It demonstrates a common framework for simplification, conversion, and prefiltering, with view parallax limiting some results.

  • Problem

    The paper seeks ways to match a reference scene's rendered appearance across complex assets and varied shape and appearance representations.

    Source: §1 · p. 1

  • Method

    The method jointly optimizes triangle-mesh shape and shading parameters through differentiable rendering to minimize image-space differences from reference renders.

    Source: §1 · pp. 1–2

  • Results

    The paper demonstrates a common inverse-rendering framework for a wide spectrum of modeling tasks, including simplification, conversion, and prefiltering.

    Source: §7 · p. 12

  • Takeaways & Limitations

    The framework supports joint optimization over shape, appearance, and animation parameters, and the authors identify automatic-differentiation and optimization as making extensions to new applications and latent representations easy.

    Source: §7 · p. 12

    View parallax can blur material parameter textures when geometry differences are large, especially for transparent aggregate geometry with extreme polygon reduction.

    Source: §6 · p. 11

Abstract

from arXiv · show

We present a suite of techniques for jointly optimizing triangle meshes and shading models to match the appearance of reference scenes. This capability has a number of uses, including appearance-preserving simplification of extremely complex assets, conversion between rendering systems, and even conversion between geometric scene representations. We follow and extend the classic analysis-by-synthesis family of techniques: enabled by a highly efficient differentiable renderer and modern nonlinear optimization algorithms, our results are driven to minimize the image-space difference to the target scene when rendered in similar viewing and lighting conditions. As the only signals driving the optimization are differences in rendered images, the approach is highly general and versatile: it easily supports many different forward rendering models such as normal mapping, spatially-varying BRDFs, displacement mapping, etc. Supervision through images only is also key to the ability to easily convert between rendering systems and scene representations. We output triangle meshes with textured materials to ensure that the models render efficiently on modern graphics hardware and benefit from, e.g., hardware-accelerated rasterization, ray tracing, and filtered texture lookups. Our system is integrated in a small Python code base, and can be applied at high resolutions and on large models. We describe several use cases, including mesh decimation, level of detail generation, seamless mesh filtering and approximations of aggregate geometry.

1 INTRODUCTION

The paper treats matching a rendered reference scene as an inverse problem, targeting applications where reference views can be synthesized under controllable viewing and lighting. It proposes jointly optimizing shape and appearance through image-space error, across multiple modeling tasks.

1 INTRODUCTION

  • The broader graphics challenge is to choose shape and appearance representations, model a scene with them, and render it efficiently.The paper contrasts image-directed inverse modeling with common tools that instead ask users to edit scene representations iteratively.Source: §1 · p. 1
  • The goal is to automatically find shape and appearance representations that match a user-provided reference scene when rendered.The scope assumes reference views can be programmatically synthesized under arbitrary, controllable viewing and lighting conditions.Source: §1 · p. 1
  • Applications include geometric level-of-detail simplification, joint shape-appearance simplification, aggregate-geometry simplification, animation, and conversion between rendering systems or shape representations.The listed representations include displacement maps, normal maps, spatially varying BRDFs, and signed distance fields.Source: §1 · pp. 1–2
  • Unlike specialized algorithms that focus on a single task or parts of an object, inverse rendering and nonlinear optimization support joint optimization of representation components affecting appearance.The paper notes that image-space-driven joint optimization can let geometry and a normal map share geometric detail, for example by baking detail into the map as the mesh is smoothed.Source: §1 · p. 2

2 PREVIOUS WORK

Prior work addresses simplification, appearance capture, prefiltering, and differentiable rendering with methods often tailored to particular tasks or representations. This paper instead uses image-space visual error and differentiable optimization to jointly adjust geometry and appearance across a broader set of settings.

2 PREVIOUS WORK

  • Image-driven simplification has used rendered images to guide mesh reduction, while this work uses visual differences with gradient-based optimization through differentiable rendering.Traditional mesh decimation commonly uses geometry-based error, with view-dependent simplification as a silhouette-oriented exception.Source: §2 · p. 2
  • For aggregate-geometry approximation, the method similarly exploits the scene graph but directly optimizes visual appearance, unlike heuristics that preserve it; Cook et al. [2007] stochastically remove and alter elements.Source: §2 · p. 2
  • For appearance prefiltering, prior methods filter normal distributions or use volumetric representations, while this work matches a supersampled reference by optimizing mesh shape and material parameters.The experiments restrict shading to one diffuse lobe and one specular GGX lobe, avoiding runtime overhead or changes to a typical game engine.Source: §2 · p. 2
  • Unlike sparse real-world appearance capture, which often uses multistage processing to generalize from limited measurements, this setup can synthesize target views under needed viewing and lighting conditions.Related approaches include optimizing SVBRDF maps from a handful of photographs while assuming planar geometry [Gao et al. 2019; Guo et al. 2020].Source: §2 · pp. 2–3
  • Other scene representations trade rendering properties: NeRF requires dense sampling at inference and is static, while this work outputs triangle meshes and materials for real-time rendering in a standard 3D engine.Related work also includes learned texture and image-space decoding for view interpolation on reconstructed meshes [Thies et al. 2019], and compact meshes without materials or fine normal-map detail [Chen et al. 2020].Source: §2 · p. 3
  • Prior differentiable scene methods include material-only BRDF conversion [Sztrajman et al. 2017] and single-image mesh generation, whereas this work uses multiple observations to optimize geometry and materials together.The paper also extends compared approaches to more complex shading, geometry, normal and displacement maps, and transparency [Chen et al. 2019; 2020].Source: §2 · p. 3

3 OUR METHOD

The method represents a scene with a triangle mesh and spatially varying material parameters, then optimizes them to match images from a reference renderer. Differentiable rendering supplies gradients for image-space loss across varied views and lighting, while the reference renderer can remain a black box.

3 OUR METHOD

  • The latent representation combines a triangle mesh with textured spatially varying material parameters, producing an asset that can render in modern game engines.The output can use hardware-accelerated rasterization, ray tracing, and filtered texture lookups.Source: §3 · pp. 3–4
  • A differentiable pipeline renders the latent scene, compares it with a reference-rendered target under identical camera and lighting, and computes gradients for vertex positions and texture contents.The pipeline consists of mesh operations, rasterization, and deferred shading.Source: §3 · p. 4
  • The method minimizes empirical image-space risk over camera and light conditions using stochastic gradient descent and gradients obtained through differentiable rendering.Training iterates over many image pairs with randomized cameras and a randomized point light.Source: §3 · p. 4
  • The reference renderer communicates only target images and need not be differentiable or implemented in the same framework, enabling conversion across renderers and geometric representations such as SDFs.The final representation can be a triangle mesh even when the reference is a signed distance field.Source: §3 · p. 4
  • The reference rendering algorithm should match intended use: when both sides omit effects such as shadows, optimization targets the model's materials; reference effects can instead be baked into materials.The pipeline does not render shadows or other global effects itself.Source: §3 · p. 4
  • Good initialization can affect convergence and result quality; because topology and texture coordinates are fixed, the initial mesh needs adequate tessellation and non-overlapping texture parameterization.Initial meshes may come from decimation tools, domain knowledge, or a tessellated sphere; unavailable material textures start randomly initialized.Source: §3 · p. 4
  • The implementation combines differentiable rasterization primitives with mesh operations and deferred shading in PyTorch, adding a Laplacian regularizer to encourage well-formed meshes.The setup supports differentiable rendering at high resolutions and polygon counts at interactive rates.Source: §3 · p. 4

4 APPLICATIONS

The applications use image-driven optimization to simplify assets, prefilter shape and appearance, fit animated characters, approximate aggregate geometry, and convert between representations and rendering systems.

4.1 Joint Shape-Appearance Simplification

  • Starting from a Simplygon-reduced 9k-triangle mesh considerably improves normal-map baking and slightly outperforms Simplygon’s baker, unlike discovering the shape from a sphere.The sphere-start result approximates a 735k-triangle reference, though some high-frequency detail is missing.Source: §4.1 · pp. 4–5, Figure 3 · p. 5
  • Displacement fitting assigns overall form to the coarse base mesh, mid-scale detail to displacement, and the finest detail to the normal map.The displacement is constrained to the normal direction, and the dancer’s appearance closely matches the reference.Source: Figure 4 · p. 5, §4.1 · p. 6
  • With diffuse/GGX shading, optimization closely matches the bronze sculpture’s silhouette, normals, and appearance despite softened speculars; cleanup improves artifacts in a reduced character mesh.The Paragon mesh remains non-watertight because topology changes are unsupported.Source: Figure 5 · p. 5, §4.1 · p. 7

4.2 Shape and Appearance Prefiltering

  • Single-sample renders of prefiltered meshes match supersampled references without apparent aliasing; materials vary by mip level, while tests keep geometry topology fixed.Geometry-level prefiltering can use multiple vertex sets, but the experiments obtain no triangle-count reduction.Source: §4.2 · pp. 7–8, Figure 8 · p. 7

4.3 Animation and Skinning

  • Animated simplification optimizes bind-pose geometry, normals, SVBRDF, and skinning weights over sequences, assuming known bone transforms and time-constant appearance.The method blends transformed vertex positions using skinning weights and is demonstrated on rigged characters.Source: §4.3 · p. 8, Figure 9 · p. 8

4.4 Approximating Aggregate Geometry

  • Replacing foliage leaves with textured quads and image-loss optimization yields plausible Moana approximations at 0.8% and 0.4% of reference triangle counts.The approximations can be instanced 3000× and rendered in a path tracer.Source: §4.4 · p. 8, Figure 10 · p. 9

4.5 Generalizations

  • Image-only supervision transfers across representations and renderers: ray-marched implicit surfaces become textured meshes, while path-traced shadows are baked into rasterized materials under matched viewing conditions.The implicit-surface conversion uses a tessellated sphere as its initial geometry; the path-traced example uses a static light.Source: §4.5 · p. 9, Figure 11 · p. 9

5 IMPLEMENTATION

The implementation combines differentiable mesh operations, a differentiable rasterizer, and nonlinear optimization to fit meshes and materials to rendered references. It supports animation, displacement, spatially varying materials, and training-time antialiasing.

5.1 Mesh Operations

  • Differentiable mesh operations update tangent frames, optimize skinning weights, and support animated meshes.The skinning operator uses a dense representation to allow any vertex-bone association during optimization.Source: §5.1 · p. 10
  • Displacement mapping subdivides a coarse mesh and offsets tessellated vertices along interpolated normals using a displacement texture.Because topology changes are not differentiable, the tessellation factor and topology are fixed before optimization, while vertex positions are recomputed each iteration.Source: §5.1 · p. 10

5.2 Differentiable Renderer

  • Deferred shading uses a G-buffer, while spatially varying specular scale, roughness, and metalness provide an engine-compatible material representation.The specular-scale parameter replaces the occlusion channel in a three-component texture resembling ORM textures.Source: §5.2 · p. 10
  • Training uses MSAA to improve silhouette visibility gradients while retaining one shading sample per pixel; final 1 spp results are rendered without MSAA.MSAA integrates visibility differently from 1 spp, creating a small training-to-rendering mismatch that the authors report is not an issue in practice.Source: §5.2 · pp. 10–11

5.3 Optimization

  • The image loss applies an L1 norm to tone-mapped colors, mapping HDR radiance through a logarithm and the sRGB transfer function.The tone mapping is intended to make the objective robust across the renderer’s floating-point range.Source: §5.3 · p. 11
  • A Laplacian regularizer helps preserve mesh surfaces early in optimization and is combined with image loss using a gradually reduced weight.The regularizer targets the input mesh’s differential, or zero for a poor initial approximation; its initial weight can be set heuristically to 0.25 Limage/Lδ.Source: §5.3 · p. 11
  • Adam uses a decaying learning rate and mini-batches of one to eight; final runs use 10k steps at 2048×2048 resolution and typically take a few hours on one NVIDIA V100 GPU.The initial learning rate is higher for coarse guesses and lower for fine-tuning auto-decimated meshes.Source: §5.3 · p. 11

6 LIMITATIONS

The approach has practical and formulation limits: optimization needs use-case-specific tuning, image-space losses can blur textures under parallax, and rasterization cannot represent several view-dependent effects directly.

Hyperparameter tuning

  • Robustly configuring the Laplacian weight and learning rate across use cases remains an open challenge, despite a heuristic for setting them.The challenge is pronounced across varying tessellation rates and initial mesh quality.Source: §6 · p. 11

View parallax

  • Image-domain L1/L2 losses can blur material textures when latent and reference geometry differ, especially in transparent aggregate geometry with extreme polygon reduction.Parallax discrepancies converge toward median or mean values; the figure notes that optimizing vertex positions can move latent geometry closer to the reference.Source: §6 · p. 11, Figure 13 · p. 12
  • VGG and adversarial losses gave no or limited benefit, adding film-grain-like details or noise instead of improving visual quality.They were also less robust in HDR and did not properly preserve its high dynamic range.Source: §6 · pp. 11–12

Rasterization-based rendering model

  • The rasterization-based renderer cannot handle refraction, specular interreflections, or subsurface scattering directly; highly view-dependent effects are captured only on average.The system instead bakes such reference-renderer effects into available material parameters as well as possible.Source: §6 · p. 12

7 CONCLUSIONS AND FUTURE WORK

The paper demonstrates a common inverse-rendering framework for varied modeling tasks and joint optimization of shape, appearance, and animation parameters. It identifies representation scope and optimization scalability as directions for further work.

7 CONCLUSIONS AND FUTURE WORK

  • The framework demonstrates simplification, conversion, and prefiltering across varied shape and appearance models, with joint optimization of shape, appearance, and animation parameters as a central strength.Coupling automatic differentiation and optimization is described as making extensions to new applications and latent representations easy.Source: §7 · p. 12
  • The evaluated applications are limited to triangle meshes and a commonly used material model so optimized representations work unmodified in existing renderers.The paper proposes hybrids of traditional graphics models and learned representations as a possible extension.Source: §7 · p. 12
  • Highly tessellated meshes are difficult to optimize because large-scale deformations often require moving many neighboring vertices together.A hierarchical approach that progressively generates triangles as results converge might also provide a natural LOD representation.Source: §7 · p. 12
  • For applications without an initial mesh guess, a volumetric grid tessellated with marching cubes is proposed as an alternative geometric representation.The paper also envisions semi-automated tools where artists address visible problems such as under-tessellated regions or insufficient mesh genus.Source: §7 · p. 12

A COMPARISON WITH NORMAL MAP BAKERS

On a scanned skull with the same 9k-triangle input mesh and 2048 × 2048 normal maps, the image-optimized result is reported as slightly higher quality than two production normal-map bakers. Jointly optimizing vertex positions and normal-map texels outperforms optimizing texels alone.

A COMPARISON WITH NORMAL MAP BAKERS

  • With the same 9k-triangle input mesh and 2048 × 2048 normal-map resolution, image-based optimization of positions and texels yields a slightly higher-quality skull result than Simplygon 8.3 and Substance Painter v2020.2.2.The optimization uses 10k steps with randomized viewing conditions and lighting.Source: A COMPARISON WITH NORMAL MAP BAKERS · p. 14
  • Restricting optimization to normal-map texels gives slightly lower quality than jointly optimizing normal-map texels and base-mesh vertex positions.The comparison is between “Our, normals only” and “Ours, normals & positions.”Source: A COMPARISON WITH NORMAL MAP BAKERS · p. 14
  • For the Ewer sculpture, the 7k-triangle initialization reaches 24.49 PSNR (dB) and 0.0588 FLIP, versus 18.13 PSNR (dB) and 0.124 FLIP without the Laplacian regularizer.The reference has 300k triangles; disabling the normal map gives 18.15 PSNR (dB) and 0.127 FLIP.Source: A COMPARISON WITH NORMAL MAP BAKERS · p. 15, Figure 16 · p. 15

B CONVERGENCE

The paper reports convergence across five examples, with difficulty and training conditions varying by task. Normal maps capture micro-detail, while the Laplacian regularizer stabilizes optimization and improves mesh quality.

B CONVERGENCE

  • All five examples converge, though aggregate geometry is harder than the skull normal-map baking example and includes transparency and PBR shading.The displacement-map example jointly optimizes shape, normal maps, and displacement maps with a lower initial learning rate; aggregate examples use MSE for path-tracer transfer, which is more sensitive to outliers than the tone-mapped L1 variant.Source: B CONVERGENCE · p. 15
  • The Ewer example combines high-frequency specular materials and high-dynamic-range lighting, while randomized lights and cameras leave some expected training-error noise.The plots show image-space training error across randomized conditions.Source: B CONVERGENCE · p. 15

C INFLUENCE OF NORMAL MAP AND LAPLACIAN

  • Normal maps help capture micro-detail, while the Laplacian regularizer stabilizes optimization and improves mesh quality.Without the regularizer, large initial steps may corrupt the mesh or create self-intersections that are hard to recover from.Source: C INFLUENCE OF NORMAL MAP AND LAPLACIAN · p. 15

D MESH DECIMATION: VARYING TRIANGLE COUNT IN THE REDUCED MESH

The section studies how the triangle count of the initial reduced mesh affects approximation quality. Even a 500-triangle dancer mesh estimates the reference’s overall appearance, while details and silhouettes benefit from more triangles.

D MESH DECIMATION: VARYING TRIANGLE COUNT IN THE REDUCED MESH

E TESSELLATION AS LEVEL OF DETAIL

The section examines how tessellation and the starting mesh affect appearance matching. A shared base mesh with displacement and normal maps supports multiple tessellation levels, while the base mesh constrains details that displacement can recover.

E TESSELLATION AS LEVEL OF DETAIL

  • The initial-guess study compares six meshes from 130 to 18k triangles against a 300k-triangle reference, with Chamfer-L1 scores scaled by 10^3.Source: Figure 17 · p. 16
  • With varying initial triangle counts, small details and silhouettes benefit from more triangles, while cloth folds are captured even at low counts through normal mapping.Chamfer-L1 scores use 1M point samples and are scaled by 10^3.Source: Figure 18 · p. 16
  • Increasing tessellation improves silhouette edges and details when rendering from a shared base mesh, displacement map, and normal map.The dancer example starts from a 1k-triangle base mesh and is optimized at 2048 × 2048 pixels for 5k iterations.Source: E TESSELLATION AS LEVEL OF DETAIL · pp. 15–16
  • The dancer’s fingers remain inaccurate even at the highest tessellation because the base mesh lacks sufficient geometry, and displacement mapping cannot introduce concavities.The passage notes that more base-mesh polygons or a better initial guess in that region could address the limitation.Source: E TESSELLATION AS LEVEL OF DETAIL · pp. 16, 18
  • The level-of-detail example uses one base mesh, displacement map, and normal map across tessellation levels, with insets showing improved silhouettes and details as tessellation rises.Source: Figure 19 · p. 17
  • The prefiltering figure shows four rendering resolutions: geometric detail and shading smooth as resolution decreases, with the optimized result compared against an antialiased reference.Source: Figure 21 · p. 18

F APPROXIMATING AGGREGATE DETAIL: COMPARISON WITH STOCHASTIC SIMPLIFICATION

For aggregate foliage, the method optimizes image appearance rather than relying on heuristic adjustments, achieving a high-quality approximation after removing most triangles. The section also shows appearance prefiltering across rendering resolutions.

F APPROXIMATING AGGREGATE DETAIL: COMPARISON WITH STOCHASTIC SIMPLIFICATION

  • The stochastic baseline removes 90% of leaves, scales each remaining leaf by 1/λ, and adjusts texel colors to change texture contrast.The contrast adjustment uses each texel’s deviation from the average texture color.Source: F APPROXIMATING AGGREGATE DETAIL:… · p. 18
  • The Elephant and Snail examples extract meshes and materials from ray-marched distance fields; the Snail example includes subsurface scattering.Source: Figure 22 · p. 18, Figure 23 · p. 18
  • With 99.6% of triangles removed, the optimized foliage still produces a high-quality approximation by moving geometry detail into transparency and normal maps.The method optimizes shape and material parameters from image observations rather than using heuristic scaling or contrast adjustments.Source: F APPROXIMATING AGGREGATE DETAIL:… · p. 18

G SHAPE AND APPEARANCE PREFILTERING

  • Across four rendering resolutions, optimized results match the antialiased reference well despite different sample counts, while geometric detail and shading smooth as resolution decreases.All results use the same initial guess.Source: G SHAPE AND APPEARANCE PREFILTERING · p. 19

H LEARNING MESH AND MATERIALS FROM IMPLICIT SURFACES

The section demonstrates learning a mesh and materials to approximate ray-marched distance-field scenes, including an example with subsurface scattering. The result depends on the initial guess, and generalizing to more complex assets would require further work.

H LEARNING MESH AND MATERIALS FROM IMPLICIT SURFACES

Related papers

Loading 2104.03989v1…