PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Geometric Texture Synthesis

Amir Hertz, Rana Hanocka, Raja Giryes, Daniel Cohen-Or

arXiv:2007.00074v1cs.GRcs.CVcs.LG

TL;DR

Generative models for irregular structures, particularly meshes, remain relatively underexplored. This paper learns local geometric texture statistics from one reference mesh and synthesizes them on target meshes; it demonstrates transfer across different genera and texture variations from different latent codes.

  • Problem

    Generative models for irregular structures, particularly meshes, have received limited research attention.

    Source: §1 · p. 1

  • Method

    A neural network learns local triangular-patch statistics from a single reference mesh and probabilistically synthesizes them as vertex displacements on target meshes without parameterization.

    Source: §1 · pp. 1–2

  • Results

    The method transfers geometric textures to varied target meshes, including meshes of different genus, and synthesizes variations based on different latent codes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report results that are difficult to achieve with state-of-the-art graphics tools or require a tailored solution for each target shape.

    Source: §5 · p. 10

    The method synthesizes local textures rather than large structures, assumes stationary and isometric textures, and requires locally uniform triangulation with well-behaved subdivision.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recently, deep generative adversarial networks for image generation have advanced rapidly; yet, only a small amount of research has focused on generative models for irregular structures, particularly meshes. Nonetheless, mesh generation and synthesis remains a fundamental topic in computer graphics. In this work, we propose a novel framework for synthesizing geometric textures. It learns geometric texture statistics from local neighborhoods (i.e., local triangular patches) of a single reference 3D model. It learns deep features on the faces of the input triangulation, which is used to subdivide and generate offsets across multiple scales, without parameterization of the reference or target mesh. Our network displaces mesh vertices in any direction (i.e., in the normal and tangential direction), enabling synthesis of geometric textures, which cannot be expressed by a simple 2D displacement map. Learning and synthesizing on local geometric patches enables a genus-oblivious framework, facilitating texture transfer between shapes of different genus.

1 INTRODUCTION

Generative models for irregular structures, particularly meshes, have received limited attention despite mesh generation being fundamental in computer graphics. This work learns local geometric texture statistics from one reference mesh and synthesizes them on targets without parameterization, including targets of different genus.

1 INTRODUCTION

  • The paper addresses limited research on generative models for irregular structures, particularly meshes, despite mesh generation being fundamental in computer graphics.Recent CNN methods on irregular structures had shown promise for discriminative tasks such as classification and segmentation.Source: §1 · p. 1
  • A CNN learns local triangular patches from a single reference mesh and uses them to generate offsets that match its local statistics on a target mesh.The paper illustrates transferring geometric spikes from a gold reference model to a giraffe target.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The framework requires no parameterization, supports target meshes of arbitrary genus, and probabilistically synthesizes learned patches rather than copying or mapping them.It can displace vertices both normally and tangentially, allowing textures that simple 2D displacement maps cannot express.Source: §1 · p. 2
  • The authors demonstrate transfers to varied target meshes, including between meshes of different genus, and synthesize texture variations using different latent codes.Figure 2 depicts learning from a genus-one cat and transferring to a genus-four fertility statue.Source: §1 · p. 2, Figure 2 · p. 2

2 RELATED WORK

Prior work explores generative models for 3D shapes across voxel, point-cloud, parameterized-surface, and mesh representations, while mesh-texture techniques often rely on mappings or parameterization. This paper's related work positions its approach as generative synthesis of local mesh geometry rather than mapping patches.

2 RELATED WORK

  • Single-image GANs learn local patch distributions, an idea this work applies to irregular 3D triangular meshes.The cited single-image methods include Zhou et al. [2018], Shocher et al. [2018], Gandelsman et al. [2019], Shaham et al. [2019], and Sun et al. [2019].Source: §2 · p. 2
  • 3D generative approaches use voxels, point clouds, parameterized or implicit surfaces, and meshes, each with different representation trade-offs.Voxel CNNs can incur large memory costs at high resolution [Wu et al. 2015, 2016], while point clouds struggle to portray fine-grained detail [Hertz et al. 2020].Source: §2 · p. 2
  • MeshCNN [Hanocka et al. 2019] uses edge operations for discriminative tasks, whereas this work proposes a generative network that synthesizes new mesh geometries and transfers textures across arbitrary genera.Existing explicit mesh-generation techniques typically deform a template, preserving its genus and connectivity.Source: §2 · pp. 2–3
  • Earlier mesh-texture methods map patches or extend image-space synthesis using local parameterization; this method learns source geometries and synthesizes them on the target instead.The cited synthesis methods include Ying et al. [2001], Turk et al. [2001], and Xu et al. [2009]; simple 2D displacement maps may not represent complex local geometries.Source: §2 · p. 3

3 DEEP GEOMETRIC TEXTURE SYNTHESIS

The framework learns local geometric-texture statistics from one mesh using a coarse-to-fine hierarchy of generators trained on multiscale inputs. Face-based features and fixed mesh subdivision support synthesis on novel target meshes.

3 DEEP GEOMETRIC TEXTURE SYNTHESIS

  • A hierarchical GAN learns local triangular-patch statistics from a single mesh and synthesizes geometric texture through progressively refined mesh levels.Each level’s output is subdivided and used as input to the next generator.Source: §3 · p. 4, §3.4 · p. 8
  • Multiscale training meshes are created by repeatedly subdividing and optimizing a low-resolution template to fit the reference, then used to train the hierarchy.The reference mesh is discarded after preparing these training inputs.Source: §3 · p. 4, §3.3 · p. 6

3.1 Triangular Mesh Representation

  • Per-face features encode each edge’s length and opposite-vertex coordinates in a local frame, remaining invariant to mesh translations and rotations.The three edge-feature rows contain enough information to reproduce the mesh in any global position and orientation.Source: §3.1 · p. 5

3.2 Symmetric Face Convolution

  • Face convolutions combine a face embedding with its three neighboring face features and use max operations to remove dependence on face ordering.The initial per-face features are embedded before subsequent convolutions operate on neighboring face embeddings.Source: §3.2 · p. 5
  • The generator predicts per-face displacements that are projected onto local edge axes and averaged across incident faces to move shared vertices.A vertex can move in any direction because it receives displacements from all incident faces.Source: §3.2 · p. 5

3.3 Subdivision and Multi-Scale Meshes

  • Uniform subdivision splits each face into four and behaves consistently across mesh connectivities, supporting transfer to novel target meshes.The fixed operation places vertices at edge midpoints.Source: §3.3 · p. 6
  • The training-mesh fit is optimized with bidirectional Chamfer distance and normal similarity, plus regularizers for uniform edge lengths and smoothness.The optimization uses vertex locations as its variables and is solved through back-propagation.Source: §3.3 · p. 6

3.4 Learning from a Single Mesh

  • At inference, the fully convolutional generators synthesize texture on novel meshes with arbitrary connectivity and resolution after scaling average edge length to one.The target’s given connectivity does not need to be remeshed.Source: §3 · p. 4, §3.4 · p. 8

4 EXPERIMENTS

The experiments demonstrate texture synthesis across target meshes and resolutions, including different genera and latent variations. The comparison with OptCuts highlights the method’s ability to represent tangential movement and its faster processing time.

Implementation

  • The models use 5–7 scales, depending on texture complexity, and each level is trained for 2,000 iterations with the Adam optimizer.The implementation uses seven face-convolution layers per generator and discriminator; reference meshes come from Thingi10K or were built by hand.Source: §4 · pp. 8–9

Hierarchical Generation

  • Starting generation at higher levels with higher-resolution inputs preserves global structure and deforms only local regions; starting at the lowest level can produce different global structures.The hierarchy also supports inference on meshes with different resolutions from a chosen level, with the appropriate level depending on the training meshes and their scale spaces.Source: §4 · p. 9

Texture Synthesis

  • The method synthesizes geometric structures on unseen target meshes without parameterization, and examples show complex 3D displacements and transfer to different target shapes.Figure 10 highlights tangential movement in a cross-section illustration.Source: §4 · p. 9, Figure 10 · p. 9
  • The method transfers textures between meshes of different genera, including from a genus-one cat to a genus-four target.Examples also include a torus and pig.Source: §4 · p. 9

Texture Scale

  • Target mesh resolution determines the scale of the synthesized texture, even when the same generator scales trained on the spiky ball are used.The torus examples use target meshes at different resolutions.Source: §4 · p. 9

Latent Space Interpolation

  • Different noise vectors generate varied textures from the same reference, and interpolating the latent variable enables smooth texture animation.The reported variations on a given target shape are attributed solely to the noise vector because the generator was trained on one reference mesh.Source: §4 · p. 9

Comparison

  • OptCuts takes 10 minutes to compute a parameterization, while the proposed technique takes only a few seconds and captures tangential movement absent from a 2D displacement map.Figure 11 compares OptCuts’ normal-direction displacement of a manually created map with direct synthesis from the 3D reference mesh.Source: §4 · pp. 9–10, Figure 11 · p. 10

5 DISCUSSION AND FUTURE WORK

The work introduces a generative approach that learns local structures from a triangular mesh and synthesizes them on target models. It avoids parameterization and supports displacements in normal and tangential directions, while its scope and remeshing requirements impose limitations.

Contribution

  • The method learns local structures from a single triangular mesh and synthesizes them on different target models; the authors describe it as the first generative model to learn from a single mesh.It matches the local statistics of a specified mesh rather than copying or mapping reference patches.Source: §5 · p. 10
  • The method needs no parameterization and synthesizes geometry by displacing vertices in normal and tangential directions, including for textures not given as displacement maps.The discussion notes that arbitrary textured surfaces have no generic decomposition into a base surface and displacements.Source: §5 · p. 10

Limitations

  • The method is limited to local, stationary, isometric textures and requires locally uniform triangulation; remeshing may fail on thin, intertwined, or geometrically complex shapes.Anisotropic texture transfer would require estimating and synthesizing from a directional field, which the authors describe as difficult.Source: §5 · p. 10, Figure 12 · p. 10

Future Work

  • The framework may also be used to transfer color or other attributes, or to interpolate between positions of the same shape for controlled animation.Source: §5 · p. 10
  • A proposed two-step application first builds texture on a simple shape such as a sphere or torus, then uses it as an intermediate shape for transfer to the target.The authors compare this approach to Bier and Sloan’s texture mapping work [1986].Source: §5 · p. 10

Conclusion

  • The authors present face convolution as facilitating a GAN framework for triangular meshes and report results difficult to achieve with state-of-the-art graphics tools or requiring tailored solutions.Source: §5 · p. 10

Related papers

Loading 2007.00074v1…