PaperlayerView exact revision on arXiv

Source-linked AI summary

TM-NET: Deep Generative Networks for Textured Meshes

Lin Gao, Tong Wu, Yu-Jie Yuan, Ming-Xian Lin, Yu-Kun Lai, Hao Zhang

arXiv:2010.06217v3cs.GR

TL;DR

Existing deep shape-generation methods do not produce textured models, and textured-shape generation has often relied on image guidance. TM-NET separately encodes part geometry and texture, then conditionally generates part textures; the paper reports evaluations across multiple generation and texturing scenarios and identifies material correlation as a remaining limitation.

  • Problem

    Existing deep shape-generation methods do not generate textured models, and prior textured-shape generation has predominantly relied on image guidance.

    Source: §1 · p. 1

  • Method

    TM-NET separately encodes part geometry and texture and uses conditional autoregressive generation to produce compatible part textures.

    Source: §1 · p. 2

  • Results

    The paper evaluates TM-NET qualitatively and quantitatively across mesh autoencoding, novel generation, conditioned texturing, and latent-space interpolation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TM-NET provides a unified framework for generating novel textured meshes, texturing given shapes, interpolation, and image-guided generation.

    Source: §6 · pp. 12–13

    The work models texture dependence on geometry and structure but does not account for correlations between texture, material properties, and object functionality.

    Source: §6 · p. 13

Abstract

from arXiv · show

We introduce TM-NET, a novel deep generative model for synthesizing textured meshes in a part-aware manner. Once trained, the network can generate novel textured meshes from scratch or predict textures for a given 3D mesh, without image guidance. Plausible and diverse textures can be generated for the same mesh part, while texture compatibility between parts in the same shape is achieved via conditional generation. Specifically, our method produces texture maps for individual shape parts, each as a deformable box, leading to a natural UV map with minimal distortion. The network separately embeds part geometry (via a PartVAE) and part texture (via a TextureVAE) into their respective latent spaces, so as to facilitate learning texture probability distributions conditioned on geometry. We introduce a conditional autoregressive model for texture generation, which can be conditioned on both part geometry and textures already generated for other parts to achieve texture compatibility. To produce high-frequency texture details, our TextureVAE operates in a high-dimensional latent space via dictionary-based vector quantization. We also exploit transparencies in the texture as an effective means to model complex shape structures including topological details. Extensive experiments demonstrate the plausibility, quality, and diversity of the textures and geometries generated by our network, while avoiding inconsistency issues that are common to novel view synthesis methods.

1 INTRODUCTION

TM-NET targets textured-mesh generation beyond image-guided approaches, with part-aware textures and the ability to generate meshes or texture a given shape. It combines deformable-box texture mapping with separate geometry and texture encodings and conditional texture generation.

1 INTRODUCTION

  • TM-NET aims to generate novel textured meshes or predict textures for a given shape, including part-specific textures, addressing the limited prior work on generative textured meshes.The introduction also identifies image guidance as a common basis for existing textured-shape generation.Source: §1 · p. 1
  • Deformable template boxes provide a shared part-level representation for geometry and texture, and unfolding them into UV space yields a consistent texture atlas.The box-based mapping is intended to minimize distortion and accommodate shapes with structural or geometric variation.Source: §1 · p. 2, Figure 3 · p. 2
  • Separate PartVAE and TextureVAE encodings support learning texture distributions conditioned on part geometry, while conditional autoregressive generation promotes compatibility across parts.For non-seed parts, the model conditions on geometry and VGG features of the seed-part texture.Source: §1 · p. 2
  • TextureVAE adapts VQ-VAE-2 with vector quantization to support a high-dimensional texture latent space and generate high-frequency details with less blurriness.The paper motivates the design by the tendency of conventional VAEs to produce blurry outputs.Source: §1 · pp. 2–3
  • TM-NET uses texture transparency through the alpha channel to reproduce the appearance of topological details such as holes.This allows hole-like appearance even when a part is represented by a genus-zero box.Source: §1 · pp. 2–3
  • The paper evaluates autoencoding, novel generation, shape-conditioned and image-guided texturing, and latent-space interpolation through qualitative and quantitative comparisons and ablations.Comparisons include SDM-NET, BSP-Net, and Texture Fields.Source: §1 · p. 3

2 RELATED WORK

Prior work spans 3D shape generation, image texture synthesis, and textured-shape generation, but the section identifies gaps in directly generating complete, richly structured textured meshes. TM-NET is positioned as a part-based approach that can generate geometry and texture together.

Deep Generative Models for 3D Shapes

  • Existing 3D shape generators use varied representations, but the cited methods do not generate textured models, despite textures’ importance to visual realism and rich appearance.TM-NET builds on SDM-NET’s structured deformable-box representation.Source: §2 · p. 3

Texture synthesis

  • Neural texture-synthesis methods primarily address image textures, while point-cloud and voxel representations face detail or resolution constraints for surface textures.The section also distinguishes stochastic solid textures from general textures associated with object surfaces.Source: §2 · p. 3

Generating textured 3D Shapes

  • Image-supervised textured-shape methods may produce partial colored point clouds or rely on fixed-topology meshes, limiting coverage of complex structures and geometry.The cited fixed-topology methods rely mainly on input-image constraints.Source: §2 · pp. 3–4
  • Consistent UV mapping is difficult across shape collections, while alternatives cited here incur 3D-space costs or leave atlas coverage and seamlessness unresolved.The section contrasts these approaches with TM-NET’s ability to generate geometry and texture simultaneously.Source: §2 · p. 4

Novel view synthesis (NVS)

  • Novel-view synthesis generates images from unknown viewpoints using an input image, but its implicit intermediate texture cannot be directly mapped to 3D shapes and cross-view consistency is not guaranteed.TM-NET instead generates textured meshes directly without requiring a single-view image input.Source: §2 · p. 4

3 METHOD

TM-NET represents textured meshes through part-level geometry and texture codes, then generates textures conditioned on geometry and other parts. Its design supports varied textures, cross-part compatibility, and texture-based representation of topological detail.

3 METHOD

  • Consistent deformable boxes provide part domains that unfold into texture atlases, avoiding extra distortion at this stage.The atlas remains consistent across shapes whose parts may differ in topology.Source: §3 · p. 4

3.1 Part-based Texture Representation

  • Transparency in texture maps lets genus-zero box parts visually represent high-genus structures such as holes.Pixels that project outside the input surface receive zero alpha, allowing rays to continue through them during rendering.Source: §3.1 · p. 5

3.2 Part-level Texture and Geometry Encoding

  • Two-level dictionary-based vector quantization captures multiscale texture characteristics and supports faithful representation of high-frequency detail.TextureVAE encodes four-channel RGB-plus-alpha patches and decodes the discrete feature maps into images.Source: §3.2 · pp. 5–6
  • PartVAE and TextureVAE separately encode geometry and texture into their own latent spaces.The texture latent concatenates discrete feature maps from two levels.Source: §3.2 · pp. 5–6
  • TextureVAE’s loss combines reconstruction, seam, and embedding terms, with seam penalties targeting boundary pixels that correspond across box edges.The embedding loss updates both dictionary vectors and encoder parameters.Source: §3.2 · p. 6

3.3 Geometry Guided Texture Generation

  • Conditional autoregressive generation samples varied textures for a part from its geometry-conditioned distribution.For other parts, conditioning also uses the seed part’s texture feature to promote coherence across the shape.Source: §3 · p. 4, §3.3 · p. 6

3.4 Applications

  • Shape-conditioned texturing samples multiple seed-part textures, then conditions other parts on the seed texture’s VGG feature.The TextureVAE decoder converts sampled discrete codes into textures.Source: §3.4 · pp. 6–7
  • TM-NET supports random textured-shape generation, latent-space interpolation, and texturing guided by a single-view image.Image-guided texturing conditions on masked-image VGG features and part geometry features.Source: §3.4 · p. 7

4 DATASETS AND NETWORK IMPLEMENTATION

The implementation uses segmented textured meshes from SDM-Net and a staged architecture and training pipeline. Evaluation combines rendered-image similarity measures with human judgments of texture realism.

4.1 Dataset

  • The experiments use segmented textured meshes from the SDM-Net dataset, a ShapeNet Core V2 subset with part segmentations.The dataset is split randomly into training and test sets at a 9:1 ratio.Source: §4.1 · p. 7, §4.4 · p. 7

4.2 Network Architecture

  • TM-NET combines PartVAE, TextureVAE, conditional PixelSNAIL, and SP-VAE; TextureVAE uses vector quantization to encode and decode texture images.The conditional network adds fully connected layers to match its conditioning input to the discrete index matrix size.Source: §4.2 · p. 7

4.5 Training Details

  • Training proceeds through PartVAE and SP-VAE, then TextureVAE, and finally conditional PixelSNAIL.For the car dataset with seven parts, training takes about five days and generating a textured shape takes about 10 minutes.Source: §4.2 · p. 7, §4.5 · p. 8

4.3 Metrics for Evaluation

  • Rendered meshes are evaluated with LPIPS and SSIM, while texture realism is assessed by a human-judgment fooling rate.SSIM compares same-view renders under the same lighting across 12 viewpoints.Source: §4.3 · p. 7

4.4 Parameters

  • Hyperparameters are fixed across object-category datasets, and each dataset uses a random 9:1 training-test split.The reported SSIM settings include α1 = 1 and β1 = 0.25.Source: §4.4 · p. 7

5 EXPERIMENTAL RESULTS AND EVALUATIONS

TM-NET is evaluated on textured-mesh autoencoding and generation, shape- and image-guided texturing, interpolation, and component ablations. Results include strong autoencoding and texture comparisons, with texture quality varying by category and resolution.

5.1 Reconstruction and Generation of Textured Meshes

  • TM-NET outperforms other methods on multi-view SSIM across all four ShapeNet categories, and transparency reproduces the appearance of topological details.TM-NET and SDM-NET produce clean structures; only TM-NET reproduces the appearance of topological details.Source: §5.1 · p. 8, Table 2 · p. 8
  • Randomly sampled meshes show diverse, novel textured generations, though car and plane textures tend to be blurrier than chair and table textures.The paper attributes the blur to textures mixing sharp features with smooth or constant regions, which are harder for its TextureVAE to encode.Source: §5.1 · p. 9, Figure 11 · p. 9
  • For a given shape, varying the sampling produces plausible, diverse textures, with different parts of the same shape textured differently.Some generated part textures may appear incompatible, but the training set also contains more drastic examples.Source: §5.1 · pp. 9–10, Figure 13 · p. 10, Figure 14 · p. 10
  • TM-NET's texturing results were judged more realistic and more diverse than Texture Fields results.The comparison used LPIPS to measure diversity and a user-study fooling rate to assess realism.Source: §5.1 · p. 10, Table 4 · p. 10, Figure 15 · p. 10
  • For image-guided generation, TM-NET can sample multiple textured shapes and generally produce a result more faithful to the input image than DVR.The sampling strategy makes it easier to find a generated shape close in appearance to the guidance image.Source: §5.1 · p. 10, Figure 16 · p. 11
  • Chair interpolations from TM-NET were more plausible and faithful in geometry and texture than those from TF and VON; part-based alpha blending was also close.The authors attribute alpha blending's similarity to part-by-part blending and the box-based parts' support for correspondence.Source: §5.1 · p. 11, Figure 17 · p. 11

5.2 Self evaluations

  • Dividing texture images into patches yielded higher-quality generated textures, while changing the seed part changed results that remained plausible and compatible across the shape.The seed part conditions other parts on its generated texture features and their geometry features.Source: §5.2 · p. 11, Figure 18 · p. 12, Figure 19 · p. 12

5.2 Self evaluations; 5.3 Ablation Studies

  • Texture autoencoding quality was best at l=256 with 1,024 × 768 texture resolution per part; seam loss removed boundary artifacts, and seed-texture conditioning avoided incompatible parts.The paper cautions that higher resolution does not always improve reconstruction, as generalization may be compromised.Source: §5.2 · p. 12, Table 5 · p. 12, §5.3 · p. 12

6 CONCLUSION, LIMITATION, AND FUTURE WORK

TM-NET represents textured meshes part-wise using deformable-box atlases and separate geometry and texture VAEs. Its conclusion notes limits in texture encoding, fine-detail reconstruction, and image-guided generation.

6 CONCLUSION, LIMITATION, AND FUTURE WORK

  • TM-NET uses deformable-box atlases and separate part-level VAEs to represent geometry and texture, with dictionary-based vector quantization for high-frequency details.The framework supports texturing a given shape, generating novel textured meshes, latent-space interpolation, and image-guided generation.Source: §6 · pp. 12–13
  • Without part-compatibility conditioning, generated table-leg or chair-seat textures can conflict with textures on neighboring parts.The examples compare omitting tabletop VGG features as a condition with enforcing compatibility from a chair-back seed.Source: Figure 21 · p. 13
  • The model conditions texture on geometry and structure, but does not model correlations between texture, material properties, and object functionality.The authors give sofa fabrics and leather as an example of functionality-linked material choices.Source: §6 · p. 13
  • VQ-VAE texture encoding is less effective for complex, varied car and plane textures than for furniture, producing blur and cross-fading during interpolation.The codebook is not constrained to map every entry to a reasonable texture image.Source: §6 · p. 13, Figure 22 · p. 13
  • TM-NET can represent high-genus shapes with texture alpha, but may fail to decode small geometric details such as tabletop slits accurately.Slit details occupy small texture regions that are difficult to encode and decode precisely, and some gaps close in decoded shapes.Source: §6 · p. 13
  • Image-guided conditional sampling does not guarantee that the input texture will be reproduced or closely approximated, and may yield unexpected textures.The authors suggest more deterministic reconstruction using the input image through neural rendering.Source: §6 · p. 13, Figure 23 · p. 13
  • Future directions include procedural textures, additional input and encoding types, and learning styles with substantial texture variation across parts.The current shape-conditioned texturing cannot generate the strongly divergent per-part styles shown in Figure 14.Source: §6 · p. 13

Related papers

Loading 2010.06217v3…