PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging 2D Data to Learn Textured 3D Mesh Generation

Paul Henderson, Vagia Tsiminaki, Christoph H. Lampert

arXiv:2004.04180v1cs.CVcs.LG

TL;DR

Existing 3D generative models do not produce textured objects, and available mesh datasets lack detailed textures. This paper learns textured 3D meshes from images by reconstructing each image as a rendered foreground mesh over a background, and introduces a parametrization that avoids self-intersections. The authors report diverse, realistic textured samples across five object classes, including from natural images.

  • Problem

    Existing methods generate 3D shapes without textures, while ShapeNet lacks detailed textures for most instances and ModelNet lacks color information.

    Source: §1 · p. 1

  • Method

    The model learns from image collections by reconstructing each image as a generated 3D foreground mesh rendered over a background, using an encoder to infer latent parameters.

    Source: §1 · p. 1, §5 · p. 5

  • Results

    The method generates diverse, realistic textured meshes for five object classes, including from natural images and without ground-truth segmentation masks.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The resulting textured meshes may be used in downstream applications such as visual effects and games.

    Source: §1 · p. 1, §7 · p. 10

    The MASK training setting assumes approximate camera calibration, an instance segmentation mask, and a background image, though these can be estimated automatically from weakly annotated data.

    Source: §5 · p. 5

Abstract

from arXiv · show

Numerous methods have been proposed for probabilistic generative modelling of 3D objects. However, none of these is able to produce textured objects, which renders them of limited use for practical tasks. In this work, we present the first generative model of textured 3D meshes. Training such a model would traditionally require a large dataset of textured meshes, but unfortunately, existing datasets of meshes lack detailed textures. We instead propose a new training methodology that allows learning from collections of 2D images without any 3D information. To do so, we train our model to explain a distribution of images by modelling each image as a 3D foreground object placed in front of a 2D background. Thus, it learns to generate meshes that when rendered, produce images similar to those in its training set. A well-known problem when generating meshes with deep networks is the emergence of self-intersections, which are problematic for many use-cases. As a second contribution we therefore introduce a new generation process for 3D meshes that guarantees no self-intersections arise, based on the physical intuition that faces should push one another out of the way as they move. We conduct extensive experiments on our approach, reporting quantitative and qualitative results on both synthetic data and natural images. These show our method successfully learns to generate plausible and diverse textured 3D samples for five challenging object classes.

1. Introduction

Existing 3D generative methods produce shapes without textures, while common 3D datasets lack detailed texture information. This work learns textured 3D meshes from images and introduces a mesh parametrization that prevents self-intersections.

1. Introduction

  • Existing generative methods produce shapes without textures, and ShapeNet and ModelNet lack detailed textures or color information.Surface appearance strongly influences how people perceive and interact with objects, and textures matter for uses such as visual effects and games.Source: §1 · p. 1
  • Learning from images requires resolving 3D projection ambiguity, handling single views and cluttered backgrounds, and learning object shape and texture together.The approach models each training image through a foreground mesh and background rather than fitting each image independently.Source: §1 · pp. 1–2
  • The method trains a textured 3D mesh model on image collections by reconstructing images as rendered foreground objects over backgrounds.An encoder predicts latent parameters for each image, while a limited-capacity latent space encourages complete, coherent objects rather than shapes tailored to individual views.Source: §1 · pp. 1–2
  • The proposed mesh parametrization guarantees non-intersecting surfaces without regularization while retaining flexibility for complex shapes such as chairs and airplanes.It models faces as pushing one another out of the way as they move.Source: §1 · p. 2
  • Experiments on four synthetic object classes and two natural-image classes report diverse, realistic samples, including without ground-truth segmentation masks.The method also represents angular and concave classes such as chairs, airplanes, and sofas.Source: §1 · p. 2

2. Related work

Most 3D generative models learn from 3D-shape datasets, while several approaches using 2D data depend on silhouettes or produce untextured outputs. Related work also includes textured reconstruction and mesh methods that do not guarantee non-intersection.

2. Related work

  • Most 3D generative models learn from 3D-shape datasets, unlike this work, which learns from image collections.Earlier work learning generative 3D shapes from 2D data includes voxel GANs trained to match silhouette distributions [8] and MP-GAN, which uses viewpoint-specific discriminators.Source: §2 · p. 2
  • Silhouette-supervised reconstruction methods require masks, while the closest cited mesh-generation approach [15] handles untextured renderings and lacks a self-intersection mechanism.The cited works reconstruct voxels, meshes, or point clouds from images; unlike this paper, they do not learn an explicit prior for sampling new meshes a priori [36] [56] [51] [47] [24] [46] [22] [19] [14] [15] [30] [23].Source: §2 · pp. 2–3
  • Other texture-generation and reconstruction approaches rely on full supervision, including textured 3D data, or generate texture in image space after sampling textureless shapes.Examples include implicit texture functions [35], image-space texture generation, colored-voxel reconstruction [43], and texture priors for multi-image reconstruction [29] [38].Source: §2 · p. 3
  • Local smoothness regularizers do not prevent separate smooth surfaces from passing through each other, and cited intersection-processing methods do not construct non-intersecting meshes a priori.This distinction matters when a neural network directly predicts mesh shape.Source: §2 · p. 3

3. Modelling textured meshes

The model represents each textured object as a fixed-topology mesh and generates its geometry and face colors from separate latent codes. The shape code can also inform color generation, while a more structured shape decoder is needed to avoid self-intersections.

3. Modelling textured meshes

  • Each mesh has fixed topology, with generated vertex positions defining its shape and per-face RGB values defining its texture.The model samples latent codes and passes them through decoders to produce these attributes.Source: §3 · p. 3
  • The shape decoder maps zshape to vertex positions, while the color decoder maps zcolor to face colors.The shape and color codes are separate, but the shape code can also condition color generation to capture shape–color dependencies.Source: §3 · p. 3
  • A dense decoder that directly predicts vertex locations can produce irregular, self-intersecting meshes.The paper introduces a more structured decoder to guarantee non-intersecting surfaces.Source: §3 · p. 4

4. Non-intersecting mesh parametrization

The parametrization deforms an initially non-intersecting mesh in steps, making faces push others aside rather than intersect. It detects potential collisions in a projected plane and adjusts vertex displacements to preserve face ordering.

4. Non-intersecting mesh parametrization

  • The decoder repeatedly moves vertices along a shared direction from an initially non-intersecting mesh, with per-vertex displacement distances adjusted to prevent intersections.The process uses a sequence of deformation steps and a physical intuition of surfaces pushing one another out of the way.Source: §4 · p. 4
  • The adjusted distances minimize total vertex motion subject to each vertex meeting its proposed lower bound and the no-intersection constraints.These linear constraints form an LP; the buffer distance ε is included in the ordering constraint.Source: §4 · p. 4
  • Potentially colliding face pairs are found by projecting triangles onto a plane perpendicular to the shared motion direction and checking for overlap.The overlap regions are projected back onto the faces in 3D.Source: §4 · p. 4
  • For each candidate pair, the method preserves the initial ordering of projected intersection points along the motion direction.The face points’ displacements are expressed using their barycentric coordinates.Source: §4 · p. 4
  • Gradients pass through the LP solution by differentiating through the active constraints’ linear system.At the solution corner, active constraints define a system of linear equations whose solution is the displacement vector.Source: §4 · p. 4

5. Training from images

The model is trained from images by reconstructing each as a 3D foreground mesh over a background, with different available supervision in MASK and NO-MASK settings. Its sampled meshes show diverse, realistic geometry and textures across the classes illustrated.

5. Training from images

  • MASK training uses approximate camera calibration, an object mask, and a background image; NO-MASK uses only approximate camera calibration.The MASK setting adds a multiscale silhouette IoU objective and local mesh smoothness regularizers.Source: §5 · pp. 5–6
  • The image-based training objective models each image as a rendered 3D foreground mesh over a 2D background and uses an encoder to infer its latent variables.The model places the mesh relative to a perspective camera and renders it over the background.Source: §5 · p. 5
  • In NO-MASK, a low-dimensional latent code generates the background, constrained so it cannot capture high-frequency foreground details.The background decoder uses a 16-dimensional code and three transpose-convolutional layers, then upsamples the image.Source: §5 · p. 5
  • Training maximizes a modified variational objective, replacing the Gaussian log-likelihood with MS-SSIM and scaling the KL-divergence term.The model and encoder are trained jointly to reconstruct images while keeping the latent posterior close to the prior.Source: §5 · p. 5
  • Cars and sofas use NO-MASK with DENSE parametrization, while chairs and airplanes use MASK with PUSHING and have no self-intersections despite finely detailed geometry.Figure 4 shows five views of each sampled mesh; the caption describes the samples as diverse and realistic in geometry and texture.Source: Figure 4 · p. 6
  • The encoder groups image pixels by the mesh face they would occupy, then uses group mean colors to estimate the color latent distribution.A separate group represents background pixels; this makes the color code account for pixel location relative to the reconstructed surface.Source: §5 · pp. 5–6

6. Experiments

Across ShapeNet and natural-image experiments, the model generates plausible textured meshes, including without masks; the non-intersecting parametrization trades slightly worse generation metrics for meshes usable where intersections matter. The learned model also reconstructs natural-image objects, including plausible occluded regions.

6. Experiments

  • Experiments cover five classes across synthetic ShapeNet renderings and natural images, using IS, FID, and KID to evaluate rendered samples.IS is better when larger; FID and KID are better when smaller. Hyperparameters were selected manually for perceived generation quality, not directly optimized for these metrics.Source: §6 · p. 6

6.1. Validation on ShapeNet

  • Across four ShapeNet classes, both MASK and NO-MASK models generate plausible, reasonably diverse samples; NO-MASK gives slightly poorer IS, while FID and KID are similar across settings.For FID and KID, cars and sofas are better with MASK, while airplanes and chairs are better with NO-MASK.Source: §6.1 · p. 7, Table 1 · p. 7
  • PUSHING avoids the substantial self-intersections produced by DENSE, but all three generation metrics are slightly worse, making parametrization choice an application-dependent trade-off.PUSHING is recommended when non-intersection matters; DENSE may be more appropriate otherwise. Both parametrizations produce reasonable renderings, and PUSHING can produce realistic shapes despite fewer degrees of freedom.Source: §6.1 · pp. 7–8, Table 2 · p. 7, Figure 5 · p. 7

6.2. Generation results on natural images

  • NO-MASK birds perform significantly better than any other natural-image combination, while 3D models outperform the 2D VAE for birds but perform somewhat worse for cars.For cars, NO-MASK is slightly worse than MASK; generated car samples are less diverse and have blurrier colors, while bird performance shows no perceptible decrease without masks.Source: §6.2 · p. 9, Table 3 · p. 10

6.3. Single-image 3D reconstruction on natural images

  • The encoder and decoder produce textured single-image reconstructions without 3D supervision or, in NO-MASK, segmentation masks; visible details and occluded regions are often reconstructed plausibly.Occluded regions do not affect the photometric reconstruction loss; plausible completions are associated with learning shapes that reproject across training images under a low-dimensional latent space constrained by the KL term.Source: Figure 7 · p. 9, §6.3 · p. 10

7. Conclusion

The paper presents a generative model of textured 3D meshes trained from images by reconstructing them as foreground meshes over backgrounds. It generates diverse, realistic textured meshes and introduces a parametrization that avoids intersections a priori, with a slight metric trade-off.

7. Conclusion

  • The model learns from images alone by reconstructing them as foreground meshes rendered over backgrounds, generating diverse and realistic textured meshes across five object classes.Natural images can be used without multiple views of an instance or ground-truth segmentation masks.Source: §7 · p. 10
  • The resulting textured meshes may be used immediately in downstream applications such as visual effects and games.Source: §7 · p. 10
  • The new mesh parametrization avoids intersections a priori, but produces samples scoring slightly lower than a naive parametrization on IS/FID/KID metrics.The intersection guarantee is useful for applications such as physical simulation or 3D printing.Source: §7 · p. 10

A. Additional qualitative results

The appendix provides additional uncurated samples and reconstructions across synthetic-rendering and natural-image settings, with comparisons across model settings and parametrizations.

A. Additional qualitative results

  • Random, uncurated samples generated by models trained on ShapeNet renderings are shown for cars (Figures 8–9), chairs (10–12), airplanes (13–15), and sofas (16–18).Section 6.1 discusses these results.Source: A. Additional qualitative results · p. 12
  • The paper presents random (uncurated) samples from models trained on natural images of cars in Figures 19–20 and birds in Figures 21–22, with discussion in Section 6.2.Source: A. Additional qualitative results · p. 12
  • The paper reports additional reconstruction results on natural images of cars in Figure 23 and birds in Figure 24, and directs readers to Section 6.3 for discussion.Source: A. Additional qualitative results · p. 12

B. Network architectures

The appendix specifies the neural architectures for the model’s encoders, decoders, and baseline VAE, alongside conventions for reading their layer notation.

B. Network architectures

  • Layer notation defines Conv, Dense, Upsampling, Reshape, concatenation, and sigmoid, and specifies how layers with additional inputs are indicated.Source: B. Network architectures · p. 12
  • A shared CNN extracts image features, which feed dense encoders predicting latent variables for shape, texture, background, and pose.The feature extractor outputs a 128D feature vector; shape, texture, and background encoders are specified in the listed architectures.Source: B. Network architectures · pp. 12–13
  • The shape decoder maps a 32D shape embedding to 3D offsets for the base-mesh vertices, while the texture decoder takes 128D texture and 32D shape embeddings and predicts RGB colors for the mesh faces.The background decoder is used only in the NO-MASK setting.Source: B. Network architectures · p. 13
  • The baseline VAE uses the main model’s feature extractor, predicts a 160D image embedding, and decodes it into an RGB image.Source: B. Network architectures · p. 13

Related papers

Loading 2004.04180v1…