PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Generative Models of Textured 3D Meshes from Real-World Images

Dario Pavllo, Jonas Kohler, Thomas Hofmann, Aurelien Lucchi

arXiv:2103.15627v2cs.CVcs.GRcs.LG

TL;DR

Existing textured-mesh generation approaches require camera poses estimated from costly keypoint annotations, limiting their use on many real-world datasets. This work combines mesh templates and inferred part semantics to learn textured triangle-mesh generators without keypoints. It reports results on par with keypoint-based work and new baselines on ImageNet categories.

  • Problem

    Textured-mesh generation from 2D images has required camera poses estimated from keypoint annotations, which are expensive and usually available for only a few datasets.

    Source: §1 · p. 2

  • Method

    The framework uses category mesh templates and inferred semantic part segmentations to estimate poses, resolve ambiguities, and train a GAN for textured triangle meshes.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    The approach is on par with state-of-the-art on Pascal3D+ and CUB by FID without keypoints, and sets new baselines on ImageNet categories without class-specific hyperparameter tuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework learns textured-mesh generators on real-world categories without keypoint annotations, including a single generator for all categories.

    Source: §1 · p. 2, §5 · p. 8

    The framework requires segmented images and an untextured mesh template for each image category; it uses part semantics to address silhouette-based pose ambiguity.

    Source: §3 · p. 3

Abstract

from arXiv · show

Recent advances in differentiable rendering have sparked an interest in learning generative models of textured 3D meshes from image collections. These models natively disentangle pose and appearance, enable downstream applications in computer graphics, and improve the ability of generative models to understand the concept of image formation. Although there has been prior work on learning such models from collections of 2D images, these approaches require a delicate pose estimation step that exploits annotated keypoints, thereby restricting their applicability to a few specific datasets. In this work, we propose a GAN framework for generating textured triangle meshes without relying on such annotations. We show that the performance of our approach is on par with prior work that relies on ground-truth keypoints, and more importantly, we demonstrate the generality of our method by setting new baselines on a larger set of categories from ImageNet - for which keypoints are not available - without any class-specific hyperparameter tuning. We release our code at https://github.com/dariopavllo/textured-3d-gan

1. Introduction

Textured 3D mesh generators can disentangle shape, appearance, and pose, but existing approaches rely on camera poses estimated from costly keypoint annotations. This work removes that requirement and reports results on par with keypoint-based work as well as new ImageNet baselines.

Motivation

  • Keypoint-based pose estimation restricts prior textured-mesh learning approaches because annotations are expensive and usually available for only a few datasets.These approaches train with 2D supervision but generally require camera poses estimated from keypoint annotations.Source: §1 · pp. 1–2

Contributions

  • The framework learns textured triangle-mesh generators without keypoint annotations, using category mesh templates and inferred semantic part segmentations to refine poses and resolve ambiguities.Its reported results are on par with prior work on Pascal3D+ and CUB and set new baselines on ImageNet categories.Source: §1 · p. 2

Results

  • The method is evaluated on Pascal3D+ and CUB, where it is quantitatively on par with state-of-the-art on FID despite not using keypoints.It also sets new baselines on a larger set of ImageNet categories without class-specific hyperparameter tuning.Source: §1 · p. 2

Contributions

  • The paper evaluates pose-estimation performance under different template and semantic assumptions and demonstrates mesh generation from semantic-map sketches.It also studies a single generator for all categories and the models’ disentanglement properties.Source: §1 · p. 2

2. Related work

Prior mesh reconstruction work spans synthetic 3D supervision and single-view methods, while the closest keypoint-free approaches separately use mesh templates or semantic templates. This work combines those ideas for generation and adapts prior textured-mesh GAN supervision to avoid keypoint requirements.

Keypoint-free pose estimation

  • Single-view triangle-mesh reconstruction methods include template-based fitting and semantic-template matching, but the authors report that neither idea individually generalized to generation in early experiments.Their approach combines mesh templates and semantics, adds optional multiple templates and ambiguity resolution, and uses a general object-part segmentation framework instead of class-tuned SCOPS.Source: §2 · pp. 2–3

Differentiable 3D representations

  • Prior 3D reconstruction methods often require synthetic 3D supervision, while multiple-view methods assume viewpoints that the authors consider unrealistic for natural-image collections.Triangle meshes support UV texture mapping and are used with the differentiable renderer DIB-R.Source: §2 · p. 2

Mesh generation

  • Among 2D-supervised mesh generators, uses face colors in a VAE, cannot supervise textures directly from image pixels, and models UV-mapped textures using keypoint annotations.This work builds on ’s GAN architecture while reworking its supervision strategy to relax the keypoint requirement.Source: §2 · p. 3

3. Method

The method prepares segmented real-world images, estimates poses using mesh templates and semantics, then trains a UV-space GAN to generate textured meshes. Semantics help resolve silhouette-based pose ambiguities and also regularize generation.

3. Method

  • The method requires segmented images and at least one untextured mesh template per category; part semantics help resolve pose ambiguities that silhouettes alone can leave unresolved.Images are automatically filtered for objects that are too small, truncated, or occluded, with the filtering tuned for high precision and low recall.Source: §3 · p. 3

3.1. Pose estimation framework

  • Pose estimation fits rendered template silhouettes to target silhouettes from multiple camera hypotheses, then uses confident, unambiguous poses to infer a 3D semantic template.The process repeats with semantics to resolve ambiguities and potentially reinstate images previously discarded.Source: §3.1 · pp. 3–4
  • The ambiguity score weights pairwise camera-rotation distances by pose confidence, rejecting images with vagr > 0.3.The score targets cases where different camera hypotheses fit the silhouette similarly but describe substantially different rotations.Source: §3.1 · p. 4
  • Semantic templates are inferred by fitting mesh vertex colors to 2D semantic segmentations from the top 100 unambiguous images assigned to each template.For continuous semantic templates, the final pose-scoring step uses class-averaged mIoU and a smooth weighted-Jaccard generalization.Source: §3.1 · p. 5

3.2. Generation framework

  • The generation pipeline reconstructs displacement-map meshes, projects image pixels onto their UV maps as partial textures, and trains a 2D convolutional GAN in UV space.The discriminator uses masked inputs to compensate for texture occlusions.Source: Figure 4 · p. 5, §3.2 · p. 5
  • The reconstruction model predicts silhouettes, RGB textures, and semantic textures; its losses include smoothness and template-vertex supervision on the top 10% of images by IoU.The vertex prior requires templates remeshed to a common topology and helps model fine details where silhouettes provide weak signals.Source: §3.2 · p. 6
  • The all-category model conditions on category through BatchNorm parameters and shares semantic output channels for parts such as wheels across categories.Semantic parts are shared between categories, while the learned template probability maps are smoother than their displayed colors suggest.Source: Figure 2 · p. 3, §3.2 · p. 6, Figure 5 · p. 6

4. Experiments

The experiments evaluate pose estimation and textured-mesh generation across CUB, Pascal3D+, and ImageNet. The method improves on prior work for car and airplane generation without keypoints, while results also examine pose estimation, multi-class generation, and qualitative model behavior.

4.1. Datasets

  • Evaluation spans CUB, Pascal3D+, and 13 categories, using GD and recall for pose estimation and FID for generated meshes.CUB and Pascal3D+ have keypoint annotations; ImageNet is used to establish new baselines. CUB uses about 6k training images, and ImageNet classes vary from about 300 usable giraffe images to thousands for car.Source: §4 · p. 6, §4.1 · p. 7
  • Figure 6 shows wireframe and textured meshes for all 13 classes, plus additional textured samples rendered from random viewpoints.Each class has a wireframe mesh at left, its textured counterpart at right, and two more textured meshes on the second row.Source: Figure 6 · p. 7

4.2. Results

  • Adding semantic information improves both pose geodesic distance and recall, while multiple templates yield no substantial difference.Repeating ambiguity detection and semantic template inference a second time does not improve results.Source: §4.2 · p. 7, Table 1 · p. 7
  • A single generator trained across classes scores in line with separate class models, with no clear winner across all categories.The shared model has the same capacity as the individual models; airplane scores are significantly better, possibly from regularization.Source: §4.2 · pp. 7–8
  • Latent directions correlate with scene style factors such as lighting and color across classes, and fixed-code class interpolation preserves style without observable artifacts.The authors describe this disentanglement analysis as preliminary and report that the factors were learned without explicit supervision.Source: Figure 7 · p. 7, §4.2 · p. 8
  • The full pipeline outperforms on P3D car and airplane FID, but performs slightly worse on CUB bird.The comparison is within the same class; FID is affected by the number of images.Source: Table 2 · p. 8, §4.2 · p. 8
  • Across animal meshes, textures are generally accurate but facial details sometimes fail and legs can merge; vehicle shapes are generally faithful, though rare-class textures can be incoherent.The authors attribute animal issues mostly to pose misalignment with rigid templates and suggest articulated templates; they note that more data could in principle mitigate rare-class issues.Source: §4.2 · p. 8
  • A proof-of-concept car model takes a semantic layout in UV space as input, allowing manipulation of details such as headlight shape or semantic-part placement.The figure demonstrates progressively building a car by sketching its parts.Source: Figure 8 · p. 8, §4.2 · p. 8

5. Conclusion

The framework learns textured 3D mesh generators without keypoint annotations, matches prior work using keypoints, and sets new baselines on a subset of ImageNet categories.

5. Conclusion

  • The framework learns generative models of textured 3D meshes without keypoint annotations, enabling use on real-world datasets.Source: §5 · p. 8
  • The method matches prior work that uses ground-truth keypoints, without relying on that information.Source: §5 · p. 8
  • The method sets new baselines on a subset of ImageNet categories where keypoints are unavailable.Source: §5 · p. 8

A.1. Implementation details

The implementation prepares filtered images and remeshed templates, estimates camera poses, and infers semantic templates before training generators. It also describes optimizers and a separate semantic-layout generation model.

Dataset preparation

  • Images are filtered for confident object detections, sufficient mask area, no border contact, and no collisions; part maps retain frequent semantic classes and weight overlapping detections by score.The part detector uses a 0.2 threshold and keeps classes appearing in at least 25% of category images, yielding about 10 classes per category.Source: §A.1 · p. 12

Mesh templates and remeshing

  • Templates are remeshed by deforming a 32 × 32 UV sphere to match them, reducing complexity and aligning topology for batching and reconstruction supervision.The deformation minimizes rendered pixel-space MSE and uses smoothness, Laplacian, and edge-length regularizers.Source: §A.1 · p. 12, Figure 9 · p. 12
  • Remeshing combines smoothness, Laplacian, edge-length, and optional within-category template-alignment losses; the non-squared L2 alignment penalty encourages only a sparse set of vertices to change.The reported loss weights are 0.00001 for Lflat, 0.003 for Llap, 0.01 for Llen, and 0.001 for Lalign.Source: §A.1 · p. 12

Pose estimation

  • Pose estimation starts with 40 camera hypotheses per image and optimizes them with full-matrix-preconditioned Adam, progressively increasing render resolution and pruning low-IoU hypotheses.The implementation optimizes each hypothesis for 100 iterations and prunes the worst half after iterations 30 and 60.Source: §A.1 · pp. 12–13

Full-matrix preconditioning

  • Full-matrix preconditioning makes the camera optimizer rotation invariant while remaining practical because each camera hypothesis has only eight parameters.The method modifies Adam to use full-matrix preconditioning rather than its usual diagonal preconditioning.Source: §A.1 · p. 13

Semantic template inference

  • Semantic templates average 2D part maps in vertex space using differentiable-renderer gradients and a closed-form accumulator that requires one dataset pass.The inference step selects the top 100 images by IoU among unambiguous poses and favors left/right views within 45 degrees.Source: §A.1 · pp. 13–14

Generative model

  • The single-category reconstruction networks train for 130,000 iterations, while the multi-category model trains for 1,000 epochs with class-balanced sampling.Both use Adam and generated-mesh symmetry; the multi-category model uses four GPUs and synchronized batch normalization.Source: §A.1 · p. 14

Semantic mesh generation

  • For semantic-layout mesh generation, four convolutional layers replace the generator’s latent-code input projection, progressively downsampling the layout from 128 × 128 to 8 × 8.The discriminator concatenates a resized semantic map to its input.Source: §A.1 · p. 14

A.2. Additional results

Additional results describe pose-error patterns, common generation and semantic-template failures, and qualitative demonstrations of latent-space controls. The ImageNet table reports synsets and counts of raw images and retained object instances.

Pose estimation

  • Pose errors are often bimodal, with many near-zero errors and a smaller cluster near the maximum; using semantics mitigates ambiguity and increases available images, particularly for birds.Errors are more concentrated for rigid cars and airplanes, while birds have a longer error tail, likely reflecting less-defined poses for articulated objects.Source: §A.2 · p. 14, Figure 11 · p. 14

Qualitative results

  • Generation failures include incoherent vehicle textures, especially for trucks, and animal errors such as missing facial detail, merged or distorted legs, and occasional mesh distortion.Lower truncation can mitigate these issues at the expense of sample diversity.Source: §A.2 · p. 15, Figure 12 · p. 15

Semantic templates

  • Most learned semantic templates are coherent, but one or two truck templates can be mostly empty when too few images are assigned, helping explain why multiple templates do not consistently outperform one.Source: §A.2 · p. 15

Demo video

  • The supplementary video demonstrates interpretable shared latent directions for lighting and shadows, style-preserving interpolation between classes, and mesh generation from hand-drawn semantic layouts.Source: §A.2 · p. 15

Dataset information

  • The ImageNet data table reports synsets and, for each category, raw-image counts and extracted instances passing size, truncation, and occlusion checks.Source: Table 3 · p. 15

A.3. Dataset information

The experiments use specified ImageNet synsets drawn from ImageNet22k because some required synsets are unavailable in ImageNet1k.

Dataset information

  • The ImageNet experiments use the synsets specified in Table 3.Source: §A.3 · p. 15
  • Some required synsets are not available in ImageNet1k.Source: §A.3 · p. 15
  • The data are therefore drawn from the larger ImageNet22k set.Source: §A.3 · p. 15

A.4. Negative results

The explored alternatives exposed unreliable optimization paths: discrete template hypotheses replaced interpolation and bandit selection, while repeated semantic-guided pose optimization and 3D Chamfer remeshing were unsuccessful.

A.4. Negative results

  • Replicating camera hypotheses across mesh templates and pruning replaced interpolation, which varied across categories, and bandit selection, which performed slightly worse.The discrete formulation increased computation time, motivating pruning.Source: §A.4 · p. 15
  • Some truck semantic templates received few images, leading to incoherent semantics.Source: Figure 13 · p. 16
  • Repeated semantic-guided pose optimization did not work reliably because color gradients were too uninformative for camera-pose updates.The authors tried DIB-R and SoftRas and also smoothed rendered images, without success.Source: §A.4 · pp. 15–16
  • 3D Chamfer remeshing caused artifacts, merged animal legs, and sensitivity to initialization, while silhouette-based differentiable rendering was more reliable.Source: §A.4 · p. 16

Related papers

Loading 2103.15627v2…