PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine Detailed Texture Learning for 3D Meshes with Generative Models

Aysegul Dundar, Jun Gao, Andrew Tao, Bryan Catanzaro

arXiv:2203.09362v1cs.CV

TL;DR

The paper addresses realistic textured 3D reconstruction from images, where viewpoint control and consistency remain challenging and real multi-view data is difficult to annotate. It adapts pretrained reconstruction models and uses a texture GAN; the authors report improvements on multi-view and single-view datasets. [p]

  • Problem

    Realistic 3D reconstruction from images is challenging because viewpoint changes can alter appearance, while real multi-view data and its annotations are difficult to obtain.

    Source: §1 · p. 1

  • Method

    The method adapts a pretrained category-specific reconstruction model to images and then uses a GAN with learnable position encodings and attention to generate texture maps.

    Source: §1 · pp. 1–2

  • Results

    The authors report significant qualitative and quantitative improvements on both multi-view and single-view datasets, with superior textured 3D models.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The approach reconstructs consistent textured 3D models from multi-view images without human-annotated labels for novel sequences.

    Source: §5 · p. 10

    The framework still requires estimates of camera parameters and mask silhouettes from pretrained networks.

    Source: §5 · p. 11

Abstract

from arXiv · show

This paper presents a method to reconstruct high-quality textured 3D models from both multi-view and single-view images. The reconstruction is posed as an adaptation problem and is done progressively where in the first stage, we focus on learning accurate geometry, whereas in the second stage, we focus on learning the texture with a generative adversarial network. In the generative learning pipeline, we propose two improvements. First, since the learned textures should be spatially aligned, we propose an attention mechanism that relies on the learnable positions of pixels. Secondly, since discriminator receives aligned texture maps, we augment its input with a learnable embedding which improves the feedback to the generator. We achieve significant improvements on multi-view sequences from Tripod dataset as well as on single-view image datasets, Pascal 3D+ and CUB. We demonstrate that our method achieves superior 3D textured models compared to the previous works. Please visit our web-page for 3D visuals.

1 INTRODUCTION

The paper targets realistic, controllable textured 3D reconstruction from images, addressing limitations in single-view supervision and synthetic multi-view data. It adapts pretrained reconstruction models and adds a texture GAN, reporting improvements across multi-view and single-view settings.

Motivation

  • Single-view reconstruction can look faithful from the input viewpoint yet unrealistic from other views, while synthetic multi-view training can leave a domain gap on real images.Single-view methods lack guidance for consistent predictions in invisible regions; synthetic multi-view data also may not capture realistic details across views.Source: §1 · p. 1

Method

  • The framework adapts a category-specific reconstruction model to unannotated real multi-view images, using pretrained estimates rather than costly per-frame 2D keypoints, camera poses, or 3D meshes.The framework is described as robust to noisy camera and silhouette predictions and as improving those initial estimates.Source: §1 · pp. 1–2
  • A UV-space texture GAN adds realistic detail and handles image variation within a sequence, using learnable position encodings and attention in its generator and discriminator.The design exploits spatial alignment in UV maps, where corresponding parts such as tires appear in the same location across examples.Source: §1 · p. 2

Results

  • The method reports superior textured 3D models with coherent geometry and convincing textures, improving results on both multi-view sequences and single-view datasets.The authors describe the results as realistic synthesis using a differentiable renderer and mesh representation while training solely on images.Source: §1 · p. 2

2 RELATED WORK

Prior work highlights limitations in image generation and 3D reconstruction: viewpoint changes can affect identity or appearance, single-view supervision leaves unseen regions unconstrained, and real multi-view data is costly to annotate. The paper positions mesh-based reconstruction from real multi-view images as a way to address these challenges.

2.1 Generative Adversarial Networks (GANs)

  • Viewpoint control in GANs remains limited by entanglement: identity or appearance can change as viewpoint changes, and 3D-aware models do not guarantee strict 3D consistency [19], [20], [2], [11], [33], [34].Earlier image disentanglement is limited to 2D space, while style-based GANs provide implicit 3D knowledge without strict shape-appearance disentanglement.Source: §2.1 · pp. 2–3

2.2 3D Object Reconstruction from Single-view Images

  • Single-view reconstruction is ill-posed because invisible mesh and texture regions receive no training gradients, and results can look unrealistic from a different viewpoint, [9], [14], [18], [24].Learning shape and viewpoint without ground-truth camera parameters has produced lower-quality results than methods using known camera parameters [9].Source: §2.2 · p. 3

2.3 3D Object Reconstruction from Multi-view Images

  • Multi-view datasets offer more supervision but are small because camera-pose or keypoint annotation is expensive; synthetic data can transfer poorly to real images, and NeRF methods overfit to sequences [31], [32], [47], [3], [41], [42], [53].The paper favors efficient mesh representations and complete texture modeling from 360° object rotations.Source: §2.3 · p. 3

3 METHOD

The method adapts a pretrained reconstruction network in two stages: first learning geometry, then training a GAN for texture maps. It uses UV-map alignment to guide attention in the generator and positional embeddings in the discriminator.

3.1 Mesh Learning

  • The first stage adapts an encoder-decoder with differentiable rendering to learn consistent geometry, using perceptual, silhouette, mesh-smoothness, and camera losses.For multi-view adaptation, it reconstructs a randomly selected target view; single-view training reconstructs the input view.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 Fine Detailed Texture Learning

  • The texture stage projects images onto predicted meshes in UV space to create pseudo-ground-truth maps, with visibility masks used to mask textures and GAN losses.Pseudo-ground-truth accuracy depends on mesh and camera estimates; projected mesh silhouettes are used to mask inputs because Mask-RCNN borders can be imprecise on Tripod.Source: §3.2 · pp. 4–5

3.2.1 Position-based Attention in Generator

  • The generator uses learnable positional queries and keys to form attention maps that relate UV locations, while prior-layer features supply values.The attention layer is added early in the generator; a second layer showed no benefit.Source: §3.2.1 · p. 5

3.2.2 Learnable Position Embedding in Discriminator

  • The discriminator receives a learnable positional embedding so its patch predictions can specialize by location despite a limited receptive field.The paper notes that a 16 × 16 receptive field works reasonably well but cannot identify which texture-map patch it classifies.Source: §3.2.2 · p. 5

4.1 Datasets and Preprocessing

The multi-view experiments use unannotated Tripod car sequences with noisy masks and camera estimates, then prune frames with incorrect camera estimates. The single-view evaluations use Pascal 3D+ and CUB data with camera poses and masks available or estimated.

4.1.1 Multi-view Datasets.

  • Tripod contains unannotated car sequences of about 90–120 images captured across a 360-degree rotation, with masks and camera estimates affected by domain gaps.Mask-RCNN masks are less accurate on Tripod than on COCO, while camera estimates from a Pascal 3D+ model are noisy; camera parameters are jointly optimized.Source: §4.1.1 · p. 5
  • Frames with large camera-estimate changes between consecutive views are removed, and training on the pruned dataset improves FID and the reconstructed headlight and plate areas.The authors report that pruning also prevents incorrect camera estimates from producing wrong pseudo-ground-truth texture maps.Source: §4.1.1 · p. 6, Figure 4 · p. 6

4.1.2 Single-view Datasets

  • The single-view evaluations use Pascal 3D+ and CUB with the DIB-R train-test split, using available ground-truth masks or Mask-RCNN masks and estimated camera parameters.The comparison of pseudo-ground-truth mask sources illustrates that mask-prediction errors can leak black pixels into generated textures.Source: §4.1.2 · p. 6, Figure 5 · p. 6

4.2 Evaluation

Evaluation reports FID for rendered novel views and uses qualitative figures to show texture and rendering behavior. Tripod FID results are averaged across 19 sequences.

4.2 Evaluation

  • FID is calculated from novel-view renders scaled to 299 × 299; Tripod reports the average of each sequence’s best score across 19 sequences.For Pascal 3D+ and CUB, the reported averages use five runs with different random seeds.Source: §4.2 · p. 6
  • In Fig. 6, the first column shows generated texture maps and later columns show renders from different viewpoints; grey pixels appear in the maps.Their visibility varies between the first and second rows.Source: Figure 6 · p. 6
  • Table 2 presents Tripod FID scores for multiple setups, averaged across 19 sequences, with lower scores preferred.Fig. 7 compares texture maps and renders from different viewpoints, including the baseline and final models.Source: Figure 7 · p. 7, Table 2 · p. 8

4.3 Model and Training Details

The method uses a reconstruction network followed by a GAN pipeline for texture generation. The section describes their architectures, training setup, and single-view evaluation protocol.

4.3 Model and Training Details

  • The reconstruction network shares a feature extractor for deformation and texture maps, and uses a pretrained ResNet18 camera-pose module when ground-truth camera parameters are available.Training uses batch size 50 on one GPU; pretrained training takes 18 hours and adaptation training takes 8 hours.Source: §4.3 · p. 8
  • The texture generator uses position-based attention, while a multi-scale discriminator receives texture maps, visibility masks, and learnable positional embeddings.The GAN pipeline trains on four GPUs with batch size 32 for about 18 hours.Source: §4.3 · p. 8
  • Pascal 3D+ car and CUB bird tables report averaged best FID scores from five runs with different random seeds, with lower scores preferred.The best scores among the runs are averaged for the tables.Source: Table 3 · p. 8, Table 4 · p. 8

4.4 Quantitative Results

The quantitative results compare Tripod setups and report consistent FID improvements from the proposed generative components on single-view datasets. Mask-derived pseudo-ground-truth textures can also contain distracting grey pixels.

4.4.1 Multi-view Datasets

  • Tripod has no prior work for 3D textured-model reconstruction, and single-view-trained models perform poorly on that dataset.The authors note those models are not comparable to their sequence-adaptation approach.Source: §4.4.1 · p. 8
  • On Tripod, Mask-RCNN masks degrade FID relative to renderer-projected masks because mask impurities make black pixels visible in pseudo-ground-truth textures.The resulting grey pixels can be visually distracting and appear in generated texture maps.Source: §4.4.1 · pp. 8–9
  • Tripod pseudo-ground-truth textures vary with lighting and reflections as cars rotate, and GAN-based training can synthesize diverse texture maps in this setting.The texture differences reflect changes in lighting direction and reflections on glass.Source: Figure 8 · p. 8
  • Learned discriminator embeddings improve Tripod FID from 82.19 to 66.57, and generator position-based attention further improves it.The authors attribute the embedding improvement to spatially aligned discriminator inputs and the attention improvement to signals from distant features.Source: §4.4.1 · p. 9

4.4.2 Single-view Datasets

  • Learned discriminator embeddings and generator position-based attention consistently improve FID on Pascal 3D+ and CUB; position-based attention further improves Tripod FID.The reported Tripod FID improves from 82.19 to 66.57 with learned discriminator embeddings.Source: §4.4.2 · p. 9, §4.4.1 · p. 9

4.5 Qualitative Results

The qualitative results show the roles of per-sequence adaptation and GAN-based texture generation on multi-view sequences, then compare novel-view generations on single-view datasets. The final models are described as producing more detailed results with fewer artifacts than the baseline.

4.5.1 Multi-view Datasets

  • TexRecon and NerfMM perform poorly on the challenging multi-view sequences, which have inconsistent lighting across views.Dib-r and GANVerse3D are not directly comparable because they are not adapted to each sequence.Source: §4.5.1 · p. 9
  • Per-sequence reconstruction estimates shape accurately, but view-averaged texture predictions blur colors and details; the GAN models lighting and reflection variation to produce more varied textures.The final model visibly improves on the baseline, and the text also reports FID improvements.Source: §4.5.1 · p. 9

4.5.2 Single-view Datasets

  • On Pascal 3D+, the final model produces fewer artifacts and more details than the baseline in novel-view generations.The text gives a baseline example with three tires, an artifact absent from the proposed model’s output.Source: §4.5.2 · p. 10, Figure 9 · p. 10
  • On CUB, the method produces more diverse and detailed novel-view results than the baseline, whose bird generations often lack eyes.Fig. 10 shows rendered novel views for both models.Source: §4.5.2 · p. 10, Figure 10 · p. 10

5 CONCLUSION

The method reconstructs consistent textured 3D models from unannotated multi-view images using pretrained networks, and reports qualitative and quantitative improvements on multi-view and single-view datasets. Its scope is limited by sphere-based mesh deformation and the need for pretrained estimates of camera parameters and silhouettes.

5 CONCLUSION

  • The method reports significant qualitative and quantitative improvements on multi-view and single-view datasets.The improvements are attributed to changes in the data-generation pipeline and network architectures.Source: §5 · p. 10
  • The final meshes are deformed from a sphere and cannot handle objects with holes, preventing joint modeling of a car's interior.Source: §5 · p. 10
  • Novel sequences still require pretrained-network estimates of camera parameters and mask silhouettes.Source: §5 · pp. 10–11

Related papers

Loading 2203.09362v1…