PaperlayerView exact revision on arXiv

Source-linked AI summary

CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model

Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, Jun Zhu

arXiv:2403.05034v1cs.CVcs.LG

TL;DR

CRM addresses scarce 3D data and transformer-based methods’ failure to exploit triplane geometric priors. It generates six orthographic views and CCMs, maps them with a convolutional U-Net, and uses Flexicubes for end-to-end textured-mesh reconstruction. The model produces detailed textured meshes in 10 seconds, while remaining sensitive to atypical viewpoints, inconsistent views, and limited grid resolution.

  • Problem

    3D data is scarce, while transformer-based triplane methods do not exploit triplane geometric priors and can yield sub-optimal quality with long training times.

  • Method

    CRM generates six orthographic images and CCMs from one input image, uses a convolutional U-Net to create a triplane, and decodes it with Flexicubes into a textured mesh.

  • Results

    CRM produces detailed, high-fidelity textured meshes from a single image in 10 seconds without test-time optimization.

  • Takeaways & Limitations

    Spatial correspondence between orthographic inputs and triplanes supports high-quality textured meshes with significantly less training cost than previous transformer-based methods.

  • Takeaways & Limitations

    Results can be unsatisfactory for inputs with large elevation or different FoV, inconsistent multi-view diffusion outputs, or geometry requiring detail beyond the grid size of 80.

Abstract

from arXiv · show

Feed-forward 3D generative models like the Large Reconstruction Model (LRM) have demonstrated exceptional generation speed. However, the transformer-based methods do not leverage the geometric priors of the triplane component in their architecture, often leading to sub-optimal quality given the limited size of 3D data and slow training. In this work, we present the Convolutional Reconstruction Model (CRM), a high-fidelity feed-forward single image-to-3D generative model. Recognizing the limitations posed by sparse 3D data, we highlight the necessity of integrating geometric priors into network design. CRM builds on the key observation that the visualization of triplane exhibits spatial correspondence of six orthographic images. First, it generates six orthographic view images from a single input image, then feeds these images into a convolutional U-Net, leveraging its strong pixel-level alignment capabilities and significant bandwidth to create a high-resolution triplane. CRM further employs Flexicubes as geometric representation, facilitating direct end-to-end optimization on textured meshes. Overall, our model delivers a high-fidelity textured mesh from an image in just 10 seconds, without any test-time optimization.

1 Introduction

CRM addresses limited 3D data and transformer-based triplane reconstruction by incorporating geometric priors through aligned orthographic inputs and convolutional design. It combines diffusion-generated views and CCMs with Flexicubes to produce textured meshes efficiently.

  • 3D datasets remain much smaller than image datasets because creating 3D data requires specialized expertise and considerable time.Objaverse contains millions of 3D contents, compared with 5 billion images in Laion.
  • Transformer-based triplane generators do not exploit inherent geometric priors, resulting in sub-optimal quality and long training times.
  • CRM uses six orthographic images and a convolutional U-Net to map spatially aligned inputs to a rolled-out triplane.The U-Net leverages pixel-level alignment and substantial bandwidth to produce detailed triplane features.
  • CCMs add geometric information to the reconstruction network, enriching its understanding of spatial relations and geometry.
  • Diffusion models generate the six orthographic images and CCMs from a single input image, using a filtered Objaverse dataset for training.Training improvements include Zero-SNR, random resizing, and contour augmentation.
  • Flexicubes enables end-to-end optimization with textured meshes as the final output, and CRM generates a high-fidelity textured mesh within 10 seconds.

2 Related Works

Prior 3D-generation methods trade quality, speed, or reconstruction simplicity in different ways. CRM is positioned among score-distillation, sparse-view, and feed-forward approaches as a fast pipeline that avoids test-time reconstruction optimization.

  • 2.1 Score Distillation for 3D Generation: Score-distillation methods use image diffusion models to iteratively refine 3D models, but generating one object usually takes minutes to hours.SDS and VSD are examples of score-distillation techniques.
  • 2.2 3D Generation with Sparse View Reconstruction: Sparse-view methods generate multi-view-consistent images before reconstruction, but commonly require test-time optimization that adds computation and can compromise quality.
  • 2.3 Feed-forward 3D Generative Models: Feed-forward methods provide significantly faster generation than score-distillation and other iterative approaches.
  • 2.3 Feed-forward 3D Generative Models: CRM generates six orthographic images, CCMs, and a final textured mesh through a feed-forward pipeline taking around 10 seconds on an A800 GPU.The pipeline uses multi-view diffusion, CCM generation, and CRM reconstruction.

3 Method

CRM converts a single image into six orthographic views and CCMs, then uses a convolutional U-Net, triplane decoding, and Flexicubes to reconstruct and supervise textured meshes.

  • Multi-view Diffusion Model: Given a single image, multi-view diffusion models generate six orthographic images and canonical coordinate maps for CRM.
  • Multi-view Diffusion Model: Zero-SNR training, random resizing, and contour augmentation improve the quality and robustness of the multi-view diffusion models.
  • UNet-based Convolutional Network: A convolutional U-Net maps the six images and CCMs to a rolled-out triplane, which is reshaped into the triplane representation.
  • UNet-based Convolutional Network: The triplane projects query grid cells onto xy, xz, and yz planes, whose aggregated features are decoded into SDF, deformation, color, and Flexicubes weights.
  • Canonical Coordinates Map (CCM): CCM encodes each point’s canonical-space coordinates in three channels with values in [0, 1], supplying extra geometry information beyond RGB.
  • UNet-based Convolutional Network: The convolutional U-Net preserves input information through larger bandwidth and uses triplane-image spatial correspondence to accelerate convergence and stabilize training.The reconstruction model can use batch size 32, compared with LRM’s 1024, and its training cost is 1/8 that of LRM.
  • Flexicubes: Flexicubes features are converted into meshes by dual marching cubes, while texture is obtained by querying color at surface points.
  • Loss Function: Training supervises rendered texture with MSE and LPIPS losses and geometry with depth, mask, and mesh-quality regularization losses.

4 Experiments

The reconstruction model is trained using a filtered high-quality subset of Objaverse with rendered multi-view supervision and a compact experimental configuration.

  • The filtered Objaverse training set contains around 376k valid high-quality objects after removing scene-level objects and low-quality meshes.
  • Each shape provides 16 rendered images at 256 × 256 resolution, supplemented by six orthographic images and CCMs rendered with matching lighting and resolution.
  • The reconstruction model contains around 300M parameters and uses a Flexicubes grid size of 80.

Network Architecture

The reported implementation trains the reconstruction and diffusion components with separate multi-GPU configurations, optimizers, iteration counts, and batch settings.

  • Reconstruction Model: The reconstruction model used 8 NVIDIA A800 80GB GPUs for 6 days and 110k iterations with batch size 32.Adam optimization used learning rate 1e−4.
  • Diffusion Models: The diffusion models for six orthographic images and CCMs used 8 NVIDIA A800 80GB GPUs for 2 days and 10k iterations.Gradient accumulation of 12 produced a total batch size of 1536, with DDIM sampling for 50 steps.
  • Evaluation Setup: The evaluation compares CRM with Wonder3d, SyncDreamer, Magic123, One-2-3-45, and OpenLRM using official or project-provided implementations.OpenLRM serves as the comparison implementation because LRM is not open-sourced.

Qualitative Results

Qualitative evaluations present high-fidelity textured meshes from single images and compare CRM with prior methods on texture and geometry.

  • Mesh Comparisons: CRM produces better texture than LRM, smoother geometry than LRM, and more geometric detail than LGM.The comparison is shown for generated meshes in Fig. 6.
  • Additional Results: The method generates high-fidelity textured meshes from a single image across additional examples.These results are presented in Fig. 7.
  • Mesh Comparisons: CRM is qualitatively compared with LRM and LGM using visualized generated meshes.The comparison focuses on texture quality and geometry smoothness or detail.
  • Evaluation Protocol: The evaluation uses 30 GSO shapes rendered from a single 256 × 256 input image, with geometry and texture assessed using separate metrics.Geometry uses Chamfer Distance, Volumn IoU, and F-Score; texture uses rendered views at 512 × 512 resolution.
  • Texture Quality: The generated textured meshes surpass all baselines in appearance according to the reported texture-quality evaluation.The evaluation uses PSNR, SSIM, LPIPS, and Clip-Similarity on rendered views.

4.3 Ablation Study and Analysis

The analysis reports that CRM yields reasonable reconstruction results very early in training and attributes fast convergence to its geometry prior.

  • Early Training: Reasonable reconstruction results appear after barely 280 iterations, corresponding to only 20 minutes of training.The early-stage results are shown in Fig. 8.
  • Geometry Evaluation: Geometry quality is quantitatively compared using Chamfer Distance, Volumn IoU, and F-score on the GSO dataset.Table 1 reports these metrics for CRM and baselines.
  • Texture Evaluation: Texture quality is quantitatively compared using PSNR, SSIM, LPIPS, and Clip-Similarity on the GSO dataset.Table 2 reports the texture-quality metrics.
  • Analysis: The authors conjecture that fast convergence results from the strong geometry prior in CRM’s architecture.This is presented as a conjecture rather than an established causal result.

Training Time of CRM

CRM’s reconstruction model has substantially lower reported training cost than LRM and LGM, attributed to exploiting spatial correspondence as a strong prior.

  • Training Cost: The spatial correspondence between six orthographic images, CCMs, and triplanes serves as a strong prior that makes training easier.The authors connect this design to CRM’s smaller training time relative to the baselines.

Importance of Input CCM

The ablation shows that Canonical Coordinate Maps (CCMs) substantially improve reconstructed geometry, particularly for complex shapes.

  • Removing CCM input substantially degrades the reconstructed geometry.The RGB-only reconstruction remains reasonable but is much worse than the full model with CCMs.
  • CCMs provide important geometry information when the reconstructed geometry is complex.
  • The full model with CCM input produces better geometry than the model using only six RGB images.

Design of Multi-view Diffusion

The ablation evaluates multi-view diffusion design choices on GSO using novel-view similarity metrics and qualitative contour behavior. Zero-SNR and random resizing improve quantitative results, while contour augmentation improves a contour-sensitive back-view prediction despite not improving metrics.

  • Design of Multi-view Diffusion: The ablation study evaluates PSNR, SSIM, and LPIPS on GSO for novel-view synthesis.The study compares generated novel views with ground-truth images.
  • Design of Multi-view Diffusion: Starting from ImageDream fine-tuning with two additional views, the experiment sequentially adds the proposed training techniques.The baseline naively fine-tunes the pretrained ImageDream model.
  • Design of Multi-view Diffusion: Zero-SNR and random resizing improve the quantitative metrics in the ablation.These techniques are added sequentially to the baseline during training.
  • Design of Multi-view Diffusion: Contour augmentation does not improve quantitative metrics, but it produces a more reasonable back-view color prediction when contours affect sensitivity.The figure contrasts predictions without and with contour augmentation and illustrates how the input is augmented during training.

5 Conclusion

CRM creates high-quality textured meshes from a single image by exploiting spatial relationships between input images and the output triplane. The method is trained end to end, produces meshes in 10 seconds, and remains constrained by input-view conditions, multi-view consistency, and Flexicubes resolution.

  • 5 Conclusion: CRM creates high-quality 3D models from a single image and directly outputs textured meshes through end-to-end training.The approach uses spatial relationships between input images and the output triplane.
  • 5 Conclusion: 10 seconds: CRM generates a high-fidelity textured mesh without test-time optimization.This speed is reported alongside the method's high-fidelity output.
  • 5 Conclusion: Inputs with large elevation or different field of view sometimes produce unsatisfactory results.The paper identifies this as a limitation also present in ImageDream.
  • 5 Conclusion: Inconsistent images from the multi-view diffusion model can degrade the resulting 3D reconstruction.The paper notes that fully consistent multi-view generation is difficult to ensure.
  • 5 Conclusion: An 80-cell Flexicubes grid cannot represent very detailed geometry because of limited computing resources.The stated grid size is 80.
  • 5 Conclusion: CRM may be used to generate malicious or fake 3D content, requiring additional caution.The paper lists this as a potential negative impact shared with many generative models.
Loading 2403.05034v1…