Source-linked AI summary
Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, Shenghua Gao
TL;DR
Directly generating 3D shapes from images or text is difficult because 3D shape distributions differ from those modalities and the task is ill-posed. The paper aligns shapes, images, and text before generation, using SITA-VAE and ASLDM to produce higher-quality, more diverse, and more semantically consistent conditional shapes.
Problem
Conditional 3D generation from images or text is limited by complicated shape topologies, an ill-posed mapping, and distribution differences between 3D shapes and image or text spaces.
Method
The framework aligns 3D shapes with pre-aligned image and text features using SITA-VAE, then learns image- or text-to-shape-latent mapping with an aligned shape latent diffusion model.
Results
The method generates higher-quality and more diverse 3D shapes with greater semantic consistency to conditional image or text inputs.
Takeaways & Limitations
A shape-image-text-aligned space supports cross-modal 3D shape generation by narrowing the domain gap between shape latents and image or text spaces.
Takeaways & Limitations
The method requires ground-truth 3D shapes for training and converts meshes into watertight occupancy fields, which can degrade original mesh quality.
Abstract
from arXiv · showhide
We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to producing inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Our extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.
1 Introduction
The paper addresses cross-modal 3D shape generation by aligning shape, image, and text representations before generation. Its SITA-VAE and ASLDM framework is designed to produce detailed, diverse shapes that conform semantically to image or text conditions.
- Motivation: Direct image- or text-to-3D generation is difficult because 3D shapes have complex topologies and distributions that differ substantially from 2D images and text.The problem is also ill-posed because a single image or description can correspond to multiple 3D shapes.
- Method: SITA-VAE aligns 3D shape embeddings with pre-aligned CLIP image/text features and decodes them into neural occupancy or SDF fields.A perceiver-based transformer encodes shapes, while a transformer-based implicit decoder reconstructs high-quality 3D shapes.
- Method: ASLDM learns the probabilistic mapping from image or text conditions to the aligned shape latent space for conditional 3D generation.The aligned representation is intended to close the domain gap before diffusion-based sampling.
- Experiments: The framework is evaluated on ShapeNet and a collected 3D Cartoon Monster dataset with geometric details.The authors state that the codes will be publicly available.
- Results: The authors report higher-quality, more diverse 3D shapes with stronger semantic conformity to conditional images or text.The reported outcome supports the effectiveness of the shape-image-text-aligned space for cross-modality generation.
2 Related Work
Related work develops topology-free neural-field representations and conditional generators, while multimodal contrastive learning has mainly focused on 3D recognition. The paper positions cross-modal 3D generation as an under-explored extension of these directions.
- 3D Shape Representation: Neural fields represent occupancy, SDF, density, or features over 3D coordinates in a topology-free form that neural networks can process.Common latent structures include global latents, regular latent grids, and sets of latent points.
- Conditional 3D Generation: Optimization-based methods use text-image matching or diffusion-model guidance to optimize per-scene neural radiance fields for 3D generation.Examples include Dreamfields, AvatarCLIP, DreamFusion, and Magic3D.
- Conditional 3D Generation: Optimization-free methods directly learn conditional generative models from paired texts or images and 3D shapes.CLIP-Forge uses normalizing flows, while AutoSDF, ShapeFormer, and 3DILG explore autoregressive modeling.
- Contrastive Learning in 3D: Contrastive learning connects vision and language in CLIP and has been extended to 3D representation learning for recognition tasks.Examples include CrossPoint and PointCLIP.
- Contrastive Learning in 3D: Figure 2's alignment-before-generation pipeline combines SITA-VAE, which aligns modalities, with ASLDM, which maps image or text conditions into shape latents.The resulting aligned embeddings are decoded into 3D shapes.
3 Our Approach
Michelangelo first aligns 3D shapes with images and texts, then learns conditional generation in that shared latent space. SITA-VAE reconstructs detailed neural fields, while ASLDM maps image or text conditions to shape latents.
- Overview: The alignment-before-generation strategy addresses inconsistency caused by distribution gaps between 3D shapes and 2D images or texts.SITA-VAE aligns the modalities before ASLDM learns conditional generation.
- SITA-VAE: SITA-VAE uses fixed CLIP image and text encoders, a trainable 3D shape encoder, and a neural field decoder.The encoders provide image, text, and shape representations for multimodal alignment and reconstruction.
- SITA-VAE: The perceiver-based 3D shape encoder injects point-cloud features into learnable global and local query tokens through cross-attention and self-attention.The global token captures high-level semantics, while local tokens contain low-level geometric structure.
- SITA-VAE: Contrastive learning projects shape, image, and text tokens into same-dimensional embeddings to align the three modalities.The shape projector is learnable, whereas the CLIP image and text projectors remain pretrained and frozen.
- SITA-VAE: The decoder reconstructs high-quality 3D neural fields from shape embeddings, while KL divergence encourages a continuous latent distribution and projection layers compact the representation.The SITA-VAE training objective combines contrastive, reconstruction, and KL-regularization terms.
- ASLDM: ASLDM learns the conditional distribution of aligned shape latents with a transformer-based latent diffusion model, then denoises sampled Gaussian noise during inference.Classifier-free guidance randomly drops conditions during training; its guidance scale trades off sampling fidelity and diversity.
4 Experiments
The experiments evaluate reconstruction and conditional generation on ShapeNet and a collected 3D Cartoon Monster dataset using quantitative, visual, and ablation studies. Results indicate that the method reconstructs shapes faithfully, maps image and text conditions effectively, and benefits from aligned-space training, CLIP, and 512 learnable query embeddings.
- Datasets, baselines, and evaluation: The evaluation uses ShapeNet and a collected 3D Cartoon Monster dataset, with comparisons against established reconstruction and conditional-generation baselines.The experiments include data preparation, baseline comparisons, metrics, numerical results, visual comparisons, and ablation studies.
- Numerical comparison: IoU results show that the method achieves the best reconstruction performance across 55 overall categories and faithfully reconstructs selected categories.IoU measures reconstruction accuracy, with larger values indicating better performance.
- Numerical comparison: The method achieves the best SI-S and ST-S scores, while P-FID and P-IS indicate realistic and diverse generated 3D shapes.SI-S and ST-S measure shape-condition similarity for images and texts; P-FID and P-IS evaluate generation quality and diversity.
- Visual comparison: Visual comparisons show smoother, more detailed, and more condition-conforming outputs than baselines for both image- and text-conditioned generation.3DILG produces over-smooth shapes with missing details, while 3DS2V produces noisy, discontinuous surfaces; the proposed method handles abstract and detailed text conditions.
- Ablation studies and analysis: Training the generative model in the aligned space produces samples that conform more closely to conditional texts than training without aligned space.The ablation compares upper samples trained in the aligned space with lower samples trained without it.
- Ablation studies and analysis: Using CLIP and 512 learnable query embeddings yields the best reported reconstruction and classification performance in the ablations.The studies assess both the vision-language model choice and the number of learnable query embeddings.
5 Disscusion and Conclusion
The framework aligns 3D shapes with images and text to support cross-modal generation while reconstructing detailed 3D shapes. Its limitations are reliance on ground-truth 3D training data and watertight mesh conversion.
- Training requires ground-truth 3D shapes, although 3D data is usually an order of magnitude smaller than 2D data.
- Representing shapes as occupancy fields requires watertight mesh conversion, which inevitably degrades the original mesh quality.
- The framework aligns 3D shapes, 2D images, and text in a compatible space for cross-modal 3D shape generation.
- The aligned representation reconstructs high-quality 3D shapes from latent embeddings while incorporating semantic information.
- The method generates higher-quality and more diverse shapes with greater semantic consistency to conditional image or text inputs.
A Training Details
Training uses separate stages for SITA-VAE and ASLDM, with specified sampling, optimization, diffusion, and classifier-free guidance settings.
- Stage 1: SITA-VAE: SITA-VAE encodes 4096 point clouds with normal features and trains for around 200,000 steps with batch size 40.
- Stage 1: SITA-VAE: SITA-VAE reconstruction samples 1024 volumetric and 1024 near-surface points from normalized watertight meshes.
- Stage 1: SITA-VAE: The SITA-VAE loss uses λc = 0.1 and λKL = 0.001.
- Stage 2: ASLDM: ASLDM uses 1000 diffusion steps with β ∈ [0.00085, 0.012], batch size 64, and around 500,000 training steps.
- Stage 2: ASLDM: Inference applies classifier-free guidance with guidance scale λ = 7.5.
B Details in zero-shot classification experiments
The zero-shot classification experiment evaluates SITA-VAE encoders on ModelNet40 by comparing shape embeddings with text-derived category representations.
- ModelNet40 contains 12311 synthetic 3D CAD models across 40 categories.
- The dataset uses 9843 models for training and 2468 models for testing.
- The experiment trains SITA-VAE on ShapeNet before directly applying its encoders to ModelNet40 test classification.
- Classification computes cosine similarity between each query shape and category text formed with the phrase “a 3D model of {}”.
- Performance is reported using top-1 and top-5 accuracy.
C Template in building shape-image-text data pair
The paper constructs shape-image-text training pairs with predefined templates, replacing a placeholder with each 3D shape’s tag.
- The template set adds the phrase “a 3D model of {}” alongside templates from previous work.
- Predefined templates are used to build shape-image-text pairs during training.
- The placeholder “{}” is replaced by the 3D shape tag in each template.
D Visualization for image/shape retrieval
Contrastive alignment lets the framework compare images with 3D shapes and retrieve database shapes that are visually similar to a query image.
- The model computes image–3D shape similarities across a database and returns the shape most similar to a query image.Figure 7 visualizes the top retrieval, marked as the golden model.
- Retrieved shapes show visual information close to the query image, indicating alignment between image information and 3D shape information.
E More visual comparison
Visual comparisons show that the framework captures image details, completes single-view inputs with plausible geometry, and maps text keywords to corresponding 3D parts.
- Image-conditioned generation: The model captures image details and generates 3D shapes faithfully while completing geometry from single-view information with plausible solutions.
- Image-conditioned generation: Figure 8 compares the framework with 3DS2V and 3DILG for image-conditioned 3D shape generation.
- Text-conditioned generation: Text-conditioned results show that the model understands language and maps keywords to corresponding parts in 3D shapes.
- Text-conditioned generation: Figure 9 presents a comparison of the framework with 3DS2V and 3DILG for text-conditioned 3D shape generation.
F Test in the wild
Tests on internet images and manually designed text indicate that the framework can handle out-of-domain visual inputs and produce shapes conforming to text keywords.
- Test in the wild: The wild-data evaluation includes images from the internet and manually designed text as model conditions.
- Conditional 3D shape generation on images from the Internet: Internet-image tests indicate that the model robustly maps some out-of-domain visual inputs to 3D shapes.The corresponding results are shown in Figure 10.
- Conditional 3D shape generation on manual input text: Manually designed text tests show that the model captures keyword information and produces shapes conforming to the text.The corresponding results are shown in Figure 11.