Source-linked AI summary
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, Kamal Rahimi Malekshan
TL;DR
Text-to-shape generation lacks the large paired datasets available for text-to-image generation. CLIP-Forge addresses this with a two-stage model trained on unlabelled shapes and a pre-trained image–text network, and demonstrates zero-shot generation with efficient inference and multiple outputs per prompt.
Problem
Text-to-shape generation remains difficult because large-scale paired text–shape data is unavailable, unlike in text-to-image generation.
Method
CLIP-Forge learns a shape latent space with an autoencoder and trains a conditional normalizing flow using rendered-shape image features, replacing them with text features at inference.
Results
CLIP-Forge demonstrates promising zero-shot generation qualitatively and quantitatively, with efficient generation and multiple shapes available for each text prompt.
Takeaways & Limitations
The method can leverage shape-only datasets and generate semantically aligned 3D shapes without text–shape labels or inference-time optimization.
Takeaways & Limitations
Generation quality remains limited, the method focuses on geometry rather than texture, and performance is constrained by CLIP’s trained data distribution.
Abstract
from arXiv · showhide
Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data at a large scale. We present a simple yet effective method for zero-shot text-to-shape generation that circumvents such data scarcity. Our proposed method, named CLIP-Forge, is based on a two-stage training process, which only depends on an unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our method has the benefits of avoiding expensive inference time optimization, as well as the ability to generate multiple shapes for a given text. We not only demonstrate promising zero-shot generalization of the CLIP-Forge model qualitatively and quantitatively, but also provide extensive comparative evaluations to better understand its behavior.
1. Introduction
Text-to-shape generation is difficult because large-scale paired text–shape data is unavailable. CLIP-Forge bridges this gap by training on rendered shapes and aligned image–text embeddings, enabling direct zero-shot generation.
- Text-to-shape generation is challenging because, unlike text-to-image generation, large-scale paired text–shape datasets are impractical to acquire.
- CLIP-Forge uses automatically rendered images of 3D shapes and pre-trained image–text joint embeddings to connect shapes with natural language.
- The method first learns a shape latent space with an autoencoder, then trains a conditional normalizing flow using image features to model shape embeddings.
- At inference, text embeddings replace image embeddings as the flow condition, producing shape embeddings that the decoder converts into 3D shapes.
- CLIP-Forge generates 3D shapes directly without paired text–shape labels and supports efficient, multi-shape generation for a text prompt.
- The method is evaluated extensively in qualitative and quantitative zero-shot generation settings.
2. Related Work
Prior work established zero-shot generation and image–text alignment, while CLIP-Forge extends these ideas to 3D shapes using a two-stage autoencoder and conditional normalizing flow.
- Zero-Shot Learning: Zero-shot learning uses external knowledge to make predictions for classes or outputs not observed during training, and CLIP connects text and images in a shared latent space.
- Zero-Shot Learning: CLIP-Forge applies zero-shot generation to 3D shapes, where prior text-to-image methods benefit from abundant paired data unavailable for text–shape learning.
- Stage 1: Shape Autoencoder: The first stage trains a voxel-encoder and implicit-decoder autoencoder to obtain a latent shape embedding for each shape.
- Stage 1: Shape Autoencoder: Gaussian noise is added to the shape embedding, and the implicit decoder predicts occupancies from the embedding concatenated with 3D query points.
- Stage 2: Conditional Normalizing Flow: The second stage trains a RealNVP conditional normalizing flow to model shape embeddings from rendered-image features.
- Inference: At inference, text features condition the flow, whose reverse path samples shape embeddings; repeated sampling yields multiple shapes for one query.
4. Experiments
Experiments evaluate CLIP-Forge on ShapeNet using reconstruction, generation, diversity, and human perceptual criteria, alongside supervised baselines. The method generates diverse, semantically responsive shapes from text and supports multiple shapes and interpolations.
- Experimental Setup: The evaluation uses ShapeNet(v2), with reconstruction, generation quality, category diversity, and human perceptual criteria.Generation quality uses FID and MMD; category diversity uses Acc.
- Comparison with Supervised Models: CLIP-Forge significantly outperforms both supervised baselines across all reported evaluation metrics.The supervised baselines use the paired Text2Shape dataset and share the CLIP text encoder and occupancy-network components for comparison.
- Qualitative Results: CLIP-Forge generates multiple and diverse shapes from a single text query.The qualitative results also cover category, sub-category, semantic-word, and shape-attribute prompts.
- Qualitative Results: Interpolation between two text inputs produces results suggesting a smooth conditioning space.The method also generates shapes from common names, sub-categories, and attributes.
- Human Perceptual Evaluation: Detailed prompts were correctly identified by human evaluators for 70.83% of image pairs.Attribute prompts were more readily recognized than sub-category prompts; “A circular bench” was recognized by 8/9 evaluators, while “A laboratory bench” was recognized by none.
- Prompt Selection: Prompt prefix selection affects generation quality and diversity, with some prefixes producing worse visual results.The authors identify prompt engineering or prompt tuning as a future research direction.
5. Ablation Studies
Ablation studies examine the autoencoder, conditional flow, rendering views, and CLIP architecture. Results favor noise augmentation, affine-layer conditioning, dimension masking with RealNVP, multiple views, and ViT-based CLIP models.
- Stage 1 Autoencoder Design Choice: Adding Gaussian noise to the autoencoder latent space improves reconstruction, generation quality, and shape diversity.The model also remains reasonable with a smaller latent size of 128 and can use point-cloud encoder inputs.
- Stage 2 Prior Design Choice: Conditioning the affine coupling layers is more effective than conditioning only the prior network.The authors attribute this to conditioning at multiple coupling layers rather than only once.
- Stage 2 Prior Design Choice: Dimension masking is more effective than checkered masking, while MAF is less effective than RealNVP.The remaining ablations use dimension-masked RealNVP.
- Views and CLIP Architectures: Using more rendering views generally improves generation quality and diversity.The authors connect this to better capturing CLIP’s output distribution across viewpoints.
- Views and CLIP Architectures: ViT-based CLIP models outperform ResNet-based CLIP models, while increasing ViT size from ViT-B/32 to ViT-B/16 has little effect on text-based generation.The authors hypothesize that patch-based ViT methods focus more on foreground objects in image renderings.
6. Limitations and Future Work
The authors identify limitations in generation quality, representation scope, and dependence on CLIP’s training distribution. They also discuss possible benefits and disruption from lowering barriers to 3D content creation.
- Generation quality: Generation quality remains limited, motivating future integration with local implicit methods.
- Representation scope: The current method focuses on geometry, leaving texture integration as future work.
- Data distribution: CLIP-Forge is constrained by CLIP’s trained data distribution, suggesting fine-tuning for specific datasets.
- Potential impact: Language-driven 3D modeling may lower technical barriers and reduce tedious tasks for modelers and animators.
- Potential impact: The authors frame democratized 3D content creation as a greater benefit for the general public.
7. Conclusion
CLIP-Forge efficiently generates multiple 3D shapes from text while preserving semantic meaning, without requiring text-shape labels. The method can also extend to other 3D representations such as point clouds.
- CLIP-Forge efficiently generates multiple 3D shapes while preserving semantic meaning from a text prompt.
- Training requires no text-shape labels, enabling use of shape-only datasets such as ShapeNet.
- The model generates results on other representations, including point clouds.
- The authors thoroughly study different components of the method.
A. Architecture and Experiment Details
The appendix specifies the encoders, decoders, point-cloud components, flow priors, visualization procedure, and threshold selection used in CLIP-Forge experiments.
- Visualization: Visualizations use 643 resolution shapes and select category-specific thresholds that yield the best visual result.
- Evaluation procedure: Ablation experiments average results across three second-stage seeds, whereas main experiment results use the best seed.
- Encoders: VoxEnc and ResVoxEnc use four 3D convolution layers followed by a linear layer, while the point-cloud encoder is inspired by PointNet.
- Decoders: RN-OccNet and CBN-OccNet are OccNet-inspired decoders, and FoldingNet provides the point-cloud generation decoder.
- Prior model: RealNVP uses five coupling-layer blocks with translation, scale, batch normalization, inverted masking, and 1024-dimensional hidden vectors.
B. Comparison with Supervised Models
The supervised comparison uses T2S and SN13 settings with different available supervision, while CLIP-Forge is evaluated against supervised baselines on common text queries and ShapeNet data. The reported comparison favors CLIP-Forge’s zero-shot generalization and scaling without text-shape labels.
- Evaluation setup: The comparison uses the original text-query subset and ShapeNet v2 test data to evaluate generation across categories and queries.
- Reported comparisons: Table 7 provides additional detailed comparisons with supervised models, while Table 6 lists the text queries used in the evaluation.
- Datasets: T2S contains annotated text-shape descriptions mainly concerning texture, whereas SN13 is a 13-category ShapeNet v2 subset.
- Supervised baseline: Because T2S labels cover only chairs and tables, its supervised baseline connects a pretrained CLIP text encoder to a pretrained occupancy decoder using latent-space L2 loss.
- Supervised baseline: For SN13, the supervised model uses category names as training text because text labels are unavailable.
- Results: CLIP-Forge outperforms the supervised method on SN13, indicating stronger zero-shot generalization and better scaling with more data without text-shape labels.
C. Category-wise Accuracy Results
CLIP-Forge generates shapes across all ShapeNet categories, but category-wise accuracy varies, with airplane and car outperforming boat and loudspeaker.
- CLIP-Forge generates shapes across all ShapeNet categories, according to the category-wise classifier results.
- Accuracy is higher for airplane and car than for boat and loudspeaker.
- The authors hypothesize that category differences may reflect unequal numbers of training examples.
E. Effect of Threshold Parameter
Threshold choice strongly affects occupancy-based results and visual quality, while additional experiments examine out-of-distribution queries, prefixes, descriptive prompts, and qualitative shape coverage.
- Effect of Threshold Parameter: Different shapes require different occupancy thresholds depending on category and local geometric details.The authors use a constant threshold of 0.05 for metrics and human evaluations but grid-search the threshold for visual results.
- Effect of Threshold Parameter: Out-of-distribution queries prompt shapes that the method tries to align with its training dataset.The authors identify broader out-of-distribution generalization as future work.
- Effect of Threshold Parameter: Prefix changes produce small variations, while the prefix “a rendering of” can worsen visual results.
- Effect of Threshold Parameter: Longer descriptive prompts capture some shape-related attributes, but color descriptions are not captured and may bias generation.The authors suggest semi-supervised learning for finer text control.
- Effect of Threshold Parameter: Qualitative results cover sub-categories, synonyms, attributes, common names, multiple-shape generation, and interpolation, but some details and sub-categories remain inaccurate.Examples include incomplete details for “a swivel chair” and inaccurate shapes for “an operating table.”
J. Human Perceptual Evaluation
Human perceptual evaluation compares shapes generated from category names with shapes generated from detailed prompts containing subcategory or attribute information.
- Human Perceptual Evaluation: Crowd workers viewed image pairs consisting of a category-name generation and a detailed-prompt generation.
- Human Perceptual Evaluation: Each detailed prompt contained either subcategory or attribute information and was shown alongside the prompt itself.
- Human Perceptual Evaluation: Nine crowd workers evaluated each image pair by selecting which image best described the detailed text prompt.
- Human Perceptual Evaluation: Each detailed prompt received a score from 0 to 9 based on how often its generated image was selected.