CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

• Text-to-shape generation is challenging because, unlike text-to-image generation, large-scale paired text–shape datasets are impractical to acquire.Source: §1 · p. 1
• CLIP-Forge uses automatically rendered images of 3D shapes and pre-trained image–text joint embeddings to connect shapes with natural language.Source: §1 · p. 1, Figure 2 · p. 2
• The method first learns a shape latent space with an autoencoder, then trains a conditional normalizing flow using image features to model shape embeddings.Source: §1 · p. 2
• At inference, text embeddings replace image embeddings as the flow condition, producing shape embeddings that the decoder converts into 3D shapes.Source: §1 · p. 2
• CLIP-Forge generates 3D shapes directly without paired text–shape labels and supports efficient, multi-shape generation for a text prompt.Source: §1 · p. 2
• The method is evaluated extensively in qualitative and quantitative zero-shot generation settings.Source: §1 · p. 2

Zero-Shot Learning

• Zero-shot learning uses external knowledge to make predictions for classes or outputs not observed during training, and CLIP connects text and images in a shared latent space.Source: §2 · p. 2
• CLIP-Forge applies zero-shot generation to 3D shapes, where prior text-to-image methods benefit from abundant paired data unavailable for text–shape learning.Source: §2 · p. 2

Stage 1: Shape Autoencoder

• The first stage trains a voxel-encoder and implicit-decoder autoencoder to obtain a latent shape embedding for each shape.Source: §3 · p. 3, §3.1 · p. 3
• Gaussian noise is added to the shape embedding, and the implicit decoder predicts occupancies from the embedding concatenated with 3D query points.Source: §3.1 · p. 3

Stage 2: Conditional Normalizing Flow

• The second stage trains a RealNVP conditional normalizing flow to model shape embeddings from rendered-image features.Source: §3.2 · pp. 3–4

Inference

• At inference, text features condition the flow, whose reverse path samples shape embeddings; repeated sampling yields multiple shapes for one query.Source: §3.3 · p. 4

Experimental Setup

• The evaluation uses ShapeNet(v2), with reconstruction, generation quality, category diversity, and human perceptual criteria.Generation quality uses FID and MMD; category diversity uses Acc.Source: §4 · pp. 4–5

Comparison with Supervised Models

• CLIP-Forge significantly outperforms both supervised baselines across all reported evaluation metrics.The supervised baselines use the paired Text2Shape dataset and share the CLIP text encoder and occupancy-network components for comparison.Source: §4.1 · p. 5

Qualitative Results

• CLIP-Forge generates multiple and diverse shapes from a single text query.The qualitative results also cover category, sub-category, semantic-word, and shape-attribute prompts.Source: §4.2 · p. 5
• Interpolation between two text inputs produces results suggesting a smooth conditioning space.The method also generates shapes from common names, sub-categories, and attributes.Source: §4.2 · p. 5, Figure 7 · p. 7

Human Perceptual Evaluation

• Detailed prompts were correctly identified by human evaluators for 70.83% of image pairs.Attribute prompts were more readily recognized than sub-category prompts; “A circular bench” was recognized by 8/9 evaluators, while “A laboratory bench” was recognized by none.Source: §4.3 · p. 5

Prompt Selection

• Prompt prefix selection affects generation quality and diversity, with some prefixes producing worse visual results.The authors identify prompt engineering or prompt tuning as a future research direction.Source: §4.4 · p. 5

Stage 1 Autoencoder Design Choice

• Adding Gaussian noise to the autoencoder latent space improves reconstruction, generation quality, and shape diversity.The model also remains reasonable with a smaller latent size of 128 and can use point-cloud encoder inputs.Source: §5.1 · p. 7, Table 3 · p. 8

Stage 2 Prior Design Choice

• Conditioning the affine coupling layers is more effective than conditioning only the prior network.The authors attribute this to conditioning at multiple coupling layers rather than only once.Source: §5.2 · p. 7, Table 4 · p. 8
• Dimension masking is more effective than checkered masking, while MAF is less effective than RealNVP.The remaining ablations use dimension-masked RealNVP.Source: §5.2 · pp. 7–8

Views and CLIP Architectures

• Using more rendering views generally improves generation quality and diversity.The authors connect this to better capturing CLIP’s output distribution across viewpoints.Source: §5.3 · p. 8, Table 5 · p. 8
• ViT-based CLIP models outperform ResNet-based CLIP models, while increasing ViT size from ViT-B/32 to ViT-B/16 has little effect on text-based generation.The authors hypothesize that patch-based ViT methods focus more on foreground objects in image renderings.Source: §5.4 · p. 8, Table 5 · p. 8

Generation quality

• Generation quality remains limited, motivating future integration with local implicit methods.Source: §6 · p. 8

Representation scope

• The current method focuses on geometry, leaving texture integration as future work.Source: §6 · p. 8

Data distribution

• CLIP-Forge is constrained by CLIP’s trained data distribution, suggesting fine-tuning for specific datasets.Source: §6 · p. 8

Potential impact

• Language-driven 3D modeling may lower technical barriers and reduce tedious tasks for modelers and animators.Source: §6 · p. 8
• The authors frame democratized 3D content creation as a greater benefit for the general public.Source: §6 · p. 8

• CLIP-Forge efficiently generates multiple 3D shapes while preserving semantic meaning from a text prompt.Source: §7 · p. 8
• Training requires no text-shape labels, enabling use of shape-only datasets such as ShapeNet.Source: §7 · p. 8
• The model generates results on other representations, including point clouds.Source: §7 · p. 8
• The authors thoroughly study different components of the method.Source: §7 · p. 8

Visualization

• Visualizations use 643 resolution shapes and select category-specific thresholds that yield the best visual result.Source: A. Architecture and Experiment Details · p. 9

Evaluation procedure

• Ablation experiments average results across three second-stage seeds, whereas main experiment results use the best seed.Source: A. Architecture and Experiment Details · p. 9

Encoders

• VoxEnc and ResVoxEnc use four 3D convolution layers followed by a linear layer, while the point-cloud encoder is inspired by PointNet.Source: A. Architecture and Experiment Details · p. 9

Decoders

• RN-OccNet and CBN-OccNet are OccNet-inspired decoders, and FoldingNet provides the point-cloud generation decoder.Source: A. Architecture and Experiment Details · p. 9

Prior model

• RealNVP uses five coupling-layer blocks with translation, scale, batch normalization, inverted masking, and 1024-dimensional hidden vectors.Source: A. Architecture and Experiment Details · p. 9

Evaluation setup

• The comparison uses the original text-query subset and ShapeNet v2 test data to evaluate generation across categories and queries.Source: B. Comparison with Supervised Models · p. 9

Reported comparisons

• Table 7 provides additional detailed comparisons with supervised models, while Table 6 lists the text queries used in the evaluation.Source: Table 7 · p. 9, Table 6 · p. 9

Datasets

• T2S contains annotated text-shape descriptions mainly concerning texture, whereas SN13 is a 13-category ShapeNet v2 subset.Source: B. Comparison with Supervised Models · p. 10

Supervised baseline

• Because T2S labels cover only chairs and tables, its supervised baseline connects a pretrained CLIP text encoder to a pretrained occupancy decoder using latent-space L2 loss.Source: B. Comparison with Supervised Models · p. 10
• For SN13, the supervised model uses category names as training text because text labels are unavailable.Source: B. Comparison with Supervised Models · p. 10

Results

• CLIP-Forge outperforms the supervised method on SN13, indicating stronger zero-shot generalization and better scaling with more data without text-shape labels.Source: B. Comparison with Supervised Models · p. 10

• CLIP-Forge generates shapes across all ShapeNet categories, according to the category-wise classifier results.Source: C. Category-wise Accuracy Results · p. 10
• Accuracy is higher for airplane and car than for boat and loudspeaker.Source: C. Category-wise Accuracy Results · p. 10
• The authors hypothesize that category differences may reflect unequal numbers of training examples.Source: C. Category-wise Accuracy Results · p. 10

Effect of Threshold Parameter

• Different shapes require different occupancy thresholds depending on category and local geometric details.The authors use a constant threshold of 0.05 for metrics and human evaluations but grid-search the threshold for visual results.Source: E. Effect of Threshold Parameter · p. 10
• Out-of-distribution queries prompt shapes that the method tries to align with its training dataset.The authors identify broader out-of-distribution generalization as future work.Source: F. Out of Distribution Generation · p. 10
• Prefix changes produce small variations, while the prefix “a rendering of” can worsen visual results.Source: G. Visual Results for Different Prefixes · p. 10
• Longer descriptive prompts capture some shape-related attributes, but color descriptions are not captured and may bias generation.The authors suggest semi-supervised learning for finer text control.Source: H. Visual Results for more Descriptive… · p. 10
• Qualitative results cover sub-categories, synonyms, attributes, common names, multiple-shape generation, and interpolation, but some details and sub-categories remain inaccurate.Examples include incomplete details for “a swivel chair” and inaccurate shapes for “an operating table.”Source: §I · pp. 10–11

Human Perceptual Evaluation

• Crowd workers viewed image pairs consisting of a category-name generation and a detailed-prompt generation.Source: J. Human Perceptual Evaluation · pp. 11, 13
• Each detailed prompt contained either subcategory or attribute information and was shown alongside the prompt itself.Source: J. Human Perceptual Evaluation · p. 13
• Nine crowd workers evaluated each image pair by selecting which image best described the detailed text prompt.Source: J. Human Perceptual Evaluation · p. 13
• Each detailed prompt received a score from 0 to 9 based on how often its generated image was selected.Source: J. Human Perceptual Evaluation · p. 13