Source-linked AI summary
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, Tiberiu Popa
TL;DR
The paper addresses text-to-3D generation without large paired 3D-text datasets or 3D supervision. It directly optimizes a subdivided mesh, texture map, and normal map using CLIP scores from differentiably rendered images, with regularization and augmentation techniques. The resulting assets are diverse, deployable meshes, and the method is reported to be 100 times faster than Dreamfields in similar configurations, while remaining limited by template genus and CLIP-induced artifacts.
Problem
Text-to-3D generation lacks large varied datasets pairing 3D objects with natural-language descriptions, while existing approaches may not produce directly usable meshes and texture maps.
Method
CLIP-Mesh directly optimizes a Loop-subdivision control mesh with texture and normal maps using CLIP scores from differentiably rendered images, augmented by regularization and embedding-prior techniques.
Results
The method generates diverse textured mesh assets, outperforms Dreamfields across the reported generation and evaluation models, and takes 50 minutes per shape on a single 16GB NVIDIA P100 GPU versus over 24 hours on four NVIDIA A100 GPUs for Dreamfields.
Takeaways & Limitations
The generated meshes, texture maps, and normal maps can be directly loaded as assets in games and modeling applications.
Takeaways & Limitations
The generated object’s genus is set by the initial template mesh, and CLIP can project image artifacts such as people or text onto the mesh.
Abstract
from arXiv · showhide
We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without any 3D supervision our method deforms the control shape of a limit subdivided surface along with its texture map and normal map to obtain a 3D asset that corresponds to the input text prompt and can be easily deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimization on mesh parameters directly to generate shape, texture or both. To constrain the optimization to produce plausible meshes and textures we introduce a number of techniques using image augmentations and the use of a pretrained prior that generates CLIP image embeddings given a text embedding.
1 INTRODUCTION
CLIP-Mesh targets accessible text-driven 3D asset creation by directly optimizing mesh geometry and maps with differentiable rendering, without 3D supervision. Its regularization, render augmentations, and embedding prior address the scarcity of varied text-aligned 3D data and unstable mesh optimization.
- Motivation: Existing mesh-based 3D assets require time-consuming creation and texturing, while shape-generation methods often produce representations needing artifact-prone mesh conversion.These methods commonly use point clouds, voxel grids, or implicit functions and remain restricted to fixed shape categories.
- Motivation: The target asset format is intended to improve accessibility and integration because meshes with texture maps can be loaded into existing game engines and modeling software.The paper frames arbitrary text-described shape generation as a way to broaden 3D asset development.
- Motivation: Large varied datasets pairing 3D objects with natural-language descriptions are scarce compared with large-scale image-text datasets.The paper contrasts roughly 50 object categories in Shapenet and CO3D with 21,000 in Imagenet-21K and CLIP’s 400 million image-text pairs.
- Approach: CLIP-Mesh renders a mesh differentiably and optimizes its shape and texture to maximize agreement between rendered images and the input text prompt.Naive optimization can produce tangled, noisy meshes, motivating additional constraints and techniques.
- Contributions: The method directly generates 3D meshes with texture and normal maps using zero-shot text guidance and a differentiable renderer.The contributions also include Loop subdivision as an implicit regularizer, render augmentations, and a text-to-image embedding prior.
2 RELATED WORK
Prior text-to-3D methods commonly depend on paired 3D-text data, category-limited 3D training, or stylization of predefined shapes. CLIP-Mesh instead belongs to the zero-shot line that optimizes directly toward text while producing deployable meshes and maps.
- CLIP guidance: CLIP-guided 2D image methods demonstrate text conditioning through a shared image-text embedding space, which provides context for applying CLIP-based guidance to 3D rendering.The cited examples include StyleCLIP, VQGAN-CLIP, and GLIDE.
- Text-to-3D: Earlier text-to-3D methods trained joint 3D-shape and text embeddings with generative adversarial networks, but were not zero-shot and depended on paired datasets.Their applicability was limited by the availability of matched 3D models and text descriptions.
- Text-to-3D: CLIP-Forge reduces dependence on paired text and 3D data but remains restricted by the categories in its 3D training data and does not produce meshes or textures.The paper presents these constraints as only a partial solution to text-to-3D generation.
- Stylization: Other approaches stylize predefined human shapes or arbitrary meshes to match text prompts rather than generating the paper’s directly deployable mesh assets.The related work distinguishes stylization from the paper’s direct generation objective.
- Zero-shot generation: Dreamfields performs zero-shot text-guided generation with a NeRF, but does not directly generate a mesh and incurs substantial computation through raycasting and neural-parameter optimization.The paper reports that its figures were generated on a single 16GB GPU, contrasting the computational setup with Dreamfields.
3 METHOD
CLIP-Mesh optimizes a subdivided mesh, texture map, and normal map directly from rendered views compared with text and generated image embeddings. Differentiable rendering, smooth subdivision, regularization, camera/view augmentations, and a diffusion prior constrain the optimization toward plausible text-aligned assets.
- 3 METHOD: The model comprises a Loop subdivision mesh, texture map, and normal map optimized as the principal 3D representation.Control vertices are subdivided into the surface used for rendering.
- 3 METHOD: A differentiable renderer produces views from sampled camera positions, allowing gradients to optimize mesh vertices, texture, and normals.Camera azimuth spans 0°–360°, while elevation sampling emphasizes a front view but includes other elevations for texture coverage.
- 3 METHOD: The Loop subdivision limit surface acts as an implicit smoothness regularizer, while a Laplacian term keeps each vertex near the average position of its one-ring neighbors.The Laplacian regularizer weight starts high and decays toward a minimum as the shape stabilizes.
- 3 METHOD: Rendered images and the text prompt are encoded by CLIP, whose normalized embeddings provide cosine-similarity losses for optimization.Differentiability of both subdivision-surface computation and rendering makes the full pipeline differentiable through the chain rule.
- 3 METHOD: A diffusion prior samples CLIP image embeddings conditioned on the text embedding, and its loss can be combined with or used instead of the direct CLIP loss.The prior is pretrained on 400 million image-text pairs and supplies embeddings in the same space as rendered-image encodings.
- 3 METHOD: Random camera fields of view, distances, backgrounds, and object positions reduce reliance on texture, background, or image centering during CLIP optimization.The camera field of view varies from 30° to 60° and distance from 3.0 to 7.0.
4 RESULTS AND EVALUATIONS
The evaluation demonstrates diverse single- and multi-object mesh generation, comparisons with Dreamfields, and improvements from the proposed pipeline components. The method also supports controllable optimization of multiple objects, fixed geometry, textures, and normal maps.
- Evaluation setup: The evaluation covers single-object generation, comparisons with Jain et al. (2021), complex modeling scenarios, quantitative tests, and ablations.The quantitative evaluation uses two shapes per caption across 153 captions and evaluates held-out-pose CLIP-R precision.
- 4.1 Single Object Generation: The method generates diverse household objects and recognizable landmarks from text prompts using CLIP ViT/B-32.Figures include household objects, varied shapes without textures, and landmarks such as the Pyramid of Giza, Eiffel Tower, and Taj Mahal.
- 4.1 Single Object Generation: The method is faster than Dreamfields by a factor of 100, requiring 50 minutes per shape on one NVIDIA P100 GPU versus over 24 hours on four NVIDIA A100 GPUs.The authors attribute this difference to fewer optimized parameters and faster rasterization-based rendering.
- 4.2 Complex Modeling Scenarios: The method supports simultaneous optimization of multiple objects while freezing the geometry or appearance of selected scene elements.Examples include a boat with a red lighthouse and scenes where chairs, monitors, or other objects are generated alongside fixed planes or furniture.
- 4.4 Ablation Studies: Ablations show that limit subdivision, image augmentations, higher-resolution rendering, and prior loss improve results, with center-offset augmentation providing the largest augmentation boost.Higher-resolution rendering helps all evaluated models except the largest ViT-L/14, for which it reduces performance; the prior gives the best overall results.
5 CONCLUSIONS, LIMITATIONS AND FUTURE WORK
The method generates diverse, directly usable 3D assets from text prompts, but its outputs remain constrained by the initial mesh genus and can inherit image-model artifacts.
- The generated assets include a mesh, texture map, and normal map for direct use in games and modeling applications.
- The generated object's genus is determined by the initial template mesh, with transparency providing only a partial workaround.
- The ablation study reports systematically improved R-Precision as limit subdivision, augmentation, large rendering, and the generative prior are added to the baseline.Higher R-Precision is better.
- CLIP-based generation can project image artifacts onto meshes, including small people or written labels on reconstructed objects.
- Future work targets stronger shape-based constraints and greater user control over the generative process.