Source-linked AI summary
TextDeformer: Geometry Manipulation using Text Guidance
William Gao, Noam Aigerman, Thibault Groueix, Vladimir G. Kim, Rana Hanocka
TL;DR
TextDeformer addresses text-guided deformation of existing meshes while preserving their source structure and semantic correspondences. It optimizes per-triangle Jacobians with view-consistency and identity regularization, and demonstrates plausible deformations across varied shapes and prompts, from large shape changes to fine details.
Problem
Automating text-guided mesh deformation must balance plausible target shapes, high-quality geometry, and adherence to the input, despite noisy gradients and conflicting view-dependent signals.
Source: §1 · pp. 1–2Method
TextDeformer uses differentiable rendering to guide per-triangle Jacobian deformations with pretrained image encoders, view consistency, and identity regularization.
Source: §1 · pp. 1–2Results
Experiments show plausible deformations across a large class of source shapes and targets, with meaningful source-to-result correspondences and both low- and high-frequency changes.
Source: §1 · pp. 1–2Takeaways & Limitations
The framework provides controllable deformations that retain global characteristics of the source while matching highly dissimilar text targets.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present a technique for automatically producing a deformation of an input triangle mesh, guided solely by a text prompt. Our framework is capable of deformations that produce both large, low-frequency shape changes, and small high-frequency details. Our framework relies on differentiable rendering to connect geometry to powerful pre-trained image encoders, such as CLIP and DINO. Notably, updating mesh geometry by taking gradient steps through differentiable rendering is notoriously challenging, commonly resulting in deformed meshes with significant artifacts. These difficulties are amplified by noisy and inconsistent gradients from CLIP. To overcome this limitation, we opt to represent our mesh deformation through Jacobians, which updates deformations in a global, smooth manner (rather than locally-sub-optimal steps). Our key observation is that Jacobians are a representation that favors smoother, large deformations, leading to a global relation between vertices and pixels, and avoiding localized noisy gradients. Additionally, to ensure the resulting shape is coherent from all 3D viewpoints, we encourage the deep features computed on the 2D encoding of the rendering to be consistent for a given vertex from all viewpoints. We demonstrate that our method is capable of smoothly-deforming a wide variety of source mesh and target text prompts, achieving both large modifications to, e.g., body proportions of animals, as well as adding fine semantic details, such as shoe laces on an army boot and fine details of a face.
1 INTRODUCTION
TextDeformer automates text-guided deformation of existing meshes while seeking to retain their semantic correspondence and support both large shape changes and fine details. It uses Jacobians to reduce artifacts from noisy gradients, view consistency to encourage coherent geometry, and identity preservation to keep the deformation tied to the input.
1 INTRODUCTION
- The framework targets a gap between detail-preserving, low-frequency handle-based deformations and the laborious addition of geometric details.Its stated aim is to automate deformation toward a text-described target while preserving semantic correspondence.Source: §1 · p. 1
- Per-triangle Jacobians and a global Poisson solve replace direct vertex updates, favoring smooth large-scale changes and reducing localized noisy-gradient artifacts.The paper attributes this global effect to each Jacobian influencing all vertices and, consequently, all rendered pixels.Source: §1 · p. 2
- A view-consistency loss encourages vertices to have similar CLIP features across viewpoints, addressing conflicting view-dependent signals that can produce incoherent geometry.The motivating example is one viewpoint pulling a cow’s vertex toward a trunk and another toward a tusk.Source: §1 · p. 2
- An identity-preserving term keeps optimization from straying too far from the input mesh, preventing it from ignoring the input geometry.The introduction identifies adherence to the source geometry, including corresponding parts, as a required property.Source: §1 · pp. 1–2
- Experiments show plausible text-driven deformations across a large class of source shapes and targets, while retaining resemblance to the input and meaningful correspondences.The framework supports both low-frequency shape changes and high-frequency details, including elongated animal proportions and added scales.Source: §1 · pp. 1–2
2 RELATED WORK
Related work spans text-guided image and 3D synthesis, as well as neural shape deformation. TextDeformer differs by using differentiable rendering and pretrained visual models to deform explicit input geometry without requiring paired 3D training data.
2 RELATED WORK
- Text-guided 3D methods include dataset-trained approaches and zero-shot methods, but NeRF-based systems do not straightforwardly yield editable explicit meshes.Dreamfields optimizes a NeRF against text, while DreamFusion and Wang et al. distill 2D diffusion models; extracting and editing explicit meshes is not straightforward.Source: §2 · pp. 2–3
- Surface-based approaches use differentiable rendering and CLIP, with Text2Mesh stylizing a template and CLIP-Mesh deforming a sphere to synthesize new geometry.The paper frames its own focus as deforming explicit geometry rather than generating it from scratch.Source: §2 · p. 3
- Traditional handle-based and variational deformation methods span only a subset of possible deformations and limit fine-grained detail control.The cited approaches include energy-minimizing formulations and skinning with weights interpolating coordinates around handles.Source: §2 · p. 3
- Learned deformation methods often face limited semantic capability because paired shape-caption 3D datasets are scarce; TextDeformer instead uses differentiable rendering and visual models such as CLIP.The related-work discussion contrasts its approach with explicit supervision for template-shape deformation.Source: §2 · p. 3
3 METHOD
TextDeformer guides an input mesh toward a user-specified shape using text and differentiable rendering. It optimizes per-triangle Jacobians, with regularization for source preservation and consistency across views.
Deformations through Jacobians
- Per-triangle Jacobians define a deformation map reconstructed as the closest least-squares fit to those Jacobians.The Jacobians are initialized to identity, so the initial map is unchanged.Source: §3 · pp. 3–4
Text-driven deformation
- The same source can receive either local details or broad shape changes, including shoe laces, a giraffe form, and changes to a guitar body or animal shape.Fig. 5 illustrates both high- and low-frequency deformations.Source: Figure 5 · p. 4
Language Guidance
- CLIP compares rendered-shape and prompt embeddings, while relative embedding directions provide stronger guidance when the optimization landscape is unclear.The relative direction uses a base caption describing the input mesh.Source: §3 · p. 4
Text-driven deformation
- For a shared target such as “a cactus,” deformations adapt to different source meshes while retaining aspects of their input geometry.Fig. 6 shows different vases and meshes guided toward cactus and skyscraper targets.Source: Figure 6 · p. 4
Jacobian Regularization
- A Jacobian regularizer penalizes deviation from identity to keep the deformation from straying too far from the input geometry.Its strength is controlled by α.Source: §3 · p. 4
View Consistency
- Cross-view regularization encourages each vertex to have similar deep features across render viewpoints, addressing conflicting view-dependent CLIP signals.The method associates visible vertices with nearby CLIP ViT patch centers; finer-resolution features are obtained using overlapping patches and interpolated positional encodings.Source: §3 · pp. 4–5
4 EXPERIMENTS
Experiments show that TextDeformer handles a wide variety of source meshes and target prompts. The examples span adjective, related-shape, and unrelated-shape targets.
4.1 Generality of TextDeformer
- TextDeformer produces high-quality deformations across a wide variety of source meshes and target prompts.The examples include both organic and man-made shapes.Source: §4.1 · p. 5
- Adjective prompts deform cats, vases, aliens, and shoes while preserving the source shape’s overall structure.Examples include “fluffy,” “heart-shaped,” “scary,” and “high-heeled” targets.Source: §4.1 · p. 5
- Related-shape prompts transform an acoustic guitar to a banjo, a candle to an incense stick, and an Eiffel tower to a pagoda.Other examples include a vase deformed into a royal goblet.Source: §4.1 · p. 5
- Unrelated prompts deform an ant into a snake, a clothing iron into a kettle, and a cow into a turtle.These targets are described as far from the source mesh.Source: §4.1 · p. 5
4.2 Expressiveness of TextDeformer
TextDeformer produces both broad shape changes and fine details, while preserving semantic correspondence through the deformation process.
4.2 Expressiveness of TextDeformer
- TextDeformer expresses low-frequency shape changes and high-frequency details, including animal transformations, giraffe spots, and shoe laces or shoe creases.It also changes guitar-body shape while preserving the neck.Source: §4.2 · pp. 5–6
- During optimization, Max Planck’s facial features map to corresponding features in the Einstein and Obama targets at intermediate steps.The nose maps to the nose and the eyes to the eyes.Source: Figure 8 · p. 5, §4.2 · p. 6
4.3 Identity Preservation
TextDeformer adapts the deformation to the source mesh while preserving meaningful aspects of its geometry; the regularization weight controls how far the shape changes.
4.3 Identity Preservation
- When deforming varied vases into a cactus, branches grow from source appendages; skyscraper results likewise retain source-specific structures.Examples include preserving the candle’s base and the guitar’s body, while a flatter iron becomes a dome rather than a needle.Source: §4.3 · p. 6
- The Jacobian-regularization weight α controls deviation from the source: α = 25 limits meaningful change, while α = 0 may introduce artifacts.Intermediate values offer the best results in the reported examples.Source: §4.3 · p. 6
4.4 Viewpoint Consistency
Viewpoint consistency reduces unrealistic geometry, while Jacobians support coherent deformations and higher-quality surfaces than vertex displacements. Quantitative evaluation reports comparable R-Precision to CLIP-Mesh and fewer self-intersections.
4.4 Viewpoint Consistency
- Without viewpoint consistency, deformations often contain unrealistic features, including a crooked gaming-chair backrest, a skewed skyscraper tip, and inward-curved poodle backs or bat wings.The authors note that outlier sampled camera views can cause these abnormalities.Source: §4.4 · p. 6
4.5 Effect of Jacobians
- The Jacobian ablation keeps the additional losses LΔP and LVC while replacing Jacobians with vertex displacements.This isolates the deformation representation in the comparison.Source: §4.5 · p. 6
- Replacing Jacobians with vertex displacements drastically reduces surface quality, causing inward collapse, template-violating details, and self-intersections.The ablation figure contrasts vertex displacements in the top row with Jacobians in the bottom row.Source: §4.5 · pp. 6–7, Figure 10 · p. 6
- Gradients propagated through Jacobians influence a large surface area, yielding globally coherent deformations unlike local vertex updates.In the Obama example, vertex displacements produce spurious face mirroring on the back of the head, which Jacobians avoid.Source: §4.5 · p. 7
- TextDeformer and CLIP-Mesh achieve quantitatively comparable R-Precision; TextDeformer produces higher-quality geometry with significantly fewer self-intersections, and all methods outperform Text2Mesh in R-Precision.The comparison combines the reported retrieval evaluation with qualitative and quantitative geometry-quality findings.Source: Table 1 · p. 7
4.6 Qualitative Comparisons
Across five text prompts, TextDeformer produces more semantically accurate deformations than CLIP-Mesh and smoother geometry in examples where CLIP-Mesh’s results are semantically correct.
4.6 Qualitative Comparisons
- Across five prompts, TextDeformer produces more semantically accurate results than CLIP-Mesh, including a more elongated alligator, larger chair, and rounder Chinese lantern.For camel and octopus prompts, CLIP-Mesh is semantically correct but produces irregular triangulation and self-intersections, whereas TextDeformer produces smoother, more realistic geometry.Source: §4.6 · p. 7
4.7 Quantitative Evaluation
The evaluation compares TextDeformer with CLIP-Mesh and Text2Mesh, and tests view-consistency and vertex-displacement variants. Qualitative results favor TextDeformer’s surface quality, while vertex-displacement pipelines can disregard mesh triangulation.
4.7 Quantitative Evaluation
- TextDeformer produces more semantically correct, higher-quality surfaces than CLIP-Mesh in the shown examples, while Text2Mesh fails to produce semantically meaningful deformations.The prompts include an alligator, camel, comfortable chair, Chinese lantern, and octopus.Source: Figure 11 · p. 7
- View-consistency loss increases retrieval quality, while vertex-displacement pipelines achieve the highest R-Precision without respecting the initial template geometry.All tested methods outperform Text2Mesh on the retrieval task.Source: §4.7 · p. 7
- Vertex-displacement pipelines disregard the shape’s triangulation, and the shown vertex-optimized variant introduces further surface distortion and self-intersections.The comparison also shows distortion on the goggles’ temple arms when view-consistency losses are removed.Source: §4.7 · p. 8, Figure 12 · p. 8
4.8 Qualitative Comparison with Stable Dreamfusion
In the qualitative comparison, DreamFusion meshes have less smooth surfaces than TextDeformer’s Jacobian-based deformations. DreamFusion also often shows the Janus effect, while TextDeformer’s View-Consistency loss helps produce more coherent geometry.
4.8 Qualitative Comparison with Stable Dreamfusion
- DreamFusion’s extracted geometries lack the smoothness of TextDeformer’s Jacobian-based deformations.The comparison uses DreamFusion’s extracted geometry and a representative neural rendering.Source: §4.8 · p. 8
- DreamFusion often exhibits the Janus effect even with prompt augmentation, whereas TextDeformer’s View-Consistency loss helps produce more coherent geometry.The comparison describes the Janus effect as repeated or incoherent geometry across views.Source: §4.8 · p. 8
- The comparison presents extracted DreamFusion geometry alongside a representative neural-rendering image, and shows TextDeformer’s initial mesh.Source: §4.8 · p. 8
5 DISCUSSION AND FUTURE WORK
TextDeformer uses per-face Jacobians for zero-shot text-guided mesh deformation, aiming to produce smooth geometry while retaining source characteristics. Its view-consistency and identity regularization address coherence and deformation magnitude, and future work includes learning prompt-driven deformations.
5 DISCUSSION AND FUTURE WORK
- Per-face Jacobians predict smooth deformations that retain source characteristics and mitigate local artifacts commonly caused by vertex displacements.TextDeformer does not require 3D training data or annotations.Source: §5 · p. 8
- View-consistency loss reduces visual artifacts by discouraging view-specific geometry and keeping a region roughly similarly interpreted across viewpoints.The method also includes identity regularization to control deformation magnitude.Source: §5 · p. 8
- TextDeformer is described as retaining global source-shape characteristics while matching highly dissimilar targets, with user-controllable, expressive deformations.The comparison with Stable Dreamfusion notes heavy surface artifacts and frequent Janus effects in Dreamfusion outputs, which View-Consistency helps alleviate for TextDeformer.Source: §5 · p. 8, Figure 13 · p. 8
- Future work would learn prompt-driven deformations across shapes, potentially predicting Jacobians in one feed-forward pass and improving results through neural regularization.The authors also propose connecting the method to a retrieval module for exploring source and prompt combinations.Source: §5 · p. 8