PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Mesh: Text-Driven Neural Stylization for Meshes

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, Rana Hanocka

arXiv:2112.03221v1cs.CVcs.CLcs.GR

TL;DR

Text2Mesh addresses text-driven 3D stylization by separating a fixed mesh’s content from learned color and local geometric details. It uses a neural style field optimized with CLIP similarity over augmented multi-view renderings, producing varied styles without a pretrained generative model or specialized 3D dataset. The method handles low-quality meshes and arbitrary genus, but mismatched prompts can erase source-shape content unless the object category is included.

  • Problem

    Text-driven 3D stylization requires editing appearance while preserving the underlying mesh content and properly formulating content, style, and their representations.

  • Method

    A fixed input mesh is coupled with a neural style field that predicts color and local geometric details, optimized using CLIP similarity from augmented multi-view renderings.

  • Results

    Text2Mesh stylizes a wide variety of meshes into many text-guided styles without a pretrained GAN or specialized 3D dataset, including low-quality meshes with arbitrary genus.

  • Takeaways & Limitations

    The framework provides intuitive text control for semantic 3D mesh stylization while retaining the advantages of explicit mesh surfaces and neural fields.

  • Takeaways & Limitations

    Mismatched prompts can cause stylization to ignore or erase source-shape content; including the object category in the prompt adds a content-preservation constraint.

Abstract

from arXiv · show

In this work, we develop intuitive controls for editing the style of 3D objects. Our framework, Text2Mesh, stylizes a 3D mesh by predicting color and local geometric details which conform to a target text prompt. We consider a disentangled representation of a 3D object using a fixed mesh input (content) coupled with a learned neural network, which we term neural style field network. In order to modify style, we obtain a similarity score between a text prompt (describing style) and a stylized mesh by harnessing the representational power of CLIP. Text2Mesh requires neither a pre-trained generative model nor a specialized 3D mesh dataset. It can handle low-quality meshes (non-manifold, boundaries, etc.) with arbitrary genus, and does not require UV parameterization. We demonstrate the ability of our technique to synthesize a myriad of styles over a wide variety of 3D meshes.

1. Introduction

Text2Mesh frames 3D style editing as text-driven manipulation that preserves mesh content while synthesizing color and local geometry. It combines explicit meshes, neural style fields, CLIP guidance, and regularization to produce diverse stylizations, including on low-quality meshes.

  • Text2Mesh separates 3D content, represented by a fixed mesh’s global structure and topology, from style, represented by color and local geometric details.
  • The neural style field predicts colors and normal-direction displacements over the input mesh surface, with its weights optimized to match a target text prompt.
  • CLIP compares augmented renderings of the stylized mesh from multiple 2D views with the target text, providing the optimization signal.
  • Directly maximizing CLIP similarity can produce noisy, degenerate solutions because the joint embedding space contains false-positive matches between text and artifacts.
  • Mesh geometry, neural-network weights, frequency-based positional encoding, and 2D augmentations regularize optimization while supporting smooth, high-frequency stylization.
  • The framework supports varied text-driven styles across 3D shapes, including low-quality meshes with arbitrary genus, while preserving underlying content in demonstrated examples.

2. Related Work

Prior work addresses text-driven image or 3D synthesis, geometric style transfer, texture transfer, and neural fields through specialized models, datasets, or mesh processing. Text2Mesh instead uses compact text specifications to guide a broad range of mesh styles without parameterization.

  • Text-Driven Manipulation: Text-guided image methods commonly use CLIP with pretrained generative networks, while CLIPDraw demonstrated text-guided synthesis without such a network or dataset.
  • Geometric Style Transfer in 3D: Existing 3D geometric style-transfer methods analyze shape correspondences, separate content and style, or target specialized categories and patterns.
  • Geometric Style Transfer in 3D: Unlike these approaches, Text2Mesh uses an intuitive text specification to guide a wide range of 3D styles.
  • Texture Transfer in 3D: Texture-transfer methods often rely on mesh parameterization with strict requirements such as manifoldness, non-intersection, and low or zero genus.
  • Texture Transfer in 3D: Text2Mesh avoids parameterization by using a neural field to modify appearance over the mesh.
  • Neural Priors and Neural Fields: Neural-field approaches can entangle geometry and appearance and may struggle with sharp features, rendering speed, and editability, whereas Text2Mesh complements an explicit mesh with a neural style field.

3. Method

Text2Mesh keeps the input mesh fixed as content while a neural style field predicts color and local geometric changes guided by CLIP similarity between augmented rendered views and target text. Fourier features, multiple views, augmentations, and geometry-only supervision regularize the optimization and preserve coherent detail.

  • Neural Style Field Network: Text2Mesh fixes an input mesh and learns a neural style field that maps surface points to vertex colors and normal-direction displacements.The resulting stylized mesh modifies color and local geometry while retaining the source mesh as content.
  • Neural Style Field Network: The style field uses Fourier positional encoding and branched MLPs to synthesize high-frequency style details while constraining displacements to (−0.1, 0.1).Coordinates are normalized to a unit bounding box; one branch predicts color and another predicts displacement.
  • Text-based Correspondence: The method renders the stylized mesh and displacement-only mesh from multiple views, applies global and local 2D augmentations, and embeds the results with CLIP.The target text is also embedded with CLIP, and averaged view representations provide the inputs to the similarity loss.
  • Text-based Correspondence: The CLIP-based loss combines full, local, and displacement-only representations, with the displacement-only term updating the shared and geometric branches.Separating geometry-only supervision from geometry-and-color supervision encourages meaningful geometric changes.
  • Text-based Correspondence: An anchor view is selected by CLIP similarity, then five nearby views sampled around it are averaged to encourage view consistency.The sampled views follow a Gaussian centered at the anchor with σ = π/4; increasing the view count beyond five does not meaningfully change results.
  • Text-based Correspondence: Random perspective and crop augmentations are critical regularizers: local crops cover 10% of the image and focus optimization on fine-grained surface adjustments.The ablation compares removing these augmentations and other priors, including positional encoding, the style field network, geometry-only supervision, and 3D learning.

4. Experiments

Experiments show that Text2Mesh provides controllable, content-aware stylization across diverse meshes and target modalities, while its priors improve fidelity and avoid degenerate CLIP-driven solutions. A user study finds higher perceived quality, content preservation, and style matching than the VQGAN baseline.

  • Experimental Setup: Text2Mesh stylizes diverse meshes, including low-quality inputs with non-manifold edges, non-manifold vertices, and boundaries, without particular input-quality constraints.The evaluated meshes average 79,366 faces, 16% non-manifold edges, 0.2% non-manifold vertices, and 12% boundaries.
  • Neural Stylization and Controls: Increasing text-prompt granularity yields styles with increasing detail and specificity while retaining details across successive target descriptions.Examples progress from generic lamp and iron prompts to descriptions specifying materials, colors, and surface structure.
  • Neural Stylization and Controls: Increasing positional-encoding frequency increases the frequency of style details and produces sharper, more frequent normal-direction displacements.The frequency range is controlled by the standard deviation σ of matrix B in the positional encoding.
  • Neural Stylization and Controls: Changing the input sphere’s protrusion frequency changes the generated cactus style’s frequency and structure, showing adaptation to pre-existing surface structure.The target prompt remains fixed to “cactus” while source geometry varies.
  • Text2Mesh Priors: Removing key priors causes distinct failures: noise, prompt mismatch, loss of fine detail, shading-based geometry simulation, or loss of global structure.The ablation identifies the style field network, 2D augmentations, Fourier features, cropping, geometry-only similarity, and the 3D prior as contributors to robust stylization.
  • Beyond Textual Stylization: Text2Mesh successfully stylizes meshes toward 2D image targets, 3D mesh targets, and combined mesh-plus-text targets.For mesh targets, the target embedding averages CLIP embeddings from renderings using the source mesh’s sampled views.
  • Incorporating Symmetries: A symmetry prior enforces style consistency across a symmetry axis, remaining effective even when mesh triangulation is not perfectly symmetrical.For bilateral symmetry across the X-Y plane, the positional-encoding input can use γ(x, y, |z|).

5. Conclusion

Text2Mesh predicts colors and local geometric details from target text while preserving mesh content, without requiring a pre-trained GAN or 3D dataset. It supports diverse meshes and styles, but future work is needed to manipulate or jointly stylize mesh collections.

  • Text2Mesh predicts colors and local geometric details for input meshes from target text prompts.
  • Including geometric content in the target prompt improves stylization when the target style might otherwise be unrelated to the mesh.
  • Texture direction is driven by rendered 2D images, avoiding directional fields and mesh parameterization.
  • The method stylizes many meshes into diverse styles without a pre-trained GAN or specialized 3D dataset.
  • The current neural style field is tailored to one mesh; future work could generate meshes from scratch or stylize collections feed-forward.

A. Additional Results

Additional results show that Text2Mesh synthesizes wood-style textures on a chair while aligning them to sharp features and curves.

  • Wood-style chair textures automatically align with the mesh’s sharp features and curves.

B. High Resolution Stylization

Text2Mesh produces high-fidelity stylization on high-resolution meshes and can synthesize finer styles after mesh resolution is increased.

  • A 1670x2720 render demonstrates high-resolution stylization with high-fidelity style synthesis.
  • Inserting degree-3 vertices at triangle barycenters enables finer neural-field style synthesis on coarse meshes.

C. Choice of anchor view.

The method selects an anchor view using CLIP similarity and evaluates stylization across sampled views. CLIP scores correlate with semantically meaningful views, but their narrow range limits expressiveness, while more than five sampled views changes little.

  • The anchor view is selected as the view with the highest CLIP similarity to the target text.
  • The additional figures provide example views with CLIP similarities and compare views around stylized meshes, including chair, vase, and cactus-donut prompts.
  • CLIP scores show a strong positive correlation with semantically meaningful views, supporting automatic anchor-view selection.
  • For the camel, CLIP scores range only from 0.2 to 0.35, limiting the metric’s expressiveness across views.
  • Increasing the number of sampled views beyond 5 does little to change output stylization quality.

D.1. Network Architecture

The network maps mesh vertices into Fourier features and processes them through shared and branched MLP layers to predict stylization outputs. Zero-initialized final layers preserve the original mesh at initialization, while residual color prediction constrains colors to [0.0, 1.0].

  • D.1. Network Architecture: Each vertex p ∈R3 is mapped to a 256-dimensional Fourier feature before entering the network.The Gaussian matrix B typically uses standard deviation 5.0, adjustable by the user.
  • D.1. Network Architecture: The shared network contains four 256-dimensional ReLU linear layers, while displacement and color branches each contain two such layers.Both branches apply tanh after their final linear layer.
  • D.1. Network Architecture: Style outputs are sampled across different views for prompts such as “A horse made of cactus.”The cited figure passage identifies view sampling and the target prompt but does not specify the network operation producing it.
  • D.1. Network Architecture: The final branch-layer weights are initialized to zero so the original content mesh remains unchanged at initialization.This initialization applies to both output branches.
  • D.1. Network Architecture: Color outputs are divided by 2 and added to [0.5, 0.5, 0.5], enforcing predicted colors within (0.0, 1.0).Initializing mesh color to grey and adding the output residually helps prevent undesirable early-training solutions.

D.2. Training

Training uses Adam with scheduled learning-rate decay for 1500 iterations on a single RTX2080Ti GPU. The supplementary material also shows baseline results and user-study screenshots for evaluation examples.

  • D.2. Training: Training uses Adam with an initial learning rate of 5e−4 and a 0.9 decay factor every 100 iterations.The model is trained for 1500 iterations on a single Nvidia GeForce RTX2080Ti GPU.
  • D.2. Training: The reported training time is around 25 minutes on a single Nvidia GeForce RTX2080Ti GPU.The same setup uses 1500 training iterations.
  • D.2. Training: Global augmentation uses a random perspective transformation, while local augmentation crops to 10% of the original image before another random perspective transformation.Images are normalized per channel before CLIP encoding.
  • D.2. Training: VQGAN baseline examples are shown alongside Text2Mesh results in Figures 22 and 23.The passage identifies these as examples for the VQGAN baseline described in Section 4.
  • D.2. Training: Figure 24 provides screenshots of the user study shown to users.The passage does not report the study outcome.

F. Societal Impact

Text2Mesh can make biases in CLIP’s joint image-text embedding space directly visible through stylization. The authors report societal bias in some outputs, including female features added to a male input for a nurse prompt.

  • F. Societal Impact: The framework uses a pre-trained CLIP embedding space that has been shown to contain bias.This inherited bias is a stated consideration for the societal impact of the framework.
  • F. Societal Impact: Text2Mesh enables direct and transparent visualization of biases through styles synthesized from target text prompts.The claim concerns biases present in joint image-text embeddings.
  • F. Societal Impact: For a nurse style, the authors observed female features being added to an input male shape.This is presented as evidence of societal bias in one stylization.
  • F. Societal Impact: The authors identify creating a datasheet for CLIP and future image-text embedding models as an important future direction.The proposed system is described as offering an early opportunity to observe these embedding biases through stylization.
Loading 2112.03221v1…