PaperlayerView exact revision on arXiv

Source-linked AI summary

MagicClay: Sculpting Meshes With Generative Neural Fields

Amir Barda, Vladimir G. Kim, Noam Aigerman, Amit H. Bermano, Thibault Groueix

arXiv:2403.02460v4cs.GR

TL;DR

Neural fields support shape generation but lack incremental geometric control, while meshes offer intuitive control but do not lend themselves to neural optimization. MagicClay jointly optimizes an SDF and mesh for prompt-guided local sculpting; it reports higher-quality generation and localized edits than competing approaches.

  • Problem

    Neural fields lack localized geometric control, while meshes provide intuitive control but are difficult to optimize with neural generation methods.

    Source: §1 · pp. 1–2

  • Method

    MagicClay jointly optimizes a mesh and SDF, using consistency losses and representation-specific priors to support prompt-guided local edits.

    Source: §1 · p. 2

  • Results

    MagicClay produces higher-quality unconditional text-to-3D meshes than other representations using similar generative techniques and higher-quality, more localized edits than competing methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MagicClay brings neural shape generation closer to iterative artistic workflows, with incremental editing and greater precision and control over the result.

    Source: §5 · p. 7

    MagicClay is not interactive, taking 1 hour per prompt on an A100 GPU, with SDS loss as the bottleneck.

    Source: §5 · p. 7

Abstract

from arXiv · show

The recent developments in neural fields have brought phenomenal capabilities to the field of shape generation, but they lack crucial properties, such as incremental control - a fundamental requirement for artistic work. Triangular meshes, on the other hand, are the representation of choice for most geometry related tasks, offering efficiency and intuitive control, but do not lend themselves to neural optimization. To support downstream tasks, previous art typically proposes a two-step approach, where first a shape is generated using neural fields, and then a mesh is extracted for further processing. Instead, in this paper we introduce a hybrid approach that maintains both a mesh and a Signed Distance Field (SDF) representations consistently. Using this representation, we introduce MagicClay - an artist friendly tool for sculpting regions of a mesh according to textual prompts while keeping other regions untouched. Our framework carefully and efficiently balances consistency between the representations and regularizations in every step of the shape optimization; Relying on the mesh representation, we show how to render the SDF at higher resolutions and faster. In addition, we employ recent work in differentiable mesh reconstruction to adaptively allocate triangles in the mesh where required, as indicated by the SDF. Using an implemented prototype, we demonstrate superior generated geometry compared to the state-of-the-art, and novel consistent control, allowing sequential prompt-based edits to the same mesh for the first time.

1 INTRODUCTION

Neural fields offer robust generation but lack localized geometric control, while meshes support intuitive editing but are difficult to optimize generatively. MagicClay jointly maintains an SDF and mesh, enabling prompt-guided local edits alongside higher-quality generation.

1 INTRODUCTION

  • Neural fields lack localized editing and surface-based priors, while artists expect updates to retain existing geometry, textures, and topology.Meshes provide intuitive control, but sparse gradients limit stable large deformations; two-stage pipelines can become trapped in local minima and do not extend to editing meshes with pre-existing UVs and textures.Source: §1 · pp. 1–2
  • The method jointly optimizes an input mesh and its SDF, using multi-view consistency and representation-specific priors to keep them synchronized.It uses differentiable renders of RGB, opacity, and normals; adaptive topology updates split faces, collapse edges, and flip edges as needed.Source: §1 · p. 2
  • The hybrid representation uses mesh-localized SDF sampling for faster, higher-resolution rendering and adaptive tessellation for geometry where needed.Triangle supersampling helps maintain mesh properties, such as animation vertex groups, through topology changes.Source: §1 · p. 2
  • Mesh and SDF synchronization supports localized sequential edits, preserving existing mesh information in some regions while allowing radical semantic changes in others.The method also reports higher generated geometric quality than using an implicit representation alone.Source: §1 · p. 2
  • MagicClay produces higher-quality unconditional text-to-3D meshes than other representations using similar generative techniques, and higher-quality, more localized edits than competing methods.The editing workflow preserves geometry, texture, tessellation, and rigging parameters in unselected regions.Source: §1 · p. 2

2 RELATED WORK

Prior work spans implicit and mesh-based generation, local editing, and hybrid representations. MagicClay is positioned as a jointly optimized hybrid that combines SDF-driven topology updates with mesh-based detail and existing-mesh control.

2 RELATED WORK

  • Earlier generative methods use NeRFs, SDFs, mesh refinement, or separate geometry and appearance optimization, while MagicClay keeps a mesh and SDF synchronized with regularization and dynamic projection.The related methods include SDS- and VSD-guided generation and differentiable mesh refinement.Source: §2 · pp. 2–3
  • Two-stage methods use implicit functions for coarse generation and meshes for high-resolution refinement, but can be prone to local minima and cannot edit existing meshes with precomputed UVs.MagicClay instead jointly optimizes an SDF and mesh in a one-step pipeline, combining SDF topology updates with mesh detail.Source: §2 · pp. 2–3
  • Prior local-editing methods use implicit representations and prompting, sketches, or localization losses; reconstructed meshes can lose properties in non-editable regions.MagicClay exposes a mesh for region selection and aims to preserve unselected mesh properties.Source: §2 · p. 3
  • Traditional sculpting and example-based modeling offer geometry-editing workflows, but the latter can depend on restricted domains and heavily annotated assets.MagicClay uses pretrained 2D generative priors for semantic local, iterative modeling without pre-annotated 3D data.Source: §2 · pp. 3–4

3 METHOD

MagicClay jointly optimizes a mesh and an SDF, using SDF guidance for larger-scale shape changes and the mesh for localized control. Their consistency and representation-specific priors support prompt-guided editing while preserving the mesh representation.

3.1 Hybrid Representation

  • The hybrid representation jointly renders a triangular mesh and an SDF with a shared appearance network, using the SDF for larger-scale guidance and the mesh for localized edits.The SDF receives text guidance through SDS; consistency losses and topology updates propagate changes to the mesh.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 4
  • Mesh colors come from a shared appearance network and are applied through adaptive face subdivision, balancing compact flat regions with high-frequency textures.The authors avoid per-vertex coloring and UV parameterization because of triangulation sensitivity and the mesh’s ongoing topological changes.Source: §3.1 · p. 4

3.2 Hybrid Shape Guidance

  • Mesh intersections center three SDF samples per ray, enabling 512x512 and larger SDF renders instead of the typical 512 samples per ray.The authors also apply regular SDS to 128x128 VolSDF renders, which they report slightly improves results.Source: §3.2 · p. 5

3.3 Representation Priors

  • Consistency losses align RGB, normal, and opacity renders, while freeze, Laplacian, Eikonal, opacity, and normal losses regularize localization and representation quality.The mesh freezes vertices outside the selected region; a sampling-based loss favors unchanged SDF regions around fixed vertices, and the mesh Laplacian encourages smoothness.Source: §3.3 · p. 5

3.4 Updating the Mesh Topology

  • ROAR uses SDF-derived curvature scores to split triangles and edge scores to collapse them, with each iteration decreasing its energy while preserving manifoldness.The method projects supersampled triangle vertices onto the SDF zero-level set before scoring the resulting piecewise-linear surface.Source: §3.4 · p. 5

3 METHOD

  • On a benchmark of 20 prompts, MagicClay has the highest CLIP Score for texture-less meshes; CLIP Score is cosine similarity, scaled by 100, between prompt embeddings and 25 normal renders.The 25 normal renders are taken in an equidistant circular pattern around each generated object. MagicClay is compared with five state-of-the-art methods.Source: Table 1 · p. 6

4 EXPERIMENTS

The experiments compare MagicClay with generative and editing methods, demonstrate mesh sculpting, test robustness to noisy guidance, and ablate design choices. They report geometry quality and editing behavior alongside controlled noise and component analyses.

4.1 Comparison with Generative Methods

  • MagicClay produces recognizable, smooth texture-less meshes and achieves the highest Clip Score among the competing approaches.The comparison evaluates mesh renderings without texture.Source: §4.1 · p. 6
  • Changing the text-to-image backbone changes generated quality, while the hybrid representation is orthogonal to backbone choice.The paper shows MagicClay with SD 1.5, SD 2.1, and DeepFloyd.Source: §4.1 · p. 6

4.2 Mesh Sculpting

  • Users can preserve non-selected surface regions and transfer their texture, tessellation, and rigging while sculpting selected regions to match a prompt.The resulting mesh can be iteratively refined with new elements.Source: §4.2 · p. 6

4.3 Comparison with Contemporary 3D Editing Pipelines

  • Compared with the editing baselines, MagicClay adds complex large-scale details while restricting changes to user-selected regions.The cited comparison includes a text-driven mesh deformation baseline and methods based on soft attention.Source: §4.3 · p. 6

4.4 Analysis of Mesh and SDF Robustness to Noise

  • At noise standard deviation 5, mesh reconstruction degenerates to a blob while SDF reconstruction remains recognizable; the hybrid performs better than both individual representations at every tested noise level.The figure reports L2 reprojection error on novel views. Curves are not strictly increasing, which the authors attribute to noise possibly preventing overfitting; they state this experiment tests robustness to noise, not hybrid superiority over SDF.Source: Figure 3 · p. 7, §4.4 · p. 7

4.5 Ablations

  • 512x512 SDF renderings significantly affect generated shape quality, supporting a role for the mesh component that accelerates SDF rendering.Source: §4.5 · p. 7
  • Only substantial differences in the initial sculpt produce different results; removing localization constraints can cause unintended global changes.Source: §4.5 · p. 7
  • Without topology updates, fixed-resolution meshes either differ only marginally from the input when resolution is high or lack fine details when it is low.Source: §4.5 · p. 7

5 CONCLUSION

MagicClay combines an SDF and mesh representation for generative sculpting, using the mesh to localize SDF ray samples. The conclusion notes noisy SDS gradients and slow per-prompt runtime, and identifies diffusion-based approaches as future work.

5 CONCLUSION

  • MagicClay’s hybrid SDF-and-mesh representation uses mesh-localized ray sampling to render the SDF, a key part of its generative sculpting process.The paper identifies this rendering strategy as key to the process.Source: §5 · p. 7
  • Noisy, view-dependent SDS gradients make fine details harder to emerge, and a prompt takes about one hour on an A100 GPU.The authors identify SDS loss as the runtime bottleneck.Source: §5 · p. 7
  • Future work could use inpainting and depth-conditioned diffusion models to speed SDS convergence and reduce noise by keeping unchanged image regions consistent with localized edits.Source: §5 · p. 7

Related papers

Loading 2403.02460v4…