Source-linked AI summary
3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation
Dale Decatur, Itai Lang, Kfir Aberman, Rana Hanocka
TL;DR
Precise text-driven local editing on meshes remains limited: existing localizations can be coarse, and local editing work does not natively produce mesh texture maps. 3D Paintbrush jointly learns a localization and texture map and uses multi-stage cascaded diffusion supervision. It produces detailed local stylizations across varied meshes and semantic regions, with CSD providing control over detail and global understanding.
Problem
Prior local-editing methods often produce coarse regions, and existing local editing work does not natively operate on meshes or produce texture maps for them.
Source: §1 · p. 1Method
3D Paintbrush jointly produces a mesh texture map and localization region, using CSD to distill supervision from multiple cascaded diffusion stages.
Source: §1 · p. 1Results
Takeaways & Limitations
Its texture maps integrate into standard graphics pipelines, and explicit localization supports layering local edits onto existing textures.
Source: §1 · p. 1The paper identifies extending localized editing beyond texturing as future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
In this work we develop 3D Paintbrush, a technique for automatically texturing local semantic regions on meshes via text descriptions. Our method is designed to operate directly on meshes, producing texture maps which seamlessly integrate into standard graphics pipelines. We opt to simultaneously produce a localization map (to specify the edit region) and a texture map which conforms to it. This synergistic approach improves the quality of both the localization and the stylization. To enhance the details and resolution of the textured area, we leverage multiple stages of a cascaded diffusion model to supervise our local editing technique with generative priors learned from images at different resolutions. Our technique, referred to as Cascaded Score Distillation (CSD), simultaneously distills scores at multiple resolutions in a cascaded fashion, enabling control over both the granularity and global understanding of the supervision. We demonstrate the effectiveness of 3D Paintbrush to locally texture a variety of shapes within different semantic regions. Project page: https://threedle.github.io/3d-paintbrush
1. Introduction
3D Paintbrush addresses the need for precise, text-driven local edits on meshes by producing a texture map together with a localization region. It combines these outputs and multi-resolution diffusion supervision to create detailed local stylizations across shapes and semantic regions.
1. Introduction
- Existing local editing methods often yield coarse regions, while mesh-based texture maps are not natively supported by prior local-editing work.Fine-grained localization helps confine edits and prevent unrelated changes.Source: §1 · p. 1
- 3D Paintbrush produces mesh texture maps that integrate into graphics pipelines, and its explicit localization masks support layering local edits onto existing textures.The predicted localizations and textures can be composited without unwanted fringes.Source: §1 · p. 1, Figure 2 · p. 2
- The method jointly produces a localization mask and texture, with each improving the other’s detail and helping the style respect the localization boundary.The localization explicitly masks the texture, while the texture encourages a more detailed localization.Source: §1 · p. 2
- Cascaded Score Distillation (CSD) distills scores at multiple resolutions, combining supervision for granularity and global understanding.The method incorporates independently trained cascaded stages together, unlike standard SDS, which uses only the low-resolution stage.Source: §1 · p. 2
- 3D Paintbrush produces diverse local textures across varied shapes and semantic regions, with CSD enabling control over detail and global understanding.The method uses all stages of the diffusion model to achieve higher-resolution textures and localizations than standard SDS.Source: §1 · p. 2
2. Related Work
Prior work covers neural stylization, text-driven analysis, global 3D generation and editing, and high-resolution score distillation, but leaves gaps in precise text-driven local mesh editing. 3D Paintbrush combines localization with texturing on meshes and uses a visual loss to keep localization boundaries sharp.
2. Related Work
- Neural stylization methods focus on stylization rather than localization, while text-driven analysis methods produce localizations but not textures.Prior mesh-localization methods still produce smooth regions that miss high-frequency detail needed for sharp local edits.Source: §2 · p. 2
- The method overview maps mesh-surface points to localization, texture, and background outputs, then uses rendered variants with text conditions to compute CSD loss.The variants are rendered from the same viewpoint.Source: Figure 3 · p. 3
- Global text-to-3D editing methods lack explicit edit regions, so they cannot guarantee that changes stay within the intended region.The cited methods may therefore struggle with highly specific local edits without affecting other components’ appearance.Source: §2 · p. 3
- Existing 3D local-editing approaches either require user-drawn regions or use post-edit attention masks that do not ensure edits conform to localization bounds.Attention-based regions may be coarse and larger than the actual edit, risking changes outside the intended region.Source: §2 · p. 3
- 3D Paintbrush applies visual loss to localizations for sharp boundaries and enables text-driven local editing directly on meshes.The method is presented as the first purely text-driven local editing approach to enable this on meshes.Source: §2 · p. 3
- High-resolution SDS improvements remain orthogonal to CSD because they modify base-stage supervision and can also be incorporated at super-resolution stages.CSD instead extracts supervision from multiple cascaded diffusion stages.Source: §2 · p. 3
3. Method
3D Paintbrush takes a mesh and text description to produce a local texture and localization region. It represents these maps with neural fields and uses simultaneous localization–texture optimization and Cascaded Score Distillation (CSD) for local stylization.
3. Method
- Given a mesh and text description, the system produces a local texture guided by CSD supervision at multiple resolutions.Source: §3 · pp. 3–4
3.1. Local Neural Texturing
- MLPs map 3D surface coordinates to localization probabilities and RGB values, yielding smooth textures across UV seams and supporting super-resolution queries.Source: §3.1 · p. 4
3.2. Visual Guidance for Localized Textures
- Three visual-guidance branches supervise the masked local texture, localization alone, and background outside the mask to discourage overly broad regions.Jointly optimizing localization and texture improves both: the texture encourages a more intricate localization, while the localization masks the texture to keep the style within its boundary.Source: §3.2 · pp. 4–5
3.3. Score Distillation and Cascaded Diffusion
- Standard SDS uses only the low-resolution base stage; CSD extends score distillation to cascaded super-resolution stages using a training-based formulation.Source: §3.3 · pp. 5–6
3.4. Cascaded Score Distillation
- CSD combines gradients from multiple diffusion stages and lets users weight each stage to control supervision detail and corresponding localization.The mesh is rendered at each stage's resolution, and the weighted gradients update the neural texture parameters.Source: §3.4 · pp. 6–7
4. Experiments
Experiments show detailed local textures and localizations across varied meshes and prompts, with CSD and joint localization-texture optimization supporting detail, placement, and composition.
3D Paintbrush generality
- Detailed localizations and textures span organic and manufactured meshes, including out-of-domain prompts such as shinguards on a giraffe or a shirt on a goat.The placements are described as semantically meaningful, despite those combinations being unusual for the objects.Source: §4 · p. 7
3D Paintbrush precision and composition
- Precise localization boundaries let multiple textures compose on one mesh without layering artifacts or changing regions outside an edit.The tie-dye apron can be composited over a Superman emblem while leaving the emblem outside the apron boundary unaffected.Source: §4 · p. 7
3D Paintbrush specificity and robustness
- High-resolution local edits closely follow prompt details, including intricate jersey badges and differences between a tapered cape and a boxier poncho.Multiple edits on the same mesh and edits across different meshes demonstrate robustness to prompts and meshes.Source: §4 · pp. 7–8
Impact and granular control of CSD
- CSD combines low-resolution localization with high-resolution detail, while stage 2 alone often mislocalizes edits and stage 1 alone produces low-resolution results.Increasing stage 2 weight progressively increases detail and granularity, allowing users to control the balance between detail and global understanding.Source: Figure 9 · p. 7, §4 · p. 8
Simultaneous localization and texture
- Simultaneous localization and texture optimization yields more detailed textures and intricate localizations than sequential or independent optimization.Independent optimization can leave the texture misaligned with the localization and produce fringe artifacts when the texture is masked.Source: §4 · p. 8
Background loss ablation
- Learning a separate background map supports accurate, tightly coupled localization and texture; sharing the edit map misaligns them, while removing background loss can add input-shape features to the localization.The cited example of the latter is a duck's bill appearing in an additional localization region.Source: §4 · pp. 8–9, Figure 11 · p. 8
5. Conclusion
3D Paintbrush produces detailed mesh textures that adhere to predicted localizations, while CSD extracts supervision from multiple diffusion stages. The authors identify expanding beyond texturing and co-texturing shapes as future work.
5. Conclusion
- 3D Paintbrush produces highly detailed mesh textures that adhere to predicted localizations, supporting seamless post-processing such as compositing without unwanted fringe.The method is also described as producing non-obvious local textures across a wide variety of meshes.Source: §5 · p. 9
- CSD extracts supervision from multiple stages of a cascaded diffusion model, whose weights control the resulting textures' detail and global understanding.The authors note that CSD can also be applied to images, videos, and alternative 3D representations.Source: §5 · p. 9
- Future work includes localized edits beyond texturing and co-texturing multiple shapes with a shared local texture map.The authors suggest shared maps may provide correspondences between shapes.Source: §5 · p. 9