PaperlayerView exact revision on arXiv

Source-linked AI summary

Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes

Hmrishav Bandyopadhyay, Subhadeep Koley, Ayan Das, Ayan Kumar Bhunia, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, Yi-Zhe Song

arXiv:2312.04043v2cs.CVcs.AI

TL;DR

The paper addresses sketch-to-3D generation that can reflect limited drawing skill and lacks paired human-sketch/shape training data. It aligns part-level shape and sketch representations for latent-diffusion generation and editing, reporting improved sketch-conditioned metrics and efficient generation.

  • Problem

    Prior sketch-to-3D methods can produce deformed shapes from poorly drawn sketches, and paired human-sketch and 3D-shape data is scarce.

    Source: §1 · p. 2

  • Method

    The method aligns part-level implicit shape latents with sketch-part representations derived from projected 3D regions and CLIPasso edgemaps, then conditions latent diffusion on sketches.

    Source: §1 · p. 2

  • Results

    For sketch-conditioned generation without human-sketch training, the method outperforms LAS-D [90] and SENS [9] by 0.0076/0.049 and 0.0012/0.05 on average CD/EMD, respectively.

    Source: §4 · p. 7

  • Takeaways & Limitations

    The approach supports fine-grained shape control from doodles across abstraction levels, in-position sketch edits, and multi-view improvement, while the conclusion describes it as more efficient than state-of-the-art 3D generative models.

    Source: §5 · p. 8

Abstract

from arXiv · show

In this paper, we democratise 3D content creation, enabling precise generation of 3D shapes from abstract sketches while overcoming limitations tied to drawing skills. We introduce a novel part-level modelling and alignment framework that facilitates abstraction modelling and cross-modal correspondence. Leveraging the same part-level decoder, our approach seamlessly extends to sketch modelling by establishing correspondence between CLIPasso edgemaps and projected 3D part regions, eliminating the need for a dataset pairing human sketches and 3D shapes. Additionally, our method introduces a seamless in-position editing process as a byproduct of cross-modal part-aligned modelling. Operating in a low-dimensional implicit space, our approach significantly reduces computational demands and processing time.

1. Introduction

The paper targets sketch-to-3D generation that remains precise when users lack drawing skill, without requiring paired human-sketch and 3D-shape data. It uses aligned part-level representations to model abstract sketches, generate shapes, and support in-position editing.

1. Introduction

  • Sketches capture intent for fine-grained creation, but text can lack the nuanced cues needed to convey precise ideas.The paper frames sketches as a complementary input modality for fine-grained shape creation.Source: §1 · p. 1
  • The paper aims to turn abstract sketches into precise 3D shapes without paired human-sketch and 3D-shape data.Prior sketch-to-3D methods can produce deformed shapes from poorly drawn sketches, while the paper targets a workflow that does not depend on drawing skill.Source: §1 · pp. 1–2
  • The approach models abstract forms with part-level representations, whose flexible parts accommodate deformations found in human sketches.The authors build on the hypothesis that part-level modelling aids generalisation to complex shapes [18, 76].Source: §1 · p. 2
  • Starting from a pre-trained auto-decoder, the method obtains part-disentangled latents, aligns parts to shared indices for similar components, and trains a diffusion pipeline to generate aligned latents decodable into meshes.In low-dimensional implicit space, it uses 0.06×params and runs 3×faster than prior art [90].Source: §1 · p. 2
  • The same decoder supports sketch modelling by matching CLIPasso edgemaps to projected 3D part regions, without collecting sketch data.Once sketch regions correspond to 3D parts, the system can identify which parts an in-position sketch edits and condition generation part by part.Source: §1 · p. 2
  • The paper’s contributions include precise 3D generation from abstract sketches, cross-modal part alignment, and in-position editing.The authors present editing as a byproduct of their cross-modal part-aligned modelling.Source: §1 · p. 2

2. Related Works

Related work covers explicit and implicit shape representations, part-level modelling, 3D generation, and sketch-to-3D methods. The paper positions its approach at the intersection of implicit representations, part decomposition, latent diffusion, and part-aware sketch conditioning.

2. Related Works

  • Explicit representations include point clouds, voxel grids, and meshes, while continuous implicit functions map coordinates to occupancy or signed-distance values.Sampling these functions on a grid can produce explicit shapes at chosen resolutions.Source: §2 · p. 2
  • Part-level representations decompose shapes for part-specific editing and generation, including unsupervised decompositions based on Gaussian mixtures.Prior approaches also use hyperplanes, quadratic surfaces, and superquadrics.Source: §2 · p. 3
  • Prior 3D generation spans explicit structures and latent implicit functions; this work combines implicit representations and part decomposition in a part-aware latent diffusion model.The cited work describes explicit 3D diffusion as slow and compute-heavy, while latent approaches may limit diversity and editability.Source: §2 · p. 3
  • Sketch-to-3D is framed as generative because inaccurate hand-drawn views can correspond to multiple plausible shapes; this work conditions latent diffusion on fine-grained part-aware sketch encodings.The approach aligns sketch parts with underlying shape parts to capture fine-grained shape information.Source: §2 · p. 3

3. Proposed Methodology

The method represents shapes and sketches with corresponding parts in an implicit latent space, then uses sketch-conditioned latent diffusion to generate shapes. Alignment, rendered part annotations, and cross-attention provide part-level control from abstract sketches.

3.1. Baseline Sketch-to-3D Generation:

  • Sketch-to-3D is ill-posed as a deterministic mapping because one sketch can correspond to multiple shapes, and synthetic-edgemap training may not generalise to human doodles.Global implicit codes also make local edits difficult because they lack feature disentanglement.Source: §3.1 · p. 3

3.2. Part-aware Neural Implicit Shape modelling

  • A pre-trained decoder decomposes meshes into part-latents and maps them to implicit codes, with part Gaussians describing part positions and orientations.Part structures and Gaussian parameters are combined with self-attention to form implicit codes that can be sampled into meshes.Source: §3.2 · p. 4

3.3. Part-level Alignment in the INR Latent Space

  • The method aligns part indices across shapes by minimizing Wasserstein distance between part-volume Gaussians and a selected template latent.For each template part, it replaces the corresponding part with the closest-matching part from the shape latent.Source: §3.3 · p. 4, Figure 2 · p. 4

3.4. Unsupervised Part Discovery for Sketches

  • To obtain sketch-part labels without paired human sketches and shapes, the method renders multiple shape views, generates synthetic edgemaps, and projects 3D part segmentations onto them.The resulting sketch segment maps correspond to the parts of their associated shapes and support part-disentangled sketch representations.Source: §3.4 · p. 4

3.5. Latent Diffusion for 3D Generation

  • The diffusion model generates part-latents, while a sketch encoder and shared decoder learn part-disentangled sketch features from the projected part annotations.Cross-attention uses sketch features as queries and intermediate diffusion outputs as keys and values; the denoising loss is conditioned on the sketch.Source: §3.5 · pp. 5–6
  • For multiple views, the method aggregates corresponding part-aware sketch representations to reconstruct shapes more accurately.The per-view encodings are described as viewpoint-independent representations of the same explicit parts.Source: §3.5 · p. 5

3.6. Inference Pipeline

  • Inference samples a noisy latent, iteratively denoises it conditioned on the sketch, then decodes the result into an implicit code and mesh.The mesh is constructed by sampling occupancies on a coordinate grid and applying Marching Cubes.Source: §3.6 · p. 6

4. Experiments

Experiments evaluate sketch-conditioned and unconditional 3D generation, including abstraction robustness, editing, multi-view input, and efficiency. The method is tested on hand-drawn sketches despite training on synthetic sketches.

  • Training uses edgemaps and synthetic sketches from 6,755 chairs, producing about 80K samples; evaluation includes unseen hand-drawn viewpoints and sub-20-second Quick-Draw doodles.Figure 6 compares detailed artist sketches, amateur perspective-distorted sketches, and abstract Quick-Draw examples; SENS was trained on ProSketch-3D sketches.Source: Figure 5 · p. 6, §4 · p. 6, Figure 6 · p. 7
  • The method outperforms LAS-D and SENS by 0.0076/0.049 and 0.0012/0.05 on average CD/EMD, respectively, in sketch-conditioned generation.Table 1 compares conditional generation and model efficiency on AmateurSketch-3D and ProSketch-3D.Source: §4 · p. 7, Table 1 · p. 7
  • For unconditional shape generation trained on ShapeNet Chairs, our model performs at par with other diffusion-based 3D generative frameworks, including LAS-D [90] and Diffusion-SDF.Source: §4 · p. 7, Table 2 · p. 7
  • For increasingly abstract CLIPasso sketches, average generated-shape CD rises from 0.0078 to 0.0241 for SENS, 0.0084 to 0.0130 for our method, and 0.0102 to 0.0390 for edgemap-trained LAS-D.Source: §4 · p. 8, Figure 7 · p. 8
  • Aggregating part representations from multiple sketch views improves reconstruction accuracy on AmateurSketch-3D and ProSketch-3D, reducing CD/EMD from 0.0109/0.093 to 0.0097/0.085, respectively.Figure 8 attributes slight shape variance to occlusion from specific views or poor-quality sketches and says multi-view sketches can fix it.Source: §4 · p. 8, Figure 8 · p. 8
  • Removing latent alignment increases CD by 0.0062; replacing part-aware sketch encodings adds 0.0053, while 1/2/4 attention layers add 0.0018/0.0005/0.0001.The reported ablation uses the AmateurSketch-3D dataset.Source: §4 · p. 8, Table 3 · p. 8
  • Part-aware encodings support local sketch edits without disturbing global structure; sketch-representation interpolation morphs shapes smoothly, and inference time is minimal versus SOTA generative networks.The paper also describes fine-grained control over generated shapes through sketch edits.Source: §4 · p. 8, Figure 9 · p. 8

5. Conclusion

The paper presents a latent diffusion pipeline that generates precise 3D shapes from abstract sketches using part-disentangled sketch representations. It reports fine-grained control and editing, multi-view improvement, and greater efficiency than SOTA 3D generative models.

  • The latent diffusion pipeline generates precise 3D shapes from abstract sketches using part-disentangled sketch representations.The paper reports fine-grained sketch edits and generation across abstraction levels from Quick-Draw! to ProSketch-3D.Source: §5 · p. 8

A. Segmenting Hand-Drawn Sketches

The auxiliary segmentation task tests whether a model trained on synthetic sketches generalizes to hand-drawn sketches. The authors report fairly accurate segmentation and semantic maps for all 16 parts.

B. Further Details on Shape Decomposition

Shape decomposition represents a mesh with part-latents decoded into an implicit representation. The method optimizes part disentanglement beyond matching the final shape alone.

  • To represent a mesh, the decoder maps part-latent Z to an implicit code, which predicts occupancy values for known coordinates; pretraining uses binary cross-entropy against recorded occupancies.Source: B. Further Details on Shape… · p. 12
  • Ideally, each latent code independently represents a shape component, but binary cross-entropy alone matches the output shape and does not enforce part-level correspondence.Source: B. Further Details on Shape… · p. 12
  • For disentanglement, each part gains a structural representation and a volumetric descriptor: a parametric 3D Gaussian estimating whether a coordinate belongs to that part.The coordinate-to-part relationship supports decomposing the shape volume and disentangling overlapping or nearby parts.Source: B. Further Details on Shape… · p. 12

C. Shape Interpolation

The paper presents shape interpolation across four sketches and extends sketch-based 3D generation to additional object categories.

C. Shape Interpolation

  • Shapes are interpolated among four sketches placed at the corners, with results ordered left to right and top to bottom.Source: Figure 11 · p. 12

D. Generation of Shapes from Other Categories

G. User feedback for generated shapes:

Thirty users rated generated shapes and sketch-based edits in a demo, with higher mean opinion scores for the paper’s method than LAS-D across the reported measures.

Related papers

Loading 2312.04043v2…