PaperlayerView exact revision on arXiv

Source-linked AI summary

Locally Attentional SDF Diffusion for Controllable 3D Shape Generation

Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, Heung-Yeung Shum

arXiv:2305.04461v2cs.CVcs.GR

TL;DR

Existing 3D generators leave a quality gap and offer limited intuitive local control for users. LAS-Diffusion uses two-stage SDF diffusion with view-aware local attention for sketch-conditioned generation, producing plausible shapes with reported controllability and generalizability advantages over existing methods.

  • Problem

    Existing 3D generators have a quality gap relative to training data, while users lack intuitive control over local geometry and must spend considerable time on prompt engineering.

  • Method

    LAS-Diffusion uses two-stage diffusion: occupancy-diffusion models a coarse shape shell, while SDF-diffusion generates finer geometry within occupied regions, guided by view-aware local attention.

  • Results

    The model produces good-quality shapes matching user sketches, remains robust to synthetic and free-hand sketches, and demonstrates superior local controllability, generalizability, shape quality, and diversity across evaluated tasks.

  • Takeaways & Limitations

    The view-aware local attention mechanism provides a simple, flexible module for controllable multimodal-conditioned 3D content generation and can extend to color, depth, and point-cloud inputs.

  • Takeaways & Limitations

    The model is trained only on synthetic data and is not well adapted to highly distorted lines, oversketches, or seriously inconsistent perspectives.

Abstract

from arXiv · show

Although the recent rapid evolution of 3D generative neural networks greatly improves 3D shape generation, it is still not convenient for ordinary users to create 3D shapes and control the local geometry of generated shapes. To address these challenges, we propose a diffusion-based 3D generation framework -- locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch image input. Our method is built on a two-stage diffusion model. The first stage, named occupancy-diffusion, aims to generate a low-resolution occupancy field to approximate the shape shell. The second stage, named SDF-diffusion, synthesizes a high-resolution signed distance field within the occupied voxels determined by the first stage to extract fine geometry. Our model is empowered by a novel view-aware local attention mechanism for image-conditioned shape generation, which takes advantage of 2D image patch features to guide 3D voxel feature learning, greatly improving local controllability and model generalizability. Through extensive experiments in sketch-conditioned and category-conditioned 3D shape generation tasks, we validate and demonstrate the ability of our method to provide plausible and diverse 3D shapes, as well as its superior controllability and generalizability over existing work. Our code and trained models are available at https://zhengxinyang.github.io/projects/LAS-Diffusion.html

1 INTRODUCTION

Existing 3D generators leave gaps in geometry quality and intuitive local control. LAS-Diffusion addresses these issues with two-stage SDF diffusion conditioned by sketches and view-aware local attention.

  • Motivation: Existing 3D generation methods produce shapes with a substantial quality gap relative to their training datasets and offer limited intuitive user control.The introduction identifies representation quality and generative-model capability as contributors to the quality gap.
  • Motivation: Global sketch features limit local controllability and generalization to unseen shapes, while 2D sketching offers a natural interface without language barriers.The paper contrasts sketching with text conditioning, which requires prompt engineering and has limited paired 3D-text data.
  • Method: LAS-Diffusion uses occupancy-diffusion to generate a coarse shape shell, followed by SDF-diffusion to synthesize high-resolution geometry within occupied voxels.The two-stage design reduces the memory and computation required by naïve high-resolution 3D SDF diffusion.
  • Method: The method introduces view-aware local attention that lets 2D sketch features guide 3D voxel feature learning for local control and improved generalizability.The attention mechanism interacts with the 3D diffusion models using learned attention and image-patch information.
  • Results: Experiments show plausible, diverse shapes and superior controllability and generalizability across sketch-conditioned and category-conditioned generation tasks.The model also matches user sketches and is robust to both synthetic sketches extracted from images and free-hand sketches.

2 RELATED WORK

Prior 3D generation research spans discrete fields, points, meshes, implicit representations, autoregressive models, diffusion, and diverse conditioning inputs. Sketch-based methods commonly frame the task as reconstruction or use projection information, while this paper belongs to the broader effort toward controllable conditional generation.

  • Shape representations in 3D generation: Early methods use low-resolution occupancy fields or fixed-number point sets, whose discrete representation limits shape fidelity and motivates refinement.Polygonal meshes and implicit representations are also explored for 3D generation.
  • GAN-based 3D generation: GAN-based methods learn latent shape spaces or directly discriminate 3D outputs, including approaches combining local and global discriminators to improve quality.These methods commonly decode latent codes into implicit occupancy fields or directly model 3D outputs.
  • Autoregressive-based 3D generation: Autoregressive approaches generate hierarchical occupancy structures or model latent representations of implicit-function voxel patches with transformers.Some methods avoid encoding empty voxels when defining their latent representations.
  • Diffusion-based 3D generation: Diffusion-based 3D generation has focused heavily on point clouds, but converting them into continuous shapes requires additional nontrivial processing.Other work directly uses SDF-related coefficient volumes for coarse and detailed generation.
  • Conditional 3D generation: Conditional 3D generation uses text, images, coarse voxels, sparse points, and bounding volumes to support content creation and downstream reconstruction or completion tasks.The related methods cover multiple forms of conditional generation and shape processing.
  • Sketch-based shape reconstruction and generation: Sketch-based 3D research includes single- or multi-image reconstruction methods and approaches incorporating view information or 2D projection losses.These works formulate sketch-to-3D as reconstruction or generation from image evidence.

3 VIEW-AWARE LOCALLY ATTENTIONAL SDF DIFFUSION

LAS-Diffusion uses two-stage diffusion to generate detailed 3D shapes efficiently, while view-aware local attention uses projected sketch patches to guide local voxel features.

  • 3.1 Method Overview: Discrete SDFs represent shape geometry continuously enough for Marching Cubes extraction, avoiding some limitations of discrete occupancy or point representations.The SDF records signed distances on a regular grid, and its zero isosurface yields a mesh.
  • 3.1 Method Overview: Two-stage diffusion first generates a low-resolution surface-occupancy shell, then generates high-resolution SDF values only within the occupied region.This design addresses the cubic memory and computation cost of full-grid high-resolution SDF diffusion.
  • 3.3 Occupancy-diffusion Module: The occupancy-diffusion module denoises a 64^3 grid, reserves predicted occupied voxels, and subdivides them into a 128^3 sparse grid for SDF diffusion.The second module uses sparse-voxel processing and Marching Cubes to produce the final mesh.
  • 3.2 Self-conditioning Continuous Diffusion Model: Self-conditioning feeds a previous estimate of the clean sample into the diffusion network during training, with probability 0.5 by default.Gradient backpropagation through the previous estimate is disabled to reduce training time.
  • 3.5 View-aware Local Attention: View-aware local attention projects each voxel center onto the sketch and cross-attends to nearby image patches, improving local controllability and robustness to small view errors.The method uses projected patch neighborhoods rather than a single global image feature; rough view information can therefore suffice.

4 SKETCH-CONDITIONED SHAPE GENERATION

LAS-Diffusion is evaluated for sketch-conditioned 3D generation using predefined, perturbed views and comparisons with representative sketch-to-3D methods. The experiments examine quality, view robustness, local controllability, generalizability, ablations, scalability, and feature-based shape manipulation.

  • Experimental setup: LAS-Diffusion uses five predefined perspective views, with perturbed sketch views during training to improve inference robustness.The views are left, side-left, front, side-right, and right; each predefined view is perturbed ten times.
  • Model evaluation: View robustness tests show consistent, good geometry across side-left and side-right inputs, while strongly mismatched view information can distort or incompletely generate some parts.The model remains capable of producing chair-like shapes in the stress test despite wrong view information.
  • Model evaluation: View-aware local attention captures modified table-bar structures with high probability of structurally correct, geometry-plausible results, unlike Sketch2Model.The test changes the number of horizontal bars in a table sketch.
  • Model evaluation: LAS-Diffusion generalizes to unseen structural variations, unseen categories, and freehand sketches, although competing methods can fail on unseen geometry or produce low-quality outputs.Wing-like chair structures appear across generated samples, unseen-category sketches yield meaningful results, and freehand inputs are supported.
  • Ablation and extensions: Ablations find that global attention has limited generalizability, whereas view-agnostic attention often adds wrong geometry; swapping ViT patch features also generates novel meaningful shapes.The feature-manipulation examples combine patches from different chairs, cars, and airplanes without drawing new sketches.

5 CATEGORY-CONDITIONED SHAPE GENERATION

The category-conditioned experiments evaluate LAS-Diffusion across five ShapeNet categories using FID and additional fidelity, coverage, and diversity metrics. Results show high-quality, diverse, and generally competitive or superior generation, including on small datasets.

  • Experimental setup: The evaluation covers category-conditioned generation, including single-category and multi-category-conditioned training configurations.The dataset uses chair, car, airplane, table, and rifle categories; the multi-category model encodes class names with CLIP features.
  • Experimental setup: Shading-image-based FID evaluates generated-shape quality and diversity from 20 uniformly distributed rendered views.Lower FID indicates better generation quality and diversity.
  • FID comparison: LAS-Diffusion outperforms other methods in all five categories for single-category generation based on FID.The multi-category-conditioned model remains better than other methods except for airplane versus Wavelet-Diffusion and rifle versus SDF-StyleGAN; comparison with 3DILG is only for reference because training data differ.
  • Additional metrics: On chairs, LAS-Diffusion and Wavelet-Diffusion are comparable on MMD and 1-NNA, while LAS-Diffusion achieves better COV(EMD) than the other methods.These metrics use 2,048 uniformly sampled mesh points and assess fidelity, coverage, and diversity.
  • Diversity: Generated chairs are mostly different from the training set, and a nearest-neighbor example shows a novel structure.The analysis uses Chamfer-distance histograms and retrieves the four most similar training shapes.
  • Small datasets: LAS-Diffusion generates plausible, good-quality shapes when trained on small cap and mug datasets.The cap category contains 56 objects and the mug category 214 objects; the trained SDF-diffusion module is reused.

6 CONCLUSION AND PERSPECTIVES

LAS-Diffusion combines two-stage diffusion with view-aware local attention to generate plausible shapes with controllability and generalizability. The paper identifies appearance, multi-view input, and sketch-domain coverage as directions or limitations for future work.

  • Conclusion: The paper presents a diffusion-based technique whose view-aware local attention is integrated with two-stage diffusion for greater controllability and generalizability.The mechanism is described as simple and flexible for extension to color, depth, and 3D point-cloud inputs.
  • Limitations: The model is trained only on synthetic data, and its sketch style is tied to the rendering pipeline.It is not well adapted to highly distorted lines, oversketches, or seriously inconsistent perspectives; an oversketch example fails to generate a chair arm structure.
  • Shape appearance: The model currently focuses on shape geometry and does not provide vivid shape appearances.The authors plan to combine 2D sketches with language descriptions for geometry-compatible shape appearances.
  • Multi-view sketches: Single-view sketches do not convey the complete idea of designers, motivating future use of multi-view sketches and a supporting user interface.This is presented as a future direction rather than a demonstrated capability.
Loading 2305.04461v2…