PaperlayerView exact revision on arXiv

Source-linked AI summary

SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation

Ruihui Li, Xianzhi Li, Ka-Hei Hui, Chi-Wing Fu

arXiv:2108.04476v1cs.CV

TL;DR

Part-aware generation can depend on part annotations, while direct point-cloud generators can struggle with fine detail, noise, and structure control. SP-GAN pairs a sphere prior with local random codes to generate point clouds and establish implicit correspondence, and the authors report detailed, less noisy shapes and structure-aware manipulation. The point-cloud representation does not directly provide surface or topological information.

  • Problem

    Part-based structure-aware generation depends on part annotations, while direct point-cloud generators can struggle with fine details, noisy samples, and controllable part correspondences.

    Source: §1 · p. 1

  • Method

    SP-GAN pairs uniformly distributed unit-sphere points with local random latent codes, using the sphere for spatial guidance and the codes for local detail.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    SP-GAN generates diverse, realistic point clouds with finer details and less noise than previous works, while enabling shape-wise or part-wise latent-code modification and interpolation.

    Source: §7 · p. 11

  • Takeaways & Limitations

    The implicit dense correspondence supports part editing, part-wise interpolation, and multi-shape part composition without requiring part annotations for training.

    Source: §1 · p. 2, §5.4 · p. 9

    Point clouds do not directly provide surface or topological information, so surface reconstruction requires post-processing and can produce distorted edges or holes.

    Source: §6.4 · p. 11

Abstract

from arXiv · show

We present SP-GAN, a new unsupervised sphere-guided generative model for direct synthesis of 3D shapes in the form of point clouds. Compared with existing models, SP-GAN is able to synthesize diverse and high-quality shapes with fine details and promote controllability for part-aware shape generation and manipulation, yet trainable without any parts annotations. In SP-GAN, we incorporate a global prior (uniform points on a sphere) to spatially guide the generative process and attach a local prior (a random latent code) to each sphere point to provide local details. The key insight in our design is to disentangle the complex 3D shape generation task into a global shape modeling and a local structure adjustment, to ease the learning process and enhance the shape generation quality. Also, our model forms an implicit dense correspondence between the sphere points and points in every generated shape, enabling various forms of structure-aware shape manipulations such as part editing, part-wise shape interpolation, and multi-shape part composition, etc., beyond the existing generative models. Experimental results, which include both visual and quantitative evaluations, demonstrate that our model is able to synthesize diverse point clouds with fine details and less noise, as compared with the state-of-the-art models.

1 INTRODUCTION

Part-based generation offers structure-aware control but depends on the availability and quality of part annotations, while direct point-cloud models can struggle with detail, noise, and controllability. SP-GAN uses sphere-guided global and local priors to generate point clouds and support part-aware manipulation without part annotations.

1 INTRODUCTION

  • Part-based methods enable structure-aware manipulation but depend on part annotations, while direct point-cloud models can struggle with fine details, noisy samples, and structure controllability.A single latent code characterizes overall shape variation, making plausible correspondence between parts and latent dimensions or across generated shapes difficult.Source: §1 · pp. 1–2
  • SP-GAN is trained without part annotations and is reported to generate diverse, high-quality point clouds with finer details and less noise than state-of-the-art models.The paper presents these capabilities as supported by experimental evaluations.Source: §1 · p. 2
  • SP-GAN pairs a fixed unit-sphere prior for unbiased spatial guidance with a local random code at every sphere point for detail.The authors describe this as disentangling global shape modeling from local structure adjustment, which they say eases learning and enhances generated-shape quality.Source: §1 · p. 2
  • An implicit dense correspondence between sphere points and generated shapes supports part-aware editing and interpolation across shapes.Changing selected local codes can alter associated parts, while other parts may shift slightly for structural compatibility.Source: §1 · p. 2, Figure 2 · p. 2

2 RELATED WORK

Prior 3D shape-generation methods use different representations and conditional or unconditional strategies, with trade-offs in detail, flexibility, and computational cost. SP-GAN instead uses a sphere prior to guide point-cloud generation while implicitly embedding dense correspondence.

2 RELATED WORK

  • Part-conditioned methods compose shapes from labeled parts, whereas unconditional methods synthesize directly from a random distribution without predefined part structures.Part-conditioned methods depend on part labels and predefined or learned part distributions.Source: §2 · p. 2
  • Unconditional point-cloud generators include autoregressive, flow-based, and GAN-based approaches; autoregressive generation is iterative, while flow models face invertibility-related representation limits and may produce blurry or noisy samples.PointGrow samples points sequentially [Sun et al. 2020]; flow methods include transformations from generic priors [Yang et al. 2019; Klokov et al. 2020].Source: §2 · pp. 2–3
  • Unlike GAN approaches that use a single latent code, SP-GAN uses a fixed sphere prior in a disentangled model for high-quality, detailed outputs and structure-aware controllability.It attaches local noise codes to sphere points, with their relations defining the generated shape’s global structure.Source: §2 · p. 3
  • Voxel grids can produce coarse shapes, implicit fields require extensive surface sampling, and mesh methods face genus constraints and often require conditional inputs.Point clouds allow flexible topology and are easy to manipulate and interpolate.Source: §2 · p. 3
  • SP-GAN embeds dense correspondence implicitly and automatically, in contrast to the explicit deformed implicit-field template approach of Deng et al. [2021].P2P-Net attaches independent noise vectors to point features, whereas SP-GAN attaches latent codes to sphere points.Source: §2 · p. 3

3 OVERVIEW

SP-GAN feeds a generator a prior latent matrix pairing sphere points with local random codes, then learns to produce point clouds resembling repository shapes. The resulting implicit correspondence supports localized editing and part-wise interpolation.

3 OVERVIEW

  • The generator input pairs N uniformly distributed unit-sphere points with d-dimensional random latent vectors, and adversarial training matches generated point clouds to repository shapes.The discriminator distinguishes generated P from sampled repository point clouds, while the generator learns to produce P resembling those samples.Source: §3 · p. 3
  • The sphere proxy establishes dense point-wise correspondence to generated shapes, and that correspondence extends across outputs.The paper illustrates the association with smoothly varying point colors on the sphere and generated shapes.Source: §3 · p. 3, Figure 3 · p. 4
  • For part editing, changing selected sphere-point codes changes associated local parts, with other parts sometimes adjusted for compatibility.The chair example replaces codes associated with its back; the regenerated back follows the new code.Source: §3 · p. 4
  • Interpolating codes associated with a selected part across generated shapes produces a sequence whose corresponding parts morph between those shapes.The overview illustrates this with the lower parts of two chairs.Source: §3 · p. 4

4 SP-GAN

SP-GAN combines sphere-derived spatial features with per-point latent styles, and uses shape-level and point-level discriminator scores; alternating training holds the sphere fixed while resampling latent codes.

4.1 Generator

  • Sphere coordinates supply spatial features through relation-weighted graph attention, while per-point latent styles modulate them via adaptive instance normalization before PointNet reconstructs coordinates.Source: §4.1 · pp. 4–5

4.2 Discriminator

  • The discriminator predicts both a per-shape score and per-point scores, regularizing global structure and local detail.Source: §4.2 · p. 5, Figure 6 · p. 5

4.3 Training and Implementation details

  • Least-squares training combines shape and averaged point losses for the discriminator and balances terms in the generator objective; alternating optimization keeps S fixed while resampling z.Source: §4.3 · pp. 5–6
  • At inference, the trained generator reuses the training sphere and produces diverse point clouds from different random z codes; implementation uses Adam for 300 epochs.Source: §4.3 · p. 6

5 SHAPE GENERATION AND MANIPULATION

This section demonstrates how SP-GAN’s generated point clouds support shape generation, localized editing and interpolation, composition, and co-segmentation. Sphere-linked correspondence underlies these structure-aware operations, while experiments report detailed, low-noise outputs and favorable benchmark results.

5.1 Shape Generation

  • Generated shapes show rich global and local variety, fine details, and little noise; corresponding sphere regions align with semantic parts across shapes, including animals with different poses.The sphere visualization uses matching colors for associated points.Source: §5.1 · p. 6, Figure 7 · p. 6

5.2 Single-shape Part Editing

  • Modifying latent codes associated with selected parts changes their local styles, while other parts may shift slightly for compatibility.The interface supports lasso selection and a real-time preview of the manipulation.Source: Figure 8 · p. 6, §5.2 · p. 8, Figure 10 · p. 8

5.3 Shape Interpolation

  • Interpolating between whole-shape latent codes produces smooth transitions, including between human poses, while intermediate shapes retain fine details and little noise.The interpolation weight ranges from zero to one.Source: §5.3 · p. 8, Figure 11 · p. 8
  • Part-wise interpolation changes selected latent codes while unselected points may shift slightly; a two-dimensional variant varies interpolation weight across rows and selected-region size down columns.The selected region’s style transitions toward the target as interpolation progresses.Source: §5.3 · pp. 8–9, Figure 13 · p. 9

5.4 Multi-shape Part Composition

  • Repacking parts and their latent codes from different shapes produces compositions whose local parts generally inherit the parent shapes’ styles.The selected parts are combined in a new prior latent matrix with the sphere prior.Source: §5.4 · p. 9, Figure 14 · p. 9

5.5 Part Co-Segmentation

  • 87.7% mIoU on 100 manually labeled SMAL shapes; one-shot co-segmentation also spans different poses and animal types, with 48.5% user preference.Only one shape in the set is manually segmented; the remaining shapes receive labels through dense correspondence.Source: §5.5 · p. 9, Figure 15 · p. 9
  • SP-GAN achieves the best quality (lowest MMD, largest COV, and lowest FPD) on Airplane and Chair, with the fewest parameters and shortest training time among five methods.Visual comparisons also show finer details and less noise than other methods.Source: Table 1 · p. 10, Figure 16 · p. 10

6 EVALUATION AND DISCUSSION

SP-GAN outperforms five comparison models on all three evaluation metrics for Chair and Airplane shapes, with finer details and less noise in qualitative comparisons. Ablations support the contribution of each component, while retrieval analysis shows distinct local details and the discussion identifies limits in training-data needs, thin structures, surface reconstruction, and explicit part relations.

6.1 Comparing with Other Methods

  • The evaluation trains on ShapeNet Chair and Airplane shapes, generates 1,000 shapes per category, and uses MMD, COV, and FPD.The five comparison models use their publicly released trained networks.Source: §6.1 · p. 10
  • SP-GAN outperforms five comparison methods on MMD, COV, and FPD for both datasets, with finer details and less noise in qualitative comparisons.The authors also report fewer learnable parameters and less training time.Source: §6.1 · pp. 10–11

6.2 Ablation Study

  • Removing any tested component—GAM, AdaIN, PPS, or the sphere prior—reduces performance on Chair relative to the full pipeline.The ablation compares the full pipeline with four simplified variants.Source: §6.2 · p. 11, Table 2 · p. 11

6.3 Shape Diversity Analysis

  • Generated shapes resemble retrieved training shapes in overall structure but retain different local details, supporting the reported shape-diversity analysis.The authors also report less noise in their generated points than in those from other unsupervised generative models.Source: §6.3 · p. 11, Figure 17 · p. 11

6.4 Discussion on Limitations

  • Training requires many shapes, and limited data or complex or thin structures can yield blurry or noisy shapes; surface reconstruction can produce distorted edges and holes.The point-cloud representation does not directly provide surface or topological information, so surface reconstruction requires post-processing.Source: §6.4 · p. 11, Figure 18 · p. 11

7 CONCLUSION AND FUTURE WORK

SP-GAN separates shape generation into global shape modeling and local structure regulation using sphere points and attached latent codes. Its implicit dense correspondence supports shape-wise and part-wise manipulation, and the authors propose extending the approach to user feedback and other 3D representations.

7 CONCLUSION AND FUTURE WORK

  • SP-GAN combines a global sphere-point prior with local random codes and uses style embedding and adaptive instance normalization to regulate local structure.The design frames generation as global shape modeling alongside local structure regulation.Source: §7 · p. 11
  • Implicit dense correspondence supports modifying or interpolating latent codes by shape or part, while experiments show diverse shapes with finer details and less noise than previous works.Source: §7 · p. 11
  • Future work will explore different forms of user feedback and generation of polygonal meshes or implicit surfaces.Source: §7 · pp. 11–12

Related papers

Loading 2108.04476v1…