PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised 3D Shape Completion through GAN Inversion

Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, Chen Change Loy

arXiv:2104.13366v2cs.CV

TL;DR

Supervised shape completion often struggles to generalize from paired training shapes to other partial forms and real-world scans. ShapeInversion searches a pretrained GAN’s latent space for a complete shape matching the partial input. On ShapeNet, it outperforms the leading unsupervised method and is comparable to supervised methods, while also generalizing across partial forms and real-world scans.

  • Problem

    Most shape-completion methods rely on paired partial-complete training data and can struggle to generalize to other partial forms or real-world scans.

    Source: §1 · p. 1

  • Method

    ShapeInversion searches a pretrained GAN’s latent space for a complete shape that best reconstructs the partial input, without requiring paired training data.

    Source: §1 · p. 1

  • Results

    ShapeInversion significantly outperforms unsupervised pcl2pcl [7] on ShapeNet and is comparable to supervised methods, with robust results on real scans and varied partial inputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For ambiguous partial inputs, ShapeInversion can produce multiple valid complete shapes that remain faithful to the visible parts.

    Source: §1 · p. 2

    The reported shape completion and manipulation are conducted on models pretrained with a single category; improving multi-class model fidelity remains future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Most 3D shape completion approaches rely heavily on partial-complete shape pairs and learn in a fully supervised manner. Despite their impressive performances on in-domain data, when generalizing to partial shapes in other forms or real-world partial scans, they often obtain unsatisfactory results due to domain gaps. In contrast to previous fully supervised approaches, in this paper we present ShapeInversion, which introduces Generative Adversarial Network (GAN) inversion to shape completion for the first time. ShapeInversion uses a GAN pre-trained on complete shapes by searching for a latent code that gives a complete shape that best reconstructs the given partial input. In this way, ShapeInversion no longer needs paired training data, and is capable of incorporating the rich prior captured in a well-trained generative model. On the ShapeNet benchmark, the proposed ShapeInversion outperforms the SOTA unsupervised method, and is comparable with supervised methods that are learned using paired data. It also demonstrates remarkable generalization ability, giving robust results for real-world scans and partial inputs of various forms and incompleteness levels. Importantly, ShapeInversion naturally enables a series of additional abilities thanks to the involvement of a pre-trained GAN, such as producing multiple valid complete shapes for an ambiguous partial input, as well as shape manipulation and interpolation.

1. Introduction

Supervised shape completion performs well in-domain but struggles with real-world scans and other partial forms. ShapeInversion searches a pretrained GAN’s latent space for a complete shape matching the partial input, using PatchVariance and k-Mask to address challenges specific to point clouds.

1. Introduction

  • ShapeInversion applies GAN inversion to completion by searching for a latent code whose generated complete shape best reconstructs the partial input.Using a pretrained GAN prior requires no paired training data or assumptions about partial-input form.Source: §1 · p. 1
  • Unstructured point clouds make visible-region correspondence difficult, while non-uniform point distributions can create holes and undermine completion.Unlike lattice-aligned image pixels, corresponding regions of two shapes may occupy different locations in 3D space.Source: §1 · pp. 1–2
  • PatchVariance samples surface patches and penalizes variance in average distances from patch centers to nearest neighbors, acting as an object-level soft uniformity regularizer during GAN training.The paper reports improved uniformity across categories, from bulk to fine structures, while preserving shape plausibility and variety.Source: §1 · p. 2
  • k-Mask dynamically estimates visible-region correspondences by letting each partial-input point select its k nearest neighbors in the predicted shape.The resulting mask supports reconstruction loss and is reported to outperform predefined voxel or distance-threshold masks, including when corresponding semantic parts are spatially distant.Source: §1 · p. 2
  • ShapeInversion significantly outperforms unsupervised pcl2pcl [7] on ShapeNet and is comparable to supervised methods, while generalizing across partial forms and real-world scans.It can also produce multiple valid completions faithful to visible parts when the input is ambiguous.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Work

Prior completion methods are mostly supervised on paired shapes and may generalize poorly beyond their training partial-shape form. The paper situates ShapeInversion at the intersection of unsupervised completion and GAN inversion, applying the latter to 3D shape completion for the first time.

2. Related Work

  • Most existing completion methods use paired partial and complete shapes, achieving strong in-domain results but potentially generalizing poorly to real scans or other partial forms.Unsupervised pcl2pcl [7] instead maps partial-shape latent codes to complete-shape latent codes using separate auto-encoders.Source: §2 · p. 2
  • Prior GAN inversion work uses a pretrained generator as a prior, finding a latent vector that reconstructs an input image for restoration or manipulation tasks.Latents may be optimized by gradient descent or projected by an encoder; some methods perturb or fine-tune the generator.Source: §2 · p. 3
  • ShapeInversion is presented as the first GAN-inversion approach to shape completion, using a pretrained GAN prior and also producing multiple reasonable completions.The paper describes its results as more faithful, particularly for real scans, than the related unsupervised approach.Source: §2 · p. 3

3. Method

ShapeInversion completes partial shapes by inverting a pretrained 3D GAN, optimizing its latent code and generator parameters against a degraded reconstruction of the input. The method addresses point-cloud nonuniformity with PatchVariance, uses k-Mask for point correspondences, and combines structural and feature losses during inversion.

3. Method

  • ShapeInversion jointly optimizes a latent code and generator parameters so a pretrained GAN produces a complete shape whose degraded form matches the partial input.It initializes the latent code by sampling hundreds of candidates, then updates the code and generator by gradient descent.Source: §3 · p. 3, Figure 2 · p. 3

3.1. Enhancing Point Cloud Uniformity

  • PatchVariance regularizes whole-shape point uniformity during GAN training by penalizing variance in average distances across sampled surface patches.The authors report that it significantly improves generated-shape uniformity and works independently of GAN architecture.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. Degradation in the 3D Space

  • k-Mask forms a degraded shape from the generated shape’s k-nearest neighbors for each input point, providing more accurate and robust correspondences than voxel-Mask or τ-Mask.The alternatives use fixed voxel-size or distance thresholds and cannot adapt to regional point-density changes.Source: §3.2 · pp. 4–5

3.3. Loss Function for Inversion

  • Inversion combines a structural loss with discriminator-feature matching to compare the degraded generated shape with the partial input.The authors use CD-T because EMD is infeasible when the degraded shape and input have different point counts; feature matching uses an intermediate discriminator layer after max-pooling.Source: §3.3 · p. 5

4. Experiments

ShapeInversion is evaluated on ShapeNet completion, cross-domain partial forms, real scans, ambiguous inputs, and shape manipulation, alongside ablations of its components. It outperforms the unsupervised benchmark baseline, is comparable to supervised methods, and shows robustness across domains and additional shape-generation uses.

4. Experiments

  • Experiments cover synthetic partial shapes, real scans, and standard completion metrics, with a single-class, 2048-point tree-GAN used throughout.The synthetic inputs include virtual scans, ball-holed shapes, and PartNet incompleteness; real scans come from KITTI, ScanNet, and MatterPort3D.Source: §4 · pp. 5–6

4.1. Ablation Study

  • The ablation study finds that PatchVariance improves generated-shape uniformity, while k-Mask and feature distance improve completion-stage performance.PatchVariance beats expansion penalty and repulsion loss; alternative degradation functions perform worse than k-Mask, and feature distance adds semantic information to structural loss.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.2. Shape Completion on Virtual Scan Benchmark

  • ShapeInversion beats pcl2pcl across all eight ShapeNet categories and is comparable to supervised methods, despite completing in a single stage.The comparison uses the virtual-scan benchmark; some supervised methods calibrate coarse outputs using the partial input during refinement.Source: §4.2 · p. 6, Table 3 · p. 7

4.3. Robustness to Varying Partial Forms

  • ShapeInversion gives the best results in almost all cross-domain tests, while methods—especially supervised ones—often drop substantially on out-of-domain inputs.The comparison evaluates CD across partial forms, with in-domain results listed for reference.Source: Table 4 · p. 7, §4.3 · p. 7, Figure 6 · p. 7

4.4. Completion of Real-World Scans

  • On real scans, ShapeInversion significantly outperforms pcl2pcl on UCD and achieves comparable UHD, with better UHD after adding UHD to its loss.The added UHD objective brings a small compromise in UCD; pcl2pcl is retrained with real-world scans.Source: §4.4 · p. 7, Table 5 · p. 8

4.5. Multiple Valid Outputs under Ambiguity

  • For more severely incomplete inputs, ShapeInversion produces multiple diverse, reasonable completions that each remain faithful to visible details.A larger loss threshold and fewer inversion iterations offer a more flexible diversity-fidelity trade-off.Source: §4.5 · pp. 7–8, Figure 9 · p. 8

4.6. Shape Manipulation

  • After reconstruction, latent-space perturbation enables jittering, while interpolation between latent vectors and generator parameters enables morphing.The resulting new shapes and transitions are reported as plausible.Source: §4.6 · p. 8, Figure 7 · p. 8

5. Conclusion

ShapeInversion uses GAN inversion and two components to address challenges in unsupervised point-cloud completion, and supports multiple completions and shape manipulation. Its current completion and manipulation models are pre-trained on a single category.

5. Conclusion

  • ShapeInversion addresses domain gaps across real-world scans and varied partial shapes using GAN inversion, PatchVariance, and k-Mask.The pre-trained GAN captures shape geometry and semantics; the framework also supports multiple reasonable completions, shape jittering, and interpolation.Source: §5 · p. 8
  • Both shape completion and manipulation are currently conducted with models pre-trained on a single category.The authors identify improving multi-class model fidelity as future work.Source: §5 · p. 8

A. Additional Details of Evaluation Metrics

The evaluation metrics assess generated-shape fidelity, input-to-output distance, one-sided Hausdorff distance, and point-cloud accuracy and completeness.

A. Additional Details of Evaluation Metrics

B. Additional Details of Datasets

The evaluation details specify dataset splits and partial-shape construction for simulated and real-world scans, including differing training-data requirements for ShapeInversion and pcl2pcl.

B. Additional Details of Datasets

  • PartNet test shapes also present in ShapeNet training data are removed when evaluating the PartNet target domain.Virtual scans and ball-holed partial shapes use ShapeNet’s train-test split; PartNet uses the split provided by MPC.Source: B. Additional Details of Datasets · p. 9
  • Ball-holed partial shapes are generated by removing exactly 512 points from each complete shape.Higher incompleteness levels are tested in the main paper’s Figures 1 and 9.Source: B. Additional Details of Datasets · p. 9
  • For MatterPort3D, pcl2pcl uses a model retrained on ScanNet, whereas ShapeInversion is evaluated directly on the test set without real-scan training data.ScanNet and KITTI are split into train and test sets, and pcl2pcl’s mapping GAN is retrained on the real-scan training set.Source: B. Additional Details of Datasets · p. 9

C. Additional Details on Implementation

Implementation details cover GAN pre-training, PatchVariance sampling, and staged inversion settings for different shape categories.

C. Additional Details on Implementation

  • Tree-GAN is trained for 2000 epochs on eight Nvidia V100 GPUs, with batch sizes of 512 for four larger categories and 128 for four smaller categories.The larger categories are plane, car, chair, and table; the smaller categories are cabinet, lamp, sofa, and boat.Source: C. Additional Details on Implementation · p. 9
  • PatchVariance uses 100 patches of 30 points each, a setting also used for r-GAN.The passage states this samples the entire generated 2048-point shape via FPS.Source: C. Additional Details on Implementation · p. 9
  • Inversion uses k = 5 for k-Mask and four learning-rate sub-stages, with 30 iterations per stage for bulk structures and 200 for thin structures.The learning rates differ for latent code z and generator parameters θ across the four stages.Source: C. Additional Details on Implementation · p. 9

D. Additional Qualitative Results

The additional qualitative results cover ShapeNet and real-world scans, cross-domain inputs, PatchVariance on r-GAN, and ShapeInversion’s shape-jittering and interpolation capabilities.

D. Additional Qualitative Results

  • PatchVariance enhances generated point-cloud uniformity for r-GAN and is invariant to generator architecture.Figure 10 depicts point density, with darker regions indicating higher density.Source: Figure 10 · p. 10
  • Without using real scans, ShapeInversion reconstructs high-fidelity shapes more faithful to real-world partial inputs than pcl2pcl retrained with real-world partial shapes.The pcl2pcl retraining uses real-world partial shapes together with synthetic complete shapes in an unpaired manner [7].Source: Figure 11 · p. 10
  • The ShapeNet qualitative comparison uses 2048 points for each partial input, complete shape, and ground truth.The section also includes qualitative results for virtual scans, cross-data validation, real scans, jittering, and morphing.Source: D. Additional Qualitative Results · p. 9, Figure 12 · p. 11
  • ShapeInversion provides plausible, accurate outputs across partial forms, while supervised CRN and PF-Net show significant performance drops under domain change.Unsupervised pcl2pcl and MPC show relatively better out-of-domain results; CRN uses partial inputs during refinement, whereas PF-Net predicts missing regions and combines them with the input.Source: Figure 13 · p. 12
  • ShapeInversion’s latent-space perturbations change a shape into other plausible shapes with different geometries.Figure 14 shows additional results for this random jittering.Source: Figure 14 · p. 13
  • Interpolating between corresponding latent vectors z and generator parameters θ produces a sound transition between shapes.Figure 15 presents additional shape-interpolation results.Source: Figure 15 · p. 14
Loading 2104.13366v2…