PaperlayerView exact revision on arXiv

Source-linked AI summary

Unpaired Point Cloud Completion on Real Scans using Adversarial Training

Xuelin Chen, Baoquan Chen, Niloy J. Mitra

arXiv:1904.00069v3cs.CV

TL;DR

Real-scan completion is difficult to train with paired data, which is often unavailable and may not match synthetic training scans. This paper maps partial point clouds to complete shapes without paired correspondence, producing plausible real-scan completions and outperforming supervised baselines in reported plausibility comparisons.

  • Problem

    Many scan-completion methods require paired partial and complete training data, which is difficult to collect for real scans, and synthetic-to-real distribution differences can limit generalization.

    Source: §1 · p. 1

  • Method

    A GAN-based adaptation network maps partial point-set latent representations toward clean, complete shape representations, using a partial Hausdorff-distance reconstruction loss.

    Source: §3.2 · pp. 4–5

  • Results

    On real scans, the method produces plausible completions and is reported superior to 3D-EPN and PCN in plausibility; it also generates plausible cars from sparse KITTI inputs.

    Source: §4.1 · pp. 5–6

  • Takeaways & Limitations

    The method can use unpaired real scans and virtual object repositories as training data without requiring examples of real complete scans.

    Source: §1 · p. 1

    Real scans lack completion ground truth, so their quantitative comparison uses plausibility rather than direct accuracy measures.

    Source: §1 · p. 2, §4.1 · p. 6

Abstract

from arXiv · show

As 3D scanning solutions become increasingly popular, several deep learning setups have been developed geared towards that task of scan completion, i.e., plausibly filling in regions there were missed in the raw scans. These methods, however, largely rely on supervision in the form of paired training data, i.e., partial scans with corresponding desired completed scans. While these methods have been successfully demonstrated on synthetic data, the approaches cannot be directly used on real scans in absence of suitable paired training data. We develop a first approach that works directly on input point clouds, does not require paired training data, and hence can directly be applied to real scans for scan completion. We evaluate the approach qualitatively on several real-world datasets (ScanNet, Matterport, KITTI), quantitatively on 3D-EPN shape completion benchmark dataset, and demonstrate realistic completions under varying levels of incompleteness.

1 INTRODUCTION

Real scans are often incomplete, while paired training data is difficult to collect and synthetic-to-real distribution mismatch limits generalization. The paper proposes unpaired point-based completion and evaluates it on real and synthetic data.

1 INTRODUCTION

  • Paired-supervision methods require complete counterparts for incomplete training scans, which are difficult to collect for real scans and may not generalize across synthetic-to-real distribution gaps.Prior methods commonly simulate partial scans from virtual objects to obtain paired data.Source: §1 · p. 1
  • The method performs well when synthetic complete scans and real partial scans have different data distributions, unlike methods requiring paired supervision.The method uses no explicit correspondence between partial scans and complete shape models.Source: §1 · p. 1
  • A GAN adaptation network maps raw partial point sets toward clean, complete point sets using separately learned latent manifolds for scanned and synthetic-object data.The generator transforms input into a latent representation that the discriminator cannot distinguish from latent variables obtained from complete shape models.Source: §1 · pp. 1–2
  • The approach is demonstrated on ScanNet, Matterport3D, and KITTI real scans, with quantitative evaluation on a synthetic dataset that has completed versions.The real-scan examples cover chairs and tables in ScanNet and Matterport3D and cars in KITTI.Source: §1 · p. 2

2 RELATED WORK

Prior completion work includes voxel-based and point-based methods, but many point-cloud approaches depend on paired partial-complete supervision generated through virtual scanning. This paper positions its unpaired GAN and point-set losses against earlier voxel-based real-data and latent-space approaches.

2 RELATED WORK

  • Point-cloud completion methods fill missing parts directly but commonly train by regressing partial inputs to paired ground-truth shapes, often generated by virtual scanning, and do not easily generalize to real scans.Voxel-based completion is also widely used, though voxel quantization loses geometric information.Source: §2 · p. 2
  • Stutz & Geiger (2018) work on voxelized real-world data with a latent space and maximum-likelihood reconstruction loss; this method instead uses a GAN between latent spaces and Hausdorff distance on point sets.The paper describes its point-set approach as avoiding voxelized input.Source: §2 · p. 2
  • Earlier 3D GAN completion methods use ground-truth supervision, while this method is designed for unpaired data and real-synthetic distribution differences.The cited supervised approaches include Yang et al. (2018) and Wang et al. (2017).Source: §2 · p. 2
  • PointNet and PointNet++ provide point-cloud analysis backbones, and Achlioptas et al. (2018) report advantages from training GANs in an autoencoder-produced latent space.Inspired by that latent-space approach, this paper translates between two latent spaces for unpaired completion.Source: §2 · pp. 2–3

3 METHOD

The method completes noisy partial point sets without paired correspondences by mapping between learned latent spaces for scanned inputs and clean, complete shapes. An adversarial objective encourages plausible complete outputs, while a reconstruction term preserves their match to the partial input.

3 METHOD

  • The method learns an unpaired mapping from noisy partial scans to clean, complete point sets using class-specific latent spaces and adversarial training.It assumes access to clean, complete point sets but does not require explicit correspondence between them and the partial inputs.Source: §3 · p. 3

3.1 LEARNING LATENT SPACES FOR POINT SETS

  • Autoencoders learn compact latent representations of clean-complete and noisy-partial point sets, allowing the method to operate directly on point sets rather than voxel grids.Using the clean-complete autoencoder’s latent space for noisy-partial inputs worked slightly better in experiments than training a separate noisy-partial autoencoder.Source: §3.1 · pp. 3–4

3.2 LEARNING A MAPPING BETWEEN LATENT SPACES

  • An adversarial generator maps noisy-partial latent codes toward the clean-complete latent space, while a Hausdorff reconstruction loss encourages the completion to match the input partially.The loss uses partial supervision because no complete ground-truth scan is available; using Earth Mover’s Distance for this term led to worse completion results.Source: §3.2 · pp. 4–5

4 EXPERIMENTAL EVALUATION

The evaluation spans real-world scans and synthetic benchmarks, testing baseline performance, distribution shift, increasing incompleteness, completion diversity, and network design choices.

4 EXPERIMENTAL EVALUATION

  • Experiments cover ScanNet, Matterport3D, KITTI, 3D-EPN, and synthetic ShapeNet-derived scans, using plausibility for real data and accuracy, completeness, and F1 where ground truth is available.The method is trained on ScanNet and tested on Matterport3D to evaluate generalization to a new data source.Source: §4 · p. 5

4.1 EXPERIMENTAL RESULTS ON REAL-WORLD DATA

  • On real scans, our method produces plausible completions, exceeds supervised methods in plausibility, and generates plausible KITTI cars despite sparse inputs.3D-EPN trained on simulated partial data failed on real-world scans.Source: §4.1 · pp. 5–6, Figure 4 · p. 6

4.2 COMPARISON WITH BASELINES ON 3D-EPN DATA

  • On 3D-EPN, the unpaired method outperforms 3D-EPN and achieves comparable results to PCN; adapted Ours+ achieves similar performance to PCN.Both 3D-EPN and PCN require paired data.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 EFFECT OF DATA DISTRIBUTION DISCREPANCY AND VARYING INCOMPLETENESS

  • As input incompleteness increases, AE performance declines dramatically while ours retains high plausibility and F1; supervised methods also degrade on shifted-distribution scans.The Table 3 caption attributes robustness to restricting completions through learned latent shape manifolds.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 DIVERSITY OF THE COMPLETION RESULTS

  • JSD is 0.06 vs. 0.46 on cars, 0.05 vs. 0.61 on chairs, 0.04 vs. 0.53 on planes, and 0.04 vs. 0.59 on tables (ours vs. extreme collapse).Lower JSD is better; the reference sets repeat a randomly selected point cloud.Source: §4.4 · p. 8

4.5 ABLATION STUDY

  • The full method performs best among ablations testing a partial autoencoder, EMD reconstruction, removing GAN, or removing reconstruction loss.The ablations vary these design choices and modules.Source: §4.5 · p. 8, Table 4 · p. 8

5 CONCLUSION

The conclusion presents an unpaired point-based framework that maps raw partial scans to clean, complete point clouds, with plausible completions reported on real and virtual scans.

  • Evaluations on real and virtual scans show consistently plausible completions and superior performance to other methods.Source: §5 · p. 8
  • An adaptation network maps raw-scan latent encodings to clean, complete object-scan encodings, with transfer restricted to the respective data manifolds.Source: §5 · p. 8
  • The method does not produce fine-scale details and assumes canonically oriented input scans.Source: §5 · p. 8
  • The authors identify scene-level completion and combining point and image features for geometry and texture as future directions.Source: §5 · p. 8

A DETAILS OF DATASETS

The experiments use ShapeNet-derived complete point sets alongside real and synthetic partial scans and the 3D-EPN benchmark, with different data sources serving distinct training and evaluation roles.

  • ShapeNet models from eight categories are virtually scanned and downsampled to N points to learn the clean-complete point-set manifold.The models are used without pose or scale augmentation.Source: A DETAILS OF DATASETS · p. 12
  • ScanNet contributes approximately 550 chairs and 550 tables, manually aligned to ShapeNet orientation and split 90%/10% for training and testing.Source: A DETAILS OF DATASETS · p. 12
  • Matterport3D contributes 20 chairs and 20 tables; models trained on the ScanNet training split are tested on these unseen data.Source: A DETAILS OF DATASETS · p. 12
  • KITTI car observations use ground-truth bounding boxes and exclude observations with fewer than 100 points.Source: A DETAILS OF DATASETS · p. 12
  • The 3D-EPN benchmark covers eight categories and 25,590 instances, and the method uses its provided point-cloud representations rather than richer SDF training data.The test set contains 5,384 models.Source: A DETAILS OF DATASETS · p. 12
  • Synthetic partial scans use ShapeNet cars, chairs, planes, and tables, with incompleteness controlled by removing nearest neighbors and Gaussian noise added at σ=0.01.Resulting point sets are duplicated to contain N points.Source: A DETAILS OF DATASETS · p. 12

B NETWORK ARCHITECTURE DETAILS

This section specifies the encoder, decoder, generator, and discriminator architectures used in the network.

B.1 AE ARCHITECTURE DETAILS

  • The encoder applies five pointwise 1-D convolutional layers, then feature-wise maximum pooling to produce a 128-dimensional latent code.Its layers use ReLU and batch normalization; the filter counts are 64, 128, 128, 256, and 128.Source: §B.1 · pp. 12–13
  • The decoder uses three fully connected layers to reconstruct an N × 3 point set from the latent vector.The layers have 256, 256, and N × 3 neurons; the first two use ReLUs.Source: §B.1 · p. 13

B.2 GAN ARCHITECTURE DETAILS

  • The latent-space generator maps noisy-incomplete codes to clean-complete codes using two fully connected layers, while the discriminator outputs one scalar per code.The generator layers have 128 neurons each; the discriminator has 256, 512, and 1 neurons.Source: §B.2 · p. 13

C TRAINING DETAILS

Training first pretrains the autoencoders that establish the latent spaces, then updates only the generator and discriminator during GAN training.

  • The autoencoders are pretrained, their weights retained, and only generator and discriminator weights updated during GAN training.Source: C TRAINING DETAILS · p. 13
  • Autoencoders use Adam with learning rate 0.0005, β1 = 0.9, batch size 200, and up to 2,000 epochs.Source: C TRAINING DETAILS · p. 13
  • Generator and discriminator training uses Adam with learning rate 0.0001, β1 = 0.5, batch size 24, and alternating updates for up to 1,000 epochs.Source: C TRAINING DETAILS · p. 13

D QUALITATIVE RESULTS ON 3D-EPN DATASET

On 3D-EPN, the method produces qualitatively plausible completions despite not being quantitatively the best.

  • The method’s completions are qualitatively plausible, although they are not quantitatively the best.Figure 5 compares the partial input, AE, 3D-EPN, PCN, Ours, Ours+, and ground-truth point sets.Source: D QUALITATIVE RESULTS ON 3D-EPN DATASET · p. 13

E VISUAL COMPARISON ON TEST DATA WITH DISTRIBUTION DIFFERENT TO TRAINING DATA

On synthetic data with a different training distribution, the method retains high-quality completions while 3D-EPN and PCN produce much worse results. With ground truth available, the EMD-only variant outperforms the adversarial variant.

E VISUAL COMPARISON ON TEST DATA WITH DISTRIBUTION DIFFERENT TO TRAINING DATA

  • On synthetic data with a different training distribution, our method retains high-quality completions, while 3D-EPN and PCN produce much worse results.Ground truth for this synthetic data is available only for evaluation; the authors attribute the baselines’ weaker results to the training-data distribution difference.Source: E VISUAL COMPARISON ON TEST DATA WITH… · p. 13

F VARIATIONS OF LEVERAGING GROUND TRUTH SUPERVISION

G MORE STATISTICS FRO THE BASELINE COMPARISON

H USER STUDY ON REAL-WORLD DATA COMPLETION

In a user study of real-world scan completions, users selected the authors’ results in 52% of choices and PCN in 45%, though users often favored preserving the input. On unseen classes, the table network produces some reasonable chair structures but fails on airplanes.

H USER STUDY ON REAL-WORLD DATA COMPLETION

  • In 1,000 valid user selections, the authors’ completions were preferred in 52% and PCN’s in 45%.The authors note their method did not hold a dominant position; users often favored embedding the partial input and neglected plausibility.Source: H USER STUDY ON REAL-WORLD DATA… · p. 15
  • Users often favored completions that preserve the partial input, while the authors’ generator trades off Hausdorff distance and plausibility.A PCN chair completion without clear chair structure was often selected over the authors’ trade-off completion.Source: H USER STUDY ON REAL-WORLD DATA… · pp. 15–16

I GENERALIZATION TO UNSEEN CLASSES

  • The table network produced some reasonable structures for unseen chairs but could not add a seat back, and it failed on unseen airplanes.The chair shares structure with tables, whereas airplanes are described as rather dissimilar to tables.Source: I GENERALIZATION TO UNSEEN CLASSES · pp. 16–17
Loading 1904.00069v3…