PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion

Muhammad Sarmad, Hyunjoo Jenny Lee, Young Min Kim

arXiv:1904.12304v1cs.CVcs.AI

TL;DR

Incomplete point clouds are difficult to complete because acquisition can leave large missing regions and the data are unstructured. RL-GAN-Net uses reinforcement learning to control a GAN through a learned latent representation, completing shapes in approximately one millisecond and improving PointNet classification accuracy from 50% to 83% with 70% missing points.

  • Problem

    Point clouds often contain large missing regions and are unstructured, limiting direct use of conventional convolutional networks for shape completion.

  • Method

    RL-GAN-Net trains an autoencoder and GAN on latent representations, then uses an actor-critic reinforcement-learning agent to select the generator input for completion.

  • Results

    The method completes shapes in approximately 1 millisecond, handles up to 70% missing points, and raises PointNet accuracy from 50% to 83% on data with 70% missing points.

  • Takeaways & Limitations

    RL-GAN-Net provides real-time point-cloud completion and can improve classification performance when point-cloud data contain substantial missing regions.

  • Takeaways & Limitations

    Failure cases occur when the internally inferred semantic category does not exactly align with the observed shape, while the hybrid approach achieves better Chamfer distance.

Abstract

from arXiv · show

We present RL-GAN-Net, where a reinforcement learning (RL) agent provides fast and robust control of a generative adversarial network (GAN). Our framework is applied to point cloud shape completion that converts noisy, partial point cloud data into a high-fidelity completed shape by controlling the GAN. While a GAN is unstable and hard to train, we circumvent the problem by (1) training the GAN on the latent space representation whose dimension is reduced compared to the raw point cloud input and (2) using an RL agent to find the correct input to the GAN to generate the latent space representation of the shape that best fits the current input of incomplete point cloud. The suggested pipeline robustly completes point cloud with large missing regions. To the best of our knowledge, this is the first attempt to train an RL agent to control the GAN, which effectively learns the highly nonlinear mapping from the input noise of the GAN to the latent space of point cloud. The RL agent replaces the need for complex optimization and consequently makes our technique real time. Additionally, we demonstrate that our pipelines can be used to enhance the classification accuracy of point cloud with missing data.

1. Introduction

RL-GAN-Net addresses large missing regions in unordered, noisy point clouds by using a reinforcement-learning agent to control a GAN in latent space. The framework completes severely incomplete shapes within milliseconds without complex optimization and supports downstream point-cloud processing such as classification.

  • Problem: Point-cloud acquisition commonly produces large missing regions because of limited viewpoints, occlusions, sensor resolution, unstable measurements, or specular materials.The raw data are unstructured and unordered, making conventional image-oriented CNN processing unsuitable.
  • Results: Input data missing 70% of the original points are completed within milliseconds, including severely distorted cases with high fidelity compared to an autoencoder approach.The method is designed to fill missing regions without prior knowledge of visibility or noise characteristics.
  • Method: RL-GAN-Net completes partial point clouds by selecting an appropriate GAN input through a reinforcement-learning agent.An autoencoder first provides a latent-space representation, which is then used to train the GAN.
  • Novelty: The paper presents the first reported combination of deep reinforcement learning and GAN control for point-cloud shape completion.The authors formulate a unique reinforcement-learning problem for this task.
  • Contributions: The framework provides real-time GAN control that generates desired outputs without complex optimization.Its real-time operation enables preprocessing for other point-cloud pipelines, including classification.

2. Related Works

Related work frames shape completion as recovering holes in sensor-derived point clouds and reviews voxel, mesh, and point-cloud representations alongside deep learning, GAN, and RL approaches. Prior methods face resolution, deployment, missing-data, or GAN-optimization limitations that motivate RL control of GAN latent space.

  • Shape Completion and Deep Learning: Real-world 3D sensors produce point clouds with large holes caused by occlusions, limited field of view, and unreliable measurements.The passage lists multi-view stereo, structured-light sensors, RGB-D cameras, and lidars as affected modalities.
  • Shape Completion and Deep Learning: Voxel-based completion benefits from 3D CNNs but is limited in resolution because network complexity and computation increase drastically with resolution.Voxel grids also require covering large empty spaces when representing point measurements.
  • Shape Completion and Deep Learning: Point clouds are efficient raw acquisition outputs, yet most successful deep learning architectures cannot be deployed directly on them.Prior work consumed incomplete point clouds but used a pretrained decoder to produce voxelized complete shapes.
  • Shape Completion and Deep Learning: Autoencoder and GAN-based point-cloud representation methods provide some shape completion but degrade considerably as input missing points increase.The cited architecture was not designed specifically for shape completion.
  • GAN and RL: GANs learn real-data distributions through competition between a generator synthesizing fake examples and a discriminator distinguishing real from fake data.The related work notes that GANs are hard to train and susceptible to local optima, motivating architectural, loss-function, and training-practice improvements.
  • GAN and RL: The proposed pipeline uses deep reinforcement learning to control GAN latent space through agent actions and rewards.Earlier RL agents guided computer-vision tasks such as bounding-box locations and segmentation seed points using DQN, whereas this work controls GAN latent space.

3. Methods

RL-GAN-Net combines separately trained autoencoder, latent-space GAN, and reinforcement-learning components to complete noisy, incomplete point clouds. The RL agent selects GAN inputs using loss-based rewards, while a discriminator-based hybrid preserves semantic completion and local details in real time.

  • Pipeline: The pipeline separately trains an autoencoder, latent-space GAN, and RL agent, with the agent trained alongside pretrained autoencoder and GAN components.The autoencoder is trained first, its encoded data trains the latent-space GAN, and the RL agent uses both pretrained networks.
  • Forward pass: The encoder maps an incomplete point cloud to a noisy global feature vector, the RL agent selects a GAN seed, and the decoder reconstructs the completed shape.The generator produces a clean global feature vector, while a discriminator observes generated and autoencoder-processed feature vectors.
  • Latent-space GAN: Training the GAN on global feature vectors provides more stable results than training on raw point clouds, avoiding direct optimization over the high-dimensional representation.The global feature vector is an efficient intermediate representation produced by the autoencoder, with Chamfer distance selected over Earth Movers distance for efficiency.
  • Reinforcement learning: The RL environment combines the autoencoder, latent-space GAN, intermediate losses, and input–prediction discrepancy, with negated Chamfer, GFV, and discriminator losses forming the reward.The reward is r = wCH · rCH + wGF V · rGF V + wD · rD, and DDPG is used because the action space is continuous.
  • Hybrid completion: A discriminator-based hybrid selects between vanilla RL-GAN-Net and autoencoder outputs, robustly preserving semantic shapes and local details in real time.This hybrid is motivated by autoencoder degradation as missing data increases and limited semantic variation in vanilla completions.

4. Experiments

Experiments on 26,829 ShapeNetCore shapes show that RL-GAN-Net completes highly incomplete point clouds with semantically reasonable outputs in real time. The hybrid variant improves geometric completion by selecting between AE and vanilla RL-GAN-Net, while classification remains comparable to the vanilla approach.

  • Experimental setup: Experiments use four ShapeNetCore categories—cars, airplanes, chairs, and desks—comprising 26,829 centered and normalized shapes sampled with 2048 ground-truth points.The implementation uses PyTorch and trains all networks on a single Nvidia GTX Titan Xp graphics card.
  • Shape completion results: With 70% of points missing, input Chamfer distance rises to 16% of the shape diagonal, whereas AE, vanilla RL-GAN-Net, and hybrid RL-GAN-Net remain below 9%.For 20% missing data, AE can sometimes complete shapes better, motivating the hybrid selector.
  • Runtime: Around one millisecond is required for the RL-controlled forward pass to produce the GAN input, enabling real-time preprocessing instead of back-propagation-based optimization.A prior method reports 324 seconds for a batch of 50 shapes.
  • Hybrid and classification analysis: The hybrid approach improves Chamfer distance by selecting AE outputs when the discriminator finds them more reasonable, but its classification performance is comparable to vanilla RL-GAN-Net.The RL agent often primarily identifies the correct semantic category in latent space, and failure cases include wrong or semantically different categories.
  • Loss analysis: Chamfer and GFV losses increase as missing data grows, while discriminator loss remains almost constant because the pretrained generator follows its learned distribution.Using Chamfer loss alone was inefficient because a single scalar does not sufficiently guide semantic 3D completion.

5. Conclusion and Future Work

RL-GAN-Net uses an RL agent to control a GAN for robust, real-time point-cloud shape completion, replacing costly optimization with reward-based control. It completes shapes with up to 70% missing points and improves PointNet classification accuracy from 50% to 83% in that setting.

  • Conclusion: RL-GAN-Net replaces costly, complex GAN optimization with an RL agent that converts loss functions into rewards for generator control.The prior optimization process required a minimum of 324 seconds to process a batch of inputs.
  • Reward function analysis: The reward analysis finds that Chamfer loss alone cannot complete the shape, while GFV loss helps recover the correct shape and discriminator loss preserves semantic meaning.The analysis evaluates different combinations of losses in the reward function.
  • Conclusion: Approximately one millisecond is required to complete a point-cloud shape, enabling real-time processing.The framework produces completion results for data with up to 70% missing points.
  • Classification use case: 50% to 83% is the PointNet classification-accuracy improvement achieved by applying RL-GAN-Net to data with 70% missing points.The use case demonstrates that the real-time network can improve other point-cloud processing networks.

6. Implementation Details

RL-GAN-Net is implemented as an autoencoder, latent-space GAN, and actor-critic reinforcement-learning controller. The implementation specifies their architectures, training objectives, latent dimensions, and DDPG training procedure.

  • Autoencoder: The autoencoder maps 2048-point 3D inputs to a 128-dimensional global feature vector and reconstructs them using five encoder convolution layers.The encoder channels are 64, 128, 128, 256, and 128.
  • Latent-space GAN: The latent-space GAN uses the autoencoder encoder, a generator, and a discriminator trained with WGAN-GP loss using λgp = 10 for one million iterations.The generator output dimension matches the autoencoder bottleneck size of 128.
  • Reinforcement learning: The RL controller uses an actor-critic architecture for continuous control of the latent-space GAN, with a 128-dimensional feature-vector state and a single-dimension z-vector action.The actor and critic are fully connected networks, and the actor has four layers with 400, 400, 300, and 300 neurons.
  • Training details: DDPG training collects one experience at a time in episodic interaction, stores state-action-reward tuples in a replay buffer, and trains with batches of 100 memories.Policy evaluation was carried out after 5000 iterations.

7. Additional Results

Additional experiments show that RL-GAN-Net remains effective as missing data increases and is robust to jitter, while shape completion improves classification accuracy when more than 30% of the shape is missing.

  • Shape completion results: RL-GAN-Net’s completion performance remains prominent as the missing portion increases from 20% to 70%.Examples also cover missing portions of 30%, 40%, and 50%.
  • Robustness to jitter: The completed shape is semantically similar to the original and is almost unaffected by jitter.These robustness results use a different dataset provided by Dai et al.
  • Comparison with voxel-based completion: Compared with Dai et al.’s voxel-based approach, RL-GAN-Net is superior in Chamfer distance against the ground truth.The comparison includes voxel resolutions of 32^3 and 128^3 alongside point-cloud approaches.
  • Qualitative comparison: Voxel-based completion produces smoother results but is limited in resolution and local-detail recovery, whereas RL-GAN-Net preserves semantic symmetries and details.The qualitative comparison includes Voxel 32^3, Voxel 128^3, AE, and RL-GAN-Net.

RL-GAN-Net and GT

RL-GAN-Net is evaluated qualitatively for point-cloud shape completion under increasingly severe missing-data conditions, from 20% to 70% of the original points. The figures report completion results at each tested missing-point level.

  • RL-GAN-Net and GT: Qualitative shape-completion results are shown for inputs missing 20% of their original points.The evaluation is presented in Figure 10.
  • RL-GAN-Net and GT: Qualitative shape-completion results are shown for inputs missing 30% of their original points.The evaluation is presented in Figure 11.
  • RL-GAN-Net and GT: Qualitative shape-completion results are shown for inputs missing 40% of their original points.The evaluation is presented in Figure 12.
  • RL-GAN-Net and GT: Qualitative shape-completion results are shown for inputs missing 50% of their original points.The evaluation is presented in Figure 13.
  • RL-GAN-Net and GT: Qualitative shape-completion results are shown for inputs missing 70% of their original points.The evaluation is presented in Figure 14.

Pin AE RL-GAN-Net Ground Truth · RL-GAN-Net and GT

The paper evaluates RL-GAN-Net against ground-truth and alternative classification pipelines using robustness tests, performance comparisons, and missing-data classification experiments. The evaluations include raw scans, Gaussian-noise perturbations, resolution comparisons, and varying proportions of missing points.

  • RL-GAN-Net and GT: RL-GAN-Net completes raw scanned point clouds in robustness-test examples using data provided by Dai et al. [7].The examples use the provided raw scan data.
  • RL-GAN-Net and GT: Adding zero-mean Gaussian noise with standard deviation 0.01, clipped at 0.05, produces additional robustness-test completion results.The perturbation is applied to point-cloud data provided by [7].
  • RL-GAN-Net and GT: RL-GAN-Net is compared with Dai et al. [7] at 323 and 1283 resolutions after converting their distance-function output to the point-cloud domain.Dai et al. additionally use mask information, whereas RL-GAN-Net operates directly on scanned points.
  • RL-GAN-Net and GT: The missing-data classification architecture includes a vanilla PointNet classifier.This is one of the network-architecture variations shown for point-cloud classification with missing data.
  • RL-GAN-Net and GT: Another missing-data classification pipeline combines RL-GAN-Net with a PointNet classifier.This is presented as an alternative network-architecture variation.
  • RL-GAN-Net and GT: Figure 18 presents variations of network architecture for point-cloud classification with missing data.The displayed variations include vanilla PointNet and RL-GAN-Net combined with PointNet.
  • RL-GAN-Net and GT: Table 6 compares classification accuracy after RL-GAN-Net processing with vanilla and AE processing across percentages of missing data points.The table evaluates classification accuracy for point-cloud inputs with varying missing-data percentages.
Loading 1904.12304v1…