Source-linked AI summary
3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-scale 3D Point Clouds
Fangyu Liu, Shuaipeng Li, Liqiang Zhang, Chenghu Zhou, Rongtian Ye, Yuebin Wang, Jiwen Lu
TL;DR
Large-scale point-cloud parsing is challenging because point distributions are unorganized and uneven, while existing approaches can miss consistency and complementary information among features. The paper combines a 3D CNN, DQN-guided eye window, and Residual RNN for automatic parsing; it reports high-quality classification performance. The authors identify insufficient 3D CNN training as a limitation, especially for learning discriminative features from small point clouds.
Problem
Large-scale point clouds are difficult to parse because their points are unorganized and unevenly dense, and existing feature approaches may not capture feature consistency and complementarity.
Source: §1 · p. 1Method
3DCNN-DQN-RNN uses a 3D CNN to learn point features, a DQN to guide an eye window over target objects, and a Residual RNN to classify points within the window.
Source: §1 · p. 2Results
The authors report high-quality classification performance from the Residual RNN’s merged multiscale point features.
Source: §1 · p. 2Takeaways & Limitations
The framework provides an automatic parsing process that integrates object localization, detection, retrieval, and classification.
Source: §1 · p. 2The 3D CNN is not trained well enough and has difficulty learning discriminative features from small point clouds.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Semantic parsing of large-scale 3D point clouds is an important research topic in computer vision and remote sensing fields. Most existing approaches utilize hand-crafted features for each modality independently and combine them in a heuristic manner. They often fail to consider the consistency and complementary information among features adequately, which makes them difficult to capture high-level semantic structures. The features learned by most of the current deep learning methods can obtain high-quality image classification results. However, these methods are hard to be applied to recognize 3D point clouds due to unorganized distribution and various point density of data. In this paper, we propose a 3DCNN-DQN-RNN method which fuses the 3D convolutional neural network (CNN), Deep Q-Network (DQN) and Residual recurrent neural network (RNN) for an efficient semantic parsing of large-scale 3D point clouds. In our method, an eye window under control of the 3D CNN and DQN can localize and segment the points of the object class efficiently. The 3D CNN and Residual RNN further extract robust and discriminative features of the points in the eye window, and thus greatly enhance the parsing accuracy of large-scale point clouds. Our method provides an automatic process that maps the raw data to the classification results. It also integrates object localization, segmentation and classification into one framework. Experimental results demonstrate that the proposed method outperforms the state-of-the-art point cloud classification methods.
1. Introduction
Large-scale point-cloud parsing is challenging because points are unorganized, unevenly dense, and may contain noise, outliers, or undersampling. The proposed 3DCNN-DQN-RNN combines learned features, DQN-guided localization, and recurrent classification in an automatic parsing framework.
1. Introduction
- Unorganized, uneven-density point clouds corrupted by noise, outliers, or undersampling make large-scale semantic parsing challenging.Source: §1 · p. 1
- The method combines a 3D CNN, DQN-controlled eye window, and Residual RNN to localize class objects and classify their points.The DQN adjusts the eye window using reward information from the 3D CNN; the Residual RNN classifies point features, coordinates, and colors within the window.Source: §1 · pp. 1–2
- The framework integrates object localization, detection, retrieval, and classification, while reducing intensive parameter-tuning cost through learned parameters.Source: §1 · p. 2
- The Residual RNN merges multiscale 3D CNN features with each point’s coordinates and colors into robust, discriminative representations, with high-quality classification performance reported.Source: §1 · p. 2
- The DQN-controlled eye window is presented as an accurate, automatic, and less expensive way to localize and segment class objects.Source: §1 · p. 2
2. Related Work
Prior point-cloud parsing approaches used hand-crafted descriptors and classifiers, while deep-learning methods learned features from 3D data but often did not directly address large-scale point-cloud parsing. Image-based segmentation methods and detection pipelines also faced challenges transferring to large, incomplete 3D point clouds.
2. Related Work
- Earlier point-cloud classifiers used hand-crafted descriptors, and the authors argue that these approaches underuse feature consistency and complementarity needed to capture high-level semantics.Source: §2 · p. 2
- Deep-learning methods learned features from 3D data, but the reviewed object-recognition approaches were difficult to apply directly to large-scale point-cloud parsing.Detection-based approaches also required hand-crafted choices such as sliding windows or box searches; PointNet is noted as addressing scene semantic parsing.Source: §2 · p. 2
- Image-oriented FCN and related segmentation methods were discussed alongside high-dimensional segmentation work, but the cited methods were difficult to apply to large-scale point clouds with significant missing data.Source: §2 · pp. 2–3
3. Proposed Approach
The approach uses a 3D CNN and DQN to search for class objects with an adjustable eye window, then applies a Residual RNN to classify the points inside it. The framework combines localization, segmentation, and classification in a single pipeline.
3.1.1 Learning Multi-Scale Features Using the 3D CNN
- Multi-scale 3D CNN features encode points in the eye window, while the reward vector guides the DQN toward a window with high class-object confidence.The reward vector is used to calculate the probability that the class object lies within the window.Source: §3.1.1 · p. 3
3.1. Network 1: Recognition and Localization through 3DCNN-DQN
- The 3D CNN evaluates eye-window contents and the DQN uses that feedback to adjust window position and size until it localizes a class object.The window search follows an evaluation-feedback and action cycle.Source: §3.1 · p. 3, §3.1.2 · p. 4
3.1.2 Object Class Detection by the DQN
- Winner replay selects a previously successful action half the time and reduces DQN searching time by 73% on average.The other half of action selections use normal N-step simulation.Source: §3.1.2 · p. 5
3.2.1 Residual RNN Structure
- Each RNN input vector combines a point's coordinates and color with its 3D CNN features, supplementing information lost through point-cloud rasterization.The input vector is [x, y, z, r, g, b, f1, f2, f3].Source: §3.1.2 · p. 6
3.2. Network 2: Residual RNN for Meticulous Parsing
- The Residual RNN receives point sequences in their spatial arrangement and combines multi-scale features to classify points within the localized window.The method uses an LSTM cell for long-term memory and residual blocks to address degradation in deep networks.Source: §3.2 · p. 6, §3.2.1 · p. 6
3.2.2 LSTM Cell
- The LSTM cell is used to learn long-term dependencies among features in the long point sequence.The points in each eye window remain numerous, making the input sequence long.Source: §3.2.2 · p. 6
3.2.3 Residual Block
- Residual blocks connect overlapping cells to address degradation, which the paper describes as training error increasing when a network becomes too deep.The approach is motivated by Deep Residual Network [7].Source: §3.2.3 · p. 6
3.2.4 Deep RNN - A Multi-Layer Classifier
- The multi-layer neural network fuses feature vectors and learns discriminative representations of point locations, spatial relationships, and color.The authors argue that shallow SVM or Random Forest classifiers may restrict self-adjustment of these representations.Source: §3.2.4 · p. 6
4. Experiments
The experiments compare the framework with a 3D FCN and examine the roles of the RNN, DQN, and 3D CNN. They report qualitative comparisons and describe dataset setup, localization time, and limitations.
4. Experiments
- The Stanford dataset contains 271 rooms across six areas and 12 classes; 70% of rooms in each area are used for training and the remainder for testing.The SUNCG point clouds cover 84 annotated classes; the authors report classification results in supplementary materials.Source: §4 · pp. 6–7
4.1. Comparison of Our Method with the FCN
- The 3D FCN performs poorly compared with the proposed method, while deeper FCN models would require more computation and time.FCN-1 fails to recognize borders; FCN-6 and FCN-12 have very low accuracy, and FCN-12's loss does not converge.Source: Table 1 · p. 7, §4.1 · p. 7
4.2. RNN Helps to Raise Precision
- Network 2 raises precision for two classes and improves accuracy for classes parsed later by returning wrongly included points to the scene.When nearby objects of different classes enter the eye window together, Network 1 can include irrelevant points.Source: §4.2 · p. 7, Figure 6 · p. 8
4.3. Influences of the DQN on Point Cloud Parsing
- 5000 DQN iterations per class take an average of 13 hours to localize all 12 classes in the first dataset, and longer runs usually raise accuracy.The eye window spends most of its time moving around objects, though some areas are not worth exploring.Source: §4.3 · p. 8, Figure 7 · p. 8
4.4. 3D CNN is Not Well Trained
- The authors identify insufficient 3D CNN training as the main limitation, noting confusion between similar objects and imprecise eye-window boundaries.A small bookcase may resemble a table, and a window boundary may include surrounding wall points; larger-dataset training is proposed to improve classification performance.Source: §4.4 · p. 8
5. Conclusion
The conclusion presents 3DCNN-DQN-RNN as a framework for parsing large-scale point clouds by combining eye-window localization with point classification. It proposes coordinating multiple eye windows with A3C in future work.
5. Conclusion
- The framework combines multiscale 3D CNN features with DQN-controlled eye-window localization and Residual RNN classification of points inside each window.The framework also supports object localization, detection, and retrieval.Source: §5 · p. 8
- The Residual RNN receives each window point's 3D CNN feature, coordinates, and color to produce the parsing result.Source: §5 · p. 8
- Future work proposes using asynchronous advantage actor critic (A3C) to coordinate multiple eye windows for more efficient object derivation.Source: §5 · p. 8
3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-scale 3D Point Clouds: Supplementary Materials
The supplementary figure juxtaposes 3D models of tables and neighboring chairs with their corresponding point-cloud data.
- Panel (a) shows 3D models of tables and neighboring chairs.Source: Figure 1 · p. 11
- Panel (b) shows the point-cloud data corresponding to the 3D models.Source: Figure 1 · p. 11
- The figure presents the models and their point clouds as paired representations.Source: Figure 1 · p. 11
A. Overview
The supplement provides a detection demo and SUNCG classification results, with a comparison to Armeni et al. In the demo, the eye window rapidly finds approximate table locations but spends more time tracing boundaries and can confuse tables with bookcases.
A. Overview
- The supplement adds a demo video and SUNCG classification results, using Stanford data and 20% of SUNCG for training and the remainder for testing.It also compares results with Armeni et al. on the same training and testing data.Source: A. Overview · p. 11
B. A Demo Video of the Detection Process
- The demo shows the eye window enveloping tables; the DQN rapidly localizes approximate positions and spends more time detecting their boundaries.The experiment classifies a point only after the eye window detects it more than five times.Source: B. A Demo Video of the Detection Process · p. 11
- The window also spends time around bookcases, which resemble tables in height and structure; the authors plan to add context information to improve detection.Although video boundaries appear rough, the reported final classification performance remains high.Source: B. A Demo Video of the Detection Process · p. 11
C.1. Training data and testing data
The method outperforms Armeni et al. on the reported SUNCG classification comparison. Training combines Stanford data with 20% of SUNCG, while the remaining SUNCG data is used for testing.
C.1. Training data and testing data
- The 3D CNN is trained on Stanford data, adapted with 20% of SUNCG data, and tested on the other 80%; training takes 48 hours and converges to approximately 10% error.The procedure transfers trained network parameters to the testing data.Source: §C.1 · p. 11
C.2. Comparison between our method and the method of Armeni et al.
- Table 1 reports classification results in which our method outperforms Armeni et al.; our method is N1+N2, while Armeni et al.’s method is S1.Source: §C.1 · p. 11, Table 1 · p. 11
C.3. Analysis of the experimental result
Performance on SUNCG is lower than on Stanford, with the authors attributing the gap to more varied, complex scenes and limited hardware. They identify object variability and aggressive down-sampling as constraints on feature learning and generalization.
C.3. Analysis of the experimental result
- Classification performance on SUNCG is lower than on Stanford.The authors give object complexity and hardware limits as two reasons.Source: §C.3 · p. 12
- SUNCG's varied chair styles and more complicated object relationships leave too few similar training examples for accurate 3D CNN scores and DQN actions.Object patterns are also harder to recognize than in Stanford scenes.Source: §C.3 · p. 12
- Hardware limits constrain the 3D CNN to six layers and require fierce down-sampling, reducing feature discrimination and generalization.The authors expect deeper CNNs with less down-sampling to improve accuracy on complex SUNCG scenes.Source: §C.3 · p. 12