PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully-Convolutional Point Networks for Large-Scale Point Clouds

Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari

arXiv:1808.06840v1cs.CV

TL;DR

Large-scale 3D learning must efficiently process point clouds while capturing spatial structure across variable scene sizes and tasks. FCPN addresses this by transforming unordered point inputs into organized internal representations for fully convolutional processing. It supports single-pass large-space processing and performs across semantic segmentation, part segmentation, and 3D captioning benchmarks.

  • Problem

    Existing 3D architectures primarily maintain either unordered point sets or organized representations, limiting the combined benefits of memory-efficient inputs and 3D convolutions for large-scale processing.

  • Method

    FCPN accepts unorganized point clouds, transforms them internally into ordered representations, and applies hierarchical multi-scale 3D convolutions with pooling and feature merging.

  • Results

    FCPN demonstrates single-pass scalability and general-purpose feature representations across semantic voxel labeling, semantic part segmentation, and 3D captioning.

  • Takeaways & Limitations

    The learned spatially ordered descriptor supports large-scale scene processing and higher-level tasks such as captioning from unordered point-cloud inputs.

  • Takeaways & Limitations

    Uniform sampling is required to cover occupied and unoccupied space comprehensively, whereas furthest point sampling is suited to occupied structure.

Abstract

from arXiv · show

This work proposes a general-purpose, fully-convolutional network architecture for efficiently processing large-scale 3D data. One striking characteristic of our approach is its ability to process unorganized 3D representations such as point clouds as input, then transforming them internally to ordered structures to be processed via 3D convolutions. In contrast to conventional approaches that maintain either unorganized or organized representations, from input to output, our approach has the advantage of operating on memory efficient input data representations while at the same time exploiting the natural structure of convolutional operations to avoid the redundant computing and storing of spatial information in the network. The network eliminates the need to pre- or post process the raw sensor data. This, together with the fully-convolutional nature of the network, makes it an end-to-end method able to process point clouds of huge spaces or even entire rooms with up to 200k points at once. Another advantage is that our network can produce either an ordered output or map predictions directly onto the input cloud, thus making it suitable as a general-purpose point cloud descriptor applicable to many 3D tasks. We demonstrate our network's ability to effectively learn both low-level features as well as complex compositional relationships by evaluating it on benchmark datasets for semantic voxel segmentation, semantic part segmentation and 3D scene captioning.

1 Introduction

FCPN addresses the challenge of processing large-scale 3D data by combining memory-efficient point-cloud input with convolution-friendly internal representations. It supports single-pass processing across scales and multiple scene-understanding tasks.

  • Motivation: 3D semantic information supports applications in computer vision, scene understanding, augmented reality, robotics, and autonomous driving.Semantic scene information can support virtual-content placement, descriptions for visually impaired people, and intelligent robotic interaction.
  • Motivation: Large-scale 3D learning must handle noise, holes, occlusion, partial scans, viewpoint changes, and 3D transformations.These issues motivate architectures that learn robust representations from real-world data.
  • Approach: FCPN accepts unorganized point clouds, transforms them internally into organized representations, and processes them with 3D convolutions.This hybrid design combines memory-efficient input with convolutional processing and supports multiple spatial scales.
  • Scalability: FCPN can process large point clouds in a single pass and can be trained on 2.4×2.4×2.4-meter regions before larger-volume inference.Figure 2 illustrates predictions for a depth image, a partial reconstruction, and an entire hotel-suite reconstruction from the same network.
  • Evaluation: The network is evaluated on semantic voxel labeling, semantic part segmentation, and 3D captioning, including a dataset with human-annotated captions.The captioning task targets meaningful descriptions of partial indoor reconstructions.

2 Related Work

Prior 3D learning methods primarily use either ordered volumetric representations or unordered point sets. FCPN occupies a hybrid position, combining direct point-set processing with multi-scale 3D convolutions.

  • Volumetric methods: Volumetric methods use dense ordered grids that enable constant-time neighbor lookup but explicitly represent empty space.Larger spaces often require lower sampling density because memory limits constrain voxel resolution.
  • Volumetric methods: OctNet and kd-tree methods reduce memory through hierarchical spatial partitioning but still impose minimum spatial discretization.FCPN instead aims to mitigate memory limitations without discretizing the input.
  • Point-based methods: PointNet processes unordered points directly and avoids imposing an input sampling resolution, while PointNet++ adds hierarchical point-based processing.PointNet++ nevertheless redundantly computes and stores context for spatially overlapping points and cannot exploit 3D-convolution benefits.
  • FCPN: FCPN is presented as a hybrid method that processes point sets directly while using 3D convolutions for multi-scale feature encoding.This positions it between volumetric and point-based approaches.

3 Fully-Convolutional Point Network

FCPN combines point-based input handling with organized internal feature processing. Its architecture uses hierarchical abstraction, multi-scale feature learning, pooling, and feature merging.

  • Architecture: FCPN uses unorganized input, applies PointNet as a low-level descriptor, and transforms local point features into an ordered representation.Uniform sampling captures precise local geometry before subsequent convolutional processing.
  • Multi-scale encoding: FCPN encodes indoor spaces at three spatial scales, with skip connections linking feature resolutions.Figure 4 depicts these scales using spherical 15cm features and cubic features at the other scales.
  • Architecture: The network consists of abstraction layers, multi-scale feature learners, weighted-average pooling, and a hierarchical merging stage.These modules form the main processing pipeline of FCPN.
  • Hierarchical abstraction: The abstraction hierarchy captures basic geometry, complex structure, and structure in contextual relationships.The levels progress from edges and corners toward higher-level contextual patterns.

Abstraction Layers

FCPN uses hierarchical, non-overlapping spatial abstraction followed by multi-scale convolutions and skip connections. These operations reduce memory while preserving contextual information across resolutions.

  • Abstraction Layers: The first abstraction level uses radius grouping, 1×1×1 convolutions, and max pooling to create a 3D feature volume of local geometry.This volume feeds the subsequent abstraction layers.
  • Abstraction Layers: Higher abstraction levels use stride-2 convolutions so eight preceding octants become one subsequent cell.The non-overlapping partitioning significantly reduces memory requirements.
  • Multi-scale features: At each abstraction level, 1×1×1 and 3×3×3 convolutions and skip connections provide features at the native scale and three times that scale.This broadens the encoded spatial context and mitigates strictly non-overlapping partitioning.

Feature Learners

The weighted average pooling layer adds long-range contextual information by weighting cells according to distance from a 1m sphere around each cell. It emphasizes nearby semantic identities over exact spatial configurations, improving discrimination and scalability.

  • Feature Learners: Weighted average pooling incorporates long-range context by distance-weighting responses from all cells relative to a 1m sphere around each cell.Cells closest to the sphere receive the greatest weights.
  • Feature Learners: The layer emphasizes nearby objects' semantic identities rather than their exact spatial configurations when distinguishing boundary cases.For example, a nearby sink can help distinguish a toilet from a chair more than the sink's precise relative position.
  • Feature Learners: Because it is parameterless and averages spatial context, the layer is cost-effective and avoids learning poorly generalizable larger-scale configurations.It retains semantic identities while removing exact local arrangements.
  • Feature Learners: The weighted pooling layer provides the flexibility needed to scale the network to larger spaces during inference.

Merging

The merging stage progressively combines features across abstraction levels through concatenation, 3D deconvolution upsampling, and 1 × 1 × 1 convolutions. Output variants support organized predictions, pointwise semantic mapping, or whole-input summarization.

  • Merging: Skip connections from each abstraction level are concatenated and upsampled to 2× spatial resolution using 3D deconvolutions.This progressively merges features across abstraction levels.
  • Merging: 1 × 1 × 1 convolutions add expressive power between deconvolution layers.
  • Merging: An additional deconvolution layer produces organized outputs at the desired point density.
  • Merging: Latent nearest-neighbor interpolation maps semantic predictions back onto each input point for end-to-end processing.
  • Merging: Fully-connected layers summarize the entire input for tasks such as scene captioning.

3.3 Uniform vs. Furthest Point Sampling

Sampling choice depends on whether the network describes occupied structure or entire occupied and unoccupied spaces. The network labels the supplied volume by restoring sampling density and padding feature volumes symmetrically.

  • 3.3 Uniform vs. Furthest Point Sampling: Furthest point sampling describes occupied structure effectively without assuming a spatial data distribution.Uniform sampling is instead required to cover every part of occupied and unoccupied spaces.
  • 3.3 Uniform vs. Furthest Point Sampling: Uniform sampling is the only strategy that ensures every part of an entire space, including unoccupied regions, is considered.
  • 3.3 Uniform vs. Furthest Point Sampling: FCPN labels the full input volume by upsampling feature maps to the original sampling density.
  • 3.3 Uniform vs. Furthest Point Sampling: Symmetric padding of feature volumes supports predictions across the complete spatial volume.The nearest edge cell is assumed likely to share occupancy characteristics with directly outside regions.

3.5 Scalability

FCPN can be trained on smaller samples and scaled during inference to process much larger spaces in one shot. It successfully handles an 80m2 space containing 200k points, with still larger extents possible at sparser density.

  • 3.5 Scalability: FCPN can be trained on smaller samples and scaled during inference to process spaces multiple times larger in a single shot.
  • 3.5 Scalability: 80m2 space consisting of 200k points is processed at once.
  • 3.5 Scalability: Even larger spatial extents can be processed at sparser point density, extending the network's versatility to use cases such as autonomous driving.

4 3D Captioning

The paper introduces 3D Captioning, a scene-understanding task that generates meaningful textual descriptions of spaces for assistive navigation and interaction.

  • 4 3D Captioning: 3D Captioning generates meaningful textual descriptions of spaces.The task is designed to answer questions about a space’s function, movement, and interaction.
  • 4 3D Captioning: The dataset contains human-annotated captions based on ScanNet and covers diverse spaces with validated annotations.Roughly half of ScanNet was annotated using a dedicated Scene Caption Annotation Tool, and outliers were removed during validation.
  • 4 3D Captioning: Captions address what functional value a space has, how a person can move, and how they can interact with it.Every 100th frame may receive zero or more applicable captions from the selected sentence set.
  • 4 3D Captioning: The authors release the captioning dataset together with the source code.

5 Evaluation

FCPN is evaluated on large-scale semantic voxel labeling, point-cloud part segmentation, 3D captioning, and memory scaling, showing competitive or strong task performance alongside large-input processing efficiency.

  • 5.1 Semantic Voxel Labeling: The semantic voxel-labeling network uses 15cm, 30cm, and 60cm spatial scales, pooling features at 60cm before producing 5cm3 output density.
  • 5.1 Semantic Voxel Labeling: 82.6% weighted and 54.2% unweighted accuracy are achieved on ScanNet semantic voxel labeling across 20 classes.These results exceed ScanNet’s 73% weighted and 50.8% unweighted accuracy but trail PointNet++ MSG+DP’s 84.5% and 60.2%.
  • 5.1 Semantic Voxel Labeling: FCPN processes ScanNet scenes from small bathrooms to whole apartments in a single shot.PointNet++ instead combines sliding-volume predictions with majority voting.
  • 5.2 Part Segmentation: FCPN outperforms the state of the art in 12 of 16 ShapeNet object categories for semantic part segmentation.The benchmark covers 50 part categories across 16 object types.
  • 5.3 3D Captioning: The captioning baseline replaces semantic-labeling upsampling layers with three fully connected layers, freezes the FCPN weights, and trains only those layers.
  • 5.4 Memory and Runtime: Clouds 5x as large with 10x as many points require only 40% more memory on a Titan Xp GPU.The evaluation varies input cloud size in both spatial extent and point count.

6 Conclusions

The paper presents FCPN as a fully-convolutional network for unordered point sets that scales to larger spaces and serves as a general-purpose descriptor across multiple 3D tasks.

  • 6 Conclusions: FCPN is presented as the first fully-convolutional neural network operating on unordered point sets.
  • 6 Conclusions: Its fully-convolutional design processes substantially larger spaces than the current state of the art in a single shot.
  • 6 Conclusions: The learned descriptor is evaluated on semantic scene segmentation and part-based object segmentation at different scales.
  • 6 Conclusions: The spatially ordered descriptor opens the door to higher-level scene understanding tasks such as captioning.
Loading 1808.06840v1…