PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion

Shi Qiu, Saeed Anwar, Nick Barnes

arXiv:2103.07074v2cs.CV

TL;DR

Real-world point clouds are challenging for fine-grained semantic segmentation because they are scattered, irregular, unordered, and unevenly distributed. The paper proposes bilateral local-context augmentation and point-level adaptive fusion of multi-resolution features, achieving competitive performance on three real-scene benchmarks.

  • Problem

    Real point cloud scenes require fine-grained semantic labeling, but their scattered, irregular, unordered, and uneven distributions make large-scale semantic segmentation challenging.

  • Method

    The network bilaterally augments local point context using geometric and semantic features, then adaptively fuses multi-resolution features at point level.

  • Results

    The approach achieves competitive performance against state-of-the-art methods on S3DIS, Semantic3D, and SemanticKITTI, including 83.1% average class accuracy and 72.2% mIoU on S3DIS.

  • Takeaways & Limitations

    The bilateral augmentation and adaptive fusion structures are adaptable and may be studied with different frameworks and extended to other 3D tasks.

  • Takeaways & Limitations

    Existing real-world point-cloud approaches can require time-consuming preprocessing and postprocessing, while intermediate representations may lose surrounding context.

Abstract

from arXiv · show

Given the prominence of current 3D sensors, a fine-grained analysis on the basic point cloud data is worthy of further investigation. Particularly, real point cloud scenes can intuitively capture complex surroundings in the real world, but due to 3D data's raw nature, it is very challenging for machine perception. In this work, we concentrate on the essential visual task, semantic segmentation, for large-scale point cloud data collected in reality. On the one hand, to reduce the ambiguity in nearby points, we augment their local context by fully utilizing both geometric and semantic features in a bilateral structure. On the other hand, we comprehensively interpret the distinctness of the points from multiple resolutions and represent the feature map following an adaptive fusion method at point-level for accurate semantic segmentation. Further, we provide specific ablation studies and intuitive visualizations to validate our key modules. By comparing with state-of-the-art networks on three different benchmarks, we demonstrate the effectiveness of our network.

1. Introduction

Real point cloud semantic segmentation is difficult because large, unstructured scenes contain ambiguous neighboring points, redundant information, and weakened global representations. The paper addresses these issues with bilateral local-context augmentation and adaptive multi-resolution feature fusion in a point-based network.

  • Large real-world point clouds are scattered, irregular, unordered, and unevenly distributed, making semantic segmentation challenging at million- or billion-point scale.
  • Existing approaches may require time-consuming preprocessing and postprocessing while losing surrounding context through intermediate representations.
  • Nearby points near class boundaries create ambiguity because fixed-neighbor construction can include outliers and overlapping neighborhoods.
  • The method represents geometric and semantic clues through a bilateral structure to reduce redundant feature processing and construct compact point-cloud representations.
  • Adaptive fusion integrates information from different resolutions to recover comprehensive point-level representations for semantic prediction.
  • Contributions: The network introduces bilateral local-context augmentation, adaptive multi-resolution feature fusion, and evaluation on three large-scale real-scene benchmarks.

2. Related Work

Prior point-cloud methods process unstructured data directly or through image-like representations, but often underuse geometric information and struggle to preserve local details across resolutions. This paper combines bilateral geometric-semantic context augmentation with adaptive fusion for efficient point-wise representations.

  • Point-Based Approaches: Point-based approaches directly process unstructured 3D data using point-wise MLPs and neighborhood operations such as ball queries or k-nearest neighbors.
  • Point Clouds Feature Representations: Point-cloud feature methods typically define neighborhoods using spatial distances or embedding similarities and aggregate local features with CNN-based modules.
  • Point Clouds Feature Representations: Existing methods may miss local details by omitting geometric restrictions, adding computational cost through descriptors, or repeatedly combining coordinates across scales.
  • Semantic Segmentation Networks: 2D segmentation architectures commonly use fully convolutional or encoder-decoder structures, with some methods combining color and depth through bidirectional gating.
  • Semantic Segmentation Networks: For large 3D scenes, encoder-decoder sampling can weaken local-context interpretation, while receptive-field expansion is costly and random sampling can sacrifice accuracy and stability.
  • Semantic Segmentation Networks: The proposed network instead augments multi-resolution point clouds bilaterally and adaptively fuses point-wise features to represent comprehensive context efficiently.

3. Methodology

The method combines bilateral geometric-semantic context augmentation with mixed local aggregation and adaptive point-level fusion across resolutions for real-scene point-cloud segmentation.

  • Point-cloud features comprise explicit 3D coordinates for geometric context and learned features encoding semantic context.
  • Bilateral Context Module: The Bilateral Context Module learns offsets from bilateral geometric and semantic inputs, shifts neighbors, and aggregates their augmented local context.It processes point clouds through cascaded blocks at decreasing resolutions.
  • Bilateral Context Module: The method concatenates projected augmented geometric and semantic contexts before forming the point representation.
  • Mixed Local Aggregation: Mixed local aggregation combines maximum neighbor features with a learned high-dimensional barycenter to represent local details more precisely.The barycenter uses learnable weights over the k neighbors.
  • Adaptive Fusion Module: The Adaptive Fusion Module progressively upsamples multi-resolution feature maps to full size and regresses point-level fusion parameters from summarized point-level information.The pipeline takes M multi-resolution feature maps and outputs a full-sized representation for semantic segmentation.

4. Implementation Details

Implementation uses an MLP feature extractor, cascaded multi-resolution bilateral blocks, efficient neighborhood processing, nearest-neighbor upsampling, adaptive fusion, and augmentation-aware losses.

  • The network contains a Feature Extractor, Bilateral Context Module, and Adaptive Fusion Module for real point-cloud semantic segmentation.
  • Feature Extractor: A single-layer MLP provides preliminary semantic knowledge from supplied point-cloud information before processing with 3D coordinates.
  • Bilateral Context Module: The Bilateral Context Module uses CUDA-based Farthest Point Sampling and cascaded blocks to process progressively lower-resolution point clouds.An example resolution sequence is N→N/4→N/16→N/64→N/256, while output dimensions increase from 32 to 1024.
  • Bilateral Context Module: Efficient k-nearest-neighbor search accelerates bilateral augmentation, with k=12 used in all experiments.Mixed local aggregation applies max pooling and reweights neighbors using an MLP and softmax.
  • Adaptive Fusion Module: The Adaptive Fusion Module integrates channels, upsamples with nearest-neighbor interpolation, and adaptively weights full-sized feature maps point by point.Fusion parameters are normalized with softmax before the upsampled maps are combined.
  • Loss Function: Training combines cross-entropy classification loss with point-level augmentation losses weighted across Bilateral Context Blocks.The block-specific weights are denoted by ωm.

5. Experiments

Experiments evaluate the network on three real-scene benchmarks, use ablations and visualizations to inspect its modules, and analyze accuracy and complexity. The method reports strong results across indoor, natural, and traffic scenes, while remaining comparable in complexity to related models.

  • Experimental Settings: Experiments cover S3DIS indoor spaces, Semantic3D natural rural and urban scenes, and SemanticKITTI outdoor traffic scenarios.S3DIS uses six-fold evaluation; Semantic3D includes semantic-8 and reduced-8 test sets; SemanticKITTI evaluates single scans.
  • S3DIS Results: 83.1% mAcc and 72.2% mIoU are reported on S3DIS, significantly outperforming competitors on these metrics.The authors note that overall accuracy can be misleading under class imbalance, whereas mAcc and mIoU better indicate segmentation ability.
  • Analysis and Complexity: The Adaptive Fusion Module visualizes upsampled multi-resolution feature maps and adaptive weights, and the model’s complexity and capacity are comparable to cited methods.The complexity analysis concerns inference on SemanticKITTI; another cited method requires multiple evaluations to reduce random-sampling effects.
  • Semantic3D Results: 94.9% OA and 75.4% mIoU are achieved on Semantic3D across two billion testing points, with IoUs above 90% for three stated categories.The method surpasses other methods in three of eight classes and performs well on humanmade and natural terrains, cars.
  • SemanticKITTI Results: SemanticKITTI results exceed other approaches in four of 19 classes and improve mIoU by 5.6% over the cited latest point- and grid-based methods.The network performs well on small objects in dense scans but remains slightly behind a sparse-tensor state-of-the-art method.
  • Ablation Studies: Ablations select bilateral offsets with geometric augmentation loss and mixed local aggregation, while point-level adaptive fusion outperforms alternative fusion strategies.Regular summation and multiplication are described as undesirable; concatenation helps, and adaptive point-level fusion performs better.

6. Conclusions

The paper presents a network for semantic segmentation of real point cloud scenes that bilaterally augments local context and adaptively fuses multi-resolution features at point level. It reports outstanding performance on S3DIS, Semantic3D, and SemanticKITTI, supported by ablations and visualizations.

  • The network targets semantic segmentation of real point cloud scenes using bilateral local-context augmentation and adaptive multi-resolution feature fusion.
  • It achieves outstanding performance on the S3DIS, Semantic3D, and SemanticKITTI benchmarks.
  • Ablation studies and intuitive visualizations analyze the properties and effects of the proposed modules.
  • The authors identify real-time efficiency, transfer to different frameworks, and extension to object detection and instance segmentation as future directions.

Supplementary Material

The supplementary material adds network details, experimental results, and visualizations for the semantic segmentation system. It specifically complements the main paper's architecture and Bilateral Context Block descriptions.

  • The supplementary material provides additional network details, experimental results, and semantic segmentation visualizations.
  • It expands the main paper's presentation of the general network architecture and Bilateral Context Block.

B.1. Key Modules

The key modules extract semantic context, process point clouds across resolutions with Bilateral Context Blocks, and adaptively fuse upsampled features point by point. The network combines these representations for semantic prediction and trains with cross-entropy plus augmentation losses.

  • Feature Extractor: A single-layer MLP with eight 1×1 kernels extracts semantic context F from the input information I for N points.
  • Feature Extractor: The extracted feature F is forwarded with 3D coordinates P to the Bilateral Context Module.
  • Bilateral Context Module: Five Bilateral Context Blocks with Farthest Point Sampling produce multi-resolution feature maps for the Bilateral Context Module.
  • Bilateral Context Module: The module's downsampling ratios and feature dimensions are adopted from prior work because the study focuses on structure design rather than hyperparameter fine-tuning.
  • Adaptive Fusion Module: Extracted maps {S1, S2, S3, S4, S5} are gradually upsampled to full-sized maps, each with dimension R^N×32.
  • Adaptive Fusion Module: Fully connected layers summarize point-level information, softmax normalizes fusion weights, and the resulting parameters adaptively combine the five maps.
  • Semantic Prediction: Three fully connected layers and dropout predict confidence scores for Q candidate semantic classes from the fused feature map.
  • Training Objective: The overall loss combines cross-entropy with augmentation losses for each Bilateral Context Block, applying weights 0.1, 0.1, 0.3, 0.5, and 0.5.

C. Experiments

The supplementary experiments report per-area S3DIS results and online Semantic3D results on the reduced-8 test set. The authors state that performance on large-scale Semantic3D data is relatively better than on the main paper's semantic-8 setting.

  • S3DIS: Table 7 reports each S3DIS area using overall accuracy, average class accuracy, and IoUs for 13 semantic classes, training on the other five areas.
  • Semantic3D: Table 8 reports online semantic segmentation results on Semantic3D's reduced-8 test set, containing four scenes and about 0.1 billion points.
  • Semantic3D: The authors conclude that semantic segmentation performance on large-scale Semantic3D data is relatively better than on the semantic-8 setting.

C.3. Ablation Study

The ablation studies examine variants of the network, while visual and geometric analyses assess bilateral context and multi-resolution fusion behavior.

  • C.3. Ablation Study: The study evaluates baseline, efficient, and dilated variants on Area 5 of S3DIS.The baseline replaces both proposed modules, the efficient model uses random sampling, and the dilated model uses dilated-kNN with d = 2.
  • C.3. Ablation Study: Equal-weighted Bilateral Context Blocks support multi-resolution processing, while fewer blocks reduce performance but consume less GPU memory.Model N4 illustrates the network’s flexible assembly with a lower-memory configuration.
  • C.3. Ablation Study: Bilateral Context Blocks produce different responses for nearby points belonging to different semantic classes.The intermediate features are visualized as heat maps for an office scene in S3DIS.
  • C.3. Ablation Study: Shifted neighbors become closer to centroids and form more compact neighborhoods in both 3D and feature spaces.Both neighbor-to-centroid distances and neighborhood variances decrease after applying bilateral offsets.

D.2. Visualizations and Failure Cases

Visualizations compare predictions with inputs and ground truths across indoor, urban, rural, cathedral, street, and traffic scenes, while highlighting characteristic failure cases.

  • D.2. Visualizations and Failure Cases: On S3DIS, the network struggles to distinguish objects with similar shapes because it relies on local neighborhoods without object-level geometric information.Beams may be classified as doors, while walls may be predicted as boards or clutter.
  • D.2. Visualizations and Failure Cases: Semantic3D visualizations cover urban squares, rural farms, cathedral scenes, and street views, with predictions described as visually plausible despite unavailable test ground truths.The figure organizes examples by scene type rather than by quantitative comparison.
  • D.2. Visualizations and Failure Cases: On SemanticKITTI, the network detects some small objects distinct from the background but remains inaccurate when using only 3D coordinates as input.The examples compare predictions with ground truth on validation Sequence 08 and mark failure cases in red.
Loading 2103.07074v2…