PaperlayerView exact revision on arXiv

Source-linked AI summary

JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds

Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, Chiew-lan Tai

arXiv:2007.06888v1cs.CV

TL;DR

3D semantic edge detection and joint learning with semantic segmentation were largely unexplored. JSENet introduces a two-stream network with joint output refinement and a dual semantic edge loss, achieving strong segmentation results on S3DIS and ScanNet and outperforming semantic-edge baselines.

  • Problem

    3D semantic edge detection and joint learning of semantic segmentation with edge detection had received little attention despite their close relationship.

  • Method

    JSENet uses two fully-convolutional streams whose segmentation and edge outputs are jointly refined, with a dual semantic edge loss encouraging sharper segmentation boundaries.

  • Results

    JSENet achieves 67.7% mIoU on S3DIS Area-5 and 69.9% mIoU on the ScanNet test set, while outperforming baseline methods for semantic edge detection.

  • Takeaways & Limitations

    The experiments support JSENet as an effective joint architecture producing high-quality semantic segmentation masks and semantic edge maps on S3DIS and ScanNet.

Abstract

from arXiv · show

Semantic segmentation and semantic edge detection can be seen as two dual problems with close relationships in computer vision. Despite the fast evolution of learning-based 3D semantic segmentation methods, little attention has been drawn to the learning of 3D semantic edge detectors, even less to a joint learning method for the two tasks. In this paper, we tackle the 3D semantic edge detection task for the first time and present a new two-stream fully-convolutional network that jointly performs the two tasks. In particular, we design a joint refinement module that explicitly wires region information and edge information to improve the performances of both tasks. Further, we propose a novel loss function that encourages the network to produce semantic segmentation results with better boundaries. Extensive evaluations on S3DIS and ScanNet datasets show that our method achieves on par or better performance than the state-of-the-art methods for semantic segmentation and outperforms the baseline methods for semantic edge detection. Code release: https://github.com/hzykent/JSENet

1 Introduction

JSENet addresses the previously unexplored joint learning of 3D semantic segmentation and semantic edge detection. Its two-stream architecture couples segmentation and edge outputs through joint refinement and a dual semantic edge loss, achieving strong results on S3DIS and ScanNet.

  • Motivation: 3D semantic edge detection had received little attention, and no prior work jointly learned it with semantic segmentation.The paper frames segmentation and edge detection as closely related, dual tasks for scene understanding.
  • Method: JSENet introduces a two-stream 3D fully-convolutional network with separate semantic segmentation and semantic edge detection streams.The segmentation stream produces SSP masks, while the edge stream produces SEP maps.
  • Method: The lightweight joint refinement module explicitly uses SSP masks and SEP maps together to refine both outputs by exploiting task duality.This design directly connects the two task outputs rather than limiting interaction to shared features.
  • Method: The dual semantic edge loss encourages semantic segmentation predictions with finer boundaries aligned to ground-truth semantic edges.This loss is presented as a separate contribution alongside the joint refinement module.
  • Results: 67.7% IoU on S3DIS Area-5 and 69.9% IoU on the ScanNet test set are reported for semantic segmentation, while semantic edge detection exceeds baseline methods.The paper constructs a new 3D semantic edge benchmark from S3DIS and ScanNet because no existing 3D SED dataset was available.

2 Related Work

Prior 3D semantic segmentation work includes multiview, voxel-based, and point-based methods, while semantic edge detection had largely remained a 2D problem. JSENet extends joint segmentation-edge learning to class-aware 3D semantic edges, whose physical object boundaries create stronger cross-class relationships than 2D occlusion boundaries.

  • 3D Semantic Segmentation: 3D semantic segmentation methods are commonly grouped into multiview image-based, voxel-based, and point-based approaches.JSENet belongs to the point-based category.
  • 3D Semantic Segmentation: Voxel-based methods can achieve strong 3D segmentation performance but require intensive computation, whereas point-based methods retain point-level information with lower computational consumption.Point-based methods include neighboring feature pooling, graph construction, attention aggregation, and kernel-based convolution.
  • Semantic Edge Detection: Before this work, learning-based semantic edge detection methods and benchmarks were developed for 2D images rather than 3D point clouds.The related 2D literature includes HED, CASENet, hierarchical supervision, and methods reducing annotation noise.
  • Semantic Edge Detection: In 2D, semantic edges across classes are treated as weakly related occlusion boundaries, but 3D semantic edges are physical object boundaries with stronger cross-class relationships.This distinction motivates different interaction patterns for 3D semantic edge detection.
  • Joint Learning: Earlier joint segmentation-edge methods often shared features while keeping predicted masks and edge maps separate from each other.JSENet instead studies joint learning for semantic segmentation and class-aware semantic edge detection with stronger output interactions.

3 JSENet

JSENet jointly learns semantic segmentation and semantic edge detection with two streams, enhanced feature extraction, and a joint refinement module that exchanges region and edge information. Its training combines standard segmentation and edge supervision with a dual semantic edge loss designed to align segmentation boundaries with semantic edges.

  • Architecture: JSENet uses a shared-encoder, two-stream architecture with an SS encoder-decoder stream and an SED stream for class-aware edge maps.The SED stream adds enhanced feature extraction and hierarchical supervision before the streams are fused.
  • Semantic Edge Detection Stream: The SED stream recovers spatial and semantic cues through skip-layer feature fusion and hierarchical supervision at intermediate layers.Early features produce binary edge maps, while later features produce SSP masks, with separate losses supervising these outputs.
  • Joint Refinement Module: The joint refinement module has separate segmentation and edge branches that fuse SSP masks and SEP maps to produce refined outputs.Its inputs are the SS stream's SSP mask and the SED stream's class-specific SEP maps.
  • Joint Refinement Module: The edge map generation sub-module converts categorical segmentation predictions into class-specific edge activation maps using mean-filtered spatial differences.Points near class boundaries receive stronger activation because neighboring class probabilities differ more after filtering and subtraction.
  • Joint Multi-task Learning: Training supervises segmentation with multi-class cross-entropy, semantic edges with weighted multi-label loss, and generated edge activations with dual semantic edge loss.The dual loss penalizes mismatches between predicted and ground-truth edge activation maps, encouraging segmentation outputs to align with semantic edges.

4 Experiments

Experiments evaluate JSENet on S3DIS and ScanNet through dataset-specific semantic-edge construction, standard metrics, controlled ablations, and comparisons with segmentation and edge-detection baselines.

  • Dataset preparation: S3DIS contains approximately 273 million points across six indoor areas with 13 classes, while ScanNet provides 1513 training and 100 test scenes with 20 classes.The datasets were selected partly because indoor scenes provide denser object connections and more detailed semantic labels.
  • Dataset preparation: Semantic edge ground truth is generated from neighboring points with different labels, using a 2cm radius for S3DIS and mesh rasterization followed by 1cm-grid downsampling for ScanNet.Points are assigned all semantic classes appearing among differently labeled neighbors.
  • Evaluation: SS is evaluated with mIoU and SED with mMF (ODS), using thicker point-cloud edges without the 2D localization tolerance.The evaluation adapts edge thickness to the sparsity of point clouds.
  • Ablation study: On S3DIS Area-5, adding the SED stream improves SS by 1.5%, while the joint refinement module adds another 1.5% over the naively combined model.The complete model improves about 3% over the standalone SS stream and more than 2% over reported KPConv results.
  • Ablation study: For SED, the standalone stream reaches 29.4% mMF (ODS), while enhanced extraction, hierarchical supervision, joint learning, and refinement provide incremental gains.The reported gains are approximately 0.5%, 0.3%, 0.3%, and 0.5%, respectively.
  • Ablation study: Removing hierarchical supervision decreases SED performance by 0.3%, and dual semantic edge losses improve predicted-mask edge-alignment F-score by 0.4%.The selected hierarchical-supervision configuration performs best among the tested choices.
  • Benchmark comparison: JSENet achieves 67.7% mIoU on S3DIS and 69.9% on ScanNet, outperforming all S3DIS baselines and ranking third on ScanNet.It achieves the best performance among point-based methods and consistently exceeds KPConv.
  • Benchmark comparison: For SED, JSENet outperforms both baselines on both datasets, while extended CASENet performs worse than the KPConv-based baseline.The comparison uses extended CASENet and a KPConv-structured network with a changed output layer.

5 Conclusions

JSENet jointly performs semantic segmentation and semantic edge detection through a two-stream architecture with explicit region–edge refinement. It achieves strong results on S3DIS and ScanNet, while annotation noise remains a limitation for semantic edge detection.

  • 5 Conclusions: JSENet uses a two-stream fully-convolutional architecture with a lightweight joint refinement module that wires region and edge information together.The network includes feature-fusion and edge-map-generation submodules, plus a dual semantic edge loss for sharper boundaries.
  • 5 Conclusions: JSENet achieves state-of-the-art results on the S3DIS and ScanNet datasets, significantly improving over strong baselines.The reported results cover both semantic segmentation and semantic edge detection outputs.
  • 5 Conclusions: Human-annotation noise in ground-truth semantic edges affects the semantic edge detection method.The authors identify special treatment of misaligned ground-truth semantic edges as a future direction.

Supplementary Material for JSENet: Joint Semantic Segmentation and Edge

The supplementary document organizes additional material on dataset preparation, model complexity, qualitative comparisons, and detailed semantic segmentation results.

  • Supplementary Material: Section A covers dataset selection and preparation.
  • Supplementary Material: Section B compares the model sizes and speeds of JSENet with other networks.
  • Supplementary Material: Section C provides qualitative comparison examples and additional ScanNet visualizations.
  • Supplementary Material: Section D presents detailed semantic segmentation results.

A Dataset selection and preparation.

The study uses indoor datasets because existing outdoor-scene point clouds are often incomplete, making semantic edges difficult to define. It also adapts edge construction to 3D point sparsity and semantics.

  • A Dataset selection and preparation: The main experiments use the indoor S3DIS and ScanNet datasets rather than outdoor-scene datasets.Outdoor scenes often contain missing parts and objects that are not densely connected in the point cloud.
  • A Dataset selection and preparation: Figure 1 contrasts an outdoor Semantic3D scene with an indoor ScanNet scene.
  • A Dataset selection and preparation: The method generates thick 3D semantic edges because point clouds are sparser than image pixels.Unlike 2D boundaries with the background, the 3D construction considers semantic edges between objects.

B Complexity of the network, in comparison with other works.

The supplementary complexity analysis compares JSENet with KPConv on ScanNet using training time, iteration speed, and convergence requirements.

  • B Complexity of the network, in comparison with other works: The complexity experiments were conducted on a PC with 8 Intel i7-7700 CPUs and one GeForce GTX 1080Ti GPU.
  • B Complexity of the network, in comparison with other works: 49h is the reported total training time for JSENet on ScanNet, compared with about 31h for KPConv.JSENet uses two training steps: 0.9s per iteration for 170K iterations, then 0.6s per iteration for 40K iterations; KPConv uses 0.7s per iteration for 160K iterations.

Training.

Runtime complexity is compared across JSENet and three state-of-the-art methods using identical ScanNet point sets.

  • Training: Table 1 compares runtime complexity for KPConv, PointConv, MinkowskiNet, and JSENet.The comparison uses the same sets of 13,000 ScanNet points.
  • Training: JSENet is largely comparable to KPConv in inference time and parameter size.
  • Training: JSENet is more efficient and compact than PointConv and MinkowskiNet on the reported runtime measures.

C Qualitative Visualization.

The qualitative section visualizes ScanNet results and compares semantic segmentation and semantic edge detection before and after joint refinement.

  • C Qualitative Visualization: Figure 2 presents qualitative results from the ScanNet validation set.
  • C Qualitative Visualization: Figures 3 and 4 visualize semantic segmentation and semantic edge detection, respectively, before and after joint refinement.
  • C Qualitative Visualization: Semantic edges are thickened for visualization, while black points in ground-truth semantic segmentation masks denote unlabeled or unconsidered points.

D Detailed semantic segmentation results.

The paper reports detailed semantic-segmentation class scores for S3DIS and ScanNet to support benchmarking and comparison with future work.

  • D Detailed semantic segmentation results: Detailed semantic-segmentation class scores are presented for S3DIS and ScanNet.The results appear in Tables 2 and 3, respectively.
  • D Detailed semantic segmentation results: Table 2 reports detailed mIoU scores (%) for semantic segmentation on S3DIS Area-5.
  • D Detailed semantic segmentation results: Table 3 reports detailed mIoU scores (%) for semantic segmentation on the ScanNet test set.
Loading 2007.06888v1…