PaperlayerView exact revision on arXiv

Source-linked AI summary

SESS: Self-Ensembling Semi-Supervised 3D Object Detection

Na Zhao, Tat-Seng Chua, Gim Hee Lee

arXiv:1912.11803v3cs.CV

TL;DR

High-quality 3D annotations are expensive to acquire, motivating semi-supervised learning for point cloud-based 3D object detection. SESS uses a Mean Teacher framework with perturbations and proposal-level consistency losses, and experiments validate its effectiveness across inductive and transductive settings on two real-world datasets.

  • Problem

    Point cloud-based 3D object detection relies on many high-quality 3D ground truths, which are expensive to acquire, while semi-supervised alternatives remain limited.

  • Method

    SESS uses a Mean Teacher framework with point-cloud perturbations, proposal alignment, and consistency losses for object locations, semantic categories, and sizes.

  • Results

    Experiments on two real-world datasets validate SESS's effectiveness in both inductive and transductive semi-supervised 3D object detection.

  • Takeaways & Limitations

    SESS is presented as a general framework applicable to both inductive and transductive semi-supervised 3D object detection.

  • Takeaways & Limitations

    Prior semi-supervised work required RGB-D input, did not work on pure 3D point clouds, and still needed many 3D labels for strong object classes.

Abstract

from arXiv · show

The performance of existing point cloud-based 3D object detection methods heavily relies on large-scale high-quality 3D annotations. However, such annotations are often tedious and expensive to collect. Semi-supervised learning is a good alternative to mitigate the data annotation issue, but has remained largely unexplored in 3D object detection. Inspired by the recent success of self-ensembling technique in semi-supervised image classification task, we propose SESS, a self-ensembling semi-supervised 3D object detection framework. Specifically, we design a thorough perturbation scheme to enhance generalization of the network on unlabeled and new unseen data. Furthermore, we propose three consistency losses to enforce the consistency between two sets of predicted 3D object proposals, to facilitate the learning of structure and semantic invariances of objects. Extensive experiments conducted on SUN RGB-D and ScanNet datasets demonstrate the effectiveness of SESS in both inductive and transductive semi-supervised 3D object detection. Our SESS achieves competitive performance compared to the state-of-the-art fully-supervised method by using only 50% labeled data. Our code is available at https://github.com/Na-Z/sess.

1. Introduction

Point cloud-based 3D object detection typically requires expensive, high-quality 3D annotations, motivating semi-supervised learning with limited labeled data. SESS addresses in-category semi-supervised detection using a Mean Teacher framework, perturbations, and consistency losses for 3D proposals.

  • 1. Introduction: Large-scale, well-annotated 3D data are costly and time-consuming to collect, creating a need for alternatives to strongly supervised detection.
  • 1. Introduction: SESS predicts 3D bounding boxes and semantic labels for an unlabeled scene after training on a mixture of labeled and unlabeled data.
  • 1. Introduction: SESS targets in-category semi-supervised 3D object detection from point clouds using few strongly labeled samples and many unlabeled samples.The labeled samples are assumed to contain all object classes of interest, with few examples per class.
  • 1. Introduction: SESS uses a Mean Teacher paradigm in which a teacher guides student predictions to remain consistent under random perturbations.The predictions are sets of 3D object proposals that are aligned during training.
  • 1. Introduction: Three consistency losses enforce agreement between teacher and student proposals on object centers, classes, and sizes.These losses target both geometric and semantic properties of predicted 3D objects.

2. Related work

Related work spans point-cloud, voxel-grid, and 2D-projection 3D detection, while semi-supervised approaches seek to exploit abundant unlabeled data. SESS differs from prior work by directly processing raw point clouds with limited labeled samples and a self-ensembling architecture.

  • 2. Related work: Existing 3D detection methods require many high-quality 3D ground truths, which are expensive to acquire and limit practical applicability.
  • 2. Related work: Prior semi-supervised 3D detection uses a cross-category or mix-supervised setting requiring all 2D box labels and some 3D box labels.
  • 2. Related work: SESS directly processes raw point clouds in one step, avoiding dependence on a separate 2D detector in the related two-step pipeline.
  • 2. Related work: Self-ensembling improves generalization by encouraging consensus among predictions under perturbations, while Mean Teacher averages network parameters rather than historical predictions.
  • 2. Related work: SESS trains student and teacher networks with labeled and unlabeled samples, using supervised loss for labeled data and consistency loss against teacher predictions.

3. Our Method

SESS adapts self-ensembling to semi-supervised point-cloud 3D detection by combining perturbed student–teacher proposals with supervised and consistency training. Its model-agnostic framework aligns proposals and enforces consistency in object centers, semantic classes, and sizes.

  • SESS Architecture: SESS trains student and teacher 3D detectors on perturbed point clouds, using supervised loss for labeled data and consistency loss for unlabeled data.The teacher is updated from the student through an exponential moving average.
  • SESS Architecture: The framework uses two sampled point sets, transforms teacher proposals into the student coordinate system, and aligns proposals by nearest center distance.Ground-truth labels receive the same transformation before supervised training.
  • Perturbation Scheme: SESS applies random sub-sampling plus stochastic flips, rotations, and scaling to preserve global object geometry while reducing reliance on unintended point-cloud properties.The transformations include x- and y-axis flips, upright-axis rotation, and scaling.
  • Consistency Loss: Proposal consistency is enforced across object centers, semantic classes, and box sizes after teacher-to-student alignment.Center consistency minimizes alignment errors, class consistency uses KL divergence, and size consistency uses mean squared error.
  • Consistency Loss: The total consistency objective is a weighted sum of the three consistency terms, with separate weights controlling their relative importance.The framework is model-agnostic, allowing existing point-cloud 3D detectors to serve as the backbone.

4. Experiments

SESS is evaluated on SUN RGB-D and ScanNetV2 under varying labeled-data proportions, inductive and transductive settings, and ablations of perturbations and consistency losses.

  • Comparison with Fully-supervised Methods: 24.51% and 28.09% improvement over VoteNet with 10% labeled data on SUN RGB-D and ScanNetV2, respectively.SESS significantly outperforms VoteNet under every labeled-data ratio, with a larger gap as labeled data decreases.
  • Comparison with Fully-supervised Methods: 50% labeled data brings SESS close to fully supervised VoteNet trained with 100% labels on both datasets.With all labeled training data, SESS further surpasses VoteNet’s fully supervised performance.
  • Transductive Semi-supervised Learning: SESS consistently outperforms fully supervised VoteNet in transductive learning across labeled-data proportions on both datasets.The result supports SESS as a framework applicable to both inductive and transductive semi-supervised detection.
  • Ablation Studies: Removing the entire perturbation scheme causes a large performance drop on both datasets, while individual perturbations have dataset-dependent effects.Rotation contributes less on axis-aligned ScanNetV2, whereas scaling contributes less on similarly scaled SUN RGB-D scenes.
  • Ablation Studies: The full combination of center-aware, class-aware, and size-aware consistency losses achieves the best performance on both datasets.Center-aware and class-aware losses contribute more individually than size-aware loss, while combinations with size-aware loss still improve performance.
  • Qualitative Results and Analysis: Qualitative examples show SESS detecting more unannotated objects and producing more accurate predictions than VoteNet with limited labeled data.On a ScanNet scene with 7 tables and 27 chairs, SESS correctly recognizes 7 tables and 26 chairs with 30% labeled data, versus VoteNet’s 6 tables and 24 chairs.

5. Conclusion

SESS is presented as a self-ensembling semi-supervised point-cloud 3D object detection framework that reduces reliance on strong labels. Experiments on two real-world datasets support its effectiveness in both inductive and transductive settings.

  • 5. Conclusion: SESS is a self-ensembling semi-supervised point-cloud 3D object detection framework that does not require many strong labels.It follows the Mean Teacher paradigm, using a point-based perturbation scheme and three consistency losses.
  • 5. Conclusion: Experiments on two real-world datasets validate SESS's effectiveness and advantage.
  • 5. Conclusion: SESS can be applied to both inductive and transductive semi-supervised 3D object detection.

B. Additional Evaluation Metric

At an IoU threshold of 0.5, SESS outperforms fully supervised VoteNet across different labeled-data ratios in both inductive and transductive evaluation.

  • B. Additional Evaluation Metric: SESS significantly outperforms fully supervised VoteNet at IoU 0.5 across labeled-data ratios in both inductive and transductive learning.The evaluations use mean average precision on SUN RGB-D and ScanNetV2.

C. Per-class Evaluation

Using all training samples, SESS achieves higher per-class average precision than fully supervised VoteNet on every SUN RGB-D class and most ScanNetV2 classes.

  • C. Per-class Evaluation: SESS is superior to fully supervised VoteNet on all 10 SUN RGB-D classes and 14 of 18 ScanNetV2 classes.The comparison reports per-class average precision using all training samples.

D. More Qualitative Results and Discussions

Qualitative evaluations show SESS detecting many difficult or weakly represented objects in SUN RGB-D and ScanNetV2 scenes, while some extremely sparse objects remain undetected. The figures compare VoteNet and SESS across multiple scene types and datasets.

  • D. More Qualitative Results and Discussions: Tables 7 and 8 report per-class AP@0.25 IoU for SUN RGB-D and ScanNetV2 validation sets using 100% training samples.
  • D. More Qualitative Results and Discussions: SESS successfully detects most challenging SUN RGB-D objects affected by occlusion, partial visibility, or extreme sparsity.Examples include unannotated chairs at the back of a classroom and a table in front of a bedroom bed.
  • D. More Qualitative Results and Discussions: SESS still fails on some extremely challenging objects with no or few representative points.
  • D. More Qualitative Results and Discussions: On ScanNetV2, SESS detects difficult objects without explicit geometric features, including bookshelves and doors.The discussion attributes improved 3D-box localization to consistency losses encoding geometric and semantic information.
  • D. More Qualitative Results and Discussions: Figures 7 and 8 compare fully supervised VoteNet with SESS using 100% training samples across four scene types in each dataset.SUN RGB-D examples cover classroom, bedroom, study space, and living room; ScanNetV2 examples cover library, kitchen, hotel, and lounge.
Loading 1912.11803v3…