PaperlayerView exact revision on arXiv

Source-linked AI summary

3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection

He Wang, Yezhen Cong, Or Litany, Yue Gao, Leonidas J. Guibas

arXiv:2012.04355v3cs.CV

TL;DR

3DIoUMatch addresses the dependence of 3D object detection on difficult-to-obtain annotations by using teacher-student learning with filtered pseudo-labels. Its IoU-aware filtering and deduplication improve results across indoor benchmarks and KITTI, while the method has a documented limitation for PV-RCNN IoU optimization.

  • Problem

    3D object detection depends heavily on difficult-to-obtain 3D annotations, motivating methods that use both labeled and unlabeled scenes.

  • Method

    3DIoUMatch uses teacher-student mutual learning with pseudo-label filtering based on class confidence, objectness, and estimated 3D IoU, plus IoU-guided deduplication.

  • Results

    3DIoUMatch consistently improves prior state-of-the-art methods on ScanNet and SUN-RGBD and surpasses a fully supervised KITTI baseline under different label ratios and categories.

  • Takeaways & Limitations

    IoU estimation makes pseudo-label filtering and deduplication aware of localization confidence across indoor and outdoor 3D detection settings.

  • Takeaways & Limitations

    Because PV-RCNN's IoU module is non-differentiable, the method does not apply IoU optimization there.

Abstract

from arXiv · show

3D object detection is an important yet demanding task that heavily relies on difficult to obtain 3D annotations. To reduce the required amount of supervision, we propose 3DIoUMatch, a novel semi-supervised method for 3D object detection applicable to both indoor and outdoor scenes. We leverage a teacher-student mutual learning framework to propagate information from the labeled to the unlabeled train set in the form of pseudo-labels. However, due to the high task complexity, we observe that the pseudo-labels suffer from significant noise and are thus not directly usable. To that end, we introduce a confidence-based filtering mechanism, inspired by FixMatch. We set confidence thresholds based upon the predicted objectness and class probability to filter low-quality pseudo-labels. While effective, we observe that these two measures do not sufficiently capture localization quality. We therefore propose to use the estimated 3D IoU as a localization metric and set category-aware self-adjusted thresholds to filter poorly localized proposals. We adopt VoteNet as our backbone detector on indoor datasets while we use PV-RCNN on the autonomous driving dataset, KITTI. Our method consistently improves state-of-the-art methods on both ScanNet and SUN-RGBD benchmarks by significant margins under all label ratios (including fully labeled setting). For example, when training using only 10\% labeled data on ScanNet, 3DIoUMatch achieves 7.7% absolute improvement on mAP@0.25 and 8.5% absolute improvement on mAP@0.5 upon the prior art. On KITTI, we are the first to demonstrate semi-supervised 3D object detection and our method surpasses a fully supervised baseline from 1.8% to 7.6% under different label ratios and categories.

1. Introduction

3DIoUMatch addresses the annotation bottleneck in 3D object detection with semi-supervised learning that filters pseudo-labels using semantic confidence and estimated localization quality. It improves prior methods across indoor benchmarks and extends semi-supervised 3D detection to KITTI.

  • Motivation: The approach targets the annotation bottleneck caused by the difficulty and cost of obtaining large, carefully labeled 3D scene datasets.It is designed to leverage both labeled and unlabeled data during training.
  • Method: The method uses teacher-student mutual learning to propagate pseudo-labels from an EMA teacher to a strongly augmented student across labeled and unlabeled scenes.VoteNet is used indoors and PV-RCNN outdoors.
  • Method: Estimated 3D IoU supplements class probability and objectness by measuring localization quality for pseudo-label filtering and NMS.The method adds a 3D IoU estimation module and applies IoU-guided Lower-Half Suppression for deduplication.
  • Results: 3DIoUMatch introduces semi-supervised 3D object detection on KITTI and surpasses a fully supervised baseline under all label ratios.

3. Method

3DIoUMatch combines teacher-student pseudo-labeling with IoU-aware filtering and selective supervision for semi-supervised 3D object detection across indoor and outdoor scenes.

  • 3.3. 3DIoUMatch for SSL on 3D object detection: 3DIoUMatch trains IoU-aware VoteNet or PV-RCNN detectors in two stages: supervised pre-training followed by pseudo-label-based semi-supervised training.The method uses VoteNet for indoor scenes and PV-RCNN for outdoor scenes, with labeled data supervising the student and teacher-generated pseudo-labels supervising unlabeled data.
  • 3.3. 3DIoUMatch for SSL on 3D object detection: An EMA teacher provides pseudo-labels for unlabeled scenes, while stronger student augmentation and asymmetric teacher augmentation support mutual learning.Teacher inputs use weak augmentation, whereas student inputs additionally undergo random flips, rotations, and uniform scaling.
  • 3.4. Pseudo-label filtering and deduplication: The method filters teacher proposals using objectness and class-confidence thresholds, then adds estimated 3D IoU to assess localization quality.IoU-aware filtering addresses the limitation that semantic confidence measures do not sufficiently capture bounding-box localization quality.
  • 3.4. Pseudo-label filtering and deduplication: IoU-guided Lower-Half Suppression removes only half of highly overlapping, lower-IoU proposals, preserving coverage when the best pseudo-label is uncertain.Unlike strict NMS, LHS is class-aware and uses dynamic thresholds among overlapping boxes to avoid discarding potentially useful supervision.
  • 3.5. Selective Supervision using Pseudo-Labels: For unlabeled scenes, the method supervises box parameters and classes near filtered pseudo-labels but omits objectness and vote losses because pseudo-labels may be incomplete.A prediction receives pseudo-label supervision when its generating vote lies within 0.3m of a pseudo bounding box.

4. Experiments

Experiments evaluate 3DIoUMatch across indoor and outdoor 3D detection benchmarks, component ablations, IoU-threshold sensitivity, and training dynamics. The method consistently improves performance over prior or supervised baselines, while IoU-based filtering and deduplication improve pseudo-label quality and localization.

  • 4.2.1 Result Comparison: 8.1 and 8.0 absolute mAP@0.5 improvements over SESS with 5% labeled data on ScanNet and SUN RGB-D, respectively.Table 1 compares 3DIoUMatch with SESS and VoteNet across labeled-data ratios; gains are attributed to train-time filtering and test-time IoU estimation.
  • 4.2.2 Ablation Study: 3.0 and 3.1 absolute improvements on ScanNet 10% mAP@0.25 and mAP@0.5, respectively, after combining IoU-based training and test-time improvements.IoU filtering and IoU-guided LHS improve over the without-IoU version by 2.3 and 1.7 points before test-time improvements.
  • 4.2.2 Ablation Study: IoU-guided LHS improves over IoU-guided NMS because it better balances pseudo-label quality and coverage.Objectness-based NMS does not further improve already filtered proposals, whereas IoU-based filtering and deduplication improve both settings.
  • 4.2.3 Result Analysis: Performance peaks at τIoU = 0.25 for mAP@0.25 and τIoU = 0.5 for mAP@0.5, while thresholds above 0.5 sharply reduce pseudo-label coverage.The differing optima reflect the stronger localization-quality preference of mAP@0.5.
  • 4.2.3 Result Analysis: Pseudo-label coverage at IoU thresholds 0.25 and 0.5 increases by about 10% during training as detection performance improves.The reported coverage is class-agnostic recall: the percentage of ground-truth objects matched by a pseudo-label above the specified IoU threshold.
  • 4.3.1 Results: 7.4 and 10.7 mAP@0.5 improvements over labeled-data-only training for pedestrian and cyclist detection on KITTI with 2% labeled data.The method improves consistently across car, pedestrian, and cyclist categories at 1%, 2%, and 100% labeled-data settings.

5. Conclusion

3DIoUMatch is a semi-supervised 3D object detection method that combines teacher-student learning with IoU-aware pseudo-label processing and inference.

  • 5. Conclusion: 3DIoUMatch combines teacher-student mutual learning with asymmetric augmentation, pseudo-label filtering, and deduplication.The student learns from an EMA teacher, while pseudo-label processing is designed to improve training quality.
  • 5. Conclusion: Its IoU estimation module makes filtering and deduplication sensitive to localization confidence.This extends confidence handling beyond classification or objectness by incorporating estimated localization quality.
  • 5. Conclusion: The method also applies IoU-guided NMS and IoU optimization at test time.

A. 3D IoU Estimation Module for VoteNet

The VoteNet IoU module estimates class-aware 3D localization quality through differentiable grid pooling, enabling IoU-based proposal filtering, deduplication, and refinement.

  • The module is designed to provide differentiable 3D IoU estimates for point-cloud detectors such as VoteNet that lack native IoU prediction.
  • It replaces hard proposal cropping with 3D Grid Pooling, interpolating features from nearby real points at virtual grid locations spanning each box.
  • Seed points, a predicted box, and a class label are processed into class-aware IoU predictions, with the class selecting the output estimate.
  • The IoU branch is appended after VoteNet proposal generation and uses 4 × 4 × 4 virtual grid points for each proposal.
  • Jittered box predictions provide on-the-fly training samples, and an L1 loss supervises the IoU estimation branch.

B. More Implementation Details for VoteNet-based 3DIoUMatch

VoteNet-based 3DIoUMatch uses extended pre-training, jitter-based IoU training, and iterative test-time box optimization alongside the grid-based IoU module.

  • The pre-training protocol runs for 900 epochs with ADAM and learning-rate decays at epochs 400, 600, and 800.
  • IoU-module training jitters predicted box centers and sizes with Gaussian noise to generate additional samples, using an averaged L1 estimation loss.
  • The IoU module combines seed features, virtual grid points, and class information to regress class-aware 3D IoU.
  • At inference, box centers and sizes are iteratively updated using gradients of the IoU estimate with respect to those parameters.
  • Ten optimization steps provide noticeable improvement without substantially slowing inference, while step sizes from [1e−4, 5e−4] perform similarly.

C. More Implementation Details for PV-RCNN-based 3DIoUMatch

PV-RCNN-based 3DIoUMatch adapts training schedules, augmentation, proposal selection, and thresholds to semi-supervised settings with limited labels.

  • Training uses batches containing eight labeled and eight unlabeled samples across eight GPUs, with repeated scene traverses during semi-supervised learning.
  • Ground-truth sampling databases are restricted to boxes from the labeled subset, while pseudo-label generation omits this augmentation because annotations are unavailable.
  • Pseudo-label generation uses PV-RCNN’s test-time RPN proposal selection instead of ground-truth-assisted proposal selection when labels are limited.
  • With 100% labeled data, ground-truth sampling is restored and class-specific thresholds are set for cars, pedestrians, cyclists, and classification confidence.

D. Overhead of the IoU module

The IoU estimation module adds moderate memory and runtime overhead during training, including the cost of computing ground-truth IoU supervision.

  • The IoU estimation module introduces moderate memory and time overhead during training.The reported measurements use batch size 8 on one GTX 1080Ti for ScanNet and SUNRGB-D.

E. IoU Module Comparison

The paper compares its differentiable IoU module with box-query designs and argues that VoteNet’s sparse seed points make querying inside small boxes difficult, while its alternative avoids that constraint.

  • The comparison includes VoteNet, SESS, a version without IoU, and 3DIoUMatch across mAP@0.25 and mAP@0.5.The supplied result labels identify the compared methods and metrics, while the tables cover ScanNet and SUNRGB-D evaluations.
  • The proposed module predicts 3D IoU from queried features and is designed to remain differentiable with respect to bounding-box parameters.The comparison discusses a point-cloud-based module for IoU estimation and IoU optimization.
  • VoteNet’s small number of seed points can make box-query methods struggle to retrieve points inside small predicted boxes.The method instead avoids being restricted to points inside the predicted bounding box.
  • The implemented STD-style comparison is confounded by different backbones, different task settings, and parameter changes made for memory and seed-point constraints.The authors also report serious overfitting with the STD IoU module, suggesting limited suitability for their problem.

F. Why not Supervise Votes and Objectness in VoteNet?

The method excludes vote regression and objectness classification from unlabeled-data supervision because pseudo-labels do not reliably specify complete vote or objectness targets, and experiments show performance drops when these terms are added.

  • Pseudo-label filtering can establish that an object is near a pseudo-box, but cannot reliably identify where no nearby pseudo-box exists.This incompleteness makes objectness supervision on unlabeled scenes prone to bias toward detecting objects.
  • The method therefore supervises other VoteNet loss terms on unlabeled data while excluding vote regression and objectness binary classification.
  • Qualitative results visualize ScanNet detections at 10% labeled data and SUNRGB-D detections at 5% labeled data using IoU-based box colors.Green boxes have IoU ≥0.25 and red boxes have IoU <0.25 in the ScanNet visualization.
  • Supervising objectness or vote prediction on unlabeled data with pseudo-labels reduces performance.The reported experiments evaluate ScanNet with 10% labeled data and SUNRGB-D with 5% labeled data.

G. Per-class Evaluation

Per-class evaluation shows that 3DIoUMatch improves average precision over SESS on nearly all classes and generally exceeds the version without IoU.

  • 3DIoUMatch improves average precision on nearly all classes over SESS and performs better than the without-IoU version on most classes.The evaluations use ScanNet with 10% labeled data and SUNRGB-D with 5% labeled data.

H. Qualitative Results

Qualitative validation on ScanNet and SUN-RGBD shows that 3DIoUMatch produces more accurate predictions while reducing false positives.

  • H. Qualitative Results: 3DIoUMatch gives more accurate predictions and significantly reduces false positives in qualitative validation.The comparisons use ScanNet with 10% labeled training data and SUN-RGBD with 5% labeled training data.
  • H. Qualitative Results: The visualizations distinguish predictions with IoU ≥0.25 from those below 0.25 using green and red bounding boxes.
  • H. Qualitative Results: Qualitative results are reported on the ScanNet and SUN-RGBD validation sets under limited-label training settings.
Loading 2012.04355v3…