PaperlayerView exact revision on arXiv

Source-linked AI summary

MonoDistill: Learning Spatial Features for Monocular 3D Object Detection

Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, Wanli Ouyang

arXiv:2201.10830v1cs.CV

TL;DR

Monocular 3D detection is challenging because a single image lacks reliable depth cues. MonoDistill projects LiDAR signals into the image plane to train a teacher that transfers spatial knowledge to a monocular detector. On KITTI, it ranks first among monocular-based methods.

  • Problem

    Accurate 3D detection from monocular images is difficult because they lack depth cues, despite being a cheaper alternative to stereo or LiDAR.

    Source: §1 · p. 1

  • Method

    MonoDistill projects LiDAR signals into the image plane, trains an image-aligned teacher, and distills its learned spatial knowledge into the monocular detector.

    Source: §1 · p. 2

  • Results

    On KITTI, MonoDistill ranks first among monocular-based models when applied to a simple baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach transfers spatial cues directly rather than through estimated depth maps, without changing the baseline architecture or adding inference computational cost.

    Source: §1 · p. 2

    The compared depth estimators were pretrained on KITTI Depth, which overlaps with the KITTI 3D validation set.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

3D object detection is a fundamental and challenging task for 3D scene understanding, and the monocular-based methods can serve as an economical alternative to the stereo-based or LiDAR-based methods. However, accurately detecting objects in the 3D space from a single image is extremely difficult due to the lack of spatial cues. To mitigate this issue, we propose a simple and effective scheme to introduce the spatial information from LiDAR signals to the monocular 3D detectors, without introducing any extra cost in the inference phase. In particular, we first project the LiDAR signals into the image plane and align them with the RGB images. After that, we use the resulting data to train a 3D detector (LiDAR Net) with the same architecture as the baseline model. Finally, this LiDAR Net can serve as the teacher to transfer the learned knowledge to the baseline model. Experimental results show that the proposed method can significantly boost the performance of the baseline model and ranks the $1^{st}$ place among all monocular-based methods on the KITTI benchmark. Besides, extensive ablation studies are conducted, which further prove the effectiveness of each part of our designs and illustrate what the baseline model has learned from the LiDAR Net. Our code will be released at \url{https://github.com/monster-ghost/MonoDistill}.

1 INTRODUCTION

Monocular 3D detection offers a lower-cost alternative to stereo and LiDAR, but sparse depth cues make accurate localization difficult. MonoDistill transfers spatial knowledge from an image-aligned LiDAR teacher without adding inference cost to the monocular detector.

1 INTRODUCTION

  • Monocular cameras are cheaper and easier to deploy than stereo or LiDAR sensors, but monocular 3D detection lacks depth cues and struggles with localization.Depth-estimation approaches add computational cost or train the estimator and detector separately; an end-to-end alternative is also reported to run at 630 ms per image.Source: §1 · pp. 1–2
  • MonoDistill projects LiDAR signals into the image plane to train an image-version teacher, bridging 3D LiDAR and 2D image feature representations for distillation.The teacher uses projected LiDAR maps, and the monocular detector learns under its guidance.Source: §1 · p. 2
  • The method transfers spatial cues directly from the teacher rather than through estimated depth maps, avoiding proxy-task information loss without changing the baseline architecture.The unchanged architecture introduces no extra inference computational cost.Source: §1 · p. 2
  • MonoDistill ranks first among monocular-based models on KITTI when applied to a simple baseline.The experiments also attribute the improvements to introduced spatial cues rather than other unaccountable CNN factors.Source: §1 · p. 2

2 RELATED WORKS

Prior monocular 3D detection work addresses missing depth information through varied monocular strategies, depth-estimation pipelines, and knowledge distillation. MonoDistill differs by distilling depth-informed features while retaining an image-based model without extra inference cost.

2 RELATED WORKS

  • Monocular 3D detection approaches include network design, loss formulation, 3D priors, geometric constraints, and perspective modeling.These approaches address the challenge of unreliable depth from monocular images.Source: §2 · p. 2
  • Depth-based methods augment RGB inputs or features, back-project estimated depth into pseudo-LiDAR, or lift 2D features using an end-to-end depth distribution.Unlike these methods, MonoDistill learns feature representations under depth-map guidance rather than integrating depth maps, maintaining efficiency.Source: §2 · p. 3
  • Knowledge distillation transfers learned knowledge from larger CNNs to smaller models and has been explored in vision tasks, but rarely in monocular 3D detection.This work applies a distillation-based paradigm to introduce depth cues efficiently.Source: §2 · p. 3
  • Compared with LIGA Stereo, MonoDistill targets monocular data and trains an aligned teacher from projected LiDAR; the authors report about 35 ms per image versus LIGA Stereo’s 350 ms.The passage also distinguishes their conclusions about soft-label and result-space supervision, attributing differences to the distinct tasks and data.Source: §2 · p. 3

3 METHOD

MonoDistill trains a LiDAR-based teacher with image-aligned LiDAR maps, then transfers its spatial guidance to a monocular student. Three complementary distillation schemes guide the student during training, while inference uses only that student.

3.1 OVERVIEW

  • MonoDistill combines a monocular 3D detector, an aligned LiDAR-based detector, and side branches that transfer teacher guidance to the student.The proposed framework uses three distillation schemes and runs only the student network at inference.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 BASELINE MODEL

  • The MonoDLE student uses DLA-34, while its teacher is trained on sparse and interpolated dense LiDAR depth maps with the same network architecture.The baseline also normalizes predicted-object confidence using estimated depth uncertainty.Source: §3.2 · p. 4

3.3 MONODISTILL

  • Scene-level distillation matches affinity maps of high-level teacher and student features, transferring pairwise structure rather than directly matching modality-specific features.Grouping features into local regions reduces the affinity map’s quadratic computational and storage cost, with no observed performance drop.Source: §3.3 · pp. 4–5
  • Object-level feature distillation matches teacher and student features within ground-truth 2D bounding boxes to limit transfer from noisy background regions.The object-feature mask is generated from the center point and bounding-box size.Source: §3.3 · p. 5
  • Object-level result-space distillation uses teacher predictions as soft labels, sampling thresholded pixels from a Gaussian-like foreground region based on the center and 2D box.The figure distinguishes center-point foreground labels from generated regions; 2D boxes serve as the foreground for L_of.Source: §3.3 · p. 5, Figure 4 · p. 6
  • Feature-space losses use attention-based fusion and are restricted to the last three backbone blocks; the student objective combines the baseline loss with all three distillation losses.The restriction avoids enforcing cross-modal agreement on low-level features that may differ between LiDAR and images.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments on KITTI evaluate MonoDistill against a baseline and other methods, ablate its design, and examine what the teacher transfers. The reported findings include stronger benchmark performance, benefits from complementary distillation choices, and better localization in qualitative comparisons.

4.1 SETUP

  • Experiments use KITTI, report final results on its test set, and conduct ablations on the validation set, focusing mainly on Car.The training data is divided into 3,712 training and 3,769 validation samples; Pedestrian and Cyclist results are provided in the appendix.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Compared with the baseline, the full model improves 3D detection by 3.34, 5.02, and 2.98, and BEV detection by 5.16, 6.62, and 3.87 on moderate, easy, and hard settings, respectively.Scene-level feature, object-level feature, and object-level result distillation each improve accuracy, with complementary gains; feature fusion also helps.Source: §4.2 · p. 6, Table 1 · p. 6
  • Foreground-region guidance improves moderate 3D detection accuracy by 0.72 over whole-image guidance, while generated region labels outperform sparse point labels.Label diffusion increases positive samples for supervision, improving performance.Source: §4.2 · pp. 6–7

4.3 MORE DISCUSSIONS

  • Cross-model evaluation attributes stronger teacher guidance to location and dimension estimation, with limited valuable cues for classification and orientation.A sparse-map teacher drops from 42.45% to 22.05% AP|40, yet its student benefits are comparable to those guided by the dense-map teacher; complementary information matters more than teacher performance for student improvement.Source: §4.3 · p. 7, Table 5 · p. 8, Table 6 · p. 9

4.2 MAIN RESULTS

  • On the KITTI test set, MonoDistill improves prior-best 3D detection by 1.83, 0.50, and 1.53 on moderate, easy, and hard, and BEV by 2.51, 1.21, and 2.46, respectively.The method outperforms existing methods in all reported metrics; all listed methods require only RGB images at inference.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 MORE DISCUSSIONS

  • Using monocular depth estimation as an intermediate step lags behind direct LiDAR-based learning, despite more training data and ground-truth rather than noisy depth maps.The authors attribute this result to information loss in monocular depth estimation; the cited depth-based methods use KITTI Depth pretraining that overlaps the KITTI 3D validation set.Source: §4.3 · pp. 7–9, Table 7 · p. 9

4.4 QUALITATIVE RESULTS

  • Qualitative comparisons show better localization for the proposed method than for the baseline.Figure 5 marks baseline detections in green, proposed-method detections in blue, ground truth in red, and circles the main differences.Source: §4.4 · p. 9, Figure 5 · p. 9

5 CONCLUSION

MonoDistill transfers spatial cues to monocular 3D detection through knowledge distillation while leaving the target model unchanged. On KITTI, it ranks first among monocular 3D detectors at 25 FPS.

5 CONCLUSION

  • On KITTI, MonoDistill ranks 1st among all monocular 3D detectors at 25 FPS.Source: §5 · p. 9
  • MonoDistill introduces spatial cues to monocular 3D detection through knowledge distillation.Source: §5 · p. 9
  • Unlike previous schemes with the same motivation, MonoDistill leaves the target model unchanged and learns spatial features from a model rich in them.The paper describes the design as performing well in both performance and efficiency.Source: §5 · p. 9

A.1 MORE DETAILS OF THE BASELINE MODEL

The baseline uses a DLA-34 backbone and seven detection heads for 2D and 3D predictions. Training uses uncertainty-aware depth scoring and a fixed teacher after teacher pretraining.

A.1 MORE DETAILS OF THE BASELINE MODEL

  • The baseline uses DLA-34 without deformable convolutions and predicts through seven heads: three for 2D components and four for 3D components.Source: §A.1 · p. 13
  • Its 3D heads predict projected-center offset, depth, dimensions, and orientation, while the 2D heads predict heatmap, keypoint-to-box-center offset, and box size.Source: §A.1 · p. 13
  • Depth uses a modified L1 loss with heteroscedastic aleatoric uncertainty, and predicted depth uncertainty also normalizes each box's confidence score.The score is intended to indicate depth uncertainty.Source: §A.1 · p. 13
  • Training runs for 150 epochs with Adam, warm-up, and random flip and center-crop augmentation; the teacher is pretrained and then fixed.The initial learning rate is 1.25e−4, decayed at epochs 90 and 120, with a five-epoch warm-up.Source: §A.1 · p. 13

A.2 PEDESTRIAN/CYCLIST DETECTION.

The method improves pedestrian and cyclist detection over the baseline in validation experiments and is competitive on the KITTI test set. These categories remain more challenging than cars, with limited training samples noted as a factor in result variability.

A.2 PEDESTRIAN/CYCLIST DETECTION

  • The method increases pedestrian AP by 0.69 on the hard difficulty level and is competitive with current state-of-the-art methods on the KITTI test set.Source: §A.2 · p. 13, Table 8 · p. 13
  • Across three runs with different random seeds, the method effectively boosts baseline pedestrian and cyclist accuracy on KITTI validation.Table 9 reports means across experiments at 0.25 and 0.5 IoU thresholds and standard deviations over random seeds.Source: §A.2 · p. 13, Table 9 · p. 14
  • Pedestrians and cyclists are harder to detect than cars because of their small sizes, non-rigid structures, and limited training samples.The paper notes that accuracy for these categories fluctuates more than car detection because training samples are insufficient.Source: §A.2 · p. 13, Table 10 · p. 14

A.3 DEPTH ERROR ANALYSIS

The depth-error analysis compares the baseline and full model across depth for valid car samples. The authors report improved depth-estimation accuracy at different distances.

A.3 DEPTH ERROR ANALYSIS

  • The full model improves depth-estimation accuracy at different distances compared with the baseline.Source: §A.3 · p. 14, Figure 6 · p. 14
  • The analysis uses valid Car-category samples projected into the image plane and fits depth errors against ground truth with least squares.Source: §A.3 · p. 14
  • Figure 6 plots depth on the x-axis, with the baseline on the left and the full model on the right.Source: Figure 6 · p. 14

A.4 THE EFFECTS OF STEREO DEPTH

Stereo-depth guidance achieves almost the same accuracy as LiDAR guidance at 0.5 IoU, with a small performance drop at 0.7 IoU. The passage attributes this to stereo depth estimation producing more accurate absolute depth than monocular estimation.

A.4 THE EFFECTS OF STEREO DEPTH

  • At 0.5 IoU, stereo-depth guidance achieves almost the same accuracy as LiDAR guidance, with only a small performance drop at 0.7 IoU.The comparison reports performance under stereo-depth and LiDAR guidance.Source: §A.4 · p. 14
  • Stereo depth estimation uses geometric constraints and pixel-wise matching to obtain absolute depth, which is more accurate than monocular depth estimation.Source: §A.4 · p. 14

A.5 GENERALIZATION OF THE PROPOSED METHOD

The appendix reports that MonoDistill benefits another monocular detector and remains effective with lower-resolution LiDAR signals. It also compares knowledge distillation with dense depth supervision and presents qualitative KITTI results.

A.5 GENERALIZATION OF THE PROPOSED METHOD

  • Applying MonoDistill to the two-stage monocular detector GUPNet also boosts its performance.The experiments are conducted on the KITTI validation set.Source: §A.5 · pp. 14–15, Table 12 · p. 15
  • MonoDistill significantly boosts the baseline under all tested LiDAR resolutions, although the improvement is slightly reduced as resolution decreases.The sparse-signal experiments use simulated 32-beam and 16-beam LiDAR signals to train the teacher, with MonoDLE as the baseline.Source: §A.5 · p. 15, Table 13 · p. 15
  • KITTI test performance suggests generalization across camera parameters, while generalization to new scenes with different statistical characteristics remains a challenge for existing 3D detectors.Source: §A.5 · p. 15

A.6 COMPARISON WITH DIRECT DENSE DEPTH SUPERVISION.

  • Dense depth supervision improves the baseline, but the proposed KD-based method performs significantly better than both the baseline and the dense-supervision control.The comparison is conducted on the KITTI validation set.Source: §A.6 · pp. 15–16, Table 14 · p. 16

A.7 MORE QUALITATIVE RESULTS

  • On KITTI, qualitative 3D comparisons show improved depth-estimation accuracy over the baseline; Figure 8 uses red for ground truth, green for baseline, and blue for MonoDistill.Source: §A.7 · p. 16, Figure 8 · p. 17

Related papers

Loading 2201.10830v1…