PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Pseudo-LiDAR Representation

Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, Wanli Ouyang

arXiv:2008.04582v1cs.CV

TL;DR

Pseudo-LiDAR improves monocular and stereo 3D detection, but the paper investigates whether its benefit comes from the representation itself or from coordinate transformation. The authors introduce PatchNet, an image-based CNN detector that organizes transformed 3D data as an image and reports stronger KITTI performance than pseudo-LiDAR counterparts, while noting failure cases involving occlusion and truncation.

  • Problem

    Pseudo-LiDAR methods improve KITTI 3D detection, but the mechanism behind their benefit and the importance of their data representation remain unclear.

  • Method

    PatchNet compares image and pseudo-LiDAR representations while integrating 3D coordinates produced by projecting image coordinates into world or LiDAR coordinates.

  • Results

    PatchNet-vanilla achieves almost the same accuracy as pseudo-LiDAR, while PatchNet outperforms other pseudo-LiDAR-based detectors on the KITTI dataset.

  • Takeaways & Limitations

    Coordinate system transformation, rather than point-cloud representation itself, is the key factor, and image representations can use mature 2D CNN technologies for competitive or superior detection.

  • Takeaways & Limitations

    The method often misestimates headings for truncated or occluded objects, and missed 2D detections can cause samples to be ignored.

Abstract

from arXiv · show

The recently proposed pseudo-LiDAR based 3D detectors greatly improve the benchmark of monocular/stereo 3D detection task. However, the underlying mechanism remains obscure to the research community. In this paper, we perform an in-depth investigation and observe that the efficacy of pseudo-LiDAR representation comes from the coordinate transformation, instead of data representation itself. Based on this observation, we design an image based CNN detector named Patch-Net, which is more generalized and can be instantiated as pseudo-LiDAR based 3D detectors. Moreover, the pseudo-LiDAR data in our PatchNet is organized as the image representation, which means existing 2D CNN designs can be easily utilized for extracting deep features from input data and boosting 3D detection performance. We conduct extensive experiments on the challenging KITTI dataset, where the proposed PatchNet outperforms all existing pseudo-LiDAR based counterparts. Code has been made available at: https://github.com/xinzhuma/patchnet.

1 Introduction

The paper investigates why pseudo-LiDAR improves monocular and stereo 3D detection, finding that coordinate transformation—not data representation—is the essential factor. It introduces PatchNet, an image-based detector that uses transformed 3D information with CNNs and achieves strong KITTI performance.

  • LiDAR-based 3D detection is accurate but depends on expensive sensors, while RGB-only methods lag because image depth is unobserved.
  • Estimated depth maps help 3D scene understanding, but their performance improvement remains limited when used directly to augment image inputs.
  • Pseudo-LiDAR methods substantially improve KITTI detection by transforming estimated depth maps and applying LiDAR-based detectors, but the underlying reason was unclear.
  • PatchNet-vanilla matches pseudo-LiDAR performance despite using image representation, showing that data representation has no substantial impact on 3D detection accuracy.
  • Ablations identify transformation from image coordinates to LiDAR coordinates, which encodes camera calibration information, as the key performance factor.
  • PatchNet extends the image-based approach with stronger CNN backbones and outperforms pseudo-LiDAR counterparts on KITTI while supporting end-to-end detection.

2 Related Work

Prior image-based detectors address 3D localization using geometric, structural, or depth-related cues, while pseudo-LiDAR methods estimate depth, transform it using calibration, and process the result as a point cloud. PatchNet instead retains an image-organized representation for subsequent processing.

  • Early image-based 3D detectors follow 2D detection paradigms but address difficult absolute 3D localization using geometric and object-structure cues.
  • Recent methods improve monocular 3D detection by estimating depth from RGB images and transforming it into pseudo-LiDAR using camera calibration.
  • PatchNet shares the intermediate-task generation and coordinate projection of pseudo-LiDAR methods but organizes the resulting data as an image representation instead of LiDAR signals.
  • Pseudo-LiDAR methods process transformed data with LiDAR-oriented detectors, while related approaches add RGB fusion, background segmentation, or 2D–3D consistency constraints.

3 Delving into pseudo-LiDAR representation

The section compares pseudo-LiDAR processing with an image-based equivalent and finds that coordinate transformation, rather than input representation, drives the performance gain.

  • Pseudo-LiDAR pipeline: Pseudo-LiDAR detectors estimate depth, generate 2D proposals, convert RoIs into 3D coordinates, and process them as point clouds.The generated coordinates use camera calibration, while point-wise CNNs process the resulting unordered point sets.
  • PatchNet-vanilla: PatchNet-vanilla shares pseudo-LiDAR’s depth estimation, proposals, and generated 3D data, differing only in Step 4 input processing.It organizes each pixel’s (x, y, z) coordinates as image data and applies 2D CNNs instead of point-wise CNNs.
  • Preliminary conclusion: PatchNet-vanilla achieves almost the same accuracy as pseudo-LiDAR, indicating that input representation has no substantial impact on 3D detection tasks.The comparison uses reproduced pseudo-LiDAR results to reduce the influence of implementation details.
  • Implications: Image-based representation can use established 2D CNN technologies to develop high-performance 3D detectors.The PatchNet framework extends this direction beyond the vanilla equivalent implementation.

4 PatchNet

PatchNet extracts spatially informed image features, focuses pooling on foreground regions, and routes examples to prediction heads according to difficulty.

  • Input and spatial recovery: PatchNet obtains position and depth information from CNNs, crops each detected 2D proposal from the depth map, and recovers its spatial information.The recovered RoI data is then processed by a backbone network for 3D detection.
  • Backbone: The backbone uses an SE-ResNet-18 without pooling layers so output features retain the input patch resolution.Mask global pooling then extracts features corresponding to foreground objects.
  • Foreground feature extraction: Mask global pooling selects foreground features before global pooling using a binary mask with the same resolution as the input image.The mask is formed by thresholding each patch’s depth map at an offset from its mean depth.
  • Difficulty assignment: Three parallel branches handle different difficulty levels, selected by an additional module that predicts each instance’s difficulty.All branches share the same architecture but learn different parameters.
  • Loss function: The loss combines center, size, heading, and corner terms, with λ weighting the corner loss.The corner term is intended to alleviate potential sub-optimal solutions.

5 Experiments

Experiments on KITTI evaluate PatchNet through metric protocols, representation and input-channel ablations, backbone comparisons, pooling and assignment studies, benchmark results, and qualitative failure analysis. The results support coordinate transformation and image-based feature extraction as central design factors, while revealing limitations on difficult or occluded cases.

  • Evaluation protocol: KITTI evaluation uses easy, moderate, and hard subsets, with AP|R11 and AP|R40 reported for comprehensive comparison.The validation split contains 3,769 images after dividing the 7,481-image training set into 3,712 training and 3,769 validation images.
  • Representation analysis: PatchNet-vanilla achieves comparable performance to pseudo-LiDAR, and matched performance again when adapting the detector based on AM3D.These comparisons indicate that input data representation is not the key factor in the observed performance improvement.
  • Input ablation: Coordinate transformation is more important than depth alone: depth-only input nearly fails, while adding spatial coordinates greatly improves predicted 3D-box accuracy.World coordinates (x, y) using camera information outperform image coordinates (u, v), supporting coordinate-system transformation as the decisive factor.
  • Backbone comparison: 2D convolution backbones improve 3D-box accuracy, especially on hard examples, because point-wise CNNs struggle to extract local features efficiently.The authors associate hard cases with occlusion, truncation, or distance, where pose estimation depends more on contextual information.
  • State-of-the-art comparison: PatchNet ranks first among published KITTI methods by moderate performance and exceeds AM3D under hard settings by 0.65/1.56/2.34 on three listed metrics.It also outperforms existing pseudo-LiDAR approaches, but trails AM3D on the easy testing split.
  • Qualitative results: For distant objects, size and heading estimates remain accurate, whereas center localization is difficult; truncated or occluded objects often produce heading errors.Strong occlusion can also cause the 2D detector to miss objects, causing those samples to be ignored in later stages.

6 Conclusions

PatchNet argues that pseudo-LiDAR gains primarily arise from projecting image coordinates into world coordinates, not from point-cloud representation itself. This coordinate representation can be integrated with image-based detection and 2D CNN technologies.

  • PatchNet identifies camera-parameter-based projection from image coordinates to world coordinates as the key factor behind pseudo-LiDAR detectors’ performance.
  • World-coordinate information can be integrated into image representation, enabling more flexible use of mature 2D CNN technologies for 3D detection.
  • Experiments on KITTI support the proposed explanation and demonstrate the potential of image-representation-based 3D detectors.

A Overview

The supplementary document adds latency analysis, stereo-image experiments, pedestrian and cyclist results, and visualization examples to the main paper.

  • The document provides additional analysis and experiments beyond the main paper.These include latency comparisons, stereo-image performance, pedestrian and cyclist detection, and visualization examples.
  • The supplementary sections cover latency, stereo images, pedestrian and cyclist detection, and qualitative visualizations.

B Runtime Analysis

PatchNet-vanilla and pseudo-LiDAR share the same processing flow but differ in input representation, yielding nearly identical runtime. PatchNet is more efficient than two compared methods and has runtime similar to pseudo-LiDAR.

  • PatchNet-vanilla and pseudo-LiDAR use the same processing stages, with different input representations, so their runtimes are almost the same.The comparison was tested on a single 1080 GPU.
  • PatchNet shares the same 2D detector and depth estimator while reporting runtime for different 3D-detection backbone models.
  • 12ms for PointNet-18 versus 28ms for PatchNet-vanilla; removing foreground segmentation and using dynamic thresholding saves about 18ms.The best PatchNet backbone runs in 26ms, with runtime similar to pseudo-LiDAR for 3D detection.
  • Method adds a ResNet-34 for image features, while adds a 2D instance-segmentation network that increases computational overhead.The passage gives about 200ms as an example overhead for Mask RCNN.
  • PatchNet is more efficient than and has runtime similar to.

C Stereo Images

The stereo experiment replaces monocular depth maps with stereo depth maps to test PatchNet on binocular images using KITTI validation data. PatchNet-vanilla achieves almost the same accuracy as pseudo-LiDAR.

  • PatchNet is evaluated with stereo depth maps from PSMNet on the KITTI validation set to test its applicability to binocular images.Performance is reported using AP|R11 for comparison with previous works.
  • PatchNet-vanilla achieves almost the same stereo 3D detection accuracy as pseudo-LiDAR.
  • Table 10 reports stereo 3D detection performance for the Car category on KITTI validation with a 0.7 IoU threshold.Best results are highlighted in bold.

D Pedestrian and Cyclist

PatchNet is evaluated for Pedestrian/Cyclist 3D detection on KITTI using monocular and stereo images, with results reported under AP|R11. It outperforms pseudo-LiDAR, while performance varies substantially because these categories have fewer training samples than Cars.

  • PatchNet achieves better Pedestrian/Cyclist detection performance than pseudo-LiDAR in both monocular and stereo settings.The comparison uses KITTI validation results and pseudo-LiDAR was evaluated using its official code.
  • Pedestrian/Cyclist accuracy fluctuates more than Car accuracy because the training set contains only 2,207/734 samples versus 14,357 Car instances.The paper suggests more training data or stronger augmentation as possible ways to reduce this issue.
  • The evaluation reports AP|R11 with an IoU threshold of 0.5 on the KITTI validation dataset.

E More Qualitative Examples

Qualitative KITTI validation results compare monocular and stereo PatchNet detections. Stereo inputs generally improve object detection through better depth estimation, while monocular results remain competitive for many nearby objects.

  • Stereo images detect objects more accurately than monocular images, primarily through better depth estimation rather than improved size or heading estimation.
  • For most close-range objects, monocular images are visually not inferior to stereo images, despite some remaining failure cases.
  • Figure 6 shows monocular results on the left and stereo results on the right, with predictions in red and ground truth in green.LiDAR signals are used only for visualization.
Loading 2008.04582v1…