PaperlayerView exact revision on arXiv

Source-linked AI summary

ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

Siyuan Qiao, Yukun Zhu, Hartwig Adam, Alan Yuille, Liang-Chieh Chen

arXiv:2012.05258v1cs.CV

TL;DR

ViP-DeepLab addresses the inverse projection problem by restoring semantically interpreted 3D point clouds from perspective image sequences. It jointly performs monocular depth estimation and video panoptic segmentation through the DVPS task, metric, and datasets. The model achieves state-of-the-art results across the reported sub-tasks, including a 5.1% VPQ improvement on Cityscapes-VPS and first place on KITTI depth estimation and KITTI MOTS pedestrian.

  • Problem

    The paper targets the ambiguous inverse projection problem of recovering 3D environmental information from limited 2D image signals.

  • Method

    ViP-DeepLab jointly performs monocular depth estimation and video panoptic segmentation as Depth-aware Video Panoptic Segmentation, evaluated with DVPQ on two derived datasets.

  • Results

    ViP-DeepLab achieves state-of-the-art performance across the reported sub-tasks, including a 5.1% VPQ improvement on Cityscapes-VPS and first place on KITTI depth estimation and KITTI MOTS pedestrian.

  • Takeaways & Limitations

    The paper provides a unified model, task, metric, and datasets for studying depth-aware video panoptic perception and related sub-tasks.

Abstract

from arXiv · show

In this paper, we present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision, which we model as restoring the point clouds from perspective image sequences while providing each point with instance-level semantic interpretations. Solving this problem requires the vision models to predict the spatial location, semantic class, and temporally consistent instance label for each 3D point. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. We name this joint task as Depth-aware Video Panoptic Segmentation, and propose a new evaluation metric along with two derived datasets for it, which will be made available to the public. On the individual sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The datasets and the evaluation codes are made publicly available.

1. Introduction

ViP-DeepLab frames inverse projection as jointly recovering 3D spatial location, semantic class, and temporally consistent instance labels from image sequences. It introduces DVPS, datasets, DVPQ, and a unified model that jointly performs video panoptic segmentation and monocular depth estimation.

  • Motivation: The inverse projection problem maps limited 2D image signals back to the 3D environment and its visual information.The paper targets machine perception of 3D scene layout from image observations.
  • Depth-aware Video Panoptic Segmentation: DVPS formulates inverse projection as joint video panoptic segmentation and monocular depth estimation.The task requires predicting spatial location, semantic class, and temporally consistent instance labels for 3D points.
  • Resources: Two derived datasets and the DVPQ metric support evaluation of the new task, with datasets and evaluation codes intended for public release.The datasets are produced by augmenting or converting existing datasets for DVPS.
  • Model: ViP-DeepLab jointly performs video panoptic segmentation and monocular depth estimation for each image-plane pixel.The model extends the Panoptic-DeepLab framework with a depth prediction head and a video tracking approach based on center regression.
  • Results: ViP-DeepLab ranks first on Cityscapes-VPS, KITTI MOTS pedestrian, and KITTI monocular depth estimation.On KITTI depth estimation, it outperforms DORN by 0.97 SILog without additional depth training data.

2. Related Work

Prior work covers image and video panoptic segmentation and monocular depth estimation. ViP-DeepLab relates to these approaches by using center regression to associate instance pixels across consecutive video frames.

  • Panoptic Segmentation: Image panoptic segmentation methods are grouped into top-down proposal-based and bottom-up box-free approaches.Top-down methods generate object proposals before producing panoptic predictions through regional computations.
  • Video Panoptic Segmentation: Video methods extend panoptic segmentation by grouping instance pixels across clips or learning spatio-temporal embeddings.ViP-DeepLab instead implicitly tracks objects by clustering instance pixels in two consecutive frames using center regression.
  • Monocular Depth Estimation: Monocular depth estimation predicts depth from a single image and can be learned with supervised, stereo, video, or relative-order training.

3. ViP-DeepLab

ViP-DeepLab extends Panoptic-DeepLab into a unified model for video panoptic segmentation and monocular depth estimation. Its design combines temporal instance handling, depth prediction, sequence stitching, and depth-aware evaluation.

  • ViP-DeepLab extends Panoptic-DeepLab to jointly perform video panoptic segmentation and monocular depth estimation.
  • Video Panoptic Segmentation: Video panoptic segmentation is formulated through concatenated images, connecting image panoptic segmentation with temporal prediction.This formulation motivates extending image panoptic segmentation models with temporal modifications.
  • Video Panoptic Segmentation: The next-frame instance branch predicts center offsets for pixels in frame t + 1 relative to object centers in frame t.During training, features from the two frames are concatenated before the next-frame instance branch.
  • Video Panoptic Segmentation: Sequence stitching propagates instance IDs between consecutive panoptic predictions using mutually largest mask IoU matches among regions with the same class.Objects without propagated IDs become new instances.
  • Monocular Depth Estimation: The depth prediction head performs dense monocular depth regression from decoded semantic-branch features, with MaxDepth set to 88 for KITTI’s approximately 0–80 m range.The depth loss combines scale-invariant logarithmic error and relative squared error.
  • Depth-aware Video Panoptic Segmentation: DVPS assigns semantic class, instance ID, and depth to each pixel, while DVPQ extends VPQ by filtering pixels according to an absolute relative depth-error threshold.VPQ and the depth inlier metric can be viewed as approximate special cases of DVPQ.

4. Datasets

The paper creates Cityscapes-DVPS and SemKITTI-DVPS by augmenting or converting existing datasets for joint depth-aware video panoptic segmentation. The conversion includes depth generation, 3D-point projection, and filtering of problematic points.

  • Dataset construction: Two datasets, Cityscapes-DVPS and SemKITTI-DVPS, are created for evaluating Depth-aware Video Panoptic Segmentation.Cityscapes-DVPS derives from Cityscapes-VPS, while SemKITTI-DVPS derives from SemanticKITTI.
  • Cityscapes-DVPS: Cityscapes-DVPS adds depth annotations to Cityscapes-VPS, whose original annotations cover six frames from each 30-frame video sequence.The depth annotations are converted from stereo-derived disparity maps and improved using modern disparity estimation methods.
  • SemKITTI-DVPS: SemKITTI-DVPS projects SemanticKITTI’s panoptic-labeled 3D point clouds into the image plane.The source point clouds provide semantic classes and instance IDs.
  • Point-cloud processing: A disparity consistency check removes projected points with large relative errors, including points recorded behind visible camera content.This filtering addresses points that are not visible from the camera despite being present in the labeled point cloud.
  • Point-cloud processing: Non-foreground suppression reduces far-away background points that invade projected regions of thin objects such as poles.The preprocessing removes problematic background points from small image patches around thin objects.

5. Experiments

Experiments evaluate ViP-DeepLab on the new DVPS task and on video panoptic segmentation, depth estimation, and multi-object tracking and segmentation. The reported results show strong temporal consistency and state-of-the-art performance across several benchmarks.

  • Depth-aware Video Panoptic Segmentation: ViP-DeepLab is evaluated on Cityscapes-DVPS and SemKITTI-DVPS using DVPQ, which combines depth and video panoptic segmentation evaluation.The evaluation varies the relative-depth threshold λ and video-clip length k.
  • Depth-aware Video Panoptic Segmentation: 3.2% is the DVPQ0.5 drop from k = 1 to k = 5 on SemKITTI-DVPS, compared with 7% from k = 1 to k = 2 on Cityscapes-DVPS.Smaller performance drops indicate higher temporal consistency; the paper relates the difference to annotation frame rates and offset prediction.
  • Video Panoptic Segmentation: 5.1% VPQ is the test-set improvement over VPSNet on Cityscapes-VPS, while the validation-set improvement is 5.6% VPQ.The results are reported for the Cityscapes-VPS validation and test sets.
  • Ablation study: Image panoptic pretraining mainly improves image PQ, while DenseContext narrows the gaps between VPQk values and therefore improves temporal consistency.The ablation compares Mapillary Vistas and Cityscapes pretraining with architectural changes in the next-frame instance branch.
  • Multi-object Tracking and Segmentation: 67.7% and 80.6% sMOTSA are achieved for pedestrians and cars, respectively, on the KITTI MOTS benchmark.The benchmark evaluates tracking and segmentation for pedestrians and cars.

6. Conclusion

The paper introduces DVPS, its DVPQ metric and two derived datasets, then presents ViP-DeepLab as a strong baseline. The model also achieves state-of-the-art performance on several related sub-tasks.

  • Contributions: DVPS combines monocular depth estimation and video panoptic segmentation as a step toward solving the inverse projection problem.The task is designed to recover 3D scene information with semantic and temporally consistent instance interpretations.
  • Contributions: DVPQ and two derived datasets are proposed for evaluating the new task, with ViP-DeepLab serving as a strong baseline.The conclusion presents these as the paper’s central task and benchmark contributions.
  • Results: ViP-DeepLab achieves state-of-the-art performance on monocular depth estimation, video panoptic segmentation, and multi-object tracking and segmentation.The conclusion summarizes performance across the paper’s related sub-tasks.

A. Stitching Algorithm

The stitching algorithm forms temporally consistent panoptic predictions by pairing adjacent images and aligning instance IDs between the two predictions.

  • Stitching procedure: The algorithm concatenates an image pair t and t + 1, then splits the panoptic prediction at the pair’s midpoint.The resulting left and right predictions are denoted Pt and Rt.
  • Stitching procedure: Pt represents image t, while Rt represents image t + 1 with instance IDs made consistent with Pt.This alignment supports temporally consistent predictions across the video sequence.
  • Associated evaluations: The surrounding evaluation materials include KITTI MOTS validation results and experiments with different training weights for Ldepth on Cityscapes-DVPS.These are identified in the accompanying table captions.

B. More Experiments

ViP-DeepLab is evaluated on KITTI MOTS and depth validation, with additional analysis of depth-loss weighting and official metric conventions.

  • ViP-DeepLab with a Kalman filter outperforms previous methods by a large margin on the KITTI MOTS validation set.
  • The depth-loss ablation varies Ldepth’s training weight from 1.0 to 10 and 0.1 to measure its effect on monocular depth performance.
  • Table 8 marks whether higher or lower values are better for each KITTI depth metric.
  • The official KITTI depth validation set contains 1,000 cropped images, and ViP-DeepLab outperforms previous methods by a large margin on all metrics.

C. Cascade-ASPP

Cascade-ASPP provides dense multi-scale context for the next-frame instance branch by cascading four densely connected ASPP modules, increasing its receptive field.

  • Cascade-ASPP is used as Dense Multi-scale Context in the next-frame instance branch.
  • It cascades four ASPP modules with their outputs densely connected.
  • Its motivation is to dramatically increase the receptive field of the next-frame instance branch.
  • Cascade-ASPP improves video panoptic segmentation performance on Cityscapes-VPS compared with a single ASPP variant.

D. More Visualizations

The visualizations cover Cityscapes-DVPS and SemKITTI-DVPS sequences, showing temporally consistent panoptic predictions, monocular depth, and reconstructed point clouds.

  • Cityscapes-DVPS visualizations cover four validation sequences, each containing six frames.
  • SemKITTI-DVPS visualizations show two 16-frame video clips from the validation sequence.
  • The visualizations report accurate video panoptic predictions and high-quality depth predictions for ViP-DeepLab.
  • Each visualization presents the input image, temporally consistent panoptic prediction, monocular depth prediction, and point cloud visualization from left to right.
Loading 2012.05258v1…