PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Panoptic Segmentation

Dahun Kim, Sanghyun Woo, Joon-Young Lee, In So Kweon

arXiv:2006.11339v1cs.CV

TL;DR

Video panoptic segmentation extends image panoptic segmentation to require temporally consistent panoptic masks and instance identities across frames. The paper introduces datasets, VPSNet, and VPQ, reporting state-of-the-art image PQ and stronger VPQ performance against baselines while identifying temporal-length and stuff-class challenges.

  • Problem

    Video panoptic segmentation lacked suitable datasets and evaluation metrics despite requiring temporally consistent panoptic segmentation and instance-id associations across frames.

  • Method

    The paper creates VIPER- and Cityscapes-based VPS datasets, adds temporal feature fusion and object tracking in VPSNet, and introduces VPQ for tube-level evaluation.

  • Results

    The paper reports state-of-the-art image PQ on Cityscapes and VIPER and higher VPQ than strong and simple baseline methods.

  • Takeaways & Limitations

    The datasets, baseline network, and VPQ metric establish initial resources for developing and evaluating video panoptic segmentation.

  • Takeaways & Limitations

    Performance drops as temporal length increases, and the task remains challenging for stuff classes even with a 15-frame window.

Abstract

from arXiv · show

Panoptic segmentation has become a new standard of visual recognition task by unifying previous semantic segmentation and instance segmentation tasks in concert. In this paper, we propose and explore a new video extension of this task, called video panoptic segmentation. The task requires generating consistent panoptic segmentation as well as an association of instance ids across video frames. To invigorate research on this new task, we present two types of video panoptic datasets. The first is a re-organization of the synthetic VIPER dataset into the video panoptic format to exploit its large-scale pixel annotations. The second is a temporal extension on the Cityscapes val. set, by providing new video panoptic annotations (Cityscapes-VPS). Moreover, we propose a novel video panoptic segmentation network (VPSNet) which jointly predicts object classes, bounding boxes, masks, instance id tracking, and semantic segmentation in video frames. To provide appropriate metrics for this task, we propose a video panoptic quality (VPQ) metric and evaluate our method and several other baselines. Experimental results demonstrate the effectiveness of the presented two datasets. We achieve state-of-the-art results in image PQ on Cityscapes and also in VPQ on Cityscapes-VPS and VIPER datasets. The datasets and code are made publicly available.

1. Introduction

The paper extends panoptic segmentation from images to videos, introducing video panoptic segmentation, datasets, VPSNet, and the VPQ metric to support and evaluate this task.

  • Video panoptic segmentation jointly predicts classes, boxes, masks, instance-id associations, and semantic segmentation while assigning a unique answer to every video pixel.
  • The task addresses a video-domain gap associated with missing datasets and evaluation metrics for direct VPS training and assessment.
  • VPSNet adds temporal feature fusion and object tracking to an image panoptic segmentation network, correlating target and reference frames at pixel and object levels.
  • VPQ evaluates spatial-temporal consistency by measuring tube-level overlap across spans of video frames.
  • The method achieves state-of-the-art image PQ on Cityscapes and VIPER and outperforms strong and simple baselines in VPQ.
  • The paper presents VIPER and Cityscapes-based VPS datasets, which are described as complementary for constructing accurate VPS models.

2. Related Work

Related work covers panoptic, video semantic, and video instance segmentation, distinguishing VPS by requiring both semantic and instance-level temporal consistency.

  • Panoptic Segmentation: Panoptic segmentation unifies semantic and instance segmentation into a joint thing-and-stuff recognition task and has become a standard visual recognition benchmark.
  • Video Semantic Segmentation: Video semantic segmentation predicts semantic classes for all video pixels but does not require instance discrimination or explicit cross-frame tracking.
  • Video Semantic Segmentation: Cityscapes-VPS is a superset of a video semantic segmentation dataset and can therefore benefit that independent field.
  • Video Instance Segmentation: Video instance segmentation combines detection, segmentation, and instance tracking in videos, exemplified by Mask-Track R-CNN.

3. Problem Definition

Video panoptic segmentation groups frame-level segments into temporally associated tubes and evaluates them with VPQ across sliding temporal windows. Larger windows make matching more difficult, while k=0 recovers image PQ.

  • Task Format: A video tube links frame-level segments across a k-frame snippet using semantic class and instance id.Thing instances receive positive ids, whereas stuff classes use id 0.
  • Evaluation Metric: VPQ_k slides a temporal window through each video, averages snippet-level scores over frames, and then averages across selected window sizes.The final score aggregates VPQ_k values across temporal settings.
  • Evaluation Metric: VPQ matches predicted and ground-truth tubes by IoU, counting matches above 0.5 as true positives and defining false positives and negatives accordingly.With annotations every λ frames, matching considers only the annotated indices within each snippet.
  • Temporal Difficulty: k=0 makes VPQ equivalent to image PQ, whereas longer windows require consistent semantic and instance labels across frames.The evaluation uses k ∈ {0, 5, 10, 15}; larger windows are more challenging.
  • Temporal Difficulty: For Cityscapes-VPS with λ=5, many object associations disconnect when k>15, so k should be reset when annotation frequency changes.The authors describe this as setting the dataset’s difficulty level.

4. Dataset Collection

The paper constructs two video panoptic datasets to address the lack of suitable data and support transfer between image and video panoptic segmentation. They reformat VIPER and add temporal panoptic annotations to Cityscapes.

  • Motivation: Existing image panoptic benchmarks do not provide the video annotations required for training and evaluating video panoptic segmentation.The paper emphasizes high annotation cost and seeks datasets with both quality and quantity.
  • Dataset Construction: The authors present two datasets: a reformatted virtual VIPER dataset and new video panoptic annotations based on Cityscapes.Both datasets are designed to be adaptable to existing image-based panoptic benchmarks.
  • Dataset Statistics: Table 1 compares high-level statistics of reformatted VIPER and Cityscapes-VPS with prior video instance and semantic segmentation datasets.The comparison includes YouTube-VIS and Cityscapes-related datasets.
  • Cityscapes-VPS: Cityscapes-VPS extends the Cityscapes validation benchmark with video panoptic labels and serves as a benchmark for video panoptic, instance, and semantic segmentation.The dataset supplements the original Cityscapes benchmark.

5. Proposed Method

VPSNet extends an image panoptic segmentation baseline with temporal fusion and object tracking, jointly handling detection, masks, tracking, and semantic segmentation. Its design uses video context at pixel and object levels.

  • Network Design: VPSNet performs object detection, mask prediction, tracking, and semantic segmentation simultaneously on video sequences.The network is designed to exploit temporal and motion context for panoptic prediction.
  • Network Design: The architecture uses a pixel-level Fuse module for temporal feature fusion and an object-level Track module for cross-frame instance association.These modules jointly address temporal inconsistency in class labels and instance ids.
  • Implementation: The method builds on UPSNet and uses MaskTrack for tracking, with ResNet-50 FPN as the feature extractor and standard panoptic-training settings.Training uses ground-truth reference-frame boxes and initializes Fuse and Track layers separately.
  • Pixel-Level Fusion: The Fuse pipeline aligns reference features to the target with learned flow warping, then applies spatial-temporal attention before downstream instance and semantic branches.Initial optical flow is refined for deep-feature alignment.
  • Object-Level Tracking: The Track branch enhances RoI features with temporally fused features and uses similarity-based association, with an additional panoptic-head IoU cue during inference.The added IoU cue improves video panoptic quality for thing classes according to the authors.

6. Experimental Results

Experiments evaluate VPSNet and variants on VIPER, Cityscapes-VPS, and image-level Cityscapes using image-quality and video-quality measures. The results show complementary gains from pixel-level fusion and object-level tracking, while longer temporal windows and crowded-object overlaps remain challenging.

  • Experimental Setup: Experiments cover VIPER, Cityscapes-VPS, and image-level Cityscapes, assessing per-frame prediction and cross-frame association with IPQ and VPQ.The video datasets evaluate temporal consistency, while Cityscapes evaluates image-level panoptic quality.
  • Experimental Setup: Cityscapes-VPS contains 500 videos split into 400/50/50 train, validation, and test videos, with 30 predicted frames and six evaluated annotated frames per video.VIPER evaluation uses 10 validation videos and the first 60 frames of each, totaling 600 images.
  • Image-Level Results: Each Fuse and Track module improves the VIPER image-level baseline by +1% PQ, while removing either module causes a -3.4% PQ drop.The best PQ is achieved when both modules are used together.
  • Image-Level Results: +1.0% PQ over the state-of-the-art baseline is achieved by VPSNet with Fuse on Cityscapes, reaching 62.2% PQ after VIPER pretraining and 79.0% mIoU.VIPER pretraining adds +1.6% PQ over the corresponding baseline.
  • Video-Level Results: +9.7% VPQ is obtained by IoU-Match over the image-level baseline, while Track adds +1.2% VPQ using learned RoI matching with semantic and spatial cues.Cls-Sort struggles when multiple instances share the same class.
  • Video-Level Results: +17.0% VPQ over the image-level base model is achieved by full FuseTrack, which also exceeds the only-Track variant by +6.1% VPQ and the Cityscapes-VPS Track variant by +1.1% VPQ.Table 4 reports VPQ, VPQTh, and VPQSt on VIPER; Table 5 reports the same metrics on Cityscapes-VPS validation and test sets.

7. Conclusion

The paper defines video panoptic segmentation, introduces datasets and VPSNet, and proposes VPQ for evaluation. It positions these contributions as early steps toward video understanding in real-world settings.

  • Contributions: Video panoptic segmentation extends panoptic segmentation to video with consistent pixel-level outputs and instance identities across frames.The task combines temporal feature fusion and object tracking with single-frame panoptic segmentation.
  • Datasets: The paper adapts synthetic VIPER and extends Cityscapes with Cityscapes-VPS to provide two video panoptic datasets.VIPER emphasizes annotation quantity and quality, while Cityscapes-VPS extends a popular image-level benchmark.
  • Method: VPSNet combines temporal feature fusion and an object-tracking branch with a single-frame panoptic segmentation network.The method is presented as a new approach for the proposed task.
  • Evaluation: The proposed video panoptic quality measure evaluates the new task by accounting for temporal consistency.VPQ provides an evaluation basis for early exploration of video panoptic segmentation.
Loading 2006.11339v1…