PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Semantic Video Segmentation with Per-frame Inference

Yifan Liu, Chunhua Shen, Changqian Yu, Jingdong Wang

arXiv:2002.11433v2cs.CV

TL;DR

Video segmentation needs accurate, temporally consistent predictions without the latency and computational burden of multi-frame inference. The paper trains compact networks with temporal loss and temporal-consistency distillation while processing frames independently, and reports better accuracy–speed trade-offs than keyframe-based methods on Cityscapes and Camvid.

  • Problem

    Existing independently trained per-frame models can be temporally inconsistent, while multi-frame and keyframe methods add computational cost, latency imbalance, or propagation errors.

  • Method

    The paper combines compact per-frame segmentation networks with a motion-guided temporal loss and temporal-consistency knowledge distillation during training.

  • Results

    The proposed compact models outperform previous semantic video segmentation methods in the trade-off among accuracy, temporal consistency, and inference speed on Cityscapes and Camvid.

  • Takeaways & Limitations

    Temporal consistency can be embedded during training so compact models retain independent per-frame inference without extra computation or post-processing.

  • Takeaways & Limitations

    Keyframe-based and multi-frame alternatives remain constrained by propagation errors, unbalanced latency, or computational cost, while video training also involves limited labeled frames.

Abstract

from arXiv · show

For semantic segmentation, most existing real-time deep models trained with each frame independently may produce inconsistent results for a video sequence. Advanced methods take into considerations the correlations in the video sequence, e.g., by propagating the results to the neighboring frames using optical flow, or extracting the frame representations with other frames, which may lead to inaccurate results or unbalanced latency. In this work, we process efficient semantic video segmentation in a per-frame fashion during the inference process. Different from previous per-frame models, we explicitly consider the temporal consistency among frames as extra constraints during the training process and embed the temporal consistency into the segmentation network. Therefore, in the inference process, we can process each frame independently with no latency, and improve the temporal consistency with no extra computational cost and post-processing. We employ compact models for real-time execution. To narrow the performance gap between compact models and large models, new knowledge distillation methods are designed. Our results outperform previous keyframe based methods with a better trade-off between the accuracy and the inference speed on popular benchmarks, including the Cityscapes and Camvid. The temporal consistency is also improved compared with corresponding baselines which are trained with each frame independently. Code is available at: https://tinyurl.com/segment-video

1 Introduction

Semantic video segmentation must balance accuracy, temporal consistency, and real-time efficiency. The paper addresses this by training compact per-frame models with temporal constraints and distillation, avoiding inference-time propagation and extra computation.

  • The method targets fast and accurate video segmentation for applications such as autonomous driving and robotics, where video processing is especially costly.The paper frames real-time deployment as a central challenge for video semantic segmentation.
  • Keyframe propagation accelerates inference but creates unbalanced latency and can accumulate warping errors that harm non-key-frame accuracy.These methods use optical flow to propagate outputs or feature maps from keyframes.
  • Independent frame inference can enable real-time execution, but independently trained models often produce temporally inconsistent video predictions.The proposed approach keeps per-frame inference while addressing consistency during training.
  • A motion-guided temporal loss propagates predicted labels between frames to impose consistent labeling along the time axis.A motion estimation network predicts pixel motion between consecutive input frames and supplies supervision for the next-frame prediction.
  • Temporal-consistency knowledge distillation transfers pairwise and multi-frame dependencies from large teacher models to compact networks.Pairwise similarities and ConvLSTM-based embeddings encode temporal information during training.
  • Compact per-frame models with temporal training outperform prior methods across accuracy, temporal consistency, and inference speed on Cityscapes and Camvid.The reported comparison weighs all three objectives rather than optimizing inference speed alone.

2 Approach

The approach embeds temporal consistency into compact per-frame segmentation models during training through motion-guided loss and temporal knowledge distillation. After training, auxiliary temporal modules are removed, enabling independent inference without extra computational cost or post-processing.

  • Motion Guided Temporal Consistency: Temporal loss constrains moving pixels to receive consistent semantic labels across frames, using motion estimation and warped predictions as supervision.An occlusion mask removes warping noise, and the resulting constraint is applied during training.
  • Temporal Consistency Knowledge Distillation: Temporal consistency distillation transfers relations from a teacher to a compact student through pair-wise frame dependency and multi-frame dependency.Pair-wise dependency captures motion correlations between frame predictions, while multi-frame dependency encodes correlations in intermediate feature maps with ConvLSTM.
  • Optimization: The training objective combines cross-entropy, single-frame distillation, temporal loss, and temporal-consistency distillation terms.The teacher is pretrained with segmentation and temporal losses, while the motion-estimation and teacher networks remain fixed during student optimization.
  • Inference: After training, only the student segmentation network remains, so temporal consistency and segmentation accuracy improve without extra inference parameters or post-processing.The motion-estimation network, teacher, and distillation modules are removed before per-frame inference.

3 Implementation details

The experiments use compact segmentation networks with separate training-time motion and teacher modules, evaluate accuracy, efficiency, and temporal stability, and report complementary training components on Cityscapes.

  • Network structures: Experiments use PSPNet18 with ResNet18 as the main compact model, alongside MobileNetV2 and light-weight HRNet-w18 for generalization.A PSPNet101 with ResNet101 supplies soft distillation targets, while pretrained FlowNetV2 estimates motion during training.
  • Training policy: Training samples randomly selected frames before and after a labeled frame to form triplets that capture both short-term and long-term temporal correlations.
  • Evaluation: Evaluation reports mIoU, pixel accuracy, parameter count, frames per second, and mean flow warping error adapted to semantic segmentation for temporal stability.

4 Experiments

Experiments show that temporal-loss and distillation training improve compact segmentation models, accuracy, and temporal consistency without added inference cost. Across Cityscapes and CamVid comparisons, per-frame models offer strong accuracy–speed trade-offs and more stable predictions than keyframe-based methods.

  • Effectiveness of proposed methods: 73.06% mIoU and 70.56% temporal consistency are achieved by combining temporal loss with distillation terms without increasing parameters or per-frame inference cost.The combined scheme treats the temporal loss and distillation terms as training-time regularizers.
  • Impact of the teacher net: The temporal-loss-enhanced teacher improves temporal consistency by 1.97% and mIoU by 0.69%, while distillation transfers temporal consistency to the student.Using the enhanced teacher changes student temporal consistency from 69.27 to 70.10, with comparable mIoU of 70.26 versus 70.32.
  • Qualitative visualization: Keyframe-based Accel produces severe inter-frame jitters and uneven quality away from keyframes, whereas the proposed method gives more stable frame-by-frame segmentation.The qualitative results show worse keyframe-method predictions at frames such as k + 3, while the proposed outputs remain consistent.
  • Comparison with single-frame based methods: Both segmentation accuracy and temporal consistency improve when temporal information is added to compact PSPNet18, HRNet-w18, and MobileNetV2 backbones.Table 4 reports mIoU, temporal consistency, inference speed, and parameter counts for these models.
  • Additional experiments on CamVid: 78.2 versus 75.2 segmentation results are reported on CamVid, while the proposed method reaches 78 fps compared with VideoGCRF's 22 fps.The CamVid comparison uses MobileNetV2 and pretrained Cityscapes weights.

5 Conclusions

The work combines compact per-frame inference with temporal consistency constraints during training, including temporal distillation across frames. Its components improve semantic accuracy and temporal consistency without extra inference computation or latency.

  • 5 Conclusions: Per-frame inference avoids latency and post-processing while the proposed compact networks improve temporal consistency and semantic accuracy without extra inference cost.The conclusion attributes these gains to temporal loss and temporal-consistency knowledge distillation.
  • 5 Conclusions: Pixel-wise distillation uses the cumbersome model’s class probabilities as soft targets, while pair-wise distillation transfers self-similarity structure.The pair-wise loss is built on a self-similarity map, complementing single-frame pixel-wise supervision.
  • 5 Conclusions: The multi-frame distillation aligns the final ConvLSTM memory state of teacher and student networks to transfer temporal consistency.The memory state is treated as a distillation item containing multi-frame dependency.

B.1 Dataset

The evaluation uses Cityscapes urban-scene snippets and CamVid automotive videos with different annotation densities and train/test splits.

  • B.1 Dataset: Cityscapes contains 30-frame street-scene snippets recorded at 17 frames per second, with pixel-level annotations for selected frames.The supplied passage identifies the dataset as an urban-scene understanding benchmark.
  • B.1 Dataset: CamVid contains five automotive videos, with ground-truth labels every 30 frames and separate training and test video sequences.Three training videos contain 367 frames, while two test videos contain 233 frames.

B.2 Training and inference.

Training uses mini-batch SGD with dataset-specific crops and standard augmentation, including random scaling and flipping on Cityscapes.

  • B.2 Training and inference: Cityscapes training runs for 200 epochs with mini-batch SGD, 769 × 769 crops, random scaling from 0.5 to 2.1, and random flipping.CamVid uses a 640 × 640 crop size.

B.3 Details of the evaluating temporal consistency

Temporal stability is evaluated using flow-warping agreement between consecutive segmentation maps, summarized with mean warp IoU over validation sequences.

  • B.3 Details of the evaluating temporal consistency: Temporal stability is measured from flow warping error between consecutive frames, using mIoU instead of mean square error.The metric evaluates semantic segmentation results rather than raw image differences.
  • B.3 Details of the evaluating temporal consistency: The evaluation compares each predicted segmentation map with the segmentation map warped from the preceding frame.Qt denotes the prediction at frame t, while the warped preceding prediction is compared at the same evaluation stage.
  • B.3 Details of the evaluating temporal consistency: Cityscapes temporal stability uses 100 validation sequences containing 3000 images, while CamVid uses the test sequence seq05.The reported statistic averages warp IoU across each sequence and then across the validation set.

C Description of videos and visualization results

The visualizations compare the proposed method with independently trained and keyframe-based baselines across validation, Cityscapes demo, and CamVid sequences. They show improved accuracy and temporal stability, while also revealing residual inconsistencies in some cases.

  • Video comparisons: The supplementary videos compare proposed and baseline predictions on validation and Cityscapes demo sequences, including consecutive-frame visualizations.In val.mp4, the proposed method appears left of the divider; in demo seq00.mp4, it appears above the divider.
  • Video comparisons: The proposed method produces stable and smooth sequences with high accuracy, outperforming baseline, Accel, and SKD comparisons in the visualizations.The comparison identifies jitters in Accel and inconsistent results from GAN-based distillation, while noting that both the proposed method and baseline can still fail in some situations.
  • CamVid visualization: On CamVid, the proposed model outperforms the baseline in the highlighted region and keeps predictions similar on still content.The comparison contrasts MobileNet trained with cross-entropy against MobileNet trained with temporal loss and distillation items.
  • Video comparisons: The Baseline SKD Accel Ours video places each method in a separate quadrant and exposes jitters between keyframe and normal-frame results for Accel.The figure layout is baseline top left, SKD top right, Accel bottom left, and the proposed method bottom right.

D Results on each class

Class-wise Cityscapes results compare PSPNet18 trained independently with the proposed temporal loss and distillation training. Improvements are strongest for moving objects, while several static classes mainly gain temporal consistency.

  • Cityscapes class-wise results: For train and bus, the proposed method significantly improves both segmentation accuracy and temporal consistency over the independently trained PSPNet18 baseline.These are moving objects with regular structures.
  • Cityscapes class-wise results: For road, sidewalk, and terrain, temporal consistency improves while accuracy shows only limited improvement.The table reports accuracy as mIoU and temporal consistency as TC for each class.
Loading 2002.11433v2…