PaperlayerView exact revision on arXiv

Source-linked AI summary

Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Video

Samvit Jain, Xin Wang, Joseph Gonzalez

arXiv:1807.06667v4cs.CVcs.LG

TL;DR

Semantic video segmentation is expensive when each frame is processed independently, and feature warping can accumulate error under complex dynamics. Accel combines warped keyframe features with an adjustable current-frame update branch, achieving a broad accuracy-throughput range and outperforming comparable and prior systems.

  • Problem

    Independent frame processing is computationally expensive, while feature warping can accumulate error as video dynamics become complex.

  • Method

    Accel fuses high-detail features warped from a reference keyframe with adjustable-quality features computed on the current frame to correct temporal error.

  • Results

    Accel spans accuracy-inference-speed modalities, with versions that are faster and more accurate than comparable single-frame models and an ensemble that outperforms prior work across keyframe intervals.

  • Takeaways & Limitations

    The modular, end-to-end trainable architecture provides a general approach for efficient dense prediction on video across different accuracy-throughput requirements.

  • Takeaways & Limitations

    Feature warping remains constrained by fast scene evolution and moving observers, with error compounding under repeated warping.

Abstract

from arXiv · show

We present Accel, a novel semantic video segmentation system that achieves high accuracy at low inference cost by combining the predictions of two network branches: (1) a reference branch that extracts high-detail features on a reference keyframe, and warps these features forward using frame-to-frame optical flow estimates, and (2) an update branch that computes features of adjustable quality on the current frame, performing a temporal update at each video frame. The modularity of the update branch, where feature subnetworks of varying layer depth can be inserted (e.g. ResNet-18 to ResNet-101), enables operation over a new, state-of-the-art accuracy-throughput trade-off spectrum. Over this curve, Accel models achieve both higher accuracy and faster inference times than the closest comparable single-frame segmentation networks. In general, Accel significantly outperforms previous work on efficient semantic video segmentation, correcting warping-related error that compounds on datasets with complex dynamics. Accel is end-to-end trainable and highly modular: the reference network, the optical flow network, and the update network can each be selected independently, depending on application requirements, and then jointly fine-tuned. The result is a robust, general system for fast, high-accuracy semantic segmentation on video.

1. Introduction

Accel targets the high computational cost and limited temporal use of semantic video segmentation by combining warped keyframe features with current-frame updates. Its adjustable update branch spans accuracy-throughput trade-offs, with reported gains over comparable single-frame and prior video-segmentation systems.

  • Motivation: Semantic segmentation becomes especially expensive on high-resolution video because pixel-level predictions scale with the temporal frame rate.The cited context gives 30 frames per second as an example.
  • Motivation: Frame-by-frame approaches can run below 1.5 frames per second on 30 fps video while failing to exploit preceding frames for improved accuracy.These approaches treat video as uncorrelated still images and incur the full computational cost.
  • Limitations of prior approaches: Feature warping reduces computation by reusing keyframe representations, but fast scene changes and moving observers cause errors that compound across repeated warps.The approach relegates expensive feature extraction to selected keyframes and projects features forward using copying or optical flow.
  • Accel: Accel fuses a reference branch carrying incrementally warped keyframe features with an update branch processing the current frame.The update branch can correct warping error when shallow or anchor predictions with current-frame features when deep.
  • Accel: ResNet-18-to-ResNet-101 update networks let Accel operate across a broad accuracy-throughput spectrum.The paper reports both a faster and more accurate low-cost model than a comparable DeepLab model and a higher-accuracy model than DeepLab-101.

2. Related Work

Related work reduces video-segmentation cost through feature reuse, warping, adaptive propagation, partial updating, and keyframe selection. Accel instead adds a cheap per-frame branch that corrects accumulated temporal error without the auxiliary scheduling complexity of several adaptive schemes.

  • Image semantic segmentation: Fully convolutional, encoder-decoder, dilated-convolution, and CRF methods have improved semantic image segmentation on larger, more realistic datasets.These approaches form the image-segmentation background from which video methods develop.
  • Video semantic segmentation: Semantic video segmentation labels every pixel in a full frame and remains less studied than video object segmentation.Its relevance has grown with applications in autonomous control and video analysis.
  • Existing video methods: Video methods reduce computation through selective feature re-execution, optical-flow feature warping, or fixed-budget LSTM keyframe policies.The cited comparison identifies optical-flow warping as the strongest of these three contenders on cost savings and accuracy.
  • Adaptive methods: Adaptive propagation, partial updating, and adaptive keyframe selection require auxiliary representations to decide when and how features should be recomputed or propagated.These techniques also do not fundamentally address mounting warping error.
  • Accel: Accel uses a simple second branch that processes every frame cheaply and corrects accumulated temporal error in the reference representation.This design directly contrasts with schemes that optimize feature propagation or scheduling complexity.

3. Approach

Accel seeks faster video segmentation by combining warped keyframe features with current-frame processing, while preserving a modular interface for accuracy–throughput choices.

  • 3.2. Operation Model: A single-frame segmentation network is accurate but slow because it processes each video frame independently without temporal continuity.The target video network should retain accuracy while running faster than the single-frame baseline.
  • 3.2. Operation Model: Accel executes its expensive feature network only on keyframes, warps cached features forward with optical flow, and runs a shallow task network on every frame.This operation model reuses higher-level features across frames while updating predictions continuously.
  • 3.3. Accel: Accel uses an independent reference feature network on keyframes and a configurable update feature network, with ResNet-18 through ResNet-101 spanning accuracy–performance goals.The reference feature network is ResNet-101, while the update network can be selected according to the desired operating point.
  • 3.3. Accel: The reference and update branches provide complementary views of current-frame labels that are merged by a 1 × 1 convolutional score-fusion layer.The fused scores are passed through the output block to produce the segmentation for the current frame.
  • 3.3. Accel: The system is modular because independently instantiated task networks share a fixed interface that can accept update feature networks with the required representation dimensionality.Reference, update, and optical-flow components are independently trainable and can be jointly fine-tuned.
  • 3.3. Accel: Keyframe scheduling is orthogonal to network design, so adaptive scheduling methods remain compatible with Accel despite their added complexity.The paper reports that adaptive scheduling did not provide a commensurate accuracy gain in its practical evaluation.

4.1. Setup

The evaluation uses Cityscapes and CamVid with mIoU for accuracy and average seconds per frame for performance, under experiments generally run at keyframe interval 5.

  • 4.1. Setup: The evaluation covers Cityscapes and CamVid, benchmarks for complex urban-scene understanding and semantic video segmentation.Cityscapes contains 30-frame street-scene snippets from 50 European cities, recorded at 17 fps.
  • 4.1. Setup: Accuracy is measured by mean intersection-over-union, averaged across valid semantic classes.mIoU is the average Jaccard index over the dataset’s valid classes.
  • 4.1. Setup: Performance is reported as average inference time in seconds per frame over the full dataset, the inverse of throughput.This reporting convention expresses speed as lower seconds per frame rather than higher frames per second.
  • 4.1. Setup: Figure 3 compares four Accel variants with single-frame DeepLab models and other related work on Cityscapes using accuracy versus inference time.All plotted results use keyframe interval 5 and draw from Table 1.
  • 4.1. Setup: The experiments use Tesla K80 GPUs and keyframe interval 5 unless otherwise stated, with an open-source MXNet implementation.The training procedure includes joint training and a reference implementation trained with the same reference and optical-flow components.

4.2. Results

Accel establishes a state-of-the-art accuracy-throughput trade-off for semantic video segmentation by combining reference and update processing across configurable model variants. Its update branch corrects warping error while fusion improves accuracy over individual component networks, with fusion location trading accuracy for speed.

  • Accel establishes a state-of-the-art accuracy-throughput trade-off curve for semantic video segmentation.
  • All Accel models exceed 72 mIoU on Cityscapes and 66 mIoU on CamVid, while Accel-101 surpasses DeepLab-101 in accuracy.
  • The four Accel variants form a Pareto curve that supersedes single-frame DeepLab and visibly outperforms related efficient video-segmentation methods.
  • Above keyframe interval 1, Accel-18 consistently exceeds DFF accuracy; beyond interval 8, DFF approaches 60 mIoU while Accel remains roughly 70-75 mIoU.The authors attribute this gap to compounding warping error corrected by Accel’s update branch, while Accel incurs additional update-network cost.
  • Fusion improves each unmodified Accel model beyond either component subnetwork, with Accel-18 gaining 6.8 accuracy points over the reference network.The gain combines high-detail but dated reference representations with lower-resolution, temporally current update representations.
  • Late score fusion is slightly less accurate but faster than early feature fusion, so it is used except in Accel-101; reducing fusion channels also lowers accuracy without noticeable speed gains.Feature fusion uses higher-dimensional inputs, whereas score fusion is cheaper because scores have fewer channels.

5. Conclusion

Accel combines reference and update pathways to correct accumulated temporal error while supporting a range of accuracy–inference-speed trade-offs. Experiments show consistent advantages over prior approaches, including qualitative correction of warping distortions and gains over individual components.

  • Accel combines a reference pathway that warps keyframe features with an update pathway that corrects accumulated temporal error.The update branch processes the current frame, while the reference branch maintains warped keyframe representations.
  • The model family spans accuracy–inference-speed modalities, from high-throughput variants outperforming comparable single-frame models to high-accuracy variants exceeding state of the art.
  • Accel consistently outperforms previous work across keyframe intervals, and ablations show significant accuracy gains over its individual components.
  • Qualitative results show Accel correcting DFF’s warping-related distortions and avoiding DeepLab’s misclassifications on pedestrians, vehicles, vans, and vegetation.
Loading 1807.06667v4…