Source-linked AI summary
Low-Latency Video Semantic Segmentation
Yule Li, Jianping Shi, Dahua Lin
TL;DR
Video semantic segmentation must process high-throughput streams at low latency despite the cost of fully convolutional networks. The paper introduces adaptive feature propagation and adaptive key-frame scheduling, achieving competitive performance on Cityscapes and CamVid while reducing latency from 360 ms to 119 ms.
Problem
Video segmentation must handle 15 to 30 frames per second and strict latency requirements, while existing methods can neglect maximum latency or fall short in the performance/latency tradeoff.
Method
The framework adaptively propagates features with spatially variant convolution and schedules keyframes using accuracy prediction, coordinating computation to reduce cost and latency.
Results
360 ms to 119 ms: the framework substantially reduces latency while maintaining competitive performance on Cityscapes and CamVid.
Takeaways & Limitations
The combined scheduling and propagation design provides a better accuracy-latency tradeoff for online video segmentation under stringent latency constraints.
Takeaways & Limitations
Further model-compression approaches are needed to reduce computation cost and latency for a practical system.
Abstract
from arXiv · showhide
Recent years have seen remarkable progress in semantic segmentation. Yet, it remains a challenging task to apply segmentation techniques to video-based applications. Specifically, the high throughput of video streams, the sheer cost of running fully convolutional networks, together with the low-latency requirements in many real-world applications, e.g. autonomous driving, present a significant challenge to the design of the video segmentation framework. To tackle this combined challenge, we develop a framework for video semantic segmentation, which incorporates two novel components: (1) a feature propagation module that adaptively fuses features over time via spatially variant convolution, thus reducing the cost of per-frame computation; and (2) an adaptive scheduler that dynamically allocate computation based on accuracy prediction. Both components work together to ensure low latency while maintaining high segmentation quality. On both Cityscapes and CamVid, the proposed framework obtained competitive performance compared to the state of the art, while substantially reducing the latency, from 360 ms to 119 ms.
1. Introduction
Video semantic segmentation must handle far more data than image segmentation while meeting strict latency requirements. The proposed framework combines adaptive feature propagation with adaptive scheduling to reduce cost and maximum latency while maintaining competitive performance.
- Challenges: 15 to 30 frames per second make video segmentation substantially more resource-intensive than image analysis, while real-world systems impose strict response-latency requirements.These constraints are especially relevant to applications such as autonomous driving.
- Limitations of prior work: Prior feature-reuse methods reduce amortized computation but neglect maximum latency and treat all spatial locations uniformly despite differences between smooth regions and boundaries.Consequently, previous methods fall short in either performance or latency according to the paper’s comparison.
- Proposed framework: The framework uses spatially variant convolution to adaptively combine features from preceding frames, improving the use of temporal information locally.The combination weights adapt to different spatial locations rather than applying one uniform treatment.
- Proposed framework: An adaptive scheduler allocates keyframes on demand using accuracy prediction and coordinates keyframe computation with feature propagation in parallel.This design targets more efficient resource use and reduced maximum latency.
- Results: The method achieves competitive state-of-the-art performance on Cityscapes and CamVid with substantially lower latency.The introduction reports this result as a central contribution of the framework.
2. Related Work
Related work spans image-based segmentation, temporal modeling for video, and methods that reuse features to reduce computation. These approaches differ in whether they target per-frame accuracy, temporal continuity, computational cost, or online suitability.
- Image Semantic Segmentation: Image segmentation methods predict per-pixel labels using FCNs, with later work extending them through CRFs, multiscale architectures, encoder-decoder designs, and efficient networks.These methods are primarily developed for static images.
- Image Semantic Segmentation: Image-based methods applied independently to video frames do not exploit temporal relations between frames.The related-work discussion distinguishes this per-frame setting from video-specific approaches.
- Video Semantic Segmentation: One video-segmentation category exploits temporal continuity through recurrent models, future-frame prediction, or feature warping to improve accuracy.Examples include spatial-temporal LSTMs, gated recurrent units, predicted future frames, and wrapped previous-frame features.
- Video Semantic Segmentation: A second category reduces computation by reusing preceding-frame features, but simple replication can fail under significant scene changes and optical-flow propagation adds a separate network cost.Clockwork Net and Deep Feature Flow illustrate these respective tradeoffs.
- Video Semantic Segmentation: Some video methods use different task or processing settings, including foreground-object segmentation or bidirectional traversal, making them unsuitable for the online setting considered here.The paper distinguishes these settings from parsing the entire scene online.
3. Video Segmentation Framework
The framework reduces video-segmentation cost and latency by adaptively selecting key frames and propagating high-level features spatially across non-key frames. A fast-track scheduler further lowers maximum latency while preserving competitive accuracy.
- Framework overview: The framework uses low-level CNN features to decide whether each frame becomes a new key frame or reuses propagated high-level features.Key frames are processed through the expensive higher network; other frames use predicted kernels for spatially variant propagation.
- Framework overview: Adaptive propagation maintains reasonably high accuracy for up to 7 frames from a key frame, while key-frame decision costs 20 ms.Feature propagation costs 38 ms compared with 299 ms for computing high-level features through the higher network.
- Adaptive key-frame selection: Key-frame selection predicts segmentation-map deviation from low-level feature differences and triggers a new key frame when deviation exceeds a threshold.The predictor uses convolutional layers, global pooling, and a fully connected layer; predicted deviation generally increases over time.
- Adaptive feature propagation: The method addresses limitations of fixed-kernel and optical-flow propagation, which can be costly or inflexible under complex scene and motion changes.Different scene parts may move in different directions and therefore require different propagation weights.
- Adaptive feature propagation: Spatially variant convolution propagates previous-key-frame high-level features using location-specific kernels predicted from current low-level features.Kernel weights vary by spatial location but are shared across channels, allowing different scene regions to receive different neighbor weights.
- Low-latency scheduling: The fast-track scheduling scheme reduces maximum latency from 360 ms to 119 ms while reducing accuracy from 76.84% to 75.89%.The paper states that this balance depends on combining low-latency scheduling with adaptive key-frame selection and adaptive feature propagation.
4. Implementation Details
The implementation uses low-level CNN features to support adaptive key-frame selection, spatially variant feature propagation, and feature fusion while balancing accuracy and computation.
- Network Architecture: The basic network is an ImageNet-pretrained ResNet-101 split at conv4_3 into lower and higher parts.The low-level feature representation from this layer has 1024 channels.
- Adaptive Key-Frame Selection: The adaptive key-frame selector compares low-level features from the current frame and previous key frame to predict deviation.It reduces both inputs to 256 channels, computes their differences, and applies global pooling followed by a fully connected layer.
- Adaptive Feature Propagation: The propagation module predicts spatially variant convolution weights from concatenated low-level features of the frame pair.Its predictor reduces inputs to 256 channels and outputs 81-channel weights using a 1 × 1 convolution.
- Feature Fusion: AdaptNet processes low-level features and fuses them with propagated features through convolutional layers.The fusion input concatenates both feature types and is processed by a 3 × 3 convolution with 256 channels.
- Training: Training first fixes the ground-truth-trained basic network, then fine-tunes propagation and scheduling modules on frame pairs separated by 2 to 10 steps.Each pair treats the first frame as the key frame and the second as the annotated frame.
5. Experiment
Experiments evaluate the framework on Cityscapes and CamVid, including propagation, scheduling, visual quality, and latency. The method maintains competitive segmentation performance while substantially reducing runtime and maximum latency.
- Datasets: The framework is evaluated on Cityscapes and CamVid against state-of-the-art video segmentation methods.Cityscapes uses 19 semantic categories, while CamVid contains 11 annotated classes from driving videos.
- Latency Analysis: 360 ms to 151 ms is the runtime reduction from adaptive feature propagation with fixed-interval scheduling, with a 4.9% performance decrease.Adaptive key-frame selection then boosts performance by 1.6%, but these schemes alone do not reduce maximum latency because key frames remain computationally heavy.
- Latency Analysis: 119 ms is achieved by the low-latency scheduler while maintaining comparable performance with only a minor drop.This scheduler addresses the maximum-latency problem left by the preceding runtime-reduction schemes.
- Feature Propagation: The adaptive propagation module outperforms globally learned unified propagation and propagation based directly on input-pixel differences.The results indicate that spatially variant learned weights are important for the proposed solution.
- Feature Propagation: 75.26% versus 68.41% is the reported improvement from fusing current-frame low-level features through AdaptNet.The authors attribute the gain to complementary information learned by the fusion model.
- Scheduling: The adaptive schedule outperforms fixed-rate and threshold schedules under every tested key-frame interval.The authors attribute the advantage over threshold scheduling to noise and irrelevant factors in intermediate feature maps used for key-frame selection.
- Latency Analysis: 119 ms is the reported pipeline latency with low-latency scheduling, equal to 33% of the basic network time.The latency sums the lower network part, adaptive key-frame selection, and adaptive feature propagation.
- CamVid and Visual Results: The method outperforms previous methods on CamVid across pixel accuracy and average per-Class Accuracy.The paper also reports qualitative results for both Cityscapes and CamVid, including frames with significant variation.
6. Conclusion
The framework combines adaptive feature propagation with adaptive key-frame scheduling to improve the accuracy–latency tradeoff in video semantic segmentation. Future work will explore model compression to further reduce computation cost and latency for practical systems.
- The framework combines adaptive feature propagation with adaptive key-frame scheduling for efficient video semantic segmentation.
- The method yields a substantially better accuracy–latency tradeoff than previous methods on Cityscapes and CamVid.
- Future work will explore model compression to further reduce computation cost and latency for a practical system.