Source-linked AI summary
Semantic Video CNNs through Representation Warping
Raghudeep Gadde, Varun Jampani, Peter V. Gehler
TL;DR
Semantic video segmentation needs temporal consistency while retaining strong image CNNs and practical computation. NetWarp warps intermediate representations using adjacent-frame optical flow and integrates this operation into existing CNNs. The resulting online, end-to-end-trainable models improve benchmark performance and set new state-of-the-art results on CamVid and Cityscapes.
Problem
Frame-by-frame image CNNs ignore temporal information and can produce jitter, while existing CRF-based video methods are often too slow for practical use.
Method
NetWarp transforms optical flow and uses it to warp intermediate CNN representations from adjacent frames within existing image segmentation architectures.
Results
NetWarp consistently improves several baseline CNNs and sets new state-of-the-art performance on CamVid and Cityscapes.
Takeaways & Limitations
The approach provides online, end-to-end-trainable video CNNs with only small computation overhead and improved temporal consistency, including better coverage of thin structures.
Takeaways & Limitations
Training uses only two adjacent frames at a time because GPU memory limits longer temporal unrolling, especially with memory-demanding base models.
Abstract
from arXiv · showhide
In this work, we propose a technique to convert CNN models for semantic segmentation of static images into CNNs for video data. We describe a warping method that can be used to augment existing architectures with very little extra computational cost. This module is called NetWarp and we demonstrate its use for a range of network architectures. The main design principle is to use optical flow of adjacent frames for warping internal network representations across time. A key insight of this work is that fast optical flow methods can be combined with many different CNN architectures for improved performance and end-to-end training. Experiments validate that the proposed approach incurs only little extra computational cost, while improving performance, when video streams are available. We achieve new state-of-the-art results on the CamVid and Cityscapes benchmark datasets and show consistent improvements over different baseline networks. Our code and models will be available at http://segmentation.is.tue.mpg.de
1. Introduction
Video semantic segmentation needs temporal consistency without sacrificing the strengths of image CNNs or practical runtime. NetWarp converts image CNNs into online video CNNs by warping intermediate representations, improving benchmark performance with little overhead.
- Image CNNs substantially outnumber video segmentation models, while frame-by-frame processing ignores temporal information and can cause boundary jitter.
- Existing CRF-based video methods can operate on condensed label representations and are often too slow for practical use.
- NetWarp uses optical flow and learned operations to warp intermediate representations from the previous frame into the current frame.
- NetWarp processes frames online, using only the present and previous frames when predicting the present segmentation.
- NetWarp consistently improves frame-by-frame baselines and achieves new state-of-the-art results on CamVid and Cityscapes.
2. Related Works
Prior semantic video segmentation methods commonly propagate frame-level predictions with CRFs or filtering, while other approaches use geometry, correspondences, or task-specific video architectures. NetWarp instead propagates information through intermediate CNN representations for more integrated training and fast runtime.
- Many semantic video segmentation systems first predict each frame and then use CRFs or filtering to improve temporal consistency.
- 3D scene-structure methods improve segmentation using geometric and motion features, but obtaining 3D information is costly.
- Graphical-model approaches connect video pixels with temporal links or long-range spatio-temporal energy terms before or alongside CNN predictions.
- NetWarp avoids CRF construction by filtering intermediate CNN representations directly, enabling fast runtime and gradient-descent training.
- Existing filtering techniques propagate information after semantic labels are computed, whereas NetWarp propagates intermediate representations during CNN training.
- Unlike video-level recognition and captioning models, semantic video segmentation requires local pixel-level information propagation across frames.
3. Warping Image CNNs to Video CNNs
The method converts image segmentation CNNs into online video CNNs by transforming optical flow, warping previous-frame activations, and combining them with current activations. It is differentiable, trainable end to end, and improves segmentation across architectures with small computational cost.
- Task formulation: The video model predicts segmentation for each frame online, using only previous frames when processing the present frame.
- NetWarp module: A NetWarp module warps kth-layer representations from the previous frame and combines them with the corresponding present-frame representations.
- Empirical findings: NetWarp improves semantic predictions across CNN architectures, especially at object boundaries and thin structures, while adding only tiny computation cost.
- NetWarp module: NetWarp uses dense optical flow and performs flow transformation, representation warping, and representation combination.
- Flow transformation: The FlowCNN transforms pre-computed optical flow using the two frames and their difference, with all parameters learned from semantic segmentation loss.
- Representation warping: Previous-frame representations are aligned with present-frame locations through bilinear interpolation using transformed flow.
- Representation combination: The warped and present representations are linearly combined with learned per-channel weights before entering the remaining image CNN layers.
- Training: Training uses two adjacent frames at a time because GPU memory limits make longer temporal unrolling impractical for memory-demanding base models.
4. Experiments
Experiments on CamVid and Cityscapes show that NetWarp consistently improves several CNN segmentation architectures over frame-by-frame baselines, with low runtime overhead. Gains are especially evident near object boundaries and on thin structures, while the method reaches strong benchmark performance.
- CamVid: 70.2% IoU is achieved by the best PlayData-CNN model with two NetWarp modules on CamVid.This improves over the 68.9% PlayData-CNN baseline, with modules performing consistently better across insertion locations.
- CamVid: 1.8% IoU improvement is obtained with Dilation-CNN at a 15-millisecond runtime increase.The runtime assumes the previous frame's result is already available, as in online video segmentation.
- Cityscapes: On Cityscapes single-scale PSPNet, NetWarp improves IoU by 1.2, tIoU by 2.4, and iIoU by 1.4 with 24 milliseconds of overhead.The multi-scale model also improves IoU by 0.7, tIoU by 1.8, and iIoU by 1.4%.
- Cityscapes: NetWarp achieves 80.5 versus 80.6 IoU against the current top method and leads reported methods in iIoU on the Cityscapes test server.The paper reports especially strong parsing of thin structures, including an approximately 4% IoU improvement for poles.
- Qualitative and boundary analysis: Trimap analysis attributes IoU gains especially to better performance near object boundaries, while temporal representations also recover thin structures and correct larger regions.The effect is reported for both single-scale and multi-scale networks.
5. Conclusions and Outlook
NetWarp transforms image CNNs into efficient, end-to-end trainable video CNNs by transferring intermediate representations across adjacent frames using optical flow. Experiments show improved performance across strong baselines, with better temporal consistency and thin-structure coverage, while training here used only two adjacent frames because of memory constraints.
- NetWarp transfers intermediate CNN filter activations between adjacent frames using transformed optical-flow estimates.
- Experiments show consistent performance improvements across different image CNN hierarchies on Cityscapes and CamVid.
- NetWarp produces more temporally consistent predictions and better coverage of thin structures such as poles and traffic signs.
- Flow changes radically after transformation with FlowCNN trained using segmentation loss, and qualitative results suggest boundary optical flow is important for semantic segmentation.
- Training used only two adjacent frames at a time because GPU memory limitations and memory-demanding base models made multi-frame training difficult.