Source-linked AI summary
Semantic Video Segmentation by Gated Recurrent Flow Propagation
David Nilsson, Cristian Sminchisescu
TL;DR
Semantic video segmentation is difficult because accurate models require extensive frame-level annotation, while existing systems have limited ability to exploit partially supervised video. The paper introduces an end-to-end trainable architecture that propagates segmentation through optical flow and uncertainty-gated recurrent fusion. Experiments indicate improved segmentation accuracy and temporal consistency across CityScapes and CamVid using unlabeled neighboring frames.
Problem
Frame-level annotation is difficult for semantic video segmentation, and current systems have limited ability to exploit partial supervision in temporally coherent video.
Method
The method augments a convolutional single-frame segmentation model with optical-flow warping and spatio-temporal recurrent units that adaptively gate and fuse temporal predictions.
Results
Experiments indicate improved segmentation accuracy and temporal labeling consistency on CityScapes and CamVid across multiple deep semantic models using unlabeled neighboring frames.
Takeaways & Limitations
The gated recurrent flow propagation component can be plugged into static semantic segmentation architectures to use sparsely labeled video without additional annotation cost.
Takeaways & Limitations
Jointly refining optical flow receives only weak semantic-segmentation supervision, which can be noisy and limit learning when flow quality is low.
Abstract
from arXiv · showhide
Semantic video segmentation is challenging due to the sheer amount of data that needs to be processed and labeled in order to construct accurate models. In this paper we present a deep, end-to-end trainable methodology to video segmentation that is capable of leveraging information present in unlabeled data in order to improve semantic estimates. Our model combines a convolutional architecture and a spatio-temporal transformer recurrent layer that are able to temporally propagate labeling information by means of optical flow, adaptively gated based on its locally estimated uncertainty. The flow, the recognition and the gated temporal propagation modules can be trained jointly, end-to-end. The temporal, gated recurrent flow propagation component of our model can be plugged into any static semantic segmentation architecture and turn it into a weakly supervised video processing one. Our extensive experiments in the challenging CityScapes and Camvid datasets, and based on multiple deep architectures, indicate that the resulting model can leverage unlabeled temporal frames, next to a labeled one, in order to improve both the video segmentation accuracy and the consistency of its temporal labeling, at no additional annotation cost and with little extra computation.
1. Introduction
Semantic video segmentation requires learning from temporally coherent video despite costly frame-level annotation and the limitations of single-frame models. The paper proposes augmenting existing image CNNs with optical-flow warping and adaptive recurrent fusion to exploit sparsely labeled video.
- Frame-level annotation makes fully trainable semantic video segmentation difficult, motivating methods that use partial supervision across video.
- Sparsely labeled frames can provide temporal information that is propagated and aggregated to reduce uncertainty during learning and inference.
- The proposed architecture retains single-frame CNNs and adds spatial transformer structures that warp information along optical flow.
- Adaptive recurrent units learn to fuse estimates from unlabeled frames with nearby temporal information according to uncertainty.
- The model is differentiable and end-to-end trainable, and video segmentation could support applications including indexing, robotics, navigation, and manufacturing.
2. Related work
Related work spans deep semantic image segmentation, classical and optical-flow-linked video segmentation, and deep temporal modeling for recognition. The paper builds on these areas while targeting semantic segmentation over video.
- Deep convolutional architectures form the foundation of many successful semantic image segmentation methods.
- Earlier video segmentation methods used normalized cuts, random fields, tracking, motion segmentation, and hierarchical graph formulations.
- More recent proposal methods generate figure-ground or superpixel estimates per frame and link them over time using optical flow.
- Deep action-recognition models combine image and optical-flow streams, sometimes passing their outputs to recurrent networks for later labeling.
3. Methodology
The methodology propagates semantic predictions through optical-flow warping and convolutional GRU units that adaptively fuse temporal and per-frame evidence. Forward and backward variants can aggregate information around a labeled center frame while training with sparse supervision.
- Spatio-Temporal Transformer Warping: The model warps the previous hidden segmentation state along optical flow, then combines it with a current-frame CNN estimate through a learned GRU.
- Gated Recurrent Flow Propagation: GRFP can operate across multiple timesteps in forward and backward directions while using sparsely labeled center frames and unlabeled temporal neighborhoods.
- Architecture: The architecture consists of spatio-temporal transformer warping, gated recurrent units, and forward-backward implementations.
- Spatio-Temporal Transformer Warping: Optical-flow warping uses a differentiable bilinear interpolation with only four non-zero kernel terms, allowing gradients to reach both features and flow efficiently.
- Gated Recurrent Units for Semantic Video Segmentation: Flow confidence gates reliance on warped predictions, while softmax replaces tanh for normalized segmentation probabilities and λ compensates for scaling differences.
- Gated Recurrent Units for Semantic Video Segmentation: The semantic-video GRU replaces fully connected layers with convolutions and represents hidden states and inputs as H × W × C tensors of class probabilities.
- Implementation: Experiments use deep static segmentation networks, including Dilation and LRR, with standard log-likelihood training and memory-saving framewise computation.
4. Experiments
Experiments on CityScapes and CamVid evaluate semantic accuracy, temporal consistency, frame-count effects, architectural portability, flow gating, and forward-backward variants. GRFP improves static baselines, but gains saturate after four propagated frames and depend on optical-flow quality.
- Semantic video segmentation: Using more than one frame improves CityScapes segmentation performance, but gains saturate beyond four frames.The model was trained using five frames, while inference benefits level off after four.
- Semantic video segmentation: GRFP improves CityScapes test-set mIoU by 0.7 percentage points over Dilation10 and 1.0 percentage points over LRR-4x.The comparison is against single-frame baselines that do not use video.
- Semantic video segmentation: Temporal propagation and consistency reasoning account for most gains beyond the refined per-frame GRFP(1) model.GRFP(1) matches the pretrained Dilation10 average, whereas GRFP(5) adds temporal information.
- Semantic video segmentation: The method improves labeling across all classes when LRR is used as the backend, confirming portability beyond the Dilation architecture.This experiment uses Flownet2 for optical flow.
- Semantic video segmentation: Qualitative examples show better segmentation of cars, walls, and poles when earlier frames provide information unavailable from the current frame alone.The reported examples include uniform-surface objects and a difficult pole.
- Forward-backward models: Averaging forward and backward predictions performs best with five forward frames plus one backward prediction, while learned stacking adds no accuracy.The forward-backward comparison is reported on the CityScapes validation set.
- Flow and end-to-end training: End-to-end flow refinement remains competitive, but noisy semantic supervision and dependence on accurate flow can hinder learning the flow network.Lower-quality flow reduces STGRU performance, especially when training begins.
5. Conclusions
The paper presents an end-to-end trainable methodology that uses unlabeled temporal frames to improve semantic video segmentation accuracy and temporal-label consistency, without additional annotation cost and with little supplementary computation.
- The methodology jointly refines recognition, optical flow, and temporal propagation modules in an end-to-end trainable video segmentation system.
- A convolutional architecture and spatio-temporal transformer recurrent layer propagate labeling information through optical flow with locally uncertainty-based gating.
- Experiments on CityScapes and CamVid with multiple deep semantic models indicate improved segmentation accuracy and temporal-label consistency.
- The method uses unlabeled frames without additional annotation cost and requires little supplementary computation.