Source-linked AI summary
Clockwork Convnets for Video Semantic Segmentation
Evan Shelhamer, Kate Rakelly, Judy Hoffman, Trevor Darrell
TL;DR
Per-frame image segmentation is too computationally demanding for video and fails to exploit temporal continuity. The paper introduces clockwork convnets that schedule network stages at fixed or adaptive rates according to semantic stability. Across video settings, the framework supports lower latency or computation while maintaining recognition accuracy, including adaptive processing that fully processes 52% of frames with minimal mean IU loss.
Problem
Applying still-image segmentation networks independently to every video frame is computationally demanding and ignores temporal continuity that could reduce computation.
Method
Clockwork convnets partition networks into stages, persist state and outputs across frames, and schedule stages using pipeline, fixed-rate, or data-driven adaptive clocks.
Results
The framework reduces latency by 59% with a 3-stage pipeline and adaptively fully processes only 52% of frames with minimal mean IU loss at θ = 0.25.
Takeaways & Limitations
Clockwork scheduling tunes video segmentation computation for latency, overall computation, and recognition performance while handling dynamic and static scenes alike.
Abstract
from arXiv · showhide
Recent years have seen tremendous progress in still-image segmentation; however the naïve application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video. We propose a video recognition framework that relies on two key observations: 1) while pixels may change rapidly from frame to frame, the semantic content of a scene evolves more slowly, and 2) execution can be viewed as an aspect of architecture, yielding purpose-fit computation schedules for networks. We define a novel family of "clockwork" convnets driven by fixed or adaptive clock signals that schedule the processing of different layers at different update rates according to their semantic stability. We design a pipeline schedule to reduce latency for real-time recognition and a fixed-rate schedule to reduce overall computation. Finally, we extend clockwork scheduling to adaptive video processing by incorporating data-driven clocks that can be tuned on unlabeled video. The accuracy and efficiency of clockwork convnets are evaluated on the Youtube-Objects, NYUD, and Cityscapes video datasets.
1 Introduction
Video semantic segmentation is computationally demanding when image networks process every frame, but temporal continuity and slower semantic change enable scheduled computation. Clockwork convnets exploit these properties with fixed and adaptive schedules evaluated across several video datasets.
- Motivation: Per-frame application of image segmentation networks is computationally expensive and ignores temporal continuity that could reduce computation.Video segmentation supports applications including autonomous driving and UAV navigation.
- Motivation: Feature velocity varies across layers, with deeper features changing more slowly than shallower features over video sequences.This motivates assigning different update rates to different network stages.
- Approach: Clockwork convnets group layers into stages and schedule their execution at separate update rates, fusing computations from multiple frames through skip connections.The schedule exploits the lower resolution and slower change of deeper layers.
- Schedules: The framework includes pipeline schedules for latency, fixed-rate schedules for computation, and adaptive schedules that respond to video content.Adaptive clocks are tuned to update more during dynamic scenes and preserve cached representations during static scenes.
- Evaluation: The approach is evaluated on synthetic PASCAL VOC sequences and videos from NYUDv2, YouTube-Objects, and Cityscapes.
2 Related Work
Prior work established strong fully convolutional image segmentation and several temporal video architectures, but existing approaches did not adequately address efficient, real-time semantic segmentation of longer videos. Clockwork networks target this computational gap by incorporating temporal scheduling into network execution.
- Fully Convolutional Networks: Fully convolutional networks provide state-of-the-art pixelwise image prediction, but their standard dense computation does not temporally amortize sequential video inputs.FCNs operate on arbitrary-sized inputs and produce corresponding pixelwise outputs.
- Video Networks and Frame Selection: Video networks use spatiotemporal filtering, temporal fusion, recurrence, or recurrent convolutional structures to model information across frames.
- Network Acceleration: Network acceleration research shows that architectural choices can trade computation and memory against segmentation accuracy, while model search can improve efficiency.
- Semantic Segmentation: Existing video-segmentation work largely targets supervoxels, motion-driven object segmentation, or weakly supervised tagged-video segmentation rather than semantic segmentation.
- Semantic Segmentation: > 15s/frame and a separate recognition step is needed for semantic segmentation, whereas a standard FCN computes full semantic segmentation in 0.1s/frame.
3 Fast Frames and Slow Semantics
Deep semantic features change more slowly across video frames than shallow features, supporting temporal persistence and selective updates. Measurements across videos motivate caching deeper-layer scores while adapting updates to scene motion.
- Temporal Coherence: Temporal skip connections persist deep features across frames because deep semantic representations remain relevant while shallow details vary more rapidly.
- Measurement: Score-map differences can be measured with a distance function based on Hamming distance between one-hot class encodings.For score layer ℓ, outputs have dimensions K × H × W, where K is the number of categories.
- Measurements: On average across YouTube-Objects videos, the deepest score layer changes an order of magnitude less than shallower layers.This motivates caching deeper-layer scores to inform current-frame inference.
- Interpretation: Pooling and learned invariances contribute to slower change in deeper layers by increasing robustness to translation and noise and tuning features toward supervised classes.
- Video Dynamics: In both high-motion and relatively static videos, fc7 differences are smaller than pool4 differences, while motion level changes the magnitude of both differences.
- Adaptive Scheduling: Because pool4 changes more than fc7 but reflects motion-dependent variation, pool4 differences can guide adaptive updates of deeper stages.
4 A Clockwork Network
The clockwork FCN treats execution schedules as architectural elements, partitioning a network into stages whose computations and cached features update at different rates. It supports pipelined, fixed-rate, and adaptive schedules for trading video-segmentation latency, computation, and accuracy.
- Clockwork FCN: The clockwork FCN adapts image FCNs to video by carrying temporal information across frames and persisting features and outputs.Its skip-layer structure is repurposed for staged computation, allowing cached features to inform current-frame predictions.
- Generalized Clockwork: The generalized framework recovers standard recurrent, clockwork recurrent, and clockwork FCN networks through time-varying input, hidden-state, output, and transition operations.In the clock FCN, state persists through the identity operation and output is fused on every frame, while modules retain hierarchical layer composition.
- Execution as Architecture: Clockwork architectures partition networks into modules or stages executed according to different schedules, with clocks acting as dynamic cuts in the computation graph.When a clock is on, execution traverses the next modules; when off, execution is blocked or a prior stage can be cached using complementary clocks.
- Fixed-Rate: Fixed-rate scheduling reduces computation by executing deeper, semantically slower layers less often while persisting their features across skipped frames.Stage clock rates are free parameters that exchange inference speed and accuracy, and stage composition can be rebalanced for latency, spatial refinement, or output classes.
- Adaptive Clockwork: Adaptive clocks use data-dependent thresholds or learned functions to adjust processing to motion, with sensitivity tunable on unannotated video through proportional temporal label differences.The method thresholds differences at a shallower stage to schedule deeper stages, seeking savings during little motion while maintaining accuracy during dynamic scenes.
5 Results
Experiments evaluate pipelined, fixed-rate, and adaptive clockwork FCNs across video segmentation settings, showing latency and computation savings with limited accuracy loss in supported schedules.
- Evaluation: The evaluation uses synthetic translated PASCAL sequences and video datasets including NYUD, YouTube-Objects, and Cityscapes.Metrics include mean IU and frequency-weighted IU for semantic segmentation.
- Pipelining: The pipelined FCN segments translated PASCAL sequences with higher accuracy than every-other-frame full-FCN evaluation in the same time envelope.The comparison includes standard masks and a 10-pixel boundary band.
- Pipelining: Pipelined clockwork networks reduce latency with minimal accuracy loss relative to standard FCN evaluation on every frame.Deeper representations persisted from previous frames are fused with current low-level predictions.
- Pipelining: A qualitative YouTube-Objects example shows the pipeline detecting car occlusion as it happens, unlike the lagging every-other-frame baseline.The network also quickly recognizes the car’s reappearance.
- Fixed-Rate: Exponential clockwork degrades accuracy while requiring 1.5× the computation of every-other-frame evaluation, motivating adaptive scheduling instead.Alternating clockwork achieves higher accuracy than the every-other-frame reference.
6 Conclusion
The paper presents clockwork FCNs that persist features and schedule layers at different rates, supporting latency, computation, and recognition-performance trade-offs.
- Conclusion: Clockwork FCNs persist features across time and schedule deeper layers more slowly than shallower layers.Adaptive clocks schedule processing online for dynamic and static scenes while maintaining accuracy.
- Conclusion: Fixed and adaptive schedules tune processing for latency, overall computation time, and recognition performance.The clockwork perspective connects convolutional networks with event-driven vision architectures.
- Conclusion: The generalized clockwork framework suggests further architectural variations for spatiotemporal video processing.