Source-linked AI summary
Temporally Distributed Networks for Fast Video Semantic Segmentation
Ping Hu, Fabian Caba Heilbron, Oliver Wang, Zhe Lin, Stan Sclaroff, Federico Perazzi
TL;DR
Video semantic segmentation requires strong features but must also meet low-latency demands, which deep per-frame computation makes difficult. TDNet distributes shallow sub-networks across sequential frames and uses attention propagation plus grouped distillation to reconstruct strong features efficiently. Across Cityscapes, CamVid, and NYUD-v2, it consistently outperforms previous methods in accuracy and efficiency.
Problem
Deep feature extraction improves video segmentation accuracy but its computation cost makes high-accuracy, low-latency processing difficult.
Method
TDNet distributes feature-group extraction across sequential frames, recomposes groups with attention propagation, and applies grouped knowledge distillation.
Results
TDNet consistently outperforms previous video segmentation methods in both accuracy and efficiency across Cityscapes, CamVid, and NYUD-v2.
Takeaways & Limitations
Temporal distribution lets TDNet retain high accuracy while significantly reducing video-frame processing latency.
Abstract
from arXiv · showhide
We present TDNet, a temporally distributed network designed for fast and accurate video semantic segmentation. We observe that features extracted from a certain high-level layer of a deep CNN can be approximated by composing features extracted from several shallower sub-networks. Leveraging the inherent temporal continuity in videos, we distribute these sub-networks over sequential frames. Therefore, at each time step, we only need to perform a lightweight computation to extract a sub-features group from a single sub-network. The full features used for segmentation are then recomposed by application of a novel attention propagation module that compensates for geometry deformation between frames. A grouped knowledge distillation loss is also introduced to further improve the representation power at both full and sub-feature levels. Experiments on Cityscapes, CamVid, and NYUD-v2 demonstrate that our method achieves state-of-the-art accuracy with significantly faster speed and lower latency.
1. Introduction
Video semantic segmentation needs strong features for accuracy, but deep networks impose high computation costs that hinder low-latency applications. TDNet distributes shallow feature extraction across frames, then recomposes feature groups with temporal attention and grouped distillation to improve the speed–accuracy trade-off.
- Motivation: Deep networks provide strong features for accurate video segmentation but are costly for latency-sensitive applications.The paper highlights self-driving cars, robot sensing, and augmented reality as relevant settings.
- Motivation: Applying a deep segmentation model independently to every frame fails to exploit temporal information and recomputes all features.
- TDNet: TDNet divides deep-model features into groups approximated by distinct shallow sub-networks with comparable total model capacity.The sub-networks share an architecture scaled to 1/N of the original deep model’s size.
- TDNet: Sequential circular assignment computes one new sub-feature group per frame while reusing complementary groups from earlier frames.This distributes computation over time and avoids recomputing all feature groups at every frame.
- TDNet: Attention propagation compensates for motion-induced spatial misalignment when reassembling feature groups across frames.Grouped knowledge distillation further improves representation at both full-feature and sub-feature levels.
- Reported contributions: TDNet reports more than 2× lower latency at comparable accuracy and better accuracy and latency than prior methods across Cityscapes, CamVid, and NYUD-v2.
2. Related Work
Prior video segmentation methods either add temporal layers to repeated deep inference or propagate keyframe features. These approaches trade accuracy against computation or face motion-induced spatial misalignment during feature reuse.
- Image semantic segmentation: Image segmentation research has pursued either efficient models or high-quality models after deep learning enabled dense label prediction.
- Video semantic segmentation: Temporal-context methods repeatedly apply a deep model and aggregate features, improving accuracy over single-frame approaches but adding per-frame computation.
- Video semantic segmentation: Keyframe-based methods reduce computation by propagating and reusing high-level features, but motion can misalign pixel-level information over time.
3. Temporally Distributed Network
TDNet distributes feature extraction across sequential video frames, then recomposes complementary sub-features with attention propagation to produce segmentation features efficiently despite motion-induced misalignment.
- 3.1. Distributed Networks: TDNet divides a deep model’s feature representation into sub-features approximated by distinct shallow sub-networks.The sub-networks share a shallow architecture sized to preserve similar total model capacity while separating feature subspaces.
- 3.1. Distributed Networks: Sequential circular assignment computes one new sub-feature group per frame and reuses earlier groups to avoid redundant feature extraction.The full feature representation is formed by aggregating previously computed feature groups with the current one.
- 3.2. Feature Aggregation: APM addresses motion-induced spatial misalignment by propagating attention between feature groups extracted at different time steps.Unlike optical-flow warping, the module is based on non-local attention extended for spatio-temporal variation.
- 3.2. Feature Aggregation: For four sub-networks, each four-frame window covers all sub-networks, allowing frame t to merge its current map with maps from t-3, t-2, and t-1.The next frame reuses the corresponding three preceding sub-feature maps in a shifted temporal window.
- 3.2. Feature Aggregation: Downsampling and local temporal propagation reduce attention cost while restricting comparisons to neighboring frames with smaller motion.The attention complexity is reduced by a factor of n^2, and experiments find n=4 effective for preserving spatial information while lowering computation.
- 3.2. Feature Aggregation: Grouped knowledge distillation aligns both full student outputs and sub-network outputs with corresponding teacher representations.The grouped objective conditions the teacher output on each respective feature subspace.
4. Grouped Knowledge Distillation
Grouped knowledge distillation transfers knowledge from a strong deep teacher to TDNet at both the whole-model and sub-feature-group levels. This encourages complementary information across feature paths while supervising the student’s segmentation outputs.
- Grouped distillation transfers knowledge at the subspace level so information from different feature paths becomes more complementary.
- A deep baseline such as PSPNet101 serves as the teacher, while TDNet with m sub-networks serves as the student.
- The loss aligns student and teacher output distributions both for the complete model and for individual sub-feature groups.
- The teacher’s feature reduction layer is evenly divided into m independent sub-convolution groups to produce the teacher sub-feature groups.
- The final loss combines ground-truth cross-entropy supervision, whole-model KL-divergence distillation, and feature-group KL-divergence distillation.The paper sets α and β to 0.5.
5. Experiments
Experiments on Cityscapes, Camvid, and NYUD-v2 show that TDNet maintains or improves segmentation accuracy while reducing computation, latency, and temporal-load imbalance. Ablations further support attention propagation, grouped distillation, distributed feature paths, and moderate attention downsampling.
- Setup and overall results: TDNet achieves state-of-the-art accuracy with faster speed and lower, evenly distributed latency on Cityscapes, Camvid, and NYUD-v2.The evaluation uses PSPNet and BiseNet-based backbones and reports speed on a single Titan Xp with batch size one.
- Cityscapes: TD2-PSP50 reduces Cityscapes per-frame time from 360ms to 178ms while improving accuracy relative to PSPNet101 with similar total parameters.TD4-PSP18 reduces latency further to one quarter, but its shallower subnetworks reduce performance relative to PSPNet101.
- Cityscapes real-time models: TD2-Bise34 outperforms prior real-time methods at comparable speed and is roughly three times faster than similarly sized BiseNet∗101.TD4-Bise18 reaches nearly 50 FPS with lower accuracy, while both distributed models improve over their single-path baselines at similar time cost.
- Camvid: TD2-PSP50 surpasses Netwarp by about 9% mIoU while being roughly four times faster on Camvid.Compared with similarly capacitated PSPNet101 baselines, it approximately halves computation with comparable accuracy; TD4-PSP18 halves latency again with an accuracy decrease.
- NYUD-v2: On NYUD-v2, TD2-PSP50 outperforms STD2P in accuracy and speed, halves PSPNet101 latency while preserving accuracy, and improves over PSPNet18 by about 1.6% mIoU without increasing latency.TD4-PSP18 is more than 5× faster but has worse accuracy.
- Method analysis: APM outperforms optical-flow warping and vanilla spatio-temporal attention in accuracy and latency, remaining robust with a six-frame sampling gap.Without motion compensation, simple feature addition has the worst accuracy; flow-based methods fail at the six-frame gap while APM drops only slightly.
6. Conclusion
TDNet combines temporally distributed feature computation with attention-based merging to improve video segmentation efficiency while retaining accuracy. Grouped knowledge distillation further boosts performance, and the method consistently outperforms prior approaches in accuracy and efficiency.
- TDNet merges feature maps computed across different frames with attention propagation to improve processing latency while retaining high accuracy.The attention propagation module handles feature alignment when recombining temporally distributed features.
- Grouped knowledge distillation further boosts TDNet’s performance.
- TDNet consistently outperforms previous methods in both accuracy and efficiency.