Source-linked AI summary
Improving Semantic Segmentation via Video Propagation and Label Relaxation
Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao, Bryan Catanzaro
TL;DR
Pixel-wise semantic-segmentation annotation is costly, motivating methods that expand training data without labeling every new frame. The paper synthesizes image-label pairs with video prediction, joint propagation, and boundary label relaxation, achieving state-of-the-art results on Cityscapes, CamVid, and KITTI.
Problem
Semantic segmentation needs large amounts of pixel-wise annotation, but dense labeling is prohibitively expensive and boundaries can be inherently ambiguous.
Method
Video prediction models synthesize future image-label pairs, while joint propagation reduces misalignment and boundary label relaxation makes training more robust to propagation artifacts.
Results
83.5% mIoU on Cityscapes, 82.9% on CamVid, and 72.8% on KITTI are reported as state-of-the-art results.
Takeaways & Limitations
Synthesized training samples improve segmentation accuracy across several datasets, while boundary relaxation also supports longer-range propagation.
Takeaways & Limitations
The approach remains sensitive to the balance between hand-annotated and synthesized samples, with cumulative propagation causing a slight performance drop.
Abstract
from arXiv · showhide
Semantic segmentation requires large amounts of pixel-wise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks. We exploit video prediction models' ability to predict future frames in order to also predict future labels. A joint propagation strategy is also proposed to alleviate mis-alignments in synthesized samples. We demonstrate that training segmentation models on datasets augmented by the synthesized samples leads to significant improvements in accuracy. Furthermore, we introduce a novel boundary label relaxation technique that makes training robust to annotation noise and propagation artifacts along object boundaries. Our proposed methods achieve state-of-the-art mIoUs of 83.5% on Cityscapes and 82.9% on CamVid. Our single model, without model ensembles, achieves 72.8% mIoU on the KITTI semantic segmentation test set, which surpasses the winning entry of the ROB challenge 2018. Our code and videos can be found at https://nv-adlr.github.io/publication/2018-Segmentation.
1. Introduction
Semantic segmentation benefits from large annotated datasets, but pixel-wise labeling is expensive and boundaries are often ambiguous. The paper addresses this by synthesizing image-label training pairs through video prediction, joint propagation, and boundary label relaxation.
- Annotating all pixels in a 1024 × 2048 Cityscapes image takes on average 1.5 hours.
- Video prediction models propagate labels to future unlabeled frames, creating additional image-label training samples.The approach uses both label propagation with original future frames and joint image-label propagation with predicted frames.
- Joint image-label propagation reduces misalignment by propagating frames and labels with the same transformation.
- Boundary label relaxation maximizes the likelihood of the union of neighboring class probabilities, improving robustness to boundary noise and enabling longer-range propagation.
- Training with synthesized samples improves accuracy across several datasets and produces state-of-the-art semantic segmentation results.
- The paper compares video prediction-based propagation with standard optical flow-based approaches using segmentation performance.
2. Related Work
Prior label-propagation methods mainly use patch matching or optical flow, each with practical weaknesses. This work instead uses video-prediction motion vectors and jointly propagates images and labels to reduce misalignment.
- Patch-matching methods are sensitive to patch size and thresholds and may assume prior knowledge of class statistics.
- Optical-flow methods depend on highly accurate flow estimation, which is difficult to achieve.
- Video-prediction motion vectors can handle occlusion, remain class agnostic, and are typically learned through self-supervision.
- Joint image-label propagation reduces misalignment by applying the same propagation process to frames and labels.
3. Methodology
The methodology synthesizes training image-label pairs from sparsely annotated videos using learned motion vectors for label-only or joint image-label propagation. It also relaxes boundary labels so segmentation training tolerates ambiguous annotations and propagation distortions.
- Training Data Synthesis: The approach synthesizes k × m new training samples from videos with labels on only a subset of frames.The propagation length k is applied to each labeled input frame.
- Video Prediction: Video prediction uses a 3D CNN to predict motion vectors from past frames and optical flows, then bilinearly samples the latest frame.The learned vectors account for disocclusions that optical flow alone cannot represent.
- Label Propagation: Label propagation reuses the learned motion vectors to sample past labels and synthesize future labels for new training pairs.The transformation model is pretrained for future-frame prediction and then applied to labels.
- Joint Image-Label Propagation: Joint propagation predicts both frames and labels, pairing them so the same learned transformation produces aligned synthesized samples.This addresses misalignment that occurs when propagated labels are paired with original future frames.
- Joint Image-Label Propagation: Reversed propagation adds past samples, scaling the dataset by a factor of 2k + 1; the method also applies when raw frames and labels are both scarce.Forward and backward propagation are performed from labeled input pairs.
4. Experiments
Experiments evaluate video-based sample synthesis, propagation strategies, boundary label relaxation, and benchmark performance across Cityscapes, CamVid, and KITTI. The results show consistent gains from synthesized samples and relaxation, with strongest reported outcomes on all three benchmarks.
- Stronger Baseline: 1.72% mIoU and 1.14% mIoU gains from Mapillary pre-training and class uniform sampling establish a 79.46% Cityscapes baseline.The gains are measured as 76.60% → 78.32% and 78.32% → 79.46%, respectively.
- Propagation Strategy: 0.8% mIoU improvement from joint propagation exceeds label propagation’s 0.33% gain over the 79.46% baseline.Both methods achieve their highest mIoU at ±1 propagation length.
- Video Reconstruction: 1.08% absolute mIoU improvement over baseline is obtained with video reconstruction, while accuracy begins dropping from ±4 propagation length.Video reconstruction outperforms video prediction at all propagation lengths, with ±1 achieving the best result.
- Boundary Label Relaxation: 0.81% mIoU improvement raises video reconstruction performance from 80.54% to 81.35% at ±3 when boundary label relaxation is used.Relaxation improves mIoU at every propagation length, and the gap over no relaxation widens with longer propagation.
- Boundary Label Relaxation: 1.39% mIoU improvement at zero propagation, from 79.46% to 80.85%, shows boundary label relaxation also benefits ordinary image segmentation.The authors interpret this as handling ambiguous boundary labels beyond propagation artifacts.
- Benchmark Results: 83.5% mIoU on Cityscapes and 3.3% mIoU over previous state-of-the-art on KITTI are reported, using a single model on KITTI.The method also reports significantly better performance than prior methods on all four KITTI evaluation metrics.
5. Conclusion
The paper proposes video-based data synthesis, joint propagation, and boundary label relaxation to scale semantic-segmentation training and mitigate label noise. These methods achieve state-of-the-art results on Cityscapes, CamVid, and KITTI, while relaxation also applies to human annotations.
- Video prediction-based synthesis scales semantic-segmentation training sets, while joint propagation alleviates misalignments and boundary relaxation mitigates label noise.
- 83.5% mIoU on Cityscapes, 82.9% on CamVid, and 72.8% on KITTI are reported as state-of-the-art results.
- Boundary label relaxation can be used with human-annotated labels as well as synthesized labels.
- The authors suggest data augmentation methods such as GANs as a route toward cheaper dataset collection and improved target-task accuracy.
A. Implementation Details of Our Video Prediction/Reconstruction Models
The video prediction model uses a motion-vector predictor and bilinear sampling to generate future frames, with a U-net architecture trained on Cityscapes video crops. A related reconstruction model differs in its input.
- The video reconstruction model has similar architecture and training details to the video prediction model, but uses a different input.
- Future frames are generated by predicting motion vectors from input frames and optical flow, then bilinearly sampling the latest input frame.
- The implementation uses a fully convolutional U-net with 10 encoder layers, 6 decoder layers, skip connections, and a final convolutional layer.
- Training uses Cityscapes frames randomly cropped to 256 × 256, Adam optimization, batch size 128 across 8 V100 GPUs, and 400 epochs.The learning rate is 1 × 10^-4 with weight decay 1 × 10^-4; Adam parameters are β1 = 0.9 and β2 = 0.999.
B. Non-Accumulated and Accumulated Comparison
The comparison evaluates non-accumulated and accumulated synthesized data, alongside a ground-truth baseline, using boundary relaxation and joint propagation on Cityscapes validation. Augmentation improves mIoU, but non-accumulated synthesis outperforms accumulation.
- The accumulated case includes synthesized data from all timesteps up to ±k, making the augmented dataset 2k + 1 times larger than the original.The non-accumulated case uses only synthesized data from timesteps ±k and excludes intermediate synthesized data.
- Table 6 reports the accumulated and non-accumulated comparison with sample standard deviations shown in brackets.
- The experiment compares a ground-truth-only baseline with non-accumulated ±3 data and accumulated ±3, ±2, and ±1 data.
- The configurations use boundary label relaxation and joint propagation, with segmentation accuracy reported on the Cityscapes validation set.
- Augmented datasets always improve mIoU, while the non-accumulated case performs better than the accumulated case.The authors suspect accumulation reduces the probability of sampling hand-annotated examples and overweights synthesized samples and their imperfections.
C.1. More Training Details
The authors use three city-based cross-validation splits and select cv2 for the final test submission because it produced higher test-set performance.
- Three cross-validation splits partition Cityscapes validation cities, with the remaining cities used for training in each split.The splits are cv0, cv1, and cv2; cv0 is the standard validation split.
- The final test submission uses cv2 because models trained on that split produced higher test-set performance.
C.2. Failure Cases
The model fails on class confusions, especially among visually similar categories, and on scenes that are distant, heavily occluded, or structurally ambiguous. It also encounters annotation ambiguities, although it correctly predicts the labels in two shown training samples whose annotations may be wrong.
- The model struggles to distinguish car from truck, person from rider, wall from fence, and terrain from vegetation.
- Objects that are far away, strongly occluded, or overlapping other objects create additional challenging cases.
- A distant object can be difficult to label as bus or train, while more than 95% occlusion makes car-versus-bus prediction difficult.
- A bicycle hanging on a car creates ambiguity about whether it is part of the car, a painting, or a separate object.
- In two potentially mislabeled training samples, the model predicts that the rider is on a motorcycle and that a fence is in front of the building.
C.3. More Synthesized Training Samples
The authors provide synthesized video-based training examples and compare propagation methods under the large inter-frame motion in Cityscapes. Their video reconstruction model performs best qualitatively and experimentally, while the method substantially improves CamVid performance over prior state of the art.
- Each synthesized Cityscapes sample is an 11-frame clip with only the fifth frame labeled and the other ten frames generated.
- The video reconstruction model produces the best propagated frames and labels in visualization and segmentation experiments.
- Cityscapes’ 17Hz recording rate creates large inter-frame motion, making propagation artifacts especially visible near image borders.
- The model achieves the highest mIoU on 8 of 11 CamVid classes, excluding tree, sky, and sidewalk.
- 7.7% mIoU separates the proposed method from previous state of the art on CamVid.
D.2. More Synthesized Training Samples
CamVid demonstrations cover propagation between sparsely annotated frames using forward and backward video reconstruction. The figure summarizes failure cases involving class confusion, distant or occluded objects, and overlapping structures.
- The ‘006E15’ validation sequence is manually annotated every other frame, enabling one-step forward propagation to label an intermediate frame.
- For ‘0001TP’, 15 forward and 14 backward propagation steps label the 29 frames between annotations.
- Both CamVid demonstrations generate synthesized samples with the video reconstruction model.
- The failure-case figure compares an image, ground truth, prediction, and their difference from left to right.
- The displayed failures include class confusion, distant or occluded objects, and overlapping objects.