Source-linked AI summary
Video Scene Parsing with Predictive Feature Learning
Xiaojie Jin, Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang, Yunpeng Chen, Jian Dong, Luoqi Liu, Zequn Jie, Jiashi Feng, Shuicheng Yan
TL;DR
Video scene parsing must produce per-pixel labels across frames, but framewise methods ignore temporal context and finely annotated video data are limited. PEARL learns spatiotemporal features by predicting future frames and parsing maps, then uses them to guide image-parsing networks; it reports state-of-the-art performance on Cityscapes and Camvid.
Problem
Framewise parsing can produce noisy, temporally inconsistent labels, while finely annotated video data are limited compared with the volume of available frames.
Source: §1 · p. 1Method
PEARL learns spatiotemporal features by predicting future frames and available parsing maps from historical frames, then uses those features to steer an image-parsing network.
Source: §1 · p. 1, §3.3 · p. 5Results
PEARL achieves state-of-the-art performance on Cityscapes and Camvid, with the best reported results on both datasets and superiority to temporal methods such as optical flow.
Source: §1 · p. 2Takeaways & Limitations
The framework uses predictive features to guide image-parsing models toward temporally smooth, structure-preserving video scene parsing results.
Source: §1 · p. 1, §5 · p. 9Predictive parsing is challenging because it predicts parsing maps without seeing the frame being parsed; the authors identify improving prediction quality as future work.
Source: §3.2 · p. 4, §3.4 · p. 6
Abstract
from arXiv · showhide
In this work, we address the challenging video scene parsing problem by developing effective representation learning methods given limited parsing annotations. In particular, we contribute two novel methods that constitute a unified parsing framework. (1) \textbf{Predictive feature learning}} from nearly unlimited unlabeled video data. Different from existing methods learning features from single frame parsing, we learn spatiotemporal discriminative features by enforcing a parsing network to predict future frames and their parsing maps (if available) given only historical frames. In this way, the network can effectively learn to capture video dynamics and temporal context, which are critical clues for video scene parsing, without requiring extra manual annotations. (2) \textbf{Prediction steering parsing}} architecture that effectively adapts the learned spatiotemporal features to scene parsing tasks and provides strong guidance for any off-the-shelf parsing model to achieve better video scene parsing performance. Extensive experiments over two challenging datasets, Cityscapes and Camvid, have demonstrated the effectiveness of our methods by showing significant improvement over well-established baselines.
1. Introduction
Limited pixel-level video annotations and framewise parsing that ignores temporal context motivate PEARL. It learns from future-frame prediction and steers image-parsing models, achieving state-of-the-art performance on Cityscapes and Camvid.
1. Introduction
- Framewise image parsing ignores temporal context, causing noisy, inconsistent labels, while pixel-level video annotations are limited relative to the volume of available frames.Cityscapes has 2,975 finely annotated training samples among 180K video frames.Source: §1 · p. 1
- Predictive feature learning trains on historical frames to predict a future frame and, when available, its parsing map, learning spatiotemporal features from unlabeled video.The learned features capture object structures, motions, and other temporal cues.Source: §1 · pp. 1–2
- PEARL achieves state-of-the-art performance on Cityscapes and Camvid, outperforming strong VGG16/Res101 parsing baselines and temporal methods such as optical flow.The paper reports the best results on both datasets.Source: §1 · p. 2
- Prediction steering parsing uses temporal-aware features to guide an image-parsing network, producing temporally smooth, structure-preserving maps with cross-frame consistency and richer local detail.The two parsing networks are jointly trained end-to-end.Source: §1 · pp. 1–2
2. Related Work
Related work spans image-parsing architectures and methods for incorporating temporal information. PEARL is distinguished by using labeled and unlabeled video sequences to learn temporal context for video scene parsing.
2. Related Work
- Framewise image-parsing CNNs, including FCNs and graphical-model combinations, and recurrent models commonly lack cross-frame consistency when temporal information is omitted [21] [3] [44] [29] [12] [26] [32].The section notes that these framewise results can also be low quality.Source: §2 · p. 2
- Temporal approaches use 3D motion and structure features, spatiotemporal CRFs [23], or optical flow [11] [30], with CRF inference described as computationally expensive.PEARL instead uses both labeled and unlabeled video sequences to learn richer temporal context.Source: §2 · pp. 2–3
- Although PEARL also uses adversarial training for frame prediction [22] [24], it applies the resulting spatiotemporal features to video scene parsing.The cited frame-prediction methods are presented as related frame-prediction work.Source: §2 · p. 3
3. Predictive Feature Learning for VSP
The approach uses past video frames to learn spatiotemporal features and adapt them for parsing, addressing temporal inconsistency and limited pixel-level annotations. It also explores fully unsupervised feature learning and future parsing.
3.1. Motivation and Notations
- The framework targets cross-frame consistency and effective parsing despite insufficient training data, using causal inference based only on past frames.This setting is described as better aligned with real-time applications where future frames cannot be seen in advance.Source: §3.1 · p. 3
3.2. Predictive Feature Learning
- FPNet generates future frames from past frames rather than random noise, with alternating generator and discriminator training combining reconstruction and adversarial supervision.The discriminator distinguishes predicted frames from real ones, while the generator is trained to produce similar and authentic predictions.Source: §3.2 · p. 4
- FPNet predictions retain object and stuff structures and moving-car motion, supporting the claim that frame prediction learns spatiotemporal features.The examples are described as visually similar to ground-truth frames.Source: §3.2 · p. 4, Figure 3 · p. 5
- PPNet predicts a future parsing map from past frames; its encoder is initialized from FPNet, and a deconvolution network produces the map.Training PPNet from scratch is reported to risk overfitting because annotated data are insufficient.Source: §3.2 · p. 4
- PPNet maps are temporally smooth but tend to miss small objects such as signs and poles, partly because blurry predictions smooth high-frequency details.The authors suggest multi-scale parsing as a possible mitigation and future investigation.Source: §3.2 · p. 5
3.3. Prediction Steering Parsing
- PSPNet connects PPNet and a framewise IPNet through AdapNet, adapting temporal features for parsing while combining temporal context with per-pixel discrimination.AdapNet adjusts feature norm, scale, and feature space before the IPNet decoder, easing the transformation to parsing maps.Source: §3.3 · pp. 5–6
3.4. Discussion
- FPNet* VGG11, trained fully unsupervised, performs similarly to FPNetVGG16 initialized with pretrained weights; deeper unsupervised models remain future work.The discussion also identifies predicting parsing maps seconds ahead as a potential task, while calling for higher-quality and longer-term predictions.Source: §3.4 · p. 6
4. Experiments
PEARL outperforms the reported baselines on Cityscapes and Camvid, with ablations supporting contributions from predictive features, joint training, and AdapNet. The experiments also report qualitative parsing improvements and give dataset, metric, and inference settings.
4.1. Settings and Implementation Details
- Cityscapes evaluation uses finely annotated frames from 30-frame clips; reported metrics are mIoU for Cityscapes and PA and CA for Camvid.The dataset passage reports 2,975/500/1,525 finely annotated train/val/test frames and 180K total video frames; PEARL uses only the finely annotated frames for training.Source: §4.1 · p. 6
4.2. Results
- PEARL’s Cityscapes parsing maps are described as smoother and more discriminative for small objects than the VGG16 baseline’s maps.The examples compare maps for frame T+4; the stated improvements are attributed to jointly learning temporally consistent and locally discriminative features.Source: §4.2 · p. 7, Figure 5 · p. 8
4.2.1 Cityscapes
- Cityscapes ablations report a rise from 63.4 to 68.6 mIoU with FPNet features, a further 0.5 mIoU with PPNet features, and the best performance after joint training.Adding AdapNet improves PEARLVGG16/PEARLRes101 by 1.1/0.3 mIoU; deeper AdapNet layers yield only marginal gains at greater computation cost.Source: §4.2.1 · pp. 7, 9
- On Cityscapes test, PEARLRes101 achieves the best performance among the reported top methods, using single-scale testing without CRF post-processing.The paper reports 0.8 seconds per 1,024×2,048 frame on a modern GPU, with no post-processing required.Source: §4.1 · p. 7, Table 3 · p. 8, §4.2.1 · p. 9
- PEARLVGG16 and PEARLRes101 improve their corresponding Cityscapes val baselines by 6.4/2.4 mIoU, respectively, and PEARLVGG16 exceeds the temporal skip network [31] by 5.4 mIoU.PEARL achieves the best performance among compared methods on both network architectures.Source: Table 4 · p. 8, §4.2.1 · p. 9
4.2.2 Camvid
- On Camvid, PEARL improves the Res101-baseline by 1.6% PA and 2.5% CA, and outperforms the reported competing methods.The comparison uses loss re-weighting and reports advantages over optical-flow methods and [20].Source: Table 5 · p. 9, §4.2.2 · p. 9
5. Conclusion
The conclusion presents PEARL as a two-component approach to video scene parsing, with experiments on Cityscapes and Camvid demonstrating its effectiveness.
5. Conclusion
- PEARL combines predictive feature learning from future-frame and parsing-map prediction without extra annotations with prediction steering that guides single-frame parsing toward temporally smooth, structure-preserving results.The conclusion reports effectiveness in experiments on Cityscapes and Camvid.Source: §5 · p. 9
A.1. More Results of Frame Prediction from FPNet in PEARL
The appendix points readers to Figure 6 for additional FPNet frame-prediction results.
A.1. More Results of Frame Prediction from FPNet in PEARL
- Additional FPNet frame-prediction results are referenced in Figure 6.Source: §A.1 · p. 11
A.2. More Results of Video Scene Parsing
The appendix shows FPNet frame predictions and PEARL video-scene parsing examples. The figures contrast predicted frames with ground truth and illustrate parsing consistency across a sequence.
A.2. More Results of Video Scene Parsing
- Figure 6 presents eight Cityscapes video sequences, with eight ground-truth frames above the corresponding FPNet predictions.The caption says the predictions model object and stuff structures and moving-object motion.Source: Figure 6 · p. 12
- Figure 9 contrasts framewise VGG16-baseline maps, PEARL predictive parsing maps, and combined PEARL maps against the target ground-truth map.The caption describes inconsistent regions in the baseline as classified consistently by predictive parsing, and says the combined maps have better accuracy and temporal consistency.Source: Figure 9 · p. 15