PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Model Distillation for Efficient Video Inference

Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, Kayvon Fatahalian

arXiv:1812.02699v2cs.CV

TL;DR

Most cameras observe narrower, changing image distributions than general-purpose vision models target, motivating adaptation to live video. The paper uses online distillation to specialize a compact segmentation student with intermittent teacher supervision. The resulting models closely approximate their Mask R-CNN teacher at substantially lower inference cost, including on non-stationary video.

  • Problem

    Most cameras capture a narrow, changing subset of real-world images, unlike the general distributions targeted by many computer-vision models.

    Source: §1 · p. 1

  • Method

    The paper adapts compact semantic-segmentation models online on live video as frames arrive, using model distillation.

    Source: §1 · p. 1

  • Results

    7 to 17× lower inference runtime cost (11-26× in FLOPs) lets online-distilled models closely approximate their Mask R-CNN teacher, even on non-stationary video.

    Source: §1 · p. 1

  • Takeaways & Limitations

    For common real-world video streams, compact models can achieve semantic-segmentation accuracy comparable to an expensive, high-capacity teacher.

    Source: §6 · p. 8

    Fast-motion blur can make teacher predictions unreliable and disrupt updates, while short-lived objects in busy scenes call for faster online adaptation.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

High-quality computer vision models typically address the problem of understanding the general distribution of real-world images. However, most cameras observe only a very small fraction of this distribution. This offers the possibility of achieving more efficient inference by specializing compact, low-cost models to the specific distribution of frames observed by a single camera. In this paper, we employ the technique of model distillation (supervising a low-cost student model using the output of a high-cost teacher) to specialize accurate, low-cost semantic segmentation models to a target video stream. Rather than learn a specialized student model on offline data from the video stream, we train the student in an online fashion on the live video, intermittently running the teacher to provide a target for learning. Online model distillation yields semantic segmentation models that closely approximate their Mask R-CNN teacher with 7 to 17$\times$ lower inference runtime cost (11 to 26$\times$ in FLOPs), even when the target video's distribution is non-stationary. Our method requires no offline pretraining on the target video stream, achieves higher accuracy and lower cost than solutions based on flow or video object segmentation, and can exhibit better temporal stability than the original teacher. We also provide a new video dataset for evaluating the efficiency of inference over long running video streams.

1. Introduction

Most cameras observe a narrower, continuously changing image distribution than general-purpose vision models target. The paper trains compact segmentation students online on live video, using intermittent teacher supervision to adapt them to each stream.

1. Introduction

  • The approach moves away from pretraining on camera-specific datasets curated in advance, instead adapting models as new frames arrive.Source: §1 · p. 1
  • Online model distillation adapts a lightweight student to live video frames, intermittently running a high-capacity teacher to provide learning targets.The student is designed for low-cost inference and fast training, with teacher calls chosen judiciously.Source: §1 · p. 1
  • Online model distillation produces semantic segmentation models that closely approximate their Mask R-CNN teacher with 7 to 17× lower inference runtime cost (11-26× when comparing FLOPs), even when the target video distribution is non-stationary over time.The student delivers higher-accuracy output than low-cost video segmentation solutions based on flow, and its output can be preferable to the teacher in temporal stability.Source: §1 · pp. 1–2
  • The paper introduces a video dataset for evaluating inference efficiency over long-running streams.Source: §1 · p. 2

2. Related Work

The paper positions online distillation as a way to specialize compact models to changing video streams, alongside research on online learning, tracking, and temporal coherence. It distinguishes this goal from prior approaches that target other settings or incur different costs.

Distillation for specialization

  • Distillation for specialization trains compact models to mimic a teacher on a desired data subset, while this work adapts the student online to changing video contents.Prior specialization methods use careful configuration or sample selection to avoid missing rare events; prior online distillation work also creates domain-specialized models.Source: §2 · p. 2

Online training

  • Online training on arriving video frames violates traditional stochastic gradient descent’s i.i.d. assumption, and the paper demonstrates standard gradient descent for compact-model training on correlated streams.Reinforcement-learning approaches [37] are described as computationally expensive, sample-intensive, and largely offline.Source: §2 · p. 2

Tracking

  • Traditional and deep-feature object tracking learn appearance models rapidly, but track bounding boxes rather than pixel-level semantic masks.The paper’s approach models the whole scene rather than individual objects.Source: §2 · p. 2

Video object segmentation

  • Video object segmentation methods adapt high-capacity models from reference masks for short clips, whereas this work approximates a pretrained general model at low cost.Those methods require seconds to minutes of training per short clip, which the paper says precludes real-time use.Source: §2 · p. 2

Temporal coherence in video

  • For semantic segmentation, specializing a compact model on recent frames is reported as both more accurate and more efficient than flow-based temporal-coherence methods across a wide range of videos.Source: §2 · p. 2

3. Just-In-Time Model Distillation

JITNet is a compact student designed for efficient inference and rapid adaptation to live video. The method trains it from teacher predictions and adjusts the timing of teacher supervision using recent student accuracy.

3. Just-In-Time Model Distillation

  • The online-distillation pipeline runs a compact student on each frame and periodically adapts it using predictions from a high-quality teacher.The student must support fast inference and adaptation, learn from imperfect teacher output, and determine when to request supervision.Source: §3 · pp. 2–3

3.1. JITNet Architecture

  • JITNet is a compact encoder-decoder with three modified ResNet blocks, separable filters, limited high-resolution channels, and skip connections for gradient propagation.The architecture is designed for efficient high-resolution inference and fast learning; Table 1 reports 47× fewer parameters and 91× fewer inference FLOPS than MRCNN.Source: §3.1 · p. 3, Table 1 · p. 3

3.2. Online Training with Gradient Descent

  • Online training addresses correlated frames, shifting content, and potentially unstable or erroneous teacher predictions; instance-weighted cross-entropy focuses learning on objects and boundaries.The method weights pixels inside instance boxes dilated by 15% five times more than pixels outside them, and updates all layers using high learning rates and momentum 0.9.Source: §3.2 · pp. 3–4

3.3. Adaptive Online Distillation

  • Adaptive distillation uses recent teacher labels to validate JITNet, updates it when mean IoU falls below a threshold, and doubles or halves the teacher-call stride based on whether the target is met.The accuracy threshold is the only user-exposed knob; experiments use strides of 8 to 64 frames and at most 8 updates per frame.Source: §3.3 · p. 4

4. Long Video Streams (LVS) Dataset

The Long Video Streams dataset is designed to evaluate efficient inference in extended, varied video-streaming settings. It provides long HD videos spanning fixed and moving camera viewpoints, with teacher predictions used because exhaustive human labels are impractical.

4. Long Video Streams (LVS) Dataset

  • Existing large-scale video datasets largely comprise curated clips lasting seconds to a couple of minutes, while LVS targets long-running real-world camera scenarios.The motivating deployment settings include retail checkout, sports analysis, traffic monitoring, and wearable-device video analysis.Source: §4 · p. 4
  • LVS contains 30 HD videos, each 30 minutes long and at least 720p, spanning fixed cameras, moving and zooming TV cameras, handheld video, and egocentric video.The dataset totals 900 minutes, compared with 345 minutes for YouTube-VOS.Source: §4 · p. 4
  • Because human segmentations are impractical for all 900 minutes, the dataset provides MRCNN predictions for every frame to support evaluation.The authors report evaluating other segmentation models and selecting MRCNN for the highest-quality labels.Source: §4 · pp. 4–5

5. Evaluation

The evaluation compares online distillation with motion-based interpolation and video object segmentation on long video streams, measuring segmentation accuracy and cost relative to Mask R-CNN. It also compares online adaptation with offline specialization and evaluates selected methods on the LVS dataset.

5.1. Experimental Setup

  • Accuracy is measured by mean IoU over evaluated classes, excluding background, on independently moving classes with reliable Mask R-CNN predictions.Results generally cover the first 30,000 frames; car, truck, and bus are combined as “auto” for training and evaluation.Source: §5.1 · p. 5

5.2. Accuracy vs. Efficiency of Online Distillation

  • 7.5× runtime speedup with 82.5 mean IoU for JITNet 0.9 across sequences; JITNet 0.7 achieves 17.4× speedup with 75.5 mean IoU.JITNet 0.9 provides 4.9× speedup with 75.0 mean IoU on challenging egocentric sports videos; higher accuracy thresholds use more teacher-supervised frames.Source: §5.2 · pp. 6–7

5.3. Comparison with Offline Oracle Specialization

  • JITNet 0.9 is more accurate on average than the offline oracle while using 8.4% of frames for supervision, versus 20% for oracle training.On the Elephant stream, online JITNet has a smaller accuracy drop when the elephant enters water; both methods struggle on visually diverse Driving video, where online adaptation consistently performs better.Source: §5.3 · p. 7

5.4. Comparison with Motion-Based Interpolation

  • JITNet 0.7 is faster and more accurate than the fast flow variant, while JITNet 0.9 has significantly higher accuracy than the slow flow variant except in Driving/Walking.The slower flow baseline performs well on rare classes in Driving (Bike) and Walking (Auto); flow and online adaptation could be combined to handle rapid appearance shifts.Source: §5.4 · p. 8

5.5. Comparison with Video Object Segmentation

  • JITNet 0.8 is more accurate than OSVOS and two orders of magnitude faster on LVS.OSVOS is evaluated on two-minute segments and one class per video, whereas JITNet segments all classes; continuously adapted OSVOS performs worse than its re-initialized variant.Source: §5.5 · p. 8, Table 3 · p. 8

6. Conclusion

The conclusion presents online distillation of compact models as viable for semantic segmentation in common real-world video streams. It calls for further exploration of online learning and points to the Long Video Streams dataset as a resource for that work.

6. Conclusion

  • For common real-world video streams, online distillation can produce compact models with semantic segmentation accuracy comparable to an expensive high-capacity teacher.The authors hope these results encourage exploration of online distillation for domain adaptation and self-supervised learning.Source: §6 · p. 8
  • The authors argue that models could continuously evolve using data captured from specific video streams rather than relying only on offline, carefully curated datasets.They identify the Long Video Streams dataset as a resource for this line of research.Source: §6 · p. 8
  • The paper identifies online distillation as a direction for future work in domain adaptation and self-supervised learning.Source: §6 · p. 8

7. Supplementary

The supplementary material reports ablations of online distillation, compares student architectures, and evaluates JITNet configurations on DAVIS. It also describes video examples, offline training details, standalone segmentation performance, and per-stream results across accuracy thresholds.

7.1. Qualitative Comparison Videos

  • The qualitative comparison provides one-minute LVS clips with Mask R-CNN and JITNet 0.9 predictions overlaid, subsampled by 4× temporally.Source: §7.1 · p. 9

7.2. Online Distillation Ablation Study

  • A 0.001 learning rate reduces accuracy and speedup, while 0.1 destabilizes training; more updates trade accuracy and teacher samples for speedup, and stride changes affect accuracy.The study varies one parameter at a time from the JITNet 0.8 baseline, which uses 8 maximum updates, minimum stride 8, and learning rate 0.01.Source: §7.2 · p. 9
  • Width 0.5 requires more teacher supervision and loses accuracy; doubling width improves accuracy but lowers speedup, while 75% resolution retains high accuracy and removing skips reduces accuracy and speedup.Across the tested settings, the algorithm is reasonably robust to parameter changes, except very high learning rates that destabilize training.Source: §7.2 · p. 9, Table 4 · p. 10

7.3. MobileNet Student

  • JITNet has a better accuracy-efficiency spectrum for online distillation than MobileNetV2; even 75%-scale JITNet has higher accuracy and speedup than the MobileNetV2 student.The authors favor FLOPs for platform-agnostic comparison; reported V100 inference times are 38 ms for MobileNetV2 and 7 ms for JITNet.Source: §7.3 · p. 9

7.4. DAVIS Evaluation

  • On DAVIS 2016, JITNet A performs similarly to OFL, JITNet B is comparable to OSVOS at significantly lower runtime cost, and slightly noisy supervision suffices for distillation.JITNet B adapts from OSVOS-S predictions every 16 frames; JITNet C adapts from ground-truth masks at the same interval and performs comparably to using OSVOS-S predictions alone.Source: §7.4 · p. 10

7.5. Offline Training Details

  • COCO pretraining uses all 80 classes and 30 epochs; offline oracle models are then specialized per video using datasets of about 6,000 images, 20× smaller than COCO.Source: §7.5 · pp. 10–11

7.6. Standalone Semantic Segmentation

  • A JITNet architecture with twice the channels and encoder-decoder blocks is 1.5× faster than MobileNetV2, with mean IoU 67.34 versus 70.71; online experiments use a still smaller model.The larger JITNet values are reported on the Cityscapes validation set.Source: §7.6 · p. 11

7.7. Additional Results

  • The additional-results section presents each evaluated video stream at JITNet accuracy thresholds 0.7, 0.8, and 0.9.Source: §7.7 · p. 11

Related papers

Loading 1812.02699v2…