PaperlayerView exact revision on arXiv

Source-linked AI summary

NoScope: Optimizing Neural Network Queries over Video at Scale

Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, Matei Zaharia

arXiv:1703.02529v3cs.DBcs.CV

TL;DR

Large-scale video analysis is difficult because accurate neural-network inference is computationally expensive. NOSCOPE searches for video- and object-specific cascades of cheaper models, achieving two- to three-order-of-magnitude speedups on fixed-angle binary classification while losing 1-5% accuracy.

  • Problem

    Applying accurate neural networks to video at scale is prohibitively expensive, with real-time object detection requiring costly dedicated GPUs.

  • Method

    NOSCOPE automatically searches for and trains cascades of specialized models and difference detectors for a target video, object, reference network, and accuracy target.

  • Results

    Two- to three-order-of-magnitude speedups were demonstrated for binary classification on fixed-angle video, with 1-5% accuracy loss.

  • Takeaways & Limitations

    Inference-optimized model architecture search can support neural-network analysis of large video datasets at substantially lower computational cost.

  • Takeaways & Limitations

    The prototype is limited to binary classification on fixed-angle video, and model drift requires re-optimization when the scene distribution changes.

Abstract

from arXiv · show

Recent advances in computer vision-in the form of deep neural networks-have made it possible to query increasing volumes of video data with high accuracy. However, neural network inference is computationally expensive at scale: applying a state-of-the-art object detector in real time (i.e., 30+ frames per second) to a single video requires a $4000 GPU. In response, we present NoScope, a system for querying videos that can reduce the cost of neural network video analysis by up to three orders of magnitude via inference-optimized model search. Given a target video, object to detect, and reference neural network, NoScope automatically searches for and trains a sequence, or cascade, of models that preserves the accuracy of the reference network but is specialized to the target video and are therefore far less computationally expensive. NoScope cascades two types of models: specialized models that forego the full generality of the reference model but faithfully mimic its behavior for the target video and object; and difference detectors that highlight temporal differences across frames. We show that the optimal cascade architecture differs across videos and objects, so NoScope uses an efficient cost-based optimizer to search across models and cascades. With this approach, NoScope achieves two to three order of magnitude speed-ups (265-15,500x real-time) on binary classification tasks over fixed-angle webcam and surveillance video while maintaining accuracy within 1-5% of state-of-the-art neural networks.

1. INTRODUCTION

NOSCOPE addresses the high cost of applying neural networks to large-scale video by specializing models and searching for efficient cascades. On fixed-angle binary classification tasks, it reports large speedups while preserving reference-model accuracy.

  • Motivation: 30-80 fps is the throughput of fast accurate object detectors, making real-time video analysis costly at scale.The cited hardware examples include GPUs costing approximately $4000-$4600.
  • Approach: NOSCOPE searches for and trains query-specific cascades that approximate a reference network while sacrificing generality for lower computational cost.The cascade is specialized to the target video, object, and desired accuracy.
  • Results: 265-15,500× real-time speedups were achieved with 1-5% accuracy loss on binary detection tasks over fixed-angle webcam and surveillance video.The evaluation reports two- to three-order-of-magnitude acceleration while retaining high, configurable accuracy.
  • Model specialization: Specialized neural networks mimic the reference model for a target video and object, reaching up to 340× faster inference.They consult the full reference network when their confidence is below an automatically learned threshold.
  • Difference detection: Difference detectors exploit temporal similarity between frames to avoid repeatedly running expensive neural networks.They determine whether frame contents have changed and can process up to 100k frames per second on a CPU.
  • Optimization: The optimal cascade depends on the video and object, so NOSCOPE uses cost-based optimization to select models and thresholds for an accuracy target.The optimizer evaluates candidate models and sweeps feasible threshold combinations efficiently.

2. BACKGROUND

The background introduces object detection and explains why deep neural networks became dominant despite their computational demands. It also outlines neural-network structure, training, and frame-by-frame video inference.

  • Object Detection History and Goals: Object detection extracts object occurrences and locations from image data, building on classical machine-learning and image-featurization techniques.The section traces the task to at least the 1960s and mentions methods such as SIFT, HOG, deformable parts models, and selective search.
  • Object Detection History and Goals: Deep neural networks improved computer-vision accuracy to near-human or better-than-human levels and became preferred for tasks including classification and pedestrian detection.These models contain millions to billions of parameters.
  • NN Architecture: A convolutional neural network processes high-dimensional images through connected layers that produce simpler representations and predictions.The described layers include convolution, pooling, ReLU, and fully connected layers.
  • NN Architecture: The best neural-network architecture varies by task, but the field has generally moved toward deeper models as more training data becomes available.
  • NN Training: Training fits network weights to minimize empirical error on labeled data, while video object-detector training requires labeling frames by hand.The background notes that training is computationally expensive.
  • NN Inference: Video inference evaluates a neural network repeatedly, processing individual frames one at a time.This approach is convenient because neural networks are usually trained on static images, but it is expensive.

3. NOSCOPE ARCHITECTURE

NOSCOPE accepts a target object, video, and reference model for binary presence-or-absence queries. It learns a cascade of specialized models and difference detectors, then uses confidence and cost optimization to accelerate processing.

  • NOSCOPE Queries and Goal: NOSCOPE targets binary classification queries that identify whether a specified object class is present or absent over time.Users select an object class and target video, while the reference model supplies the desired labels.
  • System Components: The architecture has three components: specialized models, difference detectors, and an inference-optimized cost-based optimizer.The system first labels a subset of the video with the reference model, learns a cheaper cascade, and applies it to the remainder.
  • System Components: Specialized models output confidence values and can classify frames cheaply, passing uncertain cases to the full reference model.Low and high confidence thresholds determine negative, positive, or deferred decisions.
  • System Components: Difference detectors compare the current frame with a recent labeled frame to determine whether the contents have changed.
  • System Components: The cost-based optimizer configures filters to meet target false-positive and false-negative rates, because cascade customization is critical for performance.

4. MODEL SPECIALIZATION

NOSCOPE specializes smaller neural networks to mimic a reference model for restricted video tasks, trading generality for efficient inference. Confidence thresholds and model search determine when to accept specialized predictions or defer to the reference model.

  • MODEL SPECIALIZATION: Specialized models are smaller networks that faithfully mimic a reference model on a particular task rather than across all tasks.They exploit the limited object classes and viewpoints often present in a target video.
  • MODEL SPECIALIZATION: NOSCOPE trains specialized models from reference-model outputs on a target video, reducing computational resources while approximating the reference behavior.
  • MODEL SPECIALIZATION: Specialized models use confidence thresholds to classify clear cases and defer uncertain frames to the full reference network.The lower threshold produces a negative result, the upper threshold a positive result, and intermediate confidence triggers reference-model inference.
  • MODEL SPECIALIZATION: The optimizer trades model complexity and threshold width against speed and accuracy when selecting a specialized cascade.A less accurate but faster model may be preferable if it passes more frames to the full model.
  • Results: 90% of frames can regularly be eliminated without calling the full reference model while preserving its accuracy.The result depends on appropriately setting the confidence thresholds.
  • Results: Scene-specific training data leads to better specialized-model performance than generic object-detection datasets.

5. DIFFERENCE DETECTION

NoScope uses efficient difference detectors to identify when video-frame labels may change, avoiding redundant neural-network evaluations. It combines temporal comparisons, configurable metrics, and video-dependent thresholds to trade accuracy for speed.

  • Difference detectors compare a labeled frame with an unlabeled frame to determine whether their object labels differ.
  • Up to 90× inference speedups are possible when video frame rates greatly exceed label-change rates.
  • NoScope supports fixed-reference and earlier-frame comparisons, with the optimizer selecting between them according to the video.
  • Mean Square Error compares frames globally or by grid blocks, where logistic regression can weight informative regions more heavily.
  • The difference threshold δdiff and frame-skipping interval tskip are configurable because videos and queries impose different speed, false-positive, and false-negative trade-offs.
  • NoScope evaluates MSE and blocked MSE because they generally perform within a few percent of the best alternative metrics.

6. COST-BASED MODEL SEARCH

NoScope searches for a low-cost cascade of difference detectors and specialized models while enforcing target false-positive and false-negative rates. Its optimizer estimates costs from measured selectivities and searches architectures and thresholds jointly.

  • The cost-based optimizer maximizes expected throughput subject to false positive rate < FP* and false negative rate < FN*.
  • NoScope searches model architectures, trains candidate models, and sets their parameters together rather than merely ordering fixed filters.
  • The optimizer labels sampled frames with the reference model, then splits them into training and evaluation sets for model selection.
  • For each candidate cascade, the cost model combines pass-through fractions with per-frame execution times for the detector, specialized NN, and reference NN.The expected time per frame is fsT_MSE + fsfmT_SpecializedNN + fsfmfcT_FullNN.
  • The search trains filters individually, profiles their selectivity and sensitivity, and then examines feasible threshold combinations jointly.
  • The CBO complexity is O(n_d n_c n_t), and the considered configuration counts are typically below 100, making searches often take only a few seconds.

7. IMPLEMENTATION

The prototype combines C++, TensorFlow, Python, OpenCV, multithreading, vectorized operations, and GPU batching to execute NoScope’s cascade efficiently. Runtime applies difference detection, specialized inference, and reference-model fallback in sequence.

  • The prototype uses C++ and TensorFlow for inference and Python for the cost-based optimizer.
  • Parallelized CPU operations, vectorized libraries, and GPU batching improve performance because difference detection and specialized inference eliminate many frames.
  • NoScope executes a difference detector, applies the specialized NN to passing frames, and uses the reference model on frames not confidently labeled.
  • YOLOv2 parameters remain loaded on the GPU across evaluations to avoid repeatedly reloading the large reference model.
  • Frames are loaded and resized with OpenCV, while difference detectors use hand-tuned C++ and frame-level parallelism.
  • Blocked difference detectors use scikit-learn logistic regression weights, and specialized NNs use cross-validation, early stopping, and separate evaluation data.

8. LIMITATIONS

NoScope’s current evaluation and implementation are scoped to fixed-angle video and binary classification, with additional constraints from distribution shift and batching latency. The authors describe extensions for moving cameras, broader queries, and automated drift handling as future directions.

  • The prototype’s difference detectors and specialized CNN results are limited to object classification in fixed-angle video from static cameras.
  • NoScope has been evaluated only on binary classification tasks, although the authors suggest its techniques may extend to more complex boolean queries.
  • The optimizer assumes training data and subsequent video share a distribution; major scene changes require re-optimization.
  • Periodic scene changes require training data that represents all relevant conditions, such as day and night.
  • Batching improves efficiency for historical video but can delay live processing; batching 100 frames at 30 fps can add 3.3 seconds.

9. EVALUATION

NOSCOPE substantially improves video-query throughput while preserving high agreement with YOLOv2, but its performance depends strongly on video-specific cascade choices. Evaluation shows that difference detection, specialized models, and cost-based optimization each contribute to efficient inference.

  • End-to-End Performance: 40-5400× higher throughput is achieved than state-of-the-art NN models at 99% of their accuracy, while 100-10000× is achieved at 90+% accuracy.These results are reported for binary classification on real-world webcam and surveillance videos.
  • End-to-End Performance: 30× speedup is achieved with at least 98% accuracy across all datasets, while many datasets retain this accuracy at 100× speedup.At 90+% accuracy, NOSCOPE often reaches 1000× to 10,000× speedups; even taipei reaches 30× with no accuracy loss.
  • Impact of the CBO: Optimal filter configurations vary substantially across videos, and transferring a cascade between datasets generally produces poor performance.The CBO selects different filter types, thresholds, and specialized architectures because video dynamics and target-object difficulty differ.
  • Impact of the CBO: Difference detection and specialized models improve throughput by up to 3× and 340× respectively, while frame skipping contributes up to 30×.The lesion study further finds that each cascade element contributes to overall throughput.
  • Impact of Model Specialization: Video-specific specialized models outperform same-sized models trained on MS-COCO by up to 20× across evaluated videos.The comparison supports using scene characteristics such as fixed perspective and background during specialization.
  • Comparison Against Baselines: NOSCOPE is nearly a thousand times faster than several classical baselines while achieving higher accuracy, whereas non-specialized deep networks do not reach comparable accuracy.The evaluation compares classical computer-vision methods and non-specialized deep networks using frame skipping.

10. RELATED WORK

NOSCOPE builds on visual data management, computer vision, model-management, and cascade research while introducing video-specific specialization and cost-based optimization. Its related-work distinction is exploiting temporal locality and query simplicity to trade generality for inference speed.

  • Visual Data Management: Video data-management systems have long supported storing, querying, and managing visual data, initially using classical computer vision techniques.
  • Model Cascades: NOSCOPE adapts classifier cascades to video by combining temporal difference detectors with query-specific model specialization.
  • Model Cascades: NOSCOPE’s cost-based optimizer jointly selects cascade filters and thresholds for the video stream and reference neural network, unlike approaches optimizing only accuracy.
  • Video Object Detection/Extraction and Tracking: Modern video object-detection systems commonly run still-image detection on every frame and prioritize accuracy rather than temporal locality or inference speed.
  • Video Object Detection/Extraction and Tracking: NOSCOPE addresses binary classification, while extending the system to object detection is identified as future work.
  • Improving Deep Network Speed: Model compression and distillation reduce neural-network size or evaluation time, whereas NOSCOPE specializes inference to a target video and task.

11. CONCLUSIONS

Applying neural networks to video at scale is prohibitively expensive, so NOSCOPE accelerates inference through specialized models, temporal difference detectors, and cost-based cascade optimization. On fixed-angle video, the prototype achieves two- to three-order-of-magnitude speedups with limited accuracy loss.

  • NOSCOPE accelerates video inference by combining specialized models and difference detectors in a cascade.
  • Two- to three-order-of-magnitude speedups with 1-5% accuracy loss were demonstrated for binary classification on fixed-angle video streams.
Loading 1703.02529v3…