PaperlayerView exact revision on arXiv

Source-linked AI summary

BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics

Daniel Kang, Peter Bailis, Matei Zaharia

arXiv:1805.01046v2cs.DB

TL;DR

Large-scale video analytics is limited by expensive neural-network inference, complex deployment code, and prior methods’ inability to support aggregation and limit queries. BlazeIt addresses these gaps with declarative FrameQL queries and specialized-NN optimizations, reporting up to 83× speedups while retaining accuracy guarantees. The system targets batch video analytics and requires a target-model preprocessing step; distribution drift can reduce performance without affecting accuracy.

  • Problem

    Expensive DNN inference, complex imperative deployment, and inadequate support for aggregation and limit queries hinder large-scale video analytics.

  • Method

    BlazeIt combines the declarative FrameQL language with specialized-NN optimizations, including control variates for aggregation and prioritized search for limit queries.

  • Results

    BlazeIt reports up to 83× speedups over prior video-analytics work while retaining accuracy guarantees despite potentially inaccurate specialized NNs.

  • Takeaways & Limitations

    BlazeIt supports aggregation, limit, and selection queries over large video datasets through declarative optimization rather than full object-detection materialization.

  • Takeaways & Limitations

    In streaming settings with distribution shift, specialized-NN accuracy may degrade and queries may run more slowly, although accuracy guarantees remain.

Abstract

from arXiv · show

Recent advances in neural networks (NNs) have enabled automatic querying of large volumes of video data with high accuracy. While these deep NNs can produce accurate annotations of an object's position and type in video, they are computationally expensive and require complex, imperative deployment code to answer queries. Prior work uses approximate filtering to reduce the cost of video analytics, but does not handle two important classes of queries, aggregation and limit queries; moreover, these approaches still require complex code to deploy. To address the computational and usability challenges of querying video at scale, we introduce BlazeIt, a system that optimizes queries of spatiotemporal information of objects in video. BlazeIt accepts queries via FrameQL, a declarative extension of SQL for video analytics that enables video-specific query optimization. We introduce two new query optimization techniques in BlazeIt that are not supported by prior work. First, we develop methods of using NNs as control variates to quickly answer approximate aggregation queries with error bounds. Second, we present a novel search algorithm for cardinality-limited video queries. Through these these optimizations, BlazeIt can deliver up to 83x speedups over the recent literature on video processing.

1 Introduction

BLAZEIT addresses the usability and computational costs of large-scale video analytics with declarative queries and optimizations for aggregation and limit queries. Its techniques reduce object-detection work while preserving exact answers or accuracy guarantees.

  • Motivation: Cheap cameras and DNNs have expanded batch video analytics, but processing large stored video collections remains challenging.Object detectors produce object positions and classes for each frame, enabling later analysis of collected video.
  • Challenges: 3 frames per second is the reported speed of Mask R-CNN, implying 8 GPU-decades to process 100 camera-months of video.Naive deployment also requires complex imperative programming across low-level libraries.
  • Prior Work: Approximate filtering methods do not handle aggregation and limit queries and can be inefficient when target objects are frequent.They also continue to require non-expert users to write complex deployment code.
  • BLAZEIT: FRAMEQL extends SQL with video-specific functionality, letting BLAZEIT optimize object queries without fully materializing the virtual relation.The relation represents object positions and classes and supports selection, aggregation, and limit queries.
  • Optimizations: BLAZEIT uses specialized NNs as control variates for aggregation and prioritizes likely event-containing frames for cardinality-limited queries.The aggregation method uses correlation with the target statistic, while the limit-query method searches promising frames first.
  • Results: Up to 14× and 83× improvement over prior work is reported for aggregation and limit queries, respectively.The end-to-end optimizer and execution engine avoids object detection where possible while maintaining user-specified accuracy.

2 Example Use Cases

The examples show how FRAMEQL expresses practical video-analytics tasks over objects, timestamps, predicates, and user-defined functions. They span traffic, autonomous vehicles, retail planning, and ornithology.

  • Urban planning: Urban planning queries count cars, identify busy times, and select events containing at least one bus and five cars.The example requires 10 events separated by a temporal gap.
  • Limit queries: GROUP BY with HAVING selects timestamps containing at least one bus, while GAP 300 spaces returned events by 300 frames.At 30 fps, GAP 300 corresponds to 10 seconds.
  • Aggregation: FCOUNT expresses a frame-averaged car count with specified error and confidence bounds.The example query requests maximum absolute error 0.1 at 95% confidence.
  • Selection: FRAMEQL can select red buses using redness and area predicates, track duration, and a noise-reduction constraint.The example requires area greater than 100,000 pixels and presence for more than 15 frames.
  • FRAMEQL examples: Figure 2 presents three FRAMEQL examples whose syntax is largely standard SQL.The examples cover aggregation, temporally spaced event selection, and predicate-based object selection.
  • Other applications: The use cases include autonomous-vehicle events, aisle traffic for store planning, and bird-feeding patterns differentiated by location and color.These applications use spatiotemporal object information and simple user-defined functions over object-box content.

3 BLAZEIT System Overview

BLAZEIT uses configurable detection, entity-resolution, and proxy-model components to execute FRAMEQL queries while reducing expensive object-detection calls. Its scope includes specialized-NN optimizations, but preprocessing, distribution shift, and detector capabilities constrain deployment.

  • Components: BLAZEIT combines target object detection, entity resolution, and optional user-defined functions to execute FRAMEQL queries.The detection and entity-resolution methods can be replaced, while UDFs support properties such as color, size, and location.
  • Proxy models: Proxy models infer simplified query-specific outputs and accelerate execution with accuracy guarantees.BLAZEIT typically uses specialized NNs that mimic larger models on simpler tasks and run dramatically faster.
  • Optimizations: Specialized NNs extend prior binary-detection use to aggregation and limit queries.This broadens specialization beyond the binary detection tasks used in prior work.
  • Limitations: BLAZEIT requires target-model annotations over part of the data to train specialized NNs and filters.The target-model annotated set is created as a preprocessing step, and other contemporary systems also require one.
  • Limitations: In streaming settings, distribution shifts can reduce specialized-NN performance and slow queries, although accuracy guarantees remain intact.Labeling new data, monitoring drift, or continuously retraining can mitigate this effect.
  • Limitations: BLAZEIT cannot support object classes beyond those returned by the target detector, although users can add UDFs when necessary.For example, a detector that identifies cars may not distinguish sedans from SUVs.

4 FrameQL: Expressing Complex Spatiotemporal Visual Queries

FRAMEQL extends SQL with a virtual object-per-tuple video relation and video-specific constructs for temporal spacing, error bounds, and frame-averaged counts. These features support composable visual queries while allowing BLAZEIT to populate only rows needed for each query.

  • Query language: FRAMEQL extends SQL with a table-like schema for querying spatiotemporal object information that can be automatically populated from video.Its virtual relation is populated only as necessary, unlike prior languages that assume externally created relations.
  • Data model: Each FRAMEQL tuple represents one object in one frame, so frames may contain many or no tuples and objects may span multiple frames.The schema includes time, location, class, identifiers, box contents, and detector features.
  • Syntax: FRAMEQL adds GAP, error-tolerance specifications, and FCOUNT to SQL-style selection, projection, and aggregation.It does not support joins because BLAZEIT does not optimize them in this work.
  • Query examples: FRAMEQL supports selection, aggregation, and limit-oriented visual queries while preserving composability with standard relational operators.Examples include counting average cars, selecting temporally separated bus-and-car events, and exhaustively selecting red buses.
  • Temporal and error controls: GAP ensures returned timestamps are at least the specified number of frames apart, while error bounds can constrain absolute, false-positive, or false-negative error.A GAP of 100 removes all but one result from ten consecutive event-containing frames.
  • Aggregation: FCOUNT returns a frame-averaged count, normalizing counts across videos of different lengths.A car in every frame yields 1, whereas a car in only the first of ten frames yields 0.1.

5 Query Optimizer Overview

BLAZEIT’s rule-based optimizer inspects FRAMEQL queries and applies specialized optimizations for aggregation and LIMIT queries, while preserving query accuracy guarantees. It can reuse specialized-model work and defaults to prior filtering methods for other queries.

  • Optimization selection: BLAZEIT applies novel optimizations to aggregation and LIMIT queries to reduce object-detection cost.Its rule-based optimizer selects these optimizations from the query specification.
  • Accuracy guarantees: Both novel optimizations preserve accuracy guarantees despite potentially inaccurate specialized NNs.They speed execution without affecting query accuracy.
  • Optimization selection: Queries without aggregation or LIMIT keywords default to filters similar to NOSCOPE’s.BLAZEIT also supports prior optimizations for exhaustive selection queries.
  • Work reuse: Specialized NN models and results can be stored and reused across queries.The model weights are small compared with the video.

6 Optimizing Aggregates

BLAZEIT accelerates aggregate queries by using error-tolerant sampling, direct specialized-NN answers, or control variates. These choices reduce object-detector work while retaining specified error and confidence guarantees.

  • Overview: BLAZEIT requires an error tolerance to optimize aggregation queries; exact answers otherwise require object detection on every frame.With tolerated error, it uses two aggregation optimizations.
  • Sampling: Insufficient training data triggers random sampling with absolute error bounds and variance-aware early stopping.Empirical Bernstein stopping respects the user’s error bound and confidence when it terminates.
  • Query Rewriting via Specialized NNs: A sufficiently accurate specialized NN can answer the aggregate directly over all frames, bypassing object detection.Accuracy is assessed on held-out data using bootstrap estimates at the requested confidence level.
  • Control Variates: Control variates use a cheap specialized-NN signal correlated with the target statistic to reduce sampling from the full detector.The estimator remains unbiased, and stronger correlation reduces variance.
  • Aggregation with query predicates: For predicates, BLAZEIT applies predicates to training data and trains a specialized NN for the most selective predicate set with enough data.This adapts aggregation optimization to queries such as counting large red buses.
  • Control Variates: Control-variate speedups increase as error bounds tighten or the variance difference grows.The latter typically occurs when specialized NNs are more accurate or their variance is larger.

7 Optimizing Limit Queries

BLAZEIT answers cardinality-limited video queries by ranking frames with specialized NNs and applying full detection selectively. This biased search can return exact answers while reducing frames examined, especially for rare events.

  • Overview: Limit queries seek a specified number of events, often for manual inspection, and are especially useful for rare events.Random sampling and sequential scans can be prohibitively slow when events occur infrequently.
  • Search intuition: BLAZEIT biases search toward video regions likely to contain events using specialized NNs.The approach follows the intuition of rare-event simulation.
  • Physical operator and selection: With training examples, BLAZEIT trains a specialized NN, ranks unseen frames by confidence, and runs object detection in that order.Without examples, it defaults to object detection over every frame and applicable filters.
  • Specialized NN training: BLAZEIT trains count-predicting specialized NNs rather than binary classifiers to alleviate extreme class imbalance in rare queries.For multiple object classes, it predicts each class separately, using fewer weights and typically achieving higher performance.
  • Correctness: BLAZEIT always returns exact answers because it performs object detection on every sampled frame and exhaustively searches when too few events exist.The GAP keyword can exclude surrounding frames after an event is found.
  • Time complexity: Specialized-NN-biased search is not guaranteed to be faster, but empirically reduces the number of frames considered.If requested events exceed available events, every method must examine all frames.

8 Implementation

BLAZEIT combines Python control-plane and neural-network tooling with C++ non-NN filters for efficiency. Its implementation preprocesses video and trains compact specialized NNs using standard deep-learning components.

  • System implementation: BLAZEIT implements its control plane in Python 3.5 and non-NN filters in C++ for efficiency.The prototype uses PyTorch for specialized-NN training and evaluation.
  • System implementation: Object detection uses FGFA and Mask R-CNN through MXNet and Detectron-based frameworks.The implementations were modified to accept arbitrary parts of video.
  • Query interface: BLAZEIT uses a Python Fluent DSL for FRAMEQL queries and Pandas for tuple processing.The paper describes storage and materialization costs as negligible.
  • Video ingestion: Video ingestion resizes frames separately for specialized NNs and object detection, then normalizes pixel values.Specialized-NN frames use 65×65 pixels, while detection uses a 600-pixel short side.
  • Specialized NN training: Specialized NNs use a 10-layer tiny ResNet with a starting filter size of 16.Training uses ImageNet normalization, cross-entropy loss, batch size 16, and SGD with momentum 0.9.
  • Object tracking: The default trackid method matches objects across consecutive frames using motion IOU with a 0.7 cutoff.Objects are treated as the same across frames when their pairwise IOU reaches that cutoff.

9 Evaluation

BLAZEIT’s evaluation on real-world videos shows substantial speedups for aggregation and cardinality-limited queries while maintaining the specified accuracy framework. Specialized neural networks and query-specific sampling drive these gains across varied query settings.

  • Aggregation Queries: Up to 14× speedup over naive AQP was achieved for aggregation queries, including training and threshold-computation time.The binary oracle performed poorly when videos contained many objects of interest.
  • Aggregation Queries: BLAZEIT’s aggregation error stayed within 0.1 for the evaluated videos, despite specialized NNs lacking intrinsic error guarantees.This supported using specialized NNs for query rewriting while respecting user-specified error bounds.
  • Aggregation Queries: Specialized NNs used as control variates reduced aggregation sample complexity by up to 1.7× and sped up naive AQP by up to 1.5×.The variance reduction depends on correlation between specialized NNs and the target object-detection methods.
  • Cardinality-limited Queries: For up to five cars, BLAZEIT’s sample complexity remained nearly constant while naive and binary-oracle complexity increased with the number of cars.Performance degraded with six cars, for which only 70 instances were available and the search was substantially harder.
  • Cardinality-limited Queries: Up to 966× speedups over the naive baseline and up to 81× over the binary oracle were achieved for multi-class limit queries.The binary oracle became more selective for multiple object classes, but BLAZEIT still substantially outperformed it.
  • Cardinality-limited Queries: BLAZEIT outperformed all baselines by up to 300× for limit queries, including proxy-model training time.Its advantage was especially pronounced when queries had few matches, where random sampling and NOSCOPE performed poorly.
  • Specialized Neural Networks: TRN10 required significantly fewer samples than the representative NOSCOPE NN on all evaluated videos for limit queries.For aggregation, TRN10 typically matched or beat NOSCOPE except on night-street.
  • Specialized Neural Networks: Tiny-ResNet throughput decreased with network depth and width and generally with the square of input resolution.Throughput declined further when reduced resolution allowed a larger maximum GPU batch size.

10 Related Work

BLAZEIT extends prior video-analytics and database techniques with declarative query optimization, specialized neural networks, and control-variate variance reduction. Its related-work position centers on exploiting video-query semantics and the high cost of materializing visual tuples.

  • AQP: BLAZEIT differs from standard database systems because video tuple materialization through object detection is orders of magnitude more expensive.This cost imbalance motivates control variates using specialized neural networks, a form of variance reduction not applicable to traditional relational databases.
  • Visual Data Management: Prior multimedia systems and visual query languages often relied on low-level image features or textual annotations for semantic queries.FRAMEQL instead extends SQL and focuses on automatically populating semantic video data with modern vision models.
  • Video Analytics Systems: Unlike systems that treat computation as a black-box graph, BLAZEIT uses FRAMEQL semantics to infer query-specific optimizations.Its optimizer targets declarative video queries rather than only predefined computation graphs.
  • Speeding up Deep Networks: Model compression and distillation improve neural-network inference efficiency while remaining largely orthogonal to BLAZEIT.Reducing the cost of object detection would also improve BLAZEIT’s performance.
  • Specialization: Specialized neural networks traditionally targeted simplified binary-detection pipelines, whereas BLAZEIT extends them to counting and multi-class classification.BLAZEIT also uses specialized NNs as control variates and for limit queries.

11 Conclusions

BLAZEIT combines declarative FRAMEQL queries with optimizations for video aggregation, limit, and selection workloads. Its supported techniques reduce object-detection work while preserving specified accuracy or exactness, but the language remains scoped to queries optimizable with current techniques.

  • Scope and assumptions: The framework assumes batch analytics with all video available and deterministic detection and entity-resolution methods, so FRAMEQL tuples remain fixed after those methods are chosen.Transformation time into FRAMEQL tuples is ignored because it is a tiny fraction of total time.
  • FRAMEQL: BLAZEIT uses FRAMEQL, a declarative SQL extension that represents spatiotemporal object information and supports selection, aggregation, and limit queries.Users can express queries over a virtual relation without fully materializing it.
  • Scope and assumptions: FRAMEQL does not currently support joins or global object identifiers, although the paper describes how both could be added for broader queries.The current work focuses on queries optimizable with existing techniques.
  • Aggregation: BLAZEIT’s aggregation methods use sampling, specialized neural networks, or control variates to answer approximate statistics with error guarantees.The system also supports occupancy and unique-object aggregation through specialized procedures or standard approximate query processing.
  • Execution: BLAZEIT’s filters are 3 to 6 orders of magnitude faster than object detection, so they are nearly always beneficial when they discard enough frames.A filter is beneficial when its cost is lower than the object-detection cost it avoids.

D.2 Empirical Analysis of EBS

EBS’s confidence intervals respected the target confidence in every tested run at the most stringent evaluated error tolerance.

  • Empirical guarantee: 100 out of 100 EBS runs respected the 95% confidence interval for every aggregation query tested at error tolerance ϵ = 0.01.The experiment evaluated the most stringent error tolerance considered.

D.3 Experiments with Full Frames

With full frames and no filtering of poorly detected regions, BLAZEIT still outperformed all baselines except on the Amsterdam cardinality-limited selection query.

  • Full-frame evaluation: BLAZEIT still outperformed all baselines except on amsterdam for the cardinality-limited selection query when experiments used full frames.The experiments were designed to test whether excluding poorly detected frame portions affected performance.

D.4 Content-based Selection Queries

Content-based filters substantially improve BLAZEIT’s end-to-end performance on exhaustive selection queries. Every filter class contributes nontrivial speedup for the evaluated query, although the optimizer falls back to full materialization for unsupported query types.

  • Experimental setup: BLAZEIT evaluates naive detection, a binary oracle, and its learned filtering strategy on an exhaustive red-bus selection query.The study omits AQP because sampling does not help exhaustive queries.
  • Measurement and optimization: BLAZEIT includes filter-training and threshold-selection time in runtime, while extrapolating object-detector cost from the number of detector calls.The optimizer determines applicable filters from query predicates and training data.
  • Additional aggregation results: For approximate aggregation, control variates reduce required samples in all tested cases without frame filtering, but relative speedups depend on the data.Table 11 reports percent reductions at error tolerance 0.01.
  • End-to-end performance: 54× improved throughput is achieved over naive methods for the red-bus query, while BLAZEIT outperforms the binary oracle by 6.4×.The binary oracle itself improves performance over naive detection by 8.4× because buses occupy 12% of frames.
  • Filter analysis: Every evaluated filter class adds a nontrivial speedup, and removing any class reduces performance on the red-bus query.The factor analysis adds filters incrementally, while the lesion study removes them individually.
  • Optimizer scope: For query types outside approximate aggregation, cardinality-limited queries, and content-based selection, BLAZEIT falls back to materializing the FRAMEQL table.The optimizer identifies supported classes using query keywords and predicates.
Loading 1805.01046v2…