Source-linked AI summary
Focus: Querying Large Video Datasets with Low Latency and Low Cost
Kevin Hsieh, Ganesh Ananthanarayanan, Peter Bodik, Paramvir Bahl, Matthai Philipose, Phillip B. Gibbons, Onur Mutlu
TL;DR
Large recorded-video datasets require accurate after-the-fact object queries, but conventional CNN processing is too slow and expensive. Focus combines cheap, specialized ingest indexing with selective expensive classification and object clustering, achieving substantially lower cost and latency than exhaustive baselines.
Problem
After-the-fact queries over continuously recorded camera video need low latency, but accurate CNN-based processing is expensive and slow.
Method
Focus uses compressed and video-specialized CNNs for ingest indexing, top-K classifications and query-time GT-CNN validation for accuracy, clustering to avoid redundant processing, and parameter selection to trade cost against latency.
Results
Focus provides trade-off options that are significantly cheaper than analyzing all frames at ingest and significantly faster than analyzing queried frames at query time.
Takeaways & Limitations
Focus supports low-cost ingest-time analytics that facilitate low-latency queries on recorded videos across traffic, surveillance, and news domains.
Takeaways & Limitations
Focus's applicability depends on the number of queried classes and the fraction of videos queried; high query rates can make Ingest-all a good option.
Abstract
from arXiv · showhide
Large volumes of videos are continuously recorded from cameras deployed for traffic control and surveillance with the goal of answering "after the fact" queries: identify video frames with objects of certain classes (cars, bags) from many days of recorded video. While advancements in convolutional neural networks (CNNs) have enabled answering such queries with high accuracy, they are too expensive and slow. We build Focus, a system for low-latency and low-cost querying on large video datasets. Focus uses cheap ingestion techniques to index the videos by the objects occurring in them. At ingest-time, it uses compression and video-specific specialization of CNNs. Focus handles the lower accuracy of the cheap CNNs by judiciously leveraging expensive CNNs at query-time. To reduce query time latency, we cluster similar objects and hence avoid redundant processing. Using experiments on video streams from traffic, surveillance and news channels, we see that Focus uses 58X fewer GPU cycles than running expensive ingest processors and is 37X faster than processing all the video at query time.
1. Introduction
Focus targets after-the-fact video queries by reducing ingestion cost and query latency while preserving specified precision and recall. It combines cheap CNN indexing, top-K results, object clustering, and parameter selection to balance these goals.
- Recorded-camera datasets support after-the-fact queries for frames containing specified object classes, making low query latency important for analysts and investigators.
- ResNet152-based video querying is expensive and slow: a month-long traffic video requires 280 GPU hours and costs $250 in Azure after motion filtering.
- Focus uses compressed, video-specialized CNNs to create a low-cost ingest-time index of object classes to frames.
- Focus preserves accuracy by indexing each object with top-K cheap-CNN classifications, then filtering candidates and validating them with the expensive GT-CNN at query time.
- Focus clusters similar objects at ingest time, classifies only cluster centroids with the GT-CNN, and assigns centroid labels across each cluster to reduce query latency.
- Focus selects ingest CNNs and parameters that meet user-specified accuracy targets while trading off ingest cost against query latency.
2. Background and Motivation
CNNs provide accurate object classification but are computationally costly, motivating Focus's use of video structure to avoid redundant processing. Videos contain many irrelevant frames, limited object classes, and visually similar repeated objects.
- 2.1. Convolutional Neural Networks: CNNs extract visual features and output probabilities for object classes, but deeper high-accuracy models require substantial computation and make large-video querying slow and costly.
- 2.1. Convolutional Neural Networks: Compression reduces CNN inference cost at the expense of accuracy, while specialization trains models for context-specific subsets of classes.
- 2.2. Background and Motivation: Focus is designed around excluding large video portions, exploiting limited per-video class sets, and using similarity among same-class objects.
- 2.2.1. Excluding large portions of videos: One-third to one-half of frames contain no objects or only stationary objects, making them candidates for query-time exclusion.
- 2.2.2. Limited set of object classes in each video: An object class occurs in only 0.01% of frames on average, while the most frequent classes occur in no more than 16%−43% of frames across videos.
- 2.2.2. Limited set of object classes in each video: Only 22%−33% of the 1,000 classes occur in less busy videos, and 3%−10% of the most frequent classes cover at least 95% of objects in each stream.
- 2.2.3. Feature vectors for finding duplicate objects: Feature-vector nearest neighbors belong to the same class in over 99% of object pairs, supporting duplicate detection with one classification instead of n classifications for n duplicates.
3. Overview of Focus
Focus indexes live video with cheap, specialized CNN processing and uses selective GT-CNN classification at query time to support low-cost, low-latency object queries. It clusters similar objects and exposes trade-offs among accuracy, ingest cost, and query latency.
- Focus indexes live video streams by object classes so later queries can retrieve frames containing a requested class.
- Focus lets users specify precision and recall targets and selects configurations that trade ingest cost against query latency.
- At ingest-time, Focus compresses and specializes the GT-CNN, clusters object features, and maps classes to object clusters through a top-K index.
- At query-time, Focus retrieves matching clusters, classifies their centroids with the GT-CNN, and returns frames from clusters whose centroids match the requested class.
- Focus’s architecture is designed to apply beyond image classification to other CNNs, including face recognition.
4. Video Ingest & Querying Techniques
Focus combines cheap, video-specific ingest CNNs, top-K indexing, object clustering, and parameter selection to preserve accuracy while reducing redundant processing. Its configuration balances ingest cost and query latency under user-specified accuracy targets.
- Cheap Ingestion: Increasing K raises recall for all cheap CNNs, while cheaper models require larger K values to reach comparable recall.The three models reach 90% recall at K ≥60, K ≥100, and K ≥200, respectively.
- Cheap Ingestion: Focus uses top-K classes for ingest indexing and GT-CNN classification at query time to recover precision while controlling the candidate set.Because each object is associated with K classes, the top-K index alone has average precision of only 1/K.
- Object Clustering: Object clustering reduces query-time GT-CNN work by classifying cluster centroids and assigning their labels to all objects in each cluster.Focus clusters object feature vectors rather than entire frames; clustering can affect precision and recall depending on threshold T.
- Object Clustering: The clustering algorithm assigns objects within threshold T to nearby clusters, maintains at most M clusters, and runs in O(Mn) time.
- Cheap Ingestion: Cheap ingest CNNs use compression and video-specific specialization to reduce indexing cost while retaining accuracy on stream-specific objects.Specialized models can remove 1/3 of convolutional layers and reduce input resolution 4× while achieving similar accuracy, becoming 10× cheaper than generic cheap CNNs.
- Parameter Selection: Focus selects viable model and clustering configurations on the Pareto boundary, balancing normalized ingest cost and query latency by minimizing total GPU cycles.
5. Implementation Details
Focus distributes ingest and query processing across worker processes, using object detection, CNN inference, indexing, and parallel query execution across available resources.
- Worker Processes: Each machine runs an ingest worker per video stream that receives live video and extracts moving objects using background subtraction.The object detector can be replaced with another detector.
- Worker Processes: Ingest workers send detected objects to CNNs for top-K classes and feature vectors, while query workers fetch indexed frames and classify objects with GT-CNN.
- CNN Classification: Focus parallelizes query work across worker processes when resources are idle, and CNNs may run on local or disaggregated accelerators.
6. Evaluation
Focus substantially reduces query latency and ingest cost across diverse video streams while exposing tunable trade-offs and retaining specified accuracy targets. Its benefits persist under higher accuracy targets, lower frame rates, and extreme query rates, though query-speed gains weaken in some settings.
- Setup: 13 live 12-hour video streams spanning traffic, surveillance, and news channels were evaluated against Ingest-all and Query-all baselines.The evaluation used ResNet152 as the GT-CNN and motion detection for both baselines.
- End-to-End Performance: 37× faster queries and 58× cheaper ingest were achieved on average than Query-all and Ingest-all, respectively.For a 24-hour video on a 10-GPU cluster, latency fell from one hour to under two minutes and monthly processing cost from $250 to $4.
- End-to-End Performance: 11× to 57× faster queries and 48× to 98× cheaper ingest were achieved across streams with diverse video characteristics.Query-speed gains were smaller for less busy videos because fewer dominant object classes create more GT-CNN work.
- Effect of Different Focus Components: 7× to 71× cheaper specialized models retained each stream’s accuracy target, while reducing query latency by 5× to 25×.Specialization provided substantially greater improvement than generic compression alone.
- Effect of Different Focus Components: Up to 56× lower query latency came from clustering, with negligible ingest cost because CPU clustering was pipelined with GPU inference.Clustering was applied on top of a specialized compressed model to reduce redundant query-time work.
- Trade-off Tuning: Focus exposes an ingest-cost/query-latency trade-off: Focus-Opt-Ingest was 141× cheaper to ingest and 46× faster to query, while Focus-Opt-Query was 63× faster and 26× cheaper than Ingest-all.Across representative videos, Opt-I averaged 95× cheaper ingest with 35× lower latency, while Opt-Q averaged 49× faster queries and 15× cheaper ingest.
- Sensitivity to Accuracy Target: 15×, 12×, and 8× faster queries were achieved at 97%, 98%, and 99% accuracy targets, while ingest remained 62× to 64× cheaper.Higher accuracy targets require larger top-K indexes, increasing query-time work while leaving ingest cost similar.
- Sensitivity to Frame Sampling: At 1 fps, Focus remained one order of magnitude faster than Query-all, although lower frame rates reduced clustering’s latency benefit.Ingest savings remained 58× to 64× at lower frame rates because they primarily came from specialized compressed CNNs.
7. Related Work
Focus differs from prior video-analytics approaches by decoupling compressed ingest-time CNNs from ground-truth CNNs and clustering similar objects to reduce redundant query-time processing.
- Cascaded classification: Focus decouples compressed CNNs from GT-CNNs, broadening ingest-time CNN choices and enabling trade-offs between ingest cost and query latency.This differs from cascaded-classification approaches that chain classifiers to reduce detection latency.
- Cascaded classification: Focus clusters similar objects using CNN features to eliminate redundant processing in video streams.The paper identifies this as a new technique within its system.
- Video indexing and retrieval: Video-indexing and retrieval research supports content-based queries, including shot-boundary detection and semantic video search.Most related systems target query types other than object-class frame retrieval.
8. Conclusion
Focus targets low-cost ingest-time analytics for object-class queries over recorded video, while reducing later query latency through specialized processing and object clustering. Across traffic, surveillance, and news videos, it substantially lowers GPU consumption and accelerates queries relative to current baselines.
- 8. Conclusion: Focus performs low-cost ingest-time analytics on live video to facilitate low-latency queries over recorded video.It indexes object occurrences using compressed and specialized CNNs.
- 8. Conclusion: Focus clusters similar objects to reduce query-time work and latency.It selects ingest-time CNNs and parameters to trade off ingest-time cost against query-time latency.
- 8. Conclusion: 58× lower GPU consumption and 37× faster queries were measured against current baselines on 150 hours of traffic, surveillance, and news video.These are the reported evaluation results across the three video domains.
- 8. Conclusion: The authors describe Focus as a promising approach and identify better determination of ingest-time and query-time trade-offs as future work.They also propose training a specialized, highly accurate query-time model as a next step.