Source-linked AI summary
Scaling Video Analytics on Constrained Edge Nodes
Christopher Canel, Thomas Kim, Giulio Zhou, Conglong Li, Hyeontaek Lim, David G. Andersen, Michael Kaminsky, Subramanya R. Dulloor
TL;DR
Large camera deployments cannot reliably send high-quality video to datacenters over constrained per-camera links. FilterForward filters video at the edge with application-specific microclassifiers that share a base DNN’s computation, and evaluation reports order-of-magnitude bandwidth savings alongside higher throughput and accuracy.
Problem
Large camera deployments face bandwidth constraints that make uploading high-quality video streams to datacenters infeasible for analytics.
Method
FilterForward uses edge microclassifiers to identify relevant video and shares feature extraction from a base DNN across concurrent applications before forwarding selected data.
Results
FilterForward uses an order of magnitude less bandwidth than standard compression, reaches up to 6.1× higher throughput, and achieves up to 1.3× higher accuracy than prior pixel-based DNN filters.
Takeaways & Limitations
The hybrid edge-to-cloud design makes high-fidelity video sequences available to datacenter applications while limiting wide-area bandwidth use and supporting multi-tenant edge filtering.
Takeaways & Limitations
FilterForward assumes relevant events are rare and that applications require high-quality video, while microclassifiers still incur the base DNN’s upfront overhead.
Abstract
from arXiv · showhide
As video camera deployments continue to grow, the need to process large volumes of real-time data strains wide area network infrastructure. When per-camera bandwidth is limited, it is infeasible for applications such as traffic monitoring and pedestrian tracking to offload high-quality video streams to a datacenter. This paper presents FilterForward, a new edge-to-cloud system that enables datacenter-based applications to process content from thousands of cameras by installing lightweight edge filters that backhaul only relevant video frames. FilterForward introduces fast and expressive per-application microclassifiers that share computation to simultaneously detect dozens of events on computationally constrained edge nodes. Only matching events are transmitted to the cloud. Evaluation on two real-world camera feed datasets shows that FilterForward reduces bandwidth use by an order of magnitude while improving computational efficiency and event detection accuracy for challenging video content.
1 Introduction
FilterForward addresses bandwidth bottlenecks in large camera deployments by filtering video at the edge and forwarding only relevant events, while sharing computation across application-specific microclassifiers. Its evaluation reports substantially lower bandwidth use, higher throughput, and improved accuracy under its target conditions.
- Hundreds of gigabits per second from tens of thousands of always-on cameras can overload shared networks, especially at wirelessly or cellularly connected sites.
- FilterForward identifies relevant video sequences on edge resources and forwards only that data for near-real-time datacenter processing over limited-bandwidth links.
- FilterForward assumes relevant events are rare and that datacenter applications require high-quality video, ruling out aggressive compression or resolution reduction.
- Applications install small binary microclassifiers that classify incoming frames, smooth results into events, and re-encode those events for datacenter streaming.
- Many microclassifiers share a base DNN’s feature extraction, allowing tens of concurrent classifiers on a small edge node after an upfront break-even overhead.
- 6.1× higher throughput is achieved with 50 concurrent microclassifiers, while microclassifiers are up to 1.3× more accurate and have up to a 23× lower marginal cost than prior pixel-based DNN filters.
2 Background and Challenges
Large-scale surveillance deployments face bandwidth, video-quality, and edge-compute constraints: wide-angle views make targets small, while many applications may need simultaneous analysis. FilterForward responds with semantic filtering, custom microclassifiers, and computation reuse.
- Limited Bandwidth: A camera may receive only a few hundred kilobits per second or less, whereas low-quality H.264-encoded 1080p video requires approximately 2 Mb/s.
- Limited Bandwidth: The bandwidth gap and high-quality requirement necessitate deciding at the edge which frames are relevant enough to upload to the datacenter.
- Real-world Video Streams: Wide-angle cameras make interesting objects occupy small portions of frames, while downsampling can erase details such as license plates and distant people.
- Real-world Video Streams: FilterForward uses high-resolution edge microclassifiers to detect fine-grained details without the quality degradation caused by bandwidth-driven decimation.
- Scalable Multi-tenancy: Different applications may simultaneously seek pedestrians, vehicles, shoppers, weather, foliage, and road conditions from one camera.
- Scalable Multi-tenancy: Running one full DNN per application is costly on constrained nodes; MobileNet runs at approximately 15 fps on a modest CPU while consuming more than 1 GB of memory.
- Scalable Multi-tenancy: FilterForward addresses these challenges through semantic filtering, microclassifier architectures for fine-grained details, and computation reuse for multi-tenancy.
3 Designing FilterForward
FilterForward splits video analytics between edge and cloud: a feature extractor produces reusable representations, and microclassifiers use them to filter high-fidelity video for datacenter applications.
- FilterForward reuses computation to provide accurate, multi-tenant video filtering on bandwidth-constrained edge nodes.
- The architecture contains a feature extractor and microclassifiers designed to meet bandwidth, fine-detail detection, and concurrent-application requirements.
3.1 Generating Features
FilterForward generates feature maps from full-resolution video using a shared base DNN, preserving fine details while amortizing expensive computation across microclassifiers.
- The feature extractor runs a shared reference DNN and supplies intermediate activations as feature maps to microclassifiers.
- Feature-map activations range from visually recognizable low-level filters in early layers to high-level concepts in later layers.
- MobileNet trained on ImageNet is used as the base DNN because it balances accuracy and computational demand for constrained edge nodes.The evaluation uses the 32-bit unquantized network.
- FilterForward processes full-resolution frames, retaining details such as distant pedestrians, vehicle-specific features, and faces.Evaluation uses 1920 × 1080 or 2048 × 850 frames, which contain 41.3× and 34.7× more input data than 224 × 224 frames.
- Although full-resolution feature extraction adds substantial computation, reusing its results amortizes the upfront cost as the number of microclassifiers increases.The feature maps support FilterForward’s accuracy and scalability achievements.
3.2 Finding Relevant Frames Using Microclassifiers
Microclassifiers independently filter incoming frames for application-specific content, optionally specializing their inputs spatially while prioritizing bandwidth savings.
- Microclassifiers are lightweight binary neural networks that output the probability that each frame is relevant to a particular application.An edge node can run many microclassifiers on one stream or fewer microclassifiers across several streams.
- Developers train each microclassifier offline and deploy its weights, architecture, and selected base-DNN input layer to the edge.
- Each microclassifier can select feature maps from any base-DNN layer and optionally crop them to focus on a frame region.Cropping is an independent local optimization for each microclassifier.
- Dropping irrelevant frames limits bandwidth, with false positives especially costly because they upload irrelevant video.Video redundancy provides a safety margin for false negatives.
- The edge records original streams locally so cloud applications can demand-fetch additional context around matched segments.
- Sharing base-DNN computation lets microclassifiers operate on feature maps with competitive accuracy while reducing marginal compute cost by an order of magnitude.
3.3 Microclassifier Architectures
FilterForward uses custom microclassifier architectures to detect small or localized objects in wide-angle surveillance video, optionally incorporating temporal context.
- Off-the-shelf classifiers and detectors perform poorly on wide-angle surveillance video because objects of interest are often small.
- Microclassifiers use task-appropriate feature-map layers and optional cropping to improve accuracy on real-world surveillance data.
- The full-frame object-detector microclassifier applies binary classification across convolutional feature-map locations and aggregates logits with a max operator.The max operator represents detecting at least one object.
- The localized binary classifier processes spatially cropped feature maps with two separable convolutions and a fully connected layer.It is designed to detect prominent objects within a localized region.
- The windowed localized classifier adds nearby temporal context by processing a user-specified W-frame window centered on the target frame.It predicts whether the center frame is interesting.
3.4 Choosing Microclassifier Inputs
Choosing each microclassifier’s base-DNN input layer balances spatial localization against semantic information, with the prototype using heuristic layer selection and rectangular crops.
- Selecting a base-DNN layer is critical because feature hierarchies trade spatial localization against semantic information.
- Late layers may lose small details, while early layers retain large activations and require more processing toward classification.
- The baseline chooses a layer by matching feature-map reduction to the typical object size being detected.For 40-pixel pedestrians in 1920 × 1080 video, the heuristic selects the first layer with roughly 20:1–50:1 spatial reduction.
- The prototype restricts each microclassifier to one base-DNN layer and rectangular feature crops.Combining multiple layers, using free-form or discontiguous crops, and automating parameter selection remain future-work challenges.
3.5 From Per-frame Classifications to Events
FilterForward converts per-frame microclassifier outputs into smoothed event detections and streams selected frames using configurable H.264 encoding.
- Event detection: Each microclassifier produces binary per-frame classifications that are smoothed into event detections.K-Voting aggregates N consecutive frame results and treats the middle frame as detected when at least K are positive.
- Event detection: K-Voting treats the middle frame as a detection when at least K of N consecutive classifications are positive.The evaluation uses N = 5 and K = 2.
- Event representation: A frame can belong to multiple events, with internal metadata mapping each microclassifier to its detected event.This allows overlapping application-specific detections to be represented for the same frame.
- Forwarding: Selected frames are re-encoded with H.264 at a user-configured bitrate and streamed to the datacenter.The application developer chooses a bitrate considered sufficiently high for the task.
4 Evaluation
FilterForward is evaluated on two real-world surveillance datasets and addresses bandwidth conservation, event detection quality, and scalable multi-tenancy. Across these evaluations, it substantially reduces bandwidth, scales better with many classifiers, and provides favorable accuracy–compute tradeoffs, while paying an upfront base-DNN cost.
- Bandwidth: 6.3–13× less bandwidth is used than heavily compressing and uploading the full stream when filtering rare events at the edge.The localized binary classifier achieves the larger reduction by dropping more frames, partly because it has more false negatives.
- Bandwidth: Filtering the edge stream reduces bandwidth by 6.3× with full-frame object detectors and 13× with localized binary classifiers.Matched frames are re-encoded at 250 Kb/s and 500 Kb/s, respectively; savings are independent of the selected upload bitrate.
- Accuracy: Event F1 score increases by 1.5× for full-frame object detectors and 1.9× for localized binary-classifier MCs versus similarly bandwidth-constrained full-stream compression.Filtering preserves fine-grained details by selecting frames from the original high-fidelity stream before upload.
- Scalability: By 50 classifiers, FilterForward reaches up to 6.1× higher throughput than the compared discrete classifiers.Compute sharing and base-DNN amortization make FilterForward faster than existing techniques once more than 3–4 classifiers run together.
- Scalability: The base DNN’s CPU time is equivalent to that of 15–40 microclassifiers, so adding dozens of MCs increases total execution time only modestly.This reflects reuse of feature extraction across concurrent filtering tasks.
- Accuracy and cost: FilterForward’s MCs are up to 1.3× as accurate and 23× cheaper on Jackson, and 1.1× as accurate and 11× cheaper on Roadway, than discrete classifiers.The marginal compute savings result from operating on feature maps rather than pixels.
- Accuracy and cost: The MCs incur an upfront computational overhead from the base DNN.This is the principal cost tradeoff of reusing shared feature extraction across classifiers.
5 Related Work
Prior video-analytics systems address filtering, computation reuse, and resource management, but FilterForward targets constrained-bandwidth edge deployments with scalable multi-tenancy. Its microclassifiers share feature extraction, achieving lower marginal cost and higher accuracy than discrete classifiers.
- Conventional computation reuse improves scalability, but its rigidity can sacrifice accuracy.
- FilterForward shares a base DNN’s feature extraction across microclassifiers, unlike transfer learning’s last-layer feature-vector reuse and multi-task learning’s retraining requirement when tasks change.
- Saving Compute During Bulk Analytics: FilterForward differs from NoScope by operating microclassifiers on shared feature maps rather than raw pixels, reducing marginal cost without sacrificing accuracy.
- FilterForward is designed for stationary surveillance cameras, whereas related edge systems focus on moving-camera streams and are not optimized for multi-tenant environments.
- Unlike scheduling systems that adjust bitrate, resolution, or model choice, FilterForward directly improves computational efficiency for multiple filters sharing an edge node.
6 Conclusion
FilterForward addresses real-time wide-area video analytics under bandwidth and compute constraints by filtering video at the edge with lightweight microclassifiers. The system reduces bandwidth use by an order of magnitude, preserves accuracy, and reaches up to 6.1× higher throughput than existing approaches.
- FilterForward uses lightweight per-application microclassifiers at the edge to identify relevant video segments for cloud upload.
- An order of magnitude less bandwidth is used without sacrificing accuracy, while throughput reaches as much as 6.1× higher than existing approaches.
- FilterForward’s computation sharing and hybrid edge-to-cloud design are presented as building blocks for machine-learning applications in constrained environments.