PaperlayerView exact revision on arXiv

Source-linked AI summary

ASTRA: Low-Overhead Runtime Architecture for STReam Adaptation in Video Analytics

Mahshid Ghasemi, Zoran Kostic, Javad Ghaderi, Gil Zussman

arXiv:2609.07020v2cs.NI

TL;DR

Real-time video analytics must balance accuracy and latency under resource limits and changing network conditions and video content. ASTRA provides a low-overhead runtime for online adaptation algorithms in live edge-video systems. Across evaluations, it approaches offline-optimal performance, with live-camera accuracy above or near the required threshold.

  • Problem

    Video analytics accuracy and latency vary with network conditions, computing capacity, and video content, while processing every stream at maximum resolution and frame rate is resource-intensive.

    Source: §I · pp. 1–2

  • Method

    ASTRA is a low-overhead runtime architecture that mitigates execution overheads and runs iterative adaptation algorithms as interchangeable black-box components.

    Source: B. Contributions · p. 3, §II · p. 4

  • Results

    Across single- and multi-camera experiments, ASTRA consistently approaches offline-optimal performance; live-camera accuracy remains above or close to the required threshold of 0.85.

    Source: A. Deployment in COSMOS Testbed · p. 11, C. Varying Network Latencies · p. 13

  • Takeaways & Limitations

    The evaluations support that principled online adaptation is practical and effective for live video analytics systems.

    Source: §VII · p. 14

Abstract

from arXiv · show

Real-time video analytics is crucial for smart city applications and cloud-connected vehicle control. To improve analytics accuracy, it is desirable to process the video at the highest resolution and frame rate. However, due to limited resources, streaming and processing video at the highest resolution and frame rate from all cameras is not feasible and adversely affects the analytics latency. Intelligent adaptation of cameras' resolutions and frame rates based on network conditions and the video content is crucial in order to optimize the performance. In this paper, we present ASTRA, a low-overhead runtime architecture for online adaptation of live camera analytics at the edge. ASTRA can execute various online algorithms as a black box. We deployed ASTRA in the realistic NSF COSMOS testbed and uniquely assessed its real-time performance using COSMOS' street-level cameras. We further evaluated ASTRA with up to eight emulated cameras by streaming a comprehensive video dataset under real-world network conditions. We used ASTRA's architecture to evaluate the practical performance of several classes of adaptation algorithms, including theoretical and empirical methods. The results indicate that ASTRA can provide system reliability (i.e., the probability of meeting accuracy and latency requirements) of more than 90% while maintaining performance within a deviation of less than 10% from optimal offline performance with average GPU utilization overhead of around 2% per camera.

I. Introduction

Video analytics must balance accuracy and end-to-end latency as network conditions, computing capacity, and video content change. ASTRA addresses the runtime costs and feasibility gap of online adaptation in deployed real-time systems.

I. Introduction

  • Online adaptation seeks stream configurations that maximize accuracy and end-to-end latency performance while meeting application-specific minimum-accuracy and maximum-latency constraints.A configuration is a stream’s resolution and frame rate.Source: §I · p. 2
  • Network, compute, and content variation make accuracy and latency time-varying and unknown, limiting offline optimization and motivating iterative online adaptation.Configurations can be evaluated only while deployed on live streams, without future knowledge or an explicit performance model.Source: §I · pp. 1–2
  • Prior adaptation studies largely develop algorithms and evaluate recorded video, leaving system-level overhead and real-time edge/cloud feasibility underexplored.ASTRA targets this gap with a runtime architecture that executes iterative algorithms as black boxes.Source: §I · p. 2

A. Challenges

  • Proxy accuracy measurement and sequential configuration or stream switching add computational, network, and temporal overhead to adaptation.Switching can repeat decoding, encoding, buffering, pipeline reconfiguration, data transfer, and model loading.Source: A. Challenges · pp. 2–3

B. Contributions

ASTRA profiles runtime bottlenecks and provides a modular architecture for executing adaptation algorithms as interchangeable black boxes. Its evaluations span live COSMOS cameras and reproducible emulated-camera experiments, while the paper situates its system focus against prior adaptation work.

B. Contributions

  • ASTRA combines asynchronous processing, OS-level signaling, efficient memory management, and modular design to mitigate runtime overheads and support fair algorithm comparisons.It runs GP-UCB-C and Chameleon++ as representative theoretically grounded and empirical approaches, alongside offline-optimal and AutoML++ baselines.Source: B. Contributions · p. 3
  • ASTRA reduces per-iteration overhead by an order of magnitude, reaches within 10% of optimal performance, exceeds 90% reliability, and averages around 2% GPU utilization per camera.Reliability is the probability of meeting accuracy and latency requirements.Source: B. Contributions · p. 3
  • Evaluation combines real-time adaptation on two live COSMOS traffic cameras with reproducible experiments using eight emulated cameras under realistic network latency.The emulated streams use RTP and GStreamer, enabling identical conditions across algorithms.Source: B. Contributions · p. 3

II. Related Work

  • Unlike work centered on adaptation algorithms, ASTRA targets system-level execution overheads—including switching, reconfiguration, buffering, contention, and model switching—in deployed real-time pipelines.The authors report these overheads can make online adaptation counterproductive if left unaddressed.Source: §II · p. 4

III. Online Configuration Adaptation

ASTRA models online adaptation as iterative selection and measurement of camera configurations under accuracy and latency constraints. Its runtime design periodically adapts, then uses a near-optimal configuration for the rest of each time window.

III. Online Configuration Adaptation

  • At each iteration, an algorithm samples a supported resolution-and-frame-rate configuration, measures its accuracy and latency on a short stream segment, then selects the next configuration.The process ends when a near-optimal configuration is identified.Source: §III · p. 4
  • Accuracy uses F1-score comparisons of detected boxes and labels against ground truth, while latency sums camera encoding, network transfer, edge decoding, and inference.For unsampled frames, object locations from the previous sampled frame are used to account for frame-rate effects.Source: §III · p. 4
  • Because manual annotation is infeasible in real time, ASTRA uses detections from the highest-resolution, highest-frame-rate golden configuration as proxy ground truth.The paper notes that inaccurate estimates can steer adaptation away from the actual optimum.Source: §III · pp. 4–5

IV. Architecture Design

  • Adaptation runs at each time window’s start until a near-optimal configuration is found, which is then used for the remainder; the adaptation period should stay small relative to the window.The paper gives below 10% as an example target for the adaptation-period share.Source: §IV · p. 5, Figure 4 · p. 5

A. Baseline Architectures

The baseline repeatedly switches camera channels to sample configurations, but camera switching can be too slow for short adaptation iterations. ASTRA instead uses one highest-quality stream and locally generates configurations, improving switching latency and frame alignment.

A. Baseline Architectures

  • Channel switching takes ∼400 ms to more than 1.2 s, significant when adaptation samples streams every ∼0.2 s.Measurements across three cameras show latency varies by camera type, including with internal buffering behavior.Source: A. Baseline Architectures · p. 5
  • The baseline samples each configuration by switching camera channels, while the improved architecture takes the highest-resolution, highest-frame-rate stream and locally down-samples it.The golden stream also supplies proxy ground truth, so it can be mirrored at the edge instead of requesting a second camera stream.Source: A. Baseline Architectures · p. 5
  • Approximately 20× lower switching delay is achieved with configuration switching through down-sampling rather than channel switching across Bosch, HikVision, and Axis cameras.The improved design also reduces camera-specific buffering and jitter effects, making behavior more predictable.Source: A. Baseline Architectures · p. 6, Figure 6 · p. 7
  • Using one stream aligns ground-truth and detection frames by timestamp, supporting reliable frame-level F1-score measurement.Separate camera channels can have inconsistent timing from independent buffering and NTP drift.Source: A. Baseline Architectures · p. 6
  • A one-frame drift can make the truly most accurate highest-frame-rate configuration appear least accurate in either tested resolution setting.That distortion can lead the iterative algorithm to select a suboptimal configuration.Source: A. Baseline Architectures · p. 6, Figure 7 · p. 7

B. Our Architecture: ASTRA

ASTRA turns the improved single-stream design into an asynchronous pipeline, using queued concurrent processes to reduce waiting. It estimates per-chunk network latency from receiver-side observations and RTT measurements.

B. Our Architecture: ASTRA

  • ASTRA decouples down-sampling/object detection and proxy-ground-truth processing with FIFO queues, forming an asynchronous Fork–Join pipeline without global synchronization barriers.The architecture includes queues for video chunks, sampled outputs, proxy ground truth, and measured network latency.Source: B. Our Architecture: ASTRA · p. 7, Figure 8 · p. 8
  • Chunkification groups consecutive decoded frames into small chunks; matching chunk length to the I-frame interval minimizes buffering and idle time.Longer I-frame intervals can require waiting for the full interval before emitting smaller chunks.Source: B. Our Architecture: ASTRA · p. 8
  • The golden stream is sent briefly during adaptation—about 1 s per camera versus 4–7 s of adaptation in a 60 s window—and uses approximately 10 Mbps while active.Adaptation components can be deployed near cameras with high-bandwidth connectivity, while continuous analytics can use bandwidth-constrained links.Source: B. Our Architecture: ASTRA · p. 8

C. Latency Measurement

  • ASTRA estimates chunk-level network latency at the edge using receiver-side timestamps because camera clocks and RTP do not provide precise one-way delay measurements.The method approximates one-way delay as RTT/2; the authors report its error is much smaller than errors from NTP or RTP sender timestamps.Source: C. Latency Measurement · pp. 8–9
  • The estimate treats chunk latency as transmission time plus the one-way delay of the last bit, using observable edge arrival times and RTT to approximate it.The edge cannot observe the camera-side start time directly.Source: C. Latency Measurement · pp. 8–9

V. Configuration Switching in Object Detection Pipeline

Frequent configuration changes introduce pipeline overhead that can consume a substantial part of short adaptation iterations. The main costs arise from reconfiguration, memory management, and cache disruption.

V. Configuration Switching in Object Detection Pipeline

  • Reconfiguring pipeline elements, including reloading inference engines and allocating model-specific buffers, can take tens to hundreds of milliseconds per switch.Different resolutions, frame rates, model variants, or input shapes can require recompilation and initialization.Source: §V · p. 9
  • Configuration changes trigger intermediate-buffer allocation and deallocation, adding a few to tens of milliseconds, while cache misses add further delays.GPU memory-hierarchy inefficiencies from disrupted cache continuity also take a few to tens of milliseconds.Source: §V · p. 9
  • Even a few hundred milliseconds of switching delay per iteration can add several seconds across ten adaptation iterations.The delay is consequential because adaptation iterations may need to occur back-to-back in short time slots such as ∼0.2 s.Source: §V · p. 9

A. Mitigating Switching Overhead

ASTRA reduces object-detection pipeline switching latency from hundreds of milliseconds to 10–20 ms through buffer, memory, and reconfiguration optimizations.

A. Mitigating Switching Overhead

  • 10–20 ms switching latency is achieved by capping inter-element buffers at one frame and reusing small buffer pools.This reduces draining, flushing, and reallocation overhead for small video chunks.Source: A. Mitigating Switching Overhead · p. 9
  • Frames stay in GPU device memory through decoding, preprocessing, inference, and postprocessing, avoiding higher per-chunk latency from PCIe reads.ASTRA uses device-resident buffers and custom kernels on cudaMalloc pointers rather than CUDA zero-copy host memory.Source: A. Mitigating Switching Overhead · p. 9
  • Runtime reconfiguration updates configuration-dependent state without recreating pipeline elements and selects preloaded TensorRT engines.Idle models consume memory but add no computation.Source: A. Mitigating Switching Overhead · p. 9

VI. Evaluations

The evaluations compare iterative adaptation algorithms within ASTRA against offline-optimal and ground-truth-free benchmarks. They test accuracy–latency adaptation across camera models, configurations, and network conditions.

VI. Evaluations

  • ASTRA is evaluated with GP-UCB-C, a theoretically guaranteed method, and Chameleon++, an empirical profiling-based method, against offline-optimal and ground-truth-free baselines.The offline optimum uses full knowledge of accuracy and latency and serves as an upper-bound benchmark; a better individual metric does not imply a better feasible solution.Source: §VI · pp. 9–10
  • GP-UCB-C models accuracy and latency as correlated unknown functions with Gaussian Processes and selects configurations using confidence bounds and application constraints.The algorithm updates its feasible set and stops when the current best configuration's lower utility bound is no smaller than the alternative's upper bound.Source: §VI · p. 10
  • Chameleon++ adds latency measurements and estimates unprofiled latency using the parameter-independence assumption, while retaining Chameleon's search logic.Chameleon estimates accuracy from a small profiled subset using the same independence assumption.Source: §VI · p. 10
  • The tested configuration set contains 12 resolution–frame-rate pairs, and each experiment uses an F1-score threshold of 0.7 and 0.2-second time slots.Resolution levels are 832, 608, and 416; frame rates are 30, 15, 10, and 5 fps.Source: §VI · pp. 10–11
  • Evaluation covers two live COSMOS cameras and up to eight emulated cameras, with realistic network latency based on 4G measurements.The 4G dataset provides end-user-to-nearest-cloud-site RTT and jitter measurements.Source: §VI · p. 11

A. Deployment in COSMOS Testbed

ASTRA was deployed with GP-UCB-C on two live COSMOS cameras during peak and off-peak periods. It kept accuracy at or near the 0.85 threshold, with lower latency at night as traffic density fell.

A. Deployment in COSMOS Testbed

  • The COSMOS deployment tested real-time adaptation on two live intersection cameras using separate servers for processing and periodic adaptation.The deployment emulated realistic camera-to-server network latency.Source: A. Deployment in COSMOS Testbed · p. 11
  • ASTRA consistently maintained accuracy above or close to 0.85 across both 30-minute live-camera periods, with lower latency during off-peak hours.The authors attribute the off-peak latency reduction to lower pedestrian and vehicle density allowing lower resolutions and frame rates.Source: A. Deployment in COSMOS Testbed · p. 11, Figure 10 · p. 12
  • At higher traffic density, ASTRA upgraded the configuration to preserve accuracy, resulting in higher latency.At lower density, it reduced resolution and frame rate, lowering latency while meeting the accuracy target.Source: A. Deployment in COSMOS Testbed · p. 12

B. Controlled and Reproducible Evaluation

The controlled evaluation streamed a varied COSMOS video dataset through emulated cameras under realistic network conditions. GP-UCB-C remained near the offline optimum while using low GPU utilization and meeting the accuracy threshold in most time windows.

B. Controlled and Reproducible Evaluation

  • Three Google Cloud VMs streamed H264-encoded videos over RTP while two GPU-equipped instances handled real-time processing and adaptation.The instances were hosted in the us-east1 region.Source: B. Controlled and Reproducible… · p. 12
  • The controlled setup streamed 52 paired camera videos, selected to span traffic densities and weather, rather than reading recordings directly from local storage.The videos were streamed over the network to expose the system to real-time latency and throughput constraints.Source: B. Controlled and Reproducible… · p. 12
  • Sequential adaptation within repeating 60-second windows supported evaluation with up to eight emulated cameras.Each camera was adapted once per window, with a different adaptation offset.Source: B. Controlled and Reproducible… · p. 12
  • With 2, 4, and 8 cameras, GP-UCB-C stayed within 10% of offline-optimal performance, used about 2% GPU per camera, and met the 0.85 accuracy threshold in more than 90% of time windows.GPU utilization was around 2%, 4%, and 9% for the respective camera counts.Source: B. Controlled and Reproducible… · p. 12, Figure 11 · p. 13
  • Chameleon++ used GPU utilization close to GP-UCB-C but missed the 0.85 accuracy target in more than 30% of evaluated times.AutoML++ had the lowest GPU utilization, yet failed to meet minimum accuracy in around 75%, 85%, and 90% of times for 2, 4, and 8 cameras.Source: B. Controlled and Reproducible… · p. 13

C. Varying Network Latencies

Under increased network latency and jitter, GP-UCB-C performed closest to the offline optimum and varied least. The authors interpret the tests as measuring response to increased chunk-delivery latency, a consequence of several adverse network conditions.

C. Varying Network Latencies

  • The network tests varied latency and jitter using added Gaussian delay with means from 10 ms to 150 ms and variance of 20%, under a 250 ms latency constraint.The authors note that measured latencies in the referenced datasets and similar studies were relatively small.Source: C. Varying Network Latencies · p. 13
  • GP-UCB-C was closest to the optimal performance and showed the least variation as network latency and jitter varied.AutoML++ performance was unstable, while Chameleon++ and GP-UCB-C were more consistent.Source: C. Varying Network Latencies · p. 13
  • The authors interpret increased chunk-delivery latency as a common consequence of propagation delay, jitter, bandwidth limits, and packet loss.With TCP streaming, limited bandwidth can increase transmission and queueing time, while packet loss can trigger retransmissions.Source: C. Varying Network Latencies · p. 13

VII. Conclusions

ASTRA is a lightweight architecture for running diverse iterative adaptation algorithms in real-time video analytics. Paired with a principled algorithm, it tracks near-optimal configurations and achieves performance close to an offline optimal benchmark.

VII. Conclusions

  • With a principled algorithm, ASTRA consistently tracks near-optimal configurations and performs close to an offline optimal benchmark with full accuracy and latency knowledge and no overhead.The paper describes that offline benchmark as an upper bound on achievable performance.Source: §VII · p. 14
  • ASTRA supports a broad class of iterative adaptation algorithms, from theoretically grounded methods to empirical, profile-driven approaches, with fast configuration adaptation and minimal overhead.Source: §VII · p. 14
  • Future work will scale live evaluations, adapt time windows to rapid environmental fluctuations, and extend ASTRA to geo-distributed edge/cloud systems with dynamic resource allocation.Source: §VII · p. 14
Loading 2609.07020v2…