PaperlayerView exact revision on arXiv

Source-linked AI summary

Albireo: Adaptive, Energy-Efficient Inference Framework for Video Object Detection on the Edge

Amir Taherin, José Cano, Bin Ren, Yanzhi Wang, David Kaeli

arXiv:2609.29648v1cs.CVcs.ARcs.LGcs.PFcs.RO

TL;DR

Edge video detection expends substantial energy on per-frame detector passes, while adaptive deployment approaches still leave a gap in drop-in operation. ALBIREO wraps off-the-shelf detectors with per-object uncertainty triggering, rescue, and empty-scene screening; it reports accuracy close to per-frame inference with lower energy across its evaluated detectors and platforms.

  • Problem

    Per-frame detector passes dominate edge-video pipeline cost, while existing adaptive methods do not jointly provide the deployment properties sought for drop-in edge inference.

    Source: §I · pp. 1–2

  • Method

    ALBIREO wraps an unchanged detector with per-object Kalman uncertainty gating, detector-miss rescue, and a lightweight empty-scene screen.

    Source: §I · p. 2, A. Per-Object Kalman Filter and… · p. 4, B. Detector-Flicker Rescue · p. 5

  • Results

    Across evaluated detectors and platforms, ALBIREO keeps AP@50 within 1.2 pp of Vanilla while reducing energy by 12.1–17.6%.

    Source: §I · p. 2

  • Takeaways & Limitations

    Object-level temporal state can serve as a practical drop-in efficiency layer for off-the-shelf edge video detection pipelines.

    Source: §VII · pp. 13–14

    Constant-acceleration Kalman dynamics can drift under highly erratic motion, leaving prediction-only boxes inaccurate at abrupt motion.

    Source: D. Limitations and Future Work · pp. 11–12

Abstract

from arXiv · show

Video object detection on edge devices runs computationally expensive detectors over long frame streams, causing high energy consumption and sustained GPU utilization. Although consecutive frames are highly redundant, naive frame skipping is content-blind: it skips during critical moments such as object entry, occlusion recovery, and abrupt motion, degrading detection quality. We present Albireo, a detector-agnostic, codec-free adaptive inference framework that wraps off-the-shelf detectors and decides when detector invocation can be safely skipped based on scene content and per-object temporal state, requiring no detector modification or retraining. Albireo maintains a 10-dimensional Kalman filter (KF) per active object and invokes the detector only when prediction uncertainty exceeds a threshold; on skipped frames, boxes are predicted from the KF state at near-zero GPU cost. A KF-based rescue mechanism preserves confirmed objects through brief detector misses to prevent output fragmentation, while a lightweight empty-scene screen avoids detector calls on objectless frames. We evaluate Albireo on the BDD100K MOT validation split with three architecturally distinct detectors (YOLO11x, YOLO26x, RF-DETR-Large) on two NVIDIA Jetson platforms (AGX Thor, AGX Orin). Across all configurations, Albireo keeps AP@50 within +/-1.2 pp of per-frame inference while reducing total energy by 12.1-17.6%. On YOLO26x, it improves AP@50 by +0.8 pp while reducing energy by 17.6% (Thor) and 14.4% (Orin) and per-frame energy-delay product by 24.9% and 26.1%, respectively. Thus, the default operating point improves accuracy, energy, and latency together. In contrast, FixedSkip-2, a fixed-interval baseline with a 50% skip rate, loses 8.6 pp AP@50. Source code, evaluation pipeline, and per-clip results are available at https://github.com/amirtaherin/albireo

I. INTRODUCTION

ALBIREO addresses the energy cost of per-frame edge detection and the quality loss from content-blind frame skipping with adaptive, object-state-based detector gating. It combines uncertainty triggering, detector-flicker rescue, and empty-scene screening, and reports accuracy–energy–latency gains across detectors and Jetson platforms.

Motivation and problem

  • Fixed-cadence skipping loses 8.6 pp AP@50 when every second frame is skipped and 28.8 pp when four of five frames are skipped, because schedules miss content changes.The design therefore makes skip decisions using scene content and object-level spatial and temporal state.Source: §I · pp. 1–2

Forward-looking uncertainty trigger

  • The forward-looking trigger skips detection while every active object's projected Kalman covariance trace remains below a threshold, using current-state uncertainty rather than a previous detection as its reference.On BDD100K MOT, this trigger preserves higher AP@50 than CTD at similar skip rates.Source: §I · p. 2

Empty-scene screen

  • When no active object states exist, a lightweight binary empty-scene classifier gates the full detector on objectless frames.The classifier is fine-tuned for the evaluation domain; the object detector remains unchanged.Source: §I · p. 2

Empirical accuracy–energy–latency gains

  • +0.8 pp AP@50 on YOLO26x, with 17.6% lower energy and 24.9% lower per-frame EDP on Thor; on Orin, energy falls 14.4% and EDP 26.1%.Across all evaluated detectors and platforms, AP@50 stays within 1.2 pp of Vanilla while energy falls 12.1–17.6%.Source: §I · p. 2

System design

  • ALBIREO is a detector-agnostic, codec-free, content-aware wrapper that needs no detector modification or retraining and is not tied to fixed-camera assumptions.The paper presents it as a system for improving off-the-shelf detectors' accuracy–energy–latency tradeoff on commercial edge SoCs.Source: §I · p. 2

Detector-flicker rescue

  • A rescue rule emits a confirmed object's Kalman-predicted box after a brief detector miss, raising AP@50 to 0.617 versus 0.572–0.578 for no-rescue variants.Rescue requires a high-confidence most-recent detector match and an unmatched-frame count below a fixed cap.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

The background motivates detector invocation as the energy-control point on edge SoCs, especially for moving-camera video, and explains the per-object Kalman state that ALBIREO reuses. The detector interface and covariance-based scheduling role frame the system's design.

A. Edge Inference: Detectors, Camera Motion, and Energy

  • The detector families expose class-labeled boxes and confidence scores, allowing a wrapper to control invocation without changing detector architecture or retraining.The evaluation context includes fixed- and moving-camera video, with ego-motion identified as a harder frame-skipping regime.Source: A. Edge Inference: Detectors, Camera… · p. 3
  • Detector forward passes dominate GPU workload, making detector invocation a natural energy-control point; CPU bookkeeping must remain cheap enough not to erase savings.ALBIREO targets frame-level gating rather than detector-architecture modification.Source: A. Edge Inference: Detectors, Camera… · p. 3

B. Kalman Filters for Per-Object Temporal State

  • A Kalman filter's covariance grows during repeated prediction without measurement updates, quantifying how stale an estimate has become.ALBIREO repurposes per-object covariance growth as a forward-looking detector-scheduling signal, rather than using covariance only for association gating.Source: B. Kalman Filters for Per-Object… · p. 3

C. Naive Frame Skipping Fails

Fixed-cadence skipping can harm detection quality because it ignores scene changes, while existing adaptive approaches leave deployment requirements unmet. ALBIREO uses per-object uncertainty in a detector wrapper and pairs it with rescue and empty-scene mechanisms.

C. Naive Frame Skipping Fails

  • FixedSkip-2 loses 8.6 pp AP@50 on BDD100K MOT despite skipping only every other frame, illustrating the risk of content-blind scheduling.Fixed schedules can skip when objects enter, occlusions clear, or objects accelerate.Source: C. Naive Frame Skipping Fails · p. 3

D. Existing Adaptive Approaches Are Insufficient

  • Prior approaches trade off deployment properties: feature propagation couples to detector backbones and retraining, pixel differencing falters under ego-motion, CTD uses a stale reference, and compressed-domain methods require codec metadata.ALBIREO targets the gap with per-object uncertainty gating of off-the-shelf detectors without fixed-camera or codec assumptions.Source: D. Existing Adaptive Approaches Are… · p. 4

III. ALBIREO SYSTEM DESIGN

  • ALBIREO combines per-object Kalman-covariance triggering, rescue of confirmed states through brief detector misses, and lightweight empty-scene screening without detector modification or retraining.The uncertainty threshold u is the primary control for selecting an accuracy–efficiency operating point.Source: §III · p. 4
  • The system targets causal, online, single-stream edge inference using only the current frame and carried state; it makes no hard real-time or safety-critical guarantees.Recorded clips are replayed for reproducible accuracy, latency, and power measurement.Source: §III · p. 4

A. Per-Object Kalman Filter and Uncertainty Trigger

ALBIREO uses per-object Kalman uncertainty to decide when to invoke the detector, with safety caps on prediction-only runs. A rescue rule preserves confirmed objects through brief detector misses.

A. Per-Object Kalman Filter and Uncertainty Trigger

  • A 10-dimensional Kalman filter represents each active object’s box center, size, and temporal derivatives in normalized image coordinates.It uses constant-acceleration dynamics for box centers and constant-velocity dynamics for box size.Source: A. Per-Object Kalman Filter and… · p. 4
  • Detector calls are skipped when every active state’s projected covariance trace is below threshold u, subject to caps of 15 prediction-only frames and one call every 30 frames.Lower u invokes the detector more often; higher u permits longer prediction-only stretches.Source: A. Per-Object Kalman Filter and… · pp. 4–5
  • With fixed process noise Q, covariance growth tracks time since correction rather than motion magnitude, potentially over-triggering for slow objects or lagging fast ones.Motion-adaptive process noise is identified as future work.Source: A. Per-Object Kalman Filter and… · p. 5

B. Detector-Flicker Rescue

  • On inference frames, confirmed unmatched objects can emit predicted boxes after a recent detector match with confidence ≥0.5, for at most two consecutive rescues.Rescued states receive no Kalman update, so uncertainty continues to grow until a real match or another detector invocation.Source: B. Detector-Flicker Rescue · p. 5

C. Empty-Scene Screening

When no object states are active, ALBIREO uses a lightweight empty-scene classifier to avoid full detector calls on empty frames. Its frame-state routing combines this screen with prediction, uncertainty-triggered inference, and rescue; evaluation covers multiple detectors and edge platforms.

C. Empty-Scene Screening

  • ERD, a fine-tuned five-block CNN with about 14k parameters and 1.3 ms GPU latency, gates detector calls only when no active object states exist.An empty prediction skips detection; a non-empty prediction invokes it and initializes object states.Source: C. Empty-Scene Screening · p. 5
  • New objects in non-empty scenes may be discovered only at the next detector invocation, with periodic caps bounding the delay; skipped-frame discoveries count as false negatives in AP@50..Source: C. Empty-Scene Screening · p. 5

D. Frame-State Taxonomy and Algorithm

  • The per-frame algorithm predicts and prunes object states, screens empty scenes, checks uncertainty and safety caps, then matches detections and rescues eligible unmatched states.Its mutually exclusive frame states identify whether output came from ERD skipping, prediction, detector inference, or inference augmented by rescue.Source: D. Frame-State Taxonomy and Algorithm · pp. 5–6

A. Experimental Setup

  • Evaluation uses 200 BDD100K MOT dashcam clips, three off-the-shelf detectors, two Jetson platforms, and paired runs against per-frame and frame-skipping baselines.The seven annotated object classes are evaluated; AP@50 is the primary accuracy metric.Source: A. Experimental Setup · pp. 6–7

B. Pareto Dominance Across Three Axes

On YOLO26x, ALBIREO’s default operating point improves accuracy while reducing energy and latency relative to per-frame inference. Across the target accuracy region, its threshold sweep forms the upper Pareto frontier among evaluated systems.

B. Pareto Dominance Across Three Axes

  • +0.8 pp AP@50, 17.6% lower energy per clip on Thor (14.4% on Orin), and 8.8% lower average frame time on Thor (13.7% on Orin) accompany 24.9% lower EDP on Thor (26.1% on Orin) versus Vanilla.The default operating point Pareto-dominates per-frame inference across AP@50, energy, and latency.Source: B. Pareto Dominance Across Three Axes · p. 7, Figure 2 · p. 8
  • Within AP@50 ≥0.50, ALBIREO’s threshold sweep forms the upper Pareto frontier on both platforms; on Thor, Vanilla requires 33% more EDP than the default point while achieving lower AP@50.At comparable EDP, baselines achieve lower AP@50; at comparable AP@50, they require higher EDP.Source: Figure 2 · p. 8, B. Pareto Dominance Across Three Axes · p. 8
  • At u = 3×10−3, ALBIREO reaches 55% energy savings but AP@50 falls to 0.46, outside the target operating region..Source: B. Pareto Dominance Across Three Axes · p. 8

C. Comparison to Prior Frame-Skipping Baselines

Content-blind FixedSkip, Kalman-triggered CTD, pixel-difference Statues, and ERD-only screening each expose limits that ALBIREO’s content-aware, object-state approach is designed to address. Their comparisons show accuracy losses, weaker energy savings, or added overhead under the evaluated settings.

C. Comparison to Prior Frame-Skipping Baselines

  • FixedSkip-2 loses 8.6 pp AP@50, while FixedSkip-3 and FixedSkip-5 lose 16.7 pp and 28.8 pp, respectively.FixedSkip-3 and FixedSkip-5 reach lower energy and frame times than ALBIREO only at AP@50 values of 0.44 and 0.32.Source: C. Comparison to Prior Frame-Skipping… · p. 8
  • At about 40% skipping, ALBIREO reaches AP@50 = 0.564 versus CTD-0.30’s 0.506; near AP@50 ≈0.577, CTD-0.90 requires about 16% more EDP.CTD’s confidence score references the most recent detector output, which becomes stale across consecutive skipped frames; ALBIREO triggers on covariance growth from the current object state.Source: C. Comparison to Prior Frame-Skipping… · p. 8
  • Statues-500 reaches AP@50 = 0.578 at 650 J, while ALBIREO at u = 5×10−4 reaches AP@50 = 0.593 at 602 J.Across the tested thresholds, Statues skips 3.6–15.5% of frames under dashcam ego-motion; at its lowest threshold, Statues-25 skips only 3.6%.Source: C. Comparison to Prior Frame-Skipping… · pp. 8–9
  • ERD-only skips 7.6% of frames, yet raises per-clip energy by 28% and average frame time by 134% on Thor.The ERD CNN runs on every frame, and the full detector still runs on about 92% classified as non-empty.Source: C. Comparison to Prior Frame-Skipping… · p. 9

D. Detector-Agnostic Behavior Across Edge Platforms

Across three detectors and two Jetson platforms, ALBIREO reduces energy while keeping AP@50 close to per-frame inference. Its frame-state behavior and component ablation identify rescue as a major contributor to preserving detection quality.

D. Detector-Agnostic Behavior Across Edge Platforms

  • Across six detector–platform pairs, ALBIREO keeps AP@50 within ±1.2 pp of Vanilla and saves 12.1–17.6% energy; savings are significant in every pair.On 82–95% of individual clips, ALBIREO consumes less energy than Vanilla.Source: D. Detector-Agnostic Behavior Across… · p. 9
  • Across a 4.6× power-budget spread, energy savings stay within 15.3–17.7%, while YOLO26x’s +0.8 pp AP@50 gain and 19.3% skip rate remain identical.The energy savings are largest at Orin’s strict 15 W budget; the passage identifies power-budget interaction and end-to-end latency as future work.Source: D. Detector-Agnostic Behavior Across… · p. 9, Table V · p. 9

E. Frame-State Behavior

  • Skip rates are 19.3% for YOLO26x, 21.3% for YOLO11x, and 22.2% for RF-DETR; rescue occurs on 43.5–48.9% of all frames.Rescue is used on 56–61% of detector-invocation frames, while ERD-triggered inference is about 1.3%.Source: E. Frame-State Behavior · pp. 9–10, Figure 4 · p. 10

F. Ablation: Component Contribution

  • Rescue raises AP@50 to 0.617 from 0.572–0.578 in the no-rescue variants, making it the largest AP@50 contributor among the tested components.The ablation compares Kill-only, +ERD, +10D KF, and full ALBIREO on BDD100K MOT with YOLO26x on Thor.Source: F. Ablation: Component Contribution · p. 10

G. Per-Domain Energy and Workload Shift

ALBIREO’s energy savings primarily reflect reduced GPU activity, while per-frame CPU bookkeeping has platform-dependent costs. Per-clip behavior also varies with scene content: empty-scene skipping offers large savings in sparse clips, whereas dense clips emphasize rescue and accuracy preservation.

G. Per-Domain Energy and Workload Shift

  • YOLO26x GPU energy drops by 17.5% on Thor and 19.0% on Orin, exceeding total-energy reductions.GPU power is 34 W of Thor’s 70 W total and 31 W of Orin’s 43 W total under per-frame inference; grouped rails are for trend analysis and do not necessarily sum to top-level board energy.Source: G. Per-Domain Energy and Workload Shift · p. 10, Figure 5 · p. 10
  • On Orin, ALBIREO raises average CPU power from 1.6 W to 3.6 W, while on Thor it changes from 13.6 W to 13.0 W.Per-frame Kalman prediction, matching, and rescue accounting add CPU work; GPU savings dominate and total energy still drops on both platforms.Source: G. Per-Domain Energy and Workload Shift · p. 10

H. Per-Clip Behavior

  • On Thor with YOLO26x, ALBIREO matches or exceeds Vanilla AP@50 on 144 of 200 clips; on the other 56, the median drop is below 2 pp.The per-clip comparison decomposes skip rate into ERD-empty, Kalman-predict, and combined components.Source: H. Per-Clip Behavior · pp. 10–11
  • Across the full 0–36% Kalman-predict skip range, per-clip AP@50 deltas remain centered near zero; the largest negative deltas cluster above 40% ERD-empty skipping.On sparse clips, a one-frame delay between an ERD non-empty verdict and detector recovery can shift AP@50 by several points.Source: H. Per-Clip Behavior · p. 11
  • A dense urban clip with 96% augmented-inference frames has similar ALBIREO and Vanilla power, while a sparse highway clip with 69% ERD-empty frames shows sharply reduced ALBIREO GPU power.Dense clips emphasize rescue and accuracy preservation; sparse clips expose larger savings through empty-scene skipping.Source: H. Per-Clip Behavior · p. 11

A. Comparison with the Closest Prior Work

ALBIREO differs from nearby edge-inference methods in its object-state skip signal and deployment assumptions. The discussion also identifies CPU, memory, composability, and evaluation boundaries for deployment.

A. Comparison with the Closest Prior Work

  • At a matched skip rate of ∼40%, ALBIREO reaches AP@50 = 0.564 while CTD reaches 0.506; unlike Statues, its trigger avoids a fixed-camera assumption.CTD compares predictions with the most recent detection, while ALBIREO uses covariance growth; Statues reuses stale detections and was evaluated on custom hardware.Source: A. Comparison with the Closest Prior… · p. 11

B. Deployment Implications of CPU-Side Bookkeeping

  • Despite CPU bookkeeping, total energy drops on both platforms, and ALBIREO adds only 2–27 MB of system memory, below 0.7% in every configuration.CPU power overhead is visible on Orin but not measurably higher on Thor; smaller SoCs should account for bookkeeping when choosing thresholds and rescue policies.Source: B. Deployment Implications of CPU-Side… · p. 11

C. Composability with Model Compression

  • Compression and ALBIREO act on different parts of the pipeline, but their combination was not evaluated because calibrated compressed detectors require a separate accuracy–energy study.Compression lowers the cost per detector invocation, whereas ALBIREO reduces how often invocations occur.Source: C. Composability with Model Compression · p. 11

D. Limitations and Future Work

  • Constant-acceleration Kalman predictions can drift under erratic motion, leaving prediction-only boxes inaccurate at abrupt moves; learned residuals or class-conditioned motion models are proposed.The passage gives a pedestrian making an abrupt lateral move at a crosswalk as an example.Source: D. Limitations and Future Work · pp. 11–12
  • The empty-scene screen is fine-tuned for BDD100K dashcam scenes and does not automatically transfer to indoor, aerial, or robot-mounted cameras.Future deployments should fine-tune the classifier for the target domain or replace it with a more general low-cost emptiness oracle.Source: D. Limitations and Future Work · p. 12
  • The rescue confidence threshold and consecutive-rescue cap are fixed heuristics, while object classes, sizes, and motion may call for different rescue lifetimes.An adaptive decay policy is proposed as a possible way to further improve the AP@50–energy tradeoff.Source: D. Limitations and Future Work · p. 12
  • Evaluation focuses on BDD100K ego-motion video and a single detection stream; fixed-camera, non-driving, and co-tenant workloads remain untested.The paper identifies these evaluations as future work.Source: D. Limitations and Future Work · p. 12

VI. RELATED WORK

Prior work reduces video-detection cost through temporal propagation, frame skipping, and adaptive computation. These approaches differ in detector modification, scheduling assumptions, and which system cost they target.

A. Temporal Feature Propagation

  • Temporal feature-propagation methods reuse features across frames to improve the speed–accuracy tradeoff, but typically require detector-specific modules, architecture changes, or retraining.Examples include optical-flow or learned aggregation methods and temporal memory or tubelet approaches.Source: A. Temporal Feature Propagation · p. 12
  • Seq-NMS improves weak detections through temporal consistency, but the detector still runs on every frame and therefore does not directly reduce edge-energy cost.It links detections across frames as a post-processing step.Source: A. Temporal Feature Propagation · p. 12

B. Frame Skipping and Adaptive Inference

  • Fixed-interval and heuristic frame-skipping methods share ALBIREO’s goal of reducing detector calls, but can skip frames when fresh detections are needed.Examples include optical-flow tracking between fixed-interval human detections and dynamic skipping for drone-based detection.Source: B. Frame Skipping and Adaptive Inference · p. 13
  • Below-frame computation methods reduce work through cropping, skipped convolutions, or patch routing, while learned schedulers require offline training or tuning tied to a detector, dataset, or deployment.Patch-level routing and model modification are required by the cited patch-computation approaches.Source: B. Frame Skipping and Adaptive Inference · p. 13
  • Energy-focused studies report savings from frame dropping or adaptive frame rates, while streaming-perception work targets latency-aware scheduling under asynchronous arrivals.These address different scheduling choices from whether to invoke the detector at all.Source: B. Frame Skipping and Adaptive Inference · p. 13

C. Compressed-Domain Video Inference

Compressed-domain systems reuse codec information, whereas ALBIREO operates on decoded frames; model compression and tracking-based detection exploit different forms of temporal or computational efficiency. The conclusion reports energy savings with near-per-frame AP@50 and a favorable Pareto position over evaluated skipping baselines.

C. Compressed-Domain Video Inference

  • Compressed-domain methods propagate detections using codec cues such as motion vectors and residuals, but depend on codec metadata or codec-imposed keyframe schedules.ALBIREO instead uses decoded frames and object-state covariance without codec infrastructure or a fixed GOP schedule.Source: C. Compressed-Domain Video Inference · p. 13

D. Model Compression and Tracking-Based Detection

  • Model compression reduces the cost of each detector invocation, complementing ALBIREO’s reduction in invocation frequency.The cited techniques include quantization, pruning, and knowledge distillation.Source: D. Model Compression and… · p. 13
  • Tracking-based detection methods use trained temporal modules or joint detector modification, whereas ALBIREO uses an external Kalman object state to gate inference and emit boxes after brief misses.The cited prior methods jointly train detection and tracking or predict future object locations to maintain detections through temporary misses.Source: D. Model Compression and… · p. 13

VII. CONCLUSION

  • 12.1–17.6% energy reduction while keeping AP@50 within 1.2 pp of per-frame inference across BDD100K MOT, three detectors, and two Jetson generations.On YOLO26x, the default operating point Pareto-dominates per-frame inference on AP@50, energy, and latency; its uncertainty-threshold sweep forms the upper frontier over FixedSkip-N, CTD, and Statues at AP@50 ≥ 0.50.Source: §VII · pp. 13–14
Loading 2609.29648v1…