PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Neural Assisted Actor-Critic for Latency-Efficient Edge Vision System

Alam Noor, Luis Almeida, Kai Li, Jiyan Wu, Miguel Gutiérrez Gaitán, Eduardo Tovar

arXiv:2608.16142v1cs.CVcs.AI

TL;DR

UAV edge-vision systems need lower-latency video transmission while preserving accurate object detection. The proposed GCN-assisted A2C selects correlated pixel regions for transmission, achieving 45 ms inference latency—about half that of competing approaches—with 70.72% average precision and 60.30% mean IoU.

  • Problem

    Region-of-interest transmission can reduce latency, but pixel correlations with the surrounding frame environment must be considered for accurate video analysis.

  • Method

    GCN-assisted A2C selects feature-correlated pixel regions around detected objects and optimizes their transmission latency subject to identification accuracy.

  • Results

    45 ms inference latency—approximately half that of competing approaches—was achieved alongside 70.72% average precision and 60.30% mean IoU.

  • Takeaways & Limitations

    The model reduces UAV video transmission latency while improving region-of-interest selection and detection performance relative to benchmark methods.

Abstract

from arXiv · show

UAV on-board vision systems are widely used for different activities, including monitoring in no-fly zones. In this case, the vision-equipped UAV streams a video to a ground server where an operator assists its activities. The latency of video transmission has a profound impact on the effectiveness of the operator assistance. However, most techniques available for video transmission still incur significant latency costs. In this paper, we propose a graph convolutional neural network-assisted (GCN-Assisted A2C) deep reinforcement learning (DRL) system model to find the optimal pixel-correlated area of a suspicious object. We combine the Lagrangian dual form with gradient descent to prevent lack of convergence and over- and under-penalization constraint violation during latency optimization. The proposed system model sends a sub-group pixel-correlated area of the frame from the UAV to the server rather than the transmission of the whole video frame. The proposed framework utilizes the GCN model to explore hidden representations of feature-correlated groups of pixels. Moreover, the GCN supervises the A2C model, which selects a subgroup to enhance transmission latency, thus supervising the training of UAV actions in A2C. Experimental results show that GCN-assisted A2C reduces video frame transmission latency together with false detection rate in UAV vision systems over other DRL and state-of-the-art models.

I. INTRODUCTION

UAV edge-vision applications require accurate real-time detection, but streaming full video creates latency, motivating transmission of correlated regions of interest. The proposed GCN-assisted A2C selects pixel-correlated regions and optimizes transmission delay, achieving 45 ms inference latency with improved detection metrics.

  • Motivation: UAV vision applications stream video to ground servers for real-time detection, where transmission latency affects operator assistance and detection effectiveness.Applications include surveillance in no-fly zones and detection of large and small objects in dynamic environments.
  • Motivation: Transmitting selected regions of interest reduces transferred information and transmission time, but pixel correlations and system parameters affect video-analysis performance.Relevant parameters include image resolution, encoding rate, and server-side neural-network architecture.
  • Proposed approach: The proposed GCN-assisted-A2C model uses GCN representations to identify correlated pixel regions and A2C selection to control which regions are transmitted.YOLO-Nano supplies detected-object bounding boxes, whose centers guide GCN correlation discovery; A2C selects the transmitted subgroup.
  • Proposed approach: The model combines a SLIC-supervised GCN with an A2C whose reward uses the Lagrangian dual form and dual gradient descent to optimize latency constraints.The dual update targets convergence and avoids over- or under-penalization of constraint violations.
  • Results: 45 ms inference latency, approximately half that of competing approaches, was achieved alongside 70.72% average precision and 60.30% mean IoU on a YouTube dataset.The comparison used FlexPatch, EdgeDuet, and DRL as benchmark models.

II. RELATED WORK · III. MATHEMATICAL SYSTEM MODELING

Prior work reduces UAV video latency through transport, feature-based, edge-cloud, DRL, and region-of-interest methods, but may transmit unnecessary frame areas. The proposed model selects RoIs under an AP target and formalizes their area and packet-loss-aware transmission time for UAV–Edge communication.

  • II. RELATED WORK: Low-latency UAV video systems use QUIC/TS delivery, feature-based transmission, air-to-ground coordination, and public-safety communication models.Dong et al. achieved latency below 200ms with UDP-based QUIC and TS.
  • II. RELATED WORK: Edge-cloud collaboration, splitting-merging streaming, multi-link retransmission, and DRL-based soft actor-critic models target latency and packet-loss reduction.These approaches combine computation placement, streaming strategies, retransmission, or reinforcement learning to improve transmission efficiency.
  • II. RELATED WORK: RoI and object-centric methods transmit relevant image areas, including EdgeIS segmentation, motion tracking, mask transfer, and contour-instructed edge inference acceleration.These techniques aim to reduce latency by processing or transferring object-relevant regions rather than entire frames.
  • II. RELATED WORK: Only a few prior works target flying-object detection latency, while some still transmit areas irrelevant to detection, increasing packets, bandwidth use, latency, and packet losses.The paper compares against EdgeDuet and Flexpatch, which are identified as the most closely related works.
  • II. RELATED WORK: The proposed approach identifies RoIs, transmits only those regions, and uses an A2C DRL policy to select the smallest RoIs satisfying a given AP target.The Edge server detects small objects within the selected RoIs despite poor video quality or environmental effects.
  • III. MATHEMATICAL SYSTEM MODELING: Each encoded UAV frame has spatial size W × H, while an onboard detector produces a bounding box and the transmitter chooses a centered RoI parameterized by action Et and scale factors αt and βt.The bounding box is bt = (xc, yc, ζW, ζH), with center (xc, yc) and detected-object dimensions.
  • III. MATHEMATICAL SYSTEM MODELING: For N RoIs, the n-th region has area |ft,n| = (1 + 2Et,n)^2 ζW ζH, which determines the information volume and therefore affects transmission time.The symmetric expansion uses αt = βt = 1 + 2Et, with Emin ≤ Et ≤ Emax.
  • III. MATHEMATICAL SYSTEM MODELING: The transmission-time model accounts for packet loss and stop-and-wait ARQ retransmissions using wireless rate R, min RTT, packet success probability qt,n, and NACK size B.Packet loss triggers retransmissions, extending RoI transmission time and affecting the target delay constraint.

A. Accuracy Constraints Strategy

The framework constrains detection accuracy through careful selection and scaling of transmitted RoI sizes, accounting for UAV size and object–camera distance. It estimates server-side accuracy from RoI size and uses correlation-based, differentiable optimization to meet an accuracy threshold while reducing latency.

  • Constraint formulation: Frame-cropping scale strongly influences UAV detection accuracy because UAV sizes vary and object–camera distances differ.These factors motivate careful selection of the transmitted sub-area scale.
  • Accuracy model: Server-side accuracy A_t,n is modeled as a function of RoI size using the probability of correct detection given the received RoI.The probability term represents the likelihood that the server correctly detects an intruding UAV from each transmitted sub-area.
  • Adaptive RoI scaling: The inspecting UAV estimates server-side accuracy and iteratively scales the RoI until the desired threshold A_threshold is met.The procedure updates the scaling factor E(k+1), beginning from E(0).
  • Optimization strategy: Correlation-based GNN selection is preferred over retrieval and linear optimization because it is differentiable and avoids discontinuous accuracy and low-correlation region-selection problems.The scaling update uses V_E proportional to the pixel-group correlation around the sub-area center, while GNN-based area identification produces a greater latency decrease than linear optimization.

B. Problem Formulation

The problem formulation minimizes total transmission latency by selecting and scaling cropped pixel-correlated sub-areas while accounting for packet transmissions and propagation/processing delays. It constrains each received sub-area frame to meet a server-side accuracy threshold.

  • Latency objective: The number of sub-area frame transmissions is critical for keeping delay below a single full-frame transmission delay.Total transmission time is expressed as the sum of individual sub-area frame delays.
  • Latency objective: The optimization minimizes total delay using estimated packet sizes, send rate R, and minimum one-way propagation/processing delay.The objective is stated for transmissions characterized by packet sizes S_t,n, send rate R, and min RTT/2.
  • Latency objective: Each cropped sub-area f_t,n is adapted with a variable scaling factor E_t,n per packet to reduce transmission latency.The formulation also accounts for the number of sub-area transmissions and uses pixel-correlation-based precision probability P(.) to maintain stable server-side accuracy A_t,n.
  • Accuracy constraint: The optimization selects sub-area frames {f_t,n}^N while requiring A_t,n ≥ A_threshold for n = 1, ..., N.This constraint enforces the minimum server-side accuracy for every received sub-area frame.

C. Complexity

The optimization problem is a constrained MDP with continuous actions and a nonconvex, data-driven accuracy function, making exact dynamic programming intractable. A model-free primal–dual actor–critic method is therefore used, while simplified discrete monotone cases admit independent O(NK) optimization.

  • Complexity: The constrained MDP has continuous actions and a nonconvex, data-driven accuracy function, so exact dynamic programming is intractable.The adopted method learns policies that minimize expected delay while satisfying accuracy constraints.
  • Complexity: A model-free primal–dual actor–critic method learns policies minimizing expected delay while satisfying accuracy constraints.
  • Complexity: O(NK) optimization is possible for each sub-area frame when actions use K discrete levels and accuracy and transmission time are monotone in the action.The simplified case optimizes each sub-area independently by scanning the grid.
  • Complexity: Temporal correlations, joint constraints, and stochastic packet success create a high-dimensional constrained MDP for which reinforcement learning methods are more appropriate.

IV. PROPOSED GCN-ASSISTED DRL SYSTEM MODEL · A. CMDP Formulation and Information Flow

The proposed GCN-assisted A2C model extracts correlations among pixel groups and uses them to guide latency- and accuracy-constrained region-selection decisions. Its CMDP represents detector context, GCN embeddings, and constraint multipliers, while actions select correlated bounding-box regions whose transmission outcomes update the policy online.

  • IV. PROPOSED GCN-ASSISTED DRL SYSTEM MODEL: GCN extracts feature areas related to the object center and correlations among pixel groups, while A2C uses these outcomes to train latency-optimized, accuracy-constrained actions.The GCN supports future prediction by identifying strongly related features and pixel groups.
  • IV. PROPOSED GCN-ASSISTED DRL SYSTEM MODEL: The model addresses RoI-scaling limitations by exploiting pixel correlations and complex feature distributions across UAV sizes, lighting conditions, and motion.The earlier action abstraction used the RoI scaling factor Et,n to adjust cropped sub-area frame ft,n.
  • A. CMDP Formulation and Information Flow: The CMDP state combines detector features, a GCN embedding of pixel-group correlations around (xc, yc), and the multiplier λt for latency–accuracy constraints.Detector features include bounding boxes and scores.
  • A. CMDP Formulation and Information Flow: The GCN forms gt from feature vectors derived from pixel-group sets S = {Si}Z, compactly encoding correlations between disjoint groups for A2C decision-making.The passage defines gt = V and describes it as a compact correlation encoding.
  • A. CMDP Formulation and Information Flow: The CMDP action selects a bounding box Bt,n from the GCN-predicted correlated pixel group S∗i.The action is defined through the correlated pixel-group selection.
  • A. CMDP Formulation and Information Flow: Each selected bounding box satisfies Bt,n ≡ ft,n, converting pixel correlations into RoI actions that the A2C policy transmits to the server.The bounding box defines a sub-area RoI in the original optimization formulation.
  • A. CMDP Formulation and Information Flow: The CMDP aggregates contextual information beyond the optimization variables so the RL policy can generate practical decisions ft,n that solve the transmission problem.The optimization variables are ft,n, Et,n, Tt,n, and At,n.
  • A. CMDP Formulation and Information Flow: Server feedback on accuracy and measured delay updates estimated accuracy As and success probability qt online, balancing latency Tt,n against accuracy At,n.Accuracy is observed through AP or an IoU proxy.

B. GCN-A2C for Decision Making · C. Hidden Pixels Relation Exploration

The proposed GCN-assisted A2C framework uses graph-predicted correlated pixel groups to guide latency-optimized sub-area selection instead of exhaustive action search. Its GCN explores hidden pixel relations by aggregating pixel-level features and weighting edges according to correlation strength.

  • B. GCN-A2C for Decision Making: The actor selects an action comprising a correlated pixel group S_i predicted by the GCN and a bounding box B_t,n defining the cropped sub-area.Each B_t,n corresponds to an RoI f_t,n with scaling factor E_t,n.
  • B. GCN-A2C for Decision Making: A2C optimizes latency for the OP problem formulated from equations (2) to (6).
  • B. GCN-A2C for Decision Making: Exhaustively evaluating correlated pixel groups of size M_i among K pixels requires K!/[M_i!(K−M_i)!] = O(K^M_i), making brute-force search infeasible.K is the total number of correlated pixels near the object center, and M_i = |S_i| is the selected subset cardinality.
  • B. GCN-A2C for Decision Making: The GCN predicts future feature-correlated actions for A2C, reducing search complexity while satisfying A_t,n.The system selects pixel groups and captures stronger correlations between the center and neighboring hidden pixels.
  • C. Hidden Pixels Relation Exploration: The framework partitions each image frame into different regions to identify sub-areas whose pixels have high correlations.Frame partitioning is described as a critical stage for finding regions of interest in low-level vision systems.
  • C. Hidden Pixels Relation Exploration: In the framework, the actor selects actions, the critic evaluates action values, and GCN future-state prediction regulates actor behavior to optimize latency and minimize errors.
  • C. Hidden Pixels Relation Exploration: GCN convolution operates over large areas of unevenly correlated pixels, producing aggregated pixel-level features rather than pixel-level features.
  • C. Hidden Pixels Relation Exploration: The GCN averages pixels within each set, assigning strong edge weights to similarly correlated areas and weak or no weights to uncorrelated areas.

1) State: · D. A2C for Tf Optimization with At,n Satisfaction

The state comprises disjoint correlated pixel groups whose mean features form GCN input nodes while preserving the total pixel count. The GCN predicts future correlation patterns for bounding-box cropping and is trained with focal classification objectives that balance within-box consistency against background spurious correlations.

  • 1) State:: The state is formed from aggregate groups of correlated pixels, denoted S_i, that constitute the observation.
  • 1) State:: Each correlated-pixel group S_i has cardinality M_i, and groups are disjoint while preserving all ζ_H × ζ_W pixels.
  • 1) State:: The representation summarizes every group by its mean feature vector, producing a Z-dimensional column vector V.
  • 1) State:: These averaged group features capture aggregated-region correlation structure and serve as input nodes for the GCN.
  • 1) State:: The GCN predicts future pixel-correlation patterns that guide action selection by cropping the corresponding bounding-box region.

1) Action: · 2) Reward:

The A2C action selects an adaptive bounding box around a GCN-predicted correlated pixel group, while the reward optimizes latency subject to an accuracy constraint. Dynamic Lagrangian updates address constraint-violation penalization and guide actor-critic learning toward accurate, latency-reducing transmissions.

  • 1) Action:: The A2C action is a bounding box Bt,n associated with a GCN-predicted correlated pixel group S∗i, specifying the region transmitted by the UAV.The GCN partitions images into pixel groups and supplies the correlation information used to define the transmitted region.
  • 1) Action:: A correlation-weighted centroid (xc, yc) is computed from pixel coordinates using correlation weights to locate compact rectangular or square transmission regions.The centroid uses coordinates of pixels Φ ∈ S∗i with weights CΦ.
  • 1) Action:: The bounding-box width and height adapt to pixel-group variability around the centroid, improving expressiveness and accurate capture of correlated regions.The dimensions are denoted W and H and depend on the distribution around the centroid.
  • 2) Reward:: The UAV action maps the GCN-predicted pixel group S∗i to the transmitted group Si, but reward optimization requires At,n ≥ Athreshold for positive server-side object detection.This accuracy condition prevents latency optimization from proceeding without the required detection accuracy.
  • 2) Reward:: The Lagrangian dual reward penalizes the objective when At,n < Athreshold, while dynamic adjustment of λ addresses fixed-penalty convergence and over- or under-penalization problems.The learning rate η is initially 0.01 and the weight λ is initially 1; η dynamically adjusts λ during constraint violation.
  • 2) Reward:: The combined loss Lloss = −log πθ(at | Φt) · δt + Wc · (δt)2 maximizes actor advantage while minimizing critic temporal-difference error.Wc is the weighted factor for the critic loss.
  • 2) Reward:: The policy πθ(at|Φt) maps CMDP states to action probabilities, with each action selecting a bounding box Bt,n that defines the ROI ft,n transmitted.The actor loss uses the advantage signal δt to encourage latency reduction while maintaining required accuracy.

3) Policy: … B. Ablation Study

The paper evaluates GCN-assisted A2C for UAV-to-UAV onboard detection and compares its latency, AP, and mean IoU against DQN, DDPG, and A2C. The ablation results indicate that GCN assistance improves feature extraction, scalability, robustness, and transmission efficiency.

  • V. EXPERIMENTAL WORK: The proposed GCN-assisted A2C model evaluates onboard intruding-UAV detection using 100 YouTube clips spanning 240 to 1080 resolution and varied backgrounds.Experiments ran on Ubuntu 22.04 with Python 3.9, PyTorch, and an A300 GPU.
  • A. Implementation of GCN-assisted A2C Model: The model combines multiple GCN layers with multi-layer perceptron layers to process correlated pixel features.Its GCN module contains three multi-head layers, each with two edge softmax heads.
  • A. Implementation of GCN-assisted A2C Model: UAV actions are trained in A2C for 1500 steps with a 0.99 discount factor, while the value-function loss receives coefficient 0.5.The loss averages squared differences between estimated values and actual returns.
  • B. Ablation Study: During training, A2C, DDPG, and DQN generally minimize transmission latency, while synchronous A2C performs better than deterministic DDPG.GCN assists A2C by extracting informative UAV features through aggregation and propagation across graph nodes.
  • B. Ablation Study: GCN-assisted A2C reduces dimensional focus to specific interdependent areas, supporting efficient handling of energy or bandwidth constraints.Without GCN, A2C faces scalability and computation challenges; GCN aggregation also improves robustness to noise and missing pixel information.
  • B. Ablation Study: 45 ms is the GCN-assisted A2C overall FPS transmission latency, lower than the other benchmark DRL algorithms.The ablation evaluates latency for each video-frame transmission, overall AP, and mean IoU.
  • B. Ablation Study: 30.55% AP and 28.61% mean IoU are achieved by DDPG, while DQN records 17.30% AP and 20.21% mean IoU.These values are reported alongside the latency comparison among DQN, DDPG, A2C, and GCN-assisted A2C.

C. State-of-the-art (SOTA) comparisons · D. Evaluation Results · VI. CONCLUSION

GCN-assisted A2C achieves lower transmission latency while maintaining server-side detection accuracy across SOTA comparisons and varied UAV conditions. Its graph-based pixel grouping supports A2C decisions and optimization toward AP above 50.

  • C. State-of-the-art (SOTA) comparisons: 50 ms transmission latency with a 12 ms standard deviation made GCN-assisted A2C the best-performing model versus FlexPatch and other SOTA methods.FlexPatch ranked second with 90 ms latency and a 15 ms standard deviation.
  • C. State-of-the-art (SOTA) comparisons: GCN-assisted A2C outperformed FlexPatch and EdgeDuet across transmission delay, AP, and meanIoU comparisons at different video resolutions.The evaluation examined suspicious-UAV detection accuracy, meanIoU, and end-to-end latency.
  • C. State-of-the-art (SOTA) comparisons: FlexPatch achieved 39.52% meanIoU and 110.25ms minimum latency, while its high-resolution tile offloading was unsuitable for real-time or low-bandwidth scenarios.GCN-assisted A2C instead uses connected groups of pixel areas to handle rapidly changing or unpredictable scenes.
  • C. State-of-the-art (SOTA) comparisons: GCN-assisted A2C maintained higher server-side AP and meanIoU across different frame sizes, with accuracy dropping to 50 AP as resolution decreased.The 50 AP level was the desired minimum server-side threshold.
  • D. Evaluation Results: Across three UAV sizes and cloudy, complex, and clear-sky conditions, the evaluation showed the expected trade-off between AP and transmission latency.Blue bars represented latency and olive bars represented AP.
  • D. Evaluation Results: GCN-assisted A2C consistently outperformed basic models in every tested situation, achieving higher AP with less delay, especially under cloudy and complex conditions.These results motivate smarter feature-correlated region prediction algorithms.
  • VI. CONCLUSION: The proposed model uses nano YOLO detections centered on suspicious objects and surrounding pixels, then groups related pixels through GCN to improve A2C decisions.The design targets no-fly-zone detection when onboard UAV detection accuracy is below 50.
  • VI. CONCLUSION: Using authentic onboard UAV data and DRL and SOTA comparisons, GCN-assisted A2C improved transmission latency and AP across clear skies, complex backgrounds, and partial clouds.The model optimizes until server-side AP exceeds 50.
Loading 2608.16142v1…