PaperlayerView exact revision on arXiv

Source-linked AI summary

Deadline-Aware Multi-Agent Reinforcement Learning for TSN-Based Vehicular Edge Networks

Bernardo A. C. Pereira, Marcos Carvalho, Fatih Temiz, Shavbo Salehi, Melike Erol-Kantarci, Andreas Gavrielides, Johann M. Marquez-Barja, Daniel F. Macedo

arXiv:2610.09870v1cs.NIcs.AI

TL;DR

Dynamic vehicular deadlines and dependencies among co-located TSN queues challenge existing scheduling approaches. The paper assigns an autonomous MAPPO agent to each queue to learn service order and time-slot duration. Against centralized single-agent methods, MAPPO reduces service latency by up to 66.2% and improves reliability by up to 271.8%.

  • Problem

    Existing TSN schedulers struggle with dynamic vehicular environments and inter-queue dependencies, while centralized RL policies can overlook traffic sources and changing deadlines.

    Source: §I · p. 1

  • Method

    The approach assigns each TSN queue an autonomous agent that uses MAPPO to learn service order and time-slot duration under speed-dependent deadlines.

    Source: §I · p. 1, A. XR Traffic, Latency and Dynamic… · p. 3

  • Results

    MAPPO reduces service latency by up to 66.2% and improves reliability by up to 271.8% compared with centralized single-agent methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    In the evaluated setting, MAPPO reduces latency, improves PDU-set reliability, and maintains low inference time, which the authors describe as suitable for real-time scheduling.

    Source: §VI · p. 6

    The impact of wireless networks on the proposed framework remains for future evaluation.

    Source: §VI · p. 6

Abstract

from arXiv · show

Vehicular edge computing (VEC) enables latency-sensitive applications by bringing computing and networking resources closer to vehicles. However, existing approaches often overlook network contention among co-located services with heterogeneous and dynamic latency requirements. While time-sensitive networking (TSN) provides bounded-latency communication, conventional and reinforcement learning-based schedulers struggle to adapt to highly dynamic vehicular environments and inter-queue dependencies. To address these limitations, we propose a multi-agent reinforcement learning (MARL) approach for queue-level scheduling in TSN-enabled VEC. Each TSN queue is assigned an autonomous agent that jointly learns the queue service order and time-slot duration to minimize deadline misses under speed-dependent latency requirements. We employ multi-agent proximal policy optimization (MAPPO) to enable coordinated yet autonomous scheduling decisions. Evaluation against single-agent, multi-agent, and non-learning-based baselines shows that MAPPO provides robust performance across different traffic profiles. Compared with centralized single-agent methods, it reduces service latency by up to 66.2% and improves reliability by up to 271.8%. Furthermore, unlike urgency-based heuristics, MAPPO ensures balanced scheduling while achieving lower inference times compared to other MARL methods.

I. INTRODUCTION

This paper proposes autonomous queue-level MARL scheduling to address dynamic, interdependent traffic deadlines in vehicular edge networks. MAPPO outperforms the considered approaches on latency and reliability while using less inference time than other MARL methods.

I. INTRODUCTION

  • TSN offers bounded-latency, high-reliability communication, but managing mixed deadline-critical traffic in mobile edge environments remains challenging.The paper argues that critical-service latency budgets at the edge should tighten as vehicle speed increases.Source: §I · p. 1
  • Existing RL schedulers often use centralized policies that overlook traffic sources and may miss inter-queue dependencies when deadlines vary dynamically.Optimization-based TSN schedulers also face challenges adapting rapidly to dynamic environments.Source: §I · p. 1
  • Each TSN queue receives an autonomous agent that jointly learns queue service order and time-slot duration to minimize deadline misses.MAPPO uses centralized training with decentralized execution.Source: §I · p. 1
  • MAPPO reduces service latency by up to 66.2% and improves reliability by up to 271.8% versus centralized single-agent methods, with lower inference times than other MARL methods.The evaluation also compares against non-learning-based baselines.Source: §I · p. 2

II. RELATED WORKS

  • Prior TSN work includes PPO scheduling, multi-queue slot adaptation, and routing-and-timing or routing-and-scheduling agents, while this work targets host-level queue coordination.The related-work discussion distinguishes its focus from prior network-level coordination.Source: §II · p. 2

III. SYSTEM MODEL AND PROBLEM FORMULATION

The system models downstream XR traffic from vehicles through RSU-hosted MEC servers, with each application assigned a TSN queue. It evaluates queueing and PDU-set completion against speed-adjusted application deadlines.

III. SYSTEM MODEL AND PROBLEM FORMULATION

  • RSUs co-located with MEC servers serve highway vehicles; traditional AR and semantic AR applications use dedicated TSN queues.SeAR sends compact semantic masks rather than full video content, reducing transmitted data.Source: §III · p. 2

A. XR Traffic, Latency and Dynamic Deadline Model

  • XR traffic is modeled as PDU-sets: groups of closely related PDUs carrying one application-level payload, indexed by vehicle and set.The traffic model is adopted from prior XR work and incorporates the PDU-set concept.Source: A. XR Traffic, Latency and Dynamic… · p. 2
  • PDUs enter their TSN queue by arrival time and are served first-in, first-out during each transmission opportunity.Source: A. XR Traffic, Latency and Dynamic… · p. 2
  • The age of the oldest PDU measures the waiting time of the oldest packet currently in a queue, providing a queue-level waiting-time quantity.The queue's waiting PDUs define the set over which the oldest packet is identified.Source: A. XR Traffic, Latency and Dynamic… · pp. 2–3
  • A PDU-set's total latency runs from arrival of its first PDU until completion of its last, because delaying one PDU can prevent frame or semantic-mask reconstruction.The model requires total PDU-set latency not to exceed its dynamic deadline.Source: A. XR Traffic, Latency and Dynamic… · p. 3
  • Each application's dynamic deadline scales its base deadline by a speed-dependent factor; the base is 11.11 ms for a 90 Hz AR example and 2 ms for SeAR.The deadline factor adjusts with vehicle speed, which changes the time available for transmission within RSU coverage.Source: A. XR Traffic, Latency and Dynamic… · p. 3

B. TSN Schedule Model

Each fixed-duration scheduling cycle is partitioned into queue-specific time slots, and the schedule specifies both slot durations and service order.

B. TSN Schedule Model

  • Each 1 ms scheduling cycle is partitioned into queue-specific time slots whose durations sum to the cycle duration.Source: B. TSN Schedule Model · p. 3
  • Queues are ordered by descending continuous priority score; ties give precedence to the queue with the larger index.The resulting order is represented as a permutation of the queue indices.Source: B. TSN Schedule Model · p. 3
  • The schedule is an ordered sequence of queue-and-duration pairs, combining the selected service order with each queue's allocated time slice.Source: B. TSN Schedule Model · p. 3

C. Problem Formulation

The scheduling problem maximizes on-time PDU-sets across co-located mixed-criticality XR applications, with time-slot durations and queue priorities as decision variables.

C. Problem Formulation

  • The scheduler maximizes the aggregate number of on-time PDU-sets across co-located, mixed-criticality XR applications.Source: C. Problem Formulation · p. 3
  • The decision variables are time-slot durations and queue priority scores.Source: C. Problem Formulation · p. 3
  • Allocated slots must fill the predefined cycle, application deadlines cannot exceed their base deadlines, and SeAR delay requirements are tighter than traditional AR requirements.Source: C. Problem Formulation · p. 3

D. MDP Formulation

The TSN scheduling problem is formulated as a multi-agent partially observable Markov decision process, with one agent controlling each queue. Agents use local queue observations to choose scheduling actions and receive a team reward based on the bottleneck queue.

D. MDP Formulation

  • Each queue is controlled by one agent in a multi-agent partially observable Markov decision process.Source: D. MDP Formulation · p. 3
  • Each agent observes local queue state, including backlog, missed PDU-sets, AOPDU waiting time, and urgency rate.Source: D. MDP Formulation · pp. 3–4
  • Each agent chooses a two-dimensional action that determines its time-slice allocation and service order within the scheduling cycle.Source: D. MDP Formulation · p. 4
  • The reward uses a dense urgency- and deadline-violation-based surrogate, with the global team reward set by the queue with the maximum penalty.Source: D. MDP Formulation · p. 4

A. MAPPO Framework

The MAPPO framework uses centralized training with decentralized execution: local actors schedule individual queues, while a centralized critic evaluates global system information.

A. MAPPO Framework

  • MAPPO assigns each agent a local actor and uses a centralized critic during training under a centralized-training, decentralized-execution paradigm.Source: A. MAPPO Framework · p. 4
  • The critic accesses global observations and actions to capture inter-queue dependencies and coupling, while each actor receives only its own local observation.Source: A. MAPPO Framework · p. 4
  • The centralized critic provides value estimates for policy gradients and advantages, reducing non-stationarity and variance as agents adapt their policies.Source: A. MAPPO Framework · p. 4

B. Training and Inference Procedure

During training, agents collect transitions from local observations and joint actions, then update actors and a centralized critic. At inference, each queue selects actions independently using its trained actor.

B. Training and Inference Procedure

A. Experimental Setup

Experiments use Smart Highway mobility data to model an eight-queue TSN network serving five XR applications. The setup varies speed-dependent deadlines and compares MAPPO with multi-agent, centralized single-agent, and urgency-based scheduling baselines.

A. Experimental Setup

  • Smart Highway vehicle counts and average speeds drive an eight-queue TSN network supporting five XR applications, with traffic modeled using Johnson’s SU distribution.Queues q0, q1, q3, q4, and q7 carry traditional AR traffic; the others support SeAR resolutions.Source: A. Experimental Setup · p. 4
  • Deadlines are scaled by α ∈[0.50, 0.90] for speed-dependent latency budgets, and SeAR traffic uses a 2 ms base deadline.The passage states the MEC delay budget is reduced from 90% at low speeds, but its final speed value is truncated.Source: A. Experimental Setup · p. 4
  • Each 150-step episode samples vehicle count and average speed from the dataset, clips them to 1–100 and 10–120, and distributes clients uniformly across queues with at least one user per queue.The reward penalty weight is λ = 1.0.Source: A. Experimental Setup · p. 4
  • Results are reported as the mean and standard error across three random seeds, and Table I reports MAPPO hyperparameters.Table I is identified as listing VEC and MAPPO parameters.Source: A. Experimental Setup · p. 4, Table I · p. 5
  • Baselines include queue-level MAT and HAPPO, centralized PPO and A2C, and a non-learning urgency heuristic that prioritizes queues approaching deadlines.The heuristic also specifies a time-slot allocation rule.Source: A. Experimental Setup · pp. 4–5

B. Experimental Results

MARL methods reduce service latency and improve deadline satisfaction across traffic classes, while MAPPO combines competitive execution performance with lower inference latency than other MARL methods.

Learning Performance

  • MARL methods converge faster than single-agent methods, but PPO achieves the highest and most stable final reward.Higher training rewards do not necessarily imply better network performance under heterogeneous queue deadlines.Source: Figure 2 · p. 5, B. Experimental Results · p. 5

Service Latency Analysis

  • For SeAR queues, MARL cuts average latency by up to 66.2% and p99 latency by up to 68.3% versus single-agent methods.Across AR queues, average PDU latency falls by at least 19.5% versus A2C and 14.9% versus PPO; p99 reductions reach 38.6% versus single-agent methods.Source: Figure 3 · p. 5, B. Experimental Results · p. 5
  • Urgency heuristics reduce SeAR latency by at least 29.1% but increase AR average latency by up to 315% and leave AR reliability around 30%.The AR worst-case latency reaches 80 ms, a 373% increase compared with learning-based methods.Source: B. Experimental Results · pp. 5–6

PDU-Set Reliability

  • MARL deadline satisfaction exceeds 70% across AR queues versus approximately 60% for single-agent methods, a 16.66% improvement.Urgency-based scheduling exceeds 90% reliability for SeAR queues, particularly q2 and q5, but achieves only around 30% across AR queues.Source: B. Experimental Results · pp. 5–6, Figure 4 · p. 6

Inference Time

  • MAPPO has comparable performance with significantly lower inference latency, while HAPPO and MAT exceed the scheduling cycle by more than 5 and 10 times, respectively.MAPPO and single-agent methods generate actions in one forward pass; MAT generates them autoregressively, and HAPPO uses a separate actor network per agent.Source: Table II · p. 6, B. Experimental Results · p. 6

VI. CONCLUSION

The paper applies MAPPO to co-located XR traffic scheduling in VEC-TSN with vehicle speed-dependent latency deadlines. Simulations show improved latency and PDU-set reliability over the considered baselines, with low inference time; wireless-network impacts remain future work.

VI. CONCLUSION

  • MAPPO outperforms the considered baselines in simulation, reducing latency, improving PDU-set reliability, and maintaining low inference time.The authors describe the approach as suitable for realtime scheduling.Source: §VI · p. 6
  • The approach uses a centralized critic to learn cooperative queue-level scheduling policies under dynamic, vehicle speed-dependent deadlines.The paper considers co-located XR traffic in VEC-TSN and models queue-level interdependencies.Source: §VI · p. 6
  • Future work will evaluate how wireless networks affect the proposed framework.Source: §VI · p. 6
Loading 2610.09870v1…