Source-linked AI summary
Mission-critical spectrum sharing with decentralized Multi-Agent Reinforcement Learning
Dimitrios Pylorof, Imtiaz Nasim, Humberto E. Garcia, Vivek Agarwal, Jasni A. Mannil, Mingyue Ji
TL;DR
Mission-critical spectrum sharing requires decentralized coordination when long-horizon forecasts are imperfect and secondary users may contend for the same opportunities. The paper develops lightweight decentralized MARL with waiting and fixed-power band selection, reporting up to 96.8% fewer PU-SU collisions than a forecast-aware heuristic baseline.
Problem
The paper identifies a gap in lightweight decentralized actor-critic learning that operates under imperfect long-horizon forecasts while supporting incumbent protection.
Source: §I · p. 1Method
The paper uses decentralized MARL with idle or fixed-power band-selection actions, making waiting part of agents’ access strategies.
Source: §I · p. 1Results
Up to 96.8% fewer PU-SU collisions than the forecast-aware heuristic baseline.
Source: §I · p. 1Takeaways & Limitations
Learned policies occupy low-conflict operating regions that forecast-aware access alone does not meet, while reward weights shift tradeoffs among successful transmissions and PU and SU conflicts.
Source: §VIII · p. 7, §VI · p. 7This study uses idle or fixed-power band choices; distributed joint spectrum awareness, active sensing, selectable transmit power, and explicit conflict budgets remain future scope.
Source: §VI · p. 7
Abstract
from arXiv · showhide
Motivated by emerging mission-critical applications and an increasingly congested spectrum, we develop a decentralized multi-agent reinforcement learning (MARL) model for dynamic spectrum access. The model enables secondary users to learn effective transmission strategies across shared frequency bands while minimizing collisions with high-priority primary users and among themselves. We design the agent-level learners following a Markov potential game approach, connecting independent local updates to system-level improvement. We instantiate this design using lightweight linear actor-critic learners suitable for resource-constrained edge devices, rather than computationally intensive centralized or deep multi-agent architectures. Across spectrum environments with different incumbent activities, the learned policies adapt their transmission policy and waiting behavior to preserve throughput while greatly reducing transmission collisions relative to random and forecast-aware heuristic baselines. The results establish the value of decentralized MARL and shows up to 96.8% reduction in overall collisions.
I. INTRODUCTION
The paper frames mission-critical spectrum sharing as a coordination problem: imperfect forecasts can leave secondary users contending for the same opportunities. It develops decentralized learning that makes waiting part of access strategy and evaluates collision reduction against benchmarks.
I. INTRODUCTION
- The paper targets lightweight decentralized actor-critic learning under imperfect long-horizon forecasts, with methods readable for incumbent protection.The operational behavior includes deciding when not to transmit, which can protect incumbents and reduce contention among secondary users.Source: §I · p. 1
- Agents choose between idling and fixed-power transmission on an available band, isolating coordination learned through waiting and band selection.The study asks whether this decentralized MARL setup can reduce collisions before enlarging the action space.Source: §I · p. 1
- Up to 96.8% fewer PU-SU collisions than the forecast-aware heuristic baseline.The paper also reports conservative operating points that maintain non-trivial service.Source: §I · p. 1
- Forecasting can leave secondary users synchronized on the same apparently favorable band, so spectrum awareness alone does not provide coordination.Incumbent activity is bursty, and forecasts may be inaccurate at the instant of transmission.Source: §I · p. 1
II. BACKGROUND AND DESIGN POSITION
The background positions independent learning within Markov potential games and uses an imperfect forecast as a look-ahead sensor. The environment then tests distributed temporal access strategies amid incumbent activity and peer contention.
A. Distributed learning and potential-game structure
- Markov potential games align individual decisions with a common system objective, extending the potential-game idea to long-term rewards and independent policy updates.The implementation follows these principles with independent local actors and critics, common rewards, and queue transitions driven by local actions and exogenous arrivals.Source: A. Distributed learning and… · p. 2
B. Forecasting as a look-ahead sensor
- GAD-Autoformer forecasts serve as an external sensor and heuristic baseline, with empirical residuals perturbing incumbent reference traces to create distinct local forecasts.This setup evaluates temporal access strategies under realistic but imperfect spectrum forecasts.Source: B. Forecasting as a look-ahead sensor · p. 2
III. SPECTRUM SHARING ENVIRONMENT
- The environment models queued secondary-user packets sharing incumbent-prioritized bands, where incumbent activity and simultaneous secondary-user choices govern distinct collision risks.Noisy look-ahead forecasts inform each user's choice to wait or transmit; path loss and SINR determine successful transmissions or peer collision.Source: §III · p. 2
A. Incumbent and RSS process
The environment varies incumbent activity through a stochastic RSS process, supplies forecast-error surrogates as local spectrum views, and models idle or fixed-power band actions with queued packets and distinct collision outcomes.
A. Incumbent and RSS process
- The activity knob κ shifts the semi-Markov ON/OFF process from quieter to busier regimes; realized occupancy emerges stochastically rather than being prescribed.The latent process maps to continuous RSS levels, producing gradual and noisy changes rather than binary labels.Source: A. Incumbent and RSS process · p. 2
B. Forecast-error surrogation
- Forecast-error surrogation perturbs hidden future RSS with lead-specific empirical residuals, giving agents different local views of the same evolving spectrum.Lead-indexed residual profiles show similar broad quiet/busy regimes and recurring burst-transition errors; Fig. 1 presents a representative profile used to calibrate the surrogate.Source: B. Forecast-error surrogation · p. 2, Figure 1 · p. 3
C. Actions, queues, and wireless outcomes
- Each secondary user waits or transmits at fixed power on one band; incumbent-active attempts are PU conflicts, while weaker same-band links can incur peer collisions under the path-loss/SINR model.Each attempt removes one queued packet, and the reward records the realized transmission outcome separately.Source: C. Actions, queues, and wireless… · pp. 2–3
IV. DECENTRALIZED POLICY GRADIENT FORMULATION
Each secondary user turns its queue and imperfect spectrum forecasts into a fixed local feature vector, then learns action preferences from its own observed rewards. Forecast-derived features represent busy risk, uncertainty, and later opportunities, leaving the learner to weigh waiting against carrying the queue longer.
IV. DECENTRALIZED POLICY GRADIENT FORMULATION
- Each node compresses its queue, time, and imperfect spectrum forecast into a fixed feature vector, scores local actions, and adjusts preferences from its own rewards.The loop keeps forecast information, mission priorities, and learned behavior visible.Source: §IV · p. 3
A. From forecasts to a value-of-waiting signal
- Each agent observes its queue and per-band forecasted RSS and busy probability, then converts forecasted RSS into a smooth busy-risk score.The forecast spans a look-ahead horizon of H steps; the risk mapping uses environment-matched center and scale parameters.Source: A. From forecasts to a… · p. 3
- The risk-adjusted forecast adds a weighted uncertainty feature to the busy-risk score, with β denoting the risk weight.The uncertainty term conservatively adjusts for known or assumed forecast uncertainty.Source: A. From forecasts to a… · p. 3
- The policy feature map summarizes current per-band risk, later opportunities, current and future best bands, and interactions with the queue.It includes best and average opportunities later in the forecast.Source: A. From forecasts to a… · p. 3
- A forecast-derived signal identifies when a safer opportunity is expected later than the best band available now.This is information supplied to the learner, not a guarantee that waiting is beneficial.Source: A. From forecasts to a… · p. 3
- Waiting is beneficial only when the expected interference reduction is worth carrying the queue longer, a tradeoff evaluated through independent policy-gradient iterations.Source: A. From forecasts to a… · p. 3
B. Reward weights and mission postures
The shared reward weights encode mission priorities by valuing successful delivery and penalizing primary-user conflicts most heavily. Separate terms distinguish peer collisions, link failures, attempts, and idling, while alternative coefficient sets represent different operating postures.
B. Reward weights and mission postures
- Successful delivery receives positive value, while primary-user conflicts receive the largest penalty in the shared reward.Source: B. Reward weights and mission postures · p. 3
- The reward distinguishes successful delivery, incumbent-active-band attempts, same-band secondary-user collisions, link failures, transmission attempts, and idle decisions.Each transmission attempt incurs an attempt term in addition to exactly one realized transmission outcome.Source: B. Reward weights and mission postures · p. 3
- All agents use the same coefficient vector, whose alternatives emphasize incumbent protection, queue relief, or service more strongly without changing the learning algorithm.Source: B. Reward weights and mission postures · p. 3
C. From action scores to independent policy updates
The method uses local linear actor–critic updates: actors score feasible actions from local features, while critics assess local experience without centralized values or parameter exchange. Training batches, frozen-policy evaluation, and compact parameter storage define the implementation, which follows an MPG-aligned theoretical framework.
C. From action scores to independent policy updates
- The study restricts agents to idling or fixed-20-dBm band selection, while Fig. 2 marks adaptive power and sensing as follow-on additions.Source: Figure 2 · p. 4
- Each feasible action receives a linear feature-based score, converted to action probabilities by a softmax; actor coefficients start from an independent zero-mean Gaussian.Source: C. From action scores to independent… · p. 4
- The critic fits a ridge-regularized least-squares Bellman-residual model on on-policy batches and supplies a local expected discounted return.Source: C. From action scores to independent… · p. 4
- Transitions are compared with the critic expectation through clipped TD residuals, and the resulting independent policy-gradient update adjusts the actor.Positive residuals reinforce the sampled action; negative residuals suppress it.Source: C. From action scores to independent… · p. 4
- The critic judges only recent local experience; the design has no centralized value function or exchange of learned parameters.Source: C. From action scores to independent… · p. 4
- Agents update after each batch of M transitions, process any residual training buffer once, freeze policies, then deploy and report the final Ttail steps.Algorithm 1 outlines this ordering from local feature construction and rewards through training and frozen-policy evaluation.Source: C. From action scores to independent… · pp. 4–5
- Each node stores dϕ|Ai| actor coefficients and dϕ critic coefficients; the compact construction is described as compatible with CPU-class edge execution.Source: C. From action scores to independent… · pp. 4–5
- The independent policy-gradient construction follows the MPG direction of, whose approximate ϵ-Nash convergence result has stated additional conditions for linking equilibria to potential maximizers.The authors describe the MPG alignment and related results as the architecture’s proof envelope.Source: D. ϵ-Nash convergence · p. 5
E. Implementation settings
The numerical setup uses six agents, four bands, fixed 20-dBm transmission, and stochastic packet arrivals. Forecast features span two hours, and training uses batch updates followed by frozen-policy evaluation over a seeded trace.
E. Implementation settings
- The instantiation uses N = 6 agents, B = 4 bands, ∆t = 5 min, PTX = 20 dBm, and |Ai| = 5 actions.Queues start at 8 packets, have capacity 20, and receive Bernoulli arrivals with probability 0.20 per decision step.Source: E. Implementation settings · p. 5
- Forecast features use H = 24 steps spanning two hours, with τRSS = −82 dBm, sRSS = 4.5 dB, β = 0.75, and dϕ = 82.Each node stores 410 actor coefficients and 82 critic coefficients.Source: E. Implementation settings · p. 5
- Training runs for 3000 environment steps with twelve updates; frozen-policy evaluation runs for 1000 steps and reports the final 250.Reported event counts are scaled to a per-1000-step basis.Source: E. Implementation settings · p. 5
A. Protocol and baselines
Across the five incumbent-activity settings, selected MARL operating points have lower conflict counts than the heuristic and random baselines. The protocol selects service-feasible MARL configurations, and changing reward weights shifts the learned balance between access and deferral.
A. Protocol and baselines
- Each activity setting uses a 14-day trace sampled every five minutes, yielding 4,032 timestamps.The settings are κ ∈ {0.05, 0.10, 0.15, 0.20, 0.25}.Source: A. Protocol and baselines · p. 5
- Agents train for 3,000 environment steps and are evaluated after a 1,000-step deployment over five seeds, with reported metrics averaged over the final 250 steps.The evaluation compares a selected MARL point with random access and a forecast-aware heuristic; a post-training score selects the MARL point subject to at least 100 successful transmissions per 1,000 steps.Source: A. Protocol and baselines · p. 5
B. How reward weightings move the learned operating point
- In Fig. 3, selected MARL operating points and their trial distributions lie consistently lower in conflicts than heuristic and random baseline points.The figure shows successful SU transmissions against PU conflicts and total PU+SU conflicts; bold markers are five-seed means.Source: Figure 3 · p. 6
- As incumbent activity rises, selected reward profiles shift from selective access toward stronger deferral, with balanced service reopening access at κ = .25.At κ = .25, the balanced profile yields 196.8 successes and 70.4 total conflicts, while strict, base, and queue-aware protection reach zero service.Source: B. How reward weightings move the… · pp. 5–6
C. Throughput–conflict operating regions
The learned operating points reduce collisions relative to the forecast-aware heuristic, while trading throughput and queue buildup against incumbent protection and peer coordination. Policies often wait when transmission risk is high, but a minimum-service requirement excludes solutions that achieve low conflicts simply by remaining idle.
C. Throughput–conflict operating regions
- 54.4–96.8% total-conflict reductions and 16.9–46.9% PU-conflict reductions are reported for MARL versus the forecast-aware heuristic.At κ = 0.20, MARL records 100.8 successful transmissions and lowers PU conflicts from 17.6 to 12.8.Source: C. Throughput–conflict operating regions · p. 6
- The heuristic's peer-conflict burden remains 328.8–381.6 per 1,000 steps, while MARL uses the full forecast horizon to weigh queue risk against interference.The proposed policy also anticipates that several agents may choose the same apparently safe band.Source: C. Throughput–conflict operating regions · p. 6
D. What the policy learns about waiting
- Across observations with nonempty queues in five activity settings, selected MARL policies wait in 84.3–98.1%, reduce high-risk transmissions to 1.9–15.7%, and wait for forecast improvement in 35.6–40.4% of queued states.The table reports ranges across selected operating points.Source: D. What the policy learns about waiting · p. 6, Table III · p. 6
- Waiting carries a backlog cost, and the minimum-service requirement excludes operating points that achieve low conflicts simply by remaining idle.Mean queue lengths are 10.8–18.7 packets for MARL, compared with 10.7 for the heuristic and about 0.38 for random access.Source: D. What the policy learns about waiting · pp. 6–7
VI. DISCUSSION AND ROADMAP
The discussion and conclusion frame forecast-aware decentralized MARL as a coordination layer that learns low-conflict operating regions through access, band choice, and waiting. They position joint awareness, richer actions, and field validation as next steps.
VI. DISCUSSION AND ROADMAP
- Agents learn when to access, which band to use, or when to defer, while reward weights let operators shift trade-offs among successful transmissions and PU/SU conflicts.Source: §VI · p. 7
- Exchanging compact summaries and using distributed average consensus to build joint spectrum awareness are future directions, unlike the work presented here.Source: §VI · p. 7
- Adding sensing or selectable transmit power introduces energy considerations, while replacing soft PU reward weights with explicit conflict budgets would require analysis beyond current guarantees.Source: §VI · p. 7
VII. APPLICATIONS OF PROPOSED DECENTRALIZED SPECTRUM COORDINATION FOR RESPONSE OPERATIONS
- Response networks with distributed assets and unreliable or infeasible central control align with local decisions, CPU-class learners, and forecast-aware waiting that trades urgency against incumbent protection and peer contention.Source: §VII · p. 7
- Operational deployment requires representative RF testing and network-level demonstrations to determine whether observed safety-service regions persist under realistic topologies, interference, and communication challenges.Source: §VII · p. 7
VIII. CONCLUSION
- Forecast-aware decentralized MARL learns low-conflict operating regions that forecast-aware access alone does not reach, jointly considering incumbent risk, peer contention, and future opportunities.Source: §VIII · p. 7