PaperlayerView exact revision on arXiv

Source-linked AI summary

Cooperative Dueling DQN SAC Learning for Energy Efficiency in Dynamic OWC Networks

Walter Zibusiso Ncube, Ahmad Adnan Qidan, Taisir El-Gorashi, Jaafar M. H. Elmirghani

arXiv:2610.09266v1eess.SYcs.SY

TL;DR

Dynamic OWC allocation must coordinate AP association, subcarrier counts, and power to balance energy efficiency and service requirements over time. DARA-DRL assigns discrete allocation and continuous power control to specialized agents that learn cooperatively. It remains close to the optimal solution in the reported comparison, and the paper concludes that specialization and cooperation are more effective than a single agent or independent learning.

  • Problem

    Dynamic OWC resource allocation couples discrete AP and subcarrier choices with continuous power control, service-dependent QoS, and costs from allocation changes.

    Source: §I · p. 2

  • Method

    DARA-DRL uses a branching duelling DQN for discrete allocation and a conditional SAC agent for power control, with cooperative value updates evaluating the complete allocation.

    Source: §I · pp. 2–3, C. Cooperative Learning Between the… · p. 9

  • Results

    At 12 users, DARA-DRL’s optimality gap is approximately 5.6%, compared with 26.7% for independent DQN–SAC, 37.6% for HAPPO, 41.8% for PDP-OPA, 48.1% for P-DQN, and 60.6% for reactive greedy.

    Source: B. Results · p. 11

  • Takeaways & Limitations

    The paper concludes that specialized agents handle different action types more effectively and that cooperative training matters because discrete decisions’ value depends on subsequent power allocation.

    Source: §VI · p. 12

Abstract

from arXiv · show

Growing wireless traffic is increasing pressure on the congested radio-frequency spectrum. Optical wireless communication (OWC) provides a complementary solution by using the abundant unlicensed optical spectrum. However, indoor OWC networks are dynamic: users move, enter or leave the network, and subscribe to different services. Poorly coordinated resource allocation can consequently waste subcarriers, require excessive transmission power and cause frequent AP reassignments. Energy efficiency (EE), defined as the total delivered data rate divided by the total network power consumption, therefore requires the serving AP, number of allocated subcarriers, and transmission power to be jointly adapted while maintaining QoS. Optimising these in a dynamic time series, multi-service OWC environment produces a complex sequential EE optimisation problem. To address this problem, this work proposes Dual-Agent Resource Allocation using Deep Reinforcement Learning (DARA-DRL). DARA-DRL combines a branching duelling deep Q-network for association and subcarrier allocation with a conditional soft actor-critic agent for continuous power control. The agents are coupled through a common reward and a cooperative value update that evaluates each discrete allocation together with its corresponding power decision. Simulation results show that DARA-DRL remains within 5\% of the optimal solution and, compared with state-of-the-art benchmarks, it improves EE by 28.8\% and QoS satisfaction by 10.5\%, while reducing online decision time by 14.5\%. Results demonstrate that agent specialisation simplifies mixed-action learning, and cooperation outperforms independently trained agents.

I. INTRODUCTION

Dynamic, multi-service OWC makes energy-efficient allocation a sequential problem: AP association, subcarrier counts, and transmit power must be coordinated across changing channels, loads, and service requirements. DARA-DRL addresses the mixed discrete-continuous decisions with specialized agents and cooperative learning.

Motivation

  • Dynamic users, changing service requirements, and the costs of reallocating resources make energy-efficient OWC allocation a sequential problem balancing EE, QoS, and allocation stability.EE is delivered data rate divided by total network power; an allocation suitable in one interval may require more power or become insufficient in the next.Source: §I · pp. 1–2
  • AP association, subcarrier allocation, and transmit power are coupled through AP capacity, power needs, processing consumption, and inter-AP interference.A strong optical link may have little remaining AP capacity, while a less-loaded AP may need more power to serve the same user.Source: §I · p. 1

Research gap

  • Prior OWC resource-allocation studies address learning or coordinated resource decisions, but the stated problem combines service-dependent QoS, discrete choices, continuous power, and allocation-change costs.Existing approaches include Q-learning, deep Q-learning, cross Q-learning, and prediction-assisted RL; joint optimization for one configuration does not by itself account for successive allocation changes.Source: §I · p. 2

Proposed approach

  • DARA-DRL pairs a branching duelling DQN for AP association and subcarrier counts with a conditional SAC agent for continuous power control.The agents share a system reward and replay transitions during centralized training; the power agent conditions on the discrete allocation.Source: §I · pp. 2–3
  • A cooperative value update uses the power agent’s critics to evaluate the complete allocation, so the discrete agent learns from the consequences of its choice for subsequent power allocation.The shared reward also penalizes minimum-rate shortfalls, and feasibility processing enforces association, subcarrier, and AP power limits.Source: §I · pp. 2–3

Evaluation design

  • The evaluation compares DARA-DRL with six benchmarks and uses modified versions to assess agent specialization and cooperative updates.The comparisons include convergence, optimality gap, EE, minimum-rate satisfaction, AP reassignment, and online decision time.Source: §I · p. 3

II. SYSTEM MODEL

The system model describes a dynamic indoor VCSEL-based OWC network coordinated by a central unit. Its downlink allocation represents subcarrier counts and user-level power while tracking changes between slots.

A. Room and Network Setup

  • The modeled room uses ceiling-mounted optical APs with VCSEL arrays, connected to a central unit that gathers network information and coordinates resource allocation.A Wi-Fi AP provides uplink channel feedback, user requests, and control signalling.Source: A. Room and Network Setup · p. 3, Figure 1 · p. 3
  • Users may enter, leave, or move between AP coverage regions across equal-duration slots, and subscribe to service classes with distinct minimum-rate requirements.The model treats position as constant within a slot and updates it between consecutive slots.Source: A. Room and Network Setup · p. 3

B. Downlink System Model

  • The downlink models DCO-OFDM subcarrier assignment by integer counts per AP-user link, with a deterministic mapper assigning subcarriers while avoiding reuse within each AP.Different APs may reuse subcarriers, causing inter-AP interference; power is a continuous user-level decision divided equally among that user’s assigned subcarriers.Source: B. Downlink System Model · pp. 3–4
  • The model tracks AP reassignments and subcarrier-count changes between slots so the long-term objective can distinguish useful reconfiguration from short-lived changes with little EE benefit.The optical channel is assumed flat across data subcarriers for a given AP-user link within a slot, though gains can vary across users and over time.Source: B. Downlink System Model · p. 4

C. Optical Channel Model

The optical channel model represents each VCSEL output as a Gaussian beam and derives link gain from beam geometry and receiver characteristics.

C. Optical Channel Model

  • Each VCSEL’s optical intensity is modeled with a Gaussian beam, with its output power and post-microlens beam radius specified for channel calculation.Beam-radius calculation follows Gaussian-beam propagation and the ABCD transformation; VCSEL operating-region and eye-safety calculations determine the AP power limit.Source: C. Optical Channel Model · p. 4
  • The channel gain accounts for AP-to-user geometry and receiver orientation, including photodetector area, optical-filter and concentrator gains, and the receiver field of view.The direct channel gain is zero when the incident beam is outside the photodetector face’s field of view.Source: C. Optical Channel Model · p. 4

D. SINR and Achievable User Rate

The downlink model builds real-valued OFDM transmission signals, represents the received signal and interference, and uses an optical-channel lower bound to specify achievable user rate.

D. SINR and Achievable User Rate

  • Hermitian symmetry is imposed on the complex data symbols so the inverse fast Fourier transform produces a real waveform, which is DC-biased for intensity modulation and direct detection.The waveform is constructed from users’ data symbols, subcarrier assignments, and allocated powers.Source: D. SINR and Achievable User Rate · p. 4
  • The received-symbol model includes the effective desired and interfering AP channel gains, while receiver noise contains shot, thermal, and background-noise components.The corresponding electrical SINR is defined for the assigned subcarrier.Source: D. SINR and Achievable User Rate · pp. 4–5
  • User rate is calculated using a lower bound for an IM/DD optical channel, with the assigned subcarriers and bandwidth specified in the rate model.The bandwidth is denoted by B.Source: D. SINR and Achievable User Rate · p. 5

E. Network Power Consumption

The network power model includes fixed, circuit, transmission, and allocation-change costs. The optimization jointly selects AP association, subcarrier counts, and transmit power over time to balance EE and QoS amid a computationally difficult problem.

E. Network Power Consumption

  • Network power accounts for the CU, active APs, data transmission, and the overhead of changing power and resource allocations.The model also includes incremental baseband and driver power for active user-subcarrier assignments, plus AP-reassignment and subcarrier-count-change costs.Source: E. Network Power Consumption · p. 5

III. ENERGY-EFFICIENCY OPTIMISATION PROBLEM

  • The optimization jointly selects each active user’s serving AP, subcarrier count, and transmit power to pursue EE while meeting service QoS and avoiding changes whose short-term benefit does not justify their cost.The objective is evaluated over time using current measurements and previous allocations.Source: §III · p. 5

A. Problem Formulation

  • P1 is a mixed-integer non-convex fractional problem because association and subcarrier choices are discrete, power is continuous, and rate and power decisions are coupled.Even before power allocation, a slot with N users has many discrete combinations.Source: A. Problem Formulation · p. 5
  • Exhaustive enumeration can provide an optimal benchmark, but its worst-case complexity grows exponentially and it must be rerun whenever the network changes.Reactive greedy allocation is computationally inexpensive but does not account for movement or future demand; PDP-OPA does not jointly optimize subcarrier counts and uses iterative power updates.Source: A. Problem Formulation · pp. 5–6
  • A DRL policy can be trained offline and then make operational decisions through a forward pass, learning from the effects of allocations on later EE, QoS, and reassignments.This addresses the repeated-search burden of the described optimization approaches.Source: A. Problem Formulation · p. 6

B. Markov Decision Process Reformulation

The paper reformulates resource allocation as an MDP so an RL policy can act on current network conditions while accounting for future performance. Its reward favors EE when user rate requirements are met and penalizes rate shortfalls otherwise.

B. Markov Decision Process Reformulation

  • The P1 reformulation defines the network state, action, transition, and reward so an RL policy can choose allocations based on current conditions and their effects on future performance.The policy objective is expected discounted accumulated reward, with β controlling the influence of future rewards.Source: B. Markov Decision Process Reformulation · p. 6
  • The state includes active users, service classes, channel conditions, AP loads, and previous AP-association, subcarrier, and power decisions.Source: B. Markov Decision Process Reformulation · p. 6
  • The joint action combines discrete AP-association and subcarrier-count decisions with continuous transmit-power decisions.A feasible action must satisfy P1 constraints C1–C6.Source: B. Markov Decision Process Reformulation · p. 6
  • When all active users meet minimum rates, the reward is proportional to achieved EE; otherwise it is negative according to average relative rate shortfall, and it is zero when no users are active.Only users below their required rates contribute to the penalty.Source: B. Markov Decision Process Reformulation · p. 6
  • Feasible-action construction enforces association, subcarrier, and AP power constraints, but minimum rates are encouraged by the shortfall penalty rather than guaranteed by construction.Source: B. Markov Decision Process Reformulation · p. 6

IV. DUAL-AGENT RESOURCE ALLOCATION USING DEEP REINFORCEMENT LEARNING

DARA-DRL decomposes the mixed discrete-continuous allocation into discrete resource selection followed by conditional continuous power control. A cooperative value update lets the discrete agent evaluate its choices together with the resulting power allocation.

IV. DUAL-AGENT RESOURCE ALLOCATION USING DEEP REINFORCEMENT LEARNING

  • The approach addresses the mixed action space by matching DQN to discrete decisions and SAC to continuous power, avoiding power discretization or rounding discrete outputs.The cited discussion identifies these as drawbacks of applying either algorithm across all decisions.Source: §IV · p. 7
  • DARA-DRL uses a branching duelling DQN for AP association and subcarrier counts, followed by a conditional SAC agent for continuous transmit-power control.Both agents use the same transition, reward, and shared replay buffer.Source: §IV · p. 7
  • During the discrete-agent update, the power agent’s critics evaluate the complete discrete-continuous action, while the power agent learns conditioned on the selected discrete configuration.This coordinates each discrete allocation with its corresponding power decision.Source: §IV · p. 7

A. Discrete Resource Allocation Agent

The Discrete Resource Agent selects each active user’s AP and subcarrier count through user-specific branches, then converts those choices into a feasible joint allocation. A deterministic mapper assigns the corresponding subcarrier identities.

A. Discrete Resource Allocation Agent

  • The DRA uses a shared state encoder and one action branch per supported user to rank that user’s AP-subcarrier-count choices.Training uses ε-greedy exploration, while online execution selects branch values greedily.Source: A. Discrete Resource Allocation Agent · p. 7
  • A deterministic feasibility layer accepts the highest-valued feasible user-AP-count candidates while preserving AP capacity and enough subcarriers for unassigned active users.It continues until every active user has an AP and a positive subcarrier count.Source: A. Discrete Resource Allocation Agent · p. 7
  • The feasibility layer converts branch decisions into a joint discrete allocation satisfying constraints C1–C3, provided all APs are available as choices and active users do not exceed the stated limit.Source: A. Discrete Resource Allocation Agent · p. 7
  • The deterministic mapper assigns the requested number of unassigned subcarriers from the serving AP; a subcarrier may serve at most one user per AP, while different APs may reuse it.Source: A. Discrete Resource Allocation Agent · pp. 7–8

B. Continuous Power Agent

The conditional SAC power agent generates continuous transmit power after the discrete allocation is set. Its critics evaluate power in the context of the full allocation, while projection enforces AP power budgets and QoS is addressed through the shared reward.

B. Continuous Power Agent

  • Conditioned on the feasible discrete allocation, the CPA uses a SAC policy to generate requested continuous transmit powers.The policy does not alter the AP association or subcarrier-count decisions.Source: B. Continuous Power Agent · p. 8
  • The CPA’s requested powers are bounded by each AP’s maximum power, set to zero for inactive users, and proportionally scaled when their sum exceeds an AP budget.The resulting projection satisfies AP power constraints C5–C6; requests within budget are unchanged.Source: B. Continuous Power Agent · p. 8
  • Power projection does not enforce the minimum-rate QoS constraint; rate-shortfall penalties instead enter the common reward.QoS depends jointly on AP choice, subcarrier allocation, power, channel conditions, and inter-AP interference.Source: B. Continuous Power Agent · p. 8
  • Each CPA critic evaluates power together with the network state and feasible AP–subcarrier allocation, capturing the dependence of reward on both allocation stages.The minimum of the twin critic estimates is used in the learning target to reduce overestimation; entropy supports exploration and its temperature is adjusted toward target entropy.Source: B. Continuous Power Agent · p. 8

C. Cooperative Learning Between the Agents

The agents train from shared network interactions and a common reward. Cooperation is implemented by using the target CPA critics to evaluate the full next allocation, so the DRA’s value update reflects the subsequent power decision.

C. Cooperative Learning Between the Agents

  • Both agents train on the same network interaction and common reward, with transitions stored in a shared replay buffer for both agents to sample.The shared transitions pair the same network state and allocation outcome with the global reward.Source: C. Cooperative Learning Between the… · p. 8
  • The CPA uses twin-critic SAC updates, while the DRA’s discrete feasibility layer and subcarrier mapper enforce C1–C3 and power projection enforces C5–C6.QoS condition C4 is handled through the common reward because it depends on the joint allocation and network conditions.Source: C. Cooperative Learning Between the… · p. 9
  • The cooperative DRA target evaluates the next discrete allocation together with the CPA’s subsequent power allocation, rather than using branch values alone.This gives the DRA feedback about how its AP–subcarrier choice affects power allocation and long-term network reward.Source: C. Cooperative Learning Between the… · p. 9
  • The CPA critics evaluate power in the context of the complete discrete allocation, while the DRA learns to value each AP–subcarrier choice with its subsequent power decision.The cooperative value update uses the target CPA critics to estimate the complete next allocation.Source: C. Cooperative Learning Between the… · p. 9
  • Unlike the independent branching-DQN-SAC baseline, the cooperative formulation feeds CPA critic feedback into the DRA update.The CPA still receives the DRA allocation in the independent formulation, but its critic feedback is omitted from the DRA update.Source: C. Cooperative Learning Between the… · p. 9

D. Training and Online Operation

DARA-DRL combines discrete allocation and continuous power control, with cooperative training and feasibility processing for online decisions. The evaluation reports near-optimality, higher EE and QoS satisfaction than benchmarks, and fast online decisions across dynamic scenarios.

D. Training and Online Operation

  • The DRA selects AP and subcarrier choices before the conditional CPA samples and projects power; both agents share replay, while online they use fixed parameters and feasibility processing.Training uses a common reward, and online allocation requires forward inference without replay sampling or learning updates.Source: D. Training and Online Operation · p. 9

E. Complexity Analysis

  • DARA-DRL’s online complexity is O(CD + CP + ν log ν + AL), avoiding exhaustive search but costing more than reactive greedy allocation.The learned policy accounts for long-term QoS shortfalls and allocation-change costs without forecasts.Source: E. Complexity Analysis · pp. 9–10

B. Results

  • DARA-DRL converges fastest, stabilising after approximately 5 × 10^5 interactions at a final return of about 105; independent DQN-SAC, HAPPO, and P-DQN reach about 94, 88, and 80.The plotted return is discounted episode return, with interactions measured in OWC-environment time slots.Source: Figure 2 · p. 10, B. Results · p. 10

A. Simulation Setup

  • Evaluation averages 5 independent training runs and 200 previously unseen episodes per run, using 0.1-s slots and video, browsing, and voice service classes.The classes’ minimum-rate vector is [100, 20, 2] Mbit/s.Source: A. Simulation Setup · p. 10

B. Results

  • DARA-DRL’s online decision time rises from 0.38 ms with 2 users to 1.12 ms with 12, while PDP-OPA reaches 196 ms.Reactive greedy is fastest; DARA-DRL is next fastest and provides higher EE, QoS satisfaction, and allocation stability in the evaluated scenarios.Source: Figure 3 · p. 10, B. Results · pp. 10–11
  • DARA-DRL’s optimality gap rises from 1.8% to 5% as users increase; at 12 web users it is 5.6%, versus 26.7% for independent DQN–SAC.At 12 users, gaps are 37.6% for HAPPO, 41.8% for PDP-OPA, 48.1% for P-DQN, and 60.6% for reactive greedy.Source: Figure 4 · p. 11, B. Results · p. 11
  • Full DARA-DRL achieves the highest EE and lowest AP reassignment rate, 0.62.The hybrid-agent variant retained 69.2% EE with 1.20 reassignments per user per minute; independent training retained 85.1% EE with 0.92.Source: Figure 5 · p. 11, B. Results · pp. 11–12
  • DARA-DRL maintains the highest EE and higher QoS satisfaction than the benchmark schemes.Under mixed mobility, its EE fluctuates with changing channels and resource adjustments but recovers after short reductions, without continuous degradation over 120 s.Source: B. Results · p. 12, Figure 6 · p. 12

VI. CONCLUSION

DARA-DRL addresses dynamic OWC resource allocation by coordinating discrete allocation and continuous power agents. It approaches the optimal solution, improves EE and QoS against benchmarks, and yields lessons about agent specialisation and cooperation.

VI. CONCLUSION

  • DARA-DRL approaches the optimal solution, provides higher EE and QoS satisfaction than learning-based and conventional benchmarks, and produces allocations within a few milliseconds.Source: §VI · p. 12
  • Feasibility layers enforce association, subcarrier, and AP-power constraints, while a common reward penalises violations of users’ minimum-rate requirements.Source: §VI · p. 12
  • The conclusion identifies specialised agents as more effective for different action types and cooperative training as necessary because allocation value depends on subsequent power decisions.Source: §VI · p. 12
  • Future work will consider experimental validation, individual-subcarrier power optimisation, and subcarrier reuse coordination across APs for interference management and EE.Source: §VI · p. 12
Loading 2610.09266v1…