PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks

Yiding Yu, Taotao Wang, Soung Chang Liew

arXiv:1712.00162v2cs.NI

TL;DR

The paper addresses time-slot sharing among heterogeneous wireless networks when a DLMA node lacks knowledge of coexisting MAC mechanisms. It uses deep reinforcement learning to learn MAC strategies for sum-throughput and α-fairness objectives, achieving near-optimal performance while converging faster and showing greater robustness than traditional reinforcement learning.

  • Problem

    The paper addresses how wireless networks using heterogeneous, unknown MAC protocols can share time slots according to specified objectives.

  • Method

    The paper designs DLMA, a deep-reinforcement-learning MAC protocol that learns from state-action-reward observations and uses channel action-observation states.

  • Results

    DLMA achieves near-optimal sum throughput and proportional fairness without knowing coexisting protocols, while DRL converges faster and is more robust than traditional RL.

  • Takeaways & Limitations

    DLMA can learn overall objectives during operation in heterogeneous environments without detailed knowledge of other coexisting MACs.

Abstract

from arXiv · show

This paper investigates the use of deep reinforcement learning (DRL) in a MAC protocol for heterogeneous wireless networking referred to as Deep-reinforcement Learning Multiple Access (DLMA). The thrust of this work is partially inspired by the vision of DARPA SC2, a 3-year competition whereby competitors are to come up with a clean-slate design that "best share spectrum with any network(s), in any environment, without prior knowledge, leveraging on machine-learning technique". Specifically, this paper considers the problem of sharing time slots among a multiple of time-slotted networks that adopt different MAC protocols. One of the MAC protocols is DLMA. The other two are TDMA and ALOHA. The nodes operating DLMA do not know that the other two MAC protocols are TDMA and ALOHA. Yet, by a series of observations of the environment, its own actions, and the resulting rewards, a DLMA node can learn an optimal MAC strategy to coexist harmoniously with the TDMA and ALOHA nodes according to a specified objective (e.g., the objective could be the sum throughput of all networks, or a general alpha-fairness objective).

I. INTRODUCTION

The paper develops DLMA, a deep-reinforcement-learning MAC protocol for heterogeneous wireless networks that learns spectrum-sharing strategies without prior knowledge of coexisting MAC mechanisms. It targets sum-throughput and α-fairness objectives, while emphasizing convergence speed and robustness for dynamically changing environments.

  • Motivation and contributions: The study focuses on time-slot sharing among networks using different MAC protocols and evaluates sum throughput and general α-fairness objectives.The considered coexistence scenarios include TDMA, ALOHA, and mixtures of the two.
  • Methodological novelty: The implementation adopts a deep residual network whose static architecture can be reused across different wireless network scenarios.The paper contrasts this with plain DNNs, whose optimal depth varies by scenario.
  • Motivation and contributions: DLMA uses DRL to learn MAC strategies for heterogeneous wireless networks without knowing the operating mechanisms of coexisting protocols.The protocol learns from observations, actions, and rewards, aiming to perform as if it knew the other MAC protocols in detail.
  • Results: Extensive simulations show that DLMA achieves near-optimal sum throughput and proportional fairness without identifying the coexisting TDMA and ALOHA protocols.These results are reported for coexistence with TDMA, ALOHA, and mixed TDMA-ALOHA networks.
  • Results: Compared with traditional RL, DRL converges faster and is more robust to non-optimal parameter settings.These properties are presented as important for practical deployment when wireless environments change quickly and protocol details are unknown.
  • Methodological novelty: The paper generalizes Q-learning by separating the Q function from the objective function used to select actions.This framework supports optimization of more general objectives, including α-fairness.

II. DLMA PROTOCOL

The paper models time-slotted heterogeneous networks in which nodes using TDMA, ALOHA, and DLMA share a wireless channel. It introduces the reinforcement-learning formulation in terms of states, actions, rewards, and long-term accumulated return.

  • A. Time-Slotted Heterogeneous Wireless Networks: The system consists of nodes transmitting to an access point over a shared time-slotted channel, where simultaneous transmissions cause collisions.Nodes may use different MAC protocols, including TDMA, ALOHA, and DLMA.
  • A. Time-Slotted Heterogeneous Wireless Networks: TDMA transmits in fixed slots within repeating frames, whereas q-ALOHA transmits independently in each slot with probability q.The section also describes fixed-window and exponential-backoff ALOHA variants.
  • A. Time-Slotted Heterogeneous Wireless Networks: A DLMA node receives an ACK after transmitting or listens for other nodes’ transmission results and channel idleness when waiting.These observations allow the node to select objectives such as maximizing sum throughput.
  • B. Overview of RL: In reinforcement learning, an agent observes state s_t, takes action a_t, receives reward r_t+1, and transitions to state s_t+1.The policy is updated through experience to maximize a long-term accumulated reward, potentially discounted by γ.
  • B. Overview of RL: Q-learning estimates an action-value function representing expected accumulated reward for taking an action in a state under a policy.The optimal action-value function is defined through maximization over policies and satisfies the Bellman optimality equation.
  • B. Overview of RL: The ε-greedy policy selects the highest-value action with probability 1−ε and a random action with probability ε.Random exploration helps avoid becoming stuck with an insufficiently converged action-value function.

C. DLMA Protocol Using DRL

DLMA represents channel history as state-action observations and trains a deep Q-network to choose between transmitting and waiting. Its reward design supports sum-throughput optimization, while the neural-network approach addresses the inefficiency of tabular RL in large state spaces.

  • DLMA state, action, and reward: A DLMA agent chooses TRANSMIT or WAIT and observes SUCCESS, COLLISION, or IDLENESS through acknowledgments or channel listening.The resulting action-observation pair forms the basis of the channel state.
  • DLMA state, action, and reward: The environmental state is a history of the most recent M channel action-observation pairs.Five action-observation combinations are possible because WAIT can be followed by SUCCESS, COLLISION, or IDLENESS.
  • DLMA state, action, and reward: For the sum-throughput objective, the reward is 1 after SUCCESS and 0 after COLLISION or IDLENESS.A reward vector is introduced later to generalize DLMA to α-fairness.
  • Deep Q-network: DRL approximates Q∗(s,a) with a deep Q-network that maps a state to action-value estimates and updates network weights through training.The agent selects actions from the network outputs rather than using tabular updates.
  • Deep Q-network: Each experience sample contains the current state, action, reward, and next state, and is used to train the Q-network toward a refined discounted-return target.The target combines the current reward with predicted future rewards, and the step size ρ controls weight adjustment.
  • DLMA algorithm: DLMA improves training stability using experience replay and a quasi-static target network.Experience replay stores tuples in a finite FIFO memory and samples batches for training.

III. SUM THROUGHPUT PERFORMANCE EVALUATION

This section evaluates DLMA for maximizing aggregate throughput in heterogeneous, time-slotted networks. It uses a model-free DRL framework with a ResNet-based QNN and benchmarks against model-aware operation.

  • Scope and setup: DLMA is evaluated with the objective of maximizing the sum throughput of coexisting networks across TDMA, ALOHA, and mixed scenarios.The framework is later reformulated for general α-fairness, with sum throughput as a subcase.
  • Evaluation: The reported evaluation includes simulations with TDMA nodes, ALOHA nodes, and mixtures of the two.The supplied figure and table entries identify the evaluation materials and DLMA hyper-parameters.
  • Architecture: The DLMA QNN uses a six-hidden-layer ResNet with 64 neurons in each hidden layer.It contains fully connected layers followed by two ResNet blocks with shortcut connections.
  • Benchmark: DLMA is model-free, whereas the benchmark model-aware nodes know the coexisting MAC mechanisms and use protocols optimized for them.The comparison tests whether DRL can approach optimal throughput without protocol knowledge.

A. Coexistence with TDMA networks

The TDMA experiments test whether one DRL node can learn to use slots left unused by a repeating TDMA schedule without knowing that schedule or protocol.

  • Setup: The TDMA benchmark uses a node transmitting in X specific slots within each frame of Y slots, while the model-aware node knows those slots.The model-aware node transmits in all Y − X slots not used by TDMA.
  • Setup: The DRL agent does not know the other node is TDMA or how many other nodes exist, and learns its strategy through DRL.This contrasts with the TDMA-aware benchmark, which has full schedule knowledge.
  • Results: A sum throughput close to 1 is achieved when Y = 10 and X varies from 2 to 8.The result indicates that the DRL node captures the unused TDMA slots without knowing the adopted protocol.

B. Coexistence with ALOHA networks

The ALOHA experiments apply the same DLMA algorithm across q-ALOHA, FW-ALOHA, and EB-ALOHA settings and compare it with protocol-aware benchmarks.

  • Setup: The experiments evaluate one DRL node coexisting separately with q-ALOHA, FW-ALOHA, and EB-ALOHA nodes.The ALOHA variants differ in transmission probability, fixed-window size, or backoff parameters.
  • Measurement: Unless otherwise stated, throughput is a short-term average over the preceding N = 1000 steps, and bar-chart values average the final-window rewards across 10 experiments.With 1 ms steps, the short-term window represents the past second.
  • Method: The same DLMA algorithm is used even though the coexisting protocols are no longer TDMA.Model-aware benchmark nodes instead use MACs tailored to each ALOHA mechanism.
  • Results: The DRL node learns optimal throughputs with q-ALOHA without knowing that the other node uses q-ALOHA or knowing its transmission probability q.FW-ALOHA and EB-ALOHA are tested across different window settings.

C. Coexistence with a mix of TDMA and ALOHA networks

Mixed-network experiments examine aggregate throughput, fairness, and learning speed when DRL nodes coexist with TDMA and q-ALOHA nodes. DLMA approaches optimal aggregate throughput, but uncoordinated DRL nodes can produce severe individual-throughput imbalance.

  • Mixed coexistence: With one DRL node, one TDMA node, and one q-ALOHA node, the same DLMA algorithm is tested while TDMA occupancy or q varies.The first case uses three TDMA slots out of ten; the second fixes q = 0.2 while TDMA occupancy varies.
  • Mixed coexistence: With three DRL nodes, one TDMA node using 2 of 10 slots, and two q-ALOHA nodes with q = 0.1, DLMA achieves near-optimal sum throughput.This is a more complex mixed-network setup than the single-DRL-node experiment.
  • Individual throughput: Without coordination among DRL nodes, one DRL node may preempt all slots outside TDMA occupancy, leaving the other DRL and q-ALOHA nodes with zero throughput.This observation motivates the later proportional-fairness formulation.
  • RL versus DRL: DRL approaches the optimal throughput of 1 in fewer than 5000 steps, or 5 s when each step lasts 1 ms.RL learns progressively more slowly as state history length increases, whereas DRL convergence varies only slightly.
  • RL versus DRL: Both RL and DRL eventually reach optimal throughput, but DRL converges faster and RL exhibits larger throughput fluctuations during learning.The figure uses short-term throughput over N = 1000 steps rather than cumulative throughput.
  • State exploration: RL requires 35000 steps and visits 23000 distinct states to find the optimal strategy, whereas DRL requires 10000 steps and visits around 1000 states.The comparison indicates that DRL narrows the states it visits more effectively during learning.

E. Plain DNN versus deep ResNet

The paper compares plain DNN and deep ResNet architectures for DLMA across wireless coexistence scenarios, emphasizing robustness to changing environments. It also frames α-fairness as a generalized objective beyond sum throughput.

  • Architecture robustness: Plain DNN performance varies with the architecture parameter h across wireless setups.The best h is 1 in one case and 4 in another, making a common plain-DNN architecture difficult to use.
  • Architecture robustness: Deep ResNet is presented as more suitable than plain DNN when wireless environments change dynamically.A static ResNet architecture can be reused across different scenarios, whereas the optimal plain-DNN depth may differ by case.
  • Objective formulation: The α-fairness formulation extends DLMA beyond sum-throughput optimization to proportional fairness and other fairness criteria.The parameter α selects the fairness objective, with α = 0 corresponding to sum throughput and α = 1 to proportional fairness.

B. DLMA reformulation

The reformulated DLMA system coordinates multiple DRL nodes as a virtual big agent and replaces scalar Q-learning with multidimensional action-value estimates. Actions are then selected using a general objective, allowing α-fairness rather than only discounted-reward maximization.

  • System reformulation: The gateway coordinates K DRL nodes by selecting them round-robin, creating one virtual big agent alongside L legacy nodes.The reformulated system contains L + 1 nodes: one DRL big agent and L non-DRL nodes.
  • Multi-dimensional Q-learning: Multi-dimensional Q-learning maintains an action-value vector with one elementary estimate for each node.Each Q(i)(s, a) estimates the expected accumulated discounted reward of node i.
  • Multi-dimensional Q-learning: The new algorithm is needed because conventional single-agent scalar Q-learning cannot directly satisfy arbitrary fairness objectives in a multi-node setting.The reformulation changes the reward representation from a scalar to an L + 1-dimensional vector.
  • Objective and Q-function separation: The generalized action rule applies the α-fairness objective to the vector of elementary action-value estimates.The exact throughput is unavailable, so estimated accumulated rewards are used to make decisions.
  • Objective and Q-function separation: DLMA separates the Q function from the objective used to choose actions.The Q function remains a projected weighted sum of current and future rewards, while the policy can optimize a more general objective such as α-fairness.
  • Deep-Q implementation: The reformulated DLMA protocol incorporates a QNN, experience memory, ε-greedy action selection, and target-network updates.The supplied algorithm description initializes QNN parameters and repeatedly observes transitions, stores experience, trains, and updates the target QNN.

V. PROPORTIONAL FAIRNESS PERFORMANCE EVALUATION

The evaluation studies proportional fairness for DLMA coexisting with TDMA, ALOHA, or both, using model-aware optima as benchmarks. The stated focus is whether the generalized DLMA formulation achieves proportional fairness across these heterogeneous cases.

  • Evaluation scope: The proportional-fairness evaluation covers coexistence with TDMA nodes, ALOHA nodes, and mixtures of TDMA and ALOHA nodes.The study uses model-aware optimal results for benchmarking different cases.
  • Evaluation scope: Model-aware nodes provide benchmark optima for the proportional-fairness experiments.The derivations of these benchmarks are stated to be provided elsewhere and omitted from the paper section.
  • Evaluation scope: The section treats proportional fairness as a representative example of the general α-fairness DLMA formulation.The evaluation therefore focuses on one fairness criterion within the broader formulation.

A. Coexistence with TDMA networks

For one DRL node coexisting with one TDMA node, proportional fairness coincides with maximizing sum throughput. DLMA learns to avoid TDMA-occupied slots and achieves proportional fairness across the tested allocations.

  • TDMA coexistence: In the one-DRL-node and one-TDMA-node case, proportional fairness is equivalent to maximizing sum throughput.The optimal DRL strategy transmits in slots not occupied by TDMA and remains silent in TDMA slots.
  • TDMA coexistence: The experiment tests TDMA allocations of 2, 3, 7, and 8 slots out of 10 per frame.These allocations are evaluated for coexistence between one DRL node and one TDMA node.
  • TDMA coexistence: The reformulated DLMA protocol achieves proportional fairness in the tested one-TDMA-node cases.The result is reported for all listed TDMA slot allocations in Fig. 10(a).

B. Coexistence with ALOHA networks

DLMA is evaluated in coexistence with ALOHA and TDMA networks under both individual-throughput and proportional-fairness objectives. The reformulated protocol approximates optimal outcomes without detailed knowledge of the other MAC protocols.

  • ALOHA coexistence: The experiments include one DRL node coexisting separately with q-ALOHA, FW-ALOHA, and EB-ALOHA nodes.The cases vary transmission probabilities, fixed-window sizes, and initial window sizes.
  • Mixed coexistence: With one TDMA and one q-ALOHA node, reformulated DLMA achieves individual throughputs approximating those of a model-aware node.This evaluation targets proportional fairness among all nodes.
  • Mixed coexistence: Three DRL nodes can operate as one big agent alongside one TDMA node and two q-ALOHA nodes under a proportional-fairness objective.The setup uses the same coexistence case as an earlier experiment, with optimal results derived for comparison.
  • Mixed coexistence: The reformulated DLMA protocol also approximates the optimal results in the three-DRL-node coexistence experiment.Individual throughputs are computed using a moving average with N = 5000.
  • Implication: DLMA can learn near-optimal performance for an overall objective from state-action-reward observations without knowing the detailed operating mechanisms of coexisting MACs.The framework supports objectives such as α-fairness in heterogeneous wireless environments.
Loading 1712.00162v2…