PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning based Wireless Resource Allocation with Application to Vehicular Networks

Le Liang, Hao Ye, Guanding Yu, Geoffrey Ye Li

arXiv:1907.03289v2cs.ITcs.LG

TL;DR

Wireless resource allocation becomes difficult when problems are non-convex, sequential, multi-agent, and dependent on requirements that are hard to model exactly. This paper reviews deep-learning-assisted optimization and deep reinforcement learning for wireless and vehicular networks, reporting architectures and methods that address these challenges, including one achieving about 95% of brute-force optimal performance with 36-bit local observations. It also identifies open issues for future investigation.

  • Problem

    Wireless resource-allocation problems are often non-convex, combinatorial, or sequential, while vehicular networks impose service requirements that are difficult to formulate and optimize exactly.

  • Method

    The paper reviews deep-learning-assisted optimization and deep reinforcement learning methods for wireless resource allocation, including multi-agent architectures for vehicular networks.

  • Results

    About 95% of the optimal performance from time-consuming brute-force search is achieved when each local observation is compressed to 36 bits per time step.

  • Takeaways & Limitations

    Deep learning provides approaches for difficult wireless resource-allocation problems, including cases that cannot be handled or modeled in the traditional optimization framework.

  • Takeaways & Limitations

    Most wireless optimization problems remain computationally difficult, and conventional formulations require exact models that abstract away real-world imperfections.

Abstract

from arXiv · show

It has been a long-held belief that judicious resource allocation is critical to mitigating interference, improving network efficiency, and ultimately optimizing wireless communication performance. The traditional wisdom is to explicitly formulate resource allocation as an optimization problem and then exploit mathematical programming to solve the problem to a certain level of optimality. Nonetheless, as wireless networks become increasingly diverse and complex, e.g., in the high-mobility vehicular networks, the current design methodologies face significant challenges and thus call for rethinking of the traditional design philosophy. Meanwhile, deep learning, with many success stories in various disciplines, represents a promising alternative due to its remarkable power to leverage data for problem solving. In this paper, we discuss the key motivations and roadblocks of using deep learning for wireless resource allocation with application to vehicular networks. We review major recent studies that mobilize the deep learning philosophy in wireless resource allocation and achieve impressive results. We first discuss deep learning assisted optimization for resource allocation. We then highlight the deep reinforcement learning approach to address resource allocation problems that are difficult to handle in the traditional optimization framework. We also identify some research directions that deserve further investigation.

I. INTRODUCTION

Wireless resource allocation is central to improving communication performance, but increasingly complex and dynamic networks expose limits in traditional optimization. The paper motivates deep learning as a data-driven alternative and reviews its use in wireless and vehicular networks.

  • Resource allocation complements physical-layer improvements by optimizing limited communication resources for higher throughput, lower latency, reliability, and coverage.Relevant resources include power, bandwidth, beams, time slots, sub-bands, spreading codes, base stations, and backhaul links.
  • Traditional approaches formulate objectives and constraints as optimization problems, but many formulations are difficult to solve and diverse service requirements are hard to model exactly.Solutions may rely on mathematical programming, graph theory, game theory, local optima, or heuristics depending on complexity and computation time.
  • Deep learning has improved performance over conventional methods in recent wireless resource-allocation studies, while deep reinforcement learning can address service requirements difficult to model or optimize.The paper reviews deep-learning-assisted optimization, deep reinforcement learning, and open research issues, with vehicular networks as a special application example.
  • Most wireless allocation problems are non-convex, combinatorial, or mixed-integer nonlinear, often making even suboptimal solutions computationally difficult for real-time execution.The spectrum-management problem in frequency-selective channels is described as non-convex and NP-hard, with no known polynomial-time algorithm guaranteeing optimality.
  • High-mobility vehicular networks combine V2I capacity maximization with V2V reliability requirements that are sequential and difficult to formulate exactly.Breaking the problem into isolated per-step decisions can lose the long-term balance between V2I and V2V performance and cause performance loss.

B. Deep Learning Assisted Optimization

Deep learning can accelerate optimization by learning solution mappings or optimizing objectives directly, while reinforcement learning handles sequential decisions through interaction and reward maximization. The section introduces core RL formulations and representative algorithms.

  • Deep Learning Assisted Optimization: Deep learning can approximate the mapping from optimization parameters to solutions, moving computationally complex optimization offline and producing good real-time solutions through a trained DNN.The approach uses the universal approximation capability of deep neural networks to approximate solutions generated by existing algorithms.
  • Deep Learning Assisted Optimization: A DNN can also treat an optimization objective directly as a supervised-learning loss and use stochastic gradient descent to seek an optimization solution.This direct objective-minimization approach is presented as an alternative to learning a parameter-to-solution correspondence.
  • Reinforcement Learning: Reinforcement learning addresses sequential decision making by maximizing a numerical reward while an agent interacts with an unknown environment modeled as a Markov decision process.At each step, the agent observes a state, selects an action, receives a reward, and transitions to a new state.
  • Reinforcement Learning: RL learning seeks an optimal policy that maps states to action-selection probabilities and maximizes expected cumulative rewards from an initial state.The policy is denoted π(a|s), and the framework provides native support for sequential decisions.
  • Reinforcement Learning: The section reviews Q-learning, REINFORCE, and actor-critic methods as representatives of value-based, policy-based, and hybrid reinforcement learning approaches.Q-learning estimates action values, policy gradients search policy space directly, and actor-critic combines policy and value approximations.

1) Q-Learning:

The section presents Q-learning and related policy-based methods as foundations for deep reinforcement learning, emphasizing exploration, function approximation, and experience replay. It also distinguishes interaction-based deep RL from methods that assume known transition dynamics.

  • Q-Learning: Q-learning avoids requiring environment dynamics by iteratively updating action-value estimates while following exploratory policies such as ϵ-greedy.The optimal action-value function satisfies the Bellman optimality equation, while Q-learning uses iterative updates instead of directly acquiring dynamics.
  • Q-Learning: Deep Q-learning uses a DQN to approximate action values when state-action spaces are too large for tabular storage.Transitions are stored in replay memory during exploration and used to train the network across MDP episodes.
  • Deep Q-Network with Experience Replay: Experience replay improves sample efficiency by repeatedly sampling stored transitions, while a periodically fixed target network helps stabilize learning.The target network is duplicated from the training network and held fixed for several updates.
  • Policy Gradient and Actor Critic: Policy-gradient methods directly search policy space by updating DNN parameters toward improved performance rather than estimating a value function first.REINFORCE uses accumulated future rewards as an unbiased action-value estimate but can suffer from high variance.
  • Policy Gradient and Actor Critic: Actor-critic methods reduce policy-gradient variance by learning a value-function approximation alongside the policy, but differ from value-based methods in their learning procedure.The value approximation is the critic and the approximate policy is the actor.
  • Deep Reinforcement Learning: MDP-based resource-allocation studies are not treated as deep RL when they assume transition dynamics rather than learning through interaction with an unknown environment.This distinction separates model-based uses of MDPs from trial-and-error deep reinforcement learning.

III. DEEP LEARNING ASSISTED OPTIMIZATION FOR RESOURCE ALLOCATION

The paper presents three ways to incorporate deep learning into wireless resource-allocation optimization: supervised learning, objective-oriented unsupervised learning, and learning-accelerated optimization. These approaches either learn optimization mappings, optimize objectives directly, or accelerate selected algorithmic steps.

  • Deep learning is incorporated through supervised learning, objective-oriented unsupervised learning, and learning-accelerated optimization paradigms.The paradigms respectively learn mappings to algorithmic solutions, use the optimization objective as the training loss, or accelerate selected steps of an existing optimization algorithm.
  • A. Supervised Learning Approach for Optimization: In supervised learning, a DNN learns the mapping from problem parameters to solutions produced by a traditional optimization algorithm.The optimization method acts as a supervisor, and its solutions provide training labels for the DNN.
  • A. Supervised Learning Approach for Optimization: Training uses labeled samples generated by optimization algorithms, while testing feeds new problem parameters to the trained network and compares its inferred solution with the corresponding label.
  • A. Supervised Learning Approach for Optimization: DNNs can closely mimic optimal or heuristic resource-allocation solutions while reducing computational complexity.Reported examples include approximation of WMMSE for interference-channel power allocation and virtually optimal weighted sum energy efficiency learned from branch-and-bound solutions.
  • B. Feature Embedded Paradigm: Graph embedding improves sample efficiency for D2D link scheduling by learning topology-based node representations before DNN scheduling.Each graph node represents a D2D link, and the resulting feature vector is provided to the DNN.
  • B. Feature Embedded Paradigm: The graph-embedding method achieves good link-scheduling performance with only hundreds of training samples and without accurate CSI.Performance is evaluated using classifier accuracy and normalized average sum rate relative to FPLinQ.

B. Unsupervised Learning Approach for Optimization

Unsupervised learning trains DNNs directly against wireless resource-allocation objectives rather than conventional optimization outputs. These approaches can outperform WMMSE, but their nonstandard loss functions create training challenges.

  • Unsupervised learning uses the optimization objective itself as the DNN loss, optimized through stochastic gradient descent.This avoids learning ground-truth solutions from conventional optimization algorithms.
  • DNNs have been trained with sum rate, spectral efficiency, or energy efficiency objectives for wireless resource management.Channel powers can serve as inputs, while transmit powers are DNN outputs.
  • Both cited unsupervised approaches outperform the state-of-the-art WMMSE heuristic.The comparison concerns the unsupervised methods described for power-control and resource-management objectives.
  • Functional optimization maps channel states to power allocations while maximizing long-term average wireless-system performance.Constraints such as power budgets and QoS requirements can be incorporated in the dual domain through weighted objectives.
  • Wireless resource-allocation loss functions raise theoretical and practical difficulties because they differ from standard regression and classification losses.Existing studies use special network structures, ensembles, or WMMSE pretraining to improve optimization performance.

C. Deep Learning Accelerated Optimization

Deep learning can accelerate selected components of established optimization algorithms by exploiting domain knowledge. In mixed-integer wireless allocation, learning-based pruning makes branch-and-bound substantially faster while retaining near-optimal performance.

  • Embedding deep learning within a well-behaved optimization algorithm uses domain knowledge to improve sample efficiency and accelerate computation.This contrasts with fully replacing optimization by a black-box learning module.
  • Wireless resource allocation often forms mixed integer nonlinear programs combining discrete and continuous decisions under power or QoS constraints.Such MINLP problems are generally NP-hard and difficult to solve optimally.
  • Learning-based branch-and-bound accelerates optimization by converting the time-consuming branching or pruning decision into a binary classification task.Invariant problem-independent and selected problem-dependent features support the learned pruning policy.

IV. DEEP REINFORCEMENT LEARNING BASED RESOURCE ALLOCATION

Deep reinforcement learning addresses wireless resource-allocation settings involving partial observability, changing environments, and requirements that are difficult to model exactly. Across spectrum-access examples, learned policies approach model-informed benchmarks and support distributed multi-user operation.

  • Deep RL has been applied to network slicing, vehicular-network resource orchestration, channel selection, and millimeter-wave beam selection.These applications illustrate its use across diverse wireless resource-allocation problems.
  • A. Dynamic Spectrum Access: Dynamic spectrum access models a user selecting one of N channels whose good or bad conditions vary over time.The objective is to maximize successful transmissions.
  • A. Dynamic Spectrum Access: Partial observability arises because the user senses only one channel per slot, leaving the other channel conditions unknown.A DQN therefore receives histories of previous actions and success or failure observations.
  • A. Dynamic Spectrum Access: For N = 16 strongly correlated channels, the DQN method outperforms the Whittle Index heuristic and closely approaches the genie-aided Myopic policy.The reward is +1 for successful transmission and −1 for failure, with ϵ-greedy exploration during learning.
  • A. Dynamic Spectrum Access: An actor-critic framework was proposed because the DQN approach encounters difficulty as the number of channels becomes large.The actor maps observation histories to a distribution over actions.
  • A. Dynamic Spectrum Access: In a heterogeneous network, model-free deep RL achieves performance measurably close to a theoretical upper bound requiring full model knowledge.The setting includes devices coexisting with TDMA- and ALOHA-based transmitters.
  • A. Dynamic Spectrum Access: For multiple users and orthogonal channels, deep RL supports objectives such as sum rate under successful, collision-free transmissions.Centralized training can produce DQNs that users execute independently from local observations without online coordination.
  • A. Dynamic Spectrum Access: Environment nonstationarity motivates disabling experience replay and using dueling networks or double Q-learning to improve convergence.These techniques address outdated experiences and training stability.

B. Power Allocation in Wireless Networks

Wireless power allocation adapts transmit powers to channel and user conditions, but its coupled optimization is generally non-convex and NP-hard. Deep RL offers distributed, lower-overhead solutions that track channel evolution and can outperform conventional benchmarks.

  • Power-allocation formulation: Power allocation adapts transmit power to varying channel and user conditions to optimize system metrics such as weighted sum rate.The transmit power vector contains the powers of all communication links, while weights can represent sum-rate maximization or proportional fairness.
  • Challenges: Coupled transmit powers make the dynamic allocation problem generally non-convex and NP-hard, while acquiring global CSI creates substantial signaling overhead.These challenges motivate model-free distributed learning approaches.
  • Deep RL approach: A model-free deep RL scheme tracks channel evolution and performs distributed power allocation with limited information exchange.Agents explore the unknown environment, while centralized training collects their experience before deployment.
  • Results: Deep RL outperforms WMMSE and fractional programming benchmarks without assuming prior channel knowledge, producing solutions that are better and faster.The approach also shows robustness when DQNs are trained with different initializations or numbers of links.
  • Multi-cell extension: In multi-cell networks, DQN, REINFORCE, and DDPG outperform WMMSE and FP, with actor-critic DDPG best in sum-rate performance and robustness.Each base station acts as an RL agent, with centralized training and distributed policy execution.

C. Joint Spectrum and Power Allocation: Application Example in Vehicular Networks

In vehicular networks, V2V links share the spectrum of V2I links, coupling high-rate entertainment traffic with reliable, deadline-constrained safety messaging. Deep RL is introduced as a distributed resource-allocation approach for this setting.

  • Network setting: K V2V links share the spectrum of M V2I links to improve utilization efficiency.V2I links support high-data-rate entertainment services, while V2V links disseminate safety-critical messages.
  • Service requirements: V2V reliability requires successful delivery of packets of size B within a time constraint T, which is difficult for traditional optimization approaches.The requirement combines packet delivery reliability with an explicit deadline.
  • Deep RL application: The proposed formulation therefore targets joint spectrum and power allocation across interacting V2V and V2I links.The application example uses deep RL to address the coupled vehicular-network resource-allocation problem.

1) Single-Agent RL:

The single-agent RL approach distributes learning across V2V transmitters while using centralized training to coordinate system-wide objectives. Its reward balances V2I capacity, V2V capacity, and deadline-sensitive payload delivery.

  • Single-Agent RL: Each V2V transmitter acts as a learning agent with a local DQN and observes local channel, interference, neighbor-RB, load, and latency information.The observation includes current V2V and interference channel strengths, prior interference and neighbor selections, remaining load, and remaining time.
  • Single-Agent RL: Each agent selects a resource block and discrete transmit power levels.The action space jointly controls spectrum selection and power adaptation.
  • Reward design: The reward balances V2I and V2V requirements using link capacities and nonnegative objective weights.The reward includes V2I capacity, V2V capacity, and weighting terms for balancing design objectives.
  • Reward design: Including T − Ut in the reward reminds agents of the approaching V2V deadline and improves V2V payload delivery rates.Ut denotes the remaining time to meet the latency constraint.
  • Architecture and results: Centralized training compiles experiences from all agents, while trained local DQNs execute allocation distributively; turn-taking stabilizes training under mutual interaction.Experiments show the approach outperforms a distributed benchmark and random allocation in both sum V2I rate and V2V payload delivery rate.

2) Multi-Agent RL:

Multi-agent RL addresses the suboptimality and training instability caused by turn-taking or disabled experience replay in simultaneous V2V allocation. Fingerprint conditioning and revised rewards stabilize learning, improve performance, and can support cooperation or compressed centralized decisions.

  • 2) Multi-Agent RL:: Simultaneous actions make each agent’s environment highly nonstationary, motivating turn-taking or disabling experience replay but compromising efficiency or optimality.The simultaneous action space is broader than the turn-taking subset, while experience replay is central to deep RL.
  • 2) Multi-Agent RL:: Fingerprint conditioning addresses nonstationarity caused by other agents’ policy changes during learning.The method conditions each agent’s environment on information about the evolving policies.
  • 2) Multi-Agent RL:: The revised reward uses the sum V2V rate before delivery finishes and a constant β afterward, balancing goal direction against learning efficiency.β is set larger than the largest sum V2V rate.
  • 2) Multi-Agent RL:: The multi-agent method significantly outperforms the single-agent RL approach and random baseline while stabilizing DQN training.It encourages V2V cooperation without online coordination; links can sequence transmissions and alternate to avoid strong mutual interference.
  • 2) Multi-Agent RL:: A compressed-observation architecture enables centralized decision making with low signaling overhead and achieves about 95% of brute-force optimal performance using 36 bits per time step.Each V2V transmitter compresses its local observation before feeding it back to the central base station.

V. OPEN ISSUES AND FUTURE RESEARCH

The paper identifies roadblocks for deep learning in wireless resource allocation, including architecture selection, safe transition from simulation to deployment, and multi-agent learning under nonstationarity and partial observability.

  • A. Tailoring Deep Learning Architecture for Wireless: Deep learning architectures have different strengths, but the best way to incorporate wireless domain knowledge into DNN design remains unclear.Power allocation is non-convex in the primal domain, while dual-domain solutions can be much easier and have small duality gaps in practical cases.
  • B. Bridging the Gap between Training and Implementation: Offline simulator training simplifies development, but deployment requires avoiding catastrophic actions during active exploration before effective policies are learned.Expert knowledge may constrain exploration, yet implementing this with performance guarantees remains unresolved.
  • C. Multi-Agent Consideration in Deep RL: Multi-user deep RL faces both environment nonstationarity from other agents’ exploration and partial observability from local measurements.Inter-agent communication may improve coordination and awareness, but the combined learning problem remains extremely difficult.
  • VI. CONCLUSION: The article identifies open issues and research directions while positioning deep RL as a direct approach for allocation problems traditional optimization cannot handle or model.Its scope covers deep learning for wireless resource allocation with application to vehicular networks.
Loading 1907.03289v2…