Source-linked AI summary
DeepRMSA: A Deep Reinforcement Learning Framework for Routing, Modulation and Spectrum Assignment in Elastic Optical Networks
Xiaoliang Chen, Baojia Li, Roberto Proietti, Hongbo Lu, Zuqing Zhu, S. J. Ben Yoo
TL;DR
Dynamic RMSA must make immediate routing, modulation, and spectrum decisions under changing requests and uncertain future traffic, while existing policies lack comprehensive state perception. DeepRMSA uses DNN-parameterized reinforcement-learning policies trained from provisioning experience, with episode-based and sliding-window mechanisms. Across the evaluated topologies, DeepRMSA-FLX stabilized training and reduced blocking probability by more than 20.3% and 14.3% versus baselines.
Problem
Dynamic RMSA is difficult because request arrivals and departures destabilize EON states, while existing methods use fixed policies or manually extracted features rather than holistic state perception.
Method
DeepRMSA parameterizes online RMSA policies with DNNs and trains them from dynamic provisioning experience using episode-based and window-based mechanisms adapted from A3C.
Results
DeepRMSA-FLX reduced blocking probability by 20.3% versus KSP-FF and by 14.3% versus KSP-FF in the evaluated topologies.
Takeaways & Limitations
DeepRMSA-FLX can stabilize training while achieving lower blocking probability than the reported baselines in the evaluated NSFNET and COST239 topologies.
Abstract
from arXiv · showhide
This paper proposes DeepRMSA, a deep reinforcement learning framework for routing, modulation and spectrum assignment (RMSA) in elastic optical networks (EONs). DeepRMSA learns the correct online RMSA policies by parameterizing the policies with deep neural networks (DNNs) that can sense complex EON states. The DNNs are trained with experiences of dynamic lightpath provisioning. We first modify the asynchronous advantage actor-critic algorithm and present an episode-based training mechanism for DeepRMSA, namely, DeepRMSA-EP. DeepRMSA-EP divides the dynamic provisioning process into multiple episodes (each containing the servicing of a fixed number of lightpath requests) and performs training by the end of each episode. The optimization target of DeepRMSA-EP at each step of servicing a request is to maximize the cumulative reward within the rest of the episode. Thus, we obviate the need for estimating the rewards related to unknown future states. To overcome the instability issue in the training of DeepRMSA-EP due to the oscillations of cumulative rewards, we further propose a window-based flexible training mechanism, i.e., DeepRMSA-FLX. DeepRMSA-FLX attempts to smooth out the oscillations by defining the optimization scope at each step as a sliding window, and ensuring that the cumulative rewards always include rewards from a fixed number of requests. Evaluations with the two sample topologies show that DeepRMSA-FLX can effectively stabilize the training while achieving blocking probability reductions of more than 20.3% and 14.3%, when compared with the baselines.
I. INTRODUCTION
Elastic optical networks provide flexible bandwidth and modulation, but dynamic RMSA remains difficult because changing requests destabilize network state and existing policies lack holistic adaptation. DeepRMSA applies deep reinforcement learning to learn online RMSA policies from provisioning experience.
- EONs support bandwidth-variable superchannels by grooming fine-granularity subcarriers and adapting modulation to lightpath QoT.
- Static RMSA optimization can be solved with ILP models, but the problem is NP-hard and intractable for large-scale networks.
- Dynamic RMSA is harder because lightpath arrivals, departures, and uncertain future traffic can destabilize the EON state.
- Existing dynamic RMSA methods use fixed policies or manually extracted features, lacking comprehensive perception of holistic EON states.
- DeepRMSA uses deep reinforcement learning to learn online RMSA policies with deep neural networks that perceive complex system states through repeated interactions.
II. PROBLEM FORMULATION
The dynamic RMSA problem provisions arriving and expiring lightpath requests online while jointly selecting routes, modulation formats, and contiguous spectrum under optical-network constraints. Its objective is to minimize long-term request blocking probability.
- A lightpath request is represented by its origin, destination, bandwidth requirement, and service duration within an EON topology.
- Provisioning requires an end-to-end route, a modulation format for QoT assurance, and contiguous frequency slots along the route.
- Without spectrum conversion, spectrum assignments must remain aligned across all fibers used by a lightpath.
- Dynamic RMSA services requests immediately as they arrive and expire, targeting the minimum long-term blocking probability.
III. DEEPRMSA FRAMEWORK
DeepRMSA combines centralized SDN control with deep reinforcement learning to sense EON conditions, select RMSA schemes, and adapt policies through provisioning experience. Its reward objective is long-term cumulative reward.
- A remote SDN controller collects network states and lightpath requests, distributes RMSA schemes, and coordinates local SDN agents.
- DeepRMSA receives a request and uses a learned policy to determine an RMSA scheme within the centralized control loop.
- The framework optimizes long-term cumulative reward, with future rewards discounted by γ.
- DeepRMSA can learn and adapt RMSA policies through dynamic lightpath provisioning, and parallel agents can extend it toward an autonomic EON system.
IV. DEEPRMSA DESIGN
The design section defines DeepRMSA’s state, action, and reward modeling and introduces two training mechanisms tailored to dynamic lightpath provisioning.
- DeepRMSA’s design covers state representation, action space, reward definition, and two training mechanisms for dynamic lightpath provisioning.
A. Modeling
DeepRMSA models online RMSA decisions from request information and spectrum utilization on candidate paths, while selecting routing and spectrum actions through a neural policy.
- State: The state st is a 1×(2|V|+1+(2J+3)K) array encoding the request and spectrum-utilization features of K candidate paths.It includes source, destination, service duration, available FS-block sizes and positions, required FSs, average available-block size, and total available FSs.
- State: Directly including the full two-dimensional spectrum state would require f0·|E| additional elements and create scalability issues.The paper leaves feature extraction from the large binary spectrum matrix and its EON constraints as future work.
- Action: DeepRMSA chooses one of K candidate routing paths and one of J FS-blocks, yielding K·J possible actions.Each action jointly specifies the path and spectrum block for the request.
- DNNs: The policy DNN outputs an action distribution over K·J actions, while the value DNN estimates the discounted cumulative reward of the state.The two networks share the fully connected architecture but differ in their output layers.
B. Training
DeepRMSA adapts A3C training to dynamic lightpath provisioning with episode-based and sliding-window reward scopes. The flexible mechanism keeps reward horizons fixed to stabilize learning under request uncertainty.
- A3C framework: A3C uses parallel actor-learners that interact with separate environment copies and update shared global DNN parameters from diverse experiences.DeepRMSA bases its training design on this asynchronous actor-learner framework.
- DeepRMSA-EP: DeepRMSA-EP defines an episode as servicing N lightpath requests, matching N to the training batch size and avoiding estimation of the next state’s value.This modification addresses state transitions that are difficult to model because future requests may be random.
- DeepRMSA-EP: DeepRMSA-EP computes each sample’s discounted cumulative reward from its request through the end of the episode and trains when the buffer contains N samples.The request position within the episode is added to the state representation.
- Losses: The policy loss reinforces actions with larger advantages and adds an entropy penalty to encourage exploration, while the value loss uses mean squared error.Gradients from these losses update the global DNN parameters asynchronously.
- DeepRMSA-FLX: DeepRMSA-FLX trains when the buffer contains 2N−1 samples and slides a length-N window so each cumulative reward covers N requests.The fixed reward coverage is intended to smooth oscillations caused by unpredictable dynamic-request trajectories.
A. Simulation Setup
The simulations evaluate DeepRMSA under dynamic lightpath traffic using NSFNET settings, with training configurations and cumulative rewards examined across algorithm variants.
- Traffic and topology: The NSFNET evaluation uses dynamic lightpath requests generated by a Poisson process with uniform traffic distribution.The average arrival rate is 10 time units, service duration is 15 time units, and bandwidth requirements range evenly from 25 to 100 Gb/s.
- Training configuration: The training configuration uses N=50 requests per episode, γ=0.95, α=0.01, and learning rate 10^-5.The experiments use Adam, normalize every state field before DNN input, and apply first-fit spectrum allocation after routing.
- Training measurements: Figure 3 compares cumulative rewards for DeepRMSA-FLX and DeepRMSA-EP across DNN sizes and numbers of actor-learners.Panels (a) and (c) vary DNN sizes, while panels (b) and (d) vary actor-learners.
B. Numerical Results
The numerical results show that larger models and more actor-learners speed training, while DeepRMSA-FLX stabilizes learning and lowers blocking probability across NSFNET and COST239 evaluations.
- Training configuration: Increasing DNN scale facilitates faster training for both DeepRMSA-EP and DeepRMSA-FLX.The tested configurations are 3×64, 5×128, and 8×256 hidden-layer/neuron setups.
- Blocking probability: Figure 4 reports request blocking probability for DeepRMSA-EP, DeepRMSA-FLX, SP-FF, and KSP-FF during training.The figure supports comparison of blocking-probability evolution across the proposed and baseline algorithms.
- Training configuration: Nearly 10× faster training results when actor-learners increase from 1 to 8, with slightly higher rewards and marginal gains beyond that.The later evaluations therefore use 5×128 DNNs and 16 actor-learners.
- NSFNET results: 20.3% blocking reduction versus KSP-FF is achieved by DeepRMSA-FLX after 150,000 training epochs on NSFNET.DeepRMSA-FLX surpasses KSP-FF after 37,500 epochs, while DeepRMSA-EP fluctuates around KSP-FF.
- Training stability: DeepRMSA-FLX facilitates lower value losses and stabilized training, whereas DeepRMSA-EP begins diverging after 10,000 epochs.Figure 5 reports normalized value loss and policy entropy during training.
- COST239 results: 14.3% lower blocking probability than KSP-FF and 18.9% lower than DeepRMSA-EP is achieved by DeepRMSA-FLX on COST239.The COST239 evaluation uses an 11-node topology with average arrival rate 20 and service duration 30 time units.
VI. CONCLUSION
The paper concludes that DeepRMSA learns online RMSA policies with DNNs trained from dynamic provisioning experiences, and that its A3C-based mechanisms improve blocking performance. Larger-scale topologies remain a future scalability target.
- Conclusion: DeepRMSA parameterizes online RMSA policies with DNNs and progressively trains them using dynamic lightpath provisioning experiences.The framework develops two training mechanisms based on A3C.
- Conclusion: Blocking reductions exceed 20.3% on NSFNET and 14.3% on COST239 relative to the baselines.The reported simulation results associate these reductions with the proposed training mechanisms.
- Future work: Partitioned, hierarchical, or multi-agent DeepRMSA is proposed as future work for scaling to larger and multi-domain EON topologies.The proposed settings include hierarchical cooperation within an autonomous system and peer-to-peer interaction through brokers across domains.