Source-linked AI summary
A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization
Giorgio Stampa, Marta Arias, David Sanchez-Charles, Victor Muntes-Mulero, Albert Cabellos
TL;DR
Routing optimization needs methods that can adapt to network conditions and handle complex routing decisions efficiently. The paper designs and evaluates a DRL agent that learns routing policies from network interaction to minimize delay. Experiments show promising performance, including average results within the benchmark’s first quartile, alongside operational advantages such as one-step configuration generation.
Problem
The paper addresses routing optimization by seeking an approach that generalizes to unseen network states and avoids iterative improvement steps used by traditional methods.
Method
The paper develops an off-policy, actor-critic deterministic policy-gradient agent that maps traffic-matrix states to link-weight actions using reward based on mean network delay.
Results
Across all traffic intensities, the trained DRL agent computes routing configurations that are on average within the benchmark’s 1st quartile.
Takeaways & Limitations
Once trained, the DRL agent can produce a near-optimal routing configuration in one step, an advantage for real-time network control.
Abstract
from arXiv · showhide
In this paper we design and evaluate a Deep-Reinforcement Learning agent that optimizes routing. Our agent adapts automatically to current traffic conditions and proposes tailored configurations that attempt to minimize the network delay. Experiments show very promising performance. Moreover, this approach provides important operational advantages with respect to traditional optimization algorithms.
1 INTRODUCTION
The paper applies a Deep-Reinforcement Learning agent to routing optimization, targeting reduced network delay. It reports promising benchmark performance and operational advantages.
- 1 INTRODUCTION: The paper designs and trains a DRL agent to optimize routing against a predefined target metric: network delay.The approach focuses on adapting routing optimization to network conditions through deep neural networks and reinforcement learning.
- 1 INTRODUCTION: The agent provides promising performance against an initial benchmark while offering important operational advantages.
2 STATE OF THE ART
Routing optimization traditionally uses analytical or heuristic methods, while prior reinforcement-learning approaches use table-based agents. The paper proposes DRL to generalize to unseen network states and produce near-optimal routing in one step.
- 2 STATE OF THE ART: Traditional routing optimization generally relies on analytical optimization or local-search heuristics.
- 2 STATE OF THE ART: Prior RL routing proposals use table-based agents that map state and action pairs to rewards during training.
- 2 STATE OF THE ART: DRL is proposed to generalize over unseen network states, unlike traditional table-based RL agents.
- 2 STATE OF THE ART: After training, the DRL agent is proposed to provide a near-optimal solution in one step instead of iterative optimization or heuristic improvement.
3 DEEP RL AGENT
The proposed agent uses an off-policy, actor-critic deterministic policy-gradient algorithm interacting with the network through state, action, and reward. It learns a policy that maximizes reward while minimizing network delay.
- 3 DEEP RL AGENT: The proposed RL agent is an off-policy, actor-critic, deterministic policy-gradient algorithm interacting with the network.Its interaction uses the signals state, action, and reward.
- 3 DEEP RL AGENT: The state is the traffic matrix, the action is a tuple of link weights, and the reward is based on mean network delay.
- 3 DEEP RL AGENT: The agent learns a behavior policy mapping states to actions that maximizes expected reward and minimizes network delay.Two deep neural networks, an actor and a critic, iteratively improve its knowledge of the relationships among state, action, and reward.
4 EXPERIMENTAL RESULTS
The evaluation uses a 14-node, 21-link network with 1,000 traffic configurations and compares DRL routing with 100,000 random valid configurations. The trained agent improves with training and consistently performs within the benchmark’s first quartile.
- 4 EXPERIMENTAL RESULTS: The evaluation uses a 14-node, 21-link scale-free topology and 1,000 traffic configurations spanning 10 traffic-intensity levels from 12.5% to 125%.
- 4 EXPERIMENTAL RESULTS: The benchmark contains 100,000 randomly generated routing configurations that are reachable and loop-free.The same configurations are used for each traffic-intensity and traffic-matrix combination.
- 4 EXPERIMENTAL RESULTS: The trained DRL agent produces a predicted routing solution for each of the 1,000 test traffic matrices in one step.Training lasted 100,000 steps and included stochastic action exploration to avoid local minima.
- 4 EXPERIMENTAL RESULTS: DRL performance increases with training time and, across all traffic intensities, its routing configurations are on average within the benchmark’s 1st quartile.Network delay is measured using the OMNeT++ discrete-event simulator.
5 DISCUSSION
The discussion identifies operational advantages of DRL routing optimization, including one-step configuration, model-free learning, and black-box automation. The authors conclude that a fully automated DRL agent tends to minimize network delay while noting plans to evaluate broader benchmarks and topologies.
- Conclusion: The fully automated DRL agent provides routing configurations that tend to minimize network delay.The authors plan to expand evaluation to more sophisticated benchmarks and routing topologies.
- One-step optimization: A trained DRL agent produces a near-optimal routing configuration in one step, supporting real-time network control.Traditional optimization requires many steps to produce a new configuration.
- Model-free: DRL agents learn state-action-reward dynamics from experience and handle nonlinear, complex, multidimensional systems without simplifications.Traditional optimization algorithms require analytical models based on assumptions and simplifications.
- Black-box optimization: DRL supports automatic black-box optimization, allowing different reward functions to implement different target policies without designing a new algorithm.Traditional mechanisms, particularly heuristics, are tailored to the problem being optimized.