PaperlayerView exact revision on arXiv

Source-linked AI summary

When Machine Learning Meets Congestion Control: A Survey and Comparison

Huiling Jiang, Qing Li, Yong Jiang, Gengbiao Shen, Richard Sinnott, Chen Tian, Mingwei Xu

arXiv:2010.11397v1cs.NI

TL;DR

Efficient generic congestion control is challenging across diverse network scenarios, and the paper reviews learning-based approaches and compares RL-based algorithms with traditional control. Its simulations report better performance for RL-based algorithms in some scenarios, while identifying deployment limitations and directions for future work.

  • Problem

    The paper addresses the challenge of designing efficient congestion control across diverse network scenarios and examines current learning-based approaches and their challenges.

    Source: §I · p. 1, 1) Loss Classification · p. 5

  • Method

    The paper reviews learning-based congestion control and simulates three RL algorithms against NewReno, alongside discussing challenges and future directions.

    Source: A. Compared Algorithms · p. 12, §IX · p. 20

  • Results

    The paper reports that RL-based congestion control performs better than traditional algorithms in scenarios including networks with high bandwidth and low delay.

    Source: §IX · p. 20

  • Takeaways & Limitations

    The paper identifies RL-based congestion control as a research trend and calls for lightweight approaches and an open platform for evaluating dynamic network scenarios.

    Source: B. Trends of Learning-based Congestion… · p. 20, §IX · p. 20

    For RL algorithms, computational complexity delays actions and rewards, affecting bandwidth utilization.

    Source: A. Challenges of Learning-based… · p. 19

Abstract

from arXiv · show

Machine learning (ML) has seen a significant surge and uptake across many diverse applications. The high flexibility, adaptability and computing capabilities it provides extends traditional approaches used in multiple fields including network operation and management. Numerous surveys have explored ML in the context of networking, such as traffic engineering, performance optimization and network security. Many ML approaches focus on clustering, classification, regression and reinforcement learning (RL). The innovation of this research and contribution of this paper lies in the detailed summary and comparison of learning-based congestion control (CC) approaches. Compared with traditional CC algorithms which are typically rule-based, capabilities to learn from historical experience are highly desirable. From the literature, it is observed that RL is a crucial trend among learning-based CC algorithms. In this paper, we explore the performance of RL-based CC algorithms and present current problems with RL-based CC algorithms. We outline challenges and trends related to learning-based CC algorithms.

I. INTRODUCTION

The paper examines learning-based congestion control as a response to increasingly diverse and dynamic networks, where designing efficient generic algorithms is difficult. It reviews traditional and learning-based approaches, with particular attention to data-driven methods and reinforcement learning.

A. Traditional Congestion Control

  • Traditional end-to-end congestion control infers congestion from implicit loss or delay signals, while network-assisted approaches use explicit network signals such as ECN.Loss-based methods may be misled by non-congestion packet drops, while delay measurement can be inaccurate; hybrid methods also cannot precisely identify network status from these signals.Source: A. Traditional Congestion Control · pp. 1–2

I. INTRODUCTION

  • Diverse scenarios and changing network dynamics make efficient generic congestion control difficult, motivating learning-based approaches that could serve different networks.Traditional approaches may perform well in one scenario without guaranteeing performance across diverse scenarios.Source: §I · p. 1, A. Traditional Congestion Control · p. 2

B. Learning-based Congestion Control

  • Learning-based methods make control decisions from real-time network states; this paper focuses on data-driven approaches, especially RL, rather than performance-oriented utility optimization.The paper notes that data-driven methods can adapt their parameters to current data, but may face convergence problems; RL is prominent because of its online learning capability.Source: B. Learning-based Congestion Control · p. 2
  • Learning-based congestion control remains immature: many schemes adjust CWND, leaving high-speed networks vulnerable to burstiness when multiple ACKs arrive, and most focus on end-to-end control.A general-purpose learning-based scheme for real network scenarios remains a major goal.Source: B. Learning-based Congestion Control · p. 2

C. Overall Analysis

The paper combines a review and comparison of learning-based congestion control with experiments using NS3, and discusses practical remedies and research needs. It also summarizes traditional congestion-control mechanisms and rate-adjustment strategies.

C. Overall Analysis

  • The study compares learning-based congestion control with traditional algorithms through experiments across dynamic network scenarios, using NS3 because real-stack learning decisions can be too slow.The authors propose lightweight state-action mapping tables, less frequent decisions, and asynchronous RL as potential ways to address computation time.Source: C. Overall Analysis · pp. 2–3

A. CC mechanisms

  • Traditional TCP congestion control uses slow start, congestion avoidance, retransmission, and fast recovery, while learning-based schemes can learn CWND strategies instead of following fixed rules.Slow start grows CWND rapidly; congestion avoidance increases it more slowly, and congestion detection reduces it.Source: A. CC mechanisms · pp. 3–4

B. Rate Adjustment Mechanisms of Congestion Control algorithms

  • Window-based strategies directly adjust CWND but can create burstiness when ACKs arrive together; rate-based strategies avoid burstiness but respond less quickly and can consume resources.Packet pacing aims to retain responsiveness while avoiding bursts, though it can perform worse in throughput and fairness in some scenarios, including at communication start.Source: B. Rate Adjustment Mechanisms of… · p. 4
  • Most learning-based congestion-control algorithms use window-based adjustment, while rate-based methods and pacing have also developed among traditional approaches.Source: B. Rate Adjustment Mechanisms of… · p. 4

C. Performance Metrics of Congestion Control algorithms

Congestion-control algorithms balance throughput, delay, packet loss, fairness, and responsiveness, and trade-offs between objectives can depend on the scenario. The paper also describes supervised learning as a way to predict congestion signals from network-state history.

C. Performance Metrics of Congestion Control algorithms

  • Congestion-control objectives include throughput, RTT and flow completion time, packet loss, fairness, and responsiveness, and improving some targets may require sacrificing others.The simulations measure throughput, RTT, and packet loss rate.Source: C. Performance Metrics of Congestion… · p. 4

III. SUPERVISED LEARNING-BASED CONGESTION CONTROL ALGORITHMS

  • Supervised learning maps inputs to outputs using a model trained on samples, and network applications use it to predict congestion signals or manage queue length.Congestion prediction uses current and previous states, such as queue length and network delay, on the premise that future states can be predicted from past states.Source: §III · p. 5

1) Loss Classification:

The section surveys supervised learning for classifying loss types and predicting delay, addressing congestion signals that traditional approaches may misread or handle poorly. It also identifies trade-offs in classification and a need for more complex delay-prediction methods.

1) Loss Classification:

  • Traditional wireless loss classifiers use packet inter-arrival time, ROTT, or both, but their suitability varies by topology and ZigZag is sensitive to sending rate.Biaz suits wireless last-hop topologies; Spike performs better in wireless backbones with multiple flows.Source: 1) Loss Classification · p. 5
  • Supervised learning uses multiple network parameters and diverse models to classify loss types, with simulations reporting high accuracy across different network scenarios.Methods include decision trees, ensembles, random forests, boosting, and multi-layer perceptrons.Source: 1) Loss Classification · pp. 5–6
  • Wireless, contention, and reordering losses can obscure congestion detection, making loss classification important for understanding congestion.Loss is an indirect signal, and it provides feedback only after congestion has occurred.Source: 1) Loss Classification · pp. 5–6
  • In wireless networks, misclassifying congestion loss as wireless loss can delay rate control, while the reverse can make algorithms overreact to wireless losses.Classifier parameters must balance performance across network scenarios.Source: 1) Loss Classification · p. 6
  • Boosting achieves higher accuracy than decision trees but consumes much more network resources; given limited accuracy gains, decision trees may offer an advantage.The comparison is reported in.Source: 1) Loss Classification · p. 6

2) Delay Prediction:

  • Supervised learning techniques are described as efficient at predicting future delays and reacting quickly to avoid congestion, while transmission delay reflects in-flight data and overall network load.The section identifies delay prediction as a way to address the limited flexibility of traditional delay-based CC in dynamic networks.Source: 2) Delay Prediction · p. 6, Figure 2 · p. 7
  • RTT prediction supports estimates such as RTO, while further work is needed to use more complex parameters and techniques to improve delay prediction.Research also estimates RTT from sending rate using linear regression or Bayesian techniques.Source: 2) Delay Prediction · pp. 6–7

B. Queue Length Management in Network-assisted Networks

The section surveys learning-based queue management and unsupervised loss and delay clustering, then describes RL’s use across changing network conditions. It also notes limits in the parameters considered and the computational demands of network-assisted control.

B. Queue Length Management in Network-assisted Networks

  • Queue-length prediction uses historical traffic, but these methods omit diverse network parameters that may relate to future queue length.The section also describes RL-based queue management using current state, queue length, or loading information to manage buffers and router decisions.Source: B. Queue Length Management in… · p. 7

IV. UNSUPERVISED LEARNING-BASED CONGESTION CONTROL ALGORITHMS

  • Unsupervised learning clusters unknown data by similarity, but is less widely used than supervised learning and mainly clusters loss and delay.It is also used to represent state spaces and aggregate data, but the paper says research based on this approach is limited.Source: §IV · p. 7, 1) Loss Clustering · p. 8

1) Loss Clustering:

  • Loss clustering uses loss-delay information or expectation maximization to distinguish wireless from congestion losses, or contention from congestion losses.The cited simulations report higher mean RTT for congestion losses, while wireless losses have lower mean and higher RTT variation.Source: 1) Loss Clustering · p. 8

2) Delay Prediction:

  • Delay clustering groups messages by message size, validity, distance, and type, then selects the lowest delay in each group to assign a sending rate.The paper notes that delay calculation has high processing demands and that clustering is less suited than supervised learning to dynamic, diverse environments.Source: 2) Delay Prediction · p. 8

V. RL-BASED CONGESTION CONTROL ALGORITHMS

  • RL monitors and responds to network conditions, making it more responsive to environmental changes than supervised and unsupervised approaches.The survey describes RL for updating CWND in end-to-end networks and managing queue length in network-assisted environments.Source: §V · p. 9

A. Window Updating in End-to-End Networks

The section surveys RL-based congestion control across wired, wireless, and more complex network scenarios. It describes high adaptability alongside convergence, memory, and computational constraints.

Wired networks

  • RL-based congestion control is applied to wired networks, including under-buffered, high-bandwidth bottlenecks, with reported balance between throughput and delay.The algorithm’s state includes packet inter-sending time and input ACK inter-arrival time.Source: A. Window Updating in End-to-End… · p. 10

Wireless networks

  • In ad hoc wireless networks, finite-action learning automata use limited information and computation, while continuous-action automata address discretization limits at higher resource cost.Q learning combined with a grey model is also described as using real-time throughput evaluation to adapt to dynamic environments.Source: A. Window Updating in End-to-End… · p. 10

Complex network scenarios

  • RL-based approaches also target time-varying flows, sparse rewards, and continuous, large state-action spaces.The survey cites actor-critic approaches for time-varying flows, RL in sparse-reward settings such as video games, and work on continuous, large state-action spaces.Source: A. Window Updating in End-to-End… · p. 10

Limitations

  • Across these scenarios, the survey identifies difficult convergence guarantees, challenging state abstraction, substantial memory needs, and high computational complexity.It concludes that realistic applications require further exploration of these engineering issues.Source: A. Window Updating in End-to-End… · p. 10

B. Queue Length Management in Network-assisted Networks

The paper describes RL queue management and an NS3 comparison of three RL algorithms with NewReno. It also outlines DQL’s capacity for large-scale problems and its convergence caveat.

B. Queue Length Management in Network-assisted Networks

  • RL queue management uses current state, queue length, or loading information to control buffers and optimize router decisions, but existing approaches cover limited state parameters.Compared with end-to-end window updating, network-assisted queue management can require more computation because multiple nodes, such as routers, may control congestion.Source: B. Queue Length Management in… · p. 12

VI. SIMULATION SETUP

  • The NS3 setup separates RL computation from data transmission, so RL computational complexity does not affect network communications in the simulation.The experiments compare RL-based and traditional congestion control algorithms in realistic network environments with large delay and complexity challenges.Source: §VI · p. 12

A. Compared Algorithms

  • The comparison selects DQL, DDPG, PPO, and NewReno, with NewReno serving as the traditional benchmark and NS3 default.The paper characterizes DQL as simplest and suited to relatively simple environments, while DDPG and PPO have stronger learning capabilities for more complex scenarios.Source: A. Compared Algorithms · p. 12

1) DQL-based Congestion Control Algorithms:

  • DQL uses neural networks to approximate value functions for large-scale problems, but its Q network is not guaranteed to converge and may produce a poorly trained model.TCP-Drinc combines a deep CNN and LSTM to learn from historical data and adjust CWND; the LSTM handles delay-related time-series autocorrelations.Source: 1) DQL-based Congestion Control… · pp. 12–13

2) DDPG-based Congestion Control Algorithms:

The section explains actor-critic and DDPG mechanisms for congestion control, including their convergence and training concerns. It also describes PPO’s bounded policy updates and reports Aurora’s favorable comparisons with traditional algorithms.

2) DDPG-based Congestion Control Algorithms:

  • Actor-critic combines policy and value estimation, but its interacting neural networks require gradient updates and can make convergence difficult.The section also notes that AC-based CC performance depends on the interaction of the two networks.Source: 2) DDPG-based Congestion Control… · pp. 13–14
  • Earlier actor-critic congestion-control work is reported for multimedia traffic and multipath CC, with favorable voice/video quality and traditional-algorithm comparisons, [85].The multipath approach is described as flexible for continuous action spaces.Source: 2) DDPG-based Congestion Control… · p. 14
  • DDPG uses experience replay and double actor and critic networks, and a satellite multi-path CC design using DDPG is reported as effective.DDPG outputs a deterministic policy and is described as better able than DQL to train models in complex environments, though training can be difficult and random environments pose problems.Source: 2) DDPG-based Congestion Control… · p. 14

3) PPO-based Congestion Control Algorithms:

  • Aurora uses PPO to generate stable policies for variable network conditions and is reported to outperform traditional CC algorithms in different contexts [95].PPO limits policy updates using the ratio of new to old policies and an adaptive penalty; training efficiency can remain a challenge.Source: 3) PPO-based Congestion Control… · p. 14

4) NewReno:

The section establishes NewReno as the traditional-CC benchmark, defines the performance metrics, and describes the simulated network scenarios and RL control choices.

4) NewReno:

  • NewReno modifies Reno’s fast recovery so multiple losses in one congestion event halve CWND only once; experiments use it as the traditional-CC representative.It has slow start, congestion avoidance, retransmission, and fast recovery.Source: 4) NewReno · p. 14

B. Performance Metrics

  • Performance is measured by throughput in Mbps, average RTT in seconds, and packet loss rate over a time interval.Throughput counts successfully transmitted data; RTT measures sender-to-receiver transfer time.Source: B. Performance Metrics · p. 14

1) Internet:

  • Three dumbbell-network scenarios vary bottleneck bandwidth and delay to represent high- and low-BDP conditions.Scenario I has high bandwidth and low delay; Scenario II has low bandwidth and high delay; Scenario III has low bandwidth and low delay.Source: 1) Internet · pp. 14–15

2) States:

  • The study’s states use throughput, RTT, and packet loss rate, while prior RL-based CC designs use varied congestion signals and control parameters.The literature offers no guaranteed rules for state design; states commonly combine congestion signals with CWND size or sending rate.Source: 2) States · p. 15

3) Actions:

  • The simulations adjust CWND using four actions: -1, 0, +1, and +3, following the action set in [80].Other cited designs use three or seven actions, including multiplicative CWND changes.Source: 3) Actions · p. 15

4) Rewards:

The section describes reward-function choices and the NS3 simulations used to compare RL-based controllers with NewReno. It also reports that computation delays ACK handling in realistic networks.

4) Rewards:

  • The study’s reward incorporates throughput, RTT, and packet loss rate, with bandwidth denoting bottleneck bandwidth and MinRTT the pipeline’s minimum RTT.The stated utility function is based on.Source: 4) Rewards · p. 15

VII. SIMULATIONS

  • The simulations compare NewReno with DQL, DDPG, and PPO on NS3 using throughput, RTT, and packet loss rate as states.The action adjusts CWND when a new ACK arrives; the network uses a dumbbell topology.Source: §VII · p. 15

A. Simulation Results

  • Overall simulation results are presented in Table XI and Figures 8-30, including timelines of performance changes, bars reporting performance averages and variances, and CDFs showing rough performance distributions.Source: A. Simulation Results · pp. 15, 17
  • In realistic network tests, RL controllers have much larger ACK intervals than NewReno, slowing CWND growth and affecting measured throughput and RTT.The paper attributes delayed ACKs to the time RL algorithms need to compute actions and cautions that they may not suit realistic networks.Source: A. Simulation Results · p. 17, Figure 12 · p. 17

1) CWND:

In NS3, RL-based CC performs similarly across the three algorithms and has advantages over NewReno in the high-BDP scenarios. Those gains come with higher RTT and minimal packet loss, while realistic deployment is constrained by computation time.

1) CWND:

  • RL controllers use larger CWNDs than NewReno in Scenarios I and II, but the four algorithms show little CWND difference in Scenario III.The three RL algorithms differ minimally across all three scenarios.Source: 1) CWND · p. 17

2) Throughput:

  • RL-based CC improves throughput over NewReno in Scenarios I and II, but shows no advantage in Scenario III.The paper links the expected improvement in the first two scenarios to higher average CWND.Source: 2) Throughput · p. 17

3) RTT:

  • RL-based CC has higher RTT than NewReno in all three scenarios, while RTT increases remain bounded relative to throughput increases in Scenarios I and II.The paper associates higher RTT with more aggressive CWND increases.Source: 3) RTT · p. 17

4) Packet Loss Rate:

  • NewReno’s packet loss rate is almost zero, while RL-based CC has minimal but increased packet loss across the scenarios.The paper describes the losses as bounded given the RL algorithms’ aggressiveness.Source: 4) Packet Loss Rate · p. 17

B. Analysis of Results

  • In simulated high-BDP networks, RL-based CC achieves high throughput with limited increases in RTT and packet loss; the three RL algorithms perform similarly.The experiments show advantages in high-BDP networks within the tested NS3 environments.Source: B. Analysis of Results · p. 17
  • NS3 separates computation from transmission, but real networks include RL action-computation time in ACK handling, limiting the algorithms’ realistic applicability.The paper says delayed ACKs make actual transmission-time measurement difficult and real throughput, RTT, and loss measurement almost impossible.Source: B. Analysis of Results · p. 17

C. Proposed Solutions

The section proposes three ways to address RL-based congestion-control delays: precomputed mapping tables, less frequent decisions, and asynchronous RL. Each presents a trade-off or implementation challenge.

C. Proposed Solutions

  • Precomputed mapping tables can replace online model computation, but may become large and unwieldy in continuous scenarios.The paper proposes generating the state-to-action table offline after training in network emulators.Source: C. Proposed Solutions · p. 19
  • Selecting CC algorithms at longer intervals can reduce the impact of RL computation delay, but slows updates and responsiveness.The paper identifies balancing these two performance concerns as future work.Source: C. Proposed Solutions · p. 19
  • Asynchronous RL can prevent ACKs from being blocked by the agent thread, but multiple actors and larger state spaces make training more resource-demanding.The paper cites multiple asynchronous-action approaches and identifies training difficulty as an open issue.Source: C. Proposed Solutions · p. 19

VIII. CHALLENGES AND TRENDS OF LEARNING-BASED CONGESTION CONTROL SCHEMES

Learning-based congestion control faces parameter, compute, memory, training, convergence, and compatibility challenges; the paper identifies engineering-focused evaluation, lightweight models, and realistic platforms as future directions. Its simulations report better performance for RL-based congestion control than traditional algorithms in different scenarios.

A. Challenges of Learning-based Congestion Control schemes

  • Learning-based congestion control is difficult to tune: RL state, action, reward, and hyperparameters affect performance, as do parameters in supervised and unsupervised methods.Reward designs differ, including throughput and RTT or packet loss rate and delay.Source: A. Challenges of Learning-based… · p. 19
  • High computational complexity can delay RL actions and rewards, affecting bandwidth utilization; complex learning methods can also carry high computational cost.Source: A. Challenges of Learning-based… · p. 19
  • RL training can demand substantial memory in continuous environments, while large state representations remain a limitation in complex scenarios.The paper points to state abstraction and representative data as ways to support efficient training.Source: A. Challenges of Learning-based… · p. 19
  • Learning-based CC may require extensive simulation training data and still face over- or under-fitting; convergence is not guaranteed in realistic networks, and compatibility with traditional CC remains unresolved.Source: A. Challenges of Learning-based… · p. 20

B. Trends of Learning-based Congestion Control Algorithms

  • Future RL research should address engineering issues that emulator-based simulations can overlook, including parameter selection and computational complexity.The paper identifies simulations in realistic network environments as a primary focus.Source: B. Trends of Learning-based Congestion… · p. 20
  • Lightweight learning-based CC is a future direction; domain knowledge and fewer optimally chosen parameters may help reduce model demands and improve deployability.Source: B. Trends of Learning-based Congestion… · p. 20
  • Further research needs an open platform with differentiated, realistic network scenarios; existing Pantheon coverage lacks professional settings such as flexible ad hoc wireless networks.Source: B. Trends of Learning-based Congestion… · p. 20

IX. CONCLUSION

  • RL-based congestion control performed better than traditional algorithms in different scenarios, including high-bandwidth, low-delay networks.The paper also identifies computation time, data storage, and pre-designed parameters as issues for realistic deployment.Source: §IX · p. 20
Loading 2010.11397v1…