PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Curriculum Generation for Learning Adaptation in Networking

Zhengxu Xia, Yajie Zhou, Francis Y. Yan, Junchen Jiang

arXiv:2202.05940v2cs.NI

TL;DR

RL networking policies can underperform traditional rule-based schemes and generalize poorly across environment distributions. Genet uses curriculum learning guided by the RL model’s gap to rule-based baselines, improving performance across adaptive video streaming, congestion control, and load balancing.

  • Problem

    Well-trained RL policies can perform worse than traditional rule-based schemes and often generalize poorly to different network environment distributions.

  • Method

    Genet automatically generates curricula by identifying environments where the current RL policy falls substantially behind a rule-based baseline, using Bayesian Optimization to search environment configurations.

  • Results

    Genet improves asymptotic performance by 8–25% for ABR, 14–24% for CC, and 15% for LB versus traditional RL training, while also improving performance in unseen environments.

  • Takeaways & Limitations

    Across three networking use cases, Genet improves RL policies in both performance and generalization across varied environments and workloads.

  • Takeaways & Limitations

    Genet’s performance worsens when training imposes a minimum fraction of uniformly random exploration from the original training distribution.

Abstract

from arXiv · show

As deep reinforcement learning (RL) showcases its strengths in networking and systems, its pitfalls also come to the public's attention--when trained to handle a wide range of network workloads and previously unseen deployment environments, RL policies often manifest suboptimal performance and poor generalizability. To tackle these problems, we present Genet, a new training framework for learning better RL-based network adaptation algorithms. Genet is built on the concept of curriculum learning, which has proved effective against similar issues in other domains where RL is extensively employed. At a high level, curriculum learning gradually presents more difficult environments to the training, rather than choosing them randomly, so that the current RL model can make meaningful progress in training. However, applying curriculum learning in networking is challenging because it remains unknown how to measure the "difficulty" of a network environment. Instead of relying on handcrafted heuristics to determine the environment's difficulty level, our insight is to utilize traditional rule-based (non-RL) baselines: If the current RL model performs significantly worse in a network environment than the baselines, then the model's potential to improve when further trained in this environment is substantial. Therefore, Genet automatically searches for the environments where the current model falls significantly behind a traditional baseline scheme and iteratively promotes these environments as the training progresses. Through evaluating Genet on three use cases--adaptive video streaming, congestion control, and load balancing, we show that Genet produces RL policies which outperform both regularly trained RL policies and traditional baselines in each context, not only under synthetic workloads but also in real environments.

1 INTRODUCTION

RL networking policies can struggle across broad training distributions and unseen environments, sometimes underperforming traditional baselines. Genet addresses this by curriculum learning that prioritizes environments where RL has the largest gap to a rule-based baseline.

  • 1 INTRODUCTION: Wide environment distributions can make RL policies perform poorly even on environments drawn from the training distribution, while unseen distributions can cause poor generalization.These issues can leave well-trained policies substantially behind traditional rule-based schemes in some settings.
  • 1 INTRODUCTION: Existing environment-difficulty heuristics rely on intrinsic properties or handcrafted rules that may not reflect how much the current RL model can improve.The relevant factors can differ across applications, such as queue length in congestion control or buffer length in adaptive streaming.
  • 1 INTRODUCTION: Genet defines rewarding environments as those where the current RL policy falls substantially behind a traditional rule-based baseline.A large gap-to-baseline is presented as indicative of improvement potential because the baseline supplies behavior the RL policy may learn to match.
  • 1 INTRODUCTION: Genet iteratively searches for high-gap environments and promotes them into training, using parameterized synthetic or trace-driven spaces and Bayesian Optimization for large searches.This avoids handcrafted difficulty heuristics and applies across adaptive video streaming, congestion control, and load balancing.
  • 1 INTRODUCTION: Genet does not guarantee performance against all rule-based baselines, optimization beyond training ranges, or adversarial robustness.The paper notes that a naive baseline can guide Genet toward policies inferior to stronger baselines.
  • 1 INTRODUCTION: Genet improves asymptotic performance over traditional RL training by 8–25% for ABR, 14–24% for CC, and 15% for LB, with gains also observed in unseen environments.The evaluation combines trace-driven simulation and real-world tests across the three use cases.

2 MOTIVATION

Networking RL faces poor asymptotic performance over wide environment ranges and weak generalization to unseen distributions. The motivation uses ABR, CC, and LB to show how these problems emerge in representative adaptation tasks.

  • 2 MOTIVATION: The three motivating use cases cover bitrate adaptation for video, transport-layer sending-rate adaptation, and request routing across servers.The study uses open-source implementations and focuses on improving existing RL training rather than redesigning the RL algorithms themselves.
  • 2 MOTIVATION: As environment ranges expand, RL’s performance advantage over rule-based baselines diminishes, and RL falls behind baselines in a substantial fraction of test environments.The comparison uses RL1, RL2, and RL3 synthetic ranges with increasing parameter widths across ABR, CC, and LB.
  • 2 MOTIVATION: RL training over wide distributions can converge slowly because randomly sampled batches may have dramatically different distributions and push the model in different directions.This instability makes it difficult to obtain a good policy.
  • 2 MOTIVATION: RL-based congestion control can validate against a baseline on its training range yet perform much worse on recorded cellular or Ethernet traces, with performance also degrading across different trace sets.These observations illustrate poor generalization when the test distribution differs from training.
  • 2 MOTIVATION: Traditional RL training can produce suboptimal asymptotic performance over wide environment ranges and poor generalization to unseen network environments.These two challenges motivate a curriculum-learning approach for networking.

3 CURRICULUM LEARNING FOR NETWORKING

Curriculum learning sequences network environments to improve RL training across broad target distributions, but its effectiveness depends on identifying rewarding environments. The section contrasts several environment-selection strategies and shows why simple difficulty proxies can misdirect training.

  • Curriculum learning: Curriculum learning varies the training distribution to introduce increasingly difficult environments that are more likely to improve the current RL model.It is associated with faster convergence, higher asymptotic performance, and better generalization in prior RL applications.
  • Why sequencing training environments is difficult: The central challenge is sequencing environments so training prioritizes cases where the current policy can achieve substantial reward improvement.The section frames this as determining which environments are rewarding rather than merely measuring their apparent difficulty.
  • Strawman approaches: Handcrafted environment properties can miss important differences because environments with similar measured characteristics may have different effects on training.The inherent-properties strawman distinguishes only hand-picked dimensions, which may be insufficient for complex environments.
  • Strawman approaches: Rule-based performance can distinguish environments but does not indicate how training should improve the current RL model, while optimum-gap selection can be prohibitively expensive.These limitations motivate comparing alternatives beyond generic difficulty and optimal-performance gaps.
  • Strawman approaches: Training on trace set X yields only marginal improvement on X and harms Y, whereas adding Y improves both sets, illustrating that the largest apparent performance gap need not identify the best training environment.The example contrasts a suboptimal choice based on optimum gap with the broader benefit of selecting Y.

4 DESIGN AND IMPLEMENTATION OF GENET

Genet generates curricula by finding environments where an RL policy lags behind a rule-based baseline, then iteratively promoting those environments during training. Its workflow combines fixed-round RL updates, Bayesian Optimization over parameterized environments, and continued sampling from the original distribution, while acknowledging baseline, range, trace, and forgetting limitations.

  • 4.1 Curriculum generation: Genet identifies environments with large gaps between the current RL policy and a rule-based baseline as rewarding training targets.This gap indicates plausible room for improvement and may let the RL model learn rules already demonstrated by the baseline.
  • 4.1 Curriculum generation: The gap-to-baseline is more indicative of training improvement than the gap-to-optimum, although improvement also depends on unchanged training hyperparameters.The relationship is supported by examples involving ABR and CC policies, including 73 randomly chosen synthetic CC configurations.
  • 4.1 Curriculum generation: Rule-based baselines provide a practical reference because operators often scrutinize RL performance regressions against traditionally deployed schemes.Promoting environments with large gaps directly reduces the possibility that the RL policy causes performance regressions.
  • 4.2 Training framework: Each Genet iteration updates the RL model, selects large-gap environments, and promotes them into the next training distribution.Figure 7 presents this iterative workflow; new configurations are sampled with 30% default probability while 70% of samples come uniformly from the old distribution.
  • 4.2 Training framework: The framework can augment synthetic training with trace-driven environments, and trace augmentation improves performance especially on unseen real traces from the same distribution.Even without trace-driven environments, Genet-trained policies outperform traditional RL training over real or synthetic traces.
  • 4.2 Training framework: Genet parameterizes environments and uses Bayesian Optimization to search high-dimensional configuration spaces for configurations maximizing expected gap-to-baseline.The search evaluates average rewards over randomly generated environments for each configuration and restarts when the RL model changes because rewarding environments can change.
  • 4.2 Training framework: Genet does not guarantee superiority to every rule-based baseline or performance outside the training range, and it may face forgetting as promoted environments dilute original-distribution sampling.The original distribution still accounts for about 10% after nine distribution changes, and the framework reports only mild effects from forgetting.

5 EVALUATION

Genet is evaluated across three networking use cases and synthetic, trace-driven, and real-world environments. It improves over traditional RL and rule-based baselines, while its benefits depend on training coverage and curriculum-search choices.

  • 5 EVALUATION: Adding a minimum fraction of uniformly random exploration worsens Genet’s performance, indicating that its curriculum depends on targeted environment selection.
  • 5.2 Asymptotic performance: Across three use cases, Genet improves over traditional RL by 8–25% for ABR, 14–24% for CC, and 15% for LB.
  • 5.2 Asymptotic performance: Genet improves rewards across most tested ABR parameter settings rather than gaining in some environments while degrading in others, and its LB policy beats baselines by 15% in simulation.
  • 5.3 Generalization: Genet generalizes from synthetic training to trace-driven environments, where its policies outperform traditional RL baselines, including when real traces are unavailable during training.
  • 5.4 Comparison with rule-based baselines: Using a naive baseline can prevent useful environment selection, but the impact is limited to selection because RL training remains decoupled from the baseline.
  • 5.4 Comparison with rule-based baselines: Genet-trained ABR and CC policies remain on the performance frontier and outperform other rule-based baselines, while increasing the fraction of real traces where RL beats the selected baseline.
  • 5.3 Generalization: In five real wide-area paths, Genet outperforms baselines in all but two cases; failures occur with bandwidth far above the bitrate range and queues deeper than seen in training.
  • 5.5 Design choices: Genet’s curriculum ramps up faster than alternatives, outperforms Robustifying-based variants, and Bayesian Optimization finds large gap-to-baseline environments more efficiently than random exploration.

6 RELATED WORK

Related work spans curriculum learning, adversarial environment generation, safe policy improvement, and generalization techniques for RL. Genet differs by using rule-based performance gaps to sequence networking environments for curriculum training.

  • Improving RL for networking: Unlike adversarial approaches that generate challenging traces, Genet uses Bayesian Optimization to search environment configurations with large gaps between RL and rule-based baselines.
  • Improving RL for networking: Genet differs from related deployment and generalization methods by introducing curriculum learning across multiple networking applications.
  • Curriculum learning for RL: Curriculum-learning research studies environment generation, sequencing, and transfer, but optimal sequencing remains open when networking tasks lack a clear difficulty definition.
  • Curriculum learning for RL: Genet applies curriculum learning systematically to networking, sequencing environments where the current RL policy has substantial room to improve over rule-based schemes.
  • Curriculum learning for RL: Prior methods create challenging environments using maximal regret, competitive baselines, or regret proxies; Genet proposes a networking-specific rule-based criterion instead.
  • Improving RL for networking: Safe policy improvement also uses rule-based schemes, but it uses them to avoid failures during training rather than to identify where RL can improve most.

7 DISCUSSION

Genet’s environment-selection heuristic has important limitations: gaps to rule-based baselines do not always measure improvement potential or achievable gains, and the framework assumes suitable baselines exist.

  • 7 DISCUSSION: A small gap to a rule-based baseline can conceal substantial improvement potential when the baseline itself performs poorly in an environment.This can occur when a baseline misinterprets random packet loss as congestion; stronger baselines or ensembles can mitigate the issue.
  • 7 DISCUSSION: A large gap to a rule-based baseline does not always yield large RL improvement when the model cannot match the baseline’s decision granularity.Aurora may react more slowly than Cubic or BBR during sudden bandwidth changes because it updates sending rates only after monitor intervals accumulate acknowledgments.
  • 7 DISCUSSION: Genet currently requires a rule-based baseline, but it can fall back to traditional RL training, optimal-solution gaps, or trained RL models used as baselines.The optimal-solution approach is most suitable when ground-truth-informed solutions are feasible and computationally cheap.

8 CONCLUSION

The paper presents Genet as a training framework for deep RL-based network adaptation and reports improved policy performance and generalization across varied environments and workloads.

  • 8 CONCLUSION: Genet improves RL policies in both performance and generalization across various environments and workloads, according to evaluation on three networking use cases.The conclusion identifies adaptive video streaming, congestion control, and load balancing as the evaluated use cases.
  • 8 CONCLUSION: The work introduces curriculum learning to networking and selects rewarding environments where current RL performance falls significantly behind a rule-based baseline.The framework is intended to improve training of deep RL-based network adaptation algorithms.
  • 8 CONCLUSION: The paper states that it does not raise ethical issues.

A APPENDICES

The appendices contain supporting material that has not been peer-reviewed.

  • A APPENDICES: The appendices provide supporting material for the paper.
  • A APPENDICES: The appendix material has not been peer-reviewed.
  • A APPENDICES: Readers should distinguish the appendices from the peer-reviewed main paper material.

A.1 Details of RL implementation

Traditional RL training samples configurations and environments randomly, rolls out the current policy, and updates its parameters iteratively with policy-gradient optimization.

  • A.1 Details of RL implementation: Each iteration uniformly samples K configurations from the configuration space and creates N random environments for each configuration.The configuration space is defined by ranges with minimum and maximum values for each environment dimension.
  • A.1 Details of RL implementation: Policy parameters are updated after each iteration using a gradient step with learning rate ν, and the final parameters are returned after N_iters iterations.
  • A.1 Details of RL implementation: The training procedure rolls out policy π_θ on each simulated environment and aggregates the resulting trajectories for updating.

A.2 Trace generator logic

The appendix describes synthetic workload generators for ABR, congestion control, and load balancing, alongside Genet’s iterative environment-search and training procedure.

  • Trace generators: ABR traces vary minimum bandwidth, maximum bandwidth, bandwidth-changing interval, and duration, with timestamped throughput and uniform noise.Throughput is sampled between the bandwidth bounds, while the changing interval controls temporal variation.
  • Genet training framework: Genet initializes a configuration distribution, uses Bayesian optimization to select candidate environments, and promotes selected configurations into the curriculum over exploration iterations.The curriculum mixes the newly selected configuration with existing configurations before further policy training.
  • Genet training framework: Genet evaluates each candidate by averaging the rule-based baseline reward minus the current RL policy reward across simulated rollouts.The resulting mean baseline gap is returned to Bayesian optimization as the candidate’s score.
  • Trace generators: The congestion-control generator varies bandwidth, change interval, latency, queue size, random loss, delay noise, and duration, producing dynamic bandwidth series at 0.1-second steps.Bandwidth values are sampled uniformly from 1 Mbps to the configured maximum.
  • Trace generators: The load-balancing simulator models Poisson job arrivals, Pareto-distributed job sizes, and identical processing rates across servers.The supplied table identifies the simulator’s configurable parameter ranges.

A.3 Details of Figure 4

Figure 4’s motivation traces use two bandwidth configurations to illustrate testing-reward trends across controlled network conditions.

  • Trace configurations: Trace set X uses 0–5 Mbps bandwidth with changes every 0–2 seconds, whereas trace set Y uses 0–10 Mbps with changes every 4–15 seconds.Each trace set contains 20 traces for showing the testing reward trend.

A.4 Testbed setup

The evaluation combines simulation, emulation, and real wide-area tests for ABR, congestion control, and load balancing, with rewards defined from application-specific performance metrics.

  • ABR testbed: The ABR testbed uses dash.js with MPC, BBA, and RL-based algorithms, while Mahimahi emulates FCC, cellular, and Puffer traces with an 80 ms RTT.The setup uses Chrome, Apache, and the Envivio-Dash3 video on UChicago servers.
  • Congestion-control testbed: The congestion-control testbed uses Pantheon and Mahimahi for local emulation and deploys remote experiments across the paths shown in Figure 21.Baselines include Linux-kernel BBR and TCP Cubic, plus PCC-Aurora and PCC-Vivace.
  • Real-world evaluation: Genet-trained policies are evaluated on real wide-area network paths in addition to simulated and emulated environments.The real-network experiments use four reserved nodes, a home laptop, and two cloud servers.
  • ABR evaluation: In ABR, Genet tends to reduce rebuffering and smooth bitrate selection without significantly sacrificing average bitrate.The reward combines bitrate, rebuffering time, and bitrate change over video chunks.
  • Congestion-control evaluation: In congestion control, Genet tends to lower 90th-percentile latency and packet loss without reducing throughput.The reward combines throughput, average latency, and packet loss over monitor intervals.
  • Training comparison: Doubling training iterations did not help regularly trained RL and curriculum-learning baselines as much as it helped Genet, according to the Figure 22 learning curves.This comparison tests whether Genet’s advantage can be explained by simply training competing methods longer.
Loading 2202.05940v2…