Source-linked AI summary
ReForge: Keeping ABR Algorithms Never Finished with Verified Large Language Model Edits
Zhiqiang He, Zhi Liu
TL;DR
ABR designs can overfit observed network conditions and degrade as new scenarios arrive. ReForge continually edits a rule-based policy router and accepts changes only after replaying them on previously served networks, raising mean QoE from 1.23 to 1.74, past the best single policy at 1.66 and to 94% of an oracle.
Problem
ABR designs overfit observed network scenarios, while existing generalization and continual-learning approaches do not eliminate performance loss or forgetting on later scenarios.
Method
ReForge uses an LLM to make small edits to a fuzzy-rule router, replaying each candidate across all previously served networks before accepting or reverting it.
Results
Mean QoE rises from 1.23 to 1.74 across nine held-out network families, surpassing the best single policy at 1.66 and reaching 94% of an oracle.
Takeaways & Limitations
Continual rule-based redesign can improve routing across arriving network families while preserving performance on previously served families.
Takeaways & Limitations
ReForge cannot improve a family when its frozen policy pool lacks a competent policy, as illustrated by Norway3G.
Abstract
from arXiv · showhide
Designing an ABR algorithm for one network scenario takes an engineer months, and large language models now do this work in hours, matching or beating hand-built designs. But either way, the design fits only the world visible at its birth, and fails on the world that arrives after. We ask whether an ABR algorithm can keep pace with the world, redesigned in minutes as each scenario arrives, with every change proven harmless to every scenario already served. In this work, we propose ReForge, a continual heuristic learning framework that adapts to continuously changing scenarios. ReForge runs that routine with a large language model (LLM) in the loop. Each round the LLM reads where the current design falls short and proposes one small edit, and a replay over every network served so far decides. Specifically, what it edits is a single page of fuzzy rules that routes every decision to one of a frozen pool of pre-trained policies. The LLM writes the first page from measurements alone, then keeps improving it on its own. Each round it reads where the current rules fall short and proposes one small edit, and a replay over every network served so far decides whether the edit lands. We evaluate ReForge on nine real-world network families arriving one at a time as 3G, 4G, then 5G. A few edits per arrival lift mean QoE from 1.23 to 1.74, past the best single policy at 1.66 and to 94\% of an oracle, and even repair families the loop never saw, one rising from 0.30 to 0.80. All code, data, and experiment records will be open-sourced upon cleanup.
1 Introduction
ReForge continually adapts ABR algorithms as new network scenarios arrive, using an LLM to edit a cheap fuzzy-rule router over frozen pre-trained policies. Replay validates each edit against all previously served networks, raising mean QoE from 1.23 to 1.74 across nine families.
- Motivation: ABR designs overfit observed cases and fail when later network scenarios arrive.Generalization degrades beyond its training distribution, while continual learning updates shared weights without tests showing what changed.
- ReForge design: ReForge separates behavior into five frozen pre-trained policies and routing into a one-page fuzzy IF–THEN rule set.The router uses throughput and buffer signals and meets the policy pool at every video chunk.
- Results: 1.23 to 1.74 mean QoE across nine held-out families exceeds the best single policy at 1.66 and reaches 94% of an oracle.Families arrive sequentially as 3G, then 4G, then 5G.
- Results: Three accepted edits capture nearly all improvement when an arriving family clearly favors one policy, and every edit identifies its motivating evidence.The gains are described as easy to audit.
- ReForge loop: Each round, an LLM proposes a small router edit, while replay over every network served so far accepts or undoes the change.The loop detects when the algorithm must change and validates changes against all previously served networks.
2 Motivation
ReForge is motivated by a timing problem: a router written from known networks can excel there yet fail when unseen families arrive. The design therefore favors small, reviewable rule edits that can be tested against every served network and reverted safely.
- Policy selection: A single deployed policy must face every network, exposing the cost of choosing one algorithm across nine network families.The comparison includes RobustMPC, FDASH, Pensieve, Merina, and PA-MoE, with learned policies trained on 3G traces and frozen at their best checkpoints.
- LLM-written router: The LLM wrote eight IF–THEN rules from in-pool policy scores and bandwidth statistics, replacing days of training with minutes of readable design.The rules use throughput mean, throughput variability, buffer level, and buffer drift.
- Unseen networks: The router favored PA-MoE because it was the in-pool winner, but that choice became poor when unseen families arrived, where PA-MoE was nearly the worst.The failure was attributed to timing: the router knew only the in-pool world available when it was written.
- Why retraining fails: Retraining weights lacks explanation, advance testing, and safe reversion because every weight changes and evaluating all past networks costs about as much as training.Its damage is discovered live and can only be reduced, never eliminated.
- Verified edits: A one-line rule edit provides explanation, replay testing across served families, and reversion by deletion, while keeping the underlying policies frozen.The rule page records uncovered decisions, and a small file can be rescored before deployment.
3 ReForge
ReForge has an initial stage that uses an LLM to write a one-page fuzzy-rule router over a frozen policy pool, followed by continual one-edit revisions as new network families arrive. Each revision is mechanically validated and replayed across the new and previously served families, preserving past performance under an acceptance gate.
- Stage 1: Initial router: The initial LLM-generated page maps recent-observation features to one policy in a frozen pool using fuzzy bands and IF–THEN rules.The prompt supplies policy QoE, rebuffering, smoothness, and trace bandwidth measurements from in-pool training splits.
- Stage 1: Initial router: The Stage-1 router uses four signals, eight rules, and an MPC fence to manage policy choices across network conditions.Its signals are windowed throughput mean, throughput variability, buffer occupancy, and per-chunk buffer drift; MPC was measured as risky across steady and choppy families.
- Stage 2: Continual revision: Each arriving family contributes only a training probe, while its test traces remain outside the loop; each arrival opens sixteen optimization rounds.Twenty simulated episodes score one candidate on one family, and reported numbers come from test parts alone.
- Stage 2: Continual revision: The LLM proposes one validated edit per round, using current-page scores, per-policy trials, and coverage reports to identify alternative choices and unreachable states.Edits are restricted to a closed language, mechanically accepted or refused, and typically change only one or two lines of the one-page rule file.
- Stage 2: Acceptance gate: δ = 0.05 protects every prior probe family, while ε = 10^-3 requires improvement in the mean or worst probe score before an edit is accepted.The candidate page is replayed on the new family and all older probe families against their best accepted scores.
4 Evaluation
Across nine network families arriving from 3G through 5G, ReForge improved mean QoE through small verified routing edits while preserving previously served families. The final router reached 1.738 mean QoE, ahead of every fixed policy and 94% of an oracle’s 1.854.
- Overall performance: 1.738 mean QoE after all three phases exceeded every fixed policy, approaching the oracle’s 1.854 with 94% recovery.The Stage-1 router scored 1.227 versus RobustMPC’s 1.661 across nine families.
- Overall performance: 18 accepted edits from 48 proposals changed routing among the same eight rules and five frozen policies, producing the final gains.The final page used four policies instead of two and had the fewest bad playbacks while maintaining high bitrate without more stalls.
- Continual adaptation: 3G test QoE barely changed from 1.523 to 1.518, while Lumos4G gained +0.88 and Lumos5G gained +2.73 without any served family losing more than δ on probes.Five rounds were rejected because they would have harmed already served families.
- Design trajectory: Three Phase-2 edits delivered the entire Lumos4G gain, with High-band routing to MPC after MPC scored 2.71 versus PA-MoE’s 1.79.Moving the default to MPC addressed 6.4% of Lumos4G chunks that previously fired no rule, and the next 13 rounds added nothing.
- Zero-shot transfer and limitations: Lumos5G rose from 1.02 to 3.71 through an unseen-family transfer caused by changing the default from PA-MoE to MPC.The edit affected 52.5% of 5G chunks that fired no rule; adding a coverage report then enabled accepted edits after eight earlier failures.
5 Conclusion and Discussion
ReForge makes ABR design continual: an LLM rewrites a routing-rule page as network families arrive, accepting each small edit only after replay against previously served families. The approach is limited by the frozen policy pool’s competence and by the cost of rerunning past families for every candidate edit.
- Contribution: ReForge turns ABR routing into a continually edited page of rules, with an LLM making one small change per arriving network family.Each candidate edit is replayed against every family already served before acceptance.
- Limitations: On Norway3G, no policy in the pool can serve the family, so ReForge finds no worthwhile edit and the score remains unchanged.This failure indicates that a new policy must be trained rather than another routing rule written.
- Limitations: The verification guarantee requires rerunning every past family on every candidate, remaining affordable only while the artifact is a page and episodes are cheap.Thus, scaling the approach is constrained by replay cost.
A The Stage-1 Router
Stage 1 produces an eight-rule router over five-chunk signal windows, using trapezoidal membership functions to route decisions among the frozen policy pool. Seven rules end in PA-MoE, while metadata defines MPC eligibility, buffer and drift fences, stickiness, and the default.
- Router design: The Stage-1 router uses a five-chunk signal window and trapezoidal membership functions, whose parameters define rising, flat, and falling regions.A trapezoid [a,b,c,d] rises from a to b, holds through c, and falls to d.
- Router design: Seven of the eight rules end in PA-MoE, identified as the in-pool winner.The complete Stage-1 page contains eight rules.
- Router constraints: The router’s metadata permits MPC only when variability is at most 0.35, requires at least 10 s of buffer, and allows drift no worse than −3 s per chunk.These conditions form the metadata fence for MPC selection.
- Router constraints: Stickiness retains the incumbent unless a challenger leads by 0.15 for 4 chunks, while warm-up and no-rule cases default to PA-MoE.The table metadata specifies both the switching rule and fallback behavior.
B The Stage-1 Prompt
The Stage-1 prompt asks an LLM to infer a fuzzy-rule router solely from measurements, selecting among frozen ABR experts rather than choosing bitrates directly. It specifies the router’s feature language, runtime aggregation, safety constraints, JSON interface, and expert-performance evidence for constructing the initial controller.
- B The Stage-1 Prompt: The controller declares features, fuzzy memberships, expert-selection rules, safety fences, stickiness, warmup behavior, and explanatory notes in one JSON object.The system prompt fixes the interface, vocabulary, safety fence, and output format, while the user prompt supplies measurements and profiles.
- B.1 System prompt: The LLM infers both the observed feature view and routing policy from measurements, without receiving a pre-existing heuristic or imposed feature set.Inputs include expert scores, bandwidth statistics, and observed candidate-quantity ranges.
- B.1 System prompt: The router delegates every decision to a frozen expert, executing that expert’s bitrate choice rather than selecting a bitrate itself.This delegation model defines the controller’s action space.
- B.1 System prompt: Each chunk, trapezoidal membership degrees feed product-T-norm rule firing, max-S-norm expert aggregation, safety filtering, and highest-weight selection with stickiness.If no rule fires, the router uses meta.warmup.default.
- B.1 System prompt: At most 24 rules are allowed, labels must cover each membership feature’s observed range without gaps, and features should be justified by measurements.Forward references and cycles are prohibited, and warmup.default handles initial chunks before windows fill.
- B.2 User prompt: pamoe is best in FCC-16-Train at QoE=+0.909, FCC-18-Train is best with mpc at QoE=+2.949, and pamoe leads Oboe-Train at QoE=+2.164.The remaining listed families also favor pamoe: Puffer-21-Train at +1.100 and Puffer-22-Train at +0.658.
- B.3 Model output: pamoe is the per-scenario best expert in 4 of 5 families, while merina is a consistent runner-up and fdash has the lowest stall time in 4 of 5 families.fdash records 0.00s on Oboe and 0.02s on FCC-16 but pays for that advantage elsewhere.
C The Stage-2 Continual Round Prompt
ReForge’s Stage-2 continual round prompt has the LLM propose one small, evidence-driven fuzzy-rule edit while preserving previously passing traces. The prompt exposes current rules, probe results, replay evidence, and a regression bank, with a frozen expert pool and explicit safeguards for coverage and rule strength.
- C The Stage-2 Continual Round Prompt: Across all 48 rounds, the system prompt remains fixed while the user prompt is rebuilt from the current page, probe scores, and evidence channels.The illustrated phase 2, round 3 output changed the no-rule default from PA-MoE to MPC.
- C.1 System prompt: The maintainer proposes a single small edit that improves worst-case trace performance without breaking previously passing traces.The rule base routes per-chunk decisions among several pre-trained ABR experts.
- C.1 System prompt: The prompt supplies the current rule base, per-trace QoE, fixed-expert trials, worst-trace replay, regression bank, and failed-direction ledger.These evidence channels constrain edits against both current weaknesses and historical regressions.
- C.1 System prompt: The expert pool is frozen: the LLM selects among fdash, pamoe, pensieve, mpc, and merina rather than modifying their internals.The prompt describes these as fuzzy-control, mixture-of-experts, deep-RL, model-predictive-control, and meta-RL policies.
- C.1 System prompt: Coverage must be fixed before rule edits when probes fire no rule or observe out-of-range features; occupied labels may then be split only when both halves remain occupied.The prompt states that splitting an occupied label clones existing behavior, while empty or out-of-range splits buy nothing.
- C.1 System prompt: Rule strength is the product of antecedent memberships, and each expert takes the maximum over its rules, so narrower antecedents are always weaker.A new rule cannot outrank a broader rule with a different consequent in an already covered region.
- C.3 Model output: The model output sets the warmup default to mpc because Lumos has 6.4% no-rule chunks where mpc performs best.This is the single JSON edit shown for phase 2, round 3.
- C.4 The final rule base: The final rule base’s measurement-derived shape makes pamoe the per-scenario best expert in 4 of 5 families, while merina is a dominated runner-up and pensieve is dominated everywhere.The same summary says fdash has the lowest stall time in 4 of 5 families but incurs a tradeoff.