PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Based Hierarchical Coordinated Control with Continuation-Aware Policy Learning

Changhong He, Jinda Gao, Xinkuan Liu, Le Zhang, Xizi Luo, Yu Mei

arXiv:2608.15041v1cs.AI

TL;DR

Coordinating interacting engineering units is difficult when strategies depend on heterogeneous context and system interactions are hard to model. This paper pairs an LLM coordinator with task-specific constrained controllers and trains it using Continuation-Aware GRPO. Across traffic and VPP tasks, the method outperforms baselines and remains effective without adaptation in simulators with different dynamics.

  • Problem

    System-level coordination often relies on fixed parameters or manually designed rules that are difficult to construct when strategies depend on diverse contextual signals and interactions.

    Source: Introduction · p. 1

  • Method

    An LLM interprets heterogeneous operational context to coordinate units, task-specific controllers or optimizers execute constrained actions, and Continuation-Aware GRPO evaluates subsequent system evolution.

    Source: Introduction · pp. 1–2

  • Results

    Across traffic and VPP tasks, the method outperformed baselines on unseen-pattern and higher-uncertainty scenarios and transferred without adaptation; overall traffic throughput was 16625.4 ± 84.5.

    Source: Main Results · p. 6, Conclusion · p. 7

  • Takeaways & Limitations

    Policies trained on simplified models remained effective without adaptation in more realistic simulators with different dynamics and modeling assumptions.

    Source: Introduction · p. 2, Conclusion · p. 7

    Low-level execution is constrained by the operational constraints represented in the control model.

    Source: Problem Formulation · p. 3

Abstract

from arXiv · show

Coordinating multiple interacting units in complex engineering systems is challenging when system interactions are difficult to model, operational information is heterogeneous, and low-level actions must satisfy strict constraints. We propose an LLM-based hierarchical framework in which the LLM coordinates interacting units based on heterogeneous operational context, while task-specific controllers or optimizers generate executable and constraint-aware actions. We further introduce Continuation-Aware GRPO to capture the consequences of coordination decisions over subsequent control intervals. Rather than judging a decision only by its immediate outcome, the method also evaluates how the system evolves afterward under the current policy. We validate the framework on multi-ramp traffic control and virtual power plant (VPP) energy management, using simplified system models for training and more realistic simulators for evaluation. Across both tasks, the proposed method consistently outperforms direct task-specific control and optimization, end-to-end reinforcement learning, rule-based and RL-based hierarchical coordination, and prompting-only LLM coordinators, demonstrating the value of heterogeneous-context reasoning, hierarchical execution, and continuation-aware policy learning.

Introduction

The paper proposes an LLM-based hierarchical controller that uses operational context for system-level coordination and delegates constraint-aware execution to task-specific controllers or optimizers. It combines this design with Continuation-Aware GRPO and reports consistent improvements across two engineering systems.

Introduction

  • Fixed coordination rules struggle when strategies depend on evolving trends, interactions, uncertainty, urgency, priorities, and events, while direct low-level learning faces scaling, feasibility, and generalization challenges.Prompting alone does not align language-level decisions with closed-loop performance, and direct generation of physical actions offers limited feasibility and reliability guarantees.Source: Introduction · p. 1
  • An LLM interprets operating context to coordinate interacting units, while task-specific controllers or optimizers turn its high-level decisions into executable, constraint-aware actions.Coordination can specify which units act more aggressively or conservatively and which objectives receive priority.Source: Introduction · p. 1
  • The coordination policy jointly interprets heterogeneous signals and learns from closed-loop experience without requiring a complete explicit model of unit interactions or external disturbances.Inputs include measurements, recent trends, uncertainty, resource urgency, priorities, events, and system rules; domain-specific controllers retain precise execution.Source: Introduction · p. 1
  • Continuation-Aware GRPO evaluates each decision over its execution interval and a longer continuation under a frozen copy of the current policy, capturing downstream effects without an added value model.The accumulated return is used with standard GRPO, addressing short-horizon bias toward immediate benefits that may have adverse later consequences.Source: Introduction · pp. 1–2
  • The policy is trained on simplified system models and evaluated in more realistic environments with different dynamics and modeling assumptions to test for reusable coordination strategies.The two applications are multi-ramp traffic control and virtual power plant energy management, with different dynamics, information structures, constraints, and execution mechanisms.Source: Introduction · p. 2
  • The framework consistently improves over direct-control and hierarchical-coordination baselines across diverse operating patterns and reduces performance degradation when simulator dynamics change.It is evaluated in multi-ramp traffic control and virtual power plant energy management.Source: Introduction · p. 2

Related Work

Prior control methods trade off efficient fixed feedback, model-dependent centralized optimization, and task-specific hierarchical execution; LLM control adds flexible coordination but retains application-specific interfaces. Existing LLM fine-tuning addresses static outputs or sequential interaction, but does not directly evaluate control decisions through longer closed-loop continuations.

Traditional and Learning-Based Control

  • Feedback control is efficient but often fixed, centralized optimization depends on accurate models and forecasts, and hierarchical methods retain task- and model-specific interfaces (Papageorgiou and Kotsialos 2002; Hegyi, De Schutter, and Hellendoorn 2005; Naval and Yusta 2021; Belletti et al. 2018; Liu et al. 2023; Airaldi, Schutter, and Dabiri 2025; Li, Chang, and Yang 2025).Direct low-level reinforcement learning must jointly learn coordination, numerical actions, and constraint satisfaction.Source: Traditional and Learning-Based Control · p. 2

LLM-Based Decision Making and Control

  • LLM engineering controllers select actions, supervise, or interface with numerical methods, but define outputs and execution interfaces separately for each application (Lai et al. 2025; Yuan, Lai, and Liu 2026; Sha et al. 2023; Jin et al. 2024; Wu, Ai, and Li 2025; Yu et al. 2026).Examples include traffic signal-phase selection, coupling to MPC, reinforcement learning or optimization, and language-based proposal exchange before low-level RL execution.Source: LLM-Based Decision Making and Control · p. 2

Reinforcement Fine-Tuning of LLMs

Problem Formulation

The formulation models an engineering system as interacting control units and separates high-level coordination from physical execution. The high-level policy uses operational context to select coordination decisions, while low-level controllers or optimizers produce constraint-aware actions for system-level performance.

Problem Formulation

  • The system comprises N interacting control units whose observable state evolves under exogenous disturbances, with executable actions subject to operational constraints.Disturbances include time-varying demand, renewable generation, weather, or market conditions.Source: Problem Formulation · pp. 2–3
  • The decision context combines recent state history with forecasts, uncertainty estimates, resource urgency, operational priorities, external events, and system rules.Source: Problem Formulation · p. 3
  • An LLM-based high-level policy produces a structured joint coordination decision for the units, which task-specific low-level controllers or optimizers convert into executable actions.Source: Problem Formulation · p. 3
  • Learning optimizes the high-level policy for system-level trajectory performance while low-level control executes actions subject to g(s_t, a_t) ≤ 0.The trajectory distribution depends on the high-level policy, low-level controller, system dynamics, and exogenous disturbances; performance uses discount factor γ ∈ (0, 1].Source: Problem Formulation · p. 3

Method

The method pairs an LLM coordinator that interprets heterogeneous operational context with low-level controllers that produce executable, constraint-aware actions. Continuation-Aware GRPO evaluates each initial coordination decision through its subsequent closed-loop effects.

Operational-Context Representation and Hierarchical Execution

  • The LLM turns heterogeneous operational context into structured joint decisions, while task-specific low-level controllers map them to executable actions and explicitly handle physical and operational constraints.Context descriptions include system condition, recent trends, uncertainty, urgency, priorities, external events, and applicable rules; low-level implementations may use feedback control, model predictive control, or mathematical programming.Source: Operational-Context Representation and… · p. 3

Method

Continuation-Aware GRPO

  • Continuation-Aware GRPO applies each sampled decision for one interval but evaluates it over a longer closed-loop trajectory under the same frozen policy.It adds no terminal-value model or auxiliary reward; when the evaluation horizon equals the decision interval, it reduces to short-rollout evaluation.Source: Continuation-Aware GRPO · pp. 4–5

Training and Evaluation Across Operating Patterns and Simulators

  • Training uses an efficient simulator that preserves dominant coordination dynamics, while evaluation tests frozen policies on new and unseen patterns, increased uncertainty, and a more realistic simulator without fine-tuning.The operational-context format, high-level decision space, and low-level control interface remain unchanged across simulators.Source: Training and Evaluation Across… · p. 5

Experiments

The experiments evaluate hierarchical coordination across multi-ramp traffic control and virtual power plant management, including generalization to changed patterns and uncertainty, transfer to more realistic simulators, and the contribution of continuation-aware returns.

Experimental Tasks and Settings

  • Each frozen policy is tested without target-environment adaptation on 300 scenarios split equally across seen-pattern, unseen-pattern, and higher-uncertainty subsets, after training on 150 and validating on 50.Traffic training/evaluation uses LTM/SUMO; VPP training/evaluation uses a fast single-bus model/EPRI Ckt5 in OpenDSS.Source: Experimental Tasks and Settings · p. 5

Baselines and Evaluation Protocol

  • Baselines include conventional control or MPC-MILP, PPO end-to-end RL, fixed-mode hierarchy, PPO hierarchical RL, and prompting-only Qwen3-8B, Gemini 3.1 Pro, and Claude Sonnet 4.5.Hierarchical RL and prompting-only LLMs select joint modes using task-specific low-level components; prompting-only models receive the same operational information but no task-specific parameter updates.Source: Baselines and Evaluation Protocol · p. 5

Main Results

  • 16625.4 ± 84.5 overall traffic throughput (6.62% above Feedback Control) and 32679.2±223.6 VPP operating cost (4.37% below MPC-MILP) are best in every test subset.The overall traffic result also improves over Hierarchical RL by 3.06%.Source: Table 1 · p. 6, Table 2 · p. 6, Main Results · p. 6
  • Our method outperforms fixed-mode hierarchy in both tasks; hierarchical RL beats end-to-end RL, while the learned policy further improves over hierarchical RL and prompting-only LLM coordinators.These comparisons are consistent with context-adaptive joint modes, task-specific low-level execution, and task-specific learning from closed-loop feedback.Source: Main Results · p. 6
  • Traffic throughput degrades by 4.86% from seen-pattern to higher-uncertainty scenarios, versus 8.14%–11.87% for baselines; VPP cost rises by 4.94%, versus 8.52%–16.74%.The method leads on mean performance in both unseen-pattern and higher-uncertainty scenarios.Source: Main Results · p. 6

Cross-Simulator Evaluation

  • 3.82 ± 0.33% traffic and 5.62 ± 0.54% VPP are the smallest cross-simulator degradation gaps, alongside the highest mean SUMO throughput and lowest mean OpenDSS cost.End-to-End RL has a 10.09 ± 1.07% traffic gap, and Hierarchical RL has a 9.23 ± 0.95% VPP gap; fast-simulator rankings do not fully predict realistic-simulator performance.Source: Table 3 · p. 7, Cross-Simulator Evaluation · p. 7

Ablation Study

  • 16625.4 ± 84.5 traffic throughput versus 16318.7 ± 136.4 and 32679.2 ± 223.6 VPP cost versus 33581.5 ± 327.8 show gains of 1.88% and 2.69%.The full method evaluates H = 4∆, including subsequent closed-loop trajectory, versus H = ∆ for current-interval GRPO; High-U gains reach 2.84% for traffic and 3.81% for VPP.Source: Table 4 · p. 7, Ablation Study · p. 7

Conclusion · Appendix

The paper presents an LLM-based hierarchical coordination framework with Continuation-Aware GRPO, reporting stronger performance on unseen-pattern and higher-uncertainty scenarios and effectiveness in simulators with different dynamics and modeling assumptions.

Conclusion

  • The method outperforms baselines on unseen-pattern and higher-uncertainty scenarios across multi-ramp traffic control and virtual power plant energy management.It also remains effective without adaptation in simulators with different dynamics and modeling assumptions.Source: Conclusion · p. 7
  • The LLM maps heterogeneous operational context to joint modes for interacting units, while task-specific controllers or optimizers generate low-level actions.Source: Conclusion · p. 7
  • Continuation-Aware GRPO evaluates delayed coordination effects through subsequent closed-loop trajectories.Source: Conclusion · p. 7
  • Experiments evaluate the framework on multi-ramp traffic control and virtual power plant energy management.Source: Conclusion · p. 7
  • The method remains effective without adaptation in simulators with different dynamics and modeling assumptions.Source: Conclusion · p. 7
  • Ablation results support the contribution of continuation-aware return construction.Source: Conclusion · p. 7

A Task-Specific System Models and Low-Level Controllers … B.3 Baseline Implementations

The appendices detail task-specific models and execution layers, operational context for coordinators, and continuation-aware training, including reward and rollout construction, physical feasibility versus operational targets, and the fixed-snapshot design and its limitations. They also describe how coordination decisions are evaluated through subsequent policy rollouts and specify policy training, validation, and baseline designs.

A.1 Multi-Ramp Traffic Control

  • Traffic training uses a Link Transmission Model integrated every 5 s, while the low-level controller observes and updates gates every 60 s.The 27-link corridor spans 14.8 km; eight of nine on-ramps are controlled, and merges use a fixed 0.8 mainline priority. Fine integration preserves nonzero delays on short links.Source: §A.1 · p. 10
  • Ramp gates use independent occupancy-based hysteresis, while high-level modes coordinate ramps by selecting band widths or holding a gate always open.The feedback layer has no queue override or explicit inter-ramp coupling; mode-dependent thresholds use calibrated ramp- and simulator-specific centres, with clipping sometimes narrowing the realised band.Source: §A.1 · pp. 10–11

A.2 Virtual Power Plant Energy Management

  • The VPP low-level layer re-solves a rolling-horizon MILP every five minutes, and resource-specific modes change optimizer coefficients rather than replacing the shared optimizer.The model covers a cooling-season single-bus system; OpenDSS evaluation adds feeder power flow and network constraints. Physical power and state-of-charge limits are enforced exactly, while contractual and operational targets use penalized slacks and may be violated.Source: §A.2 · pp. 11–12

B.1 Operational Context, Prompt Templates, and Output Schemas

  • Prompts serialize observations, trends, forecasts, task and mode descriptions, and output schemas, with the same prompt builder used in training and evaluation.Forecasts provide no realized future values or planned dispatch; comparisons with MPC-MILP and RL coordinators match forecast access, whereas Feedback Control and Fixed-Mode Hierarchy receive less context. Corridor prompts also give every prompt-consuming coordinator the same domain guidance.Source: §B.1 · pp. 12–13
  • Training and evaluation validate machine-checkable outputs, but handle failures differently: training retries malformed responses before assigning the group minimum, while evaluation retains the previous decision.At evaluation, all LLM policies decode greedily at temperature zero, making a frozen policy deterministic for a given context.Source: §B.1 · p. 13

B.2 Continuation-Aware GRPO and Reward Construction

  • Continuation returns use task-level rewards and compare candidate decisions from identical restored states under the same future disturbances.For the corridor, reward counts vehicles discharged at exits; for the VPP, it is negative realised operating cost, not the planner’s surrogate objective. Returns are undiscounted, and at H = 4∆ each return includes the initial decision plus three independently sampled follow-up decisions.Source: §B.2 · pp. 13, 16
  • The fixed snapshot pool standardizes comparisons but is not refreshed as the policy improves, creating distribution shift from training states to states encountered by the improved policy.The pool is built under a base policy and reused across horizon settings and ablations; the authors identify this shift as a secondary contributor to horizon-sweep saturation.Source: §B.2 · p. 16
  • After each sampled decision’s interval, the rollout continues under the frozen policy that produced it, crediting that decision with its closed-loop consequences.The continuation uses the current behavior policy rather than stopping or switching to a fixed rule.Source: §B.2 · p. 16
  • Both task policies start from Qwen3-8B without supervised warm-start, and checkpoints are selected by validation return before testing.Five independent training seeds are used, with evaluation seeds matched across methods.Source: §B.2 · p. 17
  • The policy samples response sequences, which are parsed into joint decisions; training therefore updates the response, including its justification, rather than only the emitted mode vector.The return depends on the decision parsed from the response, while the sequence-level reweighting distinguishes responses that encode the same mode vector differently.Source: §B.2 · p. 17

B.3 Baseline Implementations

  • Direct control uses a fixed Standard ramp-feedback controller for the corridor and conventional fixed-coefficient MPC-MILP for the VPP.The VPP coefficient vector lies outside the hierarchical methods’ mode space.Source: §B.3 · p. 18
  • End-to-End RL emits low-level actions, whereas Hierarchical RL emits joint modes; both use PPO and receive numeric encodings of the operational context and derived domain-prior signals.The RL baselines receive the same reward and were tuned per task on validation return.Source: §B.3 · pp. 18–19
  • Fixed-Mode Hierarchy applies one validation-selected joint mode throughout an episode, with exhaustive VPP search but only group-wise corridor candidates.Thus, its margin over the policy measures the cost of constancy on the VPP, but only bounds that cost on the corridor.Source: §B.3 · pp. 19–20
  • Prompting-only coordinators use prompts matching the learned policy’s apart from required interface envelopes, greedy decoding, and no parameter updates.Qwen3-8B provides the controlled comparison for isolating the effect of Continuation-Aware GRPO from prompting.Source: §B.3 · p. 20

B.4 Computational Cost and Training Overhead · C Simulators, Scenario Construction, and Evaluation Protocol · C.1 Dual-Fidelity Simulation Environments and Interface Correspondence

Training uses fast simulators and aligned interfaces to make repeated rollouts practical, while evaluation uses higher-fidelity environments to test frozen policies under added dynamics and constraints. Continuation-aware scoring increases wall-clock cost, and the corridor transfer claim is limited to dynamics changes after interface alignment using training and validation data.

B.4 Computational Cost and Training Overhead

  • Fine-tuning takes roughly 46 GPU-hours for the corridor and 34 for the VPP, using four A100 GPUs and about 3.1 × 10^5 generations per full run.The corridor costs more because each rollout step advances a traffic simulator, whereas the VPP step solves a small MILP; an eight-snapshot optimizer step issues 256 generations.Source: §B.4 · p. 20
  • A high-level deployment decision takes about 1.6 s for the corridor and 0.4 s for the VPP, within their 3 min and 30 min control intervals.Prompting-only API baselines are also within their control interval, but their network latency is outside the authors’ control.Source: §B.4 · p. 20
  • Extending continuation from H = ∆ to H = 4∆ raises measured wall-clock time by 3.1×, while continuation generations amortize in batched calls and simulator advances remain serial.The passage frames this overhead against 1.88% and 2.69% closed-loop gains reported in Table 4 of the main paper.Source: §B.4 · p. 20

C.1 Dual-Fidelity Simulation Environments and Interface Correspondence

  • Each task pairs an affordable fast training model with a higher-fidelity evaluation model, preserving identical context formats, joint mode spaces, and z 7→θ interfaces while changing underlying dynamics.The detailed simulator is excluded from training because repeated short rollouts require many simulator advances.Source: §C.1 · p. 20
  • For corridor evaluation, SUMO 1.21 detector observables are paired with LTM links by identity, and LTM capacity and inflow are calibrated against SUMO using only 150 training and 50 validation scenarios.No test scenario is simulated before frozen-policy evaluation, and no policy parameter is updated on the target simulator; the interface constants are fitted offline and shared across methods.Source: §C.1 · p. 20
  • The corridor transfer result is narrower than zero-shot transfer because its capacity, inflow, and eight hysteresis centres use SUMO data; the VPP carries resource parameters and its z 7→θ map over unchanged.The corridor interfaces are aligned on training and validation data, while the VPP adds a distribution network without such alignment.Source: §C.1 · pp. 20–21
  • Traffic hysteresis centres are aligned by occupancy quantiles per ramp, retaining SUMO values when centres fall outside the 5th–95th percentile range, so modes match while absolute thresholds can differ.The quantiles use congested-period occupancy from 200 training and validation scenarios; two of eight centres coincide across simulators.Source: §C.1 · p. 21, Table 12 · p. 21
  • VPP evaluation distributes resources across EPRI Ckt5 in OpenDSS, adding three-phase losses, voltage excursions, and line limits enforced by curtailing injections to feasible magnitudes.Voltage limits are [0.95, 1.05] pu and line loading is capped at 100% of rating; curtailment can later appear as unmet EV energy, comfort excursion, or import.Source: §C.1 · p. 21

C.2 Scenario Construction and Data Splits

The section specifies how scenarios are constructed from measured and generated data, and how training, validation, and test sets distinguish familiar patterns from structural shifts and greater uncertainty. It also bounds the corridor evaluation to demand levels that are neither uncongested throughout nor saturated from the start.

Data provenance and resampling

  • VPP inputs are measured records, while corridor detector data set origin volumes and generated profiles control temporal shape; observed channels use causal resampling.Measured series supply temporal shape, while amplitude is normalized and mapped to the modeled installation’s capacity. Two-sided interpolation is used only for forecast and simulator trajectories, not observed channels.Source: §C.2 · pp. 21–22, Table 14 · p. 23

Electricity tariff

  • VPP episodes span tariff tiers through the critical peak, with independently perturbed prices that preserve tier ordering.Energy withheld earlier can be bought at up to 1.25 CNY/kWh later; each tier price is scaled by an independent factor in [0.95, 1.05].Source: §C.2 · p. 22, Table 15 · p. 23

Operating-pattern families

  • Training and validation cover all four operating-pattern families; unseen-pattern tests withhold corridor temporal shapes or VPP descriptor combinations while preserving familiar individual regimes.For the corridor, the binding location is familiar but its time course is new; for the VPP, previously seen individual factors occur in withheld combinations.Source: §C.2 · p. 22

Scenario construction

  • Corridor scenarios vary mainline and eight-ramp demand levels and temporal profiles, retaining 62% of draws after excluding uncongested and initially saturated cases.Training profiles use a single plateau; unseen profiles shift the onset outside the trained window or contain two peaks. The screen rejects 24% of draws as uncongested and 14% as saturated, so corridor generalisation claims apply to the retained demand band.Source: §C.2 · p. 22

Splits and test subsets

  • Each task has 150 training, 50 validation, and 300 test scenarios, split into 100 each for seen-pattern, unseen-pattern, and higher-uncertainty evaluation.Higher-uncertainty cases increase demand and PV noise from 5% to 15%, forecast mean absolute error from 8% to 20%, and jitter VPP EV arrival and departure times by up to ±30 min. Assignments are fixed at whole-scenario granularity, with duplicate scenarios excluded.Source: §C.2 · pp. 22–23

C.3 Evaluation Protocol and Metrics · D Additional Quantitative Results

Evaluation uses frozen policies, matched simulator seeds, and episode-level metrics aligned with the training signal. The VPP cost comparison shows that the policy’s advantage over MPC-MILP comes mainly from fewer constraint penalties, not cheaper energy.

C.3 Evaluation Protocol and Metrics

  • Corridor throughput counts vehicles exiting over the 3-hour evaluation window, excludes the 15-minute warm-up, and counts each vehicle once.All methods use the same all-Standard warm-up, and simulator, controller, and random-number state are cloned at minute 15.Source: §C.3 · pp. 23–24
  • 696 CNY of the 1492 CNY advantage over MPC-MILP comes from undelivered EV energy, 665 CNY from over-cap charges, 122 CNY from energy, and 9 CNY from comfort.The breakdown connects aggregate VPP cost to physical quantities; it does not indicate that electricity was bought more cheaply.Source: §C.3 · p. 24, Table 18 · p. 24
  • VPP operating cost is energy purchased at the tariff plus penalties for import above the contracted cap, comfort-band violations, and EV energy undelivered at the deadline.The EV shortfall is charged once at episode end; reported costs are CNY per episode.Source: §C.3 · p. 24
  • Both reported metrics accumulate the same quantities used for training over the full episode, while battery wear and terminal-reserve hinge shape planning but are not charged.This makes the reported score consistent with the training signal without reporting the planner objective as cost.Source: §C.3 · p. 24
  • Each configuration is evaluated over five runs with frozen policies, greedy zero-temperature LLM decoding, and matched simulator seeds across methods.Trainable methods use five independent training seeds; the scenario set is fixed across runs, and only method-independent interface calibration uses the target simulator.Source: §C.3 · p. 24
  • Scores average 100 scenarios within each of three subsets, weight subset means equally, then report the mean and standard deviation across five run-level scores.Because each subset has 100 scenarios, the aggregate equals pooling all 300; gains use Feedback Control for traffic and MPC-MILP for VPP as baselines.Source: §C.3 · p. 24

D.1 Additional Metrics Beyond the Reported Objective · D.2 Cross-Simulator Transfer for All Methods

Beyond the trained objectives, the appendix reports metrics that reveal constraint, fairness, and decision-validity trade-offs, and extends transfer-gap comparisons to every method. The results show that the proposed policy avoids several hidden costs while continuation-aware learning retains performance with improved transfer robustness.

D.1 Additional Metrics Beyond the Reported Objective

  • The additional metrics use the same runs and aggregation as the main tables: equal-weight means over three test subsets, averaged over five matched runs.For corridor control, congestion duration counts 60 s intervals below 40% of free-flow speed, while ramp queue is averaged across eight ramps and time; spillover means a queue reaches ramp storage and blocks the upstream link.Source: §D.1 · p. 25
  • End-to-End RL raises throughput over Feedback Control but has 52% longer ramp queues and more than twice as many spillovers, while our policy has the highest bottleneck speed and shortest queues.The appendix attributes the queue result to preventing mainline breakdown, which preserves merge receiving capacity and lets held ramp traffic discharge sooner.Source: §D.1 · p. 25
  • The longest uninterrupted ramp closure averages 22 min for our policy, versus 25 for Feedback Control, 26 for Fixed-Mode Hierarchy, 31 for Hierarchical RL, and 43 for End-to-End RL.Closures are governed by occupancy crossings at the 60 s low-level cadence, so the policy’s throughput is not obtained by keeping one ramp shut for a long stretch.Source: §D.1 · p. 25
  • VPP violation steps count 5-minute intervals when commanded dispatch would breach voltage or line limits, while curtailed energy records the resulting injection reduction.Executed dispatch has no violations by construction; curtailment shifts intended energy and its shortfall reappears later, while comfort excursion integrates temperature deviation outside 22.0–26.5 °C.Source: §D.1 · p. 25
  • Our policy incurs the fewest violation steps, least curtailment, least undelivered EV energy, and smallest comfort excursion; Hierarchical RL beats MPC-MILP on cost but curtails more.End-to-End RL curtails more than five times as much as our policy, and the resulting rescheduling accounts for most of its cost penalty rather than tariff exposure.Source: §D.1 · p. 25
  • Fine-tuning reduces malformed decisions to fewer than one in 3,000 corridor decisions, compared with roughly one in twenty for the untuned 8B model.A rejected output retains the previous joint decision for one interval; the untuned model’s failures explain much of its underperformance against direct control.Source: §D.1 · p. 25

D.2 Cross-Simulator Transfer for All Methods

  • Feedback Control’s 4.31% and MPC-MILP’s 6.94% transfer gaps are the smallest after ours; continuation-aware scoring recovers robustness while retaining learning’s performance gains.Every other trained or prompted method has a larger gap than these non-learning controllers, which re-derive actions from current measurements and therefore have little to overfit.Source: §D.2 · p. 25

D.3 Continuation-Horizon Sensitivity

Continuation-horizon gains saturate: 4∆ captures nearly all available improvement and is used as an operating point, not claimed as an optimum. Gains are larger under higher uncertainty, and a compute-matched control shows that most of the improvement comes from extending rollouts rather than simply spending more compute.

D.3 Continuation-Horizon Sensitivity

  • 0.24 percentage points of throughput and 0.23 of cost reduction separate H=8∆ from H=4∆, but the residual improvement is smaller than run-to-run spread.The 8∆ setting costs twice as much training as 4∆ and 6.2 times as much as ∆; the sweep’s gains largely saturate by 2∆, with 4∆ capturing nearly all of them.Source: §D.3 · p. 26
  • The horizon sweep retrains from the same base policy and shows that 4∆ is a practical operating point, not an optimum.The curve flattens when the horizon covers the mechanism by which a decision’s consequences return: about 12 minutes for corridor merges and 120 minutes for VPP tariff transitions and PV ramps.Source: §D.3 · pp. 25–26
  • At 4∆, gains are consistently largest in the higher-uncertainty subset, reaching 2.84% and 3.81%.Under low uncertainty, immediate outcomes already proxy eventual outcomes well; under high uncertainty, decisions that initially look good may need reversal, which continuation reveals.Source: §D.3 · p. 26
  • At 4∆, the compute-matched H=∆ setting gains +0.52% throughput and +0.84% cost against +1.88% and +2.69% for H=4∆.The control uses four times as many snapshots per optimiser step, matching generation calls and simulated intervals; most of H=4∆’s gain is attributed to what the return measures, not how much it measures.Source: §D.3 · p. 26

D.4 Operational-Context Ablation … E.1 Mode-Selection Distributions

Operational-context ablations show that trends and forecasts contribute most to performance, while removing all context still leaves benefits from learned mode selection. Paired scenario analyses qualify the reported margins, and mode-selection patterns show context-responsive behavior across the corridor and VPP tasks.

D.4 Operational-Context Ablation

  • 1.91% of corridor throughput and 1.65% of VPP cost are lost when the most valuable single context categories—recent trends and forecasts, respectively—are removed.Trends and forecasts are the top two categories on both tasks, swapping order; uncertainty estimates rank third, while priorities and external events have small average effects despite being decisive in some intervals.Source: §D.4 · p. 27, Table 24 · p. 27
  • 3.86% of throughput and 4.09% of cost are lost when context is stripped to numerical measurements, yet the stripped policy still beats direct control and Fixed-Mode Hierarchy.The stripped policy improves over direct control by 2.51% on throughput and 0.45% on cost, and over Fixed-Mode Hierarchy by 1.05% and 0.76%, respectively; the full context’s joint contribution exceeds the largest single-category effect.Source: §D.4 · pp. 27–28, Table 25 · p. 28

D.5 Uncertainty of the Reported Differences

  • [+408, +574] vehicles and [+688, +1002] CNY are paired scenario-bootstrap intervals favoring the method over Hierarchical RL; all reported comparison intervals exclude zero.The intervals resample the 300 matched scenarios within subsets and recompute equal-weight aggregation over 10,000 resamples; the Hierarchical RL comparison is the narrowest among comparisons with other methods on both tasks.Source: §D.5 · p. 28, Table 26 · p. 28
  • Five-seed paired comparisons favor the policy in every Table 26 comparison except horizon-ablation throughput, where it leads in four of five runs.Scenario resampling does not capture training-seed uncertainty; the horizon-ablation throughput effect is the one comparison whose margin is of the same order as seed variation, and the overlapping intervals do not support ordering the three prompting-only coordinators.Source: §D.5 · p. 28

E.1 Mode-Selection Distributions

  • 0.86 vs 0.66 pooled corridor entropy and 0.95 vs 0.76 VPP entropy show broader selections than Qwen3-8B, but pooled entropy alone does not establish adaptivity.Fixed-Mode Hierarchy is beaten on both tasks and has pooled entropy 0.78 because its vector mixes ramp assignments, despite zero temporal entropy; the untuned model concentrates on default modes, while Always-Open appears on 4.1% of steps.Source: §E.1 · p. 29, Table 27 · p. 29
  • 44.2%, 38.6%, and 19.7% Sensitive selections on rising, falling, and flat occupancy show a nonmonotone corridor response, while proactive battery rises from 21.4% to 68.7% across VPP tariff tiers.Proactive EV moves oppositely with urgency, falling from 41.2% at low urgent share to 9.2% at high urgent share; these response directions match the modes’ effects on the low-level controllers.Source: §E.1 · p. 29, Table 28 · p. 29
  • 24.5 vs 6.3 percentage points in trend-conditioned Sensitive shares and 47.3 vs 16.0 in tariff-conditioned proactive-battery shares distinguish our policy from Qwen3-8B.Continuation-aware training changes what the policy responds to, not only how often it departs from the default.Source: §E.1 · p. 29, Table 28 · p. 29

E.2 A Representative Episode of Each Task

A representative corridor episode shows the coordinator changing ramp roles across demand phases, while the VPP episode shows resource modes shifting with tariff and operating constraints. Both illustrate how coordination responds to system context over time.

E.2 A Representative Episode of Each Task

  • The balanced-heavy corridor scenario was chosen to exercise coordination: mainline traffic nears capacity while three of eight ramps carry strong demand.Table 29 follows this scenario through five phases; its counts are out of eight controlled ramps, and occupancy is the mean at the critical merge.Source: §E.2 · p. 29, Table 29 · p. 30
  • Across the corridor’s five phases, the policy assigns Sensitive upstream of the impending bottleneck, uses Sluggish during the plateau, returns to Sensitive during decline, and uses Always-Open only after the mainline clears.The decline phase is where the episode’s queue advantage over RL baselines in Table 19 accumulates.Source: Table 29 · p. 30, §E.2 · p. 30
  • In the VPP episode, the policy conserves battery and EV energy through the shoulder tariff tier, then makes both proactive for the critical-peak tier when constraints allow.HVAC becomes proactive only when outdoor temperature makes comfort violation the binding risk.Source: §E.2 · p. 30

E.3 Where the Method Helps and Where It Does Not

The method’s gains are largest when coordination choices can change outcomes and low-level controllers retain flexibility; they are negligible when physical or controller constraints leave little to decide. Its scope is low-dimensional, semantically labelled decisions, not high-dimensional or continuous ones.

E.3 Where the Method Helps and Where It Does Not

  • Upstream-dominant corridor scenarios gain only 2.3%, while comfort-limited VPP scenarios gain 0.3% with a bootstrap interval of [−148, +361] CNY around +97, so improvement is not claimed.In comfort-limited cases, the temperature band binds for most of the episode, HVAC power cannot be deferred without an immediately penalized excursion, and the MILP has almost no feasible slack; MPC-MILP is the appropriate choice.Source: §E.3 · p. 30
  • 8.5% and 9.8% are the corridor gains in ramp-dominant and balanced-heavy scenarios, while VPP gains are between 5 and 6% across PV-, EV-, and grid-limited families.The corridor gains arise where several ramps compete for downstream capacity; VPP gains reflect different resource trade-offs. Each family contributes 75 of the 300 test scenarios.Source: Table 30 · p. 30, §E.3 · p. 30
  • The advantage is largest when the low-level layer has genuine freedom; a hierarchy over a saturated controller inherits its saturation.The method also presumes a low-dimensional, semantically labelled decision, as in four roles per ramp and three modes per resource; no claim is made for high-dimensional or continuous high-level decisions.Source: §E.3 · p. 30

Related papers

Loading 2608.15041v1…