Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

Siqi Ding; Xuanhe Wang; Pei Guo; Guoyang Shi; Changquan Yu; Yiting Wang; Xianming Song; Xiang Gu; Zhengyuan Chen; Lei Xing; Yapeng Zhang; Jianguo Chen; Tianyuan Liu

Source-linked AI summary

Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

Siqi Ding, Xuanhe Wang, Pei Guo, Guoyang Shi, Changquan Yu, Yiting Wang, Xianming Song, Xiang Gu, Zhengyuan Chen, Lei Xing, Yapeng Zhang, Jianguo Chen, Tianyuan Liu

arXiv:2608.20834v1Aug 21, 2026physics.plasm-phcs.AI

TL;DR

XPT control must maintain a secondary null on the divertor leg, while existing EXL-50U operation lacks dedicated closed-loop null feedback. The paper formulates this as multi-objective RL using Advantage Aggregation, which preserves objective-wise temporal credit before nonlinear aggregation. AdvA-PPO improves nominal worst-channel performance, remains the only learned controller completing combined-uncertainty rollouts with a usable XPT shape, and reaches all tested initials after fine-tuning.

Problem
Existing XPT operation relies on feedforward waveforms and global PID loops without dedicated closed-loop feedback for the secondary null, whose displacement degrades topology and exhaust geometry.
Method
The paper develops Advantage Aggregation, retaining channel-specific value heads and advantages before worst-objective-aware nonlinear scalarisation and adding a residual policy-update correction.
Results
AdvA-PPO raises the mean worst-channel score from 0.23 to 0.81 over Reward-PPO on a nominal 500 ms rollout, reduces X-point flux RMSE by about 20×, and is the only learned controller completing combined-uncertainty rollouts with a usable XPT shape.
Takeaways & Limitations
The results establish a simulation-based foundation for future real-time XPT validation on EXL-50U across nominal, uncertain, and varied-initial-equilibrium conditions.
Takeaways & Limitations
The evidence is simulation-based, and zero-shot AdvA-PPO fails the limiter-to-XPT topology conversion absent from its training distribution.

Abstract

from arXiv · show

Managing divertor heat loads is a central challenge for compact, high-power tokamaks. To increase local flux expansion and decouple the dissipation volume from the core, EHL-2 adopts the X-point target (XPT) divertor. This requires the secondary X-point to remain on the divertor leg; displacement degrades the topology and exhaust geometry. Current experiments, including EXL-50U discharges, rely on precomputed feedforward waveforms with PID loops on global quantities. Lacking dedicated closed-loop feedback for the secondary null, XPT operation is repeatable but not routine. We formulate XPT feedback as a multi-objective reinforcement learning (RL) control problem in a free-boundary environment calibrated to EXL-50U discharge #13906. To address strong coupling among plasma current, shape, and null constraints - where reward scalarisation collapses objective-specific temporal credit - we develop Advantage Aggregation (AdvA). AdvA preserves objective-wise temporal credit before worst-objective-aware nonlinear scalarisation and introduces a residual correction to policy updates. AdvA-PPO is evaluated against Reward-PPO and a feedforward-plus-PID baseline under nominal operation, measurement uncertainties, and unseen initial equilibria. On a 500 ms rollout, AdvA-PPO raises the mean worst-channel score from 0.23 to 0.81 over Reward-PPO, reducing X-point flux RMSE by ~20x. Under combined measurement uncertainties, it is the only learned controller completing the horizon while retaining a usable XPT shape. Multi-initialization fine-tuning enables a single AdvA-PPO policy to complete full-horizon operation across divertor and limiter initial equilibria. These results provide a simulation-based foundation for future real-time XPT validation on EXL-50U.

1 Introduction

XPT offers EHL-2 a magnetically accessible divertor topology with long-leg flux expansion, but its exhaust function depends on keeping the secondary null on the divertor leg. Existing EXL-50U control relies mainly on feedforward waveforms and global PID feedback, motivating explicit reconstructed-null closed-loop control and AdvA.

XPT combines a long divertor leg, strong local flux expansion, and a dissipation volume magnetically decoupled from the core.
Secondary-null displacement can degrade the functional XPT geometry even when plasma current and global position remain acceptable.
EHL-2 design and edge-transport studies identify outer XPT as the reference divertor configuration and define the target topology motivating this study.
EXL-50U experiments establish magnetic access to XPT operation, but primarily use precomputed PF-coil waveforms with PID feedback on global quantities.
AdvA retains one value head and GAE per objective channel before worst-objective-aware nonlinear scalarisation, with a controlled residual correction to policy updates.
The framework jointly regulates plasma current, LCFS geometry, X-point positions, and X-point flux constraints through shared poloidal-field actuators.

2.1 EXL-50U XPT control problem

The EXL-50U control stack uses FGE observations to regulate plasma current and magnetic configuration through 11 learned coil commands, while the vertical-stability coil remains under PID control. The same interfaces support deployment with real-time equilibrium reconstruction.

Twelve active coil circuits regulate plasma current and magnetic configuration, with learned policies commanding 11 CS and PS1–PS10 voltages while VS remains under PID.
FGE supplies plasma current, coil currents, LCFS, and X-point features; the policy returns 11 coil voltages before the environment advances and scores the next step.
The deployment architecture replaces FGE observations with real-time equilibrium reconstruction while preserving the same state and coil-command interfaces.

2.2 FGE simulation environment

FGE is a calibrated free-boundary equilibrium evolution environment that advances plasma and conductor dynamics from coil voltages and returns the magnetic features needed for XPT control. Validation against LIUQE reconstruction of discharge #13906 preserves the four-null XPT topology with centimetre-scale geometric offsets.

FGE advances axisymmetric equilibria on resistive timescales by coupling the Grad–Shafranov equation, conductor circuit dynamics, and plasma-current evolution.
Each 1 ms step returns plasma current, coil currents, LCFS, X-point positions, and fluxes for observations and rewards on a 66 × 65 grid with 544 passive filaments and 12 active circuits.
The environment is calibrated to EXL-50U discharge #13906 over 500–700 ms using LIUQE reference equilibria, shot-specific βp(t) and q0(t), and a fitted constant plasma resistance.
Validation advances FGE from the LIUQE equilibrium at 0.5 s to 0.7 s under recorded coil voltages and compares boundaries, nulls, and global traces.
The validation preserves all four XPT nulls, with Ip RMSE of 0.8% and LCFS/X-point offsets near the ≈1.7 cm grid-cell scale.

2.3 Control methods

The paper defines XPT magnetic control through a shared reconstructed-observation interface and compares reward-level PPO with Advantage Aggregation, which preserves per-objective temporal credit before nonlinear aggregation. AdvA-PPO adds a norm-capped residual correction while leaving reliability gating and gradient projection disabled.

FF+PID baseline: The experiment-derived FF+PID baseline adds online PID corrections to offline PF-coil and plasma-current feedforward references.Its reported scores reuse the discharge #13906 feedforward and PID gains without per-initial redesign.
Control interface: The learned controllers use a 45-dimensional reconstructed observation and command 11 coil voltages, while VS remains under a separate PID loop.The observation includes plasma-current, coil-current, X-point, and LCFS features; actions control CS and PS1–PS10.
Control interface: Tracking signals cover plasma current, four X-point positions, four X-point flux conditions, and the LCFS boundary.Each physical error is converted into a satisfaction signal in [0, 1] before aggregation.
Reward-PPO baseline: The Reward-PPO baseline scalarises channel rewards before GAE, using SmoothMax with α < 0 to emphasise poorly satisfied objectives.A single value function and GAE then provide the PPO advantage.
AdvA-PPO: AdvA computes one value head and one GAE per physical channel, then applies nonlinear SmoothMax aggregation to preserve objective-wise temporal credit.This moves scalarisation from the reward layer to the advantage layer, preventing a weak channel from collapsing every other channel’s temporal credit before advantage estimation.
AdvA-PPO: AdvA-PPO combines advantage-level SmoothMax with α = −3 and a capped residual using β = 0.2 and κ = 1.Reliability gating and gradient projection are disabled in the main controller.

2.4 Evaluation setup

Evaluation uses 500 ms closed-loop rollouts from a calibrated initial equilibrium and three withheld initial equilibria, with separate measurement-delay, noise, and combined-uncertainty cells. Scores are computed from archived equilibria using physical tracking errors, survival, actuation, and shared multi-channel scores.

Evaluation protocol: The primary evaluation uses a 500-step (500 ms) horizon with deterministic test-time actions.Training episodes last 300 steps, so the longer test window also checks temporal extrapolation.
Initial equilibria: Four initial equilibria share the frozen #13906 XPT target, with I1 calibrated for training and I2–I4 withheld for cross-initialization.The withheld cells change the FGE initial and Ip reference while retaining the same geometric target and reward operators.
Measurement uncertainties: Robustness tests inject observation-only disturbances, including 1–3 ms per-step delay jitter, diagnostic and coil noise, and their combination.R3 applies both the delay jitter and the full diagnostic/coil noise configuration.
Metrics: Archived equilibria are mapped to X1–X4 before position and poloidal-flux errors are computed against the frozen target.Flux error is reported in Wb, while mean metrics average the corresponding per-slot quantities.
Metrics: Tables report RMSE, maximum absolute error, survival length, and |∆V| over the survived horizon.Full survival corresponds to 500 ms, and short-survival errors are secondary to survival.
Metrics: The evaluation also reports time means of the ten-channel SmoothMax score and the hard worst-channel mean, with higher values preferred for both.These evaluation scores are distinct from the advantage-level SmoothMax used inside AdvA.

3 Results

AdvA-PPO provides the strongest nominal multi-objective control and is the only learned controller to survive combined diagnostic noise and observation delay while retaining a usable XPT shape. Cross-initialization exposes a transferability gap, but fine-tuning extends full-horizon operation to limiter and divertor initials.

Nominal operation: AdvA-PPO leads or matches Reward-PPO and FF+PID on Ip, primary X distance, mean X flux, and LCFS during the nominal 500 ms rollout.All three controllers survive the horizon, so this comparison measures tracking quality rather than survival.
Nominal operation: AdvA-PPO raises the shared mean worst-channel score from 0.23 to 0.81 relative to Reward-PPO, while reducing mean primary X-flux RMSE from 13.1 to 0.66 × 10^-4.The corresponding shared scores are ū = 0.56 for Reward-PPO and ū = 0.93 for AdvA-PPO.
Nominal operation: AdvA-PPO holds secondary flux errors about an order of magnitude tighter than Reward-PPO while allowing the secondary nulls to sit about 10 mm farther along the divertor leg.The reported X1 flux errors are 1.07 versus 17.1 × 10^-4 Wb, and X4 errors are 0.61 versus 8.8 × 10^-4 Wb; the residual offset is comparable to the 13.5 mm reconstruction RMSE.
Nominal operation: AdvA-PPO achieves its nominal gains with the lowest mean per-step actuation, 0.36 V versus 0.56 V for Reward-PPO and 0.45 V for FF+PID.Figure 8 shows the separatrix remaining on the target boundary and the X points close to their targets under AdvA-PPO.
Measurement uncertainties: Under combined measurement noise and observation delay, AdvA-PPO is the only learned controller that completes the horizon while retaining a usable XPT shape.Reward-PPO terminates at 322 ms, whereas AdvA-PPO and FF+PID complete the horizon; FF+PID survives but retains noise-dominated degradation.
Cross-initialization and fine-tuning: Zero-shot cross-initialization favors FF+PID for transfer stability, while multi-initialization fine-tuning enables AdvA-PPO to complete the full horizon across divertor and limiter initials.The unfine-tuned AdvA-PPO fails on limiter initial #15892 at step 96; fine-tuning raises survival to the full horizon while halving boundary error from 182 to 88 × 10^-3.

4 Conclusion

The paper establishes reconstructed-null, multi-objective RL feedback for EXL-50U XPT control in an experiment-calibrated free-boundary environment. AdvA-PPO improves nominal multi-objective performance and shows regime-dependent robustness, while experimental validation and broader generalisation remain open.

Conclusion: The framework uses reconstructed-null feedback through shared poloidal-field actuators to regulate ten physical objectives spanning plasma current, LCFS, X-point positions, and X-point fluxes.The FGE environment reproduces the experimental #13906 equilibrium with centimetre-scale boundary and null accuracy.
Conclusion: AdvA-PPO raises the shared evaluation score from 0.56 to 0.93 and the mean worst-channel score from 0.23 to 0.81 relative to Reward-PPO.It also reduces mean X-point flux RMSE by approximately 20× on the nominal 500 ms rollout.
Conclusion: Under combined measurement noise and observation delay, AdvA-PPO is the only learned controller that completes the horizon while retaining a usable XPT shape.Reward-PPO is stronger under noise alone, whereas FF+PID tracks plasma current well under delay.
Conclusion: Multi-initial adaptation extends AdvA-PPO to all four initial equilibria but exposes a coverage–specialisation trade-off.PTEFIT-in-the-loop and closed-loop machine performance remain to be tested experimentally.

5 Future outlook

Future work prioritises staged on-machine validation, robustness beyond single-initial training, and broader divertor control tasks. The proposed programme separates reconstruction and learned-control effects before extending evaluation to disturbances and richer topology objectives.

On-machine validation: Staged EXL-50U validation will first quantify PTEFIT secondary-null estimate accuracy, latency, and stability before evaluating the RL controller in the loop.Conservative actuator limits, online monitoring, and a deterministic FF+PID fallback are planned.
Robustness and generalisation: Multi-initial and uncertainty-aware training should target plasma-current targets, measurement uncertainties, actuator errors, and FGE calibration residuals.Domain randomisation, constrained policy optimisation, and stage-dependent objectives are proposed as complementary approaches.
Expanded control tasks: The control task should expand from a fixed XPT target to broader divertor configurations and controlled topology transitions.Candidate objectives include strike-point and divertor-leg targets, richer LCFS descriptors, and explicit coil-current, voltage, and slew constraints.
Expanded control tasks: Coordinating these tests with auxiliary-heating experiments will probe magnetic-configuration control under more relevant plasma conditions.The programme is intended to prepare later integration with radiation, detachment, and target-heat-flux studies.