PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Sensor Robustness in Plasma Diagnostic Models: A Systematic Evaluation on TokaMark

Neerav Gupta

arXiv:2607.11915v1physics.plasm-phcs.LG

TL;DR

Plasma diagnostic models are typically evaluated with clean, complete sensors despite failures clustering near disruptions. This paper benchmarks four architectures across realistic failures and finds disruption-proximate corruption can catastrophically impair sequence models, while imputation effects depend on the evaluation metric.

  • Problem

    Published plasma-diagnostic evaluations generally assume complete, clean sensor data despite failures occurring near the moments when disruption prediction is most critical.

  • Method

    The study benchmarks four architectures across six realistic sensor-failure scenarios and evaluates robustness using model metrics and shot-level alarm detection.

  • Results

    Disruption-proximate failure raises LSTM NRMSE by +212% and collapses alarm detection to TPR = 0.00, while mean-fill recovers TPR = 1.00.

  • Takeaways & Limitations

    Robustness evaluation and application-level alarm metrics are both necessary because imputation can have opposite effects across evaluation regimes.

  • Takeaways & Limitations

    The alarm threshold is a test-set-optimized current-drop heuristic, and false-alarm rates cannot be computed because all 50 test shots disrupted.

Abstract

from arXiv · show

Plasma diagnostic models for tokamak fusion devices are almost universally evaluated on clean, complete sensor data. In practice, fusion diagnostics fail regularly: acquisition systems start late, individual sensors die, and signal dropouts cluster precisely when a plasma disruption is approaching. We present the first systematic robustness benchmark for plasma diagnostic ML using the TokaMark dataset of 11,573 MAST shots, evaluating XGBoost, LSTM, Transformer, and the TokaMark CNN baseline across six physically-grounded failure scenarios and three imputation strategies. We introduce the Robustness Score (RS) for standardized cross-architecture comparison. Our central finding is that disruption-proximate sensor failure (corruption injected in the final window timesteps) collapses sequence model performance (LSTM +212% NRMSE) while a statistical feature model remains comparatively stable (XGBoost +37%). Forward-fill imputation eliminates nearly all degradation from random dropout for sequence models (LSTM +57% to ~0%), but offers little help when the end of the window is corrupted. Shot-level alarm evaluation using ground-truth disruption timestamps reveals that LSTM alarm detection collapses to TPR=0.00 under proximate sensor failure, while mean-fill imputation recovers it to TPR=1.00, a reversal of the pattern observed in NRMSE. Plasma current emerges as the single most critical diagnostic across all architectures (+73% to +140% upon removal). Code, data, and trained checkpoints are available at https://github.com/Neerav-Gupta/tokamark-robustness.

1 Introduction

Plasma disruption prediction is a safety requirement, yet existing diagnostic-model evaluations largely assume complete, clean sensor data despite realistic, disruption-proximate failures. This paper addresses the missing robustness evaluation with natural-missingness analysis, six physically grounded failure scenarios, cross-architecture benchmarking, alarm metrics, and imputation comparisons.

  • Motivation: 3–4% of routine plasma shots experience disruptions, making reliable real-time prediction with 50–100 milliseconds of warning a safety requirement.Disruptions rapidly dump stored plasma energy into reactor walls, while early warnings can enable graceful plasma termination.
  • Problem: Existing evaluations assume complete, clean sensor data even though acquisition delays, sensor failures, and disruption-proximate dropouts regularly produce incomplete inputs.Front-loaded acquisition gaps cover 10–16% of signal history, and failures are concentrated near plasma disruptions.
  • Problem: The study identifies a gap in TokaMark and prior work: robustness to incomplete state information is recognized as a benchmark dimension but has not been systematically evaluated.TokaMark notes that signals may be absent because of hardware issues, limited acquisition windows, or diagnostic failures.
  • Contributions: The benchmark characterizes NaN structure in 300 MAST shots, finding front-loaded, category-correlated failures that inform six physically grounded scenarios, including previously unevaluated disruption-proximate failure.Scenarios include random dropout, channel ablation, front/random/pre-event temporal gaps, correlated diagnostic-group failure, and disruption-proximate failure.
  • Contributions: The study evaluates XGBoost, LSTM, Transformer, and the TokaMark CNN baseline with Robustness Score, shot-level TPR and MWT, and zero-, mean-, and forward-fill imputation.Imputation is compared across all scenarios for both NRMSE and shot-level alarm metrics, with potentially different optimal strategies across regimes.

2 Related Work

Prior work established benchmarks, disruption-prediction systems, alarm metrics, sequence forecasting, and foundation models for fusion, but generally assumed complete sensors. This study addresses the unquantified robustness gap by systematically evaluating missing-sensor effects across architectures and failure settings.

  • TokaMark and FAIR-MAST: TokaMark introduced 14 tasks across four groups using an 80/10/10 shot-level split from FAIR-MAST, while its CNN baseline reaches NRMSE 0.429 on Task 4-4.TokaMark identifies robustness to incomplete state information as an evaluation dimension but does not evaluate it.
  • Disruption prediction: Prior disruption-prediction research established cross-device learning, real-time operation, high-resolution diagnostics, and spatiotemporal modeling across JET and DIII-D.These studies demonstrate sustained development of machine-learning disruption predictors across devices and diagnostic representations.
  • Disruption alarm systems: Real-time alarm systems established per-shot evaluation with true positive rate, false alarm rate, and warning time, while DisruptionBench provided a systematic evaluation framework.Existing systems nevertheless assume complete sensor availability.
  • Missing data in tokamak ML: Prior missing-data studies examined interferometer outages in 187 GOLEM shots or entirely absent channels during deployment on a new device, leaving broader architectural comparisons open.The cited studies are narrower than the present evaluation, which compares four model architectures across missing-sensor conditions.
  • Foundation models and XGBoost: TokaMind and related foundation-model work identify missing-channel handling as a deployment challenge but do not quantify degradation, while XGBoost natively handles missing values without imputation.These observations motivate comparing architectural inductive biases under sensor failure.

3 Background

This benchmark uses FAIR-MAST data from 11,573 MAST shots to study Task 4-4, a non-Markovian forecast of future plasma current from diagnostic history. Its corruption scenarios are grounded in observed missingness, including front-loaded kinetic gaps and perfectly correlated interferometer–D-alpha failures.

  • Dataset: 11,573 FAIR-MAST shots from MAST’s final five campaigns provide the openly available TokaMark dataset, split into 9,270 training, 1,157 validation, and 1,146 test shots.Each shot includes the ground-truth disruption timestamp t_cut, and TokaMark selects 39 signals for 14 prediction tasks.
  • Task definition: Task 4-4 predicts plasma current 100 milliseconds ahead from 150 milliseconds of diagnostic history in a non-Markovian autoregressive forecasting setup.The full input history is required rather than only the current state.
  • Task definition: 14 diagnostic input signals and 4 actuator signals span MAST’s magnetic, kinetic, radiative, and active-coil categories.The signals are represented as (600, 18) time series, as described in Table 1.
  • Observed failure patterns: 0% NaN occurs for plasma current and coil currents across all 300 sampled shots, whereas interferometer and D-alpha average 14–15% NaN per shot.The kinetic missingness is caused by acquisition delays that create front-loaded gaps at shot start.
  • Observed failure patterns: r = 1.000 characterizes the identical interferometer and D-alpha NaN percentages in every sampled shot, motivating a correlated group failure scenario.The two diagnostics share a single acquisition trigger, so late triggering removes both signals simultaneously.

4 Methodology

The study compares four architectures under six physically grounded sensor-failure scenarios, using clean-data training, masked test corruption, three imputation strategies, and the Robustness Score for standardized comparison. Evaluation combines window-level NRMSE with shot-level alarm metrics based on ground-truth disruption timestamps.

  • Model architectures: XGBoost compresses each window into 142 summary statistics, whereas LSTM, Transformer, and CNN process a fixed (600, 18) tensor using distinct sequence encoders.XGBoost discards temporal structure; LSTM uses a 2-layer recurrent encoder, Transformer a 3-layer attention encoder, and CNN 18 parallel convolutional encoders.
  • Data split and protocol: 9,950 windows from 200 training shots train the models, while 2,420 windows from 50 test shots provide evaluation under TokaMark’s shot-level split.All models are trained exclusively on clean data, and corruption is applied only to the test set.
  • Evaluation metrics: NRMSE measures window-level prediction quality, while shot-level alarms use ground-truth disruption timestamps and thresholds optimized to maximize true positive rate.The introduced Robustness Score equals 1.0 when corruption has no effect and decreases linearly with degradation; XGBoost corruption proxies are conservative estimates for temporal gaps and dropout.
  • Failure scenarios: Six failure scenarios model random dropout, channel ablation, category removal, temporal gaps, correlated group failure, and disruption-proximate failure at the window’s end.The scenarios are physically grounded in tokamak failure modes, with disruption-proximate failure targeting sensors during the final moments before the predicted event.
  • Imputation: Three post-corruption strategies are evaluated: zero-fill as the baseline, mean-fill using training-set channel means, and forward-fill using the last valid observation.Imputation fills only artificially zeroed positions identified by the corruption mask; forward-fill reflects common industrial sensor behavior.

5 Results

Results show a consistent accuracy–robustness tradeoff across architectures, with sequence models especially vulnerable to disruption-proximate corruption while forward-fill stabilizes random dropout. Alarm detection exhibits the opposite imputation pattern: mean-fill restores LSTM detection under proximate failure despite worsening NRMSE.

  • Clean-data performance: NRMSE 0.470 and RS 0.765 make the Transformer most accurate but least robust, whereas XGBoost has slightly worse clean NRMSE and the highest RS of 0.841.CNN records clean NRMSE 0.528 and RS 0.764, nearly matching the Transformer; all clean NRMSE values exceed the 0.429 TokaMark CNN baseline trained on 9,270 shots.
  • Random dropout: +201% degradation makes CNN most sensitive to 50% zero-filled dropout, compared with XGBoost +145%, Transformer +85%, and LSTM +57%.Forward-fill reduces LSTM degradation to ∼0% and Transformer degradation to −0.4% at 50% dropout, while XGBoost with mean-fill remains at +39%.
  • Channel and diagnostic-group ablation: +139.8% degradation after plasma-current removal is the largest channel-ablation result for LSTM, followed by Transformer +89.1%, XGBoost +72.7%, and CNN +27.7%.Kinetics and soft X-ray radiatives cause under +20% degradation for all models, while Mirnov spectrogram failure stays under +1%.
  • Temporal gaps: +212% NRMSE at a 20% proximate gap causes catastrophic LSTM degradation, while XGBoost rises only +37% because trajectory statistics remain partially informative.Front gaps at 60% produce LSTM +0.0% and XGBoost −1.3%; forward-fill improves LSTM from +212% to +15% at 20% proximate severity but fails at higher severities.
  • Imputation strategies: +212% →+1.7% is the LSTM recovery from forward-fill at a 10% proximate gap, but at 50% proximate gap mean-fill worsens NRMSE from 1.55 to 1.88.Forward-fill leads for dropout and temporal gaps, cutting LSTM 50%-dropout degradation from +57% to near zero and Transformer degradation from +85% to near zero.
  • Alarm detection: TPR = 1.00 under mean-fill restores LSTM alarm detection for all 50 disruptive shots after proximate failure, reversing its NRMSE ranking.Under 25% proximate failure, LSTM TPR falls from 0.52 to 0.00; under zero-fill it remains 0.00, while forward-fill reaches TPR = 0.96.

6 Discussion

The discussion shows that alarm metrics can reverse conclusions drawn from NRMSE, while robustness depends on architecture, signal choice, and failure context. It also identifies practical interventions and limitations that constrain interpretation and generalization.

  • Architectural interpretation: 0.764 vs 0.765: CNN and Transformer achieve nearly identical overall Robustness Scores but exhibit different scenario-specific vulnerabilities.CNN is more sensitive to active coil removal because losing an encoder branch prevents cross-channel compensation.
  • Signal importance: Plasma current is the most critical input across architectures, but its importance is tempered because it is actively controlled and remains relatively flat during most discharges.It can also be synthesized from Bp pickup-coil measurements through Ampere’s law when the primary Rogowski coil fails.
  • Alarm metrics: TPR = 0.00 to TPR = 1.00: mean-fill imputation recovers LSTM alarm detection under proximate failure despite worsening NRMSE.NRMSE and threshold-alarm evaluation therefore support different conclusions for continuous monitoring and disruption detection.
  • Operational implications: 12–15ms mean warning times are below the 50–100ms ITER requirement, and LSTM provides zero warning on every shot under proximate failure without imputation.This frames proximate sensor failure as a machine-protection problem rather than merely a prediction error.
  • Design interventions: Forward fill nearly eliminates random-dropout degradation for NRMSE prediction, while mean fill and forward fill strongly recover LSTM alarm detection under proximate failure.Forward fill is presented as a practical default when architectural changes are not immediately feasible.

7 Conclusion

The paper establishes a systematic benchmark for plasma diagnostic ML robustness under realistic sensor failures and identifies actionable differences across failure modes, architectures, and evaluation regimes. It concludes that both robustness and alarm metrics are necessary for fault-tolerant plasma prediction design.

  • Conclusion: The benchmark evaluates four architectures across six sensor-failure scenarios and two evaluation regimes, extending TokaMark with a systematic robustness dimension.The authors describe these findings as robust and actionable.
  • Conclusion: +212% NRMSE degradation and TPR = 0.00 occur for LSTM under disruption-proximate sensor failure.The failure is attributed to the LSTM’s learned dependence on final-window timesteps near disruption events.
  • Conclusion: Forward-fill imputation resolves the random-dropout problem for sequence models almost entirely without architectural changes.The conclusion characterizes this as having zero architectural cost.
  • Conclusion: RS = 0.764 for the TokaMark CNN baseline is nearly identical to Transformer RS = 0.765.This supports the conclusion that the accuracy-robustness tradeoff reflects raw time-series processing rather than a specific architecture.
  • Conclusion: Robustness evaluation and alarm metric evaluation are both necessary and not interchangeable for designing fault-tolerant plasma prediction systems.The authors propose the Robustness Score, alarm evaluation framework, and benchmark methodology as a foundation for ITER-class devices.

Figures

The figures show that corruption timing and diagnostic choice strongly shape model robustness, with XGBoost most robust overall and disruption-proximate failures especially damaging to sequence models. Alarm-focused evaluation further shows that imputation can reverse the NRMSE-based mitigation pattern.

  • Corruption Scenarios: Forward fill nearly eliminates random-dropout degradation for sequence models, whereas no mitigation is shown for zero-fill corruption.Figure 2 compares zero-fill, mean-fill, and forward-fill across severity levels and corruption scenarios.
  • Channel Importance: Plasma current dominates channel importance across all four architectures, while active coils rank second and cause +46.9% degradation for CNN.Kinetics and radiatives cause minimal degradation despite having the highest natural missingness rates.
  • Temporal Failure Timing: Disruption-proximate and pre-event gaps cause catastrophic, severity-independent degradation in LSTM, while Transformer and CNN degrade more gradually.The figure attributes LSTM’s vulnerability to reliance on final timesteps for disruption prediction.
  • NRMSE Mitigation: Neither forward-fill nor mean-fill effectively mitigates high-severity disruption-proximate failure under NRMSE evaluation.Forward fill is consistently most effective for dropout and temporal-gap scenarios, while mean fill partially recovers XGBoost under dropout.
  • Robustness Score: XGBoost achieves the highest overall Robustness Score (RS) at 0.841, compared with Transformer at 0.765 and CNN at 0.764.The figure defines RS = 1.0 as perfect robustness and interprets lower values as greater degradation.
  • Shot-Level Alarm Detection: Under 25% proximate failure, LSTM TPR collapses to 0.00, but mean-fill recovers it to TPR = 1.00 across all 50 disruptive shots.Forward fill also recovers LSTM to 0.96; mean fill helps alarms despite being harmful for NRMSE.
Loading 2607.11915v1…