PaperlayerView exact revision on arXiv

Source-linked AI summary

EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection

Julian Oelhaf, Georg Kordowich, Christian Bergler, Andreas Maier, Johann Jäger, Siming Bayer

arXiv:2609.28149v1eess.SYcs.LGcs.SY

TL;DR

Machine-learning protection studies use differing tasks, sensing assumptions, partitions, metrics, and generalization conditions, complicating comparisons. EvEMTBench fixes these choices in an open, executable benchmark while leaving model design open. Its reference evaluation finds that conclusions vary by function and condition, and transfer is stronger for fault detection than localization.

  • Problem

    Protection studies differ in task definitions, measurement access, partitions, metrics, and generalization conditions, making results difficult to compare.

    Source: Introduction · p. 1

  • Method

    EvEMTBench provides an executable benchmark that fixes evaluation conditions across 12 functions and 24 scored tasks while leaving model design open.

    Source: Introduction · p. 2, Benchmark scope and terminology · p. 3

  • Results

    Performance conclusions depend on protection function, measurement access, evaluation family, and generalization protocol; cross-grid transfer is stronger for fault detection than localization.

    Source: Discussion · p. 11, Conclusion · p. 12

  • Takeaways & Limitations

    EvEMTBench makes generalization evaluation explicit and reproducible, providing a shared reference point for future methods.

    Source: Conclusion · p. 12

    Evidence uses synthetic EMT simulations and does not establish field-measurement performance, deployment readiness, or operational event frequencies.

    Source: Limitations and future directions · p. 12

Abstract

from arXiv · show

Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metrics, and generalization conditions often differ. EvEMTBench addresses this gap with an open, executable, and versioned benchmark that fixes these evaluation choices while leaving model design open. Across four grids spanning 20-345 kV, it defines 12 protection and event-analysis functions instantiated as 24 scored tasks and supports structured evaluation across observability conditions, predefined distribution shifts, and zero-shot and fine-tuned cross-grid transfer. Committed partitions, leakage controls, and reproducible reporting provide a common basis for comparing future methods. A reference evaluation spanning trivial, conventional, feature-based, and deep-learning baselines shows that wider observability is not uniformly beneficial, shifted conditions can reveal failures not apparent in-distribution, and cross-grid transfer is substantially stronger for fault detection than for fault localization. Protection-relevant diagnostics identify failure modes not apparent from primary metrics alone. EvEMTBench therefore makes generalization in machine-learning-based protection an explicit and reproducible evaluation problem.

Introduction

Machine-learning protection studies are difficult to compare because they vary in tasks, sensing, partitions, metrics, and generalization conditions. EvEMTBench provides a versioned, executable benchmark that fixes these evaluation choices while leaving model design open.

Introduction

  • Reliable protection must identify and clear faults while avoiding unnecessary operation during non-fault events, including under altered fault-current magnitudes and transient responses.Machine-learning implementations require evidence grounded in clearly defined operating, sensing, and validation conditions, not test-set performance alone.Source: Introduction · p. 1
  • Evaluation choices vary substantially across studies, making reported performance differences difficult to attribute to model quality alone.Differences include grids, event types, sampling rates, observability, data splits, metrics, and leakage controls.Source: Introduction · p. 1
  • EvEMTBench is an executable, versioned evaluation component built on a separate dataset, fixing comparison conditions while leaving model design open.The infrastructure includes committed group-aware partitions, training-only preprocessing, versioned artifacts, provenance records, and released evaluation software.Source: Introduction · p. 2
  • The benchmark standardizes 12 protection and event-analysis functions as 24 scored tasks, with fixed definitions, valid-sample rules, observability views, metrics, and reporting requirements.It also provides structured evaluation across matched and predefined shifted conditions, three observability views, and zero-shot and fine-tuned cross-grid transfer.Source: Introduction · p. 2
  • The reference evaluation spans trivial, conventional, feature-based, and deep-learning baselines and examines performance across functions, observability, shifted conditions, and cross-grid transfer.Its four questions address matched in-distribution performance, measurement access, predefined shift, and transfer with and without target-grid fine-tuning.Source: Introduction · p. 2

Background and Related Work

Prior work spans distinct protection objectives, model families, datasets, and evaluation practices. EvEMTBench builds on this landscape with an executable specification intended to make comparisons across tasks and conditions interpretable.

Background and Related Work

  • This section situates EvEMTBench within prior work on machine-learning protection, available datasets and benchmarks, and evaluation frameworks.The table’s comparison concerns selected public datasets and benchmarking resources relevant to EvEMTBench.Source: Background and Related Work · p. 2, Table 1 · p. 3

Machine learning for power system protection

  • Machine-learning protection covers fault detection, classification, localization, asset protection, and emergency control, using varied model representations and approaches.Examples include feature-based, wavelet-assisted, zonal, hybrid deep-waveform, relay-embedded, topology-aware, spatiotemporal, and graph-neural methods.Source: Machine learning for power system… · p. 2

Datasets and benchmarks for power system protection

  • Public resources range from field recordings and synthetic datasets to EMT benchmarks, but differ in signal representation, physical scope, task coverage, and prescribed evaluation procedures.Table 1 compares EvEMTBench with PSML, pmuBAGE, PowerGraph, PROTECT-90, and a latency-aware EMT benchmark.Source: Datasets and benchmarks for power… · p. 2, Table 1 · p. 3

From evaluation frameworks to executable benchmarking

  • A protection-evaluation framework identifies seven dimensions for interpretable, comparable results: objective, system scope, observability, timing, targets, training protocol, and outputs or diagnostics.The framework is intended to be independent of a particular dataset, grid, or model.Source: From evaluation frameworks to… · p. 2
  • A framework alone does not ensure comparability; direct model comparison requires a shared executable specification that fixes task, valid-sample, measurement, partition, and metric choices while leaving model design open.EvEMTBench implements this for multi-grid EMT data; the authors state that, to their knowledge, no existing protection-specific benchmark combines the listed tasks, views, grid families, shifted conditions, and cross-grid transfer on point-on-wave EMT data.Source: From evaluation frameworks to… · p. 3

EvEMTBench Benchmark Design and Methods

EvEMTBench separates its dataset from its benchmark: the dataset supplies EMT signals and metadata, while the benchmark fixes evaluation conditions and reporting. It defines multiple event-analysis tasks and valid-sample rules across grid families.

Benchmark scope and terminology

  • The benchmark fixes task definitions, valid-sample rules, observability views, partitions, leakage controls, protocols, metrics, and reporting while leaving model design open.The reference experiments illustrate comparisons enabled by the benchmark but are not part of its definition; Figure 1 depicts the dataset-to-evidence flow.Source: Benchmark scope and terminology · p. 3, Figure 1 · p. 4

Dataset and generalization conditions

  • The four reference grids are the CIGRÉ European medium-voltage grid, a 110 kV protection reference grid, the IEEE 39-Bus New England system, and a double-line grid.The simulations use a synthetic EMT dataset; the dataset paper documents its simulations, parameterization, formats, and technical validation.Source: Dataset and generalization conditions · p. 3

Benchmark tasks and valid-sample definitions

  • The benchmark maps windowed records to 12 functions across supported views, yielding 24 function–view tasks, including fault detection, classification, localization, event detection, and event classification.Seven additional diagnostic functions address grounding, phase, fault category and origin, event state, and switch detection and type.Source: Benchmark tasks and valid-sample… · pp. 3–4
  • Each function uses a frozen target derivation and valid-sample mask; fault classification and localization use fault-active windows, and busbar faults are excluded from line localization.Fault detection is evaluated within fault episodes, with active windows positive and pre- or post-fault windows negative; benign switching episodes are not negative examples. The accompanying table summarizes targets and valid samples for the five primary functions.Source: Benchmark tasks and valid-sample… · p. 4, Table 3 · p. 5
  • Event detection covers active fault or switching events, whereas event classification predicts event class including no_event; inactive pre- and post-event windows supply negatives.The reference-grid families have no dedicated no-event episodes. Fixed class vocabularies retain unsupported classes at zero support, including grid-specific event classes and high-impedance fault classes.Source: Benchmark tasks and valid-sample… · p. 4

Dataset and generalization conditions

  • The dataset families define generalization conditions that differ in topology, operating state, and event-parameter variation.The article calls the archive families multi_grid, adapt_grid, and eval_grid to distinguish them from the EvEMTBench Benchmark.Source: Dataset and generalization conditions · p. 3, Table 2 · p. 5

Observability and measurement views

EvEMTBench varies measurement access through local, line, and global views, then evaluates models using separated, group-aware protocols. Its transfer design is constrained to deep baselines and fixed-width local or line inputs.

Observability and measurement views

  • Local and line views center on the ground-truth faulted element, while global observability includes all main-line bay measurements.For line faults, local view uses six channels at one predefined terminal and line view adds the opposite terminal for a 12-channel input; these are not fixed-relay selectivity tests.Source: Observability and measurement views · p. 5
  • Local and line input widths remain fixed across grids, but global width varies with topology; reference models receive only waveform channels, not explicit grid parameters or topology metadata.Direct transfer is possible with fixed-width views, whereas global transfer requires a topology-aware representation.Source: Observability and measurement views · p. 5

Evaluation protocols, partitions, and leakage control

  • The benchmark defines held-out, source-corpus pre-training, zero-shot transfer, and fine-tuned transfer protocols, with training, model selection, and evaluation separated.The eval_grid family is reserved for final target-grid evaluation.Source: Evaluation protocols, partitions, and… · p. 5
  • Held-out partitions split adapt_grid episodes 70/15/15% by sample_id and event type, keeping windows from each episode together across the five seeds.Validation supports early stopping and model selection; the adapt_grid test is in-distribution, while eval_grid provides a combined shift in operating conditions and event-parameter sweeps, not a single-factor test.Source: Evaluation protocols, partitions, and… · pp. 5–6
  • Cross-grid transfer uses source grids at 20, 110, and 345 kV, with randomized topologies disjoint from target reference topologies and group-aware source partitions.Zero-shot models use no target training or validation data; fine-tuning adapts source-pretrained models using target adapt_grid training data and selects them on target validation data.Source: Evaluation protocols, partitions, and… · p. 6
  • Transfer is limited to deep baselines with transferable weights and local or line observability, so it does not test fully topology-agnostic models.Leakage controls group episodes and source topologies, estimate dataset-level preprocessing only from training data, and check partition disjointness before scoring.Source: Evaluation protocols, partitions, and… · p. 6

Reference baselines and model selection

The reference suite spans trivial, conventional, feature-based, and deep-learning baselines, with fixed hyperparameters and a validation-based selection procedure for headline learned references.

Reference baselines and model selection

  • The suite combines a task-specific trivial predictor, a conventional overcurrent threshold for binary local fault detection, a random forest, and four waveform-based deep models.The deep models are an MLP, GRU, CNN, and ResNet; the random forest uses 200 trees and seven channel-wise summary statistics.Source: Reference baselines and model selection · pp. 6–7
  • Deep-model inputs are standardized using training-partition channel statistics, with per-window z-scoring added for multi-grid pretraining and transfer protocols.For fault localization, training-target standardization is reversed before evaluation on the original percentage-of-line scale.Source: Reference baselines and model selection · p. 7
  • The four deep baselines share AdamW training settings, use early stopping on validation loss, and are trained with five seeds without equalizing model capacity.No task-specific hyperparameter search is performed.Source: Reference baselines and model selection · pp. 6–7
  • The headline learned reference is selected by validation metric within each evaluation cell, then evaluated without using test partitions or shifted eval_grid results for selection.All candidate-baseline results are retained in the Supplementary Information.Source: Reference baselines and model selection · p. 7

Evaluation metrics and uncertainty

The benchmark specifies task-specific metrics and uncertainty reporting, while versioning and run records define which artifacts make results comparable and reproducible.

Evaluation metrics and uncertainty

  • Binary and multiclass tasks use balanced accuracy as the headline metric, with macro-F1 and accuracy as secondary metrics; detection tasks also report task-specific miss and false-alarm rates.For fault detection, those rates concern windows within fault episodes, not false trips during benign switching or other non-fault disturbances.Source: Evaluation metrics and uncertainty · p. 7
  • Multiclass balanced accuracy averages recall over supported classes, while frozen macro-F1 includes the full class vocabulary and assigns zero to unsupported classes.A companion macro-F1 averages over supported classes only; the analytical balanced-accuracy reference for K supported classes is 1/K.Source: Evaluation metrics and uncertainty · p. 7
  • Fault localization reports MAE as a percentage of line length, and detection latency is conditional on successful detection and includes the 50 ms observation window.Latency is the first correctly positive fault-active window’s start-time offset from fault inception plus 50 ms.Source: Evaluation metrics and uncertainty · p. 7
  • Five-seed comparisons are descriptive, not formal hypothesis tests, and their intervals do not establish equivalence or statistical superiority.The intervals capture training-run variation, not resampling uncertainty across episodes, operating conditions, or grids.Source: Evaluation metrics and uncertainty · p. 7

Benchmark versioning, reporting, and reproducibility

  • Results are indexed without an aggregate score, and run records include benchmark version, code and model details, seed, and leakage-control status.Changes to score-determining artifacts require a version change; v1.1.0 retains unchanged held_out splits and results from v1.0.0 while versioning topology-grouped multi_grid_pretrain splits and dependent results.Source: Benchmark versioning, reporting, and… · pp. 7–8

Results

Reference performance varies by function and grid; wider observability is not consistently beneficial, and shifted conditions produce task- and baseline-dependent results. Cross-grid transfer is strongest for detection, while localization remains difficult, and diagnostics expose failure modes that primary metrics alone do not show.

RQ1: Reference performance varies across functions and grids

  • Detection was comparatively strong (FD balanced accuracy 0.83–0.95), while localization was weakest (in-distribution MAE 16–24% of line length) and classification varied by grid.FC balanced accuracy ranged from approximately 0.86 on the Double Line grid to 0.52–0.67 on the CIGRÉ medium-voltage grid; EC ranged from approximately 0.64–0.78 across grids.Source: RQ1: Reference performance varies… · p. 8

RQ2: Observability affects protection functions differently

  • Wider observability had no monotonic benefit for FD or FC, while FL generally improved on high-voltage grids (Double Line MAE: 18.6% local, 15.9% global).For FC on the CIGRÉ medium-voltage grid, balanced accuracy decreased from 0.656 under local to 0.520 under global observability; the Double Line global estimate had a confidence-interval half-width of approximately 0.087.Source: RQ2: Observability affects protection… · p. 8, Figure 2 · p. 9

RQ3: Evaluation-family shifts are task dependent

  • Shifted eval_grid outcomes were task- and baseline-dependent: Double Line line-view FL MAE fell from 16.1% to 5.2%, while selected RF detection cells fell to approximately 0.50 balanced accuracy.The localization shift used a fixed operating point and a discrete five-position sweep, so the lower error does not establish improved robustness; conventional overcurrent also exceeded the selected learned reference in two shifted local-FD cases.Source: RQ3: Evaluation-family shifts are task… · pp. 8–9, Figure 2 · p. 9

RQ4: Cross-grid transfer differs by protection function

  • Across 72 matched source–target comparisons, confidence intervals overlapped in 66, and cross-grid pre-training gave no consistent final target-grid gain over target-only training.FD transferred most consistently, while localization transferred least effectively; fine-tuning reduced line-view localization error to approximately 8–14% on high- and extra-high-voltage grids, but error remained approximately 26% on the medium-voltage grid.Source: RQ4: Cross-grid transfer differs by… · pp. 9–10, Figure 3 · p. 10

Protection-relevant diagnostics identify failure modes

  • Diagnostics show that strong FD references still miss approximately 19–26% of in-distribution fault-active windows, while near-chance shifted results reflect lost discrimination rather than random behavior.The task excludes benign switching and other non-fault disturbances, so its false-alarm rates do not measure protection security against those events; five-seed comparisons are interpreted descriptively, not as statistical superiority.Source: Protection-relevant diagnostics… · p. 11

Discussion

The reference evaluation shows that protection performance and generalization depend on the task, measurement access, and evaluation conditions. Its findings support task-specific interpretation and careful qualification of transfer and security claims.

Discussion

  • The reference evaluation shows that performance conclusions depend on protection function, measurement access, evaluation family, and generalization protocol.Source: Discussion · p. 11

Implications for power system protection

  • Protection scores are task-specific: wider observability inconsistently helps detection and classification, localization benefits, and some shifted detectors fail where conventional overcurrent thresholds outperform them.False alarms are counted only in inactive windows within fault episodes, so these results do not establish comprehensive protection security.Source: Implications for power system protection · p. 11

Implications for generalization and transfer

  • Shifted eval_grid conditions can improve localization yet expose severe failures in selected detectors, so generalization claims should identify the physical and statistical changes evaluated.Source: Implications for generalization and… · p. 11
  • Fault detection is comparatively portable across grids, classification benefits more consistently from target-grid adaptation, and localization remains strongly dependent on target-grid examples.Source: Implications for generalization and… · p. 11
  • Fine-tuning improves zero-shot transfer, but under matched preprocessing it shows no consistent final-performance advantage over target-only training.Transfer studies should distinguish zero-shot performance, fine-tuning gains, and final target-grid performance.Source: Implications for generalization and… · pp. 11–12

Benchmark use and extensions

  • Benchmark extensions can add sensor non-idealities, missing or delayed measurements, topology-held-out evaluation, and synthetic-to-real transfer without invalidating existing evaluation cells.Reported comparisons should identify the protocol artifact versions associated with their evaluation cells.Source: Benchmark use and extensions · p. 12

Limitations and future directions

  • Evidence is synthetic; transfer excludes global observability, models omit grid metadata, and source-grid comparisons confound voltage, frequency, and corpus size.Source: Limitations and future directions · p. 12

Conclusion

EvEMTBench provides an open, reproducible benchmark for evaluating machine-learning-based power system protection across grids and generalization conditions. Its reference evaluation finds that performance does not transfer uniformly across tasks or conditions, motivating clearly defined, repeatable comparisons.

  • Cross-grid transfer is substantially stronger for fault detection than fault localization, while measurement value and shifted-condition failures also vary across tasks and conditions.Source: Conclusion · p. 12
  • EvEMTBench releases benchmark implementation, reference baselines, and evaluation resources to support comparison and further development.Source: Conclusion · p. 12
  • The benchmark makes generalization an explicit and repeatable evaluation problem for machine-learning-based power system protection.Source: Conclusion · p. 12

Data availability

The study’s EvEMTBench Dataset is publicly available through FAUDataCloud.

  • The EvEMTBench Dataset used in the study is publicly available through FAUDataCloud.Source: Data availability · p. 12
  • The dataset is described by Kordowich et al. in “A Simulation Based Dataset of Faults and Events for Machine Learning in Power Systems.”Source: Data availability · p. 12
  • The cited dataset description is identified as arXiv:2608.19777.Source: Data availability · p. 12

Use of generative AI and AI-assisted technologies

The authors report using ChatGPT and Claude to assist with drafting and language refinement of selected passages. They state that AI-assisted content was reviewed and that the authors remain responsible for the manuscript.

Author contributions statement

J.O. led benchmark conception, implementation, validation, analysis, data curation, visualization, and manuscript drafting. Other authors contributed to methodology and data generation, supervised and administered the study, and revised the manuscript.

Author contributions statement

Loading 2609.28149v1…