Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning

• Random forests were the leading method for both >M1 and >C1 flare prediction in this sample.Source: B K. Florios · p. 2
• For >M1 forecasts at a 15% probability threshold, the best method achieved ACC=0.93(0.00), TSS=0.74(0.02), and HSS=0.49(0.01).Source: B K. Florios · p. 2
• For >C1 forecasts at a 35% probability threshold, the best method achieved ACC=0.84(0.00), TSS=0.60(0.01), and HSS=0.59(0.01).Source: B K. Florios · p. 2

• Flare radiation can harm space infrastructure, instruments, and personnel, making forecasting part of space-weather forecasting.Source: §1 · p. 2
• Forecasts use near-real-time SHARP data over 2012–2016, with binary outcomes for whether either flare class occurs in the next 24 hours.Source: §1 · p. 3
• The study combines novel magnetogram-based predictors with MLP, SVM, and RF methods to forecast >M1 and >C1 flares.Source: §1 · p. 3
• The authors consider both flaring and non-flaring active regions, contrasting their N=23,134 sample with Liu et al. (2017), which used 845 flaring regions.Source: §1 · p. 3

2.1. Data

• SHARP data are matched with GOES flare records over the following 24 hours, and the sample uses NOAA active regions sampled every three hours from 2012 to 2016.Source: §2.1 · p. 4

2.2. Predictors

• The thirteen predictors combine previously proposed and new measures derived from line-of-sight and vector photospheric magnetograms.Source: §2.2 · p. 4

2.2.1. Magnetic Polarity Inversion Line (TLMPIL)

• The total length of MPIL* segments is used to quantify inversion-line structure, where MPIL* denotes segments with strong vertical-field gradients and strong horizontal fields.Source: §2.2.1 · p. 5

2.2.2. Decay Index (DI)

• The decay-index predictor uses Lhs/hmin for each MPIL and retains the peak value to represent an active region’s highest eruptive potential.Source: §2.2.2 · pp. 5–6

2.2.3. Gradient-weighted integral length of the neutral line (WLSG)

• WLSG integrates the horizontal gradient of the vertical magnetic field along neutral-line segments selected for strong horizontal field.Source: §2.2.3 · p. 6

2.2.4. Ising Energy (IsinEn1, IsinEn2)

• Ising energy measures magnetic complexity through interactions between opposite-polarity elements, with a new FLARECAST variant based on flux-partitioned elements.Source: §2.2.4 · pp. 6–7

2.2.5. Fourier Spectral Power Index (FSPI)

• FSPI fits a power law to the one-dimensional spectral density over the assumed 2–20 Mm turbulent inertial range.Source: §2.2.5 · p. 7

2.2.6. Schrijver’s R value (logR)

• Schrijver’s R value sums unsigned photospheric flux near strong-field MPILs, which serve as a proxy for free magnetic energy available for flare release.Source: §2.2.6 · pp. 7–8

3. Machine Learning Algorithms and Conventional Statistics Models

• MLPs, SVMs, and RFs represent neural-network, kernel-based, and tree-based approaches, respectively, and are evaluated against conventional statistical methods.Source: §3 · p. 8, §3.5 · p. 12

3.1. Multi-Layer Perceptrons

• An MLP maps inputs through hidden layers to outputs, and sufficient hidden nodes allow approximation of continuous nonlinear functions.Source: §3.1 · pp. 8–9, Figure 2 · p. 9

3.1.1. Classification Networks

• For binary classification, the MLP adjusts its weights to minimize negative log-likelihood, with predictor inputs and binary targets defining the training examples.Source: §3.1.1 · pp. 9–10

3.2. Support Vector Machines

• The study uses C-SVC with an RBF kernel; auxiliary runs also test class-specific regularization weights for imbalanced >M1 data.Source: §3.2 · pp. 10–11

3.3. Random Forests

• RF builds an ensemble of trees from bootstrap samples and randomly selected variables, then combines tree votes for classification.Source: §3.3 · p. 11

3.4.1. Multi-layer Perceptrons; 3.4.2. Support Vector Machines; 3.4.3. Random Forests

• Implementation uses entropy-based MLP training with tuned hidden-node counts, probability estimates for SVMs, and RF predictor-importance output.Source: §3.4.1 · p. 12, §3.4.2 · p. 12, §3.4.3 · p. 12

3.5. Conventional Statistics Models

• The conventional-statistics baselines are linear, probit, and logit regression, implemented with R’s lm and glm functions.Source: §3.5 · p. 12

4. Data preparation, Results and Discussion

• The experiments predict binary >M1 and >C1 events within 24 hours using ML and statistical methods, with skill-score profiles, ROC curves, and reliability diagrams for comparison.The statistical methods are linear, probit, and logit regression; the ML methods are MLP, SVM, and RF.Source: §4 · pp. 12–13

4.1. Data Pre-processing

• The 13 predictors come from seven quantities computed using line-of-sight or radial magnetograms, across 23,134 observations split evenly into training and test sets for 200 replications.All algorithms use identical splits, and scores are computed on the testing set.Source: §4.1 · p. 13

4.2. Tuning of ML algorithms

• MLP and SVM hyperparameters were tuned by 10-fold cross-validation on training data; after tuning, both improved significantly, while default RF settings were satisfactory.Tuning focused especially on >M1 flares, which were harder to predict than >C1 flares.Source: §4.2 · p. 14

4.3. Comparison Metrics

• ACC, TSS, and HSS assess categorical forecasts, while ROC curves, reliability diagrams, Brier scores, Brier skill scores, and AUC assess probabilistic performance.The study cautions that ACC can be artificially high for rare >M1 flares, making TSS and HSS more suitable; TSS is invariant to event frequency.Source: §4.3 · pp. 14–16

4.4.1. Prediction of >M1 Flare Events Using Machine Learning

• In one reported case, peak scores are TSS=0.780 and HSS=0.587; its ROC curve is better than those of MLPs and SVMs, and its reliability diagram shows mild under-prediction above 0.2.For the MLP, 26 hidden neurons cluster probability estimates around 0 and 1, while its reliability diagram shows over-prediction above 0.4. With C2/C1=20, weighted SVM improves ROC performance to maximum TSS=0.718, but its overall forecasting ability as measured by SSP and RD remains worse than the MLP.Source: §4.4.1 · p. 16

4.4.2. Prediction of >M1 Flare Events Using Statistical Models.

• For >M1 statistical forecasts, probit is slightly better than linear regression across skill profiles, ROC curves, and reliability, while logit behaves similarly to probit.The reliability differences between probit and linear regression are mostly within error bars.Source: §4.4.2 · p. 17

4.4.3. Monte Carlo Simulation for >M1 Flares; 4.5.3. Monte Carlo Simulation for >C1 Flares

• For >M1, RF at 15% yields ACC=0.93±0.00, TSS=0.74±0.02, and HSS=0.49±0.01; for >C1, RF at 35% yields ACC=0.84±0.00, TSS=0.60±0.01, and HSS=0.59±0.01.Across 200 replications, RF's HSS rises while its TSS falls over thresholds of 10%–25% for >M1 and 30%–40% for >C1.Source: §4.4.3 · p. 17, §4.5.3 · p. 22

4.5.1. Prediction of >C1 Flare Events Using Machine Learning

• RFs’ ROC curve is better behaved than the previous two methods (maximum TSS=0.615), and its RD is arguably best alongside MLP; SVM’s ROC is less satisfactory (maximum TSS=0.566).The SVM reliability diagram shows systematic under-prediction for most of the forecast-probability range.Source: §4.5.1 · p. 20

4.5.2. Prediction of >C1 Flare Events Using Statistical Models

• For >C1 statistical forecasts, logit has the best-behaved ROC and reliability diagrams, while linear and probit regression show over- or under-prediction patterns.Their maximum TSS values are 0.567 for logit, 0.562 for linear regression, and 0.566 for probit regression.Source: §4.5.2 · pp. 21–22

4.5.3. Monte Carlo Simulation for >C1 Flares

• For >C1, the maximum HSS is 0.60 for RF at 40%, whereas the maximum TSS is 0.61 at 30%; the authors present 35% as an appealing compromise.The authors report that jointly optimizing TSS and HSS does not yield a clear optimal threshold.Source: §4.5.3 · p. 22

4.6. Assessment of Prediction Methods and Predictor Strength

• logR and WLSG ranked highest for both flare classes, followed by Ising energy and TLMPIL..Source: §4.6 · p. 22
• Daily forecasts at 00:00 UT retained similar average skill to forecasts issued eight times daily, but had higher associated uncertainties.The comparison averages over 200 Monte Carlo iterations.Source: §4.6 · p. 22
• For >C1 prediction, ML methods made up 71% of the top 21 alternatives, versus 29% for conventional statistics; for >M1, the split was 62% versus 38%.The ranking aggregates ACC, TSS, and HSS across classifier-threshold alternatives.Source: §4.6 · p. 23
• With very low sample sizes, ML showed no advantage over conventional statistics; the study indicates N > 2,000 for K = 13 is needed for proper ML training, especially for >M1.Source: §4.6 · p. 23

4.7. Statistical Tests for Random Forest vs.

• RF outperformed the second-best method in all 8/8 TSS and HSS comparisons at thresholds maximizing either metric.The second-best method was MLP in 4/8 comparisons.Source: §4.6 · p. 22, §4.7.1 · p. 24

4.7.2. Calculation of AUC and Brier Skill Scores

• For >M1, RF led BS=0.0266 and BSS=0.4163, while RF AUC=0.9556 exceeded PR=0.9392 and LG=0.9391.Means are averaged over 200 Monte Carlo replications.Source: §4.7.2 · p. 24, Table 5 · p. 27
• For >C1, RF achieved the best BS=0.1074, BSS=0.4426, and AUC=0.8927.Other methods except SVM followed closely on AUC.Source: §4.7.2 · p. 24, Table 6 · p. 28

• Ahmed et al. (2013) reported >C1 operational forecasts with TSS=0.445 and HSS=0.539 using a neural-network variant at a 50% threshold.Source: §4.8 · p. 28
• Song et al. (2009) used a hand-picked 230-event sample with nonrepresentative flare rates; Yuan et al. (2010) used the same dataset in a cascading classifier.For C-class flares, Yuan et al. reported TSS=0.22 for logistic regression, 0.08 for SVM, and 0.09 for their cascade; Li et al. (2008) reported ACC=57.02% for SVM and 63.91% for SVM-KNN on a 2002 test year.Source: §4.8 · pp. 28–29
• Yu et al. (2009) reported TSS=0.651 for C4.5 and TSS=0.667 for LVQ with 45-observation sliding windows; HSS could not be recovered from their reported rates.Source: §4.8 · p. 29
• Colak and Qahwaji’s (2009) two-neural-network ASAP system reported HSS=49.3% for C-class and HSS=47% for M-class flares over 24 hours.Source: §4.8 · p. 29
• Wang et al. (2008) reported ACC=69% for >M1 forecasting, but sampled only flaring active regions and used a 48-hour forecast window.Their results do not permit comparison of skill scores.Source: §4.8 · pp. 29–30
• Against Bobra and Couvidat (2015), this study’s RF results were competitive: ACC=0.93±0.00, TSS=0.74±0.02, and HSS=0.49±0.01.Their SVM reported ACC=0.924±0.007, TSS=0.761±0.039, and HSS2=0.517±0.035; the samples and train/test splits differed.Source: §4.8 · p. 30
• For >C1 flares, Boucheron et al. (2015) reported TSS=0.55 and HSS=0.46, while Al-Ghraibah et al. (2015) reported approximately 0.50 and 0.40, respectively.The latter results were for static images without temporal information.Source: §4.8 · p. 30

• The models used SDO/HMI SHARP-based predictors and MLP, SVM, and RF methods; the sample covered all calendar days from 2012–2016 with a 3-hour predictor cadence.Source: §5 · p. 31
• At thresholds of 15% for >M1 and 35% for >C1, RF achieved ACC=0.93 and 0.84, TSS=0.74±0.02 and 0.60±0.01, and HSS=0.49±0.01 and 0.59±0.01, respectively.The results average 200 Monte Carlo replications; the sample contained 1,108 >M1 and 6,029 >C1 events.Source: §5 · p. 31
• For >M1 prediction, SVM and MLP need further hyperparameter tuning to produce results comparable with RF; all tested methods performed acceptably for >C1, where RF remained best.The Monte Carlo results were robust across different training/testing realizations.Source: §5 · p. 31
• Schrijver’s R and WLSG ranked among the most important predictors, followed by Ising energy and TLMPIL, consistent with the reported importance of strong, highly sheared magnetic polarity inversion lines.Source: §5 · p. 32
• RF skill continued improving as more predictors were added, while MLP and SVM gains were slight beyond about six predictors.The authors suggest additional viable predictors may further improve forecasting.Source: §5 · p. 32
• The present analysis used 3-hour predictor cadence; planned work would test finer cadence down to SHARP’s inherent 12-minute cadence and explore recurrent neural networks.Source: §5 · p. 32

• Fisher score and random-forest Gini importance were calculated for each predictor in both >M1 and >C1 flare prediction.Higher Fisher scores and Gini importance values indicate greater predictor importance.Source: A. Importance of predictors for flare… · p. 33
• The two importance measures correlated at r=0.7441 for >C1 and r=0.7535 for >M1; Schrijver’s R variants and WLSG ranked among the leaders in both cases.For >M1, TLMPILBr was also among the top four predictors.Source: A. Importance of predictors for flare… · p. 33, Figure A.1 · p. 35, Figure A.2 · p. 36

B. Prediction Models Resulting From Ranking the Predictors

• Backward elimination ranked 13 predictors by Fisher score and evaluated models containing 2–12 predictors across three ML and three conventional-statistics methods.In this appendix, MLP used four hidden nodes, and SVM γ and cost settings matched the full 13-predictor models.Source: B. Prediction Models Resulting From… · p. 37
• Above six predictors, RF outperforms the other ML methods for >C1 and >M1 flares; below six, it has no special advantage for >C1 and is worse than MLP for >M1.For >C1, the text identifies the six most important predictors as the cutoff; for >M1, it says the cutoff seems to be six.Source: B. Prediction Models Resulting From… · p. 37, Figure B.1 · p. 38, Figure B.2 · p. 39
• LM had the weakest forecasting ability, while PR and LG generally scored similarly and had flatter skill-score profiles than MLP and SVM.The flatter conventional-method profiles were reported across both flare classes.Source: B. Prediction Models Resulting From… · p. 37
• For K < 6 all methods perform similarly, while for K > 6 and N ≥ 10,000 ML methods—especially RF—perform better.For very small samples, N < 2,000, the authors also report similar performance and suggest conventional statistics methods for parsimony.Source: B. Prediction Models Resulting From… · p. 37

C. Validation Results When Predictions Are Issued Only Once a Day (at Midnight)

• The hybrid method trains at a three-hour cadence and tests at a 24-hour cadence, both with a 24-hour forecast window.The authors prefer this to training at a 24-hour cadence, which they say would leave models under-trained given the limited training sample size.Source: C. Validation Results When Predictions… · p. 40
• RF remains the best overall method in the midnight-only scenario, although uncertainty increases because testing uses one rather than eight predictions per day.For >M1, BS and AUC change little on average from the baseline, while BSS shows more significant differences; for >C1, BSS does not change significantly.Source: C. Validation Results When Predictions… · p. 40, Table C1 · p. 41, Table C2 · p. 41
• For >M1, the mean flare frequency is approximately 5%, compared with approximately 25% for >C1; the authors link this difference to >C1 BSS remaining similar across scenarios.The authors say the underlying climatology seems similar between baseline and midnight-only predictions for >C1.Source: C. Validation Results When Predictions… · p. 40
• For midnight-only RF forecasts, >M1 scores were ACC=0.93±0.01, TSS=0.73±0.04, and HSS=0.47±0.03 at 15%; >C1 scores were ACC=0.85±0.01, TSS=0.63±0.02, and HSS=0.61±0.02 at 35%.The authors report that issuing predictions only at midnight does not change ACC, TSS, and HSS much with respect to probability threshold.Source: C. Validation Results When Predictions… · p. 40, Table C3 · p. 42, Table C4 · p. 43

D. Concluding Remarks on ML versus Statistical Methods for Flare Forecasting

• The weighted-sum ranking compares 126 classifiers from six methods across a 5% probability-threshold grid, using ACC, TSS, and HSS.The three metrics are normalized across alternatives to a [0, 1] range before ranking.Source: D. Concluding Remarks on ML versus… · p. 44
• Among the top 16.6% of classifiers, ML methods account for 71% for >C1 and 62% for >M1, versus 29% and 38% for statistical methods, respectively.The frequency comparison is summarized in Figure D.1.Source: D. Concluding Remarks on ML versus… · p. 44, Figure D.1 · p. 47
• RF occupies all six best ranking positions for >C1 in Table D1 and appears seven times among the top 21 classifiers for each flare class.The top 21 correspond to the top 16.6% of classifiers; RF accounts for 33.3% of this group for both >C1 and >M1.Source: D. Concluding Remarks on ML versus… · p. 44, Table D1 · p. 45, Table D2 · p. 46