SuperNNova: an open-source framework for Bayesian, Neural Network based supernova classification

• Spectroscopic follow-up is scarce relative to the thousands of supernovae discovered and the up to million transient alerts expected nightly, making fast, accurate photometric classification crucial.Source: §1 · p. 1
• Only a handful of existing methods classify partial light curves, although early classification helps prioritize spectroscopic and photometric follow-up resources.Source: §1 · p. 2
• Photometric samples can support rates and SN Ia cosmology, but contamination bias and poorly calibrated or unavailable prediction uncertainties remain concerns.Source: §1 · p. 2
• SuperNNova classifies raw photometric time series without template matching or costly feature engineering, and incorporates Bayesian neural networks and irregular observations without augmentation or fitting.Source: §1 · p. 2
• SuperNNova reports state-of-the-art classification on both complete and partial supernova light curves, with performance further enhanced by host-galaxy redshifts.Source: §1 · p. 2
• The framework is designed for supernova cosmology but can also be applied to light-curve classification tasks across time-domain astronomy.Source: §1 · p. 2

2.1 Simulations and available datasets

• The simulations use SNANA with updated DES models, SALT2 Ia spectral-energy-distribution models, and observing logs when available; Table 1 reports subtype counts for complete and SALT2-fitted samples.Source: §2.1 · pp. 2–3, Table 1 · p. 3
• The simulations contain 1,983,213 light curves, of which 881,969 were successfully fitted with SALT2; spectroscopic redshifts are available for only a subset.Source: §2.1 · p. 3
• Binary classification uses 912,691 balanced light curves, over an order of magnitude more than SPCC, split into 80% training, 10% validation, and 10% testing.Source: §2.1 · p. 3
• The complete dataset includes all class-balanced simulated light curves, whereas the SALT2-fitted subset is non-representative because it fails to explore supernova diversity.Source: §2.1 · p. 3

ROC curve

• AUC summarizes a binary classifier’s ROC curve, which plots true-positive rate against false-positive rate; 1 is perfect and 0.5 is random.Source: ROC curve · p. 3
• For binary classification, accuracy is correct predictions divided by total predictions; purity and efficiency measure the true-positive share of selected positives and the true-positive share of actual positives, respectively.Source: Accuracy, Purity and Contamination · p. 3
• Reported metric errors are the mean and one standard deviation across five runs with different random seeds.Source: Accuracy, Purity and Contamination · p. 3

2.3 Recurrent Neural Networks (RNNs)

• RNNs model sequences through time-delayed hidden-unit connections that can retain or discard past input information; LSTMs and GRUs add memory cells and gating mechanisms.Source: §2.3 · p. 3
• An RNN applies shared operations across sequence elements to produce hidden states, which can be mean-pooled into a fixed-length representation before softmax produces class probabilities.Source: §2.3 · p. 4, Figure 1 · p. 4

2.4 RNN architecture

• SuperNNova processes photometric time series with an RNN, condenses hidden states by mean pooling or concatenating each layer’s last state, then projects to class probabilities.Source: §2.4 · p. 4
• MC Dropout and BBB provide Bayesian training options, while traditional RNNs lack model-uncertainty estimates.Source: §2.4 · p. 4, §2.3 · p. 4

• Photometric light curves are the algorithm’s only required inputs, while host-galaxy redshift and its error can be repeated across time steps as additional inputs.Source: RNN training data · p. 4
• The data supports binary Ia-versus-non-Ia, ternary Ia/II/Ibc, and seven-way classification across Ia and core-collapse subtypes.Source: RNN training data · p. 4

RNN training details

• Observations within eight hours are grouped into nightly rows, with missing-filter values, elapsed-time features, and filter-availability encodings representing irregular observations.Inputs include g, i, r, z fluxes and errors; host-galaxy redshift and its error can also be supplied.Source: RNN training details · pp. 4–5
• Global normalization pools flux or error statistics across filters, whereas per-filter normalization computes those statistics separately after log transformation and standard scaling.The available choices also include no normalization.Source: RNN training details · p. 5
• RNNs use Adam, dropout, zero-padding, and masking; randomly truncated light-curves improve incomplete-curve classification, while data augmentation was not used.The paper also implements decay-on-plateau and 1-cycle learning-rate policies.Source: RNN training details · p. 5

2.5 Random Forest (RF)

• The Random Forest uses SALT2-fit light-curve features, so redshift can enter implicitly through the fit even without an explicit redshift input.Features include stretch, color, B-band magnitude, normalization, fit goodness, and observed magnitudes with errors.Source: §2.5 · p. 5

2.6 Convolutional Neural Networks (CNNs)

• The comparison CNN uses four 1D convolutional layers, collapses the sequence to a fixed-length representation, and projects it to a final prediction.The implementation is in PyTorch.Source: §2.6 · p. 5

3.1 Hyper-parameters

• Hyper-parameter variations had limited impact, with complete-light-curve accuracy averaging 94.48 with 1% standard deviation; the selected configuration used bidirectional LSTMs and two recurrent layers.Mean pooling and randomly truncated curves were chosen to improve early classification.Source: §3.1 · pp. 5–6

3.2 Normalization

• 96.12 ± 0.07 accuracy with global normalization exceeded 94.2 ± 0.2 without normalization, while per-filter normalization reached 96.0 ± 0.2.The authors note that per-filter normalization blurs color ratios, suggesting those ratios were not important for this sample's classification.Source: §3.2 · p. 6

3.3 Comparison with other methods

• 96.3 ± 0.4 accuracy and 0.995 ± 0.001 AUC were obtained by the Baseline RNN on the SALT2 fitted dataset for Ia vs. non-Ia classification.The evaluation used five randomized training, validation, and testing splits.Source: §3.3 · p. 6

3.3.1 Baseline RNN vs. CNN

• 94.53 ± 0.07 accuracy and 0.9887 ± 0.0002 AUC were reported for the CNN without redshift, while the Baseline RNN achieved higher accuracy on this task.At 0.2 of the dataset, CNN accuracy was 93.1 ± 0.2 without redshift, 97.1 ± 0.1 with photometric redshift, and 98.98±0.07 with spectroscopic redshift.Source: §3.3.1 · p. 6

3.3.2 Baseline RNN vs. RF

• 95.15 ± 0.07 accuracy and 0.9929 ± 0.0003 AUC were reported for the Random Forest without redshift; the RNN outperformed it on larger datasets at matched redshift information.RNN performance increased with dataset size while Random Forest performance remained almost constant; SALT2 features implicitly carry redshift information.Source: §3.3.2 · p. 6

3.3.3 Baseline RNN vs. another RNN architecture

• 92.4±0.9 accuracy without redshift and 96.8 ± 0.1 with photometric redshift were reported for the Baseline RNN trained on 32,222 light-curves, comparable to Charnock & Moss (2017) results on a different sample.The authors report up to a 4% accuracy increase from redshift for this sample size.Source: §3.3.3 · p. 7

3.4 Redshift, contamination and efficiency

• Without redshift, contamination was dominated by Ib and Ic supernovae, whose contributions fell from 2.5±0.2 and 1.3±0.1 to < 0.1% with host-galaxy redshifts.Contamination estimates use simulations incorporating known supernova rates and survey detection efficiencies; IIP, IIn, and IIL1 classification was barely affected by redshift information.Source: §3.4 · p. 7
• Figure 3 compares accuracy against training-set size for the RNN, CNN, and Random Forest across redshift settings; the Baseline RNN achieves greater accuracy than the other classifiers at matched explicit redshift information.The axes show data fraction and supernova count, with five-run standard-deviation error bars.Source: Figure 3 · p. 7
• 99.61 ± 0.09 selection efficiency with host-galaxy redshift information compared with 97.3 ± 0.4 without it was reported, with efficiency nearly constant across simulated redshift.The authors state that such efficiency can enable probing high-redshift supernova populations and affect selection biases in cosmology.Source: §3.4 · p. 7
• 95.4 ± 0.5 purity without redshift increased to 99.49±0.05 with redshift, with contamination below 1% in the latter case.The paper compares this contamination level with the range for spectroscopically classified samples used in cosmological and astrophysical analyses.Source: §3.4 · p. 7

3.5 Early light-curve classification

• Figure 4's columns mark days around peak, rows contrast raw and realistic observing-night counts, and colors encode accuracy for the complete dataset without redshift.The realistic count excludes epochs more than two weeks before peak.Source: Figure 4 · p. 8
• The complete dataset includes light-curves not successfully fitted with SALT2, increasing the training-set size and diversity; 43% of it was used for a similar-sized comparison to the SALT2 fitted sample.The stated fraction was selected to provide a training set of similar size to the SALT2 fitted one.Source: §3.5 · p. 8
• Using only epochs within 14 days before peak as a realistic count, accurate classification began with an average of 2.4±1.2 to 3.3±1.4 photometric epochs.The broader average was about five observing nights to reach accuracy above 80%; multicolor observations within a night provide useful information.Source: §3.5 · p. 8
• Before maximum light, baseline RNN accuracy exceeded 83% across datasets, and adding host-galaxy redshifts raised accuracy by as much as 12 points.The section summary reports 83.6±0.6 to 97.0±0.2 accuracy for the SALT2 fitted dataset before or at maximum light.Source: §3.5 · p. 8
• For the complete dataset, spectroscopic-redshift accuracy was lower than photometric-redshift accuracy, possibly because of selection bias in the spectroscopic-redshift subset.This difference was not observed in the SALT2 fitted sample.Source: §3.5 · p. 8
• Early or near-peak Ia classifications had contamination contributions of 9.4 ± 0.3% from Ib and 3.3 ± 0.2% from Ic supernovae.IIP and IIL2 each contributed around 2%, potentially because characteristic features such as the type IIp plateau appear only after maximum light.Source: §3.5 · p. 8

3.6 Classifying many supernovae types

• Complete-light-curve ternary and seven-way classification achieves unprecedented accuracy, while early classification is harder and improves with redshift information.The tasks use 318,820 light-curves per type for ternary classification and 104,158 per type for seven-way classification.Source: §3.6 · p. 8
• Multiclass accuracy is much lower than binary accuracy, as splitting core-collapse supernovae into subclasses adds complexity.Source: §3.6 · p. 8
• For seven-way classification, type IIP supernovae dominate contamination of the SN Ia sample.Source: §3.6 · p. 9
• Early classification is severely affected by adding targets; a two-step classifier is suggested as a possible refinement, but improving multiclass predictions is out of scope.Source: §3.6 · p. 9

4.1 Variational Inference

• Variational inference approximates the intractable weight posterior with a parametric distribution and minimizes divergence while balancing prior regularization against data likelihood.Source: §4.1 · p. 10

4.2 MC dropout

• 96.87 ± 0.09 without redshift and 98.8 ± 0.1 with host photometric redshift; MC dropout uses time-locked row masks and 50 sampled predictions.Source: §4.2 · p. 10

4.3 Bayes by Backprop (BBB)

• 96.85 ± 0.05 without redshift and 98.72±0.04 with host photometric redshift; BBB samples weights from learned Gaussian distributions during training.Source: §4.3 · p. 10

4.4 BRNNs Uncertainties

• SuperNNova models epistemic uncertainty but not aleatoric uncertainty, which is already represented through simulated flux-measurement errors.Source: §4.4 · p. 11

4.4.1 Uncertainty evaluation metrics

• Mean model uncertainty averages per-light-curve uncertainty over a test set, while entropy proxies confidence, with lower entropy indicating more confident predictions.Source: §4.4.1 · p. 11

4.4.2 Uncertainty evaluation

• 0.005 and 0.007 are Δ⟨σ̂⟩ for models trained with 43% versus the whole dataset; both methods show positive ΔH > 0.01, while MC dropout uncertainties are twice BBB's.Source: §4.4.2 · p. 11
• Bayesian RNN uncertainties are consistent with expected epistemic behavior, but the authors say their differing magnitudes need further study.Source: §4.4.2 · p. 11

5.1 Calibration

• For the complete dataset without redshift, calibration dispersion fell from 0.006±0.001 (Baseline RNN) to 0.004±0.002 (MC dropout) and 0.0005±0.0004 (BBB).On the SALT2 fitted dataset, Random Forest dispersion was 0.025 ± 0.002 over five runs; RNN dispersion was an order of magnitude lower.Source: §5.1 · p. 12

5.2 Representativeness

• Training on non-representative SALT2 data reduced no-redshift accuracy by 1%; Bayesian RNN uncertainty-gap metrics were positive (∆⟨bσ⟩>0.004 and ∆H > 0.01).The comparison used the SALT2-fitted dataset as non-representative and the complete dataset as representative.Source: §5.2 · p. 13

5.3 Out of distribution light-curves (OOD)

• Out-of-distribution events were rarely classified as Ia; reverse and shuffled curves had the largest binary false-Ia rates for MC dropout (4.8% and 4.1%), while BBB rates were <3%.In ternary and seven-way classification, OOD events were most often assigned to types II, IIn, IIp, or IIL1.Source: §5.3 · p. 14
• OOD uncertainty behavior was mixed rather than uniformly high, with some seven-way predictions showing high single-class probabilities alongside large uncertainties.The entropy gap could be negative for some OOD types and model configurations.Source: §5.3 · p. 14, Figure 10 · p. 15

5.4 Type Ia supernova cosmology

• With host-galaxy redshifts, selected-sample contamination was < 2%; without redshift, selected samples tracked parent redshift, color, and stretch distributions and reached redshifts > 0.9.The redshift-informed contamination was described as within the range of spectroscopically classified samples.Source: §5.4 · p. 15
• Raising the MC dropout selection threshold to > 0.9 reduced misclassified core-collapse supernovae by a factor of four while reducing correctly classified Ia by < 5%.The authors also report that discarding events with Hubble-diagram uncertainties > 0.05 produced a similar decrease in core-collapse supernovae as raising the threshold to 0.9.Source: §5.4 · p. 15

6 SUPERNNOVA RESOURCES

• The reported setup used an 8-core Intel i7-7700K CPU, 15GB RAM, and an NVIDIA GeForce 1080Ti GPU.Database formatting and normalization were performed once for the dataset and took less than half an hour.Source: §6 · p. 15
• Training about 5,000 light-curves per epoch took 1 minute on GPU for the Baseline RNN, 1.5 minutes for MC dropout, and 2 minutes for BBB.Baseline RNN convergence took 1.5 hours on GPU or 15 hours on CPU.Source: §6 · p. 15
• BRNN inference classified tens to hundreds of light-curves per second, while Baseline RNN inference reached thousands per second.BRNN speed measurements included 50 probability-distribution samples; Table 4 reports classification speeds by model and CPU/GPU.Source: §6 · p. 15

7 DISCUSSION AND CONCLUSIONS

• The authors released SuperNNova with documentation, test sets, and a Docker environment, and say it can be adapted to other time-domain astronomy classification problems.Source: §7 · pp. 16–17
• Type Ia versus non-Ia accuracy reached 95% for whole light-curves and 98% with host-galaxy information; ternary and seven-way accuracies exceeded 90% with host-galaxy information.Source: §7 · p. 16
• Before maximum peak, SuperNNova discriminated Ia from non-Ia light-curves with accuracy up to 86% without redshift and higher than 90% with host-galaxy redshifts, classifying thousands of light-curves per second.For classification into 3 to 7 targets, accuracy reached around 63% near maximum light.Source: §7 · p. 17
• The RNNs were better calibrated than Random Forest on the same dataset, and the authors emphasize diagnosing calibration because it depends on algorithm and dataset.Source: §7 · p. 17
• The Bayesian RNNs provided improved uncertainty estimates, calibration, and out-of-sample behavior over standard neural networks, with uncertainties reflecting prediction confidence and expected epistemic behavior.Source: §7 · p. 17
• SuperNNova produced photometric Ia samples with < 2% contamination and found higher selection efficiency and better population sampling than a recent DES spectroscopic analysis; tested OOD curves were mostly rejected.The authors say these uncertainty estimates may be used in cosmology or follow-up optimization frameworks.Source: §7 · p. 17

APPENDIX: BEHAVIOUR OF OOD IN MNIST

• The MNIST experiments compare standard, MC Dropout, and BBB networks on handwritten digits classified into ten digit classes.The recurrent experiment feeds images as sequences of rows, with columns as input features.Source: APPENDIX: BEHAVIOUR OF OOD IN MNIST · p. 18
• For feedforward MC Dropout and BBB, rotation increases prediction variance, while random images yield concentrated, low-variance probabilities.The authors interpret the differing responses as uncertainty about rotated images that may resemble training data, versus confidence that randomized images are unfamiliar.Source: APPENDIX: BEHAVIOUR OF OOD IN MNIST · p. 18, Figure 2 · p. 19, Figure 3 · p. 19
• Standard recurrent networks behave like their feedforward counterpart and make even more confident predictions on random and rotated images.The recurrent inputs treat image rows as time steps and columns as features.Source: APPENDIX: BEHAVIOUR OF OOD IN MNIST · p. 18, Figure 4 · p. 19
• For Bayesian recurrent networks, random inputs can yield high-confidence single-class predictions with MC Dropout or large variance across classes with BBB.The authors caution that recurrent Bayesian predictions remain at risk of collapse on unrelated data, possibly exacerbated by large flux variations and persisting normalization artifacts.Source: APPENDIX: BEHAVIOUR OF OOD IN MNIST · pp. 18–19, Figure 5 · p. 19, Figure 6 · p. 19