Source-linked AI summary
Real-bogus classification for the Zwicky Transient Facility using deep learning
Dmitry A. Duev, Ashish Mahabal, Frank J. Masci, Matthew J. Graham, Ben Rusholme, Richard Walters, Ishani Karmarkar, Sara Frederick, Mansi M. Kasliwal, Umaa Rebbapragada, Charlotte Ward
TL;DR
Large surveys need automated ways to separate astrophysical events from bogus detections in their high-volume image-subtraction streams. The paper presents braai, a deep-learning real/bogus classifier for ZTF, and reports low error rates alongside a state-of-the-art overall performance.
Problem
Large-scale surveys can detect millions of subtraction residuals nightly, while some bogus detections arise from phenomena that are difficult to model.
Source: §1 · p. 1Method
Braai is a cutout-image-based deep-learning real/bogus classifier for ZTF, supported by Kowalski for alert and light-curve access and Zwickyverse for data labeling.
Source: The Zwicky Transient Facility · p. 2Results
Braai is reported as state of the art; at a 0.5 score threshold it yields 1.1% FNR and below 3% FPR on 3,271 test examples.
Source: §2.2 · p. 5, §4 · p. 9Takeaways & Limitations
The collected data set is intended for future deep-learning work, including specialized classifiers, and the setup should be easy to reuse or extend for other surveys such as LSST.
Source: §4 · p. 9The real/bogus score is only one reliability metric, and ZTF science groups apply additional filtering using multiple alert columns.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Efficient automated detection of flux-transient, reoccurring flux-variable, and moving objects is increasingly important for large-scale astronomical surveys. We present braai, a convolutional-neural-network, deep-learning real/bogus classifier designed to separate genuine astrophysical events and objects from false positive, or bogus, detections in the data of the Zwicky Transient Facility (ZTF), a new robotic time-domain survey currently in operation at the Palomar Observatory in California, USA. Braai demonstrates a state-of-the-art performance as quantified by its low false negative and false positive rates. We describe the open-source software tools used internally at Caltech to archive and access ZTF's alerts and light curves (Kowalski), and to label the data (Zwickyverse). We also report the initial results of the classifier deployment on the Edge Tensor Processing Units (TPUs) that show comparable performance in terms of accuracy, but in a much more (cost-) efficient manner, which has significant implications for current and future surveys.
1 INTRODUCTION AND CONTEXT
Image subtraction reveals transient events but also produces bogus detections from image differences and unmodeled noise. Because surveys can generate millions of residuals nightly and some artifacts are hard to model, automated real/bogus classification is needed.
1 INTRODUCTION AND CONTEXT
- Image subtraction compares each epochal image with a reference after matching image characteristics such as point spread functions and depth.The resulting residuals can contain false positives from unmodeled image differences and noise, including registration and source-noise errors.Source: §1 · p. 1
- Large surveys may detect millions of subtraction residuals nightly, making automated separation of genuine events from bogus detections necessary.Some false positives arise from phenomena such as radiation hits and optical ghosts that lack proper statistical models, motivating machine-learning approaches.Source: §1 · p. 1
- Real/bogus classifiers score individual sources from 0.0 (bogus) to 1.0 (real) and have been adopted by several time-domain surveys.The cited examples include PTF, iPTF, Dark Energy Survey, Pan-STARRS, and HiTS.Source: §1 · pp. 1–2
The Zwicky Transient Facility
ZTF processes survey images into alerts for transient, variable, and moving objects. This paper presents braai, a deep-learning classifier for ZTF that improves on the original random-forest classifier.
The Zwicky Transient Facility
- ZTF can visit the visible sky north of −30° nightly and scans more than 3750 deg2 per hour with a 5σ r-band limit of 20.7 mag in a 30-second exposure during new moon.Its 576-megapixel camera has a 47 deg2 field of view.Source: The Zwicky Transient Facility · p. 2
- The ZSDS subtracts reference images from calibrated science exposures, matches image properties, and uses ZOGY for PSF matching and differencing.It detects positive and negative events in matched-filtered signal-to-noise images, extracts photometry and source features, then generates alert packets.Source: The Zwicky Transient Facility · p. 2
- ZTF’s initial real/bogus classifier used a random forest trained on source features from candidate-centered cutouts and measurements from science, subtracted, and reference images.Events are distributed through ZADS, with 10^5–10^6 detected on a typical night.Source: The Zwicky Transient Facility · p. 2
- Braai is a cutout-image-based deep-learning real/bogus classifier for ZTF, reported as state of the art and superior to the original random-forest classifier.The paper also describes Kowalski for archiving and accessing alerts and light curves, and Zwickyverse for data labeling.Source: The Zwicky Transient Facility · p. 2
2 BRAAI: A DEEP LEARNING FRAMEWORK FOR REAL-BOGUS CLASSIFICATION
The paper builds braai on a broad ZTF training set and describes the supporting data-access and labeling tools. Its VGG6 classifier achieves low test-set false-negative and false-positive rates at selected score thresholds.
2.1.1 Kowalski
- Kowalski provides standardized API, programmatic, and GUI access to ZTF alerts, light curves, and external catalogs; its Caltech instance stored over 30 TB, 125M+ alerts, and 2.5B+ light curves as of June 2019.Its MongoDB backend stores light curves per source, reducing the number of read operations needed to access them.Source: §2.1.1 · pp. 3–4
2.1.2 Data labelling
- Zwickyverse is an open-source labeling tool that integrates newly labeled sets into the training workflow; 2D-Moffat fits and residual images helped identify some bogus detections.Contentious examples were also inspected individually using the ZTF Alert Lab.Source: §2.1.2 · pp. 4–5
2.1 Data set
- The training set contains over 30,000 examples with an approximately 55%/45% real-to-bogus ratio and samples alert diversity across ZTF’s parameter space.Samples came from Kowalski queries, the GROWTH Marshal’s pre-labeled events, and Zooniverse data reserved for testing.Source: §2.1 · pp. 2–3, §2.1.3 · p. 5
2.1.3 Data diversity
- The training data have imbalances across filters, Galactic-plane location, subtraction polarity, date, FWHM, PSF magnitude, and SNR, which the authors planned to mitigate.The collected set was intended to adequately sample the ZTF alert parameter space.Source: §2.1.3 · p. 5
2.2 braai architecture and training
- Braai uses VGG6, a six-trainable-layer VGG18-like model with four convolutional and two fully connected layers; deeper ResNet18 and ResNet50 models showed no performance gain.The convolutional layers use 3×3 filters, with dropout applied after pooling layers and the second fully connected layer.Source: §2.2 · p. 5
- The model inputs independently L2-normalized science, reference, and difference cutouts stacked as 63×63×3 triplets; training used class weighting, random flips, early stopping, and a 81%/9%/10% data split.Training for 200 epochs on about 30,000 images took about 20 minutes on an Nvidia Tesla P100 GPU; training and validation accuracy were about 98%.Source: §2.2 · p. 5
- At a score threshold of 0.5, braai yields 1.1% FNR on 3,271 test examples while keeping FPR below 3%; at 0.65, both FNR and FPR are 1.7%.The paper reports no performance gain from the deeper ResNet18 and ResNet50 alternatives.Source: §2.2 · p. 5
3 CLASSIFIER PERFORMANCE
Braai is evaluated on volunteer-labeled examples and several real-alert populations, with fewer misclassifications than the RF classifier in the reported comparisons. Its Edge TPU implementation provides virtually the same threshold-0.5 performance while increasing processing throughput.
3.1 Zooniverse test set
- Volunteers classified science, reference, and difference images; after selecting examples with at least six ratings, inspection found about 30% of low-score bogus labels were real.The missed real events were mainly variable stars under challenging conditions, which volunteers could not assess without light curves.Source: §3.1 · pp. 5–6
3.2 Real events
- On 3,271 test examples, braai yielded 1.1% FNR and below 3% FPR at threshold 0.5; both rates were 1.7% at threshold 0.65.Figure 9 presents FNR and FPR as functions of the score threshold.Source: Figure 9 · p. 7
- At score threshold 0.5, braai correctly classified all 803 ZTF alerts originating from objects in densely populated sky regions, while RF had 113 misclassifications (14.1% FNR).Source: §3.2 · pp. 7–8, Figure 15 · p. 8
3.1 Zooniverse test set
- Braai produced more reliable scores than RF on the 734 real and 878 bogus examples in the Zooniverse test set.The two RF-score peaks around 0.5 reflect the score ranges selected for the Zooniverse campaigns.Source: §3.1 · p. 6, Figure 12 · p. 8
3.2 Real events
- At threshold 0.5, braai misclassified 18 of 2,633 human-vetted alerts (0.7% FNR), versus 282 (10.7% FNR) for the RF classifier.The alerts came from 921 objects vetted as real after programmatic science-group filters.Source: §3.2 · p. 6, Figure 13 · p. 8
- Below 1% FNR through threshold 0.7 while keeping FPR below 2%; at threshold 0.5, braai correctly identified 99.5% of supernova alerts versus 80% by RF.The test used 2,820 alerts from 140 supernovae detected in 2019 that were excluded from training.Source: Figure 16 · p. 8, §3.2 · p. 9
- At threshold 0.5, braai’s FNR stayed below 3% across all three asteroid brightness bins, while RF FNR rose from ∼5% to ∼40% for fainter objects.The evaluation covered 21,641 alerts from known Solar system objects observed in June 2019.Source: §3.2 · p. 9
3.3 Production deployment and Edge TPUs
- On an Edge TPU, braai achieved virtually the same threshold-0.5 performance as the full model and processed up to 1,200 triplets per second, 4–5× faster.The Raspberry Pi implementation processed a typical 200,000-alert night in about ten minutes.Source: §3.3 · p. 9
4 CONCLUSIONS
Braai achieves state-of-the-art real/bogus classification with a relatively simple deep-learning model and a large, representative, uncontaminated dataset. The authors plan to update the classifier as data and the processing environment change, while noting that separate classifiers by filter may not be necessary.
4 CONCLUSIONS
- A large, representative, uncontaminated dataset paired with a relatively simple deep-learning model achieves state-of-the-art real/bogus classification performance.Source: §4 · p. 9
- Retraining and deployment of new classifiers are planned as labeled data grow, especially for the i-band and low Galactic latitudes, and after hardware or processing changes.Source: §4 · p. 9
- The RB score is only one reliability metric; ZTF science groups additionally filter using multiple alert columns.Source: §4 · p. 9
- The dataset is intended for future deep-learning work, including specialized classifiers, and the setup should be easy to reuse or extend for surveys such as LSST.Source: §4 · p. 9
- Braai code and pretrained models are available as open-source software.Source: §4 · p. 9