Source-linked AI summary
Automated Transient Identification in the Dark Energy Survey
D. A. Goldstein, C. B. D'Andrea, J. A. Fischer, R. J. Foley, R. R. Gupta, R. Kessler, A. G. Kim, R. C. Nichol, P. Nugent, A. Papadopoulos, M. Sako, M. Smith, M. Sullivan, R. C. Thomas, W. Wester, R. C. Wolf, F. B. Abdalla, M. Banerji, A. Benoit-Lévy, E. Bertin, D. Brooks, A. Carnero Rosell, F. J. Castander, L. N. da Costa, R. Covarrubias, D. L. DePoy, S. Desai, H. T. Diehl, P. Doel, T. F. Eifler, A. Fausti Neto, D. A. Finley, B. Flaugher, P. Fosalba, J. Frieman, D. Gerdes, D. Gruen, R. A. Gruendl, D. James, K. Kuehn, N. Kuropatkin, O. Lahav, T. S. Li, M. A. G. Maia, M. Makler, M. March, J. L. Marshall, P. Martini, K. W. Merritt, R. Miquel, B. Nord, R. Ogando, A. A. Plazas, A. K. Romer, A. Roodman, E. Sanchez, V. Scarpine, M. Schubnell, I. Sevilla-Noarbe, R. C. Smith, M. Soares-Santos, F. Sobreira, E. Suchyta, M. E. C. Swanson, G. Tarle, J. Thaler, A. R. Walker
TL;DR
Difference-image surveys face growing data volumes and artifacts that complicate timely transient discovery. This paper presents autoScan, a supervised Random Forest classifier developed for DES-SN. In the first-season application, it reduced candidates for human scanning while retaining more than 99 percent of fake-SN efficiency at the selected threshold.
Problem
Time-domain surveys face rising image-data volumes and difference-image artifacts, while needing to identify promising transients for follow-up.
Source: §1 · pp. 2, 4Method
The paper develops autoScan, a supervised classifier, using larger training data, generative training examples, and new classification features.
Source: §1 · p. 5Results
A factor of 13.4 fewer candidates reached scanning while fake efficiency remained above 99.0 percent at τ = 0.5.
Source: §4.4 · p. 20Takeaways & Limitations
The authors conclude that supervised machine-learning artifact rejection has improved source selection efficiency, while future surveys still need greater reliability, consistency, and transparency.
Source: §5 · p. 20Nearly 500,000 training artifacts were human-scanned, and new surveys need unseen data drawn from the same feature distributions as training data.
Source: §5.1 · p. 22
Abstract
from arXiv · showhide
We describe an algorithm for identifying point-source transients and moving objects on reference-subtracted optical images containing artifacts of processing and instrumentation. The algorithm makes use of the supervised machine learning technique known as Random Forest. We present results from its use in the Dark Energy Survey Supernova program (DES-SN), where it was trained using a sample of 898,963 signal and background events generated by the transient detection pipeline. After reprocessing the data collected during the first DES-SN observing season (Sep. 2013 through Feb. 2014) using the algorithm, the number of transient candidates eligible for human scanning decreased by a factor of 13.4, while only 1 percent of the artificial Type Ia supernovae (SNe) injected into search images to monitor survey efficiency were lost, most of which were very faint events. Here we characterize the algorithm's performance in detail, and we discuss how it can inform pipeline design decisions for future time-domain imaging surveys, such as the Large Synoptic Survey Telescope and the Zwicky Transient Facility. An implementation of the algorithm and the training data used in this paper are available at http://portal.nersc.gov/project/dessn/autoscan.
1. Introduction
Time-domain surveys must find valuable transients amid growing data volumes and numerous difference-image artifacts, while producing follow-up targets quickly. The paper presents autoScan, a supervised classifier developed for DES-SN, and describes how its training data and features inform future survey frameworks.
1. Introduction
- Difference images contain legitimate variable sources alongside subtraction and single-epoch artifacts that can vastly outnumber scientifically valuable detections.The resulting artifact burden forces detection thresholds above what Gaussian fluctuations alone would suggest.Source: §1 · p. 2
- DES-SN produced up to 170 gigabytes of raw imaging data nightly, and scanning nearly one million first-season detections would take almost a week uninterrupted.The survey routinely needed to choose roughly 10 promising candidates from hundreds, most of which were artifacts.Source: §1 · p. 4
- Automated hard cuts are easy to interpret but handle feature correlations poorly and select inefficiently as feature-space dimensionality grows.The paper contrasts this with supervised classifiers that infer mappings from numeric features to labeled classes.Source: §1 · p. 4
- autoScan is a supervised classifier developed for DES-SN to address transient and moving-object identification on difference images.The work reports methodology intended to inform similar frameworks for future time-domain surveys.Source: §1 · p. 5
- The authors extend prior work to a newer, larger data set and describe improving selection efficiency through larger training sets, generative training data, and new features.They identify LSST and ZTF as future-survey contexts for this methodology.Source: §1 · p. 5
2. The Dark Energy Survey and Transient Detection Pipeline
DES is a ground-based dark-energy experiment with a wide-area survey and a time-domain supernova program. The DES-SN pipeline extracts detections from subtracted images, groups detections into candidates, and applies a multi-night requirement before visual examination and follow-up.
2. The Dark Energy Survey and Transient Detection Pipeline
- DES combines a 5,000 deg^2 wide-area survey in five filters with DES-SN, which covers ten fields in four filters.The data described in the paper come from the first season of DES science operations, August 2013 through February 2014.Source: §2 · pp. 5–6
- After image subtraction and source extraction, detections matching within 1 arcsecond form raw candidates, and candidates meeting NUMEPOCHS become science candidates.The multi-night requirement rejects Solar System objects that move substantially between nights; science candidates are eligible for visual examination and spectroscopic follow-up.Source: §2 · p. 6
3. Classifier Development
The classifier-development section organizes the account around autoScan’s training data, features, and core classification algorithm.
3. Classifier Development
- The section covers autoScan’s training data set, classification features, and selection, properties, and optimization of its core algorithm.Source: §3 · p. 6
3.1. Training Data
The training set needed examples from both artifact classes and adequate coverage of their feature distributions, but visually scanned non-artifacts were scarce early in DES-SN. The authors combined simulated supernova detections with human-scanned artifacts to create a large labeled set.
3.1. Training Data
- Effective supervised training requires labeled examples from both classes that approximate their joint feature distributions; missing feature regions can degrade predictions there.The two classes are “Artifacts” and “Non-Artifacts.”Source: §3.1 · p. 6
- Training-set compilation early in Y1 was complicated by a lack of visually scanned non-artifact sources.Source: §3.1 · p. 6
- Fake transients can supply known positive examples, but their selected training features must approximate those of observed production transients.DES-SN overlaid simulated SN Ia fluxes onto images near real galaxies.Source: §3.1 · p. 6
- The training set used visually scanned artifacts and artificial sources, excluding photometrically typed transients to reduce false positives in the Non-Artifact class.Fake SNe were injected into images to monitor survey efficiency, while the pipeline schematic places autoScan after selection cuts and before detections are grouped into raw candidates.Source: Figure 2 · p. 7, §3.1 · p. 9
- The final training set contained 898,963 instances: 454,092 simulated SNe Ia and 444,871 human-scanned artifacts.The artifact detections were randomly sampled from objects scanned during an early processing of DES Y1 data.Source: §3.1 · p. 9
3.2. Features and Processing
autoScan constructs a feature space from image cutouts and other detection data, with preprocessing and postprocessing steps to prepare features for classification. The section describes a shift from pixel-level inputs to compact metrics, including new features designed to characterize artifacts and source morphology.
3.2. Features and Processing
- The preprocessing defines a feature space from image cutouts and other pipeline outputs, then prepares those features for training, validation, and prediction.Source: §3.2 · p. 9
3.2.1. Data Preprocessing
- 51 × 51 cutouts were compressed to 25 × 25 pixels and rescaled using a median-based scale estimate; using B instead of R or I for some features improved classifier performance.The rescaling expresses pixel values as standard deviations above the median and is intended to suppress effects from defective pixels and cosmic rays.Source: §3.2.1 · pp. 9–10
3.2.3. New Features
- The new r aper psf feature compares template-image aperture flux with difference-image PSF-fit flux to identify poorly subtracted stars and galaxies that appear as dipoles.Figure 3 shows paired difference-image and template cutouts for objects with r aper psf > 1.25.Source: §3.2.3 · p. 12, Figure 3 · p. 12
- Other new metrics capture morphology changes between template and difference images, local difference-image significance, filter identity, and pixel smoothness.The smoothness-oriented numneg and lacosmic features are intended to help identify hot pixels and cosmic rays.Source: §3.2.3 · p. 12
- colmeds targets full-column artifacts by measuring column medians, while maskfrac records the fraction of the template cutout that is masked.A column median near the background is expected when a cutout is centered on a PSF, but that expectation may fail for artifacts occupying entire columns.Source: §3.2.3 · p. 14
3.2.4. Data Postprocessing
- When a feature cannot be computed because no galaxy-coadd source lies within 5 arcsec, autoScan assigns that feature its training-set mean.Source: §3.2.4 · p. 14
3.3. Classification Algorithm Selection
The study compared Random Forest, SVM, and AdaBoost classifiers for DES-SN artifact/non-artifact classification and selected Random Forest after it modestly outperformed the alternatives. Random Forests build decision trees from labeled data, selecting feature splits that improve class purity.
3.3. Classification Algorithm Selection
- The unoptimized Random Forest slightly outperformed the SVM and AdaBoost classifiers in the DES-SN artifact/non-artifact comparison.The comparison used three rounds of testing on a 100,000-detection subset.Source: §3.3 · p. 14, Figure 5 · p. 16
- Random Forest training bootstraps labeled data and recursively selects binary feature splits that maximize class purity.Each tree considers a random, user-specified-size feature subsample at each node.Source: §3.3 · p. 14
- A tree split is rejected when the sum of its children’s Gini coefficients exceeds the parent’s; Shannon entropy is an alternative purity measure.For entropy, pNA and pA denote the proportions of non-artifact objects and artifacts.Source: §3.3 · p. 15
- The fitted forest averages tree outputs to estimate the probability that an object is not an artifact, with 0.5 adopted as the DES-SN score cut.The model’s output is the mean of the trees’ estimates.Source: §3.3 · pp. 15–16
3.4. Feature Importances
Feature importance quantifies information supplied during training and aggregates contributions across the forest. The resulting global importances are normalized to sum to unity.
3.4. Feature Importances
- Feature importances quantify how much information each feature provided during Random Forest training.The calculation is based on node splits and the objective function applied to training data.Source: §3.4 · p. 16
- A feature’s tree-specific importance weights its split nodes by the reduction in objective-function value from parent to children.The node contribution uses the number of incident training points and an indicator for whether that node splits on the feature.Source: §3.4 · p. 16
- Global feature importance is averaged across trees and normalized so all feature importances sum to unity.The average uses the number of trees in the forest.Source: §3.4 · p. 16
3.5. Optimization
The authors tuned the Random Forest through a three-fold cross-validated grid search and selected its best-performing hyperparameter configuration. Recursive feature elimination did not significantly improve performance.
3.5. Optimization
- The best classifier used 100 trees, Shannon entropy, 6 features per split, at least 3 samples to split a node, and unlimited depth.The selected model was incorporated into autoScan.Source: §3.5 · p. 17
- The authors performed 1,884 trainings in a three-fold cross-validated grid search over Random Forest hyperparameters.The search covered the hyperparameters listed for the implementation, including tree count, split criterion, feature count, depth, and minimum split samples.Source: §3.5 · pp. 16–17
- Recursive feature elimination provided no statistically significant performance improvement.It was explored as a way to improve the classifier.Source: §3.5 · p. 17
4. Performance
The paper evaluates autoScan with cross-validation and reports both classification metrics and effects on the DES-SN candidate-scanning load. At the adopted score cut, candidate numbers fell substantially while fake-SN efficiency remained above 99 percent.
4. Performance
- The performance section measures classifier statistics on production Y1 data and candidate-level effects after full Y1 reprocessing with an updated difference-imaging pipeline.The reprocessed detection pool differed significantly from production data, providing an out-of-sample set for evaluating scanning-load effects.Source: §4 · p. 17
4.1. Performance Metrics
- The ROC plots FPR against MDR as the score threshold varies, helping select a threshold that balances scanning load and potential classifier bias.The authors state that validation performance can serve as a statistical guarantee in production if validation and production data are identically distributed in feature space and detections are scanned individually.Source: §4.1 · pp. 17–18
4.2. Classification Task
- Stratified five-fold cross-validation assessed autoScan by holding out each class-balanced subset in turn for validation.The other subsets form the training set in each round, supporting assessment of variation with training and testing composition.Source: §4.2 · p. 18
4.3. Results
- At τ = 0.5, autoScan achieved an average detection-level MDR of 4.0 ± 0.1 percent at a fixed FPR of 2.5 percent.Fake efficiency was lower at S/N ≲10 and approached unity at S/N ≳100; the low-S/N detections resemble artifacts.Source: §4.3 · p. 18, Figure 7 · p. 19, Figure 8 · p. 19
4.4. Effect of autoScan on Transient Candidate Scanning Load
- 13.4× fewer first-season candidates remained eligible for scanning while fake-SN efficiency stayed above 99.0 percent at τ = 0.5.The average artifact-to-non-artifact ratio encountered by human scanners also decreased by roughly 40× after production implementation.Source: §4.4 · p. 20
5. Discussion
The discussion considers how supervised classification and fake overlays have advanced artifact rejection, then identifies design challenges for future surveys. It highlights training-data coverage and the risk that individual-detection scores can produce spurious multi-night candidates.
5. Discussion
- AutoScan and fake overlays mark progress in supervised artifact rejection, while future surveys such as ZTF and LSST demand greater reliability, consistency, and transparency.The discussion suggests simulated observations and production active learning as a way to fill gaps in classifier knowledge.Source: §5 · p. 20, §5.1 · p. 22
5.1. Automating Artifact Rejection in Future Surveys
- CNNs can infer features from images, but their abstract features may be harder to interpret than Random Forest feature importances.Feature-set revisions are trial and error, and adding features can sometimes degrade classifier performance; reducing feature count and increasing feature importance is identified as a route to lower MDR.Source: §5.1 · p. 20
- Training and testing data must represent the same feature distributions, because uncovered dense test regions can degrade production accuracy.The discussion notes that even small training-set biases may cause meaningful efficiency losses and identifies active learning as a possible way to reduce sample-selection bias.Source: §5.1 · p. 20
5.2. Eliminating Spurious Candidates
- Night-to-night score fluctuations can let artifacts pass an individual-detection threshold repeatedly, accumulating into spurious multi-night candidates.Figure 9 illustrates a poorly subtracted galaxy that passed the early-test boundary on two nights; the authors caution against inferring candidate validity from detection-level scores alone.Source: Figure 9 · p. 21, §5.2 · p. 22
5.1. Automating Artifact Rejection in Future Surveys
- The training set combined fake Type Ia supernovae overlaid on real host galaxies with nearly 500,000 human-scanned artifacts; future surveys need representative artifacts for training.For surveys with ⟨NA/NNA⟩≳100, random sampling of pipeline variability detections can form an artifact library; pipelines with fewer artifacts may need several 10^4 scanned examples.Source: §5.1 · p. 22
5.2. Eliminating Spurious Candidates
- A second classifier could use multi-night detection histories and score, magnitude, and astrometric-uncertainty changes to identify bona-fide astrophysical candidates.The proposed training examples are time-contiguous detection strings sampled from known candidates.Source: §5.2 · p. 22