Data Mining and Machine Learning in Astronomy

Nicholas M. Ball; Robert J. Brunner

Source-linked AI summary

Data Mining and Machine Learning in Astronomy

Nicholas M. Ball, Robert J. Brunner

arXiv:0906.2173v2Jun 11, 2009astro-ph.IMastro-ph.CO

TL;DR

Astronomy’s rapidly growing digital datasets create a need for effective knowledge discovery, while careless data mining can become a black box with questionable results. This review surveys the data-mining process, methods, applications, and future directions, concluding that careful algorithm selection guided by the astronomical problem can support improved science.

Problem
Astronomy faces rapidly expanding digital datasets, creating a need to extract useful information while avoiding data mining that provides little physical insight or questionable results.
Method
The review surveys knowledge discovery from data collection through result interpretation, covering machine-learning algorithms, astronomy applications, and future directions.
Results
The review identifies astronomy applications in which data mining improved science, including full photometric-redshift PDFs that improve clustering signal-to-noise by four to five times for a given dataset.
Takeaways & Limitations
Data mining is a tool for improved astronomy when the algorithm is selected appropriately and guided by the astronomical problem.
Takeaways & Limitations
Data-mining results depend strongly on input-data quality, and many algorithms do not account for astronomical measurement errors.

Abstract

from arXiv · show

We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining process, from data collection through to the interpretation of results. We cover common machine learning algorithms, such as artificial neural networks and support vector machines, applications from a broad range of astronomy, emphasizing those where data mining techniques directly resulted in improved science, and important current and future directions, including probability density functions, parallel algorithms, petascale computing, and the time domain. We conclude that, so long as one carefully selects an appropriate algorithm, and is guided by the astronomical problem at hand, data mining can be very much the powerful tool, and not the questionable black box.

1. Introduction

The review frames data mining as knowledge discovery from rapidly expanding digital astronomical data, while emphasizing that algorithms are tools within the broader scientific process. It aims to connect methods with scientific results and guide skeptical researchers toward responsible use.

Data mining turns observational data into useful information that can support hypotheses, theories, and predictions.
Astronomy’s data flood has made automated pattern discovery and translation into human-understandable information increasingly necessary.
The review emphasizes algorithms that have enabled, or could enable, useful scientific results rather than computational novelty alone.
Data mining should remain one component of the broader scientific process, applied correctly to create scientifically useful results.
The review combines an algorithmic account of the data-mining process with astrophysical applications to offer insights to both astronomy and data-mining audiences.

2. Overview of Data Mining and Machine Learning Methods

The review presents astronomy-focused data mining as knowledge discovery in broadly defined machine-readable databases. It introduces necessary technical terminology while deferring most application examples to the dedicated applications section.

The methods overview focuses on knowledge discovery in databases, treating any machine-readable astronomical data as a database.
The section is organized algorithmically to review the data-mining process without duplicating the later astronomy-application section.
Technical jargon is retained where needed for clarity and generally explained for astronomers unfamiliar with data mining.

2.1. Data Collection

Data collection includes obtaining, querying, archiving, and combining astronomical data in digital form. Cross-matching supports multiwavelength analysis but introduces practical matching issues, motivating tools for large heterogeneous datasets.

Data collection covers new observations, archival database queries, and cross-matching or combining data through data fusion.
Multiwavelength cross-matching commonly uses sky position within an astrometric tolerance when definitive object identifications are unavailable.
Virtual Observatory web services are intended to simplify large cross-matches across widely distributed, heterogeneous datasets.
FITS and plain ASCII are common astronomical data formats, while VOTable is emerging and ARFF is not widely used in astronomy.

2.2. Preprocessing of Data

Astronomical data preprocessing makes inputs meaningful and suitable for particular algorithms. It includes representation changes, cleaning, handling missing or out-of-range values, normalization, and attribute selection.

Preprocessing is problem-dependent and can significantly affect algorithmic results, so it should be applied carefully.
Algorithms may require attributes to be numerical or categorical, with transformations available in both directions.
Scalarization encodes categorical attributes numerically, but labels must not imply nonexistent ordering when algorithm outputs treat values continuously.
Binning converts numerical data into categories but can create edge, empty-bin, NaN, and floating-point comparison problems.
Attribute transformations such as magnitude differencing can create numerical issues including division by zero and loss of accuracy.
Bad, missing, out-of-range, or outlier values may require removal, imputation, sampling, or other problem-specific treatment.
Normalization can improve numerical conditioning when attributes span substantially different numerical ranges.

2.3. Attribute Selection

Attribute selection reduces a dataset’s inputs to those useful for the problem, because excessive, noisy, irrelevant, or redundant attributes can hinder generalization. The review describes both subjective selection and more formal dimension-reduction methods.

Attribute selection: Using all available attributes can worsen performance through the curse of dimensionality, which creates sparse or empty regions that hinder generalization.Noisy, irrelevant, and redundant attributes can also reduce algorithm performance.
Dimension reduction: Dimension reduction seeks to retain as much information as possible while representing the data with fewer attributes.
Attribute selection: Judgment-based subset selection is the simplest reduction method and can work well depending on the problem.
Dimension reduction: Principal component analysis provides a more sophisticated reduction approach but is limited to linear relations.PCA selects eigenvector directions containing the greatest variation in the input data.
Dimension reduction: Artificial neural networks, kernel methods, genetic algorithms, and nearest-neighbor searches can also perform or support dimension reduction.These methods respectively support autoencoder-based PCA, generalized PCA, attribute-subset selection, and reduced-dimensional searches.

2.4. Selection and Use of Machine Learning Algorithms

Machine-learning methods differ in whether they use labeled training data, discover structure without labels, or combine both approaches. The review describes representative algorithms and emphasizes that algorithm choice depends on both the dataset and the application.

Supervised and unsupervised methods: Supervised methods learn mappings from objects with known target properties and apply them to unseen objects, whereas unsupervised methods require no training set.In astronomy, supervised inputs are often photometric attributes and targets are spectroscopic properties.
Semi-supervised methods: Semi-supervised methods incorporate known priors while allowing data-driven discovery and extrapolation, but can be more complex to implement.They are potentially useful for extending photometric-survey analyses beyond the spectroscopic limit while incorporating spectroscopic information.
Artificial neural networks: Artificial neural networks use weighted interconnected nodes and nonlinear activation functions, functioning in data mining as nonlinear extensions of statistical methods.Supervised ANNs map input parameter vectors to outputs by minimizing a cost function, commonly mean-squared deviation from targets.
Decision trees: Decision trees iteratively split training populations at nodes using criteria that minimize classification error until user-defined stopping conditions are reached.Splits may be axis-parallel or oblique across input attributes.
Support vector machines: Support vector machines separate classes using support-vector-defined boundaries, with kernels mapping nonlinear relationships into a higher-dimensional space where separation is linear.
Unsupervised methods: K-means, mixture models, and self-organizing maps provide unsupervised approaches for clustering, density estimation, and high-dimensional data visualization.K-means requires an initially specified number of clusters, while mixture models represent distributions as sums of probability-density components.
Choice of algorithm: No single algorithm is best because the appropriate choice depends on both the dataset and the application.A general software framework may also fail to cover every step from raw catalog to scientific result, especially for large datasets.

2.5. Improving Results

The review describes methods for improving data-mining results by avoiding local minima, controlling overfitting, combining models, and retaining full predictive probability information.

Greedy optimization can become trapped in local minima, so repeated initializations or simulated annealing can seek solutions closer to the global minimum.Simulated annealing perturbs the search through heating and then cools it toward a better solution.
The bias-variance tradeoff limits predictive models because increasing complexity can reduce bias while increasing sensitivity to training-data variations.Testing on data excluded from training helps select an appropriate stopping point and reduce overfitting.
Regularization reduces overfitting by adding a complexity penalty to the cost function, while excessively small smoothing scales can overfit unsupervised models.The penalty can discourage configurations such as large weights in an artificial neural network.
Cross-validation estimates error by repeatedly training on subsets and testing on held-out data, with leave-one-out as the K-sized limiting case.The holdout method can have high bias or variance when the training or testing subset is small.
Committees of differently parameterized models can improve predictions by reducing sensitivity to outliers and making majority or averaged results more likely to be accurate.Boosting instead reweights misclassified objects while iteratively combining weak learners.
Using a full probability density function for a predicted property can improve astronomical results by retaining more information than a single scalar prediction.The review identifies PDF use as a direction for many astronomical applications.

2.6. Application of Algorithms and Some Limitations

The review emphasizes that data mining must be applied critically because astronomical datasets impose extrapolation, validity, measurement, parameter-selection, and data-quality constraints.

The review presents collaboration between astronomers and data-mining experts as a way some KDD problems might be ameliorated.
Supervised models should not extrapolate beyond the parameter space represented by their training data, such as using spectroscopic samples for much fainter photometric objects.
Randomly overlapping training and testing subsets can yield apparently reasonable but statistically invalid models that overfit the data.
Most data-mining algorithms do not explicitly account for astronomical measurement errors, while many algorithms also require extensive tuning of adjustable parameters.
The usefulness of any data-mining result depends most strongly on input-data quality, so insufficient, poorly collected, or incorrectly treated data produce unhelpful results.

3. Uses in Astronomy

The review surveys diverse astronomical applications of data-mining algorithms, framing them for nonexperts and noting contributions from both astronomers and data-mining specialists.

This section organizes applications by astrophysical problem rather than by data-mining algorithm, complementing the earlier algorithm-centered review.
Because no exact definition of a data-mining tool exists, the review illustrates a wide variety of existing uses and possible future applications rather than providing a complete inventory.
Astronomers use data-mining algorithms across astronomy, while data-mining experts also exploit astronomical datasets because their scale makes them suitable for the approach.Examples include SKICAT for catalog production and catalog analysis.

3.1. Object classification

Object classification organizes astronomical information for hypothesis testing and model comparison, but different applications require balancing completeness against efficiency. Data-mining methods incorporate diverse measurements and have improved automated classification across stars, galaxies, morphology, and AGN.

Evaluation: Classification organizes information for forming hypotheses and comparing astronomical models, using completeness and efficiency as key evaluation concepts.Completeness measures recovered true objects, while efficiency measures the fraction of selected objects that are truly of the target type.
Evaluation: Rare-object searches generally prioritize completeness, whereas cosmological clustering prioritizes efficiency despite reduced completeness.The preferred tradeoff depends on the scientific application and its tolerance for contamination.
Star-Galaxy Separation: Automated star-galaxy separation incorporates morphology, colors, seeing, and other object information, with most reported algorithms achieving over 95% efficiency.Artificial neural networks, decision trees, mixture models, and self-organizing maps have all been applied.
Galaxy Morphology: Supervised methods assign galaxy morphology types from measured parameters, with neural networks matching human accuracy for low-redshift T-type prediction.Inputs can be purely morphological or can include additional information such as color; related work also addresses higher-redshift and peculiar galaxies.
Other Galaxy Classifications: Mixture models improve galaxy classification over a simple u −r split, while more general hybrid approaches permit new object classes beyond existing templates.The mixture-model example separates a third class that is not visible using u −r alone.
Quasars/AGN: Data mining refines AGN selection through higher-dimensional, multiwavelength classification and can provide object-level probabilities without requiring emission-line detections.Decision-tree selection improved reported reliability to 85%, compared with 60% for simpler criteria.
Other Classifications: Object detection is often the first classification stage, and neural networks can classify faint objects as well as Bayesian methods with considerable computational speedup.Astronomical detection approaches include thresholding, maximum likelihood, wavelets, independent component analysis, mixture models, and neural networks.
Other Classifications: Stellar classification commonly reaches an accuracy of one spectral type and half a luminosity type despite many input attributes and output classes.The reviewed studies indicate that the dimensionality of the inputs does not by itself invalidate the approach.

3.2. Photometric redshifts

Photometric redshift estimation uses template and empirical supervised-learning approaches, with data-mining methods addressing complex mappings and catastrophic failures across galaxies and quasars. Accuracy is strongest for low-redshift galaxies, while higher-redshift and sparse-data regimes motivate hybrid methods and dimensionality reduction.

Motivation: Photometric redshifts trade lower individual distance accuracy for the much larger object counts available from photometric surveys.The larger sample can suppress statistical noise in ensemble calculations.
Methods: The two common photo-z approaches are template methods and empirical training-set methods, with this review focusing on supervised empirical learning.The review identifies hybrid template-empirical methods as a likely future direction and highlights full probability density functions for representing redshift uncertainty.
Galaxies: At low redshifts, template and empirical methods achieve a root-mean-square deviation of ∼0.02 in redshift for normal galaxies, near the best possible result given intrinsic property spread.At higher redshifts, fainter galaxies, sparse spectra, spectral-break shifts, and intrinsic evolution make accurate estimation more difficult.
Galaxies: Higher-redshift photo-z estimation can use infrared data, which reveals the 4000˚A break farther into the redshift range and improves results for both template and supervised approaches.Template methods require suitable infrared templates, while supervised methods incorporate the new data directly.
Limitations: Beyond the available spectral regime, supervised photo-z methods require extrapolation, while small training sets can reintroduce dimensionality problems as measured wavebands outnumber objects.Hybrid template refinement, semi-supervised learning, and dimension reduction are identified as possible responses.
Quasars/AGN: Quasar photo-zs suffer catastrophic failures from degeneracies, emission-line effects, and reddening, but multiple-neighbor and supervised methods reduce spread and can largely eliminate catastrophics for selected unimodal PDFs.Using 17 filters instead of five also improves results, although the corresponding photometric sample is much smaller.

3.3. Other Astrophysical Applications

Data mining extends across time-domain astronomy, cosmology, spectral analysis, survey cross-matching, and simulations. The reviewed applications report improved forecasting, parameter estimation, interpretable metallicity relations, and broader scientific use of large astronomical datasets.

Time domain: Data mining has already been applied across variable stars, solar-system objects, planetary atmospheres, stellar motions, exoplanets, supernovae, gamma-ray bursts, and quasar variability.The unexplored time-domain parameter space is defined by depth, sky coverage, and temporal resolution.
Cosmic microwave background: Committee ANNs provide unbiased CMB cosmological-parameter estimates in considerably less processing time than maximum likelihood, with comparable accuracy.CMB applications also include Bayesian modeling, wavelets, ICA-based foreground removal, and Sunyaev-Zeldovich cluster detection.
Survey cross-matching: Probabilistic classifiers combined with distribution estimates improve cross-matching of objects across surveys and provide probabilistic outputs.The approach exploits correlations in measured attributes, such as similar sky positions, among cross-matched objects.
Spectral analysis: ANNs whose architectures are evolved with genetic algorithms produce empirical Galactic-metallicity equations from line ratios, reducing the usual black-box element and potentially identifying new diagnostics.The resulting equations connect the prediction to measurable spectral line ratios.
Solar and planetary science: A mixture of SVM and kNN improves solar-flare forecasting, while mixture models analyze mineral or constituent abundances in soil spectra.These examples illustrate direct data mining of spectra in solar and planetary applications.
Simulations: Data mining also applies to astronomical simulations, whose size and complexity can rival or exceed real datasets, including theory incorporated into the Virtual Observatory.The Millennium Run simulation has supported over 200 papers, while simulation mining presents additional challenges compared with observations.

4. The Future

The future of astronomical data mining centers on probability distributions, scalable computation, virtual observatories, specialized hardware, and time-domain surveys. These developments offer improved scientific measurements but require careful handling of algorithm choice, data scale, and infrastructure constraints.

4. The Future: Future progress will benefit from cross-fertilization with high-energy physics, biology, and geosciences, including the development of astroinformatics.The review highlights software and informatics practices from these fields as relevant models for astronomy.
4.1. Probability Density Functions: Full photometric-redshift PDFs retain information lost by single estimates and can improve astronomical results' signal-to-noise.They support bias reduction, lower error, improved subsamples, and broader applications including clustering, galaxy-cluster detection, and weak lensing.
4.1. Probability Density Functions: Four to five times higher clustering signal-to-noise can result from using full PDFs without altering the dataset.The comparison concerns clustering measurements for a given dataset and is reported as applicable to any astronomical object.
4.2. Real-Time Processing and the Time Domain: Time-domain astronomy introduces challenges involving irregular variability, difference images, rapid event response, and artifact removal.LSST will observe the visible sky every three nights, producing a data stream that anticipates these challenges.
4.2. Real-Time Processing and the Time Domain: Petabyte-scale surveys make automated, computationally efficient mining essential, because naïve algorithms may scale as N^2 or worse.The review notes that algorithms scaling beyond NlogN may rapidly become infeasible, while LSST alone is expected to produce more than one petabyte annually.
4.5. The Virtual Observatory: Virtual Observatory services, web-based analysis, and multiwavelength data access are emerging, but fully exploiting astronomical archives remains time-consuming and difficult.Distributed datasets can make cross-matching intractable when data are difficult to move; center-based cross-matching tools are proposed as one response.
4.7. Novel Supercomputing Hardware: GPUs, FPGAs, and Cell processors provide general-purpose specialized-hardware options for speeding astronomical analyses and enabling previously impossible questions.GPU applications include n-body simulation, real-time instrument processing, and accelerated nearest-neighbor searches.

5. Conclusions

The review presents data mining as a means to improve astronomical science, requiring domain-aware use and infrastructure that supports researcher-specific analysis. Distributed data and impractical transfers constrain how this approach can be implemented.

Data mining is framed as a tool for improved science rather than an end in itself.
Astronomers need sufficient knowledge to apply data mining to their science problems, often through collaboration with technical specialists.
Research infrastructure should expose raw or minimally processed data and support user-specific codes near the data when datasets are large.
Interoperable storage schemas are needed so researchers can combine distributed datasets and apply computing and algorithms to particular science questions.
When large datasets reside at separated sites, network transfer is impractical, making whole-dataset downloading unsustainable despite continued local analysis of subsets.