PaperlayerView exact revision on arXiv

Source-linked AI summary

Star-galaxy Classification Using Deep Convolutional Neural Networks

Edward J. Kim, Robert J. Brunner

arXiv:1608.04369v2astro-ph.IMastro-ph.COastro-ph.GAcs.CV

TL;DR

Existing star-galaxy classifiers generally depend on engineered catalog features, motivating methods that learn directly from images. This paper applies ConvNets to SDSS and CFHTLenS data and finds accurate, well-calibrated probabilistic classifications that are competitive with conventional methods, while performance depends on the dataset and training regime.

  • Problem

    Catalog-based star-galaxy classification requires manually engineered features, while large surveys need automated probabilistic classification at scale.

  • Method

    The paper trains ConvNets directly on SDSS and CFHTLenS images and combines predictions across transformed inputs with Bayesian Model Combination.

  • Results

    ConvNets match random-forest accuracy on CFHTLenS and provide better-calibrated probabilities there, while on SDSS they do not outperform TPC but remain slightly better calibrated.

  • Takeaways & Limitations

    ConvNets can learn useful image features automatically, but conventional catalog-based methods remain viable when optimized features are available.

  • Takeaways & Limitations

    Performance is less reliable when a ConvNet trained on spectroscopic sources is applied to fainter photometric samples beyond the training distribution.

Abstract

from arXiv · show

Most existing star-galaxy classifiers use the reduced summary information from catalogs, requiring careful feature extraction and selection. The latest advances in machine learning that use deep convolutional neural networks allow a machine to automatically learn the features directly from data, minimizing the need for input from human experts. We present a star-galaxy classification framework that uses deep convolutional neural networks (ConvNets) directly on the reduced, calibrated pixel values. Using data from the Sloan Digital Sky Survey (SDSS) and the Canada-France-Hawaii Telescope Lensing Survey (CFHTLenS), we demonstrate that ConvNets are able to produce accurate and well-calibrated probabilistic classifications that are competitive with conventional machine learning techniques. Future advances in deep learning may bring more success with current and forthcoming photometric surveys, such as the Dark Energy Survey (DES) and the Large Synoptic Survey Telescope (LSST), because deep neural networks require very little, manual feature engineering.

1 INTRODUCTION

Large photometric surveys require automated, probabilistic star-galaxy classification at scales beyond manual expert processing. This paper presents a ConvNet framework that learns features directly from calibrated images and produces accurate, calibrated classifications with little manual feature engineering.

  • Forthcoming surveys will collect data for hundreds of millions to billions of stars and galaxies, making manual classification impossible.Probabilistic outputs also support probability cuts, population studies, and follow-up selection.
  • Machine learning has already been applied to star-galaxy classification using neural networks, decision trees, Support Vector Machines, and classifier combinations.
  • Most existing classifiers use catalog summary features that require substantial engineering and domain expertise.Deep learning instead learns multiple feature levels directly from data.
  • Deep learning had been used for galaxy morphology and photometric redshift estimation but not yet for source classification.
  • The proposed ConvNet operates directly on SDSS and CFHTLenS images and produces accurate, well-calibrated probabilistic classifications with little hand-designed feature engineering.Its performance is compared with a standard catalog-based machine-learning method.

2 DATA

The study evaluates ConvNets on SDSS and CFHTLenS data using spectroscopically labeled objects, calibrated multiband image cutouts, and held-out testing. The datasets differ in survey coverage, depth, sample composition, and image preparation.

  • The study uses photometric and spectroscopic datasets from SDSS and CFHTLenS to evaluate classification performance across different data characteristics.
  • SDSS provides five-band photometry over 14,555 square degrees, a catalog exceeding 300 million objects, and spectroscopy for more than three million stars and galaxies.
  • The binary classification setup excludes quasars and applies quality criteria to spectroscopic flags, sizes, redshift errors, and redshift values.
  • SDSS images are aligned across five bands, converted to extinction-corrected luptitudes, and represented as 48 × 48 pixel cutouts.
  • The SDSS sample contains 17,344 stars and 47,656 galaxies, split into training, validation, and blind test sets of 40,000, 10,000, and 15,000 objects.
  • CFHTLenS covers 154 square degrees in five bands and contributes 8,545 stars and 57,843 galaxies after spectroscopic cross-matching and image downscaling.

3 DEEP LEARNING

The paper describes feedforward neural networks and ConvNets as models trained by minimizing cross-entropy with gradient-based updates. Its architecture processes five-band images through convolution, pooling, nonlinearities, and fully connected layers to produce class probabilities.

  • Neural-network training: A neural network learns weights and biases by minimizing cross-entropy between predicted probabilities and binary truth labels.
  • Neural-network training: Gradient descent and backpropagation update layer parameters, while mini-batches approximate gradients using randomly selected training examples.
  • Convolutional neural networks: ConvNets apply learned filters to image feature maps, with pooling reducing dimensionality and increasing invariance to small shifts and distortions.
  • Model architecture: The model processes 44 × 44 images in five ugriz bands and uses 11 trainable layers with leaky ReLU nonlinearities and convolutional filters.
  • Model architecture: A softmax layer converts the final outputs into posterior probabilities for the star and galaxy classes.
  • Model selection: The architecture was selected after searching more than 200 configurations on SDSS validation data and then reused unchanged on CFHTLenS.
  • Model selection: A first-layer receptive field larger than 5 × 5 performed worse, while replacing it with two 3 × 3 layers also increased validation error.

4 REDUCING OVERFITTING

The ConvNet has many more learnable parameters than training images, creating a strong overfitting risk. The authors address this with data augmentation, dropout, and Bayesian model combination to improve invariance and robustness.

  • The network has 11×10^6 learnable parameters but only 4×10^4 training images, making regularization necessary.
  • Training images are augmented through rotations, reflections, translations, and Gaussian noise.
  • These augmentations make the model more invariant to image transformations and add almost no computational cost because the CPU performs them during GPU training.
  • Dropout randomly removes hidden-neuron outputs with probability 0.5, forcing the remaining network to learn more robust features.
  • Final classifications combine 64 transformed-input prediction sets using Bayesian Model Combination.

5 TREES FOR PROBABILISTIC CLASSIFICATIONS

The paper uses Trees for Probabilistic Classifications (TPC) as a catalog-feature baseline, with separate photometric and morphology-augmented models for SDSS and CFHTLenS.

  • TPC is a parallel supervised algorithm combining prediction trees and random forests for star-galaxy classification.
  • TPCphot uses magnitudes and colors, while TPCmorph adds morphology-related attributes to the photometric features.For SDSS, TPCphot has nine attributes and TPCmorph has fourteen dimensions including concentration.
  • SDSS morphology models use concentration, defined as psfMag − cModelMag, alongside five-band magnitudes and colors.The SDSS pipeline classifies an object as a galaxy when concentration > 0.145.
  • CFHTLenS morphology models use FLUX_RADIUS, A_WORLD, and B_WORLD because its catalog does not provide concentration.These features are added to five magnitudes and four corresponding colors, producing twelve dimensions.

6 RESULTS AND DISCUSSION

The evaluation section reports ConvNet classification performance on CFHTLenS and SDSS and compares it with TPC using stated performance metrics.

  • The paper evaluates its ConvNet model on the CFHTLenS and SDSS data sets.
  • ConvNet performance is compared with the performance of TPC.
  • The evaluation uses performance metrics introduced before presenting the classification results.

6.1 Classification Metrics

The paper evaluates probabilistic star-galaxy classifiers with threshold-dependent discrimination, completeness and purity, prediction error, calibration, and galaxy-count reliability measures.

  • Probabilistic classifiers provide continuous star-or-galaxy probabilities rather than only discrete labels.These probabilities allow users to choose probability cuts according to their operating conditions.
  • Fixed probability thresholds can ignore science requirements, misclassification costs, and class imbalance.A threshold of 0.5 is not necessarily optimal when galaxies outnumber stars.
  • AUC summarizes the overall performance of a binary classifier by integrating the ROC curve’s true-positive rate against false-positive rate.
  • Completeness measures correctly classified members of a class, while purity measures the fraction of predicted class members that truly belong to it.The paper evaluates galaxy purity at galaxy completeness 0.960 and star completeness at star purity 0.970.
  • Mean squared error evaluates probabilistic predictions as a calibration score or loss function.
  • Calibration compares predicted posterior probabilities with observed class proportions, using overlapping bins to reduce distortion from bin choice.The calibration error is the absolute difference between the observed galaxy probability and predicted probability in each bin.
  • The absolute error in estimated galaxy counts measures whether probability-weighted outputs can support downstream analyses.A well-calibrated classifier permits weighting each object by its estimated galaxy probability instead of counting every object equally.

6.2 CFHTLenS

On CFHTLenS, ConvNet was evaluated against TPCmorph and TPCphot using the same test set, with performance varying across metrics, magnitude, color, and calibration. ConvNet learned hierarchical shape features from images and produced competitive, well-calibrated classifications.

  • Performance metrics: ConvNet outperformed TPCmorph on five metrics, while TPCmorph performed better on two metrics in the CFHTLenS test set.The compared metrics were AUC, p_g, CAL, |ΔN_g|/N_g, log loss, MSE, and c_g.
  • Performance metrics: At operating conditions c_g = 0.96 or p_s = 0.97, ConvNet and TPCmorph outperformed the CFHTLenS pipeline across all magnitudes.The comparison used adjusted probability thresholds for the specified operating conditions.
  • Magnitude dependence: 0.9972 galaxy purity was maintained by ConvNet up to i ∼24.5, compared with 0.9963 for TPCmorph.For star completeness at the same magnitude, TPCmorph reached 0.9252 while ConvNet reached 0.8966.
  • Color dependence: TPCmorph provided slightly better completeness and purity than ConvNet for 0.8≲g −r≲1.6, while ConvNet performed better in the remaining color regions.The comparison was made using the differential-count analysis as a function of g −r color.
  • Probability calibration: ConvNet’s calibration curve was nearly diagonal, and its calibration error was about 20% lower than TPCmorph’s.The diagonal relationship indicates that ConvNet’s probabilistic output can be treated as the probability that an object is a galaxy.
  • Network interpretation: ConvNet’s convolutional layers progressed from recognizable edges and blobs to higher-level features used by the final classifier.This hierarchical abstraction enables the network to use shape information in classification.

6.3 SDSS

On SDSS, ConvNet approaches TPCmorph but does not consistently surpass it, while outperforming catalog methods based only on magnitudes and colors. Its performance varies with magnitude and color, and it can learn morphological features automatically from images.

  • Overall performance: TPCmorph outperforms ConvNet on all SDSS metrics except calibration and cross-entropy, while both outperform TPCphot.ConvNet remains closer to TPCmorph because it learns shape information automatically from images.
  • Magnitude dependence: ConvNet’s galaxy purity and star completeness begin to drop at faint magnitudes i ≳ 21, whereas TPCmorph performs slightly better across magnitudes.The authors suggest over-reliance on morphology may contribute because measurement uncertainties increase near the survey limit.
  • Interpretation: Data augmentation improves overall performance greatly but may confuse the network with faint galaxies that resemble point sources.This creates a trade-off between broad performance gains and faint-magnitude behavior.
  • Integrated counts: ConvNet maintains galaxy purity of 0.9915 up to i ∼22.5, compared with 0.9977 for TPCmorph.TPCmorph also maintains star completeness of 0.9810, while ConvNet drops to 0.9500.
  • Color dependence: ConvNet performs slightly better than TPCmorph for colors 0.7 ≲ g−r ≲ 2.0, but both TPC methods outperform it for g−r ≲ 0.8.The relative performance changes with the stellar fraction across color ranges.
  • Calibration: ConvNet may be better calibrated than TPCmorph on SDSS despite being less well-calibrated than the same model on CFHTLenS.Table 4 supports this comparison through the calibration metric, although the curves are not straightforward to compare visually.
  • Interpretation: Hierarchical feature maps enable ConvNets to learn morphological features automatically from SDSS images.The activations progress from low-level features in early layers to high-level features in later layers.

7 CONCLUSIONS

The ConvNet classifies stars and galaxies from images while learning useful features automatically, matching or exceeding conventional methods depending on the dataset. Its main boundaries are dependence on spectroscopic training data, possible overfitting, and the need for larger computational resources as training sets grow.

  • ConvNets learn classification features directly from images, avoiding the separate feature-engineering process required by traditional machine-learning algorithms.This is the principal methodological advantage identified for star-galaxy classification.
  • On CFHTLenS, the ConvNet is as accurate as the TPC random-forest algorithm and appears better calibrated, while on SDSS it does not outperform TPC but remains slightly better calibrated.The paper attributes the SDSS comparison partly to the catalog’s highly optimized concentration parameter.
  • For SDSS, TPC remains a viable conventional alternative because the catalog provides a concentration parameter highly optimized for star-galaxy classification.The paper frames this result as an instance of no single model working best for every problem.
  • Possible overfitting may explain why the ConvNet with maximal information fails to significantly outperform a catalog-based method, while additional spectroscopic training data remain costly and uncertain for future surveys.Using more training images would also require multi-GPU systems, which was beyond the scope of this work.
  • Future improvements include training multiple architectures, combining transformed-input predictions, and integrating ConvNets into broader classifier-combination frameworks.The paper reports that prior classifier combinations improved overall performance over individual methods and proposes adding ConvNets to that framework.
  • The supervised ConvNet may be less reliable on fainter photometric samples that lie beyond the available spectroscopic training distribution.The authors suggest combining ConvNets with unsupervised or template-fitting methods to extend classification beyond spectroscopic limits.
Loading 1608.04369v2…