Deep learning from the crowd Fundamentals of morphological galaxy classification

1. Introduction

• Citizen-science projects recruit volunteers to label galaxy images and have produced crowd-annotated datasets that are useful, particularly when annotators agree.Examples include Galaxy Zoo 1 and 2, DECaLS, CANDELS, and Hubble.Source: §1 · p. 1
• Classifying about 400,000 Galaxy Zoo DECaLS images took 7.5 million volunteers 4.5 years; collecting 40 labels per DESI-LS image at that rate would take about 200 years.The paper therefore motivates machine-learning methods to classify the massive image collections produced by sky surveys.Source: §1 · pp. 1–2
• CNNs are state of the art for morphology-based galaxy classification, but a comprehensive study of factors shaping their performance on crowd-sourced labels is lacking.The introduction identifies training scheme, annotator agreement, and hierarchy as the study’s three fundamental components.Source: §1 · p. 2
• The hierarchy may improve classification through its constraints and preserve useful predictions at abstract levels when fine-grained classes fail, as with low-resolution images.Morphological classification schemes organize galaxies into hierarchical classes, including Hubble’s tuning fork.Source: §1 · pp. 1–2
• Using Galaxy Zoo 1, the study examines how CNN training schemes, annotator agreement, and hierarchy affect galaxy morphology classification.The analysis is intended to inform future work using other classification schemes and sky surveys.Source: §1 · p. 2

Related work

• CNNs became the state of the art after early feature-engineering approaches proved moderately accurate, with studies applying them across multiple galaxy surveys.ResNets and Capsule Networks have also been tested, but their performance is similar or inferior to CNNs.Source: §2 · p. 2
• Pandya et al. (2023) found that symmetry-based GCNNs outperform a CNN baseline on Galaxy Zoo DECaLS, but require more training resources.The GCNNs use rotations, translations, and reflections to improve robustness to noisy images and adversarial attacks.Source: §2 · p. 2
• Researchers have addressed limited labeled data with active learning, transfer learning, simulated training sets, and data augmentation.Examples include Walmsley et al. (2019), Ghosh et al. (2020), Vega-Ferrero et al. (2021), Khramtsov et al. (2022), and Serrano-Pérez et al. (2024).Source: §2 · pp. 2–3
• Cross-survey transfer methods include adversarial validation for selecting matching training galaxies and multitask learning across Galaxy Zoo datasets.Khramtsov et al. (2022) used adversarial validation; Walmsley et al. (2024) jointly trained a classifier across datasets.Source: §2 · p. 3
• Denoising galaxy images with a VAE or combining a VAE with a GCNN has been proposed, but noise reduction is outside this work’s scope.Mirzoyan (2025) presents a U-Net VAE for JWST images; Mirzoyan & Avagyan (2026) combine a VAE and GCNN for disk/non-disk classification.Source: §2 · p. 3
• Crowd-sourced Galaxy Zoo annotations became important as expert labeling scaled poorly; Galaxy Zoo 1 has six categories, while Galaxy Zoo 2 has 11 tasks and 37 responses.Crowd-label errors may affect models trained on these data.Source: §2 · p. 3

3.1. Single level agreement

• For single-level labels, the authors select items by estimated annotator agreement, which is the inverse of label entropy, to train with hard silver labels.They use agreement thresholds because hard labels are more direct to train on than soft labels; inter-coder agreement analysis is out of scope because individual votes are generally unavailable at this scale.Source: §3.1 · pp. 3–4

3.2. Hierarchical agreement

• WHA extends classical agreement to hierarchical labels by weighting leaf-node agreement along root-to-leaf paths, but it does not account for internal-node agreement in DAGs.The authors also propose multidimensional weighted agreement (MDWA), then normalize it as NMDWA to prevent dimensions with fewer classes from disproportionately influencing the score.Source: §3.2 · p. 4
• NMDWA assigns higher values than WHA in intermediate agreement ranges and drops more sharply in lower ranges; the reported Pearson correlation between MDWA and WHA is 0.9.The distribution comparison is for annotated Galaxy Zoo 1 instances; the authors state that NMDWA is required for DAG hierarchies.Source: Figure 3 · p. 4, §3.2 · pp. 4–5

• GZ1 drew from 893,212 SDSS objects, of which 667,944 were crowd-annotated; nearly 100,000 annotators supplied an average of 38 annotations per image (Lintott & et al. 2008).Source: §4 · p. 5
• The six classes were elliptical, clockwise spiral, anticlockwise spiral, other, star or don’t know, and merger, organized in the hierarchy shown in Figure 4.Source: §4 · p. 5
• The team weighted consistent annotators and corrected redshift-related morphology bias, but these experiments used original vote fractions because corrected fractions were unavailable for every class (Lintott & et al. 2011).Small, faint, or distant galaxies can appear elliptical even when they are spirals whose arms are not distinctly visible.Source: §4 · p. 5

CNN training

• Experiments use a 208-million-parameter EfficientNetV2 XL pretrained on ImageNet, comparing frozen-backbone Last training with All-layer training on 64/16/20% train/validation/test splits.The final layer has one output per class and estimates class probabilities.Source: §5 · p. 5

Hierarchical classification

• The two-stage BCNN refines CNN class probabilities with a Bayesian network, then selects a root-to-leaf path while enforcing consistency with the hierarchy.The Bayesian network propagates probabilities so each selected class is consistent with its ancestors; results are compared with CNN-only classification.Source: §5 · p. 5

Crowd agreement

• Training data are configured at crowd-agreement thresholds from 1.0 to 0.3, balancing higher annotation agreement against fewer examples and evaluating matching test-image ranges with hierarchical measures.The methodology notes that classification difficulty and annotation bias also affect this quality–quantity tradeoff.Source: §5 · p. 5

Transfer learning

• Curriculum learning initializes each agreement range from the model trained on the preceding, easier range, and is tested on training sets of 200–1,000 examples with CNN and BCNN.Ranges proceed from agr = 1 through exclusive agreement intervals down to 0.4 > agr ≥ 0.3.Source: §5 · pp. 5–6

Ensembles

• Ensembles combine predictions from 3 or 5 pretrained models using simple majority voting.Each test image is supplied simultaneously to all models in the ensemble.Source: §5 · p. 6

6.2.1. Training on cumulative ranges of agreement

• EM ≈0.92 at CRA = 0.6 with last-layer retraining, versus EM ≈1.00 at CRA = 0.7 with all-layer retraining; all-layer retraining improves all measures by about 10% points.With all layers retrained, exact match exceeds 99% for agreement above 0.6; last-layer performance peaks at CRA = 0.6 and then declines.Source: Figure 6 · p. 6, §6.2.1 · pp. 6–7

6.2.3. Training on reduced data

• With reduced training data, BCNN can exceed CNN exact match, particularly when only the last layer is retrained; below 250 samples, last-layer training performs better than all-layer training.All-layer retraining achieves above 75% with only 100 images, while performance declines as data are reduced; above 5,000 samples and agreement ≥0.6, results are very good.Source: §6.2.3 · pp. 7–8

6.2.2. Training on exclusive ranges of agreement

• Exclusive-range transfer learning performs similarly to cumulative training, with a small advantage in some cases below 200,000 training images; the schemes converge with more data.Training proceeds from high-agreement, easier cases to lower-agreement, harder cases.Source: §6.2.2 · p. 7, Figure 9 · p. 8

6.2.4. Evaluating on the full test set

• Exact match exceeds 90% on the full test set, including cases with agreement below 0.4; models trained on all galaxies at each agreement level perform best.The full-test evaluation combines images from all exclusive agreement ranges.Source: Figure 11 · p. 8, Figure 12 · p. 8, §6.2.4 · pp. 8–9

6.2.5. Galaxy type differences

• At agr ≥0.7, the all-layer CNN and BCNN confusion-matrix experiments reach an exact match of 99.75; ACW is often confused with MERGER, which has the fewest training samples.Most predictions fall on the confusion-matrix diagonal, while MERGER shows proportionally greater uncertainty.Source: §6.2.5 · p. 9, Table 2 · p. 9

6.2.6. Results for ensemble of classifiers

• Combining the five best models yields a small improvement over the best individual models on the full test set.The ensemble combines model predictions using a simple voting scheme.Source: Table 4 · p. 9, §6.2.6 · pp. 9–10

6.3. Training Times

• All-layer retraining takes longer per epoch but generally converges in fewer epochs, so training-time differences between retraining schemes are not significant.Training stops when performance fails to improve for three consecutive epochs, so training times do not increase linearly with dataset size.Source: §6.3 · p. 10

6.4. When the crowd and ML differ

• For perfect-agreement images, the system misses 12 of nearly 2,000 test cases; difficult cases include low-signal-to-noise, faint, small, or compact galaxies.Some errors are partial classifications confusing ACW with CW or EDGE; another case appears affected by a nearby bright star.Source: §6.4 · p. 10

Annotator agreement

• For agr ≥0.6, EM exceeded 99%; with perfect agreement, the model differed from the crowd on 12 of 1837 test images (EM 99.35%).Low-agreement classifications were less useful, and the study introduces hierarchical agreement as a basis for future GZ2 work.Source: §7 · p. 10

Network training

• With sufficient training data, retraining all network layers outperformed retraining only the last layer by 10% in EM; training from scratch was not evaluated.The authors attribute the benefit to pre-trained features that are not optimal for galaxy classification; training from scratch would require more labeled data and time.Source: §7 · pp. 10–11

Hierarchical classification

• In GZ1, adding the hierarchy yielded no significant overall improvement, but Bayesian hierarchical post-processing improved performance with limited data.The authors suggest hierarchy may have greater impact in larger, more complex schemes such as GZ2.Source: §7 · p. 11

Curriculum learning

• For agr ≥0.8 evaluated on all test images, curriculum learning achieved EM 0.83 versus 0.80 for the accumulative scheme, particularly benefiting limited-data settings.Curriculum learning trains in stages from easier cases to more difficult ones.Source: §7 · p. 11

Ensembling

• Across the challenging full test set, the five-model ensemble improved every evaluation measure over individual classifiers, reaching EM 91.04%, Accuracy 93.64%, and hF 94.65%.The test set included images with very low annotator agreement.Source: §7 · p. 11

8. Conclusions

• Retraining all layers significantly differs from retraining only the last layer, though last-layer training may suit very limited data.The authors note that full-network retraining may matter more when pretraining used a different kind of image.Source: §8 · p. 11
• Incorporating the hierarchy during post-processing may improve accuracy when training data are limited.Source: §8 · p. 11
• Higher annotator agreement produces better models, with this result demonstrated for agreement levels of 0.6 or greater.Source: §8 · p. 11
• For GZ1 with 6 classes and agr = 1, 5000 or more samples suffice for full-model retraining, while accuracy falls with 1000 or fewer.The required quantity varies with the number and complexity of morphological classes.Source: §8 · p. 11
• Training progressively across agreement ranges, from easier to harder cases, can offer advantages over training on all samples at once.Source: §8 · p. 11
• An ensemble of models provides a small improvement over the best individual model.Source: §8 · p. 11
• The findings may inform deep-learning analysis of upcoming wide-field sky surveys, where visual inspection of all data becomes impractical.Source: §8 · p. 11

Appendix A: Additional information of the data and results

• Appendix A provides additional information on image counts by agreement range and hierarchical evaluation measures across the experiments.It covers cumulative ranges, exclusive ranges, and evaluation on the complete test set across agreement ranges.Source: Appendix A · p. 13

A.1. Number of images and other evaluation metrics: cumulative ranges of agreement

• Table A.1 gives the number of images associated with each class for each cumulative agreement range.Source: Table A.1 · p. 13
• Table A.2 reports CNN and BCNN evaluation measures across cumulative ranges when training is limited to the CNN’s last layer.Source: Table A.2 · p. 13, §A.1 · p. 13
• Table A.3 reports CNN and BCNN evaluation measures across cumulative ranges when training updates weights in all CNN layers.Source: Table A.3 · p. 13, §A.1 · p. 13

A.2. Number of images and other evaluation metrics: exclusive ranges of agreement

• Table A.4 gives the number of images associated with each exclusive range.Source: Table A.4 · p. 13
• Table A.5 reports CNN and BCNN results on several evaluation measures for each exclusive range with transfer learning.Source: Table A.5 · p. 13

A.3. Additional evaluation metrics for the evaluation on the full test set

• Table A.6 reports several full-test-set evaluation measures for CNN and BCNN models trained on cumulative ranges, with training limited to the CNN’s last layer.The passage does not provide the table’s cell values.Source: Table A.6 · p. 14, §A.3 · p. 14
• Table A.7 reports several full-test-set evaluation measures for CNN and BCNN models trained on cumulative ranges, with all CNN layers updated.The passage does not provide the table’s cell values.Source: Table A.7 · p. 14, §A.3 · p. 14
• Table A.8 reports full-test-set evaluation measures for CNN and BCNN models trained by exclusive ranges, with all CNN layers updated and transfer learning.The passage describes the evaluation scope and training condition but does not provide the table’s cell values.Source: Table A.8 · p. 14, §A.3 · p. 14

Appendix B: Sample images

• Only 12 of 1,837 test images differ despite perfect crowd agreement; Figure B.1 shows four such examples.The figure places crowd annotations above the ML classifications.Source: Figure B.1 · p. 15, Appendix B · p. 15
• Figure B.2 shows eight examples where the CNN differs from the crowd for difficult cases with agr ≤0.4.The figure places crowd annotations above the ML classifications.Source: Figure B.2 · p. 15, Appendix B · p. 15