PaperlayerView exact revision on arXiv

Source-linked AI summary

Gravity Spy: Integrating Advanced LIGO Detector Characterization, Machine Learning, and Citizen Science

Michael Zevin, Scott Coughlin, Sara Bahaadini, Emre Besler, Neda Rohani, Sarah Allen, Miriam Cabero, Kevin Crowston, Aggelos K Katsaggelos, Shane L Larson, Tae Kyoung Lee, Chris Lintott, Tyson B Littenberg, Andrew Lundgren, Carsten Oesterlund, Joshua R Smith, Laura Trouille, Vicky Kalogera

arXiv:1611.04596v2gr-qcastro-ph.HEastro-ph.IMphysics.ins-det

TL;DR

LIGO glitches are abundant, diverse transients that contaminate gravitational-wave analyses, while existing methods do not capture their full range. Gravity Spy combines volunteer classifications through Zooniverse with machine learning that classifies glitches and routes uncertain cases for further review. Beta testing demonstrated reliable volunteer classification and new morphological classes, while the project aims to aid detector characterization and scale citizen science; its preliminary scope includes limited O1 glitch categorization.

  • Problem

    Abundant, diverse glitches contaminate LIGO data, and existing automated methods do not capture their full morphological range.

  • Method

    Gravity Spy combines Zooniverse volunteer classifications with machine learning trained on human labels to classify glitches and identify new categories.

  • Results

    Beta testing engaged over 1400 users, produced over 45,000 glitch classifications, and demonstrated reliable classification and identification of new morphological classes.

  • Takeaways & Limitations

    The project shows utility for aiding LIGO detector characterization while providing a scalable way to combine human and computer classification.

  • Takeaways & Limitations

    Glitches appearing during O1 were not analyzed or categorized to the level of pre-existing glitches.

Abstract

from arXiv · show

(abridged for arXiv) With the first direct detection of gravitational waves, the Advanced Laser Interferometer Gravitational-wave Observatory (LIGO) has initiated a new field of astronomy by providing an alternate means of sensing the universe. The extreme sensitivity required to make such detections is achieved through exquisite isolation of all sensitive components of LIGO from non-gravitational-wave disturbances. Nonetheless, LIGO is still susceptible to a variety of instrumental and environmental sources of noise that contaminate the data. Of particular concern are noise features known as glitches, which are transient and non-Gaussian in their nature, and occur at a high enough rate so that accidental coincidence between the two LIGO detectors is non-negligible. In this paper we describe an innovative project that combines crowdsourcing with machine learning to aid in the challenging task of categorizing all of the glitches recorded by the LIGO detectors. Through the Zooniverse platform, we engage and recruit volunteers from the public to categorize images of glitches into pre-identified morphological classes and to discover new classes that appear as the detectors evolve. In addition, machine learning algorithms are used to categorize images after being trained on human-classified examples of the morphological classes. Leveraging the strengths of both classification methods, we create a combined method with the aim of improving the efficiency and accuracy of each individual classifier. The resulting classification and characterization should help LIGO scientists to identify causes of glitches and subsequently eliminate them from the data or the detector entirely, thereby improving the rate and accuracy of gravitational-wave observations. We demonstrate these methods using a small subset of data from LIGO's first observing run.

1. Introduction

LIGO’s extreme sensitivity enables gravitational-wave astronomy but also leaves its data vulnerable to abundant, diverse glitches. Gravity Spy combines citizen science and machine learning to classify these glitches, discover evolving categories, and support detector characterization.

  • Motivation: Glitches are transient, non-Gaussian instrumental or environmental artifacts that can create false positives, reduce candidate significance, bias parameter estimates, and reduce analyzable data.Approximately 10^6 glitches above SNR 6 were recorded during 51.5 days of O1.
  • Motivation: The diversity and volume of glitch morphologies exceed the coverage of existing automated methods and the capacity of small groups of scientists.Only a small number of classes were understood well enough for confident removal from data.
  • Gravity Spy: Gravity Spy integrates gravitational physics, human-centered computing, machine learning, and citizen science into a socio-computational glitch-classification system.The project is designed to leverage complementary human pattern recognition and computational analysis.
  • Gravity Spy: Volunteers label known glitch morphologies and identify new categories, while machine learning models learn from those labels and rapidly classify the full glitch dataset.The system uses machine classifications to guide information presented back to participants.
  • Paper scope: The paper describes the project architecture, data preparation, interface, classification algorithms, social-science experiments, and preliminary O1 results.The paper’s preliminary evaluation uses data from LIGO’s first observing run.

2. Characterization of transient noise in LIGO

Glitches compromise gravitational-wave searches and parameter recovery despite detector coincidence checks and existing identification tools. LIGO therefore uses classification, auxiliary-channel information, and data vetoes to characterize and mitigate these transients, while facing substantial morphological and temporal diversity.

  • Impact of glitches: Accidental coincidence remains non-negligible because glitches occur frequently in both LIGO detectors, weakening coincidence-based rejection of non-cosmic disturbances.A candidate appearing in only one detector within the inter-site light-travel window is rejected, but frequent glitches can still coincide.
  • Impact of glitches: Glitches raise search-background loudness, reduce candidate significance, impair astrophysical parameter recovery, and can reduce the signal-to-noise ratio of nearby events.These effects affect both modeled and unmodeled gravitational-wave searches.
  • Identifying glitches: LIGO groups glitches by common origins or morphologies, with some causes known and others unidentified.Blips have unknown causes in the cited example, whereas whistles arise from radio signals beating with interferometer Voltage Controlled Oscillators.
  • Identifying glitches: Automated identification algorithms search time-frequency data and auxiliary channels, parameterizing glitches by properties including time, frequency, and SNR.Auxiliary channels monitor instrumental and environmental factors that may affect the detectors.
  • Mitigating glitches: Data vetoes remove times strongly affected by noise transients when a glitch source cannot be determined or fixed.Vetoes are intended to improve gravitational-wave searches by excluding contaminated intervals.
  • Mitigating glitches: Existing methods do not capture all glitch morphologies, miss transiently appearing classes, and would benefit from category-specific information for identifying correlated auxiliary channels.These limitations motivate more detailed glitch classification and characterization.

3. Gravity Spy Project

Gravity Spy addresses data-intensive scientific analysis by combining volunteer pattern recognition with machine learning’s systematic processing. Its workflows use labeled human classifications to train models, route uncertain cases back to volunteers, and study human-computer interaction.

  • Project motivation: Large modern scientific datasets make traditional researcher-by-researcher inspection impractical, motivating crowdsourcing and more intelligent citizen-science methods.Zooniverse projects have recruited large volunteer populations and contributed to peer-reviewed research.
  • Project motivation: Gravity Spy combines human and machine classification to allocate tasks more intelligently and support analysis of increasingly large datasets.The system uses human pattern recognition for image recognition and machine learning for systematic analysis.
  • System workflow: Volunteers classify glitches through workflows and provide labeled morphological classes that train machine learning algorithms.The workflow is the human-classification interface for LIGO data-stream images.
  • System workflow: Trained models classify the full glitch dataset, assign confidence scores, and feed questionable cases back to volunteers for further analysis.This creates an iterative relationship between automated classification and human review.
  • Human-centered computing: Expert-labeled “gold standard” images evaluate individual users, guide volunteer training, and expedite glitch retirement and growth of machine-learning training sets.User performance is measured by comparison with expert classifications.
  • Human-centered computing: Gravity Spy provides a platform to study how non-experts perform classification tasks, learn the task and science, and interact with socio-computational systems.The project explicitly examines human-centered computing questions alongside detector characterization.

3.1. Data Preparation

Gravity Spy prepared O1 glitch data for human and machine classification by selecting relevant triggers, representing them as Omega Scans, and building a labeled morphological training set.

  • Preparation challenges: Data preparation addressed trigger selection, morphology presentation, and training-set construction for the diverse O1 glitch population.The project faced more than 10^6 Omicron triggers and lacked a complete catalog of O1 glitch categories.
  • Data selection: The dataset retained glitches recorded while LIGO was locked and observing, while excluding O1 periods flagged for poor data quality.These criteria targeted glitches relevant to gravitational-wave searches and data analysis.
  • Glitch representation: Omega Scans represented glitches in time-frequency-energy space using Q-transform tiles and normalized energy related to tile SNR.The loudest Q template and most significant tile were used to generate the spectrogram representation.
  • Glitch representation: Each glitch was centered in Omega Scan images and shown across four time windows: ±0.25, 0.5, 1.0, and 2.0 seconds.Both volunteers and machine-learning algorithms received all four durations for classification.
  • Training set: A small human-identified set initialized machine-learning classifiers that achieved approximately 80%-90% accuracy and helped organize unlabeled glitches by morphology.This organization supported easier visual classification and enabled identification of two new classes during beta testing.
  • Training set: A labeled O1 training set contained 7718 glitches from Livingston and Hanford, grouped into 20 morphological classes and covering 8.58 hours, or 0.7%, of O1 data.The class proportions and detector contributions were summarized in Table 1.

3.2. Citizen Science

Gravity Spy combines volunteer classification with machine learning through a staged interface that trains participants, routes images by confidence, and supports evolving glitch categories.

  • User interface: Volunteers classify individual Omega Scan images through a Zooniverse interface offering 20 known glitch classes and multiple renderings of each glitch.Users can inspect different durations, consult class information and examples, and discuss uncertain cases through Talk.
  • User interface: The interface supports discovery of new morphologies through “None of the Above” classifications, follow-up descriptions, and discussion with project scientists.New classes can be integrated as detector glitches change over time.
  • Volunteer training: Machine-learning results train volunteers systematically by progressing them through levels with increasing numbers of glitch classes and more sophisticated interface features.The system selects novel images needing classification rather than relying only on a fixed training set.
  • Volunteer training: Volunteers receive gold-standard images with immediate expert-agreement feedback; initially, 40% of beginning volunteers’ images are gold standard, declining with correct classifications.These images also support measurement of individual volunteer reliability.
  • Workflows: Images are routed to beginner, intermediate, or advanced workflows using machine-learning confidence, while volunteer reliability and expertise guide assignments.The workflow includes images with uncertain machine-learning classifications to support further human review and category discovery.
  • Workflows: Retirement decisions combine machine-learning and user-confidence posteriors, weighting classifications by prior volunteer performance rather than relying only on a fixed count.This aims to reduce wasted classifications on easy images and avoid retiring images despite strong disagreement.

3.3. Machine Learning

Gravity Spy uses deep convolutional neural networks to classify glitch images, combining machine-learning probabilities with volunteer labels and reliability models. The system routes images and volunteers through workflows based on confidence and demonstrated performance.

  • Image Classifier: The Gravity Spy system uses a deep convolutional neural network for image classification because deep learning scales to copious amounts of data.The model is described as using CNN layers and is illustrated in Figure 6.
  • Image Classifier: Four glitch durations are merged into a square input so convolution kernels can learn patterns across all durations.The merged durations are presented as the input to the CNN.
  • Image Classifier: Convolution, maxpooling, ReLU activation, and fully connected layers transform the input into features for classification.Convolutional kernels produce feature maps, which are downsampled with maxpooling before later network layers.
  • Image Classifier: The final softmax layer has 20 outputs and converts the classifier’s scores into a probability distribution over glitch classes.For image i, the output scores depend on the preceding-layer representation and the weights connecting it to each class node.
  • Crowdsource Classifier: Images are initially classified by machine learning, then routed to beginner, intermediate, or advanced volunteer workflows according to confidence.Volunteer levels are determined from classification performance and reliability estimates based on confusion matrices.
  • Crowdsource Classifier: The system fuses machine-learning probabilities with volunteer labels, updating posterior probabilities until an image is retired or escalated for further review.Retirement occurs when the highest posterior exceeds its class-specific threshold; otherwise, additional or more advanced volunteers are engaged.

3.4. Socio-Computational Research Support

Gravity Spy’s socio-computational research component evaluates how training, workflow parameters, feedback, and astrophysics instruction affect volunteer performance. These experiments are intended to identify effective settings while examining possible tradeoffs.

  • Training and Performance: The project compares volunteers who complete the training process with those who immediately receive the full classification set.The comparison tests whether trained users contribute more and perform better on classification tasks.
  • Workflow Parameters: Experiments vary the number and selection of classes introduced at each level and the class-specific machine-learning confidence cutoffs.These parameters are studied to determine optimal workflow settings.
  • Workflow Parameters: The project examines whether faster volunteer advancement improves motivation while potentially reducing performance.The paper identifies advancement speed as a setting with possible tradeoffs.
  • Volunteer Feedback: Researchers study how motivational messages, glitch novelty information, and machine-learning confidence feedback affect volunteer performance.The paper identifies possible tradeoffs in revealing machine-learning confidence scores to volunteers.
  • Volunteer Learning: Workflows include a mini-course on gravitational-wave astrophysics and LIGO detector characterization to investigate how much domain knowledge volunteers need.The paper presents this as an unanswered question about volunteer learning in the classification setting.

4. Preliminary Results

Preliminary tests show that Gravity Spy can classify real LIGO glitches effectively with CNNs while volunteers reliably classify glitches and uncover new morphological categories. CNN performance is strongest for well-sampled, morphologically consistent classes.

  • Initial Machine Learning Performance: 97.1% average accuracy was achieved on testing data from 20 glitch classes using CNNs.The initial dataset contained 7,718 glitches, split into training, validation, and test sets at 75%, 12.5%, and 12.5%.
  • Initial Machine Learning Performance: Precision and recall were near unity for most glitch classes in the CNN classification results.The confusion matrix reports predicted classes on the x-axis and true classes on the y-axis, with diagonal values equal to recall under its normalization.
  • Initial Machine Learning Performance: Lower precision and recall occurred for classes with few training samples or highly variable morphologies, including Wandering Line, Paired Doves, None of the Above, and No Glitch.The catch-all classes lack specific morphological distributions, while Paired Doves and Wandering Line had only 30 and 44 samples, respectively.
  • Gravity Spy System Beta Testing Results: Beta testing engaged over 1,400 users and produced more than 45,000 glitch classifications.Across three approximately one-week beta versions, volunteers reliably classified glitches and identified new morphological classes.
  • Gravity Spy System Beta Testing Results: Beta testing led to multiple new glitch categories from O1, including Paired Doves and Helix.Paired Doves may relate to 0.4 Hz beamsplitter motion at Hanford, while Helix may relate to auxiliary photon-calibrator lasers.

5. Conclusions and Future Prospects

Gravity Spy combines citizen science and machine learning to improve LIGO detector characterization while adapting to newly observed glitch classes and sustaining volunteer participation. Its future system integrates automated analysis, human classification, evolving categories, and additional tools.

  • Conclusions and Future Prospects: Gravity Spy aims to improve understanding of LIGO detectors and reduce harmful noise while engaging the public in gravitational-wave physics.The full launch paired machine-learning analysis with a crowdsourced classifier and tailored user workflows.
  • Conclusions and Future Prospects: Machine-learning confidence scores are paired with user classifications to optimize image retirement and classification accuracy.The project is presented as useful for LIGO detector characterization and as a test bed for socio-computational interaction.
  • Conclusions and Future Prospects: During future observing runs, new trigger images will be analyzed by current machine-learning classifiers and added to human classification sets.Retired images are added to machine-learning training data, creating an ongoing human–machine workflow.
  • Conclusions and Future Prospects: New glitch categories can be added when emerging None of the Above trends and detector classes are identified.The project is designed to evolve with changes in detector behavior and glitch populations.
  • Conclusions and Future Prospects: Future engagement tools include auxiliary-channel spectrograms, deeper morphological subclasses, and collaboration features for discovering new glitch classes.Continued interaction between project scientists and volunteers is intended to support sustained participation and empirical socio-computational studies.
  • Conclusions and Future Prospects: The growth of scientific data motivates coupling citizen science with machine learning to allocate human effort more efficiently.The paper argues that simple crowdsourcing will not suffice as modern data volumes outstrip available volunteer time.
Loading 1611.04596v2…