Source-linked AI summary
Reproducing GW150914: the first observation of gravitational waves from a binary black hole merger
Duncan A. Brown, Karan Vahi, Michela Taufer, Von Welch, Ewa Deelman
TL;DR
The paper addresses whether the GW150914 significance analysis can be directly reproduced from public artifacts despite incomplete documentation and unavailable original infrastructure. It reconstructs and runs the PyCBC workflow on the Open Science Grid, reproducing the main search result but not exactly matching the original because the original input data and metadata were unavailable.
Problem
The original GW150914 analysis was difficult to reproduce because its exact software provenance, workflow instructions, scripts, and input data were not fully public.
Method
The authors reconstruct the PyCBC workflow from public code and configuration files, supplement missing information with original-analysis notes, and execute it on the Open Science Grid.
Results
The authors reproduced the PyCBC GW150914 workflow and search result, but the reproduced background and significance differed slightly from the original.
Takeaways & Limitations
The work shows that major scientific results can be reproduced when code, data, provenance, containers, archival materials, and resource requirements are documented and shared.
Takeaways & Limitations
Exact reproduction was impossible because the original input data and metadata were unavailable, and detector calibration can change over time.
Abstract
from arXiv · showhide
In 2016, LIGO and Virgo announced the first observation of gravitational waves from a binary black hole merger, known as GW150914. To establish the confidence of this detection, large-scale scientific workflows were used to measure the event's statistical significance. They used code written by the LIGO/Virgo and were executed on the LIGO Data Grid. The codes are publicly available, but there has not yet been an attempt to directly reproduce the results, although several analyses have replicated the analysis, confirming the detection. We attempt to reproduce the result presented in the GW150914 discovery paper using publicly available code on the Open Science Grid. We show that we can reproduce the main result but we cannot exactly reproduce the LIGO analysis as the original data set used is not public. We discuss the challenges we encountered and make recommendations for scientists who wish to make their work reproducible.
THE DISCOVERY OF GW150914
The paper explains how the PyCBC matched-filtering workflow established GW150914’s significance and why reproducing it requires substantial computational infrastructure and noise-background estimation.
- THE DISCOVERY OF GW150914: The original analysis ran on the LIGO Data Grid, which is unavailable to the wider community, so this work executes it on the Open Science Grid.The authors use Pegasus and HTCondor as part of the supporting cyberinfrastructure.
- THE DISCOVERY OF GW150914: The reproduction matches the GW150914 search result but not the original background, likely because the underlying gravitational-wave strain data differ.The figure compares candidate-event counts and mean background events across the detection statistic.
- THE DISCOVERY OF GW150914: PyCBC searches for merging black holes by matched-filtering LIGO data against a bank of template binary-black-hole waveforms.The method compares detector data with modeled target waveforms.
- THE DISCOVERY OF GW150914: Non-Gaussian transients and non-stationary detector noise make candidate-event significance estimation more complex than a simple signal-to-noise threshold.The workflow estimates the noise background empirically through repeated time shifts between detectors.
- THE DISCOVERY OF GW150914: The workflow uses longer stretches of detector data to measure the noise background and determine the statistical significance of candidate events.The low-latency search identifies interesting candidates but does not provide their final statistical significance.
REPRODUCING THE ANALYSIS
The authors directly reproduce the original PyCBC analysis using public artifacts, original-analysis notes, and workflow automation. They obtain the main result while exposing missing provenance, unpublished scripts, and software-stack challenges.
- REPRODUCING THE ANALYSIS: The authors combined publicly released codes and configuration files with original-analysis notes to resolve missing information and build a reproducible workflow.They used peer programming and independently ran the scripts to test whether non-original analysts could reproduce the result.
- REPRODUCING THE ANALYSIS: This is presented as the first attempt to reproduce the original LIGO analysis rather than merely replicate its conclusion with a different setup.The earlier 1-OGC analysis independently confirmed GW150914 using a modified pipeline and recovered slightly different parameters.
- REPRODUCING THE ANALYSIS: PyCBC’s PyInstaller bundles packaged the interpreter and dependencies, allowing the original analysis code to run without recreating its entire software stack.The final workflow script and its iterative fixes were released with the paper and documented through GitHub history.
- REPRODUCING THE ANALYSIS: The discovery and technical papers did not document the exact PyCBC version, and the figure-generation script was not publicly released.The original run notes identified PyCBC v1.3.2, while the plotting script required additional reconstruction.
WORKFLOW EXECUTION
The authors adapted the public LIGO workflow to use public data and services, then executed its computationally intensive jobs across local and Open Science Grid resources. Reproduction required handling software, resource, and workflow-execution challenges.
- Public-data adaptation: The workflow was modified to replace proprietary data and metadata services with public GWOSC data and wrapper codes.The wrappers preserved the command-line API while translating requests to public repositories.
- Workflow deployment: Pegasus 4.9.3 was used to make the PyCBC v1.3.2 workflow compatible with the Open Science Grid.The workflow-generation script modified the original workflow for the newer Pegasus version.
- Resource variability: Matched-filtering memory use varied with the analyzed data, so failed jobs were retried with progressively larger memory requests.More non-Gaussian noise increased memory needed for signal-based veto calculations.
- Resource requirements: Post-processing jobs requiring more than 128 GB of memory were run on a local HTCondor cluster because suitable Open Science Grid machines were difficult to find.Cluster administrators coordinated availability of sufficiently large partitionable slots.
- Workflow outputs: The workflow produced a 2.5 Gb HDF5 file containing coincident triggers and a time-slide estimate of the search background.A compressed version was archived on IEEE DataPort.
RESULTS
The reproduction generated the data and plotting environment needed to compare workflow outputs with the original LIGO result. The search result was reproduced, but small background and significance differences remained because the original C01 data were unavailable.
- Plot generation: The workflow generated data for Figure 4, but a separate plotting script was required to create the histogram.The original plotting script and its complete software stack were not publicly available.
- Reproducibility limitation: The missing original plotting environment demonstrates that source-code release alone may not preserve reproducibility.The authors emphasize releasing containerized executables alongside source code.
- Workflow reproduction: The authors reproduced the LIGO analysis workflow after addressing workflow-generation and software-bundle issues.The workflow contained almost 42,000 tasks.
- Software environment: The reproduced result was generated with an automated script after installing the original PyCBC and LALSuite versions and required dependencies.The reproduction included the original LIGO/Virgo result for comparison.
- Resource characterization: The memory histogram spans 1 to 256 GB on logarithmic X and Y axes, while the runtime histogram spans 0 to 380,000 seconds.These distributions were provided to help future researchers estimate resource requirements.
- Search result: The search result was reproduced, but the significance lower bound was greater than 5σ rather than greater than 5.1σ.The authors attribute the difference to changes in the input data, particularly public C02 data versus the original C01 data.
CONCLUSIONS
The study shows that publicly released instructions and code can largely reproduce the GW150914 PyCBC workflow, while exact reproduction remains limited by unavailable original data and metadata. It also provides practical guidance, profiling, and scripts to support reproduction on alternative compute resources.
- Reproducibility support: The workflow can run on other resources, with local-resource instructions provided alongside the OSG configuration.The authors configured compute-intensive jobs for the OSG and released instructions for local execution.
- Resource requirements: Memory and runtime profiling supports appropriate resource selection for executing workflow tasks.The profiling is intended to help scientists choose suitable computational resources.
- Reproducibility support: The released scripts allow scientists to reproduce the LIGO analysis using public data on local resources or the Open Science Grid.The workflow uses current stable versions of Pegasus and HTCondor.
- Implementation considerations: HTCondor is retained because workflow jobs have widely varying memory requirements and benefit from failure resubmission in a relatively homogeneous environment.The authors recommend against alternative schedulers despite the possibility of modifying the workflow script.
- Recommendations: Reproducibility recommendations include access to original data and code, version documentation, containers, long-term archiving, and resource requirements.The recommendations are presented as conditions that scientists should address when releasing reproducible workflows.
- Limitations: Exact reproduction was not possible because the original input data and metadata used for GW150914 are unavailable.Only final calibrated data from LIGO’s first observing run is public, and calibration can change as detector knowledge improves.