Source-linked AI summary
Improving the Sensitivity of Gravitational Wave Detection with Weighted Conformal Prediction
Ann-Kristin Malz, Gregory Ashton, Nicolo Colombo
TL;DR
Multiple gravitational-wave search pipelines must be combined while maintaining reliable confidence estimates under shifts between simulated calibration data and unlabeled observations. The paper uses likelihood-ratio reweighting within Mondrian conformal prediction and finds that weighted conformal prediction improves coverage under dataset shift, increases confidence near the FAR threshold, and recovers signals at a sensitivity–purity trade-off.
Problem
Pipeline outputs must be combined robustly, but distribution shifts between calibration and test data can invalidate standard conformal-prediction coverage guarantees.
Method
The paper applies likelihood-ratio reweighting to Mondrian conformal prediction, comparing marginal and pseudo-label-based approximations to class-conditional shift correction.
Results
Weighted conformal prediction restores well-calibrated coverage under controlled covariate shifts and improves coverage toward the diagonal for shifts between mock datasets, while increasing confidence near the FAR threshold.
Takeaways & Limitations
Weighted conformal prediction can recover signals missed by standard conformal prediction, favoring completeness-sensitive applications, whereas standard conformal prediction remains preferable for high-purity catalogues.
Takeaways & Limitations
Approximate class-conditional likelihood ratios are biased without further assumptions, and their effect on coverage cannot be bounded without additional assumptions.
Abstract
from arXiv · showhide
In the last decade, kilometre-scale interferometric gravitational-wave detectors have observed hundreds of compact binary mergers, the majority of which are binary black holes. However, the data are noise-dominated, and multiple independent search algorithms (pipelines) are used to enhance sensitivity and improve robustness. Rather than the standard approach of selecting the most significant pipeline output, we combine the outputs from all pipelines using a conformal prediction-based framework to provide statistically rigorous confidence estimates for candidate events. While combining pipelines improves sensitivity and ranking robustness, it requires a principled statistical framework that remains valid as data properties evolve across observing runs. A key challenge is distribution shifts between simulated datasets used for training and calibration and the real, unlabelled, observations used for testing, which can invalidate coverage guarantees and bias confidence estimates. In this work, we address this challenge by incorporating likelihood-ratio reweighting into our conformal prediction framework to account for covariate shift. Using mock datasets containing simulated signals, we demonstrate that weighted conformal prediction restores well-calibrated coverage under covariate shift and increases the confidence of events near the detection threshold, recovering true signals that would otherwise be missed.
1. Introduction
Gravitational-wave searches analyze extremely weak signals in noisy data with multiple pipelines whose differing assessments must be combined into a robust significance measure. This work extends a machine-learning and conformal-prediction framework with likelihood-ratio reweighting to address distribution shifts between simulated calibration data and real observations.
- Gravitational-wave searches: Gravitational-wave detectors measure extremely weak signals embedded in noisy time-series data from compact-object mergers.Pipelines quantify candidate significance with FAR, a right-sided p-value normalized by time, using empirically estimated background distributions.
- Motivation: Multiple search pipelines can disagree because of methodological, configuration, and sensitivity differences, making a single robust candidate significance measure essential.The standard approach uses the minimum FAR across pipelines, which is simple and conservative but does not exploit informative discrepancies between outputs.
- Distribution shift: Simulated mock data may differ from real observations in detector noise, pipeline behavior, and astrophysical signal properties, creating distribution shifts that can invalidate standard conformal coverage guarantees.The framework therefore models the discrepancy through feature-distribution ratios and considers approximations that use unlabeled test data or pseudo-labels.
- Framework: The framework combines pipeline outputs with a machine-learning classifier and conformal prediction to produce one conditional confidence value for each candidate.Conformal prediction supplies distribution-free coverage guarantees and an interpretable significance measure when paired with the classifier.
- Contribution: The method applies likelihood-ratio reweighting to calibration points, extending weighted conformal prediction to label-conditional classification without retraining the underlying classifier.The paper’s methodological contribution is to compare practical marginal and pseudo-label-based approximations because exact class-conditional correction requires unavailable test labels.
- Contribution: The study demonstrates that weighted conformal prediction corrects naturally occurring dataset shifts in gravitational-wave applications, with implications for the reliability of astrophysical conclusions.Controlled shifts are first used to examine coverage and confidence behavior before applying the framework to different mock datasets.
2. Method
The framework combines multiple gravitational-wave pipeline outputs with logistic regression and Mondrian conformal prediction, then extends it with likelihood-ratio weighting for calibration–test distribution shifts. Because label-conditional likelihood ratios are unavailable without test labels, the method evaluates practical approximations and quantifies their coverage limitations.
- 2. Method: The method combines pipeline outputs with a logistic-regression classifier and uses Mondrian conformal prediction to provide label-conditional uncertainty quantification.The classifier uses pipeline features, while Mondrian CP calibrates prediction sets separately for signal and noise labels.
- 2.1. Conformal prediction: Mondrian conformal prediction provides finite-sample conditional coverage for each label, with a controlled-purity catalogue obtained by thresholding signal-class confidence.The guarantee holds conditionally within each class as well as marginally over the full dataset.
- 2.2. Conformal prediction under distribution shift: Standard conformal prediction assumes exchangeability, whereas covariate shift changes feature distributions between calibration and test data while leaving label conditionals unchanged.The framework addresses this setting by reweighting calibration samples according to their similarity to the test distribution.
- 2.2. Conformal prediction under distribution shift: Likelihood-ratio weighting restores nominal coverage when the conditional ratios are known or consistently estimated, and replaces standard empirical quantiles with weighted quantiles.The method applies estimated weights to both calibration and test points before constructing weighted prediction sets.
- 2.2. Conformal prediction under distribution shift: Weighted conformal prediction has less-tight finite-sample bounds because reweighting reduces the effective calibration sample size, especially when the test distribution is concentrated.Approximate likelihood ratios can also introduce unbounded bias in coverage without additional assumptions, motivating empirical evaluation and future theoretical work.
3. Experiments
Controlled and naturally occurring distribution shifts reveal that standard conformal prediction can miscalibrate coverage, while likelihood reweighting largely restores calibration and changes confidence near the detection threshold. On the LLPIC dataset, weighted CP recovers more signals than standard CP, but increases false positives.
- 3.1. Controlled covariate shift: Only the marginal likelihood-ratio weighting reliably recovers well-calibrated coverage across both controlled test shifts, so the study adopts it as the most robust choice.Proxy-label weighting can inherit misclassification errors and may fail when shifts are strongest.
- 3.1. Controlled covariate shift: Weighted CP returns approximately calibrated coverage across controlled shift intensities, whereas standard CP increasingly undercovers or overcovers as the feature shift grows.Lower-feature shifts produce undercoverage and higher-feature shifts produce overcoverage under standard CP; weighted CP remains within expected binomial deviations.
- 3.1. Controlled covariate shift: Conditional confidence increases under lower-feature shifts and decreases under higher-feature shifts, with the largest changes for candidates near the FAR threshold.Confidence changes little for events with high maximum-log10IFAR, where classification is less uncertain.
- 3.2. Unknown shift between two datasets: On LLPIC data, standard CP undercovers, while weighted CP moves coverage toward the diagonal but leaves some undercoverage outside the expected confidence band.The remaining mismatch suggests that the dataset shift is not purely covariate.
- 3.2. Unknown shift between two datasets: Weighted CP finds 98 additional true signals while reducing false negatives from 39% to 16%, at the cost of a sixfold increase in false positives.The authors therefore associate weighted CP with prioritizing signal recovery and standard CP with higher-purity catalogues.
4. Conclusion
The framework extends Mondrian conformal prediction with likelihood-ratio reweighting to address distribution shifts between calibration and test datasets. Weighted CP improves coverage and confidence near the FAR threshold, recovering more signals while increasing false positives.
- 4. Conclusion: Weighted CP restores coverage under controlled covariate shifts, while the marginal likelihood ratio is the most robust weighting choice examined.Class-conditional approximations depend on proxy labels whose accuracy is not guaranteed.
- 4. Conclusion: Under the unknown shift between MDC and LLPIC, weighted CP moves coverage toward the diagonal but leaves residual undercoverage, indicating that the shift is not purely covariate.The result suggests that class-conditional and marginal likelihood ratios may not be approximately equal.
- 4. Conclusion: Weighted CP increases conditional confidence especially for events near the FAR threshold, where classification is most uncertain.This adjustment recovers signals that standard CP would miss, but with a higher false positive rate.
- 4. Conclusion: The method entails a sensitivity–purity trade-off: weighted CP suits completeness-focused studies, whereas standard CP remains preferable for high-purity catalogues.Future work could improve trade-off control through adaptive thresholds or better likelihood-ratio estimation for complex shifts.