Source-linked AI summary
A morphological algorithm for improving radio-frequency interference detection
A. R. Offringa, J. J. van de Gronde, J. B. T. M. Roerdink
TL;DR
Radio-frequency interference can be missed by amplitude thresholding when received power varies across time and frequency. The paper extends threshold-generated masks with a scale-invariant rank operator and provides an exact linear-time algorithm. The method improves detection for tested RFI patterns, is used in the default LOFAR pipeline, and empirically improves the calibratability of LOFAR observations.
Problem
Variations in received RFI power can cause amplitude thresholding to miss portions of interference, while lowering thresholds increases false positives.
Method
The paper applies a scale-invariant rank operator to one-dimensional time or frequency sequences in a threshold-generated mask and implements it exactly in linear time.
Results
At η = 0.2, SIR detected 98.9% of Gaussian, 99.9% of sinusoidal, and 100% of burst vertical features, with minor false-positive increases.
Takeaways & Limitations
The method is used in LOFAR’s default pipeline and has empirically improved the calibratability of LOFAR observations.
Takeaways & Limitations
Strong intermittent transmitters can be a worst-case situation, and applying η = 0.2 could theoretically yield 40% false positives for such sources.
Abstract
from arXiv · showhide
A technique is described that is used to improve the detection of radio-frequency interference in astronomical radio observatories. It is applied on a two-dimensional interference mask after regular detection in the time-frequency domain with existing techniques. The scale-invariant rank (SIR) operator is defined, which is a one-dimensional mathematical morphology technique that can be used to find adjacent intervals in the time or frequency domain that are likely to be affected by RFI. The technique might also be applicable in other areas in which morphological scale-invariant behaviour is desired, such as source detection. A new algorithm is described, that is shown to perform quite well, has linear time complexity and is fast enough to be applied in modern high resolution observatories. It is used in the default pipeline of the LOFAR observatory.
1. Introduction
Radio observations increasingly suffer from strong man-made radio-frequency interference as telescopes become more sensitive and survey broader, more crowded spectral ranges. The paper presents morphology-based processing to extend threshold-detection masks across likely RFI gaps while preserving scale-invariant behavior.
- Modern radio telescopes observe broader, more crowded spectra, allowing man-made transmitters far stronger than celestial signals to disturb observations.
- RFI commonly affects neighboring time-frequency samples in line-shaped patterns, which existing methods such as SumThreshold exploit to improve detection accuracy.
- Because received RFI power varies over time and frequency, amplitude thresholding can miss parts of continuously received interference without simply lowering thresholds.Lowering thresholds can increase false positives and data bias.
- Standard morphological dilation is not scale invariant and can create many false positives for sharp RFI features, motivating a scale-invariant rank operator.
- The paper analyzes the morphology-based technique on simulated and real observations, compares it with ordinary dilation, and introduces a fast linear-time implementation.
2. The scale-invariant rank operator
The scale-invariant rank (SIR) operator extends flagged intervals in one-dimensional time or frequency slices according to their flagged-sample fraction, making the operation independent of feature scale. Its exact implementation runs in linear time, while one-dimensional outputs can be combined in several ways to form a two-dimensional RFI mask.
- Motivation: SIR addresses the scale dependence of ordinary morphological dilation, which can perform differently on RFI features at different scales.Scale invariance allows data with different resolutions to be processed without changing parameters, although finite discrete observations limit this property in practice.
- Definition: The operator is called the scale-invariant rank (SIR) operator because it flags points when flagged samples exceed a threshold within a neighborhood.The proposed operator was previously described as a dilation in the AOFlagger pipeline but does not satisfy all properties of morphological dilation.
- Definition: Given a one-dimensional time or frequency slice, SIR flags every sample in a contiguous interval whose flagged-sample ratio is at least 1−η.η controls aggressiveness: η = 0 adds no samples, whereas η = 1 flags all samples.
- Two-dimensional construction: The one-dimensional SIR outputs can be remapped to two dimensions using a union or by applying the operator sequentially in the two directions.Sequential processing is more aggressive and can flag samples with diagonal flagged neighbors; the processing order also affects the result.
- Scale invariance: The SIR operator is formally scale invariant because scaling the input by λ and then applying ρ equals scaling the output by λ.The proof uses the fact that the operator is defined by the fraction of flagged samples in an interval; discrete finite data impose practical resolution and size limits.
- Algorithm: The exact algorithm reduces straightforward evaluation from O(N^2) to Θ(N) time and O(N) temporary storage.It performs 3N additions or subtractions and 3N−2 floating-point comparisons, using temporary arrays whose total storage is linear in N.
3. Analysis & results
Simulations and ROC analysis show that SIR generally detects undetected line-like RFI more accurately than dilation, while remaining computationally practical for LOFAR.
- Computational performance: The linear algorithm is faster than both the approximate O(N log N) and full quadratic implementations across tested input sizes.Figure 5 measured runtime on a single 3.07 GHz Intel Core i7 core.
- Computational performance: In LOFAR, processing 100,000 time steps and 256 channels takes 3.8 seconds per sub-band and baseline, with SIR requiring 49 milliseconds (1.3%).The pipeline parallelizes baselines across cores and sub-bands across cluster nodes.
- Accuracy analysis: With η = 0.48, SIR reaches 100% TP with a 1.36% FP ratio for the tested features except the slanted feature.SumThreshold pre-detection supplied the starting point for the ROC curves, including 91.3% TP and 0.38% FP for the Gaussian feature.
- Accuracy analysis: SIR detects all previously undetected samples on straight features with only a slight false-positive increase and outperforms dilation in every tested situation.The simulations used four line-shaped RFI features and repeated each noisy experiment 100 times.
- Accuracy analysis: A setting of η ∼0.2–0.4 appears to provide a good compromise between TP and FP ratios.Changing feature SNR scales the ROC curves but preserves the relative difference between SIR and dilation.
- Two-dimensional application: Sequential two-dimensional application can amplify the difference from dilation because dilation false positives are multiplied across dimensions, while SIR fills and slightly extends complex features.The comparison concerns applying the operators first along one dimension and then using that output in the second.
- Observed data examples: For a strong broadband RFI event, SIR can falsely flag data before its sudden start, but such events are described as exceptional.A WSRT example left 1.5 hours partly unflagged after SumThreshold; SIR solved this while falsely flagging about 20 minutes before the event.
- Accuracy analysis: Absolute ROC true- and false-positive ratios are not accurate representations of actual RFI detection because the simulations use simplistic, well-defined RFI models.A realistic estimate places LOFAR RFI occupancy at a few percent of samples.
4. Conclusions and discussion
The SIR operator improves RFI detection with small false-positive increases, while remaining fast enough for LOFAR's pipeline. Its usefulness is strongest for vertical features and narrower for slanted or intermittent RFI.
- Results: At η = 0.2, SIR detected 98.9% of Gaussian, 99.9% of sinusoidal, and 100% of burst vertical features.The corresponding false-positive ratios were 0.69%, 0.95%, and 1.3%.
- Results: SIR increased vertical-feature detection by 7.6% for Gaussian, 5.8% for sinusoidal, and 6.7% for burst features.The slanted feature reached 86% detection with a 6.1% increase.
- Practical implications: The method adds few false positives and avoids the bias associated with lowering amplitude thresholds to recover remaining RFI samples.Morphological mask extension does not bias the final distribution of uncorrelated white noise, unlike thresholding.
- Practical implications: The method is extremely fast, with processing time described as almost negligible in a full RFI pipeline.The SIR operator has been used in the default LOFAR pipeline after validation of η = 0.2.
- Limitations and scope: Empirically, morphological processing improved the calibratability of LOFAR observations, although strong intermittent transmitters could theoretically produce 40% false positives.The paper identifies strong intermittent transmitters on approximately minute scales as a worst case for η = 0.2.
- Future directions: A true two-dimensional SIR operator could select rectangles with flagged-value ratio ≥η, but it is likely less attractive because it may not retain linear time complexity.The current approach combines one-dimensional operators in the time-frequency domain.