Source-linked AI summary
DiFX: A software correlator for very long baseline interferometry using multi-processor computing environments
A. T. Deller, S. J. Tingay, M. Bailes, C. West
TL;DR
VLBI correlation traditionally relied on purpose-built hardware, creating a need for flexible and economical approaches as computing environments evolved. The paper develops DiFX, an FX software correlator for parallel multi-processor systems, and reports scalable operation, broad flexibility, and agreement with existing hardware correlators. It concludes that software correlation is especially useful for small to medium arrays and specialized experiments, with efficiency and implementation limitations remaining.
Problem
VLBI correlation traditionally used purpose-built processors, motivating a flexible and economical alternative for varied observing requirements.
Method
The paper develops and evaluates DiFX, an FX software correlator running in generic multi-processor computing environments.
Results
DiFX supports relatively large telescope numbers and high data rates in real time on moderate to large Beowulf clusters, with performance trade-offs allowing fewer processors when real time is unnecessary.
Takeaways & Limitations
Software correlators are flexible and scalable across interferometric modes, making them suited to novel experiments with requirements unsupported by existing hardware correlators.
Takeaways & Limitations
Software correlation is less efficient for a given task than application-specific or FPGA-based solutions, and some optimization features are not currently implemented in DiFX.
Abstract
from arXiv · showhide
We describe the development of an FX style correlator for Very Long Baseline Interferometry (VLBI), implemented in software and intended to run in multi-processor computing environments, such as large clusters of commodity machines (Beowulf clusters) or computers specifically designed for high performance computing, such as multi-processor shared-memory machines. We outline the scientific and practical benefits for VLBI correlation, these chiefly being due to the inherent flexibility of software and the fact that the highly parallel and scalable nature of the correlation task is well suited to a multi-processor computing environment. We suggest scientific applications where such an approach to VLBI correlation is most suited and will give the best returns. We report detailed results from the Distributed FX (DiFX) software correlator, running on the Swinburne supercomputer (a Beowulf cluster of approximately 300 commodity processors), including measures of the performance of the system. For example, to correlate all Stokes products for a 10 antenna array, with an aggregate bandwidth of 64 MHz per station and using typical time and frequency resolution presently requires of order 100 desktop-class compute nodes. Due to the effect of Moore's Law on commodity computing performance, the total number and cost of compute nodes required to meet a given correlation task continues to decrease rapidly with time. We show detailed comparisons between DiFX and two existing hardware-based correlators: the Australian Long Baseline Array (LBA) S2 correlator, and the NRAO Very Long Baseline Array (VLBA) correlator. In both cases, excellent agreement was found between the correlators. Finally, we describe plans for the future operation of DiFX on the Swinburne supercomputer, for both astrophysical and geodetic science.
1. Introduction
DiFX replaces purpose-built VLBI correlators with flexible software running on parallel, general-purpose computing environments. This approach suits small to medium arrays and unusual experiments, while trading lower engineering costs and greater flexibility against lower task-specific efficiency.
- DiFX approach: DiFX is a Distributed FX software correlator designed to run in generic multi-processor environments and interface with disk-based VLBI recording systems.It was developed for the Australian Long Baseline Array but is broadly applicable beyond that system.
- Motivation: Commodity computing growth and the embarrassingly parallel correlation algorithm make inexpensive multi-processor VLBI correlation feasible.General-purpose facilities can also support computational tasks beyond correlation.
- Trade-offs: Software correlation reduces non-recoverable engineering costs but has higher cost per unit processing power than application-specific or FPGA-based hardware.The paper therefore presents software as cheaper overall for small correlators, while large EVLA- or ALMA-scale systems may justify hardware optimization.
- Benefits and applications: A software correlator can provide a million spectral points across an observation bandwidth, although conventional CPUs make such correlation computationally intensive.The same software and hardware can support generic continuum and high-resolution spectral-line experiments.
- Benefits and applications: Software can overcome hardware correlator limits on minimum integration times, maximum input data rates, and maximum output data rates.This flexibility is valuable when unusual or stringent experiment requirements are impractical to implement in existing hardware.
- Benefits and applications: Software correlators are flexible, scalable, inexpensive, and quick to produce, making them particularly useful for small to medium sized VLBI arrays.These properties also support niche facilities and specialized scientific programs.
2. The FX software correlator architecture
The paper adopts an FX architecture and separates the correlator into telescope-based operations followed by baseline-based operations. It focuses on implementing these main processing steps rather than rederiving interferometry theory.
- Architecture: The correlator architecture is FX, with implementation organized around the main processing steps used by that design.The paper refers readers elsewhere for complete radio-interferometry theory.
- Scope: Operations performed at the telescopes are outside the paper’s scope because it starts from recorded, digitized baseband data streams.The discussion assumes those data have already been produced and recorded.
- Pipeline organization: Telescope-based operations and baseline-based operations are described separately and in sequence.This division structures the correlator pipeline.
2.1. Antenna-based operations
DiFX aligns telescope data using modeled delays, applies fringe rotation and fractional-sample correction, then channelizes the corrected streams. The architecture supports pre-F and post-F fringe rotation, with post-F operation saving computation under restricted conditions.
- Alignment: CALC 95 generates geometric delays, which are interpolated quadratically to achieve delays below 1 × 10^-15 sec from exact CALC values.Station clock offsets and rates are added to the geometric delays.
- Alignment: Integer-sample delays select aligned data, while fractional-sample errors are retained for later frequency-dependent phase correction.Alignment between data streams is within ±1 sample; fractional delays can reach ±0.5 sample.
- Fringe rotation: Floating-point fringe rotation yields no fringe-rotation losses, compared with a 6.25% signal-to-noise loss for three-level digital fringe rotation in a two-level complex correlator.The comparison concerns fringe-rotation quantization losses.
- Fringe rotation: DiFX applies fringe rotation as a complex phase correction, either before Fourier transformation (pre-F) or after channelization (post-F).Pre-F uses interpolated delay for each sample, whereas post-F can apply one correction per FFT window.
- Fringe rotation: Post-F fringe rotation saves considerable computational effort when fringe rates are low and the channel count is small.Under these conditions, a single fringe rotation across an FFT window introduces minimal decorrelation.
- Channelization: After alignment and phase correction, DiFX converts time series into frequency series using an FFT or polyphase digital filterbank.Filter length controls the accuracy of the approximated rectangular channel response.
2.2. Baseline-based operations
After antenna-based correction and channelization, DiFX forms baseline visibilities by cross-multiplying channelized telescope data, accumulates them, and writes the results for downstream analysis.
- Cross multiplication: For each baseline, channelized telescope data are conjugated and cross-multiplied channel by channel to produce complex visibilities.The operation is repeated across common bands, polarizations, and baselines.
- Accumulation: Visibilities are complex-added over repeated processing cycles until the requested accumulation time is reached.DiFX records the number of good cycles per telescope, but does not currently record data weights.
- Visibility formation: Cross multiplication, accumulation, and normalization by antenna autocorrelation spectra produce the complex cross-power spectrum for each baseline.Scaling by the geometric mean of telescope powers measured in Jy converts the cross-power amplitudes to Jy.
- Output: DiFX supports RPFITS output containing complex visibilities, timestamps, and (u,v,w) coordinates for loading into AIPS, CASA, or MIRIAD.Additional formats such as FITS-IDI were planned, while RPFITS weights were not currently recorded.
2.3. Special processing operations: pulsar binning
DiFX uses incoherent dedispersion and flexible pulse-phase binning to process pulsar VLBI data. Signal-weighted bins can improve recovered signal-to-noise while reducing output volume.
- DiFX applies incoherent dedispersion to correct interstellar-medium dispersion and divide visibilities into pulse-phase bins.The bins can be arbitrary in number and phase placement, unlike typical hardware-correlator options.
- Pulsar phase-bin assignment requires ephemeris information supplied through polyco files generated by TEMPO.The polyco files provide a polynomial description of apparent pulse phase as a function of time and require prior pulsar timing.
- DiFX weights user-specified phase bins by pulse strength × bin width and retains an integrated on-pulse visibility.This filtering aims to maximise theoretical signal retrieval while reducing output data volume.
- For a simple on/off gate, only selected phase bins are accepted, whereas signal-strength weighting uses the pulse profile across bins.The described alternatives compare ungated summation, binary gating, and pulse-weighted binning.
- 6% recovered signal-to-noise improvement is obtained for a Gaussian pulse compared with an optimally placed single on/off bin.For a Gaussian main pulse plus a half-amplitude Gaussian interpulse, the improvement increases to 21%.
3. Software correlation on the Swinburne Beowulf cluster - a case study
The Swinburne case study runs DiFX on a roughly 300-processor Beowulf cluster of commodity desktop and server computers linked by gigabit Ethernet.
- ∼300 processors comprise the Swinburne Beowulf cluster used for DiFX software correlation.The cluster mixes commodity off-the-shelf desktop and server PCs connected via a gigabit Ethernet network.
3.2. Structure of the DiFX code
DiFX combines optimized C++ processing with MPI communication and separates data management, correlation, and coordination across extensible node classes. Its input abstraction supports both disk-based correlation and real-time eVLBI.
- DiFX uses C++ with Intel Performance Primitive vector routines and MPI for data transfer.The optimized vector library provides a factor of several performance gain on Intel CPUs compared with nonoptimized vector code.
- FxManager assigns time ranges from Datastream nodes to Core nodes, which return processed results to the manager.Double-buffered, non-blocking communication is used to reduce latency and maximize throughput.
- Datastream nodes can read baseband data from local disks, network disks, or network sockets without changing the rest of the system.This input abstraction allows the same correlator to handle different data sources.
- The same architecture supports production disk-based VLBI correlation and real-time eVLBI testing.Real-time transmission from Parkes, ATCA, and Mopra to computing resources at Swinburne and the University of Western Australia was tested.
3.3. Operating DiFX
DiFX is operated through a GUI that organizes configuration and helper applications before launching correlation jobs. Planned GUI development includes real-time amplitude, phase, and lag feedback.
- DiFX’s GUI edits configuration files, launches helper programs, runs the delay model generator, and reports job progress.An experiment configuration file and a second required input file provide the correlator’s operational setup.
- Helper applications extract experiment information, generate delays and (u,v,w) coordinates, and prepare correlation inputs from VEX data.The GUI organizes these tasks and presents information for configuration adjustments before a job starts.
- Real-time amplitude, phase, and lag feedback in the GUI is planned for future development.The planned display would resemble continuously available visibility spectra at connected-element interferometers.
3.4. Performance
DiFX performance depends on balancing CPU, network, buffering, and manager-node capacity. Most configurations support real-time processing, but high antenna counts and extreme resolutions can exceed communication capacity.
- Scaling: CPU-limited correlation scales linearly when compute nodes remain supplied with raw data.Data transfer and network capacity eventually create a data-limited regime.
- Networking: Double-buffered MPI communication keeps processing nodes active, although transfer overhead slightly reduces maximum throughput.Throughput remains slightly below the maximum capacity of the most heavily loaded network path.
- Data flows: The FxManager receives data at a rate set by recording rate, compression ratio, and the number of Cores.Core outputs are reduced relative to their inputs, while more Cores increase aggregate manager-node traffic.
- Buffering: Larger processing messages reduce FxManager traffic, but messages exceeding available RAM degrade performance.Message size therefore trades network efficiency against memory capacity.
- Limits: Network latency and insufficient FxManager capacity can prevent real-time processing for large arrays at very high spectral and time resolution.Time-range parallelization can mitigate manager communication limits for disk-based experiments.
- Measurements: 64 MHz aggregate bandwidth across eight 8 MHz bands provides the basis for Swinburne performance tests across array sizes and spectral resolutions.The tests used 2-bit sampling and 256 Mbps per antenna.
3.5. Correlator comparison results
DiFX was compared with the ATNF S2 and VLBA correlators using matched observational data and analysis procedures. Visibility amplitudes and phases agreed closely, after accounting for delay-model and fractional-sample effects.
- ATNF S2 comparison: The S2 comparison used simultaneous recordings from two 16 MHz RCP bands during a 20-minute observation of PKS 0208−512.Data were recorded on both S2 tapes and LBADR disks, then processed by the respective correlators.
- Comparison procedure: Both correlators used identical Tsys values, clock models, 2-second integrations, and 32 channels per 16 MHz band.Delay-model implementations differed, so differential-delay phase effects were corrected in the comparison.
- ATNF S2 comparison: DiFX and the S2 correlator showed no significant differences in fitted visibility amplitudes over time.The RMS variation for each baseline also agreed very well between the correlators.
- ATNF S2 comparison: S2’s coarse fractional-sample correction can produce up to ∼10% decorrelation at band edges on long baselines.The comparison corrected S2 amplitudes for this band-edge effect.
- VLBA comparison: The VLBA comparison used four dual-polarisation 8 MHz bands recorded with one-bit Nyquist-rate sampling on the Mk5 system.Each band was correlated with 64 spectral points and 2.048-second integrations.
- VLBA comparison: DiFX and the VLBA correlator agreed within the RMS visibility errors for amplitude and phase comparisons over time and frequency.The comparison used matched visibility products after accounting for differing delay models.
4. Scientific applications of the Swinburne software correlator
DiFX’s configurable frequency and time resolution supports spectral-line, wide-field, pulsar, and geodetic VLBI applications. Its principal advantage is enabling resolutions and observing modes unavailable on existing hardware correlators, with compute time as the main penalty.
- High-frequency spectral-line VLBI: DiFX provides very high spectral resolution for broad-band spectral-line studies such as maser mapping and kinematics.The software can increase channel counts beyond the demonstrated observation, limited primarily by compute time.
- High-frequency spectral-line VLBI: 16,384 channels across a 4 MHz band yielded 0.25 kHz channels and 0.038 km/s velocity resolution for the OH maser G345−0.2.The observation used dual-polarised hard-disk recording with the LBA.
- Wide-field imaging: Wide-field imaging requires high spectral and temporal resolution to limit time and bandwidth smearing.At 40 mas VLBI resolution, the cited ATCA example requires 50 ms time resolution and 4 kHz frequency resolution.
- Wide-field imaging: Neither JIVE nor VLBA hardware correlators can provide the cited wide-field continuum resolution, whereas DiFX can be configured for it.The configuration is performed in the same manner as a normal continuum experiment.
- Pulsar studies: Pulsar scintillation studies benefit from extremely high frequency resolution because visibility variations depend on time and frequency.DiFX correlated a 32 MHz band for B0834−04 with 131,072 frequency channels.
- Pulsar studies: No existing hardware correlator could provide the demonstrated frequency resolution over such a wide bandwidth for the pulsar observation.The resulting dynamic spectrum revealed scintillation structure across time and frequency.
- Geodetic VLBI: DiFX also accommodates geodetic VLBI requirements such as different output formats and frequent sub-arraying.It was tested on Mk5 geodetic datasets and adapted for a new Australian geodetic array.
5. Conclusions
The paper concludes that software correlation offers rapid development, flexibility, scalability, and portability for small to medium VLBI arrays. DiFX was validated against established correlators and demonstrated real-time operation on a moderate Beowulf cluster.
- Benefits: Software correlation develops rapidly because it requires algorithmic knowledge rather than intimate knowledge of digital signal-processing hardware.This lowers the barrier to developing new correlator capabilities.
- Benefits: Software is flexible and scalable across interferometric modes, including experiments unsupported by existing ASIC-based hardware.The trade-off is increased compute time or additional processing elements for improved performance.
- Benefits: Software correlators can incorporate data recorded with mixed disk-based hardware.This supports heterogeneous recording environments without requiring a dedicated hardware correlator.
- Computing environment: The highly parallel correlation algorithm suits multi-processor environments, while falling commodity-computing costs increase capability and reduce expense.Medium to large computing facilities are widely available at universities and government research institutions.
- Benefits: Code portability allows the correlator to run across a wide range of platforms with minimal recompilation effort.This extends reuse beyond the original Swinburne implementation.
- Performance and validation: Moderate to large Beowulf clusters can correlate relatively large telescope arrays and high data rates in real time.If real-time operation is unnecessary, the required number of processing elements can be reduced substantially.
- Performance and validation: Amplitude and phase comparisons with the ATNF S2 and VLBA correlators verified that DiFX operates correctly for astronomical VLBI.The paper also reports support for Australian and some global observations up to 1 Gbps per telescope, plus geodetic applications.