PaperlayerView exact revision on arXiv

Source-linked AI summary

DiFX2: A more flexible, efficient, robust and powerful software correlator

A. T. Deller, W. F. Brisken, C. J. Phillips, J. Morgan, W. Alef, R. Cappallo, E. Middelberg, J. Romney, H. Rottmann, S. J. Tingay, R. Wayth

arXiv:1101.0885v1astro-ph.IM

TL;DR

Long-baseline arrays need flexible, affordable correlator upgrades because costly recording systems constrain bandwidth and hardware correlators are less adaptable. This paper presents DiFX2’s expanded capabilities, performance improvements, and validation, reporting broad speed gains while identifying operational limits for very large phase-center outputs.

  • Problem

    Long-baseline arrays require flexible correlators compatible with wider-band recording systems despite costly recording hardware and media.

  • Method

    The paper describes DiFX2’s new correlator capabilities, performance changes, supporting infrastructure, and correctness validation against an operating DiFX1.x installation.

  • Results

    DiFX2 adds multiple phase centers, phase-calibration extraction, overlapping sub-band correlation, and transient-search data products, with performance gains exceeding a factor of 100 in some parameter regimes.

  • Takeaways & Limitations

    DiFX2 reinforces the flexibility and upgradeability advantages of software correlators over custom-designed hardware correlators.

  • Takeaways & Limitations

    With 500 phase centers, output reaches 35 MB per second of recorded data, making disk-writing overhead substantial.

Abstract

from arXiv · show

Software correlation, where a correlation algorithm written in a high-level language such as C++ is run on commodity computer hardware, has become increasingly attractive for small to medium sized and/or bandwidth constrained radio interferometers. In particular, many long baseline arrays (which typically have fewer than 20 elements and are restricted in observing bandwidth by costly recording hardware and media) have utilized software correlators for rapid, cost-effective correlator upgrades to allow compatibility with new, wider bandwidth recording systems and improve correlator flexibility. The DiFX correlator, made publicly available in 2007, has been a popular choice in such upgrades and is now used for production correlation by a number of observatories and research groups worldwide. Here we describe the evolution in the capabilities of the DiFX correlator over the past three years, including a number of new capabilities, substantial performance improvements, and a large amount of supporting infrastructure to ease use of the code. New capabilities include the ability to correlate a large number of phase centers in a single correlation pass, the extraction of phase calibration tones, correlation of disparate but overlapping sub-bands, the production of rapidly sampled filterbank and kurtosis data at minimal cost, and many more. The latest version of the code is at least 15% faster than the original, and in certain situations many times this value. Finally, we also present detailed test results validating the correctness of the new code.

1. Introduction

DiFX evolved from an Australian Long Baseline Array upgrade into a widely adopted, actively developed software correlator. DiFX2 combines new capabilities and performance improvements with documentation and community infrastructure supporting installation, testing, and use.

  • DiFX began development in 2005 for the Australian Long Baseline Array and entered production usage in 2006.
  • DiFX2 reflects continued development since the 2007 public release, with new features and performance improvements sufficient to merit a major version increment.
  • DiFX uses vector arithmetic libraries for acceleration and Message Passing Interface implementations to distribute processing across multiple nodes.
  • The correlator has been adopted by the LBA, VLBA, and MPIfR, including complete VLBA migration from its retired hardware correlator.
  • A collaboratively managed wiki and two mailing lists provide installation, testing, user, and developer support for DiFX.

2. New features

DiFX2 expanded interoperability by supporting standard and observatory-specific data formats and by improving access to recorded VLBI data. These changes reduce format-related barriers for general radio-astronomy and geodetic processing.

  • FITS-IDI and Mark4 format output: RPFITS was nonstandard and had limited support in radio-astronomy post-processing packages, motivating FITS-IDI output in DiFX.
  • FITS-IDI and Mark4 format output: DiFX translates binary DiFX output into FITS-IDI files after correlation, avoiding large FITS-library dependencies and simplifying output writing.
  • FITS-IDI and Mark4 format output: DiFX2 adds translation from binary DiFX output to Mark4 visibility datasets and direct DiFX-format import into the HOPS geodetic post-processing package.
  • Mark5 data access: Since version 1.5.0, DiFX can read Mark5 data directly, removing the prior need to export files to a Linux filesystem before correlation.

2.3. Phase calibration tone extraction

DiFX2 adds flexible phase-calibration-tone extraction for estimating instrumental delays, alongside spectral selection and averaging features that reduce processing and data-transfer costs. Spectral selection also enables correlation of heterogeneous overlapping bands.

  • Phase calibration tone extraction: DiFX2 adds configurable extraction of any number of phase calibration tones, supporting arrays that rely on the correlator for this function.
  • Phase calibration tone extraction: Detailed verification of DiFX2 phase-calibration extraction and Mark4 visibility production was deferred to a future publication.
  • Spectral selection and averaging: DiFX2 spectral selection, or zoom mode, discards unneeded channels after channelization to reduce cross-multiplication load and data returned to the manager node.
  • Spectral selection and averaging: Band-matching subdivides wider recorded bands to match narrower bands, supporting heterogeneous systems and upper-sideband data against lower-sideband data covering the same spectral range.
  • Spectral selection and averaging: Spectral averaging combines correlated spectral points before return to the manager node, improving efficiency for coarse resolution and multiple phase-center correlations.

2.5. Multiple simultaneous phase centers

VLBI’s standard time and frequency resolutions restrict its field of view, while higher resolution creates prohibitive data volumes. DiFX2 addresses this by shifting and averaging multiple known source positions during correlation, producing manageable pencil-beam datasets.

  • Multiple simultaneous phase centers: Standard VLBI resolutions produce several-arcsecond pencil beams, while mapping a tenth of the VLBA primary beam can require visibility datasets exceeding 2 TB.
  • Multiple simultaneous phase centers: Post-correlation uv shifting targets known sources but requires an initially large visibility dataset, with substantial intermediate disk I/O.
  • Multiple simultaneous phase centers: DiFX2 performs uv shifting and averaging inside the correlator, avoiding transmission and disk storage of high-spectral-resolution intermediate visibilities.
  • Multiple simultaneous phase centers: For P phase centers, DiFX returns and writes P normal-sized post-average visibility collections, with relatively small performance impact from infrequent shifting and averaging.
  • Multiple simultaneous phase centers: Online decorrelation corrections are applied to amplitudes and weights before disk output, enabling standard reduction and imaging of the resulting pencil beams.
  • Multiple simultaneous phase centers: With low-resolution finder catalogues, DiFX2 enables VLBI-resolution surveys with minimal overhead and has supported pilot observations in the Chandra Deep Field South.

2.6. Correct model accountability

DiFX2 stores an accurate representation of the applied delay and clock models, enabling correct model accountability during post-processing.

  • FITS-IDI output stores the applied delay model in binary “IM” and “MC” tables.These tables contain the sampled model polynomials used by DiFX2 and the applied clock model.

2.7. New data monitoring tools

DiFX2 adds low-cost monitoring streams for transient searches and real-time visibility assessment. These tools provide filterbank data, spectral-kurtosis diagnostics, and processed dynamic spectra during correlation.

  • 2.7.1. Autocorrelation filterbank “spigot”: The autocorrelation spigot provides time- and frequency-resolved antenna autocorrelations as independent, time-aligned filterbank streams.Its computational and multicast overhead is negligible except for very short integrations.
  • 2.7.1. Autocorrelation filterbank “spigot”: Feedback from analysis programs can trigger extraction of short baseband-data ranges after correlation for full-time-resolution analysis.This supports commensal transient searches and is used in a VLBA pipeline supporting the CRAFT fast-transients project.
  • 2.7.2. Spectral kurtosis: Spectral kurtosis estimates each filterbank channel’s probability-density form to identify spectrally confined radio-frequency interference cheaply.
  • 2.7.3. Real–time visibility monitoring: The TCP visibility monitor server sends selected visibilities to clients for real-time processing, display, and data-quality assessment during correlation.It is particularly useful for verifying array and correlator setup during non-disk-based eVLBI observations.
  • 2.7.3. Real–time visibility monitoring: A dynamic-spectrum example spans 1 second horizontally and 64 MHz vertically, with 2 ms by 500 kHz pixels.Processed data interpolate missing samples and remove bandpass shapes and the 80 Hz noise-calibration signal.

2.8. Other new functionality

DiFX2 broadens compatibility and calibration flexibility through local-oscillator correction, network and complex-sampling support, and higher-order clock models.

  • DiFX2 corrects station-specific local-oscillator offsets after channelization, typically up to tens of kHz, with little decorrelation.The correction can assist mismatched sub-band correlation, but phase-calibration tones cannot currently be extracted from corrected sub-bands.
  • DiFX2 extends network baseband input beyond the TCP-only capability of DiFX1.x.
  • DiFX2 supports complex-sampled VDIF baseband data, reducing preprocessing requirements and potentially lowering quantization losses at a given bit precision.
  • DiFX2 allows arbitrary-order station clock polynomials, enabling more accurate correction of known clock variations, especially at very high frequencies.

3. Performance improvements

DiFX2 improves correlator performance through faster phase-rotation calculations and cache-aware processing, while supporting efficient multi-phase-center correlation. Tests show substantial speed gains, with output I/O becoming a limiting factor at very large phase-center counts.

  • Phase rotation: DiFX2 reduces trigonometric work by replacing most per-element sin/cos evaluations with complex multiplications when phase increments are constant.The approach reduces trigonometric-operation cost to near-negligible levels for typical 100–1000-element vectors.
  • Phase rotation: 15–20% higher throughput is achieved for correlations with a small number of stations through more efficient trigonometric processing.The same improvement can alternatively reduce the computational resources needed for a prescribed throughput by 15–20%.
  • Cache and FFT buffering: Cache-aware traversal and FFT buffering reduce the throughput loss caused by visibility buffers exceeding CPU-cache capacity.DiFX1.x suffers a marked performance drop beyond 512 spectral points, whereas DiFX2’s reduction is much smaller despite increased FFT cost.
  • Multiple phase centers: High spectral resolution for uv shifting initially increases computational load by 2–3x over a standard 16-spectral-point continuum observation.The tested 2 kHz spectral and 26 ms temporal resolutions limit decorrelation to under 5% at the edge of the VLBA primary beam at 1.4 GHz.

4. Operational infrastructure

DiFX2 adds operational infrastructure that simplifies job setup, model generation, monitoring, logging, and integration with observatory workflows. The system also supports configurable control-file generation, client/server delay-model services, and multicast status messaging.

  • Operational infrastructure: DiFX2 simplifies routine operations through easier correlator-job configuration, model generation, monitoring, and logging.Some infrastructure is VLBA-specific but can be customized for other installations.
  • Job configuration: vex2difx can generate the complete set of correlator control files from a vex observation-description file.Users can override defaults such as integration time and spectral resolution.
  • Model generation: The calcserver client/server architecture separates DiFX model-file writing from CALC delay calculations and supports integration with existing observatory setups.CALC calculations are communicated to the standalone model-writing program upon request.
  • Monitoring and control: The difxmessage package broadcasts correlation progress, resource utilization, and equipment or configuration errors through multicast XML messages.Messages are graded from debug to fatal and support monitoring and operational control.

5. Validation testing

DiFX2 was validated primarily against DiFX1.5.4 through extensive visibility comparisons and functional testing. The results showed excellent agreement and correct multiple-phase-center compensation.

  • Validation testing: The validation program compared many recording modes and included functional tests of final astrometric or geodetic observables.Earlier validation emphasized point-by-point visibility comparisons; NRAO testing expanded the modes and test types.
  • Validation testing: DiFX2.0.0 and DiFX1.5.4 showed rms phase and amplitude deviations of 0.0007° and 0.0007%, respectively, on a representative VLBA test.The comparison used 40 seconds of 4C39.25 data, one 16 MHz band, one-second integrations, and 32 averaged spectral points.
  • Validation testing: DiFX2 uses a fifth-order delay interpolator at two-minute timescales, improving on DiFX1.x’s second-order interpolator at one-second timescales.The observed nonzero phase-error mean was attributed to DiFX1.x delay errors at approximately the 0.1 femtosecond level.
  • Validation testing: The multiple-phase-center test shifted visibilities from an offset correlation center to the true source position while retaining 0.5 MHz spectral resolution.The test used a 1024-point FFT and applied uv shifts at intervals no longer than 40 ms.
  • Validation testing: Across six baselines, mean amplitude and phase errors were 0.09% and 0.014°, while estimated decorrelation agreed with predictions to within several percent.The average decorrelation agreement across all baselines was 0.4%, and amplitude corrections produced perfect amplitude agreement.

6. Future work

Future DiFX2 development targets phased-array outputs and additional performance, spectral, and monitoring capabilities. Planned functionality is intended to support high-time-resolution studies and larger or more varied observing configurations.

  • 6. Future work: DiFX2 is planned to form phased-array outputs, including tied-array beams for high-time-resolution studies such as pulsar analysis.Selectable digital filterbank and reconstructed time-series outputs are also planned for phased-array processing.
  • 6. Future work: Frequency-division multiplexing, non-power-of-two spectral support, and expanded graphical monitoring are under investigation.The development roadmap is expected to continue adapting to community needs and future applications.

7. Conclusions

DiFX2 combines improved robustness, performance, and functionality, with adoption for production correlation at major VLBI facilities. Its features supported bandwidth expansion and broadened processing applications, while development continued.

  • 7. Conclusions: DiFX2 added phase-calibration extraction, simultaneous multiple-phase-center correlation, and high-time-resolution filterbank and kurtosis outputs.These capabilities support transient searches, RFI mitigation, and other expanded VLBI processing uses.
  • 7. Conclusions: In some parameter spaces, including deep VLBI surveys, DiFX2 processing-speed improvements exceeded a factor of 100.The conclusion also reports that DiFX2 was adopted for production by three major VLBI correlator facilities.
  • 7. Conclusions: DiFX2 development included assistance with phase-calibration tone extraction, while detailed geodetic performance verification was deferred to future work.The deferred analysis includes verification of phase-calibration extraction and production of Mark4-format visibility data.
Loading 1101.0885v1…