Source-linked AI summary
Real-time processing of the imaging data from the network of Las Cumbres Observatory Telescopes using BANZAI
Curtis McCully, Nikolaus H. Volgenau, Daniel-Rolf Harbeck, Tim A. Lister, Eric S. Saunders, Monica L. Turner, Robert J. Siverd, Mark Bowman
TL;DR
Time-domain astronomy requires automated, rapid data reduction because manual processing does not scale to high-volume, variable observations. This paper presents BANZAI, an integrated pipeline for real-time LCOGT image processing and monitoring, delivering processed images to users in under 10 minutes while supporting extensibility and quality control.
Problem
Manual inspection and reduction cannot scale to surveys producing hundreds of thousands to millions of alerts nightly, while variable sources require fast-turnaround processing without human intervention.
Method
BANZAI processes LCOGT images through chained stages that remove instrumental signatures, perform astrometric fitting and source extraction, and integrate with observatory queues, APIs, and monitoring systems.
Results
BANZAI achieves a median image-processing time of 35 seconds and makes processed images available to users less than 10 minutes after shutter closure.
Takeaways & Limitations
Real-time reduction combined with operational metrics enables LCOGT to deliver data quickly while retaining insight into telescope-network health.
Takeaways & Limitations
Automatic detection and recovery of failures remain under active development, and FFT-based pattern-noise alert thresholds require further tuning to avoid false positives and alert fatigue.
Abstract
from arXiv · showhide
Work in time-domain astronomy necessitates robust, automated data processing pipelines that operate in real time. We present the BANZAI pipeline which processes the thousands of science images produced across the Las Cumbres Observatory Global Telescope (LCOGT) network of robotic telescopes each night. BANZAI is designed to perform near real-time preview and end-of-night final processing for four types of optical CCD imagers on the three LCOGT telescope classes. It performs instrumental signature removal (bad pixel masking, bias and dark removal, flat-field correction), astrometric fitting and source catalog extraction. We discuss the design considerations for BANZAI, including testing, performance, and extensibility. BANZAI is integrated into the observatory infrastructure and fulfills two critical functions: (1) real-time data processing that delivers data to users quickly and (2) derive metrics from those data products to monitor the health of the telescope network. In the era of time-domain astronomy, to get from these observations to scientific results, we must be able to automatically reduce data with minimal human interaction, but still have insight into the data stream for quality control.
1. INTRODUCTION
Time-domain astronomy requires automated, fast-turnaround reduction as data volumes and source variability outpace manual inspection. BANZAI addresses this need by reducing LCOGT’s heterogeneous network data in real time.
- 1. INTRODUCTION: Manual inspection and reduction do not scale to surveys producing hundreds of thousands to millions of alerts per night.Upcoming surveys such as ZTF and LSST motivate automated handling of observational data.
- 1. INTRODUCTION: Rapidly varying astrophysical sources require fast-turnaround data reduction without human intervention.The relevant timescales range from seconds for fast radio bursts to days for supernovae.
- 1. INTRODUCTION: LCOGT operates 21 small-aperture telescopes worldwide, producing nearly 50,000 images per month and about 1.5 TB of uncompressed raw data.Its network operates in unison through dynamically scheduled requests, requiring homogeneous data from heterogeneous instruments.
- 1. INTRODUCTION: BANZAI reduces all LCOGT images in real time and is integrated into the observatory’s dataflow.The paper presents its design choices, integration, and lessons from operating a high-volume processing service.
- 1. INTRODUCTION: After exposure, files are queued for shipment to headquarters, processed by BANZAI, and ingested into a cloud archive for user download.This flow connects observation completion to processed data availability.
2. DATA FLOW FROM TELESCOPE TO USER
LCOGT’s queue-based observatory transfers raw images to headquarters, where BANZAI produces preview reductions in near real time and final science-quality reductions overnight. The pipeline calibrates images, extracts sources, solves astrometry, and makes products available to users through the archive.
- 2. DATA FLOW FROM TELESCOPE TO USER: LCOGT provides raw, preview, and processed data products, with preview files generated on the fly and processed files created at night’s end.Preview and processed files use the same procedure but may use different calibration files.
- 2. DATA FLOW FROM TELESCOPE TO USER: Queue observing enables rapid follow-up, but many queue-scheduled observatories still require manual data reduction.LCOGT’s workflow is designed to address this remaining processing burden.
- 2. DATA FLOW FROM TELESCOPE TO USER: Raw FITS files are transferred from telescopes to headquarters through queues and placed on the FITS Exchange for downstream processing.The Shipper moves files to headquarters, while the science archive stores raw frames in the cloud.
- 2. DATA FLOW FROM TELESCOPE TO USER: BANZAI calibrates every image by masking bad pixels, correcting overscan, bias, dark, and flat signatures, extracting photometry, and solving astrometry.Source extraction uses SEP and astrometry uses Astrometry.net.
- 2. DATA FLOW FROM TELESCOPE TO USER: 10 minutes is the median time from shutter closure to preview data becoming available for user download.Longer delays are often caused by network issues.
- 2. DATA FLOW FROM TELESCOPE TO USER: 2 minutes is BANZAI’s median image-processing time, with longer times in dense fields because photometry extraction and astrometric solving take longer.Microlensing fields near the Galactic center are given as an example of dense fields.
- 2. DATA FLOW FROM TELESCOPE TO USER: End-of-night reprocessing uses master calibration images from that night to produce science-quality reductions.The preview mode supports rapid transient response, while final processing uses better calibration data.
3. BANZAI ARCHITECTURE
BANZAI combines extensible, tested pipeline architecture with calibration utilities, containerized deployment, and operational reporting for rapid processing across LCOGT instruments.
- Code availability and implementation: BANZAI uses Python with performance-critical sections in Cython and C, supporting broad contribution while retaining high-performance components.
- Code availability and implementation: Its open-source GitHub distribution promotes transparency, maintainability, reproducibility, and project-specific customization.
- Object model: Individual processing stages are chained through a Template Method pattern, while keeping images in memory to reduce file size and improve performance.
- Performance: 35 s is the median BANZAI image-reduction time, with dense microlensing fields producing longer processing tails during photometry extraction and astrometric solving.
- Master calibrations: Calibration template classes centralize file retrieval and structure, leaving specialized stages to implement operations such as scaled dark subtraction.
- Master calibrations: Calibration safeguards compare new masters with previous good frames and use pixel-level twilight comparisons with median absolute deviation to reject outliers.
- Deployment and testing: Docker supports rapid deployment and parallel preview and end-of-night pipeline instances, while Rancher manages containers, resources, upgrades, and debugging.
- Deployment and testing: Continuous integration combines public unit-test visibility with internal deployment, integration, and end-to-end testing using large datasets.
4. FAILURE DETECTION AND PERFORMANCE METRICS
BANZAI detects failures and derives metrics automatically because the LCOGT data stream is too large for manual inspection. Monitoring combines stored quality results, reports, alerts, and long-term dashboards, while alert tuning remains under development.
- Automated failure detection: Automated checks reject failed frames and store test results and performance metrics in ElasticSearch for flexible querying and monitoring.
- Operational monitoring: Summary reports visualize calibration age and known failure modes across the network, while real-time alerts reach operators by email or Slack.
- Performance metrics: Grafana dashboards visualize long-term trends in image quality, telescope pointing, FWHM, and astrometric-solution failures.
- Camera failure modes: Frames with a significant fraction of pixels exactly equal to 1000 are automatically rejected as electric-noise failures, triggering a Slack restart alert.
- Camera failure modes: A 2D FFT detects electrical-interference pattern artifacts, but threshold tuning remains unfinished because excessive alerts created counterproductive alert fatigue.
- Development status: Failure detection and recovery remain an area of active development.
5. CONCLUSION
BANZAI provides high-throughput, rapidly available data reduction for LCOGT while supporting robust deployment, extensibility, and automated network monitoring. These capabilities address the need for minimal human intervention with continued insight into the data stream.
- 35 seconds is BANZAI’s median image-processing time, while processed images reach the science archive in less than 10 minutes.These timings support rapid delivery during high-volume operations.
- BANZAI uses containerized deployment, queue/API integration, and ElasticSearch metrics to support continuous operations and monitor network health.Rancher-managed Docker deployment facilitates upgrades and redeployment, while stored metrics generate summary plots and health indicators.
- Automated reduction with minimal human interaction, while retaining datastream insight, is presented as necessary for converting future follow-up observations into scientific results.The authors frame LCOGT’s operational lessons as relevant to surveys such as ZTF and LSST.