Source-linked AI summary
The Zwicky Transient Facility Alert Distribution System
Maria T. Patterson, Eric C. Bellm, Ben Rusholme, Frank J. Masci, Mario Juric, K. Simon Krughoff, V. Zach Golkhou, Matthew J. Graham, Shrinivas R. Kulkarni, George Helou
TL;DR
ZTF-scale alert streams make rapid prioritization and filtering a central challenge. The paper describes ZADS, a Kafka- and Avro-based distribution and processing system. In production, it distributed about one million alerts nightly, exceeding the volume ever distributed via VOEvents, and the authors argue its performance is evidence that the LSST alert problem is tractable.
Problem
High-volume survey streams make quickly prioritizing alerts of interest a bottleneck, requiring robust real-time filtering and scalable distribution for timely follow-up.
Source: §1 · p. 2Method
ZADS uses Avro-formatted alerts, Kafka distribution, and Python filters to stream and process ZTF detections.
Source: §3.1 · p. 3, §7 · p. 8Results
ZADS successfully distributed about one million alerts nightly, more than have ever been distributed via VOEvents, and the authors report ZTF alert volume approaches 10% of LSST.
Source: §7 · p. 8Takeaways & Limitations
The authors argue that ZADS’s successful at-scale distribution with more modest hardware is strong evidence that the LSST alert problem is tractable.
Source: §7 · p. 8Minor Planet Center reporting is handled separately from ZADS because science-specific interfaces require additional filtering and post-processing.
Source: §4 · p. 5
Abstract
from arXiv · showhide
The Zwicky Transient Facility (ZTF) survey generates real-time alerts for optical transients, variables, and moving objects discovered in its wide-field survey. We describe the ZTF alert stream distribution and processing (filtering) system. The system uses existing open-source technologies developed in industry: Kafka, a real-time streaming platform, and Avro, a binary serialization format. The technologies used in this system provide a number of advantages for the ZTF use case, including (1) built-in replication, scalability, and stream rewind for the distribution mechanism; (2) structured messages with strictly enforced schemas and dynamic typing for fast parsing; and (3) a Python-based stream processing interface that is similar to batch for a familiar and user-friendly plug-in filter system, all in a modular, primarily containerized system. The production deployment has successfully supported streaming up to 1.2 million alerts or roughly 70 GB of data per night, with each alert available to a consumer within about 10 s of alert candidate production. Data transfer rates of about 80,000 alerts/minute have been observed. In this paper, we discuss this alert distribution and processing system, the design motivations for the technology choices for the framework, performance in production, and how this system may be generally suitable for other alert stream use cases, including the upcoming Large Synoptic Survey Telescope.
1. INTRODUCTION
ZTF’s high-volume alert stream moves discovery’s bottleneck from distributing detections to quickly prioritizing them. The paper introduces ZADS technologies intended to support real-time filtering and scalable distribution.
1. INTRODUCTION
- Earlier electronic alert services eased dissemination, but their manageable volumes still allowed manual and visual processing.The paper cites the 4 Pi Sky database’s roughly 2.2 million VOEvents accumulated since April 2014 as a reference point.Source: §1 · p. 2
- ZTF and LSST-scale surveys stream detections of transients, variables, and moving objects, shifting the challenge toward prioritizing alerts amid the flood.The paper notes that these surveys include all sources above a specified detection threshold, rather than only likely new explosive extragalactic transients.Source: §1 · p. 2
- The paper presents ZADS technologies, their design motivations and performance, and their potential to scale to larger astronomical data streams.Rapid automated filtering and integration with distribution mechanisms are described as critical for fast follow-up of time-sensitive activity.Source: §1 · p. 2
2. ZADS OVERVIEW AND GOALS
ZADS delivers near-real-time ZTF detections to downstream brokers and science users, with a goal of making science-quality alerts available within 20 minutes of observation. It is designed to handle high-volume alerts and support time-domain science.
2. ZADS OVERVIEW AND GOALS
- ZADS obtains alerts from ZTF’s image-differencing detection and alert-generation process and delivers them to brokers and science users.Its stated goal is to make science-quality alerts available within 20 minutes of observation.Source: §2 · p. 2
- ZADS must move more than 1 million nightly alerts, each about 60 KB, to the UW archive, a public cloud system, and community brokers without data loss.The system feeds four main broker systems; planned TNS reporting is distinguished from VOEvent translation or automated GCN and ATEL distribution, which had not been attempted.Source: §2 · p. 2
- ZADS supports time-domain science including young-supernova discovery, near-Earth asteroid detection, stellar-variable identification, and searches for gravitational-wave counterparts.These are the science uses named in the paper.Source: §2 · p. 3
3. TECHNOLOGIES FOR ZADS
ZADS combines Avro-formatted alerts, Kafka distribution, and Python filters to support fast parsing, scalable delivery, and stream processing. The paper also notes technology-specific limitations and that the alert data model is tailored to ZTF.
3.1. Alert formatting
- Avro provides smaller alert messages and faster serialization and deserialization than XML, while strictly typed schemas support compact binary fields and differing writer and reader schemas.The paper reports ZTF Avro alerts at about half the size of XML with embedded schema, one-third the size without schema, and two-thirds the size of compressed XML; Maeda (2012) reports an approximately 40-fold speedup.Source: §3.1 · p. 3
- ZTF nests schemas for object identity, candidate detections, image cutouts, and extensible detection history, but its Avro model was designed specifically for ZTF.The paper contrasts ZTF’s lightcurve-oriented history and embedded postage-stamp cutouts with generalized VOEvent, and suggests merging the structures could inform future IVOA standards.Source: §3.1 · p. 4
3.2. Alert message distribution
- Kafka topics divide messages into ordered partitions with offsets, allowing consumers to scale and rewind to earlier alerts for reprocessing.Kafka tracks each consumer’s latest offset rather than maintaining a separate stream copy per consumer; the paper says this rewind feature is unavailable in the current VOEvent Transport Protocol system.Source: §3.2 · p. 4
- Kafka’s production deployment had practical hurdles: limited community expertise, terse documentation, difficult access control setup, and no open-source accounting support.Authentication, authorization, and access control are available but difficult to set up; accounting would support monitoring data transfer to clients.Source: §3.2 · p. 4
3.3. Filtering
- Python filters deserialize Avro alerts into dictionaries, apply if-then logic, then serialize passing alerts back to Kafka for downstream consumers.The same filtering code can be developed with saved batch alerts and deployed on the live stream with few changes because the data structure is the same.Source: §3.3 · p. 5
4. ZADS DEPLOYMENT IN ZTF NIGHTLY PROCESSING
ZADS prepares alerts at IPAC, then distributes them through university and cloud Kafka instances to archives, brokers, and community filters. Moving-object reporting to the Minor Planet Center remains separate from ZADS.
4. ZADS DEPLOYMENT IN ZTF NIGHTLY PROCESSING
- IPAC packages image-differencing detections as Avro alerts with measurements, image cutouts, catalog matches, and prior detections, then separates internal and public streams by date and program.Kafka buffers parallel alert production before reducing it to streams for distribution outside IPAC.Source: §4 · p. 5
- Moving-object alerts are included in ZADS, but Minor Planet Center reporting is separate because science-specific interfaces require additional filtering and post-processing.Candidate moving objects are submitted to the MPC after a Moving Object Processing System process runs.Source: §4 · p. 5
- The UW Kafka cluster mirrors IPAC streams, archives nightly data for backup and limited past-alert access, and distributes alerts to internal partners.The cloud Kafka instance makes the public stream available to community brokers and related systems; both UW and cloud brokers retain a seven-day hot cache, and 16 partitions support parallel consumers.Source: §4 · p. 5
- Cloud-hosted community filters provide a first pass at detecting and classifying targets such as supernovae or variable stars, routing passing alerts to separate topics.The filtered streams are then available for science-user analysis.Source: §4 · p. 5
5. PERFORMANCE AND METRICS
ZADS sustained alert volumes above a full observing night’s production range. Alerts became available roughly 20 minutes after observation, with data reduction dominating end-to-end time.
5. PERFORMANCE AND METRICS
- The alert packet uses nested schemas, including previous candidates from a 30-day history search; that field is null when no prior candidates or upper limits exist.Source: §5 · p. 6
- Over 2 million alerts were handled without technical issues, compared with 600,000–1.2 million alerts on a full observing night.Source: §5 · p. 6
- About 20 minutes elapsed from field observation to alert availability, with data reduction dominating end-to-end time.Source: §5 · p. 6
- About 6 seconds was the median time from alert-candidate production to packaging in IPAC Kafka; simulated transfer of 1,000 alerts through Avro and Kafka took 4.2 seconds.Source: §5 · p. 6
- Cloud transfer rates reached about 100 MBps, equivalent to over 80,000 alerts per minute or 6,600 alerts in 5 seconds.Source: §5 · p. 6
6. INTERFACE
Scientific users can receive ZTF alerts through community brokers or, for approved users, directly from filtered public Kafka topics. Direct access is restricted, and additional Kafka security controls were planned.
6. INTERFACE
- Community brokers consume the public stream, add services such as external-catalog cross-matching, and provide enriched alerts to scientific users through broker interfaces.Source: §6 · p. 6
- Approved scientific users may connect directly to pre-defined community-filtered public Kafka topics, but example code does not grant unknown users access.Source: §6 · p. 6
- Direct ZADS access is limited to broker projects with signed MOU agreements and is currently restricted by IP address; additional Kafka access controls were planned.Source: §6 · p. 8
7. SUMMARY
ZADS streams Avro-formatted ZTF alerts through Kafka to archives, cloud services, brokers, and scientific users, while community filters create source-separated streams. The system handled roughly one million alerts nightly, a volume the authors view as evidence that LSST-scale alert distribution is tractable.
7. SUMMARY
- Avro-formatted alerts travel from IPAC to the University of Washington and a cloud service through Kafka, then to downstream brokers and scientific users.The pipeline also archives alerts and uses pre-defined community filters to produce source-separated streams.Source: §7 · p. 8
- ZADS transmitted on the order of one million alerts nightly, more than had ever been distributed via VOEvents; its alert volume approaches 10% of LSST with more modest hardware.The authors consider this strong evidence that the LSST alert problem is tractable.Source: §7 · p. 8