Source-linked AI summary
The Zwicky Transient Facility: Data Processing, Products, and Archive
Frank J. Masci, Russ R. Laher, Ben Rusholme, David L. Shupe, Steven Groom, Jason Surace, Edward Jackson, Serge Monkewitz, Ron Beck, David Flynn, Scott Terek, Walter Landry, Eugean Hacopians, Vandana Desai, Justin Howell, Tim Brooke, David Imel, Stefanie Wachter, Quan-Zhi Ye, Hsing-Wen Lin, S. Bradley Cenko, Virginia Cunningham, Umaa Rebbapragada, Brian Bue, Adam A. Miller, Ashish Mahabal, Eric C. Bellm, Maria T. Patterson, Mario Jurić, V. Zach Golkhou, Eran O. Ofek, Richard Walters, Matthew Graham, Mansi M. Kasliwal, Richard G. Dekany, Thomas Kupfer, Kevin Burdge, Christopher B. Cannella, Tom Barlow, Angela Van Sistine, Matteo Giomi, Christoffer Fremling, Nadejda Blagorodnova, David Levitan, Reed Riddle, Roger M. Smith, George Helou, Thomas A. Prince, Shrinivas R. Kulkarni
TL;DR
The paper addresses how to operate a processing and archival system that delivers science-quality products from a high-rate time-domain survey close to real time. It presents the IPAC-hosted ZSDS, integrating parallel pipelines, databases, archives, and interfaces, and reports that the system meets its real-time delivery requirements while supporting future large surveys. Its scope remains shaped by cost constraints and the need to monitor evolving data and classifier conditions.
Problem
The ZTF survey requires a processing and archival system that can deliver science-quality products close to real time for fast-evolving transients.
Method
The ZSDS integrates highly parallelized processing with distributed file servers, relational databases, archiving, alert distribution, and user-access services.
Results
The system meets real-time processing and product-delivery requirements, with capacity remaining because less than half of its 1192 physical cores are currently used.
Takeaways & Limitations
The ZSDS provides a foundation for larger time-domain surveys, including the planned Large Synoptic Survey Telescope.
Takeaways & Limitations
The system’s design, products, formats, and retrieval services were shaped by a highly cost-constrained development environment, while classifier reliability requires monitoring as data conditions change.
Abstract
from arXiv · showhide
The Zwicky Transient Facility (ZTF) is a new robotic time-domain survey currently in progress using the Palomar 48-inch Schmidt Telescope. ZTF uses a 47 square degree field with a 600 megapixel camera to scan the entire northern visible sky at rates of ~3760 square degrees/hour to median depths of g ~ 20.8 and r ~ 20.6 mag (AB, 5sigma in 30 sec). We describe the Science Data System that is housed at IPAC, Caltech. This comprises the data-processing pipelines, alert production system, data archive, and user interfaces for accessing and analyzing the products. The realtime pipeline employs a novel image-differencing algorithm, optimized for the detection of point source transient events. These events are vetted for reliability using a machine-learned classifier and combined with contextual information to generate data-rich alert packets. The packets become available for distribution typically within 13 minutes (95th percentile) of observation. Detected events are also linked to generate candidate moving-object tracks using a novel algorithm. Objects that move fast enough to streak in the individual exposures are also extracted and vetted. The reconstructed astrometric accuracy per science image with respect to Gaia is typically 45 to 85 milliarcsec. This is the RMS per axis on the sky for sources extracted with photometric S/N >= 10. The derived photometric precision (repeatability) at bright unsaturated fluxes varies between 8 and 25 millimag. Photometric calibration accuracy with respect to Pan-STARRS1 is generally better than 2%. The products support a broad range of scientific applications: fast and young supernovae, rare flux transients, variable stars, eclipsing binaries, variability from active galactic nuclei, counterparts to gravitational wave sources, a more complete census of Type Ia supernovae, and Solar System objects.
1. Introduction
ZTF is a next-generation optical time-domain survey, and this paper presents the IPAC-hosted Science Data System that processes, archives, and distributes its products. The system combines pipelines, alert services, archive access, and user interfaces for near-real-time scientific use.
- Survey and Science Goals: ZTF is a next-generation optical time-domain survey operating with the Palomar 48-inch Schmidt Telescope.It extends predecessor Palomar Transient Factory programs and targets near-Earth asteroids, rare and fast-evolving transients, and Galactic variable sources.
- Science Data System: The ZSDS provides data transfer, raw-data ingestion, processing pipelines, long-term archiving, product access, alerts, quality assurance, and user support.It is housed at IPAC, Caltech, and supports retrieval and analysis of delivered science products.
- Event Processing: The system interfaces with machine-learned vetting subsystems that purify raw transient events extracted by the pipelines.More detailed algorithms, data formats, and archive-query examples are provided in the ZSDS Explanatory Supplement and archive documentation.
- Paper Scope: The paper reviews the ZSDS design, processing pipelines, science products, archive services, analysis platforms, data rates, and performance.The paper also discusses real-time runtime, alert latency, and archive-product quality.
2. Data System and Operations Overview
ZSDS is an IPAC-built, high-throughput processing and archival system designed to deliver science-quality products close to real time. Its parallel infrastructure, databases, file systems, and specialized services are organized around independent CCD-quadrant processing and sustained nightly operations.
- Data System and Operations Overview: 20 minutes is the ZSDS requirement from observation to science-quality product delivery.The budget assigns 10 minutes to data transfer to IPAC and 10 minutes to processing, primarily to support fast-evolving transients and timely follow-up.
- Data System and Operations Overview: The ZSDS uses highly parallelized, high-throughput processing integrated with distributed file servers and relational databases.The archive is hosted by NASA/IPAC IRSA, while ZTF-Depot and ZADS support internal access and alert distribution.
- Processing Architecture: Each camera exposure contains 64 independently packaged CCD-quadrant images in a compressed multi-extension FITS file.CCD quadrants are the basic processing units, and all calibrations, images, catalogs, and light curves are derived and distributed on that basis.
- Processing Architecture: CCD-quadrant processing avoids cross-quadrant and cross-CCD dependencies, keeping memory use, file sizes, and processing throughput manageable.The approach also preserves sufficient source counts for robust signal-to-noise-dependent processing steps.
- Processing Architecture: 528 concurrent threads process an incoming average of 96 quadrants per minute, with a single quadrant processed in under five minutes on average.The 66-node cluster has 1192 physical cores, and the authors report more than enough cores for the incoming data rate under current network and disk performance.
- Processing Architecture: The system meets real-time processing and product-delivery requirements while currently using less than half of its 1192 physical cores.Additional capacity supports asynchronous tasks such as reference-image generation and moving-object detection, with further real-time expansion planned.
- Archive and Interfaces: ZTF-Depot provides internal access to QA metrics, survey statistics, coverage maps, and near-real-time solar-system products.Public-release products are copied to the long-term IRSA archive, while databases track pipeline state and archive metadata.
3. Pipelines
The ZSDS organizes raw-data transfer, ingestion, calibration, archiving, and downstream products into multiple coordinated pipelines. Its infrastructure supports high-throughput operations while accounting for network limits and ingestion failures.
- Pipeline organization: Nine pipelines generate ZTF products, including separate raw-data, bias-calibration, and flat-field pipelines.Seven are depicted in Figure 2 because raw-data ingestion and splitting, and the two calibration-derivation pipelines, are counted separately.
- Data transfer: The camera produces 16 CCD-based raw files per exposure, with approximately 1.3 GB of native data before packaging.Exposures are typically separated by 40 seconds, combining 30 seconds of integration, 8 seconds of readout, and 2 seconds of overhead.
- Data transfer: Raw files are compressed by approximately 2 to 2.5 before transmission to accommodate the HPWREN bandwidth, with negligible information loss.The compression increases overall pixel noise by approximately 0.2%.
- Data transfer: Median transfer time is approximately 20 seconds per exposure, with 90th- and 99th-percentile times of approximately 24 and 40 seconds.Exposures are typically transferred to IPAC within the observing cadence without backlog, although network performance depends on usage and weather.
- Raw-data ingestion: At ingestion, files undergo metadata and checksum validation before archival, decompression, quadrant splitting, and overscan correction.Failed checks mark files unusable and exclude them from the archive; successful files are decompressed into MEF format and processed at the CCD-quadrant level.
- Operational boundary: The ingest failure rate is 16 CCD files, equivalent to one exposure of image data, per night at the time of writing.The failures are attributed to timing glitches between observation start and shutter-open times, with a solution in progress.
- Calibration products: The bias and high-frequency-flat pipelines require at least 20 input images and use trimmed per-pixel averages with nσ clipping.High-frequency flats correct relative pixel-to-pixel responsivity variations and capture night-to-night changes affecting photometric precision and gain matching.
3.5. Instrumental Calibration
Instrumental calibration is the first real-time processing phase, converting overscan-corrected quadrant images into calibrated, astrometrically and photometrically characterized science products.
- Pipeline inputs and outputs: Instrumental calibration combines science images, masks, recent calibration frames, Gaia references, PS1 calibrators, and processing parameters.Its primary products are productIDs 2, 3, 4, and 14.
- Calibration processing: The pipeline applies bias and flat corrections, detector non-linearity correction, streak and ghost masking, astrometric calibration, PSF modeling, and PSF-fit photometry.The source catalog uses a ZTF-optimized DAOPhot version and supports de-blending through multi-detection fitting.
- Pipeline integration: Standalone instrumental-calibration products can feed image differencing immediately when required inputs are available.Otherwise, the pipeline archives only science-image and catalog products, preserving flexibility for reprocessing and testing.
- Astrometric calibration: Gaia DR1 sources with 12 ≤ G ≤ 18 are used for astrometric calibration, excluding saturated ZTF detections.Astrometric solutions are derived with SCAMP in two passes.
- Photometric calibration: Photometric calibration matches filtered, stable PS1 DR1 calibrators to unsaturated PSF-fit sources from each CCD quadrant.A robust fit derives a filter-specific zero point and color term for g, r, or i.
3.6. Image Differencing and Event Extraction
The second real-time phase builds reference images, subtracts them from calibrated exposures, extracts point-source and streak events, and packages thresholded detections for downstream use.
- Pipeline scope: Image differencing extracts point-source transient, moving-object, and streak candidates from calibrated CCD-quadrant products when a usable reference image exists.The primary products include productIDs 5, 9, 12, and 14.
- Image differencing: Difference-image preparation matches photometric throughputs, resamples the reference, masks bad pixels, estimates differential background, and computes an effective PSF.QA thresholds determine whether the difference image is usable for event extraction.
- Point-source extraction: Point-source events are detected on both positive and negative match-filtered S/N images and measured with aperture and PSF-fit photometry.The negative image is the sign-reversed positive difference image produced by one ZOGY run.
- Streak extraction: Streak fitting estimates endpoint positions, length, position angle, integrated flux, uncertainties, covariances, and goodness-of-fit metrics.FindStreaks detects linear streaks in the positive difference image using a PSF-convolved line model.
- Products and review: Events, features, and QA metrics enter the operations database, while difference images, PSFs, streak metadata, and cutouts support archival and human review.A post-processing step combines event metadata with database context and generates alert packets for thresholded events.
- Reference images: Reference images are co-adds of 15 to 40 high-quality historical CCD-quadrant images selected using calibration, image-quality, noise, and background criteria.The minimum of 15 images was chosen to maximize early sky coverage and trigger differencing.
- Reference images: Reference-image construction resamples, gain-matches, and combines inputs with outlier-trimmed averaging and nσ clipping while masking bad pixels and saturation-related artifacts.QA metrics are computed for the resulting co-add products and recorded in their FITS headers.
3.8. Source Matching and Photometric Refinement
Source matching cross-matches multi-epoch PSF photometry into HDF5 matchfiles, optionally refines relative photometry, and computes statistics for light-curve queries and moving-object analysis.
- Source matching: The source-matching pipeline positionally cross-matches PSF-fit catalogs across epochs for each field, CCD quadrant, and filter.Its primary product is an HDF5 matchfile containing matched-source light curves and statistics.
- Source matching: Matchfile creation requires a survey-ready reference-image source catalog and typically runs on timescales of one month or longer.Updating existing matchfiles depends on accumulating enough new epochal data.
- Matchfile construction: Reference-image catalogs seed source tables, after which epochal PSF-fit catalogs are incorporated into the matchfile.The reference and epochal catalogs are stored as structured HDF5 tables.
- Photometric refinement: Relative photometric refinement estimates a delta-zero-point correction for each exposure from up to 5000 stable sources.Both corrected and uncorrected magnitudes are retained, and rerunning the refinement can slightly shift prior corrections as data accumulate.
- Light-curve products: Collapsed light-curve statistics include magnitude summaries, dispersion measures, shape statistics, chi-square, extrema, slopes, and Stetson indices.These statistics are stored in HDF5 tables and support user queries.
- Archive access: The objects database stores match identifiers, collapsed statistics, metadata, and reference positions rather than the individual light-curve points.A GUI provides spatial searches and range queries over the precomputed statistics.
- Moving objects: Moving-object tracks link point-source events across multiple exposures, excluding streaks from faster-moving objects.ZMODE uses recent observing-night detections and produces orbit-related reports, scores, and image cutouts.
4. Data Products
The ZSDS produces quadrant-based images, catalogs, alerts, moving-object products, and quality metrics, distributing alerts through a near-real-time system and archiving products for later access.
- Image and catalog products: Primary products include raw and calibrated science images, difference images, reference co-adds, source catalogs, and light-curve files.Most products are generated per filter and CCD quadrant.
- Solar-system products: Moving-object products include streak diagnostics, image cutouts, machine-learned scores, and linked point-source events, with likely real candidates delivered to the MPC.Products are copied to ZTF-Depot for human vetting before likely candidates are sent onward.
- Quality assurance: QA products contain image-quality, calibration, source-extraction, telemetry, and nightly depth-of-coverage metrics.Both cumulative and nightly survey coverage maps are generated per filter.
- Alerts: Alert packets encode metadata and contextual information for individual events from positive or negative difference images, including transient, variable, and moving-object detections.Packets are lightly filtered before distribution and use Apache Avro serialization.
- Alert context: Alert content includes source-specific event features, repeated object identifiers, and historical events within 1.5 arcsec over a 30-day search span.Historical events may come from all filters and either positive or negative difference images.
5. Archive Access, Services, and Tools
The ZTF archive provides near-real-time product access through graphical and programmatic interfaces, with specialized tools for moving objects, light curves, and time-series analysis.
- Archive access: Epochal science products enter the archive near-real-time, while reference co-adds are archived soon after creation.Archived file products are summarized in the paper’s product inventory.
- Graphical interface: The GUI supports sky-position, object-name, field, and CCD queries, metadata filtering, downloads, image previews, cutouts, statistics, and source overlays.Users can query positions in any coordinate system or names resolved by NED or SIMBAD.
- Programmatic access: The API enables scripted archive queries and downloads using searchable quadrant-image metadata and constructed product paths.It is intended for non-interactive use with tools such as wget or curl.
- Moving-object tools: Moving-object searches accept designations, NAIF IDs, MPC ephemerides, or orbital parameters, while MOST adds orbit plots, elements, ephemerides, and download links.MOST also provides an accompanying API for scripted queries.
- Light-curve analysis: Light-curve services support cone searches and metric filtering, and a time-series viewer provides interactive plots, image inspection, periodograms, statistics, and photometry editing.Saved light curves can be uploaded to the generic viewer for further analysis.
6. Data Volumes, Rates, and Source Statistics
The ZTF system is designed for high nightly data throughput, producing petabyte-scale archives, billions of source measurements, and large streams of machine-learned alerts and streak candidates.
- Nightly rates: A typical uninterrupted observing night spans approximately 8 hr 40 min and generates about 700 science exposures.The estimates are based on typical nightly survey operations.
- Transient alerts: Difference-image processing extracts approximately 10^5 to 2 × 10^6 events and 10^3 to 10^5 likely real 5σ machine-learned point-source alerts.The alert counts include flux transients, variables, and solar-system objects.
- Streak detections: Approximately 1.5 × 10^3 machine-learned streak candidates are expected, compared with approximately 3 real human-vetted streaks from fast-moving solar-system objects.The candidate population is largely associated with artificial satellites, aircraft trails, and CCD-bleed artifacts.
- Archive scale: The expected three-year archive contains approximately 3.2 PB of products, including about 60 TB of reference images and 50 TB of matchfiles.These projections assume 260 good-weather observing nights per year.
- Source statistics: The survey is expected to produce approximately 1.8 billion reference-source seeds, 800 billion PSF-fit measurements, and 230 billion aperture measurements.These are projected source statistics for the survey products.
7. Pipeline and On-Sky Performance
The ZTF real-time system processes observing data without backlog while delivering astrometric, photometric, sensitivity, calibration, and alert products with quantified performance. Performance varies with observing conditions, source density, and calibration quality.
- 7.1. Realtime Pipeline Runtime: ≈5.8 minutes per CCD-quadrant is typical for the full real-time pipeline, while Galactic-plane processing can exceed 12 minutes per quadrant at the 95th percentile.Phase 1 alone takes ∼1.5 to 2 minutes per CCD-quadrant; the analyzed 7 hr 30 min night was processed in approximately the same time without accumulated backlog.
- 7.3. Photometric Precision (Repeatability): 8 to 25 millimag is the typical internal photometric precision at bright unsaturated fluxes, with lower precision generally occurring at higher airmass.There is no significant deterioration in photometric precision in high-density regions such as the Galactic plane, where source de-blending is more prevalent.
- 7.4. Sensitivity Limits: 5σ sensitivity corresponds to r ≈20.5 to 21.0 mag for the quadrants shown, and median limits across mapped quadrants are g=21.067±0.003, r=21.012±0.002, and i=20.51±0.01 mag.The 5σ threshold is defined here as 20% flux RMS; model-based flux-uncertainty estimates are consistent with repeatability estimates.
- 7.5. Photometric Calibration Assessment: PSF-fit catalogs are recommended for point-source photometry because they are more immune to source-crowding effects and more optimal for faint sources.For bright sources, aperture photometry is generally superior when there is no contamination.
- 7.7. Latency of Alert Packet Generation: The alert-production latency has 5th, 50th, and 95th percentiles of ≈6.7, 8.5, and 12.6 minutes, respectively.The latency tail is associated with dense Galactic-plane fields and additional processing caused by imperfect image differencing; pipeline processing contributes ≈5.8 minutes per CCD-quadrant.
8. Advice and Lessons Learned
The authors emphasize that a big-data, small-team project must constrain requirements, document interfaces, and monitor data and classifier quality. Real-time production and quality assurance require coordinated operational safeguards and sufficient human review.
- Requirements and interfaces: Clear requirements and early community engagement are essential because adding functionality late can require costly system re-architecture and unplanned hardware.The authors describe this as a recurring challenge in a big-data, small-team project.
- Requirements and interfaces: Interfaces among the ZSDS, observing system, quality-assurance subsystem, and alert-distribution system should be documented and agreed before development.Specifications should include metrics and data-product formats, with proposed updates evaluated against available resources.
- Operational monitoring: The alert-production latency distribution includes raw-data transfer, processing, and staging in the Kafka cluster, while event counts are evaluated against science-image metrics.These measurements provide operational diagnostics for the real-time system.
- Quality assurance: Machine-learning vetting reliability must be monitored because changes in instrumentation, observing conditions, survey design, processing, references, or astrophysical environments can invalidate training assumptions.The authors specifically recommend reevaluating classifier performance as these conditions change.
- Quality assurance: Coordinated eyes-on-data during commissioning and science validation is needed to analyze products across observables and astrophysical environments.The authors report that insufficient analyst coverage during these phases limited their ability to detect issues early.
9. Summary and Future Updates
The ZTF Science Data System integrates processing, alerts, Solar System discovery, archiving, and data-access services around flexibility and robustness. Ongoing refinements target automated vetting, calibration, user analysis, and scalability to larger time-domain surveys.
- Current system: The ZSDS encompasses processing pipelines, real-time transient alerts, an SSO discovery engine, a long-term archive, and services for examining and retrieving data products.Its development was guided by experience from PTF and iPTF and continues to respond to active science programs.
- Design philosophy: Flexibility and robustness cover varied astrophysical environments and observing conditions, instrumental glitches, survey-design changes, hardware outages, and near-real-time delivery of reliable events.The design philosophy also aims for minimal tuning unless instrumental performance changes.
- Future updates: Ongoing improvements refine machine-learning vetting for point-source alerts and moving objects and improve instrumental calibrations for astrometric and photometric quality.These changes are intended to enhance quality as the survey proceeds.
- Future updates: Planned functionality includes forced photometry at user-supplied positions and custom co-add or mosaic construction from the image archive.Both additions expand user-directed analysis of archived data.
- Broader significance: Advances in algorithms, software, and data management at IPAC/Caltech support the development of larger time-domain surveys such as LSST.The paper presents this as a broader outcome of the ZSDS work.
Appendix A Acronyms
The appendices provide acronym expansions used throughout the ZTF Science Data System documentation and identify ZMODE as the moving-object track-construction component.
- Acronyms: The acronym list expands technical terms including ADES, NEA, NED, NEOCP, API, CCD, P48, SSO, IRSA, SQL, and ML.These abbreviations span astrometry, Solar System objects, infrastructure, archives, databases, and machine learning.
- Moving-object terminology: ZMODE is identified as the schematic’s system for constructing moving-object tracks.The figure presents the track-construction process conceptually.
Appendix B ZMODE: ZTF’s Moving Object Discovery Engine
ZMODE links difference-image detections into candidate Solar System object tracks using three-detection stringlets, velocity matching, and orbit-based vetting. This design reduces downstream combinations relative to two-detection tuple construction, while reported performance exceeds 90% efficiency and 98% reliability in specified PTF tests.
- Performance: >90% efficiency and >98% reliability were measured for recovering known Solar System objects in PTF data to approximately R_PTF ∼20 mag.These results describe moderately faint flux levels in PTF observations.
- Stringlet construction: ZMODE forms minimum three-detection stringlets by matching relative velocities of adjacent detection pairs, unlike MOPS, which begins with two-detection tuples.One pair must span ≤10 hours and the other ≤2.5 days; four consecutive nights support the intended cadence.
- Track construction and vetting: Stringlets are merged into candidate tracks using velocity matching on a coarser grid, followed by quality metrics including dynamically plausible orbit fits.The final merging uses both intra-stringlet mean velocities and inter-stringlet relative velocities.
- Computational design: Three-detection stringlets reduce spurious two-detection combinations and the number of combinations processed and merged downstream.The design moves part of the combinatorial challenge earlier in processing compared with MOPS.
- Track vetting: Tracks with four or more linked detections that accurately fit a dynamically valid orbit are treated as worthy candidates for further inspection.Orbit fitting supplies an orbit-quality score and RMS fit residual.
- Scope: A detailed account of the ZMODE algorithm and its performance is deferred to a future publication.The appendix therefore provides an overview rather than the complete algorithmic evaluation.