Source-linked AI summary
The Zwicky Transient Facility: Surveys and Scheduler
Eric C. Bellm, Shrinivas R. Kulkarni, Tom Barlow, Ulrich Feindt, Matthew J. Graham, Ariel Goobar, Thomas Kupfer, Chow-Choong Ngeow, Peter Nugent, Eran Ofek, Thomas A. Prince, Reed Riddle, Richard Walters, Quan-Zhi Ye
TL;DR
Time-domain imaging surveys need schedules that coordinate many observations under changing conditions while preserving cadence and operational constraints. The paper uses nightly Integer Linear Programming with temporal blocks, followed by slew optimization, and applies the approach to ZTF. The scheduler self-consistently weights airmass, seeing, and sky brightness to support transient discovery, while accepting limits on exact within-night cadence and global slew minimization.
Problem
Survey scheduling must coordinate large numbers of observations, coupled slews, changing conditions, and repeated time-domain visits efficiently.
Method
The scheduler uses Integer Linear Programming to assign observations to nightly temporal blocks, then solves a within-block Traveling Salesman Problem to reduce slew time.
Results
The algorithm was implemented for ZTF and maintained survey cadences while interleaving programs and trading filter-change costs against observation quality.
Takeaways & Limitations
The on-sky implementation demonstrates that this scheduling formalism could be feasibly applied to surveys such as LSST with an appropriately modified objective function.
Takeaways & Limitations
The scheduler cannot guarantee exact within-night cadences or globally minimize slew time, although slew overheads are small in practice.
Abstract
from arXiv · showhide
We present a novel algorithm for scheduling the observations of time-domain imaging surveys. Our Integer Linear Programming approach optimizes an observing plan for an entire night by assigning targets to temporal blocks, enabling strict control of the number of exposures obtained per field and minimizing filter changes. A subsequent optimization step minimizes slew times between each observation. Our optimization metric self-consistently weights contributions from time-varying airmass, seeing, and sky brightness to maximize the transient discovery rate. We describe the implementation of this algorithm on the surveys of the Zwicky Transient Facility and present its on-sky performance.
1. INTRODUCTION
Astronomical scheduling is difficult because large numbers of observations, coupled slews, changing conditions, and cadence requirements create a combinatorially large planning problem. The paper develops an automated, lookahead-based approach for wide-field time-domain surveys and applies it to ZTF.
- Scheduling challenge: 1,000 exposures have approximately 10^2567 possible orderings, making manual sequencing difficult at survey scale.Slews couple observations, so efficient scheduling must consider many targets together.
- Scheduling challenge: Manual scheduling is labor-intensive and cannot respond dynamically to changing conditions, weather losses, or Targets of Opportunity.Manual intervention is also required whenever the schedule changes.
- Automated scheduling: Greedy schedulers are easy to automate and adapt, but local optimization cannot deliver global optimization or reliably schedule repeated time-domain observations.Lookahead is particularly valuable when observations must be distributed across a planning interval.
- Prior approaches: Integer Linear Programming has been used to assign observations under observability and cadence constraints, including in the Las Cumbres Observatory scheduler.Related approaches include mixed-integer scheduling for ALMA and a Markov Decision Process formulation for LSST.
- Paper scope: This work schedules single-telescope wide-field imaging surveys by choosing fields and their order to maximize a transient-discovery proxy while maintaining cadence.The formalism is applied to ZTF and is intended to be relevant, with modifications, to other time-domain surveys.
2. ZTF REQUIREMENTS
ZTF required a scheduler that could improve efficiency while coordinating multiple independent surveys, supporting simulation and on-sky operations, and recovering from interruptions or weather losses.
- Design motivation: ZTF’s 8-second readout and many simultaneous programs made reducing scheduling overhead and testing survey plans especially important.PTF and iPTF had 40-second readouts.
- Operational requirements: The same scheduling code had to operate in simulation mode and on sky.
- Core requirements: The scheduler had to support several independent surveys while balancing observing time between programs.
- Operational requirements: ZTF required interfaces for Target of Opportunity observations and scheduler-status monitoring.
- Operational requirements: The scheduler had to recover appropriately from interruptions and weather losses.
- Core requirements: ZTF prioritized maximizing an observing-efficiency metric while maintaining cadence control.
3. OPTIMIZATION METRIC
The optimization metric weights exposures by the observable volume implied by their limiting magnitude, while modeling time-varying image quality and leaving cadence and program priority to explicit constraints.
- Metric definition: The scheduler maximizes the nightly total of exposures weighted by the spatial volume probed by each exposure.
- Metric definition: If observing cadences are well chosen, maximizing the weighted quantity maximizes the transient discovery rate.
- Metric definition: The volumetric weighting is derived from limiting magnitude through the limiting distance and volume relation.
- Time-varying conditions: Limiting magnitude depends on filter, seeing, airmass, and sky brightness, allowing the metric to trade these changing factors coherently through the night.
- Constraints: The metric excludes scientific priority and cadence because they lack a general quantitative relationship to the objective, so the optimization structure imposes those constraints separately.
- Scope: Other metrics can replace the ZTF weighting for surveys such as LSST that also optimize coadded depth.
- Scope: A global information model could require computationally expensive light-curve modeling inside the optimization loop and might be limited to one object class such as Type Ia supernovae.
4. ALGORITHM
The algorithm converts nightly observing requests into temporally blocked ILP decisions, using block structure to enforce cadence and filter constraints while accounting for changing observation quality.
- Inputs: Observing Programs specify fields, allocated telescope time, inter-night gaps, requested visits, and filters.
- Inputs: Request Sets combine eligible fields with requested observations, exposure times, and program metadata, while history and night length limit allowed requests.A sample request set contains three g-band and three r-band 30-second exposures.
- Nightly optimization: The scheduler uses ILP to choose request sets, assign their observations to times, and arrange slews and filter changes for efficiency.
- Temporal blocks: The night is divided into 15–25 temporal blocks, making whole-night scheduling tractable while supporting cadence constraints and filter-change minimization.
- Temporal blocks: ZTF sets the block size to 30 minutes, corresponding to the minimum desired separation between exposures.
- Decision variables: Each request’s volumetric weight varies by time block and filter as image quality, atmospheric transmission, and sky brightness change.
- Decision variables: Binary variables represent request assignments, filters used in blocks, and filter changes between adjacent blocks.
4.3. Objective
The objective maximizes volume-weighted exposures while accounting for filter-change losses. Constraints control filter choice, block duration, requested exposure counts, and program balance.
- The optimizer maximizes the volume-weighted number of exposures scheduled through the night while penalizing exposures lost to filter changes.The filter-change penalty uses the filter-change time and a weight factor approximating the maximum observation value.
- Each scheduled Request Set receives exactly the requested number of observations per filter.
- Only one filter may be used within a given time block.
- The observations assigned to a block must require no more time than the block provides.The calculation allows variable exposure times and uses a 9 sec fiducial overhead before within-block sequencing.
- A per-program exposure limit enforces the desired balance between observing programs.The allowed number is determined from allocated observing time, requested exposure lengths, and past observing history.
4.5. Sequencing Exposures within a block
After assigning observations to time blocks, the scheduler sequences each block by solving an ILP formulation of the Traveling Salesman Problem to minimize within-block slew time.
- The scheduler computes pairwise slew times between fields in each block and solves a Traveling Salesman Problem to minimize total within-block slew time.The optimized quantity is slew time between fields rather than the total length of a conventional salesman tour.
- A penalty discourages consecutive observations of the same field when multiple observing programs request it.This reduces redundancy among repeated exposures within a block.
- Hour-angle and declination slew times are time-independent for the equatorial P48, whereas dome slews are computed separately for each time block.
4.6. Re-solving within the Night
The scheduler can re-solve during the night to respond to lost time and schedule disruptions. Its implemented recovery strategy reassigned earlier unobserved requests at block boundaries rather than fully reoptimizing all sequences.
- The entire optimization problem can be resolved repeatedly during the night to account for weather, Targets of Opportunity, or other schedule disruptions.
- After time is lost, the scheduler reassigns the best earlier unobserved requests to unused time in the current block at each block boundary.This limited recompute avoids choosing program-dependent tradeoffs between completing sequences fully and accepting partial completion.
- Unlike LCO and ALMA schedulers, the ZTF approach explicitly and self-consistently accounts for observation quality because its surveys have equal priority.
4.8. Summary of Algorithm Features
The algorithm provides complete-night lookahead, cadence and program balancing, filter-change tradeoffs, and within-block slew optimization for ZTF surveys. These capabilities come with limits on exact cadence control, global slew minimization, block filling, and dynamic weather response.
- Capabilities: Complete-night lookahead schedules observations at favorable times while accounting for required counts, airmass, and sky conditions.The approach was motivated by cadenced observing and balancing several simultaneous surveys.
- Capabilities: The scheduler maintains night-to-night cadences, enforces survey time allocations, interleaves programs uniformly, and trades filter-change time against observation quality.
- Limitations: Exact within-night cadences and filter sequences are not supported; total observations per filter are guaranteed with roughly block-sized separation.A 12–15 minute g-to-i sequence cannot be specified, and even the minimum separation is not strictly guaranteed.
- Limitations: The algorithm minimizes slew time within blocks but does not globally minimize slew time between fields.In practice, the authors report that slew overheads are small.
- Limitations: Blocks may contain slack because their observation-count constraint is an upper bound, especially when requested programs do not fill available sky regions.Fallback queues and possible re-solves help use otherwise unscheduled time; without re-solves, slack is typically a few percent for balanced programs.
- Limitations: The current implementation does not dynamically adapt to changing observing conditions within the night.It does not perform cloud avoidance or adjust to seeing changes, although more extensive boundary reoptimization is proposed as a possible extension.
5. IMPLEMENTATION
The scheduler is implemented as an open-source Python library that supports simulation and operations, with nightly optimization integrated into ZTF’s robotic system. It uses sky-brightness modeling, constrained runtime resources, and list queues for programs requiring precise continuous-time sequencing.
- Implementation: The scheduler is publicly available as an open-source Python library and uses Gurobi for core integer-linear-programming optimization.It also uses Astropy, Astroplan, and pandas.
- Implementation: A gradient-boosted tree model predicts filter-specific sky brightness from telescope pointing, solar position, and lunar conditions.The model was trained on historical ZTF and initially PTF data.
- Operations: The library runs in simulation and operations through a RESTful webserver on ZTF’s primary robotic observing host.The webserver invokes the scheduling library and exposes command and status interfaces.
- Operations: The optimizer runs for five minutes before observations begin using two host-computer cores, producing satisfactory results without disrupting robotic operations.Additional CPUs could shorten runtime through parallelization.
- Operations: The robotic system handles calibration and focus updates concurrently with science observations, so these activities add no overhead.The scheduler selects on-sky science observations while the ROS obtains required calibrations and updates focus.
- Operations: More than 90% of ZTF Observing Programs use the ILP scheduler, while precisely timed programs use list queues with prescribed timing and exposure sequences.A monitoring thread checks for timed queues every ten seconds and switches to the appropriate program.
6. ZTF SURVEYS
ZTF divides observing time among public, collaboration, and Caltech TAC programs, using grid-based fields and program-specific cadences. The surveys span wide, high-cadence, Galactic Plane, twilight, and asteroid-focused observations, with associated data-release plans.
- Program allocation: ZTF allocates 40% of telescope time to public surveys, 40% to collaboration surveys, and 20% to Caltech TAC surveys, balancing these shares monthly.The monthly interval roughly matches changes in collaboration and Caltech sub-programs.
- Field grids: The primary field grid covers the sky with average overlaps of 0.29° in right ascension and 0.26° in declination, while adjacent-field slews fit within the 8.3-second CCD readout.Average field spacing is 7.2° north–south and 7.0° east–west.
- Field grids: The primary grid aligns with the Galactic Plane and covers 87.5% of the sky; a secondary offset grid fills CCD-gap coverage and adds depth in vignetted regions.The primary grid also supports coverage of nearby galaxies.
- Public surveys: The Northern Sky Survey receives 34% of total telescope time and observes eligible fields every third night in g and r, with at least 30 minutes between visits.It covers fields with δ ≥−31° and |b| > 7° and receives 85% of public time.
- Public surveys: The Galactic Plane Survey receives 6% of telescope time and visits eligible visible fields twice nightly in g and r, separated by at least 30 minutes.It uses the remaining 15% of public time and targets fields with |b| ≤ 7°.
- Collaboration surveys: Collaboration surveys included a six-visit nightly extragalactic cadence, a four-night i-band cadence, continuous Galactic Plane observations, twilight Near-Earth Object searches, and fast-rotating-asteroid observations.The extragalactic cadence used three g-band and three r-band visits nightly, while the i-band survey visited each field every four nights.
7. PERFORMANCE
Simulations found that ILP improved the survey-speed metric and sequence completion relative to greedy scheduling, while on-sky operations maintained efficient slews, cadences, program balance, and high completion.
- Simulated Performance: 9% higher survey-speed metric for ILP than greedy scheduling, despite comparable numbers of exposures per hour.ILP also produced a median airmass of 1.11 versus 1.20 for greedy scheduling, but averaged 3.6 filter changes per night versus 2.0.
- Simulated Performance: 81% of requested observations were completed by ILP across all three simulated surveys, compared with 64%, 79%, and 72% for greedy scheduling.The sequence completion fraction measures fields for which all desired nightly observations were obtained, including weather losses.
- On-Sky Performance: More than 120,000 observations were scheduled on sky while allocating 42% to MSIP, 40.2% to collaboration, and 18.8% to Caltech programs.The Caltech shortfall was partly attributed to intervals without available Caltech programs or with incomplete time allocations.
- On-Sky Performance: 9.9 sec was the median time between observations, with most slews spanning one field offset and tenth–90th percentile overheads of 9.4–14.9 sec.The corresponding tenth–90th percentile slew distances were 6.0°–16.1°.
- On-Sky Performance: Filter exchanges occurred less than once per hour on most nights, while the optimization metric traded filter changes against exchange time.The metric also preferentially scheduled observations around zenith, and its values varied sharply with moon phase.
- On-Sky Performance: 84.6% was the average completion rate for MSIP and collaboration high-cadence surveys, while 80% of observations were separated by at least 30 minutes.Major-survey intra-night cadences were delivered as expected, with minimal tailing toward longer revisit times.
8. CONCLUSIONS
The scheduler separates image quality, cadence, and sequencing efficiency, while supporting wide-field time-domain survey operations and potential adaptation to LSST. Its formalism addresses current survey scheduling needs, but general-purpose surveys cannot combine image quality and scientific-value optimization using object-specific light-curve knowledge.
- 8. CONCLUSIONS: The metric varies sharply with moon phase, and hour-angle distributions characterize ZTF’s observing behavior.The field grid produces smaller-scale metric structure because some fields transit at higher airmasses depending on declination.
- 8. CONCLUSIONS: The scheduler delineates image quality, desired cadence, and sequencing efficiency as three core concerns.Image quality includes signal-to-noise ratio, sky background, airmass, and related terms.
- 8. CONCLUSIONS: General-purpose wide-field surveys cannot combine image quality and scientific-value optimization using object-specific light-curve knowledge.Such tradeoffs are possible for surveys focused on a single class of astrophysical object, but not for general-purpose surveys.
- 8. CONCLUSIONS: The formalism could adapt to LSST by modifying the objective function to maximize a night’s contribution to total coadded depth under cadence constraints.Further work is needed to meet all LSST requirements and rigorously compare the algorithm with other scheduling approaches.
- 8. CONCLUSIONS: Future large surveys may gain cost-effective science-throughput improvements from survey-specific scheduling algorithms, metrics, and operations-research methods.Different surveys will necessarily require different algorithms and metrics.