The LSST Data Management System

1. The Large Synoptic Survey Telescope

• LSST will repeatedly image about 18,000 deg^2 of sky over 800 times during its planned 10-year survey, in six optical bands from 320 to 1050 nm.Its survey system is designed to cover approximately 10,000 deg^2 every few nights.Source: §1 · p. 2
• The survey aims to address questions ranging from discovering “killer” asteroids to examining the nature of Dark Energy.The paper describes LSST as a large-aperture, wide-field, ground-based telescope with a 3.2-gigapixel camera and a data management system.Source: §1 · p. 2
• The paper reviews the Data Management system’s science goals and technical design, covering its architecture, planned data products, software stack, and broader opportunities for the codebase.It describes the system as relevant both to LSST and the astronomical software community.Source: §1 · p. 2

2. The LSST Data Management System

• The system will automatically reduce LSST data to scientifically useful catalogs and images because its scale and processing complexity make end-user reduction impractical.The observing program is expected to produce approximately 15 TB of raw imaging data per night.Source: §2 · pp. 2–3
• Within 60 seconds of observation, Level 1 processing will archive raw images, issue alerts for new or significantly changed sources, and update relevant catalogs.The data management system is divided into infrastructure, middleware, and applications layers.Source: §2 · p. 3, Figure 1 · p. 4
• Periodic Level 2 processing will calibrate accumulated survey data uniformly, measure detected objects, and characterize their time-dependent behavior in static, self-consistent Data Releases.All releases will be archived for the operational life of LSST; the two most recent will be queryable.Source: §2 · p. 3
• The system will allocate processing, storage, and network resources to user analyses and Level 3 products, while making LSST data available through community-accepted interfaces.The infrastructure is intended to enable analysis without petabyte-scale data transfers.Source: §2 · p. 3
• Across 10 years and 11 releases, processed imaging is expected to approach 500 PB and catalog databases to exceed 50 PB.The final data release catalog database alone is expected to be approximately 15 PB.Source: §2 · p. 3
• The Archive Facility in Champaign will perform all Level 1 nightly processing and 50% of Level 2 data-release processing; the Lyon satellite center will perform the remaining 50%.The DM system components span four key facilities on three continents.Source: §2 · pp. 3–4

3. LSST Data Products

• Level 1 alerts for newly discovered or significantly changed sources will be published within 60 seconds of observation to support transient science and rapid follow-up.Solar System objects will be linked across observations over a few days, with their orbits published within 24 hours of identification.Source: §3 · p. 5
• Annual Level 2 Data Releases will include single-epoch images, deep co-adds, object, source, and forced-source catalogs, and reprocessed Level 1 products.Releases will also include metadata and software needed to reproduce data release processing.Source: §3 · p. 6
• Level 2 processing will detect objects on co-adds but measure their properties by simultaneously fitting PSF-convolved models to sets of single-epoch observations.The fitted models generally include a constrained combination of two Sérsic profiles for extended sources and a point-source model with proper motion.Source: §3 · p. 6
• For extended-source fits, LSST will retain samples characterizing the likelihood surface and posterior, with up to approximately 200 independent samples per object.For storage reasons, samples may be retained only for bands of greatest interest to weak-lensing studies.Source: §3 · p. 6
• The system will devote processing and storage equivalent to 10% of its capabilities to user-created Level 3 products and support sharing results through LSST database infrastructure.This capacity is intended for a limited number of highly specialized investigations that may require custom processing.Source: §3 · p. 6
• The reusable, extendable software stack will be made available to users to develop custom Level 3 processing code and run it on their own resources.Users may also execute customized code at LSST data centers.Source: §3 · pp. 6–7
• In the most crowded Galactic-plane regions, performance considerations may limit fitting to the point-source model.Source: §3 · p. 6

4. The LSST Software Stack

• The stack is planned as a documented, high-performance, scalable, multi-camera, open-source O/IR data processing and analysis system for LSST reductions and custom Level 3 code.Its planned components include science and quality-assessment pipelines, data-access and orchestration middleware, image and catalog storage, and user-interface components.Source: §4 · p. 7
• The software builds on prior large-survey expertise, whose pipelines demonstrated largely autonomous reduction, source detection, and extraction of scientifically useful object characteristics.The cited examples include SDSS, Pan-STARRS, DES, SuperMACHO, ESSENCE, DLS, CFHTLS, and UKIDSS.Source: §4 · p. 7
• The stack was largely written anew for performance, extensibility, maintainability, and stringent science and system quality requirements, using Python with C++ where needed.Development follows practices including modularity, defined interfaces, continuous integration, unit testing, and shared documentation and coding standards.Source: §4 · p. 7
• An SDSS Stripe 82 co-add made with LSST prototypes illustrates background matching that improves background estimation and preserves diffuse structures.The figure shows a gri color composite near globular cluster M2, with the Moon inserted for scale.Source: Figure 3 · p. 8
• The current development version of the LSST stack forms the basis of the Hyper-Suprime Cam Survey pipelines and has produced two HSC data releases.LSST pipelines have also processed CFHTLS and SDSS images, while Qserv was tested on a 150-node cluster using 55 billion rows and 30 terabytes of simulated data.Source: §4 · pp. 7–8

• A comprehensive data management system is necessary to make possible the research expected from LSST’s highly automated survey.Source: §5 · p. 9
• The LSST software stack is free under GPL version 3, with prototype code and documentation available online.Source: §5 · p. 9
• Open development, long-term project commitment, and camera-modifiable design may support imaging-data processing beyond LSST.Source: §5 · p. 9