Source-linked AI summary
The Herschel Data Processing System - HIPE and Pipelines - Up and Running Since the Start of the Mission
Stephan Ott, Herschel Science Centre, European Space Agency
TL;DR
The paper describes a need for an integrated data-processing system to support instrument testing and astronomy data analysis. It presents a system combining retrieval, pipelines and analysis, and reports successful early processing across Herschel instruments, including a PACS image produced within 30 minutes of receiving the first observation data.
Problem
The system was developed to support data analysis for Instrument Level Tests and to provide the astronomical community with an integrated processing environment.
Source: §2 · p. 2Method
Results
The first PACS M51 image was generated in HIPE within 30 minutes of receiving the observation data, while first-light HIFI and SPIRE observations were also handled by HIPE and the pipeline.
Source: §5 · p. 3Takeaways & Limitations
Products were usually made available to data owners through the Herschel Science Archive on the same day the satellite data were received.
Source: §4 · p. 3
Abstract
from arXiv · showhide
The Herschel Space Observatory is the fourth cornerstone mission in the ESA science programme and performs photometry and spectroscopy in the 55 - 672 micron range. The development of the Herschel Data Processing System started in 2002 to support the data analysis for Instrument Level Tests. The Herschel Data Processing System was used for the pre-flight characterisation of the instruments, and during various ground segment test campaigns. Following the successful launch of Herschel 14th of May 2009 the Herschel Data Processing System demonstrated its maturity when the first PACS preview observation of M51 was processed within 30 minutes of reception of the first science data after launch. Also the first HIFI observations on DR21 were successfully reduced to high quality spectra, followed by SPIRE observations on M66 and M74. A fast turn-around cycle between data retrieval and the production of science-ready products was demonstrated during the Herschel Science Demonstration Phase Initial Results Workshop held 7 months after launch, which is a clear proof that the system has reached a good level of maturity. We will summarise the scope, the management and development methodology of the Herschel Data Processing system, present some key software elements and give an overview about the current status and future development milestones.
1. Introduction
Herschel is an ESA cornerstone mission whose three instruments perform photometry and spectroscopy from 55 to 672 microns. After its May 2009 launch, it began observational tests one month later.
1. Introduction
- Herschel’s HIFI, PACS and SPIRE instruments perform photometry and spectroscopy in the 55–672 micron range.The observatory is the fourth cornerstone mission in the ESA science programme.Source: §1 · p. 1
- Herschel launched on 14 May 2009, and its cryostat lid was opened one month later for the first observational tests.Most PACS and SPIRE performance-verification and science-demonstration activities had been completed; routine operations were expected before the end of 2009.Source: §1 · p. 1
- The paper is presented on behalf of the Herschel Science Ground Segment Consortium.Source: §1 · p. 1
2. Scope, Management and Development Methodology of the Herschel Data Processing System
The Herschel Data Processing System was developed from 2002 to support instrument-level testing and provide the astronomical community with an integrated, tested, documented system. It combines retrieval, pipeline processing and scientific analysis, with joint management and contributions from ESA and NASA centres and instrument control centres.
2. Scope, Management and Development Methodology of the Herschel Data Processing System
- The system combines data retrieval, pipeline execution and scientific analysis in one environment, with instrument-expert tools and Standard Product Generation software included.It was first developed to support analysis for Instrument Level Tests, and was also used for pre-flight characterisation and ground-segment test campaigns.Source: §2 · p. 2
- The Java/Jython system is designed to be license-free and portable across operating systems, with source code freely available under the GNU Lesser General Public License.The source was already accessible to Herschel Key Programme users.Source: §2 · p. 2
- More than 200 contributors and 60 full-time equivalents work on calibration, coding, documentation, quality control, testing and tutoring.The Herschel Science Centre, the HIFI, PACS and SPIRE Instrument Control Centres, and the NASA Herschel Science Center jointly manage and contribute to the system.Source: §2 · p. 2
3. Key Software Elements of the Herschel Data Processing System
HIPE provides user-oriented access to Herschel data processing, while the system also supports product access, archive retrieval and documentation. The tools allow users to work with official pipelines and adapt them to their scientific needs.
3. Key Software Elements of the Herschel Data Processing System
- Installers were available for several operating systems, with formal support for Windows XP, Vista, Linux and Mac OS X 10.5.Formal support for Windows 7 and Mac OS X 10.6 was planned for the future.Source: §3 · p. 2
- HIPE offers both a script-driven command-line environment for developers and experts and a GUI-based, data-centric interface for end users.The same framework can download, reprocess, analyse and compare data from all three instruments.Source: §3 · p. 2
- HIPE allows users to run the official pipeline and adapt it to their scientific needs within the same environment.Its features include drag-and-drop, command-line echoing, menu and toolbar access, and customizable views.Source: §3 · pp. 2–3
- The Product Access Layer stores, queries and loads data from specialized product pools, supports collaborator data sharing, and provides batch access to raw data in the Herschel Science Archive.Source: §3 · p. 3
- The system provides framework and instrument documentation with context-sensitive search.Source: §3 · p. 3
4. Operational Use of the Herschel Data Processing System
Herschel pipelines generated products at successive reduction levels, which were ingested into the archive and usually made available to data owners on the day of reception. Every scientific observation also received quality control combining automated screening and expert inspection.
4. Operational Use of the Herschel Data Processing System
- Pipeline products progress from Level 0 raw telemetry, through Level 1 calibrated detector readouts in physical units, to Level 2 absolutely calibrated images, spectral cubes and spectra for scientific analysis.Source: §4 · p. 3
- Products are ingested into the Herschel Science Archive and usually made available to data owners on the same day the satellite data are received.Source: §4 · p. 3
- All scientific observations undergo quality control combining automatic screening and manual inspection by Technical Assistants and Instrument Calibration Scientists.Reports are electronically distributed to experts, usually within a few days, and findings are summarized with the data products.Source: §4 · p. 3
5. Handling of Herschel data during its early operational days
Early Herschel observations were processed successfully by HIPE and the operational pipeline, with a first PACS image produced within 30 minutes. Early results also included HIFI spectra, SPIRE observations, and PACS observations revealing Milky Way details.
5. Handling of Herschel data during its early operational days
- Within 30 minutes of receiving the first PACS observation of M51, HIPE generated an image, followed by the operational pre-launch pipeline.Figure 1 compares HIPE (left) and the current operational pipeline (right) on SPIRE’s first-light observation of M66.Source: §5 · p. 3, Figure 1 · p. 4
- HIFI’s first-light observation of DR21 produced CO and H2O line spectra in both the interactive environment and standard pipeline, while SPIRE’s M66 and M74 observations were handled by both.Source: §5 · p. 3
- Early results included PACS imaging spectroscopy of NGC6543 and PACS/SPIRE parallel-mode observations that revealed new details in the Milky Way.Source: §5 · pp. 3–4
6. Current Status and future Development Milestones
By the end of 2009, HIPE 1.2 was operational and HIPE 2.0 was under testing. Future releases were planned regularly, about every three months during the following year.
6. Current Status and future Development Milestones
- As of the end of 2009, HIPE 1.2 was operational while HIPE 2.0 was under testing for the Science Routine Phase.Source: §6 · p. 4
- HIPE 1.1 was made available to Herschel Key Program teams and used for the first bulk reprocessing exercise; early HIPE 1.2 supported September 2009 Science Demonstration Phase data products.Source: §6 · p. 4
- Future HIPE versions were planned for regular release, about every three months during the following year.Source: §6 · p. 4