PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated data reduction workflows for astronomy

W. Freudling, M. Romaniello, D. M. Bramich, P. Ballester, V. Forchi, C. E. Garcia-Dablo, S. Moehler, M. J. Neeser

arXiv:1311.5411v1astro-ph.IM

TL;DR

Complex astronomical instruments generate interdependent science and calibration data whose reduction can require difficult-to-modify workflows and continued supervision. The paper designs an automated workflow approach and implements it as ESO’s Reflex environment within Kepler. Reflex supports rule-based organisation, interactive and batch execution, workflow modification, and efficient re-reduction, while its design is intended to apply beyond this implementation.

  • Problem

    Complex, interdependent astronomical data-reduction workflows are difficult for individual researchers to organise, execute, optimise, and modify, while ground-based observations often still require supervision.

  • Method

    The paper designs an astronomical workflow-engine approach and implements it as Reflex, a package of customised components within the Kepler workflow engine.

  • Results

    Reflex provides automated rule-based data organisation, progress monitoring, interactive workflow modification, batch execution, result reuse, and data-processing overhead of a fraction of a second per recipe.

  • Takeaways & Limitations

    Automated workflows can improve astronomical data-reduction efficiency while preserving inspection, interaction, and modification of data organisation and flow.

  • Takeaways & Limitations

    Automatic pipelines work best with stable instruments, similar targets and signal-to-noise ratios, and predictable ambient conditions; ground-based observations often violate these conditions.

Abstract

from arXiv · show

Data from complex modern astronomical instruments often consist of a large number of different science and calibration files, and their reduction requires a variety of software tools. The execution chain of the tools represents a complex workflow that needs to be tuned and supervised, often by individual researchers that are not necessarily experts for any specific instrument. The efficiency of data reduction can be improved by using automatic workflows to organise data and execute the sequence of data reduction steps. To realize such efficiency gains, we designed a system that allows intuitive representation, execution and modification of the data reduction workflow, and has facilities for inspection and interaction with the data. The European Southern Observatory (ESO) has developed Reflex, an environment to automate data reduction workflows. Reflex is implemented as a package of customized components for the Kepler workflow engine. Kepler provides the graphical user interface to create an executable flowchart-like representation of the data reduction process. Key features of Reflex are a rule-based data organiser, infrastructure to re-use results, thorough book-keeping, data progeny tracking, interactive user interfaces, and a novel concept to exploit information created during data organisation for the workflow execution. Reflex includes novel concepts to increase the efficiency of astronomical data processing. While Reflex is a specific implementation of astronomical scientific workflows within the Kepler workflow engine, the overall design choices and methods can also be applied to other environments for running automated science workflows.

1. Introduction

Modern astronomical data reduction involves many interdependent science and calibration data types, complex methods, and changing observational conditions. The paper motivates automating data organisation and workflow execution while retaining inspection and interaction.

  • 1. Introduction: Modern instruments can produce dozens of interdependent science and calibration data types requiring complex reduction algorithms.ESO’s X-Shooter, for example, uses almost 100 data types across three simultaneously working arms.
  • 1. Introduction: General-purpose instruments, rapid upgrades, and variable atmospheric effects make ground-based data reduction difficult to execute and optimize individually.Supervision and interaction may remain essential for sufficiently high-quality results.
  • 1. Introduction: Automating data organisation, reduction-step execution, and result transfer can improve efficiency while retaining existing reduction routines.This addresses workflows whose number, complexity, and interdependence have increased despite stable general concepts.
  • 1. Introduction: Individual researchers commonly use scripting languages, but complex scripts can become difficult to modify when data organisation and bookkeeping grow complicated.Scripting works best with relatively few reduction steps and simple organisation requirements.
  • 1. Introduction: The paper presents a general workflow-engine approach for astronomical data reduction and introduces ESO’s Reflex implementation as a specific example.Reflex is discussed alongside workflow-engine methods and functionalities needed for astronomy.

2. Architecture of astronomy data reduction workflows

The architecture separates rule-based data organisation from data processing while representing file relationships as directed data graphs. It also supports efficient, reusable execution by caching intermediate results and reusing them when inputs and parameters match.

  • Data organisation: Data organisation selects data sets from larger file collections and records file types and selection reasons.It is typically among the first, complex, and time-intensive workflow tasks.
  • Data organisation: Files are classified by metadata, targets are grouped into initially incomplete data sets, and suitable calibration files are then selected.Selection criteria can use target and calibration metadata, such as matching filters or exposure times.
  • Data organisation: The selection cascade maps to a directed data graph whose nodes are actions and whose links carry metadata between raw files and target processing.Trigger files define selection properties, while products pass metadata to subsequent actions.
  • Data organisation: A data set can contain multiple purposes for a file, and graph topology may differ between data sets even for the same data type.For example, dark frames used for science and flat frames may be identical in one data set but different in another.
  • Data processing workflows: The workflow processes one data set at a time, routes files by category, and groups inputs for repeated calls of individual reduction steps.This design uses three grouping levels: data sets, file categories, and files processed together in separate reduction calls.
  • Data processing workflows: The lazy mode reuses stored results when a reduction step has identical input files and parameters, while allowing each step to disable reuse.Changed inputs or parameters trigger changed results and may require downstream re-execution.

3. Implementation

The paper presents workflow principles independently of software implementation and discusses their partial implementation in open-source workflow engines, ultimately selecting Kepler for Reflex.

  • Implementation: The workflow principles are independent of a specific implementation, but Reflex is discussed using the terminology of a selected workflow environment.Several open-source workflow engines were evaluated and some concepts were partially implemented before Kepler was chosen.

3.1. The Kepler workflow engine

Kepler provides a graphical workflow environment in which actors exchange tokens through ports and relations, while a director schedules their execution. Reflex uses the Dynamic Data Flow director to support result-dependent execution and looping.

  • The Kepler workflow engine: Kepler represents stand-alone programs as actors with named input and output ports that exchange tokens through relations.Actors emit output tokens after execution and consume input tokens when execution starts.
  • The Kepler workflow engine: A director is required because relations between actors do not themselves define temporal execution order.The director triggers actors according to the workflow's execution model.
  • The Kepler workflow engine: Reflex uses Kepler's Dynamic Data Flow director, which makes execution depend on actor results and supports looping and iteration.This couples scheduling behavior to data availability and intermediate workflow outcomes.
  • The Kepler workflow engine: Kepler's graphical interface supports creating, editing, executing, monitoring, pausing, and stopping workflows.The environment also bundles many general-purpose actors.

3.2. The Reflex Environment

Reflex uses Kepler-based actors and typed workflow tokens to route files, parameters, and products through configurable astronomical reduction recipes.

  • Recipe execution: Reflex supports command-line recipes that read categorized FITS files and produce categorized FITS products.PythonActor, IDLActor, and RecipeExecuter support recipes written in Python, IDL, or CPL.
  • Workflow tokens: SOF tokens contain file names, checksums, categories, and purposes, while SOP tokens contain named parameter values.Reflex actors process these objects as workflow tokens.
  • Purpose handling: Purpose operations let workflows pass through, universalize, or trim file purposes so recipes receive files with identical purposes.Trim removes the last action, pass-through preserves purpose, and universal acts as a wildcard.
  • Workflow interface: The X-Shooter workflow contains 8 recipe executers, interactive recipe steps, and a Flat Strategy actor that routes files according to the selected flatfielding strategy.Users can specify the raw data directory and start execution from the Kepler interface.
  • Workflow flexibility: Purpose-based operations support intermediate processing, shared files, and conditional routing without embedding arbitrary action names in workflow actors.Examples include smoothing flatfields with pass-through and sharing a bad pixel map through the universal purpose.

3.3. Reflex actors

Reflex organizes FITS inputs with human-readable OCA rules and uses specialized actors to execute recipes, route files, repeat processing, and support conditional branches.

  • Actor architecture: Reflex provides 17 essential actors grouped into data organization, token processing, recipe execution, and interactive workflow functions.These groups cover the main capabilities needed to construct astronomical reduction workflows.
  • Data organiser: DataOrganiser automatically classifies FITS files and creates complete or incomplete data sets from workflow-specific rules.It is the first actor after initialization and is triggered by an input token.
  • Data organiser: OCA classification, organisation, and association rules define file categories, trigger actions, minimum required files, and product categories.Classification uses FITS keywords, while organization rules determine actions and data-set completeness.
  • Data processing: Recipe actors filter input SOFs, execute Python, IDL, or CPL programs, and emit product SOFs with categorized outputs.Supported actors include PythonActor, IDLActor, and RecipeExecuter.
  • File routing: SOFCombiner, SOFSplitter, and SOFAccumulator route files by purpose and repeat recipe execution for separate purposes before recombining products.The resulting SOF can feed the next SOFCombiner for subsequent processing.
  • Conditional routing: Kepler actors also support conditional and iterative branches when files must be routed differently according to data properties or user choices.This extends the standard purpose-based routing mechanisms for special workflow needs.

3.4. Modularity of Reflex workflows

Reflex uses nested Kepler composite actors to present modular workflows, while configured directors make execution follow the intuitive completion order of each module.

  • Modularity: Composite actors encapsulate workflows and can be nested to arbitrary depth, allowing each processing stage and its supporting actors to form a modular unit.This produces a clean view of the complete data-processing workflow.
  • Execution order: Configured directors ensure each composite actor completes before processing proceeds to other actors, rather than relying only on diagram layout.Workflow layout alone does not uniquely determine actor execution order.

3.5. Book-keeping and product organisation

Reflex records workflow inputs, outputs, parameters, checksums, and creation dates to support product reuse, organization, evaluation, troubleshooting, and provenance tracking.

  • Book-keeping: The workflow database stores every step’s input and output files, processing parameters, file names, and checksums.This rigorous bookkeeping supports retrieval and processing during execution.
  • Product reuse: Checksums and creation dates enable lazy-mode detection of changed inputs and reuse of previously produced products.The database also organizes workflow outputs in a user-friendly way.
  • Product provenance: Each science product should be associated with the input data and parameters that generated it, including recursively linked intermediate products.Intermediate products support evaluation, troubleshooting, and investigation of science-product optimization.

4. Summary and conclusions

Reflex automates astronomical data-reduction workflows while retaining interactive control, efficient re-reduction, and support for batch processing. Its workflows are implemented in Kepler and distributed for several ESO VLT instruments.

  • Automated workflows provide progress monitoring and efficient modification of data organisation and data flow compared with scripting or monolithic programs.
  • Reflex uses rule-based data organisation, interactive data-set inspection, workflow monitoring, modification, and both graphical and batch execution modes.Users can inspect and modify selected data sets, while batch mode supports large or computationally intensive processing.
  • Reflex automatically identifies and re-executes only workflow steps affected by changed input files or parameters.This supports efficient re-reduction when a single input or parameter change triggers a dependent cascade of steps.
  • The data organiser may take about a minute initially for a typical observing run, while lazy mode greatly reduces subsequent organisation time.Workflow execution adds only a fraction of a second to each stand-alone recipe; the default additional memory allocation is 1536 MB.
  • Reflex workflows were developed for several commonly used ESO VLT instruments and distributed as pre-packaged, tutorial-supported installations.The listed instruments include FORS2, SINFONI, UVES, VIMOS, X-Shooter, and KMOS; demonstration data support modification and experimentation by novice users.

Appendix A: List of Reflex actors

The Reflex actor library provides components for constructing, routing, executing, inspecting, and controlling astronomical data-reduction workflows. Its actors cover both data-handling operations and interfaces to external scripts and recipes.

  • Data-organisation actors create, combine, accumulate, split, route, and interactively filter or choose FITS data sets.These include DataOrganiser, DataSetChooser, DataFilter, FitsRouter, SOFCombiner, SOFAccumulator, SOFSplitter, SofCreator, and SopCreator.
  • Workflow-control actors test whether file sets are empty and present tokens in human-readable form to support conditional data flows and inspection.IsSofEmpty enables different flows based on data availability, while ObjectToText renders Reflex tokens for users.
  • ProductRenamer renames FITS files using file keywords, and ProvenanceExplorer lets users inspect products from current or previous workflow runs.
  • Execution actors interface with Python, IDL, and CPL recipes, while RecipeLooper supports looping over one or several recipes.The actor list includes PythonActor, IDLActor, RecipeExecuter, and RecipeLooper.
Loading 1311.5411v1…