PaperlayerView exact revision on arXiv

Source-linked AI summary

PyWiFeS: A Rapid Data Reduction Pipeline for the Wide Field Spectrograph (WiFeS)

Michael J. Childress, Frédéric P. A. Vogt, Jon Nielsen, Robert G. Sharp

arXiv:1311.2666v1astro-ph.IM

TL;DR

WiFeS wavelength calibration requires identifying arc-lamp lines across a multiplexed instrument, a repetitive process that is time-consuming and prone to human error. PyWiFeS provides Python reduction routines, a global optical wavelength solution, and scriptable metadata and reduction workflows. The pipeline accounts for standard integral-field reduction procedures and includes sophisticated corrections designed for WiFeS.

  • Problem

    Manual arc-line identification across multiplexed spectroscopic data is time-consuming and particularly prone to human error.

    Source: §3 · p. 13

  • Method

    PyWiFeS combines Python data-processing routines with a global optical wavelength solution and an abstract metadata structure that separates file-role assignment from reduction procedures.

    Source: §5 · p. 20

  • Results

    The released pipeline accounts for standard integral-field reduction procedures and includes scattered-light flatfield corrections and a global wavelength solution based on instrument optics.

    Source: §5 · p. 20

  • Takeaways & Limitations

    Scriptable FITS-based processing supports repeat reductions, while abstract instrument variables could allow adaptation to other image-slicing integral field spectrographs.

    Source: §5 · p. 20

    The default reduction and metadata scripts cannot currently handle multiple observing modes at once, and incompatible frame sizes can cause failure.

    Source: §4.2 · p. 20

Abstract

from arXiv · show

We present PyWiFeS, a new Python-based data reduction pipeline for the Wide Field Spectrograph (WiFeS). PyWiFeS consists of a series of core data processing routines built on standard scientific Python packages commonly used in astronomical applications. Included in PyWiFeS is an implementation of a new global optical model of the spectrograph which provides wavelengths solutions accurate to $\sim$0.05 \AA\ (RMS) across the entire detector. The core PyWiFeS package is designed to be scriptable to enable batch processing of large quantities of data, and we present a default format for handling of observation metadata and scripting of data reduction.

1 WiFeS and PyWiFeS: An Introduction

WiFeS is an image-slicing integral field spectrograph whose data support a range of astronomical studies. PyWiFeS was developed to provide rapid reduction for real-time transient classification, using scriptable Python routines and flexible data formats.

1 WiFeS and PyWiFeS: An Introduction

  • WiFeS is an image-slicing integral field spectrograph used for studies including galaxy kinematics, exoplanet-host radial velocities, spatially resolved galaxy emission, and interacting supernovae.The instrument is mounted on the ANU 2.3m telescope at Siding Spring Observatory.Source: §1 · p. 1
  • PyWiFeS addresses the need for rapid reduction of WiFeS observations to support nearly real-time processing and transient classification.The earlier WiFeS pipeline used IRAF; PyWiFeS was developed to meet the rapid-reduction criterion.Source: §1 · p. 1
  • PyWiFeS uses common scientific Python libraries and processes FITS data through routines that can be called interactively or scripted for batch reduction.Its core requirements are NumPy, SciPy, and PyFITS; Matplotlib is strongly recommended but not required.Source: §1 · p. 1
  • The pipeline uses abstractly described data formats where possible, and its authors note that it could potentially be adapted to future image-slicing integral field spectrographs.The paper presents the core routines, the wavelength solution, and a default reduction script and metadata format.Source: §1 · pp. 1–2

2 PyWiFeS Data Reduction Routines

PyWiFeS routines operate at the CCD-frame, slitlet, and global data-cube levels, using instrument characteristics and wavelength-solution information stored in metadata or derived separately.

2 PyWiFeS Data Reduction Routines

  • The pipeline includes routines that process raw CCD frames, individual slitlets, and global three-dimensional data cubes.The instrument has a 25′′×38′′ field of view divided into 25 slitlets, and the slitlet profiles are separated to support nod-and-shuffle observations.Source: §2 · p. 2
  • A Python dictionary in the WiFeS metadata file stores detector characteristics, default wavelength-solution line lists, and other information required by processing routines.The dictionary is packaged in Python pickle format.Source: §2 · p. 2
  • Some routines require an instrument wavelength solution, which is derived through the detailed procedure in Section 3.For the processing routines outlined here, the wavelength solution is treated as already derived.Source: §2 · p. 2

2.1 Image Pre-processing

Image pre-processing corrects detector effects before isolating the 25 slitlets for subsequent processing. It includes overscan and bad-pixel correction, bias reconstruction, and FITS organization of slitlet data.

2.1 Image Pre-processing

  • PyWiFeS subtracts the overscan and applies the readout gain to convert science pixels from ADUs to electrons, and interpolates over defective columns in second-generation detectors.The bad columns are 1 red and 2 blue out of more than 4000; they are also flagged in data-quality extensions.Source: §2.1 · p. 2
  • Bias levels in first-generation detectors vary spatially and temporally, limiting standard median-superbias subtraction; the second-generation detector bias behavior was not yet characterized.The first-generation bias levels varied by about 1 e− within the first hour and again at night’s end.Source: §2.1 · pp. 2, 4
  • A reconstructed superbias removes large-scale spatial structure, leaving residuals flat at a sub-noise level.The reconstruction addresses spatial and temporal bias variation; the default master is built from afternoon biases, while local reconstructed biases can use night-time frames.Source: §2.1 · pp. 3–4

2.1.1 Separating the Slitlets

  • Slitlet profiles are measured from normalized flat-lamp flux versus detector row, with edges defined by a 10% flux threshold and the center set midway between them.The slitlet is extracted as the 86 CCD rows centered on that midpoint.Source: §2.1.1 · p. 4
  • The slitlet routine writes pre-processed data, variance, and data-quality extensions for each slitlet into a multi-extension FITS file.For nod-and-shuffle observations, a variant saves nodded-sky slitlets to a second MEF file.Source: §2.1.1 · pp. 4–5

2.2 Cosmic Ray Rejection

PyWiFeS adapts Laplacian-kernel cosmic-ray rejection to WiFeS’s slanted wavelength solution and 25-slitlet data. It models and subtracts the sky before thresholding and replacing contaminated pixels.

2.2 Cosmic Ray Rejection

  • The cosmic-ray routine resamples data to a rectilinear wavelength grid, smooths along the cross-dispersion direction, then transforms the sky model back to the original sampling before subtraction.This three-step sky modeling accommodates the instrument’s slanted or curved wavelength solution.Source: §2.2 · p. 5
  • After sky subtraction, a Laplacian-kernel convolution and noise-based threshold identify cosmic-ray pixels, with a lower threshold applied to neighboring pixels.Contaminated pixels are replaced by interpolation from nearby cosmic-ray-free pixels.Source: §2.2 · p. 5
  • The procedure runs over all 25 slitlets and repeats for the requested iterations, defaulting to three; parallel processing is available with multithread=True.Source: §2.2 · p. 5

2.3 Flat-Fielding

PyWiFeS builds a master flat-field response from lamp and twilight-sky observations, then corrects scattered light and optionally addresses a residual background gradient. The scattered-light correction is most consequential where the lamp is faint, while the optional desag step has a restricted scope.

2.3.1 Constructing the Response Function

  • Lamp flats provide the spectral correction and twilight-sky flats the spatial illumination correction; their product forms the master response function.The internal lamp illumination differs slightly from the on-sky optical path, motivating the combined calibration.Source: §2.3.1 · pp. 5–6

2.3.2 Accounting for Scattered Light

  • PyWiFeS reconstructs the horseshoe reflection and diffuse glow from interslit regions, then subtracts the fitted contamination from master lamp and sky flats.The procedure smooths interslit regions, fits a bivariate spline to sampled counts, and subtracts the reconstruction.Source: §2.3.2 · p. 7
  • Corrections are typically only 1–2% where lamp flux is strong, but remove scattered reflection and glow to the noise level and are effective where the lamp is faint.The horseshoe reflection can dominate the faint-lamp region and affect flat-fielding below 4500 Å if left uncorrected.Source: §2.3.2 · pp. 7–8

2.3.3 Correcting for Anomalous “Sag” in Data

  • The optional desag correction addresses a residual spectral-direction background gradient, but currently applies only to point-and-stare observations, not nod-and-shuffle data.Preliminary tests suggest it is no longer required for data taken after the detector upgrades; the authors advise considering it only for observations before January 2013.Source: §2.3.3 · pp. 8–9

2.4 Data Cube Generation

PyWiFeS assigns spatial and wavelength coordinates to detector pixels, corrects wavelength-dependent atmospheric displacement, and resamples the measurements onto a regular data-cube grid. Wire frames establish spatial zeropoints, while edge resampling can leave slightly erroneous flux.

2.4 Data Cube Generation

  • The cube records flux versus wavelength at each sky position; regular-grid resampling makes the detector’s irregular spatial and wavelength sampling usable for extracting physical quantities.Instrument x and y coordinates do not correspond to the same sky coordinates at all wavelengths because of ADR.Source: §2.4 · p. 9

2.4.1 Spatial Zeropoint Derivation

  • A wire-frame observation measures the spatial zeropoint in each slitlet; PyWiFeS fits the wire shadow per column and linearly models its profile across detector columns.The lamp spectrum is modeled and subtracted to isolate the shadow, and the resulting wire solution is saved for data-cube generation.Source: §2.4.1 · p. 9

2.4.2 Correction for Atmospheric Differential Refraction

  • PyWiFeS applies Filippenko’s ADR equations during cube generation, using observation details from the FITS header; the corrected high-airmass example remedies the star-trace elongation.The uncorrected trace agrees with the predicted wavelength-dependent deflection.Source: §2.4.2 · p. 9, Figure 10 · p. 10

2.4.3 Coordinate Rectification

  • For each slitlet, PyWiFeS assigns true y from the wire profile and ADR deflection and λ from the wavelength solution, then resamples onto a rectilinear grid.The grid spans the central 38″ of each slitlet and, by default, the wavelength range common to all slitlets.Source: §2.4.3 · pp. 9–10
  • Because ADR correction across slitlets needs a second resampling along x, unsampled detector-edge flux is copied from observed edge data and remains slightly erroneous.The authors therefore note the value of aligning observations along the parallactic angle.Source: §2.4.3 · pp. 10–11

2.5 Flux Calibration

PyWiFeS derives an instrument sensitivity function from observed and reference standard-star spectra, then applies it to convert cube counts to flux. The procedure supports multiple standards, while residual-extinction fitting is not available in the current routine.

2.5 Flux Calibration

  • The three steps are extracting standard-star spectra, deriving the sensitivity function from observed/reference spectra, and applying the solution to data cubes.The routines are extract_wifes_stdstar, derive_wifes_calibration, and calibrate_wifes_cube, respectively.Source: §2.5 · p. 11
  • Standard-star extraction subtracts a sky spectrum estimated from spaxels outside the object aperture, then sums flux within the aperture.The aperture center is set by the flux centroid; default extraction and sky radii are 5″ and 10″.Source: §2.5 · p. 11
  • The sensitivity curve can combine one or several standards, with optional normalization across stars to improve signal-to-noise despite grey extinction variations.Strong telluric-absorption regions are excluded from the fit.Source: §2.5 · p. 11
  • The calibration routine applies the interpolated sensitivity solution to convert photon counts to Fλ and also applies the Siding Spring extinction correction.The sensitivity is derived from observed/reference flux ratios, corrected for smooth atmospheric extinction.Source: §2.5 · p. 11
  • Residual-extinction fitting is unavailable in the current flux-calibration routine, though users can use IRAF-style solutions and extinction curves.The paper notes that users needing stringent flux calibration and extinction corrections can use existing IRAF routines with appropriate formatting keywords.Source: §2.5 · pp. 11–12

2.6 Correction of Telluric Absorption

PyWiFeS estimates narrow telluric absorption from smooth-spectrum sources and applies correction functions for O2 and H2O. Spectra are normalized to airmass secz = 1.0 before averaging, and the correction can then be applied at the target airmass.

2.6 Correction of Telluric Absorption

  • Spectra from smooth-spectrum sources observed at similar airmass and time can correct tellurics, or several such objects over a range of airmass can estimate nightly average absorption.Source: §2.6 · p. 12
  • Telluric absorption is estimated by fitting a low-order polynomial to unaffected regions of object spectra and taking the observed-to-predicted smooth-flux ratio.The routines are derive_wifes_telluric and apply_wifes_telluric.Source: §2.6 · p. 12
  • Each spectrum’s absorption is corrected to airmass secz = 1.0, then averaged across objects to define the final telluric correction functions.The functions account for the airmass dependence of absorption.Source: §2.6 · p. 12
  • For one observing night in 2013, the fitted logarithmic power-law indices were 0.40 for O2 and 0.72 for H2O.Both values are intermediate between the optically thick value of 0 and optically thin limit of 1; indices can be set by keyword.Source: §2.6 · p. 13
  • The O2 and H2O correction functions can be applied directly to data with the appropriate airmass dependence.Figure 12 shows the functions scaled to secz = 1.0 and a smooth-spectrum supernova example before and after correction.Source: §2.6 · p. 13, Figure 12 · p. 13

3 WiFeS Wavelength Solution

PyWiFeS automates emission-line detection and association, then fits a global optical model to derive wavelength solutions across WiFeS. The solution achieves residual dispersions of about 0.05 Å for R=7000 gratings and 0.1 Å for R=3000 gratings, while temperature changes make timely arc exposures important.

3 WiFeS Wavelength Solution

  • The wavelength-solution task is to avoid manual line identification while modeling how wavelengths vary across each slitlet and how slitlet solutions relate through instrument optics.Manual identification in multiplexed IFU data is time-consuming and particularly prone to human error.Source: §3 · p. 13

3.1 Finding and Identifying Emission Lines

  • Potential line centers are selected row by row using background flux and derivative statistics, with default cuts at the mean plus 3σ.Adjacent qualifying pixels are grouped and their middle pixel is used as the candidate center.Source: §3.1 · pp. 13–14
  • The faster loggauss fit is intended for rapid solutions, while the slower mpfit Gaussian fit is the standard for science-grade reduction.Line references are associated using a two-sided nearest-neighbor requirement, with discrepant pairs rejected at a default 5 Å cut.Source: §3.1 · p. 14

3.2 Implementing the WiFeS Optical Model

  • The optical model captures wavelength curvature from off-axis geometry and slit-number shifts from the staircase slit arrangement.It has 43 parameters, including 25 small slitlet angular deviations that can shift wavelengths by about 0.05 Å per slitlet.Source: §3.2 · p. 15
  • 0.053 Å is the final residual dispersion for the R7000 Ne-Ar example; typical optical-model residuals are about 0.05 Å for R=7000 and 0.1 Å for R=3000 gratings.The 43 model parameters are fitted to minimize offsets between predicted and reference wavelengths; Figure 15 depicts the curved detector-plane solution and residuals.Source: §3.2 · p. 15, Figure 15 · p. 16
  • Many optical-model parameters are expected to remain constant over time, but their stability is still being assessed using large volumes of WiFeS data.Future PyWiFeS versions may incorporate the resulting knowledge of static parameters.Source: §3.2 · p. 15

3.3 Stability of the Wavelength Solution

  • Arc exposures taken far from an observation can yield wavelength-calibration errors of 1–2 Å, or typically 40–60 km/s, due to temperature drift.The paper recommends regular arcs; about one hour between them is described as reasonable, with cadence left to each observer’s science needs.Source: §3.3 · p. 16

4 Data Reduction Scripting and Metadata

PyWiFeS pairs abstract metadata with a configurable reduction script so recurring procedures can be applied across datasets without manually specifying file names at each step. Its default structure tracks shared and observation-specific calibrations, while users must ensure the metadata and observing modes remain compatible.

4 Data Reduction Scripting and Metadata

  • Abstract metadata lets the default script apply recurring reduction procedures to different datasets without users manually specifying file names at each step.The metadata supplies data context that the script interprets generically.Source: §4 · pp. 16–17, §4.1 · p. 17

4.1 Default PyWiFeS Metadata Structure

  • Master calibration files are stored in lists, while science and standard-star observation groups use dictionaries containing science frames and associated local calibrations.The sci, sky, and arc lists identify co-added science frames, sky frames, and local arcs; standard-star type lists designate flux and/or telluric use.Source: §4.1 · pp. 17–18, Figure 17 · p. 18
  • The automatic metadata-generation script uses the WiFeS controller’s IMAGETYP keyword and assigns calibrations to master lists by default, so user edits are needed for observation-specific calibrations.The metadata must be carefully prepared before each reduction.Source: §4.1 · p. 18

4.2 The PyWiFeS Data Reduction Script

  • The reduction script parses a configurable list of steps, each specifying its function, run flag, file suffix, and arguments.A loop dispatches master reduction functions using the metadata and suffixes, and those functions call the relevant PyWiFeS routines.Source: §4.2 · pp. 18–19
  • The default sequence covers overscan subtraction, bad-pixel repair, bias processing, frame preparation, wavelength and spatial solutions, cosmic-ray rejection, sky subtraction, co-addition, and cube generation.Calibration preparation and operations are marked CAL; the script also includes flatfield and standard-star flux and telluric calibration steps before final reformatting.Source: §4.2 · p. 19
  • Nod-and-shuffle frames are processed automatically, with nodded sky pixels subtracted from object pixels during the sky-subtraction step.The script identifies these frames from the WIFESOBS FITS header keyword.Source: §4.2 · p. 20
  • The scripts do not currently support multiple observing modes in one reduction, and incompatible calibrations can cause failures or produce incorrect results without an error.For example, the text notes that a calibration solution for the R3000 grating can be applied to R7000 data without necessarily triggering an error.Source: §4.2 · p. 20

5 Conclusion

PyWiFeS is a Python data-reduction pipeline for WiFeS that uses existing scientific Python modules and FITS files as input and output. Its scriptable, modular design supports repeat processing, while abstract instrument variables leave possible adaptation to other image-slicing spectrographs.

5 Conclusion

  • PyWiFeS is a Python pipeline using NumPy, SciPy, and PyFITS, with no function calls outside Python or requirement for IRAF or other astronomical processing environments.The pipeline is described as requiring no special system-dependent software installation.Source: §5 · p. 20
  • FITS input and output make processing modular, allowing users to adjust step order and repeat reductions after code upgrades or on new datasets.Abstract instrument variables also make adaptation to other image-slicing integral field spectrographs a possibility.Source: §5 · p. 20
  • The default metadata format separates assigning files to calibration or processing roles from the procedures that operate on them.Lists in the metadata dictionary identify shared calibrations and science frames with associated calibrations, without requiring procedure calls to be edited for new file names.Source: §5 · p. 20
  • The release includes standard integral-field reduction procedures and algorithms for scattered-light flatfield correction and an optics-based global wavelength solution; future improvements are planned for the RSAA Wiki.Further updates may reflect improved understanding of instrument behavior and data-reduction scripting.Source: §5 · pp. 20–21
Loading 1311.2666v1…