PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hyper Suprime-Cam Software Pipeline

James Bosch, Robert Armstrong, Steven Bickerton, Hisanori Furusawa, Hiroyuki Ikeda, Michitaro Koike, Robert Lupton, Sogo Mineo, Paul Price, Tadafumi Takata, Masayuki Tanaka, Naoki Yasuda, Yusra AlSayyad, Andrew C. Becker, William Coulton, Jean Coupon, Jose Garmilla, Song Huang, K. Simon Krughoff, Dustin Lang, Alexie Leauthaud, Kian-Tat Lim, Nate B. Lust, Lauren A. MacArthur, Rachel Mandelbaum, Hironao Miyatake, Satoshi Miyazaki, Ryoma Murata, Surhud More, Yuki Okura, Russell Owen, John D. Swinbank, Michael A. Strauss, Yoshihiko Yamada, Hitomi Yamanoi

arXiv:1705.06766v1astro-ph.IM

TL;DR

Wide-field HSC surveys require reliable processing because deep data contain severe blending and are sensitive to instrumental and measurement errors. This paper presents an LSST-derived, HSC-customized pipeline with large-scale processing and specialized algorithms, showing how HSC development both supports SSP data reduction and advances the LSST stack. The pipeline’s remaining limitations include calibration imperfections, poor behavior in some seeing conditions, and serious deblending failures in crowded regions.

  • Problem

    Deep wide-field imaging requires processing that controls blending, instrumental artifacts, and small systematic errors to produce scientifically useful catalogs.

  • Method

    The paper develops an HSC-customized LSST pipeline combining image calibration, coaddition, source measurement, PSF and shape algorithms, and deblending.

  • Results

    HSC processing served as an early prototype of LSST pipelines, revealing challenging algorithmic problems through use of pipeline outputs in research and reintegrating improvements into LSST.

  • Takeaways & Limitations

    The HSC Pipeline provides a recommended reduction system for HSC observations while supplying tested algorithms and experience to the LSST codebase.

  • Takeaways & Limitations

    The pipeline has unresolved PSF-modeling problems in good seeing, calibration imperfections from scattered light and changing dome-lamp responses, and severe deblending failures in crowded regions.

Abstract

from arXiv · show

In this paper, we describe the optical imaging data processing pipeline developed for the Subaru Telescope's Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey Telescope's Data Management system, adding customizations for HSC, large-scale processing capabilities, and novel algorithms that have since been reincorporated into the LSST codebase. While designed primarily to reduce HSC Subaru Strategic Program (SSP) data, it is also the recommended pipeline for reducing general-observer HSC data. The HSC pipeline includes high level processing steps that generate coadded images and science-ready catalogs as well as low-level detrending and image characterizations.

1 Introduction

Wide-field optical surveys enable broad science but require sophisticated processing to produce reliable catalogs as survey area and depth increase. The HSC Pipeline addresses this challenge through an LSST-based system customized for HSC data and large-scale processing.

  • Motivation: Wide-field CCD surveys support rare-object searches and precise measurements of noisy quantities through their large object samples.
  • Challenges: Processing large wide-field surveys is difficult because false detections, instrumental artifacts, and small photometric or shape errors can compromise scientific catalogs.The tolerance for systematic errors decreases as survey area and depth increase relative to the survey’s statistical precision.
  • Pipeline approach: HSC adopts a dedicated catalog-producing pipeline for both Subaru Strategic Program and general-use observations, customized from the LSST Data Management software stack.The software foundation is inherited through LSST from the SDSS Photo Pipeline.
  • Paper scope: The paper presents pipeline architecture, high-level processing stages, and detailed descriptions of novel or complex algorithms rather than installation instructions.
  • HSC data regime: HSC’s depth makes blending a central processing challenge because many objects have significantly overlapping isophotes and cannot simply be excluded from analyses.
  • Data organization: The pipeline processes overlapping visits together by building coadded images within overlapping tracts and patches, then resolves catalog duplicates.SSP productions use approximately square, overlapping tracts organized into constant-declination rings.

2 Software Architecture

The HSC Pipeline is an extensible LSST-derived architecture whose HSC-specific developments were reintegrated into the main LSST stack. It separates configurable high-level tasks from efficient C++ low-level processing and uses the Data Butler to manage data products.

  • Relationship to LSST: HSC-specific features introduced in the pipeline were fully reintegrated into the LSST stack during 2016, including raw HSC data processing.
  • Relationship to LSST: Future HSC-SSP data releases are produced with a minimally modified version of the main LSST stack.
  • Extensibility: Flexibility and extensibility are core goals: algorithms are configurable and replaceable, allowing HSC to commission LSST pipelines on similar data.
  • Implementation: High-level processing uses nested Python Tasks and configuration trees, while lower-level pixel processing and computationally intensive operations use C++.Task subtasks can be retargeted through configuration without changing code, and specialized algorithms can be registered through Python imports.
  • Data management: The Data Butler organizes pipeline inputs, outputs, and intermediates by dataset names and data IDs while managing their file formats and paths.Repositories preserve common pipeline versions and configurations and can be chained between processing stages.

3 Pipelines

The HSC Pipeline uses four stages to calibrate CCD images, jointly refine astrometry and photometry, create coadds, and measure coadd sources. It processes missing or suspect data while recording reliability information through masks and catalog flags.

  • Pipeline stages: The pipeline has four stages: CCD Processing, Joint Calibration, Image Coaddition, and Coadd Processing.These stages detect and calibrate CCD sources, refine calibrations across visits, combine images, and measure coadd objects.
  • Quality handling: The pipeline preserves potentially unreliable results using integer mask planes and corresponding catalog flags rather than excluding all problematic data.Flags identify conditions including sensor defects, saturated or interpolated pixels, rejected inputs, cosmic rays, edge truncation, and nearby bright objects.
  • CCD Processing: CCD Processing independently processes each CCD to produce calibrated images and source catalogs through a semi-iterative sequence of detrending, detection, calibration, and measurement.The calexp product contains a detrended, background-subtracted image, mask image, and per-pixel variance estimate.
  • Joint Calibration: Joint Calibration uses repeated observations of sources across visits to strengthen astrometric and photometric constraints beyond those available from the external reference catalog.The astrometric model includes focal-plane distortion and CCD-specific translation and rotation terms, while photometric fitting uses visit, focal-plane, and CCD corrections.
  • Image and PSF coaddition: Safe clipping improves artifact removal while preserving the PSF, but PSF inaccuracies remain possible where input coverage changes discontinuously.A current bug prevents some affected objects from receiving the relevant flag, affecting approximately 18% of objects in the HSC-SSP Wide layer.
  • Coadd Processing: Coadd Processing alternates per-band and all-band operations to produce consistent cross-band object lists and independent measurements in every band.The deepCoadd meas catalog includes measurements even in bands where an object was not detected, which can produce poor results for flexible algorithms.

4 Algorithms

The algorithms section describes low-level instrumental corrections used to transform raw HSC data into calibrated CCD images. ISR applies ordered corrections, while remaining flat-field shortcomings are addressed by Joint Calibration.

  • Instrument Signature Removal: Instrument Signature Removal assembles amplifier-level data into CCD images and applies basic detrending plus nonlinearity and brighter-fatter corrections.The HSC implementation uses the ip_isr package with multiple processing elements applied in a specified order.
  • Instrument Signature Removal: ISR detects saturation, removes overscan, and applies flat correction using dome flats normalized to a camera-wide mean.Overscan removal fits a 30th-order Akima spline to overclock pixels along rows before subtracting it from science pixels.
  • Instrument Signature Removal: ISR alone cannot make counts perfectly proportional to sky flux because scattered light and time-varying dome-lamp responses distort the flat-field pattern.The pipeline addresses these shortcomings in the Joint Calibration component.

4.2 Brighter-Fatter Correction

The brighter-fatter correction models charge-induced flux redistribution using a potential inferred from flat-field noise correlations, reducing magnitude-dependent PSF size and ellipticity biases.

  • The brighter-fatter effect makes brighter stars larger and more elliptical because accumulated charge generates lateral electric fields.
  • The correction reapportions flux under the assumption that the charge-displacement electric field has zero curl and derives a potential from flat-field noise correlations.
  • The correction reduces the large magnitude-dependent bias in measured PSF size and ellipticity.An overall magnitude-independent bias remains uncorrected.

4.3 PSF Modeling

The pipeline selects PSF stars by clustering object sizes, models each CCD’s PSF with a polynomial basis, and evaluates residuals across visits, coadds, and focal-plane position.

  • PSF modeling: PSFEx is modified to fit a position-dependent polynomial PSF model independently on each CCD while separating star selection from PSF modeling.The implementation limits the fit to a polynomial PSF as a function of position on one CCD.
  • Star selection: A four-cluster k-means analysis of object sizes isolates stellar candidates after restricting sources to brighter than 12500 counts.
  • Star selection: Approximately 72 stars are selected per CCD on average for PSF modeling, with 20% of objects reserved for cross-validation.
  • PSF modeling: The model uses 41×41-pixel star stamps, a 12-pixel-radius aperture for flux normalization, and a second-order polynomial to interpolate kernel pixels between stars.
  • Quality assessment: PSF quality is assessed through Gaussian-moment size and ellipticity residuals and the correlation function of PSF ellipticity residuals.The same tests are applied to coadds using coadded PSF models and stars selected in at least 20% of individual visits.
  • Quality assessment: The PSF model shows unmodeled outer-focal-plane structure, reserved stars perform worse than fitted stars, and coadd PSFs perform slightly worse than per-visit PSFs.The polynomial order remained two because higher order could require too many stars in some visits.
  • Limitations: Good-seeing visits produce substantially worse PSF modeling, leading the cited data releases to reject problematic visits when building coadds.The remaining cause was not identified, and the authors associate some issues with PSFEx.

4.4 Cosmic Ray Detection

HSC cosmic-ray detection combines morphology-aware pixel tests with iterative event expansion, then interpolates contaminated pixels and records their causes in image masks.

  • Cosmic-ray morphology: HSC cosmic rays primarily appear as long, branched electron tracks or straight, wedge-like muon tracks in its thick CCDs.
  • Pixel detection: The first pass requires candidate pixels to avoid saturation, exceed local background by 6σ, and contain a peak sharper than the stellar PSF.The sharpness criterion uses neighboring-pixel gradients and PSF information.
  • Visualization: Figure 11 contrasts original images, identified cosmic-ray pixels, and Linear Predictive Code interpolation results for electron and muon events.
  • Pixel detection: Conditions are tested across four neighbor pairs; candidates must exceed the background threshold for all pairs and the sharpness condition for at least one.
  • Pixel repair: Detected cosmic-ray pixels are replaced by interpolation based on the neighbor pair triggering the sharpness condition, helping reveal additional pixels from the same hit.
  • Event expansion: Contiguous cosmic-ray events must contain more than 150e−, and adjacent contaminated pixels are searched through three additional iterations.

4.5 Bad Pixel Interpolation

The interpolation method treats bad pixels as missing samples in a stationary image process and estimates them with Linear Predictive Codes from surrounding signal statistics.

  • Method: Linear Predictive Codes estimate an image signal value from other data points and the signal’s statistical properties.
  • Image model: For point-source images, the method approximates the autocorrelation as a Gaussian derived from the squared PSF.
  • Interpolation: Bad pixels are interpolated by assigning infinitely larger noise to bad columns than to good columns in the prediction.
  • Interpolation: With five terms and α = 1, the one-dimensional interpolation weights are {−0.274, 0.774, 0.000, 0.774, −0.274}.The bad pixel receives zero weight.
  • Quality flags: The image mask records both that a pixel was interpolated and the reason, while objects with interpolated centers are considered unreliable.

4.6 Background Subtraction

The pipeline models and subtracts smoothly varying background flux using spatial binning and polynomial fitting, while iterating background estimation with source detection. Chebyshev fitting better avoids over-subtraction than spline interpolation with large bins.

  • Background subtraction models sky, optical ghosts, scattered light, and sub-threshold sources, but can also remove astrophysical object flux.
  • The algorithm averages object-masked pixels in 128×128 bins with iterative 3σ clipping, computes bin variances, and fits a 6th-order 2-d Chebyshev polynomial using inverse-variance weights.
  • The fitted Chebyshev polynomial is subtracted from the original image, with polynomial order controlling the spatial scale of included background features.
  • Chebyshev fitting better avoids over-subtraction than large-scale spline interpolation, because it is less sensitive to bin-boundary placement relative to bright objects.
  • Background subtraction and source detection are coupled because object masking depends on a threshold defined relative to the background, so the pipeline re-estimates background as detection limits become fainter.

4.7 Detection

The detection algorithm smooths images and thresholds the result to form above-threshold regions, using PSF-based matched filtering for isolated sources. Its current coadd handling is suboptimal, and it lacks a dedicated extended-source filter.

  • Sources are detected by convolving images with a smoothing filter and thresholding the result, with even a single above-threshold pixel counted as a detection.
  • A PSF-transpose smoothing filter is formally optimal for isolated point sources when sky-dominated noise and a known background apply.
  • The pipeline uses a circular Gaussian matched to the PSF RMS width instead of the full PSF for performance, with usually less than 1% S/N loss in the HSC-SSP Wide layer.
  • 1.7% ± 1.5% is the reported detection S/N loss from applying the approach to non-optimal coadds, rather than smoothing each image before combining them.
  • The algorithm produces Footprints containing above-threshold regions and lists of peaks within overlapping detections, which the deblender treats as candidate objects.
  • The point-source likelihood formulation yields maximum-likelihood flux, variance, and signal-to-noise estimates from PSF-filtered image data.
  • Optimal coaddition for detection can be obtained by PSF-smoothing per-epoch images before averaging, without the deconvolution required for general-purpose optimal coadds.
  • Spurious peaks near bright objects can steal neighboring flux during deblending and increase catastrophic failures, motivating their removal before deblending.

4.8 Deblending

The deblender assigns flux within multi-peak Footprints using approximately 180-degree rotationally symmetric templates and flux-conserving relative template contributions. It works adequately for modest blends but fails in severe, crowded configurations.

  • In the SSP Wide, Deep, and UltraDeep layers, 58%, 66%, and 74% of objects, respectively, are members of blends.
  • Deblending creates a parent record and a child image for each peak, while HeavyFootprints store irregularly bounded child-pixel arrays for later measurement.
  • The current pipeline processes each filter independently, enforcing cross-band consistency only through shared input peaks and Footprints.
  • Separating detection, deblending, and measurement is a poor choice in crowded stellar fields, where these operations should be iterated through simultaneous point-source modeling and subtraction.
  • The deblender assumes approximate 180-degree rotational symmetry about each peak, recovering morphology confused by neighbors from the opposite side of the peak.
  • The reflected pixel pair uses the lower value because it is least affected by nonnegative neighbor flux when the symmetry assumption holds.
  • The algorithm fits a linear combination of templates to blend pixels, obtaining a scale factor α_i for each peak.
  • Final child images use relative scaled-template contributions rather than scaled templates alone, preserving information from the observed blend.

4.9 Source Measurement

Source measurement characterizes astronomical sources through extensible algorithms for centroids, shapes, and fluxes, including forced photometry for cross-band consistency. The section also documents performance, blending sensitivity, and important failure modes across these measurements.

  • Measurement system: Source measurement uses configurable plug-ins to characterize individual sources, primarily through centroid, shape, and flux measurements.The system runs in both CCD and coadd processing and supports externally implemented plug-ins.
  • Measurement system: Forced photometry fixes positions and shapes from a reference image to measure consistent fluxes across bands.This is the final stage of coadd processing and uses reference-band positions and shapes.
  • Shapes: Adaptive Gaussian moments iteratively match the weight-function moments C to corrected measured moments Q, yielding Q = 2M when the procedure converges.The method can provide the highest S/N for a Gaussian weight function, but convergence is not guaranteed for low-surface-brightness objects.
  • Photometry: PSF-matched images broaden the PSF and increase blending, making very small apertures the only measurements approximately unaffected by blending.Larger fixed-aperture and Kron measurements remain useful mainly for unblended objects or cases where one object's flux dominates.
  • Photometry: PSF-matched small-aperture afterburner photometry produces lower-scatter cluster red-sequence colors and appears to improve photometric redshifts relative to CModel colors.The authors leave open whether this reflects deblender sensitivity, model complexity, or differences between galaxy cores and outskirts.
  • Classification and performance: CModel performs well for simulated galaxies down to i ∼25, while extendedness appears effective down to i ∼24 with a catalog cut of mpsf − mcmodel = 0.0164.Users may choose a different extendedness cut for their science.

5 Future Work and Conclusions

The HSC Pipeline remains broadly capable but is still limited by challenges from deep ground-based imaging, especially blending and PSF modeling. Ongoing development targets these weaknesses while feeding improvements back into LSST.

  • Current status: The pipeline is a work in progress that has enabled broad scientific discoveries but has sometimes limited use of HSC and Subaru’s high-quality data.The paper links these limitations to algorithmic challenges in extremely deep, ground-based imaging and excellent seeing.
  • Future challenges: Deblending is the biggest algorithmic challenge because the production algorithm is inadequate for severe blending in deeper ground-based data.A new algorithm is in development to support more varied experiments with deblending.
  • Future challenges: Current PSFEx models are not accurate enough for weak lensing in the best-seeing images, motivating physically based PSF modeling.The planned approach separates optical and atmospheric PSF components rather than relying solely on empirical pixel-basis models.
  • Implications for LSST: LSST will face even more severe blending and tighter PSF-accuracy requirements, making HSC a pathfinder for identifying these algorithmic problems.The HSC-SSP exposes challenges through scientists’ use of pipeline outputs in research rather than through data challenges alone.
  • Future development: Improvements underway include background subtraction, artifact masking, and galaxy photometry, with PDR1 bugs prioritized for correction and LSST integration.The pipeline is expected to continue advancing image-processing algorithms as the SSP survey progresses.

Appendix 1 Joint Calibration Mathematical Details

The joint calibration pipeline fits astrometric and photometric models independently for each tract by iteratively solving linearized systems. It uses CCD, exposure, source, and polynomial parameters to relate measurements to calibrated positions and magnitudes.

  • Astrometric calibration: The astrometric model composes focal-plane, CCD, and gnomonic transformations, with polynomial coefficients mapping detector coordinates to projected sky coordinates.Reference-matched source positions are fixed, while unmatched sources contribute fitted α_s and δ_s parameters.
  • Astrometric calibration: Astrometric residuals are combined into a χ2 objective and minimized by linearizing them in calibration, detector, and unmatched-source parameters.The fit assumes all centroid uncertainties are identical.
  • Solution procedure: The pipeline initializes free parameters from the single-visit CCD Processing astrometric fit before iteratively solving the joint calibration equations.Polynomial-vector ordering is arbitrary provided it remains consistent across the block-matrix quantities.
  • Photometric calibration: Photometric calibration solves for spatial polynomial coefficients, per-exposure magnitude offsets, and per-source true magnitudes, with matched reference magnitudes held fixed.The photometric fit iteratively solves a linearized difference between true and measured magnitudes.
  • Computation: Both calibration fits are performed independently by tract using only overlapping CCD inputs, and CCDs spanning multiple tracts are fit repeatedly.In the Wide layer, each iteration factorizes approximately 6000×6000 astrometric and 2500×2500 photometric matrices.

Appendix 2 CModel Algorithm Details

CModel fits PSF-convolved exponential and de Vaucouleurs galaxy models using flexible ellipse parameters, multi-Gaussian profiles, and Bayesian priors. A PDR1 prior-weighting bug imposed strong radius and ellipticity biases with mixed effects on measurements.

  • Model profiles: CModel fits elliptical exponential and de Vaucouleurs models whose profiles approximate modified analytic forms with sums of Gaussians.The approximations use six-Gaussian and eight-Gaussian profiles matched to SDSS fitting profiles with softened cores and truncated wings.
  • Model profiles: Multi-Gaussian profiles enable fast convolution when the PSF is represented by a compatible analytic function.The pipeline instead models the PSF with two differently scaled Gauss-Hermite expansions for greater flexibility.
  • Bayesian fitting: The fitting objective combines likelihood with Bayesian priors on ellipse parameters, using a piecewise prior on ln r_tr and a flat prior on flux.The flux prior is not conditioned on flux because it could disrupt the expected linear response of flux measurements to true brightness.
  • PDR1 limitation: A PDR1 bug made the effective prior much stronger than intended, sharply cutting off radii near 0.37′′ and biasing ellipticities toward more circular values.The radius bias produced an artificial pile-up near m_psf − m_cmodel ∼0.6, while its effect on flux was small for ellipticity bias.
  • PDR1 limitation: The biased prior can suppress spuriously large CModel radii from minor deblend failures, although the bug still substantially affects fluxes and produces artificial radius structure.Because all bands are approximately equally affected, CModel colors are less strongly biased than fluxes.

Appendix 3 Ellipse Parameterizations

The paper relates several ellipse parameterizations used across HSC algorithms, emphasizing unconstrained fitting with trace-based radii and generating transforms. These representations connect intuitive axes and angles to matrix and complex-ellipticity forms.

  • Parameterization overview: HSC algorithms use multiple ellipse parameterizations, including second moments, Kron photometry, and CModel representations.The appendix establishes relationships and conventions among these forms for consistent use.
  • Geometric representation: The intuitive {A,B,θ} representation uses semi-major radius, semi-minor axis, and position angle, with a generating transform S mapping the unit circle to the ellipse.The transform is not unique because it may be multiplied on the right by any orthogonal matrix.
  • Moment representation: Moment-based algorithms represent an ellipse with the three unique elements of a symmetric positive definite matrix Q, whose eigenvalues are A2 and B2.The rotation matrix columns provide the eigenvectors associated with the ellipse orientation.
  • Complex ellipticity: Complex ellipticity encodes axis-ratio information through magnitude and orientation through phase 2θ, supporting weak-lensing descriptions.The appendix discusses conformal shear, δ, and g as alternative ellipticity definitions with different parameter domains.
  • Radius choices: The trace radius r_tr is preferred for fitting because it remains finite when the semi-minor axis approaches zero, unlike the determinant radius.This behavior is useful for noisy, unresolved objects in CModel fitting.

Appendix 4 Correcting for Bias in Absolute Blendedness

The appendix develops a noise-robust correction for absolute blendedness by integrating a bias-adjusted absolute image, then notes that an earlier incorrect formula mainly remapped low-S/N values without expected major scientific impact.

  • Noise bias and correction: Absolute blendedness uses the absolute image rather than the image itself to remain well behaved in the presence of noise.This has little effect for objects well above the noise but biases blendedness near the noise level.
  • Noise bias and correction: The bias correction starts from Gaussian photon counts with mean µ and dispersion σ, for which the total integrated flux is defined over an area A.The derivation assumes a uniform object profile before generalizing to nonuniform profiles.
  • Bias correction: Integrating |z| is biased, so the method instead integrates |z| − d, with d generalized to d(z) for nonuniform object profiles.Estimating d(z) requires estimating µ from the photon count z(r) and dispersion σ under a positive expected-count condition.
  • Implementation caveat: The PDR1 implementation used an incorrect blendedness-bias formula that primarily remapped low-S/N values rather than preserving the intended correction exactly.Because users calibrated blendedness directly with simulations, the authors do not expect this bug to significantly affect science analyses.
Loading 1705.06766v1…