Source-linked AI summary
Second Data Release of the Hyper Suprime-Cam Subaru Strategic Program
Hiroaki Aihara, Yusra AlSayyad, Makoto Ando, Robert Armstrong, James Bosch, Eiichi Egami, Hisanori Furusawa, Junko Furusawa, Andy Goulding, Yuichi Harikane, Chiaki Hikage, Paul T. P. Ho, Bau-Ching Hsieh, Song Huang, Hiroyuki Ikeda, Masatoshi Imanishi, Kei Ito, Ikuru Iwata, Anton T. Jaelani, Ryota Kakuma, Kojiro Kawana, Satoshi Kikuta, Umi Kobayashi, Michitaro Koike, Yutaka Komiyama, Xiangchong Li, Yongming Liang, Yen-Ting Lin, Wentao Luo, Robert Lupton, Nate B. Lust, Lauren A. MacArthur, Yoshiki Matsuoka, Sogo Mineo, Hironao Miyatake, Satoshi Miyazaki, Surhud More, Ryoma Murata, Shigeru V. Namiki, Atsushi J. Nishizawa, Masamune Oguri, Nobuhiro Okabe, Sakurako Okamoto, Yuki Okura, Yoshiaki Ono, Masato Onodera, Masafusa Onoue, Ken Osato, Masami Ouchi, Takatoshi Shibuya, Michael A. Strauss, Naoshi Sugiyama, Yasushi Suto, Masahiro Takada, Yuhei Takagi, Tadafumi Takata, Satoshi Takita, Masayuki Tanaka, Tsuyoshi Terai, Yoshiki Toba, Hisakazu Uchiyama, Yousuke Utsumi, Shiang-Yu Wang, Wenting Wang, Yoshihiko Yamada
TL;DR
The paper addresses the need for a large, deep, high-quality optical survey by presenting HSC-SSP’s second public data release. It combines expanded survey data with improved processing and newly public shape measurements, while documenting known limitations that users must consider.
Problem
Large imaging surveys are needed to characterize celestial objects and support high-precision cosmology, motivating HSC-SSP’s broad, deep, multi-filter survey.
Method
PDR2 processes expanded HSC-SSP observations into imaging, catalogs, database products, and calibrated galaxy shape measurements using an updated pipeline.
Results
PDR2 expands area and depth, improves overall data quality, and makes calibrated galaxy shape measurements publicly available.
Takeaways & Limitations
PDR2 provides a broader resource for survey science and weak-lensing analyses through its imaging, catalogs, database access, and shape catalog.
Takeaways & Limitations
Users should review known issues because deblending failures persist, low-level sky residuals may affect extended low-surface-brightness galaxies, and zero-point values vary spatially.
Abstract
from arXiv · showhide
This paper presents the second data release of the Hyper Suprime-Cam Subaru Strategic Program, a wide-field optical imaging survey on the 8.2 meter Subaru Telescope. The release includes data from 174 nights of observation through January 2018. The Wide layer data cover about 300 deg^2 in all five broadband filters (grizy) to the nominal survey exposure (10min in gr and 20min in izy). Partially observed areas are also included in the release; about 1100 deg^2 is observed in at least one filter and one exposure. The median seeing in the i-band is 0.6 arcsec, demonstrating the superb image quality of the survey. The Deep (26 deg^2) and UltraDeep (4 deg^2) data are jointly processed and the UltraDeep-COSMOS field reaches an unprecedented depth of i~28 at 5 sigma for point sources. In addition to the broad-bands, narrow-band data are also available in the Deep and UltraDeep fields. This release includes a major update to the processing pipeline, including improved sky subtraction, PSF modeling, object detection, and artifact rejection. The overall data quality has been improved, but this release is not without problems; there is a persistent deblender problem as well as new issues with masks around bright stars. The user is encouraged to review the issue list before utilizing the data for scientific explorations. All the image products as well as catalog products are available for download. The catalogs are also loaded to a database, which provides an easy interface for users to retrieve data for objects of interest. In addition to these main data products, detailed galaxy shape measurements withheld from the Public Data Release 1 (PDR1) are now available to the community. The shape catalog is drawn from the S16A internal release, which has a larger area than PDR1 (160 deg^2). All products are available at the data release site, https://hsc-release.mtk.nao.ac.jp/.
1 Introduction
HSC-SSP is an ambitious wide-field optical survey designed to cover 1400 deg2 in multiple filters under excellent seeing and unprecedented depths. PDR2 expands and improves on PDR1 while adding publicly available galaxy shape measurements for weak-lensing analyses.
- Survey context: 1400 deg2 is the planned HSC-SSP coverage under excellent seeing in multiple filters and unprecedented depths.The survey uses HSC, a 1.7-degree-diameter optical imager on the 8.2m Subaru Telescope.
- Survey design: The survey comprises Wide, Deep, and UltraDeep layers, with the Wide layer targeting 1400 deg2 in five broad-band filters to about 26th magnitude.The Deep layer consists of four fields totaling about 26 deg2.
- PDR2 contribution: PDR2 is a major data-release update in area, depth, and processing quality, and is a superset of PDR1 in all aspects.The release also includes carefully calibrated galaxy shape measurements needed for weak-lensing analyses.
- Paper scope: The paper describes PDR2 data products, processing-pipeline improvements, quality-assurance tests, known issues, data-access tools, and future-release plans.It also updates information on collaborating surveys.
2 Overview of the Release
PDR2 substantially expands HSC-SSP coverage and products while improving image processing and preserving excellent seeing. The release also includes weak-lensing shape measurements, but users must account for known limitations and spatially varying data quality.
- Release scale: 174 nights of observations through January 2018 support this release, compared with 61.5 allocated nights in PDR1.Some previously disjoint fields are now connected as the survey has progressed.
- Release scale: 300 square degrees reach full-color full-depth Wide coverage, up from about 100 square degrees in PDR1.PDR2 also includes partially observed Wide regions not covered in all five filters or at full depth.
- Data products: Deep and UltraDeep COSMOS and SXDS data are jointly processed into coadded images, multiband catalogs, and database tables.The release includes useful global statistics for exposure time, seeing, limiting magnitudes, and saturation magnitudes.
- Known issues: Known issues include persistent deblending failures in crowded areas, possible low-level sky residuals near 29 mag/arcsec2, and spatial variation across the survey.Users are directed to review the maintained issue list before using the data scientifically.
- Survey progress: 80% is the approximate overall Wide-survey progress rate relative to the expected speed, with filter completion rates of 88%, 97%, 67%, 81%, and 80% for g, r, i, z, and y.The i band is slowest because weak-lensing analysis imposes a stringent seeing constraint of about 0.75 arcsec.
- Data quality: 0.6 arcsec is the median i-band seeing, demonstrating excellent image quality for the release.Seeing distributions are shown for individual visits in each filter; the processing threshold is 1.3 arcsec.
- Weak-lensing products: 136.9 deg2 of calibrated galaxy shape measurements from the S16A internal release are publicly available for weak-lensing analyses.The catalog is split across six fields and has quality-assurance cuts applied; some shape measurements in PDR2 remain withheld pending validation.
3 Hardware Updates
The release addresses spatially varying filter responses and scattered light affecting HSC imaging, while documenting the pipeline’s deblending role and software correction for pre-fix data.
- Filter updates: Radial filter-curve variations cause structured sky backgrounds and position-dependent photometry in the r and i bands.Night-sky emission lines move in and out of the bandpass as the cutoff wavelength changes with radius.
- Scattered-light hardware: Scattered-light arcs in y-band images arose from rotator-encoder LEDs reflecting off the Wide-Field Corrector lens barrel.The arcs crossed the field of view and affected y-band and some narrow-band data near 0.9–1.0µm.
- Scattered-light hardware: Screens installed on November 13, 2017 blocked the scattered-light path, but all earlier data remained affected.Software was developed to subtract the scattered light from pre-fix observations.
- Pipeline context: The processing pipeline deblends overlapping sources into deblended images called heavyFootprint.
4 Pipeline Updates
PDR2 updates hscPipe from v4 to v6 with field-wide sky subtraction, dynamic detection, and improved image processing, while preserving extended galaxy structure more effectively.
- Pipeline version: PDR2 uses hscPipe v6, replacing the hscPipe v4 processing used for PDR1.
- Global sky subtraction: The new background-subtraction algorithm models the entire focal plane to reduce sky discontinuities between neighboring CCDs.It uses 1024 × 1024-pixel superpixels, approximately 2′.8 × 2′.8.
- Global sky subtraction: The algorithm combines a large-scale empirical background model with a filter-specific sky frame built from many dithered observations.The sky frame subtracts static features smaller than the empirical background model.
- Global sky subtraction: Compared with PDR1, the new sky subtraction preserves extended wings around bright objects and improves nearby-galaxy imaging.Masks around bright stars were not yet revised and were scheduled for a future incremental release.
- Dynamic object detection: 5σ static detection thresholds missed visible sources because convolution moved variance into untracked covariance.
- Dynamic object detection: The revised detector calibrates its threshold using PSF fluxes measured on empty sky, while remaining effectively at 5σ with few fake detections.Figure 6 shows many faint sources recovered by the revised algorithm, especially relevant to UltraDeep data.
4.3 Artifact Rejection
The updated artifact-rejection system identifies transient features through PSF-matched temporal comparisons before coaddition, improving performance in Wide data but remaining less efficient for smaller-dither Deep fields.
- Algorithm: The new algorithm uses time-series PSF-matched warped images to identify optical ghosts, satellite trails, and cosmic rays before coaddition.
- Algorithm: PSF-matched warps are stacked into a 2σ-clipped static-sky model, which is subtracted from each warp to locate deviations exceeding 5σ.
- Temporal classification: Transient versus persistent classification depends on how many visits contain each artifact, using a piecewise threshold in N.For N > 5, the threshold is 2 + 0.03N.
- Performance and limitations: Tests on several PDR1 tracts found better false-positive and false-negative performance than the earlier clipping algorithm.
- Performance and limitations: Known failures occur when artifacts persist across exposures, including under-dithered overlaps with static defects or compact sources.
- Performance and limitations: The rejection algorithm performs well in the Wide layer but is less efficient in Deep and UltraDeep data because their dithers are smaller.
4.4 Scattered Light in the y-band
PDR2 removes rotator-encoder scattered light from y-band data through an empirical, angle-dependent correction, substantially cleaning affected coadds while retaining a post-fix processing caveat.
- Correction method: Pre-fix y-band scattered-light patterns vary complicatedly with rotator angle, so the correction uses empirically measured dark exposures.The calibration sequence moves the rotator from −180 to +180 degrees in 0.5-degree steps.
- Correction method: The procedure interpolates compressed calibration exposures over rotation angle and integrates them across each science exposure.It computes the rotator angles at the exposure start and end before generating the expected CCD illumination pattern.
- Results and caveat: After subtraction, the hash-like pattern disappears and y-band coadds are much cleaner than those in PDR1.
- Results and caveat: The correction was erroneously applied to data taken after the hardware fix.
4.5 Effective Transmission Curve
PDR2 adds wavelength- and position-dependent photometric transmission information to image products, while improving sky subtraction and PSF modeling. The transmission data are available for scientific use, but are not yet incorporated into the pipeline’s own calibrations.
- Image products now report photometric transmission as a function of wavelength within each band and position on the image.
- Single-epoch transmission curves combine detector, optics, atmosphere, and radially varying filter responses across the focal plane.The radial dependence is especially important for the original r and i filters.
- Coadd transmission curves average per-epoch curves using the same weights used to construct each coadd.This preserves discontinuous spatial structure but requires a complex internal data structure.
- Transmission information is available for scientific applications but is not yet used in the pipeline’s own calibrations because robust sub-band SED inference tools are unavailable.Users can extract the information with hscPipe or a compatible LSST-stack version.
- The revised PSFEx resampling approach substantially reduces fractional PSF-size residuals, enabling processing of HSC SSP data under the best observing conditions.The original residual error increases to 0.4%, while the revised approach is described as sufficiently accurate for all HSC SSP data.
- Image coaddition still makes the coadd PSF model about 0.4% larger than the observed PSF.
4.7 Lossless Image Compression
PDR2 uses lossless tiled image compression to reduce the storage burden of growing HSC data products. The release also revises photometric calibration inputs and star selection to improve robustness against spatial metallicity variations.
- Pipeline images are stored with lossless FITS tiled compression using the GZIP_2 algorithm.Compression is applied to image, mask, and variance planes and can be reversed with funpack or read through hscPipe.
- PDR2 updates photometric color terms using the Pickles atlas and filter transmissions weighted by focal-plane surface area.This better represents radial transmission variation in the r and i bands.
- Color cuts exclude late-type stars from zero-point calibration because metallicity produces substantial intrinsic color variation and spatial color offsets.The cuts reduce the calibration sample but are intended to improve robustness against Milky Way stellar-population gradients.
- In COSMOS, about 40% of bright calibration-suitable stars pass the cuts, leaving about 20 stars per CCD for zero-point calibration.This is described as more than adequate for the calibration purpose.
- Comparisons with a previous internal release show no major improvement in zero-point uniformity, although the revised procedure should be more robust to Galactic metallicity variation.
4.9 Additional Mask Planes for Coadds
PDR2 adds image- and catalog-level masks for coadd regions where PSF models are ill-defined or input pixels were affected by edges, defects, or artifacts. Users are advised to test whether these flagged effects matter for their analyses.
- Coadd PSF modeling can fail to define a single PSF when different exposures contribute to different parts of an object.
- New image mask planes and catalog flags identify objects and pixels affected by ill-defined coadd PSFs.The flags distinguish affected detections and object centers.
- The SENSOR_EDGE flag marks pixels near an input-image boundary, while REJECTED marks masked input pixels that could not be interpolated.Rejected pixels commonly arise from bad amplifiers or other known sensor defects.
- The CLIPPED mask identifies coadd pixels whose contributing inputs were flagged as artifacts by image differencing.
- For many science cases, flagged PSF inaccuracies are negligible because input observations are often sufficiently similar.Users are encouraged to compare analyses with and without filtering on these flags.
5 Data
PDR2 largely retains the prior processing flow while expanding the released data and incorporating revised calibration, sky-subtraction, coaddition, and value-added products. It provides calibrated images, catalogs, coadds, and additional data products, but some measurements remain withheld or are planned for future release.
- 5 Data: PDR2 mostly follows PDR1 processing while emphasizing differences in screening and newly added pipeline features.
- 5 Data: Raw CCDs are screened using sky brightness, seeing, and transparency criteria, with relaxed seeing limits for y- and NB921-band fringe processing.
- 5.2 Data Processing: The processing generates calibrated CCD images, warped images, coadds, and associated catalog files, followed by multi-visit calibration, coaddition, and merged multi-band detection catalogs.
- 5.3 Image and Catalog Data: All major image and catalog products are available from the data-release website, with lossless compression applied to image files.
- 5.3 Image and Catalog Data: PDR2 releases previously withheld flat files and PDR1 galaxy shape measurements, while excluding shape measurements and deblended images from the new flat files.Useful newly accessible files include deblended-image measurement files and single-epoch source catalogs.
- 5.4 Value-added Products: Value-added products include COSMOS Wide-depth stacks constructed from UltraDeep-COSMOS visits under three target seeing conditions.
- 5.4 Value-added Products: Computed photometric redshifts are not included in this release and are planned for a future incremental release.
6 Data Quality and Known Issues
PDR2 shows strong photometric and astrometric quality, but residual PSF-model errors and several known issues constrain some scientific uses. Deblending failures, bright-star mask problems, and calibration offsets remain important caveats, especially in deep crowded fields.
- Photometry: Internal Consistency: σ ∼0.01 mag scatter between Kron and PSF photometry demonstrates good internal photometric consistency across the field.The y-band is noisier at σ ∼0.015, while CModel–PSF consistency reaches σ <∼0.002 mag.
- Photometry: External Consistency: ∼0.01 mag scatter in broad-band HSC–PS1 comparisons indicates good external photometric consistency, while y-band scatter is ∼0.02 mag.NB387 is much worse at ∼0.2 mag because stellar-metallicity-sensitive color terms cannot be fully captured from PS1 photometry.
- Photometry: External Consistency: 1% photometric calibration accuracy is achieved overall, with future improvement expected from the effective transmission curve.
- Astrometry: <0.1 arcsec astrometric offsets relative to Gaia are reported even in regions with larger-than-typical deviations.Most science cases are unlikely to be significantly affected, but high-precision position users should be warned.
- PSF Model: 4 × 10^-3 fractional PSF-size residual remains, failing the stringent cosmic-shear requirement despite limited expected impact on detection and photometry.Higher-order image warping reduces but does not fully eliminate the residual.
- Known Issues: PDR2 improves overall data quality, but persistent deblending failures, residual optical ghosts, satellite trails, and bright-star mask problems remain.Crowding is especially problematic in UltraDeep COSMOS; PSF-matched aperture photometry on undeblended images is recommended for meaningful colors, but not total magnitudes of extended sources.
- Known Issues: Up to 1.5% photometric zero-point error can occur between i2-only and combined i+i2 regions, particularly for red objects.
7 Data Access
PDR2 data are available through a release website with file, image, PSF, and catalog-access tools. Users can download flat-file products, retrieve cutouts and PSFs, and query catalog tables online or from the command line.
- 7 Data Access: The release website summarizes quality-assurance plots and known issues, while raw data remain available through SMOKA.The website itself provides processed data.
- 7 Data Access: All pipeline outputs are available as flat files through tools linked from the data release website.These tools include file search and image cutout services.
- 7 Data Access: An online PSF retrieval tool provides coadd PSF images at arbitrary sky positions.
- 7 Data Access: Catalog products are loaded into a database that users can query through an online SQL editor or command-line tool.Query results can be downloaded.
8 Status of Collaborating Surveys
Collaborating surveys extend HSC-SSP coverage in the Deep and UltraDeep fields, adding deep U-band and near-infrared imaging for multiwavelength studies. These efforts support band-merged catalogs and broader wavelength coverage, although existing near-infrared surveys do not fully cover the Deep fields.
- 8 Status of Collaborating Surveys: Collaborating u-band and near-infrared surveys target the Deep and UltraDeep fields, where multiwavelength data enable galaxy-evolution science.
- 8 Status of Collaborating Surveys: CLAUDS completed deep U-band imaging and processed images to match the HSC-SSP tract/patch grid, astrometry, and pixel scale.The data cover 18.60 deg2 with median seeing of FWHM=0.92”.
- 8 Status of Collaborating Surveys: CLAUDS provides multiband U + grizy photometry using SExtractor and an adapted hscPipe pipeline for CFHT U-band images.
- 8 Status of Collaborating Surveys: Existing near-infrared surveys overlap the HSC-SSP Deep and UltraDeep fields but do not fully cover the Deep fields.
- 8 Status of Collaborating Surveys: DUNES2 acquired about 270 hours on UKIRT and covers multiple HSC-SSP fields with near-infrared depth measurements.It covers the four E-COSMOS flanking fields, DEEP2-3, and part of ELAIS-N1.
- 8 Status of Collaborating Surveys: The collaboration plans publicly released U-to-K band-merged catalogs and images at the HSC-SSP final data release.The planned products use the HSC photometry pipeline and are targeted for DR3.
9 Summary and Future Data Releases
The paper concludes that the 174-night HSC-SSP dataset is publicly available and suitable for broad scientific exploration, while warning users to consult known issues. Future release timing remains vulnerable to weather, earthquakes, and telescope problems.
- 9 Summary and Future Data Releases: 174 nights of HSC-SSP data are publicly available and intended to support a wide range of scientific explorations.
- 9 Summary and Future Data Releases: Users are advised to review the release issue list before using the data.The paper also requests appropriate acknowledgments and references for HSC-SSP, LSST, and Pan-STARRS.
- 9 Summary and Future Data Releases: The baseline plan was to make PDR3 in two years, but bad weather, Kilauea-related earthquakes, and a telescope problem significantly delayed the survey.These problems may affect the data release plan, with updates expected on the website.