PaperlayerView exact revision on arXiv

Source-linked AI summary

TPZ : Photometric redshift PDFs and ancillary information by using prediction trees and random forests

M. Carrasco Kind, R. J. Brunner

arXiv:1303.7269v1astro-ph.CO

TL;DR

Large photometric surveys require reliable photometric-redshift PDFs and better understanding of their uncertainties. TPZ uses prediction trees and random forests with measurement-error and missing-value handling, and its tests showed strong performance alongside useful training-data diagnostics. The method remains dependent on the quality and coverage of its spectroscopic training data.

  • Problem

    Large photometric surveys need accurate photometric-redshift PDFs and better understanding of systematic uncertainties in their estimates.

  • Method

    TPZ is a publicly available parallel machine-learning algorithm that uses prediction trees and random forests, incorporates measurement errors and missing values, and computes ancillary diagnostics.

  • Results

    TPZ produced accurate results on SDSS and DEEP2 samples and performed comparably to or better than other empirical methods in the PHAT1 blind challenge.

  • Takeaways & Limitations

    TPZ’s ancillary information can identify important attributes, poorly estimated regions, and promising locations for new spectroscopic training data.

  • Takeaways & Limitations

    Because TPZ is empirical, its reliability depends on the available spectroscopic training data and decreases when applied beyond the training-data parameter-space limits.

Abstract

from arXiv · show

With the growth of large photometric surveys, accurately estimating photometric redshifts, preferably as a probability density function (PDF), and fully understanding the implicit systematic uncertainties in this process has become increasingly important. In this paper, we present a new, publicly available, parallel, machine learning algorithm that generates photometric redshift PDFs by using prediction trees and random forest techniques, which we have named TPZ. This new algorithm incorporates measurement errors into the calculation while also dealing efficiently with missing values in the data. In addition, our implementation of this algorithm provides supplementary information regarding the data being analyzed, including unbiased estimates of the accuracy of the technique without resorting to a validation data set, identification of poor photometric redshift areas within the parameter space occupied by the spectroscopic training data, a quantification of the relative importance of the variables used to construct the PDF, and a robust identification of outliers. This extra information can be used to optimally target new spectroscopic observations and to improve the overall efficacy of the redshift estimation. We have tested TPZ on galaxy samples drawn from the SDSS main galaxy sample and from the DEEP2 survey, obtaining excellent results in each case. We also have tested our implementation by participating in the PHAT1 project, which is a blind photometric redshift contest, finding that TPZ performs comparable to if not better than other empirical photometric redshift algorithms. Finally, we discuss the various parameters that control the operation of TPZ, the specific limitations of this approach and an application of photometric redshift PDFs.

1 INTRODUCTION

Large photometric surveys need fast, accurate photometric redshifts and probability density functions because cosmological measurements depend on galaxy distances and sample sizes. TPZ addresses this need with an empirical random-forest approach that incorporates measurement errors and provides ancillary diagnostics.

  • Photometric surveys provide many more galaxy observations than spectroscopic surveys, but spectroscopic redshifts remain more precise and expensive to obtain.
  • Accurate photo-z estimates are increasingly important for large surveys such as DES and LSST, which observe galaxies often too faint for spectroscopy.
  • Cosmological measurements including clustering, weak lensing, baryon acoustic oscillations, and cluster mass functions depend strongly on galaxy counts and distance accuracy.
  • Reliable photo-z methods should provide probability density functions, whose full information can improve cosmological analyses.Using the full redshift PDF improved a two-point angular quasar correlation measurement by nearly a factor of four in one cited study.
  • TPZ is a publicly available parallel Python algorithm that estimates photometric redshift PDFs with prediction trees and random forests.It combines ensemble predictions and can use mean regression predictions or classification modes within terminal leaves.
  • The approach incorporates measurement errors and addresses missing values while targeting systematic uncertainties and broader diagnostic needs in empirical photo-z estimation.The introduction identifies training-set systematics and outlier identification as important functionality for modern photo-z algorithms.

2 METHODS

The methods combine prediction trees and random forests to partition photometric data, aggregate predictions, and generate redshift PDFs. TPZ extends this framework with measurement-error perturbations, missing-value handling, and ancillary diagnostics for attribute importance and poorly constrained regions.

  • Prediction trees: Prediction trees recursively split multidimensional data into terminal leaves containing objects with similar properties, where a model produces each leaf’s prediction.Splitting stops when a criterion such as minimum leaf size is reached.
  • Prediction trees: TPZ implements both classification and regression trees for photometric redshift prediction.Classification leaves can yield class probabilities, while regression leaves use a continuous prediction model.
  • Prediction trees: Classification trees select splits using information gain based on impurity, with entropy, Gini impurity, and classification error available as impurity measures.Impurity is zero for a pure node and is highest for an even two-class mixture.
  • Random forests: Random forests build many unpruned trees from bootstrap samples and combine their predictions, while random feature subsampling adds another layer of randomness.The number of variables considered at each split is an adjustable parameter related to tree strength and inter-tree correlation.
  • Ancillary information: TPZ estimates attribute importance by permuting each attribute in out-of-bag data and averaging the resulting prediction-error increase across trees.The resulting ranking can help remove uninformative attributes and guide follow-up observations.
  • Ancillary information: A proximity matrix records how often pairs of galaxies occupy the same terminal leaf, enabling identification of poorly constrained or inaccurate regions of parameter space.These diagnostics can guide spectroscopic targeting to improve the training sample.
  • TPZ extensions: TPZ extends random forests to photometric-redshift PDFs while incorporating measurement errors, ancillary training-sample information, and uncertainty control.The method can identify attribute-space regions dominated by shot noise statistics.
  • TPZ algorithm: Preprocessing optionally applies PCA, iteratively replaces missing values, and generates perturbed training samples from normally distributed measurement errors before forest construction.The algorithm then builds fully grown trees using bootstrap samples for each perturbed training sample.

3 DATA SELECTION

The paper evaluates TPZ on multiple photometric and spectroscopic data sets, including SDSS MGS, PHAT1, and DEEP2, each chosen to test different code features.

  • Data sets: TPZ is tested on photometric and spectroscopic data sets that vary in quantity and quality.The authors state that these data sets are used to demonstrate different TPZ parameter configurations and capabilities.
  • SDSS MGS: The SDSS MGS subset contains 55,000 galaxies spanning z ≈ 0.02–0.3, with 15,000 used for training and 40,000 for blind testing.The sample has a mean redshift of 0.1.
  • SDSS MGS: SDSS MGS photometric colors are used instead of magnitudes to demonstrate TPZ flexibility and generate ancillary information.
  • PHAT1: PHAT1 combines eighteen-band photometry with spectroscopy for 1,984 galaxies and provides a blind comparison setting with only 515 published training redshifts.
  • DEEP2: DEEP2 DR4 supplies secure spectroscopy for over 38,000 sources, while TPZ analyses separately calibrated SDSS- and CFHTLS-based photometry.

4 APPLICATION/DISCUSSION

TPZ is applied in regression and classification modes to SDSS MGS data, where regression performs slightly better overall and confidence cuts produce cleaner samples. Out-of-bag data additionally provide accuracy estimates and attribute diagnostics without a separate validation set.

  • SDSS MGS application: Regression and classification modes show similar central-redshift performance, but regression performs slightly better across bins, especially at low and high redshift.Classification exhibits systematic errors in the first and last bins.
  • Confidence selection: zConf is the integrated PDF probability within zphot ± σTPZ(1+zphot), using σTPZ = 0.03 as an intrinsic-scatter approximation.
  • Confidence selection: Higher zConf corresponds to more concentrated PDFs, enabling selection of cleaner samples with improved performance metrics.The authors use the regression-mode PDF mean for subsequent SDSS MGS analysis.
  • Confidence selection: At restrictive confidence cuts, regression and classification performance becomes similar, while regression retains more galaxies.
  • Confidence selection: With zConf ≥ 0.75, 91% of galaxies remain and the fractions with |∆z| < 0.001, 0.002, and 0.003 are 45.2%, 73.0%, and 89.8%.
  • Ancillary information: Out-of-bag predictions provide unbiased, a priori estimates of bias and scatter using only the training set.Each tree evaluates approximately one-third of objects excluded from its bootstrap sample, and the metrics are averaged over the forest.
  • Ancillary information: The g−r color has the largest relative importance factor, close to four, while attribute importance varies with redshift for other colors.The authors relate this behavior to spectral features crossing filter bands.

Relative Importance

TPZ’s ancillary diagnostics identify important attributes and map regions where photometric-redshift predictions are weak, helping target additional training data.

  • Relative Importance: Attribute importance is estimated by permuting each input and comparing out-of-bag performance with the unperturbed case.
  • Relative Importance: A heat map using g−r and u−g encodes binned predictive performance, with higher values indicating lower prediction success.Poor-performing bins may contain outliers or galaxies with bad photometry.
  • Relative Importance: The heat map identifies parameter-space regions where follow-up observations can improve photometric-redshift performance.

Identifying new training data

The authors test whether ancillary diagnostics can guide training-set expansion, finding that selectively adding galaxies from poorly predicted regions outperforms random additions and reaches a performance plateau.

  • Identifying new training data: The experiment starts with 1,000 randomly selected training galaxies to compare random and selectively targeted additions.
  • Identifying new training data: Selecting galaxies from low-accuracy heat-map regions produces more accurate predictions than adding galaxies randomly.Adding 1,000 targeted galaxies slightly outperforms adding 2,000 random galaxies.
  • Identifying new training data: After approximately 5,000 added galaxies, performance varies little, corresponding to roughly half the full training-set size.The comparison includes the 15,000-galaxy training set.
  • Identifying new training data: The results support constructing training sets by selectively using existing observations or obtaining new spectroscopic observations.

Error distribution

TPZ derives photometric-redshift errors directly from each estimated PDF. For SDSS MGS galaxies, the standardized errors are close to unbiased, with a slight non-Gaussian tail attributed to narrow PDFs and low-redshift overprediction.

  • Error distribution: TPZ estimates photo-z errors directly from each PDF using statistics such as the mean or mode.The demonstrated σ68 error is the PDF region centered on the mean containing 68% of cumulative probability.
  • Error distribution: The standardized error distribution is close to the unbiased expectation, indicating reliable error estimates.Its best-fit Gaussian has µ = 0.112 and σ = 0.949.
  • Error distribution: A slightly extended left-hand tail makes the standardized-error distribution depart from a perfect Gaussian.The paper attributes this to very narrow PDFs and the SDSS MGS concentration at lower redshifts, where photo-z methods tend to overpredict redshifts.

Size of forest

Increasing the number of trees improves TPZ accuracy until the predictive power stabilizes. Around 100 trees is therefore a reasonable operating point that avoids unnecessary computational cost.

  • Size of forest: Using more trees increases computational demands and slows forest training and photo-z PDF construction.The paper frames the plateau in predictive power as a practical basis for choosing the forest size.
  • Size of forest: TPZ becomes more accurate as the number of trees increases, but predictive power changes little after around 100 trees.The test holds the training sample fixed at 10,000 galaxies.
  • Size of forest: The generalized error converges as more trees are added, and TPZ does not over-fit the data in this comparison.This behavior is linked to the convergence property of random forests.

Training size

TPZ accuracy improves as the training set samples more of the multidimensional parameter space, then levels off. The method also exposes underrepresented test objects and performs robustly in the PHAT1 blind evaluation despite limited training data.

  • Training size: TPZ accuracy does not change significantly after using around 70% of the SDSS MGS galaxies for training.This threshold is dataset-dependent and reflects diminishing information from additional training galaxies.
  • Training size: Prediction accuracy increases until most of the multidimensional parameter space has been sampled, after which new training galaxies add little information.The training-size trend is shown in the bottom panel of Figure 9.
  • Training size: Underrepresented test galaxies can be identified with a cross-data proximity matrix and treated individually with other photo-z techniques.The limitation is that poor representation in the training set constrains prediction accuracy.
  • Training size: Despite approximately 500 training galaxies, about 1,500 validation galaxies, and a sparse redshift range extending to z ∼6, TPZ produced robust PHAT1 estimates.The PHAT1 setup was a blind contest with unknown test spectroscopic redshifts.
  • Training size: TPZ ranked among the most accurate predictions across four photometric samples and outperformed some template-based techniques.The samples varied the inclusion of IRAC bands and imposed R < 24 magnitude limits.

DEEP2

On DEEP2, TPZ gives consistent photo-z results across redshift for the high-confidence subset, with few outliers or catastrophic estimates. Accuracy degrades near the distribution boundaries and at higher redshift, where training sparsity and photometric errors become more important.

  • DEEP2: Selecting zConf > 0.7 retains about 81% of DEEP2 galaxies for the merged CFHTLS and SDSS photo-z analysis.The CFHTLS and SDSS photometric samples are treated independently before their photo-z results are merged.
  • DEEP2: Across DEEP2 redshifts, isodensity contours and percentile error bars indicate few outliers or catastrophic photo-z estimates.Figure 10 summarizes medians and tenth-to-ninetieth percentile ranges in spectroscopic bins of width ∆z = 0.1.
  • DEEP2: The first two redshift bins are systematically high and the last two systematically low, indicating reduced accuracy at both distribution ends.The paper links these effects partly to incomplete spectroscopic training samples at the boundaries.
  • DEEP2: At higher redshifts, lower training density and larger magnitude errors near the photometric limit reduce TPZ efficacy.These effects are associated with systematic underestimation at higher redshifts.
  • DEEP2: For redshifts up to 1.1, mean PDF skewness is near zero; at higher redshifts, increasing skewness tends to lower the PDF mean.Figure 11 reports the mean skewness and its one-σ interval as functions of spectroscopic redshift.

Relative Importance

TPZ uses out-of-bag data and permutation-based attribute importance to assess predictive reliability and identify which inputs drive photometric-redshift estimates. For DEEP2, optical bands dominate, while removing uninformative attributes can improve efficiency but excessive removal loses information.

  • Out-of-bag assessment: Out-of-bag metrics for DEEP2 show strong agreement with test-data bias and variance, supporting OOB-based accuracy assessment.The comparison also ranks the eight photometric bands and the RG radius attribute.
  • Attribute ranking: The R and r bands are the most important attributes for DEEP2 photo-z prediction, with importance peaking near redshifts 0.3–0.5.The authors associate this peak with the 4000 Å break entering these filters.
  • Attribute ranking: The B band and RG attribute rank among the least important inputs, with RG behaving like a random variable that may add noise.The RG attribute shows no clear redshift-dependent importance.
  • Attribute selection: Removing the two least important attributes improves TPZ precision and runtime, whereas removing four slightly worsens performance by discarding information.Fewer dimensions reduce tree-splitting, memory, and sampling costs; excessive removal increases scatter.
  • Attribute ranking: TPZ ranks an artificial redshift-dependent variable as highly important and a uniformly random variable as minimally important.The control test places the two artificial inputs at opposite extremes of the importance ranking.

Missing data

TPZ replaces missing or highly uncertain attributes using its proximity matrix, allowing incomplete objects to remain in training and application samples. Tests with deliberately corrupted magnitudes assess this recovery strategy against cutting incomplete data.

  • Replacement method: TPZ can replace missing or highly uncertain attributes in either training or application data using values computed from the proximity matrix.This preserves incomplete training objects and enables photo-z estimation from incomplete application data.
  • Replacement method: Replacing missing training attributes retains more data than culling incomplete objects, while application-side replacement permits predictions without discarding galaxies.The authors report that such predictions will still be reliable in most cases.
  • Robustness test: The robustness test randomly replaced 50% of training magnitudes with a bad value before applying TPZ-based attribute replacement.The corrupted sample included galaxies with multiple bad attributes and used six replacement iterations.
  • Evaluation: Photo-z estimates from the replaced and cut training samples were compared with estimates from the original clean sample using clean and corrupted test data.Table 5 reports average bias and variance for these comparisons.

Photo-z PDFs and zConf

TPZ represents photometric redshift uncertainty with PDFs and uses zConf to select galaxies with concentrated estimates. Stacking the full PDFs also improves reconstruction of the galaxy redshift distribution relative to using single PDF summaries.

  • zConf: Higher zConf corresponds to narrower, more concentrated photo-z PDFs, whereas wide and sparse PDFs have lower zConf.This pattern is shown for representative DEEP2 and SDSS galaxies.
  • zConf: Increasing the zConf cut reduces both mean absolute bias and scatter independently of redshift.The test compares cuts at 0.5, 0.7, and 0.9.
  • zConf: A zConf cut of 0.5 selects galaxies whose photo-z estimate has 50% probability of lying within ±σ_TPZ(1+zphot).Higher cuts improve accuracy but reduce catalog size, retaining 90%, 76%, and 38% at cuts of 0.5, 0.7, and 0.9.
  • Full PDF application: Using the full photo-z PDFs enables estimation of N(z), a distribution relevant to weak-lensing tomography and angular-clustering analyses.Single-value summaries such as means or modes discard information contained in the PDFs.
  • Full PDF application: The stacked PDF produces smaller fractional error across almost all redshifts and a smoother distribution resembling the spectroscopic sample.The DEEP2 sample is concentrated mainly between redshifts 0.6 and 1.2.

5 CONCLUSIONS

TPZ is a publicly released parallel random-forest algorithm that generates photometric-redshift PDFs while providing diagnostics for accuracy, data quality, attribute importance, and training-set limitations. Tests on SDSS, PHAT1, and DEEP2 demonstrate accurate estimates, benefits from full PDFs and targeted training data, and practical handling of uncertainty and missing values.

  • TPZ and its scope: TPZ generates photometric-redshift PDFs with prediction trees and random forests, supporting classification and regression modes plus ancillary information about the analyzed data.Regression is preferred for poorly sampled or non-uniform training data, whereas classification summarizes redshift distributions for constructing priors.
  • Diagnostics and sample selection: Out-of-bag diagnostics provided unbiased estimates of bias and dispersion without a separate validation set, while zConf selected samples with improved photo-z estimates and minimal outliers.These diagnostics were similar to values obtained from test data for both SDSS MGS and DEEP2.
  • Training-data optimization: Attribute-importance rankings identified poor regions and enabled targeted additions of training galaxies, while convergence with forest size and training-set size constrained computational requirements.Manually selected additions improved predictions more than randomly adding galaxies, except where targeted training data were needed to address poor regions.
  • Validation and performance: TPZ produced accurate results across the SDSS main galaxy sample, PHAT1 blind challenge, and DEEP2 survey, with PHAT1 performance comparable to or better than other empirical and some SED-fitting techniques.The DEEP2 application covered a large redshift range and identified the R- and I-band magnitudes as the most important attributes in that case.
  • Missing data: TPZ handled missing parameters by recovering values and producing photo-z estimates consistent with those from the original complete data set.This was demonstrated after artificially introducing bad or missing values in training and testing data.
  • Full-PDF application: Stacking individual photo-z PDFs recovered the underlying DEEP2 galaxy redshift distribution more precisely than using the mean of each PDF.The stacked distribution had smaller error for almost all redshifts and was smoother and more similar to a fit to the spectroscopic sample.
Loading 1303.7269v1…