Source-linked AI summary
Probabilistic Random Forest: A machine learning algorithm for noisy datasets
Itamar Reis, Dalya Baron, Sahar Shahaf
TL;DR
Astronomical ML datasets often contain measurement and label uncertainties that standard algorithms do not directly model. The paper modifies Random Forest into Probabilistic Random Forest, representing features and labels probabilistically, and finds improved classification under diverse noise conditions with a moderate runtime increase.
Problem
Most ML algorithms do not directly account for heterogeneous measurement uncertainties, limiting their use on common noisy astronomical datasets.
Method
Probabilistic Random Forest modifies Random Forest by treating uncertain features and labels as probability distributions rather than deterministic quantities.
Results
PRF outperforms RF across noisy-data experiments, improving classification accuracy by up to 10% for noisy features and 30% for noisy labels.
Takeaways & Limitations
PRF naturally represents missing values and is more robust than RF when training and test sets have different noise characteristics, supporting Transfer Learning use.
Takeaways & Limitations
The experiments are simple, and one clean-dataset experiment slightly favors RF, which the authors attribute to implementation differences.
Abstract
from arXiv · showhide
Machine learning (ML) algorithms become increasingly important in the analysis of astronomical data. However, since most ML algorithms are not designed to take data uncertainties into account, ML based studies are mostly restricted to data with high signal-to-noise ratio. Astronomical datasets of such high-quality are uncommon. In this work we modify the long-established Random Forest (RF) algorithm to take into account uncertainties in the measurements (i.e., features) as well as in the assigned classes (i.e., labels). To do so, the Probabilistic Random Forest (PRF) algorithm treats the features and labels as probability distribution functions, rather than deterministic quantities. We perform a variety of experiments where we inject different types of noise to a dataset, and compare the accuracy of the PRF to that of RF. The PRF outperforms RF in all cases, with a moderate increase in running time. We find an improvement in classification accuracy of up to 10% in the case of noisy features, and up to 30% in the case of noisy labels. The PRF accuracy decreased by less then 5% for a dataset with as many as 45% misclassified objects, compared to a clean dataset. Apart from improving the prediction accuracy in noisy datasets, the PRF naturally copes with missing values in the data, and outperforms RF when applied to a dataset with different noise characteristics in the training and test sets, suggesting that it can be used for Transfer Learning.
1. INTRODUCTION
Astronomical machine-learning applications face heterogeneous measurement uncertainties that standard algorithms generally do not model directly. The paper motivates modifying Random Forests to better use uncertainty information in noisy astronomical data.
- Motivation: Astronomical ML commonly supports supervised prediction from examples, but datasets may contain complex, object-dependent measurement noise.Such applications use measured features or derived quantities to predict discrete labels or continuous physical properties.
- Motivation: Most off-the-shelf ML algorithms were not designed for astronomical datasets with different uncertainties across features or objects.Algorithm performance depends strongly on the sample’s signal-to-noise ratio, motivating explicit use of noise information.
- Motivation: 25. Noisy features may be ignored when poor measurement quality correlates with low information, but complex noise can require explicit uncertainty information.The paper argues that measurement values alone may not compensate for sufficiently complex noise, even with more data and computation.
- Random Forest: Random Forest is a popular algorithm because it is simple, performs well across many tasks, and can support classification, regression, and unsupervised learning.This work focuses on its classifier while noting that the proposed method can generalize to other Random Forest uses.
- Random Forest: A single decision tree can overfit its training data and generalize poorly, whereas Random Forest constructs many decision trees during training.Tree nodes encode feature conditions, and predictions use the highest-probability class in a terminal node.
3. PROBABILISTIC RANDOM FOREST
Probabilistic Random Forest (PRF) extends Random Forest by representing feature and label uncertainties probabilistically. Uncertain objects propagate through tree branches with probabilities, while probabilistic impurity and selective propagation make uncertainty-aware classification computationally feasible.
- PRF is an RF-based classification algorithm that accounts for uncertainties in input data to improve predictive capabilities.It modifies the original RF algorithm rather than replacing its overall tree-ensemble framework.
- The PRF represents features as probability distributions and labels as probability mass functions instead of deterministic values.Feature distribution means equal the measured values, with variances given by squared uncertainties; labels receive probabilities across possible classes.
- Feature uncertainty makes each split probabilistic, allowing an object to propagate to both branches and potentially reach multiple tree nodes.Branch probabilities are calculated from the uncertain feature value and combined along the sequence of turns leading to a node.
- Label uncertainty propagates through the splitting criterion and into the predictive model during tree construction.The probabilistic tree treats class probabilities as random variables and uses their expectancy values when determining node impurity.
- Selective propagation prunes branches whose object-propagation probability falls below a tunable threshold, reducing runtime.The ideal PRF propagates through all branches; the implementation retains only branches above the threshold.
- The PRF changes both object propagation and cost functions, while other training details such as split stopping remain classical-RF-like.The cost function uses a weighted average of modified impurities for the two child nodes.
4. EXPERIMENTS
The experiments compare Probabilistic Random Forest with Random Forest under noisy labels, noisy features, and differing noise characteristics between training and test data. PRF generally performs better, especially for noisy labels and complex or mismatched feature uncertainties, while its approximation threshold trades runtime for convergence.
- Experimental setup: The experiments use synthetic two-class datasets with 15 features, including 10 informative features, and separate training and test sets of 5,000 objects each.The study compares PRF with the scikit-learn RF implementation across four noise settings.
- Noise in the labels: PRF outperforms RF across label-noise levels and loses less than 5% accuracy with more than 45% wrong labels.The label noise is applied to the training set, while the test set is kept clean to estimate algorithm accuracy.
- Noise in the features: PRF uses noisy feature values together with their uncertainties, whereas RF receives only the noisy feature values.The feature-noise level is defined from the average measurement scatter relative to the original dataset’s intrinsic scatter.
- Noise in the features: For simple feature noise, both accuracies decrease with noise and PRF improves accuracy by roughly 1%, because RF can already ignore poorly measured, low-information features.The improvement is small because the injected noise correlates feature quality with information content.
- Complex and mismatched noise: With different noisy features across object groups, PRF gains a larger margin over RF as the noise becomes more complex.The authors connect this result to real measurements whose noise can depend on hidden parameters.
- Complex and mismatched noise: When training and test sets have different noise characteristics, PRF significantly improves classification accuracy and cannot be matched by adding more RF trees.The result is presented as relevant to transfer learning between surveys when uncertainties are well characterized.
5. DISCUSSION
The PRF extends Random Forest by incorporating feature and label uncertainties, improving robustness across heterogeneous-noise settings while supporting missing values and transfer learning. Its benefits depend on noise complexity, with one clean-data exception attributed to implementation differences.
- PRF properties and additional implications: The PRF incorporates feature and label uncertainties into Random Forest and converges to RF when uncertainties are zero.The method assumes Gaussian noisy measurements but can be generalized to other distributions.
- Experimental findings: The experiments use synthetic datasets with varied noise, and the PRF generally improves classification as noise level and complexity increase.The paper notes that astronomical datasets typically contain more complicated, heterogeneous noise sources.
- Caveat: The single reported exception is experiment (ii) on a clean dataset, where RF slightly outperforms PRF because of implementation differences.The exception qualifies the paper’s broader claim that PRF outperforms RF across the experiments.
- Missing values: The PRF naturally represents missing feature values by propagating an object equally to both child nodes, avoiding pre-training data dismissal or imputation assumptions.This also permits prediction for unlabeled objects with missing features, although alternative implementations remain under investigation.
- Noise complexity: The PRF outperforms RF as noise complexity increases, although the simplest cases show little accuracy difference.Constant uncertainties do not change the constructed model, while object- or measurement-dependent noise provides more useful uncertainty information.
- Label purity: With 45% wrong labels, PRF accuracy decreases by less than 5% relative to training with pure labels.This robustness depends on providing label uncertainties and may support supervised learning in surveys with poor label quality.
- Transfer learning: The PRF outperforms RF when training and test sets have different noise characteristics, supporting its use in transfer learning.The method accounts for differing noise during both training and prediction.
6. SUMMARY
The Probabilistic Random Forest extends Random Forest to represent feature and label uncertainties, improving robustness across noisy astronomical datasets. It improves accuracy over RF while supporting missing values and differing training-test noise characteristics.
- The PRF treats feature measurements and class labels as probability distributions rather than deterministic quantities.This modification allows the algorithm to incorporate uncertainty information directly.
- The PRF outperforms RF on noisy datasets, despite requiring longer running time.The comparison used synthetic data with several injected noise types.
- Up to 10% higher classification accuracy occurs with noisy features, while noisy labels yield a 30% improvement over RF.
- PRF accuracy decreases by less than 5% with up to 45% misclassified objects when label uncertainties are provided.The comparison is against a dataset with pure labels.
- PRF is more robust than RF when training and test datasets have different noise characteristics, supporting its use in Transfer Learning tasks.
- As noise entropy increases, PRF outperforms RF by an increasing margin, and the authors recommend PRF when uncertainties can be estimated.The authors also suggest that the approach could generalize to other ensemble methods such as boosting.
A. IMPLEMENTATION DETAILS
The implementation uses uncertainty-aware split grids and sequential PDF updates, with parallelized Python execution. Higher accuracy can require substantially longer runtime, controlled by the probability threshold parameter.
- Implementation: The PRF is implemented in Python with Numba acceleration and parallel training and prediction across individual decision trees.Joblib provides the tree-level parallelization.
- Runtime: PRF runtime depends strongly on the probability threshold parameter, pth.Lower pth reduces the runtime increase but also produces a more moderate accuracy improvement.
- Runtime: About an order-of-magnitude runtime increase is needed to achieve maximal classification-accuracy improvement over RF.The authors consider this increase practical in most applications because RF is fast.
- Best split search: The RF split-search grid consists of observed feature values, whereas the PRF grid adds defined locations from each measurement PDF.The PRF grid can include the PDF mean and values at ±1σ, ±2σ, and ±3σ.
- Best split search: PRF selects the threshold minimizing combined impurity after summing object PDFs weighted by their probabilities of reaching the node.
- Best split search: Sequential traversal updates left and right summed PDFs by moving the integrated PDF chunk between adjacent grid points.This avoids recomputing the weighted PDF sums from scratch at every candidate split.