RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing

Madhusudan Srinivasan; Namith Nishal Raphae

Source-linked AI summary

RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing

Madhusudan Srinivasan, Namith Nishal Raphae

arXiv:2608.27704v1Aug 27, 2026cs.LGcs.AIcs.SE

TL;DR

Retraining can create costly-to-verify regression faults, motivating prioritization under limited verification budgets. RiskBlend blends four historical and cross-version risk signals with validation-learned APFD-squared weights, achieving the highest APFD across the evaluated dataset, classifier, and scenario combinations. Its scope is bounded by benchmark tabular data, selected classifier families, and constructed update scenarios.

Problem
ML regression verification is costly because ground-truth checking may require expert review, while existing prioritization relies mainly on single-model confidence and misses cross-version behavioral changes.
Method
RiskBlend is a classifier-agnostic framework combining historical failure patterns, prediction shift, decision-boundary shift, and neighborhood change through validation-learned APFD-squared weighting.
Results
RiskBlend achieves the highest APFD in all 80 classifier–scenario cells, with statistically significant improvements over all baselines.
Takeaways & Limitations
Cross-version behavioral signals complement single-model confidence for robust regression-fault prioritization, especially when confidence loses discriminative power after version transitions.
Takeaways & Limitations
Results may not generalize to unstructured data, deep learning models, production pipelines with continuous drift, or update distributions unlike the four evaluated scenarios.

Abstract

from arXiv · show

When machine learning classifiers are retrained, inputs correctly classified by the previous model version may be misclassified by the updated version, creating regression faults that are costly to detect because verifying predictions against ground truth may require human annotation, expert review, or expensive simulation rather than inexpensive model inference. Test input prioritization addresses this problem by ranking inputs so that a limited verification budget reveals as many regression faults as possible. Existing approaches rely predominantly on single-model confidence scores and do not exploit how predictions, decision boundaries, and local neighborhoods change between model versions. We propose RiskBlend, a classifier-agnostic prioritization framework that combines four complementary risk signals: historical failure patterns, prediction shift, decision-boundary shift, and neighborhood change. These signals are combined using validation-learned APFD-squared weighting. Across four datasets, five classifiers, four regression-update scenarios, and 15 random seeds, totaling 1,200 experimental configurations, RiskBlend achieves the highest average APFD in all 80 dataset-classifier-scenario combinations, with improvements of up to 0.32 APFD over the strongest baseline. Confidence-based methods remain competitive primarily for linear classifiers on sparse categorical features, which we attribute to feature-space geometry. The results show that cross-version behavioral signals provide important complementary information for prioritizing regression faults in machine learning systems.

I. INTRODUCTION AND MOTIVATION

ML model updates can introduce new misclassifications on inputs previously handled correctly, while verifying correctness remains costly because ground truth often requires human or expert review. RiskBlend addresses this constrained-verification problem by prioritizing inputs using blended, validation-learned risk signals.

ML behavior changes with retraining, feature pipelines, and evolving data, making regression testing necessary after model updates.
Verifying updated-model predictions requires ground-truth comparison that may involve human annotation, delayed outcomes, or expert review rather than cheap inference alone.
10–15 minutes per case can be required for physician review, so limited verification budgets make random selection inefficient.
Existing prioritization methods predominantly use single-model confidence, uncertainty, transformations, or mutation-derived signals.
RiskBlend blends risk signals and learns their contributions by optimizing APFD on a validation set, avoiding manually tuned weights.

B. Test Input Prioritization

Test input prioritization ranks cases by fault-revealing potential so a small oracle budget can uncover regressions early. RiskBlend positions this task as regression-specific, combining cross-version signals rather than relying only on single-model confidence or objectives from other testing settings.

Test input prioritization orders instances by estimated fault-revealing potential so top-ranked cases are verified first.
With an oracle budget B ≪ n, the goal is to recover as many true regression faults as possible from the top-B ranked instances.
Classical-ML regression-testing prioritization remains comparatively underexplored relative to deep-learning test prioritization.
MLPrior uses mutation-derived features and learning-to-rank for generic single-model misclassification, whereas RiskBlend targets regression under model evolution.
RiskBlend learns to combine historical and cross-version signals so newly introduced faults are ranked earlier under limited oracle budgets.

IV. PROPOSED APPROACH

RiskBlend prioritizes test cases likely to expose regressions between consecutive classifier versions. Its History Risk component uses validated prior failures and distance in standardized feature space to identify fault-prone regions.

RiskBlend ranks cases likely to be correctly classified by Vk but incorrectly classified by Vk+1.
History Risk: History Risk identifies validated regression-fault locations using a labeled validation set from Vk+1’s training partition.
History Risk: Test instances are scored by their minimum distance to the validated regression-fault region in standardized Vk+1 feature space.
History Risk: A higher history-risk score indicates proximity to confirmed fault locations and causes those instances to be ranked earlier.
History Risk: If no validation faults are observed, rhist is zero and the remaining three signals carry the full prioritization weight.

2) Regression-Aware Impact Risk (rimpact):

Regression-Aware Impact Risk measures how model-version transitions affect predictions, local decision boundaries, and training neighborhoods. It combines prediction and boundary changes with neighborhood and local-distribution signals to identify inputs most affected by the update.

Impact Risk uses classifier-agnostic signals derived from predictions and local behavior to measure effects of transitioning from Vk to Vk+1.
Prediction Shift: Prediction Shift measures the absolute change in positive-class probability for the same test instance across model versions.
Prediction Shift: Large prediction shifts indicate altered confidence and may signal a nearby decision-boundary change associated with higher regression risk.
Boundary Shift: Boundary Shift combines probability movement, threshold crossing at 0.5, and boundary nearness to characterize changed classification behavior.
Boundary Shift: Inputs that cross the boundary and lie near the threshold receive the highest boundary-shift scores.
Neighbourhood Change: Neighborhood Change compares k-nearest-neighbor training neighborhoods separately for Vk and Vk+1 using k = 15.

3) Final RiskBlend Score:

RiskBlend computes a final prioritization score by combining four risk signals with non-negative weights normalized to sum to one. The weights are learned from validation-set APFD performance so stronger signals receive greater influence.

RiskBlend combines four risk signals into a final prioritization score using a weighted combination.
The signal weights are non-negative and normalized to sum to one.
Validation-set APFD determines each signal’s normalized weight, giving stronger regression-fault detectors greater influence.

V. EVALUATION

The evaluation spans diverse datasets, classifiers, update scenarios, and random seeds, using held-out test data and validation-driven weighting. The setup targets realistic, measurable regression behavior while avoiding catastrophic degradation.

Experimental Setup: The study evaluates RiskBlend across four benchmark datasets from socioeconomic, healthcare, and finance domains.
Experimental Setup: Five classifiers represent ensemble, rule-based, linear, probabilistic, and instance-based learning families.
Regression-Update Scenarios: Four update scenarios vary feature enrichment, data augmentation, stronger drift, and preprocessing pipeline changes.
Regression-Update Scenarios: Update magnitudes were selected to produce measurable but non-catastrophic regression behavior suitable for APFD evaluation.
Experimental Setup: Experiments use independent model training, a held-out 20% validation partition, shared test sets, and 15 random seeds.

B. Baseline Methods

The baseline study compares uncertainty-based, distance-aware, mutation-based, and random prioritization strategies under identical implementation and evaluation conditions. DeepGini ranks test instances by prediction uncertainty derived from class probabilities.

Baseline Coverage: Four baselines cover uncertainty-based, distance-aware, mutation-based, and random prioritization strategies.
1) DeepGini: DeepGini computes a prediction-uncertainty score from each test instance’s class-probability vector and ranks instances in descending order.
1) DeepGini: For binary classification, DeepGini is equivalent to a monotonic transformation of distance from the decision boundary.

2) DATIS-Tabular (Distance-Aware Test Prioritization):

DATIS-Tabular prioritizes test instances using neighborhood distance and support rather than prediction probabilities alone. MLPrior instead estimates fault-proneness from model and input perturbations through a supervised secondary ranking model.

2) DATIS-Tabular: DATIS-Tabular uses distance-based neighborhood information and redundancy elimination instead of relying solely on prediction probabilities.
2) DATIS-Tabular: DATIS compares each test instance with k nearest training neighbors and prioritizes instances whose predicted labels receive weak local support.
3) MLPrior: MLPrior estimates test-instance fault-proneness by analyzing sensitivity to model-level and input-level perturbations.
3) MLPrior: MLPrior combines original, model-mutation, and data-mutation features in an XGBoost ranking model that predicts misclassification probability.

C. Evaluation Metric

The evaluation uses APFD to measure how early regression faults appear in a prioritized test sequence, while RQ1 and RQ2 assess overall effectiveness and individual signal contributions.

APFD measures how early regression faults appear in the prioritized ranking.
APFD uses the ranked positions of m regression faults among n test inputs to quantify prioritization effectiveness.
RQ1: RQ1 compares RiskBlend with baseline prioritization methods using APFD.
RQ2: RQ2 examines each risk signal’s contribution and informativeness across update scenarios and classifier families.

A. RQ1: Overall Prioritization Effectiveness (APFD)

RiskBlend achieves consistently strong prioritization across datasets, classifiers, and regression-update scenarios, with decision-boundary and historical signals providing the strongest contributions.

Overall effectiveness: RiskBlend achieves the highest APFD in all 80 classifier–scenario cells across the four datasets.Results are averaged over 15 independent seeds.
Overall effectiveness: +0.317 APFD is RiskBlend’s largest reported advantage over the strongest baseline, while the smallest advantage is +0.003.The largest gain occurs for Credit Card moderate regression with KNN; the smallest occurs for Adult pipeline drift with Naïve Bayes.
Classifier and scenario effects: RiskBlend’s gains are largest for Decision Tree and KNN classifiers, where confidence-based methods lose discriminative power after model transitions.On Hospital KNN, RiskBlend reaches 0.861–0.916 versus DeepGini’s 0.554–0.626.
Classifier and scenario effects: RiskBlend maintains its advantage across all four regression-update scenarios, including pipeline drift caused solely by preprocessing changes.Prediction-shift and boundary-shift signals are reported as particularly effective in that setting.
Baseline comparisons: Random selection achieves approximately 0.50 APFD, while MLPrior and DATIS trail RiskBlend across every evaluated cell.MLPrior and DATIS degrade more sharply on the smaller Hospital and Credit Card datasets.
Signal contribution: Neighborhood change is consistently the weakest signal, while prediction shift is largely redundant with boundary shift in the ablation results.Neighborhood-change weights range from 0.099 to 0.142, and removing prediction shift slightly improves APFD on average.
Signal contribution: Boundary shift is the most influential signal, receiving the highest learned weight and producing the largest APFD drop in leave-one-signal-out ablation.Its mean APFD removal drop ranges from 0.0328 on Bank to 0.0749 on Credit Card.

C. Statistical Significance

RiskBlend significantly outperforms the baselines across datasets, with statistically significant gains and generally large effect sizes. However, APFD and runtime must be interpreted alongside fault density and dataset-dependent computational cost.

Statistical significance: All 16 dataset-level comparisons are significant at p < 0.001 after Bonferroni correction, with large effect sizes in 15 of 16 cases.The exception is RiskBlend versus DeepGini on Adult, where Cliff’s δ = 0.084 and ∆APFD = +0.077.
Comparative performance: RiskBlend achieves ∆APFD ≥+0.130 against MLPrior and ∆APFD ≥+0.380 against Random across all datasets.Against DATIS-Tabular, effect sizes are uniformly large with δ ≥0.973.
Comparative performance: RiskBlend achieves the highest APFD in all 80 classifier-scenario cells, with its largest gains on tree-based and distance-based classifiers.For Decision Trees, DeepGini APFD falls to 0.547–0.623 while RiskBlend maintains 0.816–0.858.
Interpretation: Fault densities range from approximately 3.6% on Bank to 14.0% on Credit Card, creating prioritization settings of differing difficulty.Near-ceiling APFD values, such as ≥0.98, should therefore be interpreted in the context of underlying fault density.
Execution cost: RiskBlend takes 0.32–8.95 s for prioritization, and its computational cost is dataset-dependent rather than consistently lower than MLPrior’s.It is faster than MLPrior on Bank but slower on Adult, Hospital, and Credit Card.

VIII. THREATS TO VALIDITY

The study’s validity is bounded by how validation weights generalize and by evaluation on benchmark tabular settings. Its conclusions may not extend directly to unstructured data, deep learning, production drift, or update distributions beyond the four scenarios.

Internal validity: Signal weights are learned from a held-out 20% validation partition, whose size or representativeness may limit generalization to the test set.Results are averaged over 15 random seeds to stabilize estimates across partitioning choices.
External validity: RiskBlend is evaluated on four datasets, five classifier families, and four regression-update scenarios spanning financial, socioeconomic, and healthcare domains.This breadth does not establish generalization to images, text, deep learning, continuous data drift, or production pipelines.
Scope and approach: The framework fuses four generic signals into an APFD-squared score learned from validation without classifier-specific tuning.The signals are historical failure patterns, prediction shift, decision-boundary shift, and neighborhood change.
Empirical conclusion: RiskBlend achieves the highest APFD in all 80 classifier-scenario cells, with statistically significant improvements over all baselines.The conclusion is supported by evaluations across four datasets, five classifiers, and four regression-update scenarios.