PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrating Prediction Timeliness Through Multi-Objective Hyperparameter Optimization for Remaining Useful Life Prediction

Tugrul Cabir Hakyemez, Ener Uras Gokhan

arXiv:2610.01530v1cs.LG

TL;DR

Early and late RUL prediction errors have asymmetric operational consequences, but optimization commonly focuses on a single accuracy metric. This study compares three optimization regimes across five architectures and two benchmarks; multi-objective optimization reduces directional imbalance on C-MAPSS, while objectives diverge on BackBlaze.

  • Problem

    Standard regression metrics treat early and late RUL errors equally despite their asymmetric operational costs, and the effect of optimization-objective choice on prognostic behavior remains underexamined.

    Source: §1 · p. 1

  • Method

    The study evaluates five architectures under single-objective R2 maximization, single-objective NASA Score minimization, and joint NSGA-II optimization with Entropy–CRITIC selection across C-MAPSS and BackBlaze.

    Source: §1 · p. 2

  • Results

    Across C-MAPSS and BackBlaze, objective choice has limited effect on C-MAPSS aggregate accuracy but changes directional imbalance, while strategies diverge on BackBlaze; multi-objective optimization reduces C-MAPSS directional imbalance by 33%.

    Source: §4.1 · pp. 7–8, §5 · p. 10

  • Takeaways & Limitations

    Multi-objective HPO with Entropy–CRITIC selection offers a procedure for directionally balanced RUL predictions without manual preference articulation.

    Source: §5 · p. 10

    The framework covers only two objectives, BackBlaze represents a single industrial domain, and Entropy–CRITIC is static and omits domain-expert preferences.

    Source: §5 · p. 10

Abstract

from arXiv · show

In predictive maintenance, early and late RUL prediction errors carry asymmetric consequences, yet hyperparameter optimization typically targets a single accuracy metric that treats both directions equally. This study treats the optimization objective itself as a design variable. Five architectures (MLP, LSTM, XGBoost, TCN, and Transformer) are evaluated under three regimes: single-objective maximization of $R^2$, single-objective minimization of the NASA scoring function, and a multi-objective formulation that jointly optimizes both criteria. The multi-objective search employs NSGA-II with Entropy-CRITIC weighting for Pareto selection. Seventy-five model-dataset-strategy combinations are assessed on the NASA C-MAPSS turbofan and BackBlaze hard-disk drive benchmarks. On C-MAPSS, all strategies achieve comparable accuracy ($R^2 \approx 0.89$), yet multi-objective optimization reduces directional imbalance by approximately 33%, improving calibration of early versus late predictions. Model rankings prove configuration-dependent, with simpler architectures frequently outperforming deeper temporal models. On BackBlaze, the objectives shift from complementary to conflicting, producing divergent Entropy-CRITIC weights and a substantial generalization gap (best $R^2 \approx 0.34$). These results demonstrate that the optimization objective materially shapes prognostic behavior and that multi-objective search provides a practical mechanism for calibrating prediction timeliness in RUL modeling.

1 Introduction

Early and late RUL errors have asymmetric operational costs, but common accuracy metrics treat them equally. This study compares single- and multi-objective optimization across architectures and datasets to examine how objective choice shapes prognostic behavior.

1 Introduction

  • Early and late RUL errors carry different operational consequences, while RMSE and R2 treat both directions equally.The NASA Prognostics Score is an asymmetric alternative that penalizes late predictions more heavily.Source: §1 · p. 1
  • Whether single- and multi-objective optimization produce distinct prognostic profiles beyond aggregate accuracy remains largely unexamined.Prior multi-objective RUL studies commonly trade prediction error against computational cost without a principled final selection procedure.Source: §1 · pp. 1–2
  • The study compares five architectures under three optimization regimes across 75 model–dataset–strategy combinations on C-MAPSS and BackBlaze.The regimes maximize R2, minimize NASA Score, or jointly optimize both with NSGA-II and Entropy–CRITIC selection.Source: §1 · p. 2
  • The contributions are a controlled comparison of optimization regimes, Entropy–CRITIC Pareto selection, and validation across simulated and real-world benchmarks.These design choices target objective effects, preference-free selection, and the data-fidelity gap, respectively.Source: §1 · p. 2

2 Literature Review

The literature review motivates studying RUL prediction across simulated and operational data and moving beyond single-metric hyperparameter optimization. This study compares architectures and optimization regimes across both domains, with Entropy–CRITIC used to select among Pareto solutions.

2.1 RUL Prediction

  • RUL is the duration a component can continue its intended function before failure and supports condition-based maintenance.The review also describes progression from traditional machine learning toward deep architectures for sensor data.Source: §2.1 · p. 2
  • True operational datasets account for approximately 24% of data used in recent RUL literature, raising concerns about generalization from simulated benchmarks alone.C-MAPSS provides run-to-failure trajectories but may not reproduce the complexity of real operating conditions.Source: §2.1 · p. 2

2.2 HPO in RUL Prediction

  • Single-objective optimization focuses on one accuracy metric, whereas multi-objective methods seek Pareto solutions across competing objectives.Hyperparameters such as layer depth, batch size, and learning rate influence learning and generalization, and their relationship to generalization error is typically non-convex and non-differentiable.Source: §2.2 · p. 2

2.3 This Study

  • The study compares three optimization strategies across five architectures and C-MAPSS and BackBlaze, using Entropy–CRITIC weighting for Pareto-front selection.Its evaluation combines standard metrics with asymmetric scoring and directional indicators, and experiments use leakage-free temporal validation.Source: §2.3 · pp. 2–3

3 Methodology

The methodology compares single- and multi-objective hyperparameter optimization strategies for RUL prediction using benchmark datasets, candidate algorithms, optimization frameworks, and evaluation metrics.

3 Methodology

  • The experimental methodology is designed to compare single-objective and multi-objective hyperparameter optimization strategies for RUL prediction.Source: §3 · p. 3
  • The methodology describes benchmark datasets, candidate algorithms, optimization frameworks, and evaluation metrics.Source: §3 · p. 3
  • The design draws on established prognostics benchmarks and evaluation protocols while extending them through a systematic multi-objective optimization perspective.Source: §3 · p. 3

3.1 Datasets

The study uses four simulated C-MAPSS subsets and an operational BackBlaze hard-drive dataset. The datasets differ in operating conditions, trajectory characteristics, and signal noise.

3.1.1 C-MAPSS

  • C-MAPSS contains four turbofan subsets spanning one or six operating conditions and one or two fault modes, with 14 sensors recorded each cycle.RUL is capped at 125 cycles; training engines are split into training and validation sets, with separate test engines.Source: §3.1.1 · p. 3

3.1.2 BackBlaze

  • BackBlaze provides daily SMART readings from operational drives and has noisier signals, less predictable degradation, and more heterogeneous failure modes than C-MAPSS.The study retains drives with recorded failures and filters out short histories before selecting 14 features and clipping RUL at 60 days.Source: §3.1.2 · pp. 3–4
  • BackBlaze includes 2,444 failed drives, split by drive into 1,956 training and 488 test drives.Across the five datasets, trajectory lengths are 206–247 cycles on C-MAPSS versus 57.6 days on BackBlaze, and the share of observations at the RUL cap is 39–49% versus 10.9%.Source: §3.1.2 · p. 4

3.2 Candidate Algorithms

The candidate set spans feedforward and tree baselines alongside recurrent, convolutional, and attention-based temporal models, supporting comparison across model families.

3.2.1 Multilayer Perceptron (MLP)

  • The MLP processes flattened input windows without an explicit temporal inductive bias, providing a simple baseline for testing whether optimization benefits depend on model complexity.The study also uses it to assess whether simpler architectures can leverage multi-objective search.Source: §3.2.1 · p. 4

3.2.2 Long Short-Term Memory (LSTM)

  • The LSTM uses gated memory cells to capture long-range temporal dependencies, with tunable depth, hidden dimension, and dropout regularization.Source: §3.2.2 · p. 4

3.2.3 XGBoost

  • XGBoost is a gradient-boosted tree ensemble over tabular features, using sliding-window statistical aggregates to capture temporal patterns indirectly.It provides a non-neural baseline that is described as robust to hyperparameter choices and typically less computationally intensive than deep learning alternatives.Source: §3.2.3 · p. 4

3.2.4 Temporal Convolutional Network (TCN)

  • The TCN models temporal dependencies with dilated causal convolutions, whose receptive field grows exponentially with network depth.Its design also maintains parallelizability and stable gradient flow.Source: §3.2.4 · p. 5

3.2.5 Transformer

  • The encoder-only Transformer uses multi-head self-attention to model pairwise interactions and dynamically weight the relevance of time steps in a degradation trajectory.Source: §3.2.5 · p. 5

3.3 Hyperparameter Optimization Strategies

The study compares hyperparameter search strategies that differ in whether they optimize one criterion or jointly optimize accuracy and the NASA Score.

3.3 Hyperparameter Optimization Strategies

  • All three strategies use Optuna with the Tree-structured Parzen Estimator sampler.Source: §3.3 · p. 5

3.3.1 Single-Objective Optimization

  • The single-objective regimes either maximize validation R2 or minimize the NASA Prognostics Score, which penalizes late predictions more heavily than early ones.The NASA Score’s sensitivity to outliers distinguishes configurations tuned under that objective from those optimized for R2.Source: §3.3.1 · p. 5

3.3.2 Multi-Objective Optimization

  • The multi-objective regime uses NSGA-II to jointly minimize 1 − R2 and NASA Score, then Entropy–CRITIC weighting to select a configuration from the Pareto front.CRITIC incorporates objective contrast and correlation; the combined weights select the solution with the highest composite score, deriving preferences from the Pareto front when no expert weights are available.Source: §3.3.2 · p. 5

3.4 Evaluation Metrics

The evaluation uses multiple metrics to characterize accuracy, safety-critical error, prediction precision, and directional bias rather than relying on one measure.

3.4 Evaluation Metrics

  • The evaluation combines regression accuracy, safety-critical scoring, prediction precision, and directional bias because no single metric fully characterizes prognostic performance.Source: §3.4 · p. 5

3.4.1 Regression Accuracy

  • R2 measures explained variance, while RMSE measures average prediction deviation with greater emphasis on large errors.Both are widely used as baseline performance measures in prognostics.Source: §3.4.1 · p. 6

3.4.2 NASA Prognostics Score

  • The NASA Prognostics Score applies asymmetric exponential penalties, with late predictions receiving a steeper penalty than early predictions.Late predictions underestimate true RUL, whereas early predictions overestimate it.Source: §3.4.2 · p. 6

3.4.3 Alpha-Accuracy

  • The α-accuracy metric reports the percentage of predictions within a specified tolerance of true RUL, evaluated here at α = 20%.It indicates the share of predictions accurate enough for operational decision-making thresholds.Source: §3.4.3 · p. 6

3.4.4 Directional Imbalance

  • Directional Imbalance measures asymmetry between early and late predictions; 0% indicates a balanced profile, while higher values indicate greater asymmetry.Early and late rates count predictions that overestimate and underestimate true RUL, respectively.Source: §3.4.4 · p. 6

3.5 Experimental Setup

The experiments compare five algorithms, five datasets, and three optimization strategies, with held-out test metrics and training-only preprocessing.

3.5 Experimental Setup

  • The design yields 75 algorithm–dataset–strategy configurations, with 30 search trials per configuration and early stopping after 15 consecutive trials without improvement, subject to at least 10 completed trials.Source: §3.5 · p. 6
  • C-MAPSS and BackBlaze use train–validation–test splits; preprocessing is fitted only on training data, and test metrics are held out from hyperparameter selection and early stopping.C-MAPSS uses a piecewise linear RUL target capped at 125 cycles and retains 14 informative sensor channels; multi-objective configurations are selected from the NSGA-II Pareto front using Entropy–CRITIC weighting.Source: §3.5 · p. 7

4 Experimental Results

Across C-MAPSS, multi-objective optimization reduces directional imbalance, while the strategies diverge substantially on BackBlaze. Algorithm rankings also vary by dataset and optimization configuration, with simpler models often leading on C-MAPSS.

4.1 Effect of the Optimization Strategy

  • 33% lower average directional imbalance (DI) on C-MAPSS: Multi-Objective achieves this versus Single-R2 (9.06) and Single-NASA (9.11).The reduction ranges from 23% on FD001 to 45% on FD004, where NSGA-II’s population-based exploration maintains diverse candidates.Source: §4.1 · pp. 7–8, Table 3 · p. 8
  • On BackBlaze, Single-R2 leads R2 (0.335), while Single-NASA lowers NASA Score by 50% versus Single-R2 (27,641 vs. 55,431) at a 34% R2 cost.Multi-Objective achieves a NASA Score of 32,133, comparable to Single-NASA while retaining higher R2; Single-R2’s lowest DI (1.45) reflects near-random predictions, not improved calibration.Source: §4.1 · p. 8, Table 3 · p. 8
  • Entropy–CRITIC weights are nearly equal for most C-MAPSS combinations but shift by architecture on BackBlaze, where TCN and Transformer favor NASA Score and MLP favors R2.BackBlaze weights include 0.68–0.70 NASA Score for TCN and Transformer, 0.46/0.54 for XGBoost, and 0.75 for R2 with MLP.Source: §4.1 · p. 8

4.2 Algorithm Comparison

  • XGBoost ranks first on R2 (0.916, 0.911, 0.907), RMSE, and NASA Score on FD001–FD003, and has the best α-20% and DI on FD001.Table 4 averages each algorithm’s results across the three optimization strategies.Source: §4.2 · p. 8, Table 4 · p. 9
  • On FD004, LSTM leads with R2 (0.891), RMSE (14.19), and DI (1.88), while its DI remains lowest across C-MAPSS at 1.33–5.33.It overtakes XGBoost, which has R2 = 0.885 and DI = 3.09 on FD004.Source: §4.2 · pp. 8–9, Table 4 · p. 9
  • TCN and Transformer occupy the bottom two positions across C-MAPSS; TCN has the lowest R2 on all four subsets, and Transformer reaches DI 18.21 on FD002.The Transformer also has the highest NASA Scores on FD002 (1,199.0) and FD004 (1,562.7).Source: §4.2 · p. 9
  • On BackBlaze, R2 drops from 0.86–0.92 on C-MAPSS to 0.24–0.30, and α-20% falls from 61–75% to 22–24%.The ranking shifts: TCN rises from last to third (R2 = 0.276), LSTM falls to fourth (0.252), and Transformer remains last; dataset noise dominates model-specific differences.Source: §4.2 · p. 9, Table 4 · p. 9
  • Multi-objective optimization affects architectures differently: it reduces DI by 76% for TCN and 43% for MLP, while LSTM remains low and Transformer remains high.It gives XGBoost, MLP, and LSTM their highest C-MAPSS R2 values (0.907, 0.901, 0.901), but the lowest R2 for TCN (0.855) and Transformer (0.865).Source: §4.2 · p. 9

5 Conclusion

The optimization objective reshapes RUL prediction behavior more than aggregate accuracy, while architecture rankings depend on configuration. The study also identifies limits in its objective scope and evaluation domains.

5 Conclusion

  • On C-MAPSS, multi-objective optimization reduced directional imbalance by 33% while strategies reached near-identical R^2; on BackBlaze, a generalization gap and strategy divergence emerged.The objectives shifted from complementary on C-MAPSS to conflicting on BackBlaze.Source: §5 · p. 10
  • Architecture rankings depended on configuration: XGBoost led in simpler conditions, LSTM in complex multi-condition scenarios, and rankings reversed on BackBlaze.Source: §5 · p. 10
  • Entropy–CRITIC weights were balanced on C-MAPSS but diverged by architecture on BackBlaze, indicating a shift in the relationship between objectives.The authors caution against adopting simulation-derived objectives without domain-specific validation.Source: §5 · p. 10
  • The evaluation used only two objectives, one industrial domain, independent architectures, and static post-hoc Entropy–CRITIC selection without expert preferences.The authors also note that fixed compute budgets may have disadvantaged deeper architectures with larger search spaces.Source: §5 · p. 10
  • Future work should expand search budgets and objectives, evaluate more domains, test hybrid approaches and adaptive Pareto selection, and explore probabilistic RUL frameworks.Source: §5 · p. 10

A Hyperparameter Search Spaces

Table 5 presents the hyperparameter search spaces for each algorithm.

A Hyperparameter Search Spaces

  • Table 5 lists the hyperparameter search spaces for each algorithm.Source: Table 5 · p. 13

Related papers

Loading 2610.01530v1…