Source-linked AI summary
FedCMAPSS: A Benchmark for Federated Learning in Remaining Useful Life Estimation
Amelia Sorrenti, Matteo Pennisi, Concetto Spampinato, Simone Palazzo
TL;DR
Robust federated RUL estimation lacks consistent evaluation settings despite scarce and private run-to-failure data motivating federated learning. FedCMAPSS addresses this gap with five standardized C-MAPSS tasks, reproducible protocols, public splits and code, and systematic comparisons across architectures and federated methods. Results show FedAvg is strong in IID settings, drift-mitigation methods are generally more reliable under domain shift and label skew, and few-shot federation is the most challenging.
Problem
Federated RUL research lacks consistent evaluation settings and client partitions, while centralized run-to-failure data are scarce and may not be shareable.
Method
FedCMAPSS defines five C-MAPSS tasks spanning IID, statistical heterogeneity, and data scarcity, with reproducible protocols, randomized splits, public code, and comparisons across architectures and FL methods.
Results
FedAvg performs strongly in IID settings, drift-mitigation strategies are generally more reliable under domain shift and label skew, and few-shot federation is the most challenging.
Takeaways & Limitations
FedCMAPSS provides a common reproducible foundation for fair and transparent comparison of federated predictive-maintenance methods.
Abstract
from arXiv · showhide
Data-driven prognostics and health management has emerged as a key enabler for Industry 4.0, yet the development of robust remaining useful life (RUL) estimation models is often limited by the scarcity of run-to-failure data. While federated learning offers a promising paradigm to collaboratively train predictive models without sharing sensor data, research efforts have operated so far in the absence of a common evaluation framework. To address this gap, this paper introduces FedCMAPSS, a benchmark for federated RUL estimation based on the commonly-used NASA C-MAPSS dataset. We define a set of five standardized tasks designed to simulate real-world industrial challenges, ranging from ideal IID settings to extreme statistical heterogeneity, and conduct a systematic evaluation of state-of-the-art federated optimization algorithms across multiple neural architectures. By establishing reproducible baselines and making the source code and data splits publicly available, this work aims to provide a standard foundation for developing and comparing federated predictive maintenance solutions.
I. INTRODUCTION
Predictive maintenance increasingly relies on deep neural networks for RUL estimation, but robust models require scarce centralized run-to-failure data that organizations may not share. Federated learning enables collaborative optimization without transmitting raw sensor data, while inconsistent evaluation protocols prevent fair comparison of federated RUL methods.
- Deep neural networks have become standard in PHM because they automatically extract high-level representations from multidimensional sensor data.
- RUL estimation models require large centralized run-to-failure datasets, which organizations may lack or avoid sharing because of intellectual-property and privacy concerns.
- Federated learning addresses data-sharing constraints by collaboratively optimizing a global model through gradient updates or network weights rather than raw data.
- FedAvg can exhibit learning oscillations and high uncertainty when multilayer perceptrons train across heterogeneous sensor distributions.
- Existing federated RUL studies use inconsistent client partitions and evaluation protocols on C-MAPSS, making cross-paper comparisons impossible under statistical heterogeneity.
- FedCMAPSS introduces standardized C-MAPSS tasks, comparisons of federated algorithms and neural architectures, reproducible splits, and publicly available code.
II. RELATED WORK
Related work positions federated RUL estimation within data-driven PHM and highlights the importance of consistent datasets, preprocessing, client partitions, and task definitions. FedCMAPSS organizes C-MAPSS into five reproducible tasks spanning IID, heterogeneous, and few-shot settings.
- RUL estimation predicts continuous degradation trajectories and therefore requires models capable of capturing long-term temporal dependencies in sensor streams.
- Federated RUL studies commonly use FedAvg to aggregate local model parameters without sharing raw data across multilayer perceptron and recurrent architectures.
- Evaluation studies vary in client partitioning, preprocessing, and target normalization, with non-IID splits complicating training relative to homogeneous splits.
- FedCMAPSS uses C-MAPSS, which simulates aircraft-turbofan degradation with multidimensional sensor readings across four sub-datasets of varying complexity.
- The benchmark creates 10 independent splits per task and separates client data into local training, global evaluation, and full testing subsets.
- The five tasks cover IID partitioning, domain shift, label skew, feature skew, and few-shot federation through distinct C-MAPSS client assignments.
C. Architectures
The benchmark compares recurrent, convolutional, and attention-based neural architectures for federated RUL prediction. The evaluated models range from simple LSTM and RNN designs to CNN, AFT, and attention-equipped bidirectional GRU architectures.
- The benchmark evaluates recurrent, convolutional, and attention-based architectures for federated RUL prediction.
- LSTM is implemented as a single-layer recurrent network for RUL estimation.
- RNN uses five progressively narrower recurrent layers followed by a multilayer perceptron for prediction.
- CNN treats the multivariate time series as a 2D matrix, applies asymmetric convolutional kernels, and uses an MLP for RUL estimation.
- AFT processes the input sequence with an Attention-Free Transformer, two 2D convolutional layers, and an MLP.
- AttBiGRU combines a bidirectional GRU with 4head self-attention, then flattens the attention map for MLP prediction with dropout.
D. Federated methods
The benchmark compares classical and non-IID-oriented federated optimization methods. These methods differ in how they average updates, control client drift, regularize local objectives, or aggregate multiple intermediary models.
- The compared methods include classical baselines and newer approaches designed for non-IID federated training.
- FedAvg is the classical baseline, aggregating locally trained client updates through weighted averaging based typically on local sample sizes.
- SCAFFOLD mitigates client drift under heterogeneous data using control variates maintained by both server and clients.
- FedDyn uses dynamic regularization to align local optima with the global objective, particularly under non-IID data.
- FedCross replaces a single global model per round with multi-model cross-aggregation through multiple middleware models.
E. Evaluation metrics
The benchmark evaluates RUL prediction with RMSE and NASA Score, which both measure prediction performance but differ in interpretability and error sensitivity.
- RMSE measures the average magnitude of prediction errors in the same unit as the target RUL.
- The NASA Score asymmetrically penalizes late predictions, or overestimated RUL, more severely than early predictions.
- Lower values indicate better performance for both RMSE and NASA Score.
- The NASA Score is reported cumulatively over all test samples and is relevant when overly optimistic RUL estimates could delay maintenance interventions.
F. Evaluation protocol
The evaluation protocol preprocesses temporal sensor windows on clients, trains models with MSE-based objectives, and evaluates federated methods across repeated randomized splits. Federated evaluation uses fixed training rounds without a dedicated validation set.
- Sensor signals are split into fixed-length temporal windows and paired with corresponding RUL labels using client-side preprocessing.Preprocessing includes normalization and windowing before local training.
- Local optimization minimizes mean squared error between predicted and ground-truth RUL, with additional loss terms allowed for specific FL algorithms.
- Federated training runs for 100 rounds with 1, 5, or 10 local epochs per round.
- The framework omits a dedicated federated validation set and evaluates the global model from the final communication round.Alternative model-selection criteria require researchers to create local validation splits independently.
- All model and FL-algorithm combinations are evaluated on 10 independent random splits, reporting mean and standard deviation for RMSE and NASA Score.
IV. EXPERIMENTAL RESULTS
The experiments compare local training, centralized optimization, model architectures, and federated algorithms across five benchmark tasks. Local-only training is used as an empirical lower-bound baseline, and all models struggle in the isolated setting.
- The benchmark evaluates isolated local training and full-data centralized optimization across five defined tasks.
- The experiments compare neural network architectures and federated learning algorithms.
- A. Local baseline: Local-only training provides an empirical lower-bound baseline because each client uses only its isolated partition without parameter synchronization.
- A. Local baseline: All models struggle when each task is trained locally, revealing a pronounced difficulty gap in the isolated setting.Results are reported in terms of RMSE and averaged across architectures within each task.
B. Task evaluation
Across the federated task evaluations, performance depends on the data-distribution regime, architecture, and optimization method. Non-IID settings favor drift-mitigation methods in some tasks, while extreme few-shot heterogeneity is the most challenging.
- Task B’s non-IID operating regimes and fault patterns favor drift-mitigation methods, with FedCross and SCAFFOLD strongest across architectures.
- Task B’s NASA Score can produce very large, comparable means and standard deviations because exponential penalties amplify a small number of large prediction errors.The tables report NASA Score values scaled by 10^-3 for readability.
- The evaluation tables organize results by neural architecture and federated method for Tasks B–E.
- Task C has comparable difficulty under different client lifespan ranges, with FedCross most effective, especially for LSTM, AFT, and RNN.
- Task D is comparatively less critical: most methods perform solidly, federated-strategy gaps narrow, and FedDyn has significantly lower prediction accuracy.
- Task E is the most challenging setting, where highly limited and biased local datasets make LSTM and CNN the most stable architectures.
C. Sensitivity to optimization hyperparameters
The benchmark examines how local training epochs and global communication rounds affect federated RUL optimization, including communication overhead and convergence behavior. Final performance is largely robust to local-epoch count, while 100 rounds generally provide stable performance, with method-specific exceptions.
- Communication trade-offs: Communication overhead scales linearly with synchronization rounds when client count and model size are fixed.The cost is modeled as C = 2RNP, where R is rounds, N is participating clients, and P is shared trainable parameters.
- Local training epochs: Final generalization performance remains largely robust across 1, 5, and 10 local epochs in the tested non-IID tasks.The analysis averages RMSE across architectures while fixing global communication rounds at 100.
- Global communication rounds: FedDyn exhibits an almost-constant trend across communication rounds, questioning its suitability for these tasks.The paper also notes a trade-off between local computation and global synchronization in edge-oriented training.
D. Qualitative analysis of degradation trajectories
The qualitative analysis compares actual and predicted RUL trajectories across representative federated methods, architectures, and tasks. Prediction uncertainty narrows near failure, while models show a tendency to slightly overestimate RUL at the failure point.
- Task-specific targets: The target RUL’s constant-segment length varies across tasks, affecting how trajectories should be interpreted.The analysis attributes this variation to task-specific target construction.
- Trajectory uncertainty: Degradation trajectories expose task-dependent predictive uncertainty across different stages of a component’s lifecycle.The plots show average predicted RUL with confidence intervals computed across task splits.
- Trajectory uncertainty: As degradation approaches the failure threshold, increasingly pronounced fault signatures coincide with narrower error bars.This pattern reflects reduced uncertainty near failure as the degradation signal becomes clearer.
- Failure-point behavior: Predicted RUL tends to be slightly overestimated as the device reaches the failure point.The paper identifies this tendency as requiring further investigation.
V. CONCLUSIONS
FedCMAPSS addresses inconsistent evaluation in federated RUL research with a standardized, reproducible benchmark built on NASA C-MAPSS. Experiments reveal distinct method and task patterns, while further work is needed to broaden optimization, robustness, and dataset coverage.
- Benchmark contribution: FedCMAPSS defines five C-MAPSS tasks spanning IID, statistical heterogeneity, and extreme data scarcity.The benchmark also standardizes preprocessing, windowing, target construction, training schedules, and randomized splits.
- Empirical findings: FedAvg performs strongly in the IID regime, while drift-mitigation strategies are generally more reliable under domain shift and label skew.These conclusions come from experiments across multiple architectures and federated learning methods.
- Empirical findings: Feature skew is comparatively mild, whereas few-shot federation is the most challenging regime and is more sensitive to model architecture.The comparison reflects the benchmark’s tested setup.
- Optimization dynamics: Final performance is relatively robust to local-epoch count, and most methods reach a stable regime within 100 communication rounds.Additional rounds may still benefit some configurations.
- Future work: Future work includes federated hyperparameter optimization, additional FL paradigms, outlier-robust criteria, and more datasets.These planned extensions define the current scope boundary of the benchmark study.