PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Pretraining for PHM: What Transfers, What Does Not, and What Federating It Costs

José Sanz-Durán, David Solís-Martín

arXiv:2610.35280v1cs.AIcs.CEcs.LGcs.PF

TL;DR

PHM self-supervised pretraining has not been measured under leakage-controlled conditions, despite its potential fit to abundant unlabeled signals and scarce labels. The paper evaluates a harmonized corpus with datasets withheld from pretraining and compares centralized with federated pretraining. Transfer is task-dependent: classification gains do not reliably extend to RUL, and federated pretraining retains but does not match centralized classification performance.

  • Problem

    PHM combines abundant heterogeneous sensor signals with scarce labels, while cross-domain self-supervised transfer and federated pretraining have not been empirically assessed under a controlled leakage protocol.

    Source: §1 · pp. 1–2

  • Method

    The paper harmonizes public PHM datasets, withholds target datasets in full, and compares centralized and federated self-supervised pretraining using a fixed channel-independent encoder.

    Source: §1 · p. 2, §3.1 · p. 3, §4.1 · p. 5

  • Results

    Transfer is task-dependent: pretraining transfers strongly to fault classification but not reliably to CMAPSS RUL, while federated pretraining retains part of the representation without closing the classification gap.

    Source: §7 · p. 13, §9 · p. 14

  • Takeaways & Limitations

    The reported failures of transfer, loss-based checkpoint selection, and overly aggressive fine-tuning are specific and actionable within the evaluated setting.

    Source: §1 · p. 2

    Only one pretraining run per regime underlies the comparison, so its p-values do not support inference about the regimes.

    Source: §8 · p. 13

Abstract

from arXiv · show

Self-supervised pretraining has produced capable time-series foundation models, but its value for Prognostics and Health Management (PHM) has not been measured under leakage-controlled conditions. We measure it on a corpus of 47 public PHM datasets, harmonized to a single tensor contract and partitioned so that 11 datasets are withheld from pretraining in full. Transfer turns out to be task-dependent: a compact channel-independent Transformer encoder (801,808 parameters) pretrained with masked patch prediction raises fault-classification macro-F1 from 0.350 to 0.812 on CWRU and from 0.569 to 0.951 on HSG18, yet on CMAPSS remaining-useful-life estimation no pretrained checkpoint reliably improves upon a from-scratch baseline. We pretrain under two regimes---centralized, and simulated cross-silo federated with ten clients grouped by physical domain---holding architecture, objective, source data and optimization budget fixed. Two further findings are negative. At matched budget the centralized encoder leads by 15.5--20.2 macro-F1 points in four of four comparisons, although the comparison is confounded by a learning-rate schedule that favours the centralized arm and rests on a single pretraining run per regime; on RUL the two regimes are indistinguishable. And the intrinsic SSL loss is not a valid selection criterion: the centralized encoder more than halves the federated reconstruction loss without a corresponding gap on RUL, and a server-momentum sweep moves downstream validation macro-F1 by up to 0.29 while the SSL loss varies by 0.021. Transfer also hinges on the backbone learning rate: the same checkpoint yields 0.169 or 0.829 macro-F1 on CWRU depending on a single order of magnitude. We release the corpus pipeline, the protocol and the code.

1 Introduction

PHM combines abundant heterogeneous sensor data with scarce labels, while cross-domain reuse and federated self-supervised pretraining remain insufficiently measured under leakage-controlled evaluation. The paper fixes a standard backbone to measure when pretraining transfers and what it costs to federate.

1 Introduction

  • PHM faces abundant heterogeneous sensor signals but scarce, expensive, and ambiguous failure labels, motivating self-supervised pretraining.PHM tasks include fault classification, RUL estimation, and anomaly detection [2].Source: §1 · pp. 1–2
  • Random splits of overlapping windows and reuse of datasets across pretraining and testing can leak information and inflate time-series model evaluations.The paper argues transfer claims depend on the evaluation protocol as well as the model.Source: §1 · p. 2
  • The paper reports task-dependent transfer, absent reliable RUL transfer, a federated classification gap, loss-metric decoupling, and strong sensitivity to adaptation learning rate.It describes these failure modes as specific, reproducible, and actionable, while noting published evidence on failures in industrial settings is thin.Source: §1 · p. 2

2 Related work

  • Prior PHM studies often use task-specific architectures and single-benchmark protocols, leaving evidence about reusable representations limited [2].Time-series foundation-model mechanisms include channel-independent patching and masked patch prediction [5] [6], but the cited corpora and benchmarks are not built for PHM or federated pretraining.Source: §2 · pp. 2–3
  • The study measures transfer and federated-pretraining costs on withheld PHM datasets, controlling leakage and adaptation rather than proposing a new architecture.It positions its contribution as a controlled measurement of when the recipe works and when it does not.Source: §1 · p. 2, §2 · p. 3

3 Corpus and harmonization

The authors construct a 47-dataset PHM corpus, reserve 11 datasets entirely for evaluation, and harmonize accepted data to a shared tensor interface. Unit-level splits and automated checks are used to control leakage.

3.2 Tensor contract

  • Each physical-unit series is windowed into 512 samples at stride 256, then represented as 32 patches of 16 samples in a shared channel-independent tensor interface.The shared encoder processes different channel counts without changing its parameters; partial tails are retained with padding, and loss excludes padded samples.Source: §3.2 · pp. 3–4

3.1 Corpus construction and partition

  • 47 datasets met the structural contract; 36 were assigned to pretraining and 11 withheld entirely for downstream evaluation.No dataset was dropped on downstream performance; the accepted corpus followed explicit criteria applied during audit.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 Tensor contract

  • Normalization is computed per window and channel while ignoring padding, avoiding shared cross-client statistics and leakage at the cost of discarding absolute amplitude.Means and standard deviations are recorded so the normalization choice can be revisited.Source: §3.2 · p. 4

3.3 Splits and leakage control

  • Splits are formed by physical unit or trajectory before windowing, preventing overlapping windows from crossing partition boundaries.Automated checks found no unit or trajectory overlap and no test-data use in normalization; all 47 datasets passed.Source: §3.3 · pp. 4–5

4 Model and pretraining

This section defines a compact channel-independent encoder and masked-patch prediction objective, then compares centralized and simulated physical-domain federated pretraining; at matched budget, centralized training reaches less than half the federated validation reconstruction loss.

4.1 Backbone

  • PatchTSTPhm has 801,808 trainable parameters and uses a channel-independent patch formulation with a four-layer Transformer encoder.Its compact size is intended to fit the study's compute budget, keep federated transmission cheap, and isolate weight origin from capacity effects.Source: §4.1 · p. 5

4.2 Self-supervised objective

  • Masked patch prediction replaces 30% of valid patches with a learnable mask token and reconstructs them from visible context without labels.Padding is neither masked nor scored.Source: §4.2 · p. 5

4.3 Centralized regime

  • The centralized full run completed 99,961 of 100,000 requested steps, while its step-25,000 comparator uses a schedule mismatch that favors centralized training.Central-25k is a snapshot of the same run, not an independent training run.Source: §4.3 · pp. 5–6

4.4 Federated regime

  • Ten physical-domain clients perform 25,000 local steps in simulation, with heterogeneity in sensors, scales, lengths, and failure semantics.The simulation has no real organizations, network latency, client dropout, or formal privacy mechanism.Source: §4.4 · p. 6
  • At matched budget, the centralized encoder reaches less than half the federated validation reconstruction loss (Table 2).Table 2 reports intrinsic reconstruction loss, a diagnostic rather than evidence of downstream utility.Source: §4.4 · p. 6, Table 2 · p. 7

5 Downstream evaluation protocol

The evaluation tests pretrained checkpoints on two fault-classification datasets and one RUL-regression dataset using physical-unit splits. Adaptation and reporting rules distinguish validation-based model selection from test evaluation and identify limits on comparisons with single-seed baselines.

5 Downstream evaluation protocol

  • Fault classification is evaluated on CWRU and HSG18 with macro-F1, while CMAPSS RUL regression uses a target capped at 125 cycles.CMAPSS has 11,795/3,131/6,610 train/validation/test windows.Source: §5 · p. 7
  • The CWRU test split contains only three of four fault classes, so its macro-F1 averages those three; HSG18 test data are balanced across two classes.The CWRU test split is skewed (477/2378/476), while HSG18 has 2,288 windows per class.Source: §5 · p. 7
  • Adaptation compares training from scratch, linear probing, and full fine-tuning; RUL also uses a two-layer MLP head over a frozen encoder.Linear probing freezes the encoder; full fine-tuning retrains it with a small backbone step size.Source: §5 · p. 7
  • Configuration and epoch selection use validation, backbone learning rates are calibrated per dataset and checkpoint, and principal comparisons use three adaptation seeds.The candidate backbone rates are 1·10−6, 3·10−6, and 1·10−5; the test split is read once and never used for selection.Source: §5 · p. 7
  • From-scratch baselines use one seed while pretrained checkpoints use three, making their comparisons indicative of effect size rather than statistical reliability.Table 3 reports pretrained full-fine-tuning results as mean ± sample standard deviation over three seeds; scratch and linear-probing rows are single-seed references.Source: §5 · p. 7, Table 3 · p. 8
  • The budget-matched Central-25k versus federated comparisons use test macro-F1 and Welch’s t-test, but their p-values quantify adaptation noise rather than regime variability.Each arm has a single pretraining run; none of the four comparisons survives Bonferroni correction.Source: Table 4 · p. 8

6 Results

Pretraining strongly improves fault classification, but CMAPSS RUL shows no reliable benefit; federated classification trails centralized pretraining, and intrinsic loss does not reliably predict downstream utility. Transfer is sensitive to adaptation choices, especially the backbone learning rate.

6.1 Fault classification

  • CWRU macro-F1 rises from 0.350 to 0.812 and HSG18 macro-F1 from 0.569 to 0.951 with centralized pretraining; linear probing reaches 0.705 and 0.906, respectively.The linear-probe gains indicate that the pretrained representation accounts for much of the improvement, which appears in two physically unrelated domains.Source: §6.1 · p. 8
  • Central-25k leads the better federated checkpoint by 16.2 points on CWRU and 15.5 on HSG18, but the comparison rests on single pretraining runs and a schedule favoring the centralized arm.The lead holds in all four pairwise comparisons, but the CWRU ordering depends on the learning-rate rule; HSG18 is stable across the tested grid.Source: §6.1 · pp. 8–9

6.2 Remaining useful life

  • On CMAPSS, the validation-selected FedAvgM has test RMSE 44.04 versus Central-100k’s 44.16, while the single-seed from-scratch reference scores 43.67; the experiment establishes no reliable pretraining benefit.The from-scratch run uses a linear head, whereas the multi-seed pretrained rows use a two-layer MLP head over a frozen encoder, so head capacity is confounded with pretraining.Source: §6.2 · pp. 9–10, Table 5 · p. 9
  • On CMAPSS, centralized full fine-tuning scores 645,389 on the asymmetric benchmark metric versus 754,754 from scratch, despite RMSE being 0.65 cycles worse.This single-seed comparison shows that the task conclusion depends on the error metric.Source: §6.2 · p. 10, Table 6 · p. 10
  • Every CMAPSS test mode has negative R2 because test trajectories are truncated before failure, limiting interpretation of absolute prognostic quality; linear probing also reaches 53.86 test RMSE versus 43.67 from scratch.Relative comparisons remain informative only under the same protocol, and a frozen embedding with a two-layer head reaches the 42.7–44.4 test-RMSE band.Source: §6.2 · p. 10

6.3 The intrinsic loss does not predict downstream utility

  • At matched budget, centralized reconstruction loss is 0.3467 versus 0.7844 for federated pretraining, yet the regimes are indistinguishable on CMAPSS and CWRU ordering depends on adaptation rate.The FedAvgM sweep likewise has SSL validation loss varying within 0.021 while downstream validation macro-F1 varies by 0.21 on CWRU and 0.29 on HSG18.Source: §6.3 · pp. 10–11, Figure 3 · p. 11, Table 7 · p. 11
  • In a matched 5,000-step screening run, FedProx reduces reconstruction loss by 11.57% versus FedAvg’s 7.55%, and its downstream behavior is mostly better in that screening.Its advantage did not persist at 25,000 steps, where FedAvgM dominated it.Source: §6.3 · p. 11

6.4 The backbone learning rate governs whether transfer survives

  • On CWRU, the same Central-100k checkpoint scores 0.169 test macro-F1 at 1·10−4 and 0.829 at 1·10−5; validation-selected backbone rates also differ by checkpoint.The aggressive rate overwrites the pretrained representation, while per-checkpoint validation calibration is necessary for meaningful comparisons.Source: §6.4 · p. 12, Figure 4 · p. 12

7 Discussion

The discussion interprets classification and RUL transfer as task-dependent and the federated deficit as a joint cost under heterogeneous clients. These conclusions are bounded by the encoder’s scale and the study protocol.

7 Discussion

  • Classification may favor reconstruction pretraining because local signal morphology is window-level, whereas RUL requires mapping trajectory-wide evidence to a global degradation coordinate.The authors present this as an interpretation, not a demonstrated mechanism; testing it would require explicit trajectory-level aggregation.Source: §7 · p. 12
  • Federated pretraining retains classification value but trails centralized pretraining and is less stable across seeds; the experiment measures their joint cost without separating its possible causes.The proposed explanation is client drift under strong non-IID conditions, where clients optimize distinct physical distributions and the server averages once per round.Source: §7 · p. 13
  • At 0.8M parameters and under this protocol, the study finds task-dependent transfer, an unclosed federated gap, and metric decoupling; larger models could change some conclusions, especially on RUL.The authors consider metric decoupling more likely to persist across scales because it concerns the relationship between metrics rather than model capacity.Source: §7 · p. 13

8 Threats to validity and limitations

The reported comparisons are constrained by optimizer-schedule and baseline-calibration mismatches, single pretraining runs, and task-specific data and evaluation limitations.

8 Threats to validity and limitations

  • The centralized-versus-federated classification comparison is an upper bound on federation cost because the optimizer schedules differ and favor the centralized arm.Only one pretraining run was conducted per regime, so the reported p-values do not support inference about regime effects.Source: §8 · p. 13
  • From-scratch baselines use one uncalibrated learning rate, while pretrained checkpoints receive validation-selected rates, making reported classification gains upper bounds.The baseline learning-rate grid was committed but not executed.Source: §8 · p. 13
  • CMAPSS uses a separate causal-window pipeline with stride 5, and its median 184-cycle trajectory is shorter than the 512-sample window, so every window is padded.The RUL null may reflect this mismatch; the authors call for a W ∈{64, 128} ablation before attributing it to pretraining.Source: §8 · p. 13
  • Evaluation power is limited by three downstream seeds, federated standard deviations up to 0.119, and single-seed from-scratch baselines.The classification comparison also lacks a completed matched-schedule centralized run.Source: §8 · p. 13
  • The evaluation omits class coverage checks, uses windows spanning only 0.3–1.3 shaft revolutions on CWRU, and evaluates only three of eleven held-out targets across multiple seeds.The authors also note that fixed 512-sample windows do not represent a fixed physical duration and instance normalization discards potentially diagnostic absolute amplitude.Source: §8 · p. 14
  • The ten-client federated simulation models domain heterogeneity but omits latency, partial participation, secure aggregation, and differential privacy.The fixed-architecture comparison also lacks external PHM and time-series foundation-model baselines; the RUL scratch baseline uses a different head.Source: §8 · p. 14

9 Conclusion

Under a leakage-controlled protocol on 47 harmonized PHM datasets, the paper finds that self-supervised pretraining transfers differently across tasks and that its intrinsic loss does not select downstream-useful checkpoints. It recommends full dataset holdouts, checkpoint-specific learning-rate calibration, and reporting linear probing alongside fine-tuning.

9 Conclusion

  • Pretraining transfers strongly to fault classification across two physical domains but does not reliably improve CMAPSS RUL; federated pretraining retains part of the representation yet leaves a classification gap.On RUL, validation and test disagree, while lower reconstruction loss does not correspond to better RUL performance.Source: §9 · p. 14
  • The intrinsic self-supervised loss is not a valid criterion for selecting pretrained checkpoints.A regime with more than half the reconstruction loss is indistinguishable on RUL, while downstream utility can vary much more than the intrinsic metric.Source: §9 · p. 14
  • Withhold evaluation datasets in full, split by physical unit before windowing, calibrate adaptation rates per checkpoint, and report linear probing alongside fine-tuning.The authors warn that an uncalibrated adaptation rate risks attributing optimizer effects to the representation.Source: §9 · p. 14

Reproducibility

The paper makes its code, configurations, corpus construction, and audit artifacts available, with reported numbers linked to versioned run records. Raw datasets are redistributed through PHMD rather than mirrored in the repository.

Reproducibility

  • Code, configurations, corpus construction, and audit artifacts are available in the project repository, and each reported number is traceable to a versioned run record.Run records include configuration hash, seed, commit, and resulting metrics.Source: Reproducibility · pp. 14–15
  • Raw datasets are redistributed by PHMD and are not mirrored in the repository.Source: Reproducibility · p. 15
  • This article extends the first author’s 2026 master’s thesis with significance analysis, a FedProx re-examination, and the first three items of Section 8.Configurations for the three experiments identified as most needed are committed and dry-run validated.Source: Reproducibility · p. 15

A Coverage of the held-out target datasets

Eleven datasets were withheld from pretraining, and Table 8 documents the status of each held-out target dataset.

A Coverage of the held-out target datasets

B A negative case: PBCP16

On PBCP16, a strong from-scratch score leaves no headroom for pretraining, bounding its benefit on this task. The CWRU macro-F1 also conceals class-level details, including an unsupported test class and class-0 errors.

B A negative case: PBCP16

  • 0.907 macro-F1 from scratch on PBCP16 leaves no headroom for pretraining, which does not help this small, balanced five-class task.The from-scratch result bounds the claim: pretraining is not universally beneficial, and reporting only CWRU and HSG18 would overstate its scope.Source: B A negative case: PBCP16 · p. 15

C Per-class breakdown on CWRU

  • Class 3 has no test support, so CWRU's 0.8292 test macro-F1 averages only three classes.Source: C Per-class breakdown on CWRU · p. 15
  • Class 0 has precision 0.480 despite near-perfect recall, with 477 class-1 windows predicted as class 0.Whether this error matters depends on which class encodes the healthy state; macro-F1 alone does not answer that operational question.Source: C Per-class breakdown on CWRU · p. 15

Related papers

Loading 2610.35280v1…