Reliable Virtual Sensing: A Multi-Domain Benchmark for Robustness Under Sensor Failures

• MuViS-C evaluates ten sensor failure modes across channels and severities using three complementary robustness measures, including a new measure of worst-case fragility.The benchmark is open-source and extensible to additional datasets, failure modes, measures, and models.Source: §1 · p. 2
• Across nine datasets from six domains, model rankings shift substantially, and no single domain captures the full robustness picture.Source: §1 · p. 2
• Six architectures spanning gradient-boosted trees and major sequence-modeling inductive biases reveal pervasive fragility, while gradient-boosted trees provide a strong implicit-robustness baseline.Source: §1 · p. 2
• On the attention-based architecture, three representative robustification strategies establish reference points and quantify a trade-off between nominal performance and robustness.Source: §1 · p. 2

• The MuViS benchmark consolidates datasets across six physical domains, but evaluates only nominal conditions, leaving model behavior under degraded sensors open.Source: §2 · pp. 2–3
• Physical sensor taxonomies converge on recurring fault archetypes tied to sensing principles, providing a grounded vocabulary for robustness benchmarks.Source: §2 · p. 3
• Computer vision corruption benchmarks established systematic robustness evaluation, while adjacent time-series studies report sensitivity to sensor faults and data-quality issues; virtual sensing still lacks a comparable multi-domain benchmark [9] [14] [18] [19].Source: §2 · p. 3
• Robustness approaches include adversarial training, certified methods, domain adaptation, representation learning, self-supervised sensor-fault pretraining, and input sensor dropout.Source: §2 · p. 3

3 The MuViS-C benchmark

• MuViS-C perturbs individual input channels and measures the resulting change in a scalar virtual-sensing estimate from a multichannel time window.Figure 2 summarizes the dataset domains, failure injection, test grid, and aggregation of degraded RMSEs.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Datasets

• The evaluation spans nine datasets from six application domains, including environmental and health sensing, industrial and energy systems, automotive, and motorsport.The datasets target quantities such as particulate concentration, heart rate, chemical concentration, battery state of charge, vehicle lateral velocity, and tire temperature.Source: §3.1 · pp. 3–4

3.2 Severity-parameterized sensor failures

• The failure catalogue defines ten modes that act on one channel and parameterizes each mode’s perturbation intensity from minimal or unchanged at severity 0 to maximal at severity 1.After per-channel z-score normalization, each evaluation perturbs exactly one channel while the others remain clean; k is fixed across the benchmark.Source: §3.2 · p. 4

3.3 Evaluation protocol

• The protocol sweeps 20 severity levels per failure-channel pair and reports mPC for average corrupted RMSE, rPC for degradation relative to clean performance, and scross for the earliest crossing of naïve-predictor error.If no crossing occurs in the measured range, scross is undefined; reading all three measures together distinguishes absolute error, average degradation, and worst-case fragility.Source: §3.3 · pp. 5–6

3.4 Standardized interface

• Users can register a dataset and models, optionally extend failure modes or metrics, then run the testbed to obtain results for inspection and export.Source: §3.4 · p. 6

4 Experiments and results

• The study benchmarks six architectures spanning tree ensembles and sequence-modeling biases, then compares input dropout, adversarial training, and self-supervised pretraining on a shared TST backbone.Omnibus tests are significant for mPC and rPC (both p < 0.001), but not for scross (p = 0.187).Source: §4 · pp. 6–7

4.1 Gradient-boosted trees as a strong robustness baseline

• XGBoost leads non-robustified models on rPC and achieves the best mPC mean rank (2.3), while no neural architecture reaches the top position on both measures.CatBoost is second on mPC with a mean rank of 2.6; XGBoost trails only TST-PGD on rPC.Source: §4.1 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 7

4.2 Trade-off between nominal performance and robustness

• All three robustification strategies lower TST’s rPC while increasing clean-data error, and each joins the leading non-significant rPC cluster.None establishes a significant advantage over the strongest non-robustified baselines.Source: §4.2 · p. 8, Figure 3 · p. 8

4.3 Benchmark diversity across datasets

• Model rankings shift across datasets: M-TCN is best on heart-rate mPC but worst on Vehicle Dynamics, while TST-F2F leads on Batt. and REVS scross but collapses on HR and TEP.TEP is easiest, with six of nine models within 1% of clean performance; Battery is challenging, with every model at least doubling clean error.Source: §4.3 · pp. 8–9

4.4 Additional analyses

• Across failure modes, rPC ranges from 1.073 for varying trimming to 1.922 for hard faults, and no model ranks best on more than four modes.The mode leader varies: XGBoost and robustified TST variants take different wins across the ten modes.Source: §4.4 · p. 9
• With a reliable fault detector and faulty-channel replacement by its training-set mean, TST-ISensD reaches geometric-mean rPC 1.034 and mean rank 1.0 across nine datasets.The authors attribute this result to ISensD’s channel-zeroing training procedure, which prepares the model for that intervention.Source: §4.4 · p. 9

5 Limitations and future work

• The ten-mode catalogue covers dominant sensor-failure archetypes but is not exhaustive; the injection framework is designed to incorporate newly characterized modes.The authors plan to broaden the catalogue as field deployments surface additional patterns.Source: §5 · p. 9
• Robustification comparisons cover three representative methods on a shared TST backbone; other defense families and transfer across backbones remain open questions.The authors plan to use MuViS-C to study both directions.Source: §5 · p. 9
• The evaluation perturbs one sensor at a time, leaving correlated multi-sensor failure schedules and channel-coverage sweeps for future benchmark extensions.The authors describe these as a natural next step for MuViS-C.Source: §5 · p. 9
• Perturbation scale is fixed at k = 3, so sensitivity of model rankings to alternative scales remains untested.The chosen scale follows the 3σ convention on z-scored data.Source: §5 · p. 9

6 Conclusion

• Every evaluated model becomes worse than a naïve predictor on at least one failure-mode/channel combination.The result indicates that substantial degradation occurs across the evaluated models, not just one architecture.Source: §6 · p. 9
• Gradient-boosted tree ensembles achieve strong robustness without an explicit defense, setting a challenging baseline for neural methods.Source: §6 · p. 9
• Dedicated robustification can move a fragile neural baseline into the leading robustness cluster, but every method incurs a clean-data error penalty.Source: §6 · p. 9
• With reliable failure detection and input intervention, ISensD reduces average degradation to near zero.Source: §6 · p. 9
• Across the benchmark, model rankings shift across domains, and no single dataset captures the full robustness picture.MuViS-C is presented as an open-source, extensible testbed for this evaluation.Source: §6 · p. 9

A Data and preprocessing

• Dataset selection, splits, and preprocessing are reused unchanged from MuViS; Table 5 summarizes the resulting dataset statistics.The pipeline includes cleaning, resampling, channel selection, and sliding-window construction.Source: A Data and preprocessing · p. 13, Table 5 · p. 13

B Implementation details

• The experiments train nine model classes on nine datasets, producing 81 model–dataset checkpoints used throughout the evaluation.Hyperparameters are reported in the accompanying repository.Source: B Implementation details · p. 13

B.1 Baselines

• Neural models take (N, T, C) inputs and predict one scalar per window; the sequence-model baselines are adapted from multivariate forecasting, with TST used as-is.ModernTCN transposes inputs to (N, C, T); xLSTM-Mixer uses a flattened postmixer regression pathway, and PatchTSMixer uses common-channel mode with last-head aggregation.Source: §B.1 · p. 13

B.2 Optimization

• Neural networks train for 100 epochs with Adam, batch size 256, and MSE loss, using training-split feature statistics for z-score normalization.Checkpoints are selected by validation loss; test metrics use the predefined temporal test split, and runs use seed 42.Source: §B.2 · p. 13

B.3 Robustification methods

• The robustification implementations include masked-reconstruction pretraining, PGD adversarial training, and channel dropout during training.F2F pretrains with zeroing, uniform bias, and Gaussian-noise corruptions before freezing its encoder for head fine-tuning; PGD mixes clean and adversarial samples, while ISensD zeros a sampled non-empty subset of channels.Source: §B.3 · pp. 13–14

B.4 Compute resources

• A complete reproduction takes approximately 155 hours of GPU time on one NVIDIA H100, with parallel runs across configurations available to reduce wall-clock time.Hyperparameter tuning and preliminary experiments required considerably more compute.Source: §B.4 · p. 14

B.5.1 Baseline architectures

• Baseline hyperparameters come from MuViS for XGBoost and CatBoost; other baselines use 100 Optuna trials per architecture and dataset, minimizing validation RMSE.Search uses multivariate TPE on a 10% training-data subsample, with author-recommended defaults initializing the first trial when available.Source: §B.5.1 · p. 14

B.5.2 Robustness methods

• All robustness variants use TST with per-dataset hyperparameters; ISensD needs no additional tuning, while F2F adopts published settings with mask length adjusted to sequence length.Source: §B.5.2 · p. 14
• PGD tuning covers perturbation budget, step size, inner steps, and adversarial batch fraction; Table 7 lists the search space.The authors use a robustness-aware validation objective because clean validation loss alone could favor weaker perturbations rather than genuine robustness.Source: §B.5.2 · p. 14, Table 7 · p. 15
• Robustification trials minimize clean RMSE divided by a geometric-mean robustness proxy computed from relative performance across a minimal corruption subset.The proxy uses bias and noise at three severities plus a stuck-at-mean hard fault at severity 0, applied to one fixed, randomly selected feature.Source: §B.5.2 · p. 15
• The tuning subset covers only three failure modes, a few severities, and one feature, limiting overlap with the full benchmark grid; the authors recommend this minimal-subset protocol.Source: §B.5.2 · p. 16

C Examples

• Figure 4 shows the ten failure modes at two severities alongside clean signals from PPG-DaLiA and Vehicle Dynamics.The examples contrast additive, multiplicative, drift-based, saturation-based, and stuck-at corruptions.Source: C Examples · p. 16, Figure 4 · p. 16

D Nominal performance

• ModernTCN leads on six of nine datasets (GM = 1.027, mean rank 2.2), while all three robustness-aware TST variants incur a clean-data penalty of +0.27 to +0.44 in GM.Clean-data RMSE is aggregated with a best-normalized geometric mean because per-dataset RMSE spans four orders of magnitude.Source: D Nominal performance · p. 17, Table 8 · p. 17

E Normalized performance under corruption (nPC)

• nPC divides mPC by the naïve mean predictor’s RMSE; values below 1 indicate better-than-naïve performance, but its omnibus test is not significant, so results are descriptive.The naïve predictor always predicts the training-set mean.Source: E Normalized performance under… · p. 17, Table 9 · p. 18

F Metric discussion

• The ratio RMSEnaive/RMSEclean measures clean-data headroom before degradation makes a model no better than naïve; rPC normalizes corruption degradation by clean performance.nPC instead uses the naïve predictor as a shared reference, preserving differences in failure-mode difficulty.Source: F Metric discussion · p. 18

G Per-failure-mode evaluation

• Per-failure-mode mPC and rPC average over affected channels and severities; cross-dataset mPC is best-normalized per dataset before geometric-mean aggregation.rPC is dimensionless and enters the geometric mean directly; failure-mode difficulty scores aggregate across models.Source: G Per-failure-mode evaluation · pp. 18–19
• Failure-mode rankings shift across corruption types, and the easiest modes raise rPC only marginally above 1 while the hardest cause considerably more degradation.The authors argue these differences make per-failure-mode reporting useful when selecting models for deployment environments with known sensor characteristics.Source: G Per-failure-mode evaluation · p. 19, Table 10 · p. 19

H Mean imputation as a failure mitigation strategy

• TST-ISensD achieves the best geometric-mean rPC (1.034) and mPC (1.072) under hard-fault mean imputation, with near-perfect rPC across datasets.Its channel-dropout training sets input channels to zero, matching mean replacement under z-score normalization; the model has learned to predict when any single channel is absent.Source: H Mean imputation as a failure… · pp. 19–20, Table 12 · p. 20, Table 13 · p. 21
• When failure detection and input intervention are available, dropout-based ISensD is described as an effective, easy-to-implement strategy that needs neither architectural changes nor adversarial training.The authors say it makes prediction quality on average nearly unaffected in this scenario.Source: H Mean imputation as a failure… · p. 20

I Severity curves

• On HR, M-TCN has the lowest corrupted RMSE across most failure modes and best mPC, but on Vehicle Dynamics it collapses under tire-temperature corruption at high severity.The Vehicle Dynamics result is consistent with M-TCN’s worst rPC and weakest mPC ranking on that dataset.Source: Figure 5 · p. 21, Figure 6 · p. 21, I Severity curves · p. 22
• PGD degrades more gradually than vanilla TST, but its clean-data penalty raises mPC on HR; on Vehicle Dynamics, smaller relative degradation registers on both mPC and rPC.The examples show that mPC and rPC can agree or diverge depending on whether robustification incurs a nominal cost.Source: I Severity curves · p. 22, Figure 7 · p. 22, Figure 8 · p. 22

J Statistical significance testing

• The tests follow the Demšar protocol, selecting parametric or non-parametric procedures based on normality and variance tests; critical-difference diagrams show mean ranks and non-significant clusters.No post-hoc comparisons are reported for scross (Friedman, p = 0.187) or nPC (ANOVA, p = 0.069), whose omnibus tests are not significant.Source: J Statistical significance testing · pp. 22–24, Table 14 · p. 23
• On mPC, XGBoost, CatBoost, and xLSTM-M lead and significantly outrank P-TSMixer and TST-ISensD; nominally, M-TCN has the best mean rank.The nominal leading non-significant cluster includes tree ensembles, xLSTM-M, plain TST, and TST-PGD.Source: Figure 10 · p. 23, J Statistical significance testing · p. 23, Figure 9 · p. 23
• On rPC, TST-PGD, XGBoost, and P-TSMixer lead, while M-TCN ranks last; all three robustified TST variants join the top non-significant cluster.At hard-fault zero severity, TST-ISensD ranks best on both mPC and rPC; its rPC advantage is significant over CatBoost, xLSTM-M, plain TST, and M-TCN.Source: J Statistical significance testing · pp. 23–24, Figure 12 · p. 24, Figure 13 · p. 24

K Package API reference

• The package accepts prediction functions or supported model objects and custom datasets; its Results object exposes per-condition records, aggregate metrics, summaries, CSV export, and degradation plots.The input convention is (N, T, C) arrays with one-dimensional labels.Source: K Package API reference · pp. 25–26, Table 16 · p. 26