PaperlayerView exact revision on arXiv

Source-linked AI summary

Pneuma: A Large-Scale Real-World Time Series Benchmark for Commercial Aircraft Predictive Maintenance

Jiafeng Lin, Yutian Zhang, Huan Zhang, Yuxuan Wang, Jinyan Feng, Yuhui Liu, Zhongyi Pei, Jianmin Wang

arXiv:2610.37563v1cs.LG

TL;DR

Commercial aviation predictive maintenance lacks public, large-scale real-world benchmarks, limiting reproducible evaluation. Pneuma provides a multi-task benchmark built from commercial aircraft onboard time series and evaluates more than 25 models. The evaluation finds that current models still struggle with subtle fault signatures.

  • Problem

    Public, large-scale real-world aviation datasets are scarce, and proprietary studies often use fragmented tasks, arbitrary splits, and inconsistent metrics that limit reproducibility and fair comparison.

    Source: §1 · pp. 1–2

  • Method

    Pneuma uses maintenance-log-linked onboard time series from commercial aircraft to establish standardized forecasting, classification, and anomaly detection benchmarks and evaluate more than 25 models.

    Source: §1 · p. 2

  • Results

    The benchmark evaluation finds that current models still struggle with subtle fault signatures and diverse aircraft configurations.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Pneuma provides a unified, reproducible basis for evaluating predictive maintenance models on real-world commercial aircraft time series.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Predictive maintenance is vital for commercial aviation, yet progress remains hindered by the lack of public, large-scale real-world datasets. As a result, most methods are developed and evaluated on proprietary data, limiting reproducibility. We introduce Pneuma, the first publicly available large-scale benchmark for predictive maintenance using onboard time series from in-service commercial aircraft. Built around bleed air systems, Pneuma contains over 920 million multivariate time steps across four aircraft types and 9 verified fault scenarios, directly aligned with real maintenance logs. The dataset exhibits key real-world challenges, including substantial class imbalance, subtle fault evolution patterns, and significant variance across aircraft configurations. We structure the benchmark around three tasks: multivariate forecasting, supervised fault classification, and unsupervised anomaly detection, with labels derived from actual maintenance records. Under unified settings, we evaluate over 25 models from four methodological families on 9 carefully curated subsets. The results reveal that even strong contemporary models struggle with subtle fault signatures. Further analyses show consistent gains from additional in-context examples and multivariate inference, while forecasting quality is strongly associated with anomaly detection performance. Together, these findings provide practical insights into effectively modeling complex real-world aviation time series. We hope Pneuma will serve as a rigorous and reproducible foundation for advancing data-driven predictive maintenance toward real-world deployment in commercial aviation.

1 Introduction

Pneuma addresses the shortage of public, representative benchmarks for commercial aviation predictive maintenance with a large-scale dataset and unified evaluation across three tasks. Its evaluation finds that current models still struggle with subtle fault signatures.

1 Introduction

  • Existing public aviation benchmarks are scarce, while proprietary fleet studies use fragmented tasks, arbitrary splits, and inconsistent metrics that hinder reproducibility and fair comparison.The literature also lacks evidence about whether current time series models generalize across real-world aircraft configurations and fault scenarios.Source: §1 · pp. 1–2
  • Pneuma provides more than 920 million real-world onboard time steps across four aircraft types and nine fault scenarios, with labels derived from maintenance records.The benchmark focuses on bleed air systems and preserves variation in flight phases, control states, and fault manifestations.Source: §1 · p. 2
  • The benchmark unifies multivariate forecasting, supervised fault classification, and unsupervised anomaly detection with standardized splits, protocols, and metrics across nine subsets.The paper evaluates more than 25 models from four methodological families and conducts controlled experiments on representation and modeling choices.Source: §1 · p. 2
  • The evaluation finds that even state-of-the-art architectures struggle to identify subtle fault signatures in aircraft operations.The authors frame this as an open challenge for the community and provide practical guidance on representation and modeling.Source: §1 · p. 2

2 Related Work

Prior benchmarks provide standardized comparisons across general time series domains, while domain-specific benchmarks preserve application structure and evaluation needs. Public aviation resources still lack a standardized benchmark with verified faults from multiple mainline commercial aircraft types.

2.1 Time Series Benchmarks

  • General-purpose benchmarks span classification, forecasting, and anomaly detection, while domain-specific efforts show the value of representative data, defined tasks, standardized protocols, and reproducible baselines.The cited examples include UCR and UEA, M4, TFB, TIME, TSB-AD, WeatherBench, LargeST, and BatteryLife.Source: §2.1 · p. 2

2.2 Aviation Time Series Datasets

  • Existing public aviation datasets focus on general aviation, a single regional-jet type, or synthetic fault injection, leaving multi-type mainline commercial aircraft data with confirmed faults lacking.Pneuma addresses this gap with onboard time series from four mainline commercial aircraft types and unified bleed air system benchmarks.Source: §2.2 · p. 3

3 Pneuma

Pneuma comprises QAR records from routine operations across four mainline aircraft types and nine bleed-air fault scenarios. The paper describes how the data are collected and aligned, then defines benchmark pools and three task-specific evaluation setups.

3.1 Dataset Overview

  • Pneuma combines QAR measurements from routine airline operations with operational and maintenance records for documented faults across four aircraft types and nine scenarios.The dataset contains 71,451 flight records and more than 920 million time steps.Source: §3.1 · p. 3

3.2 Operational Background

  • The bleed air system regulates engine-compressed air through components such as pressure-regulating valves, with sensors tracking pressure, temperature, and operating state.Fault symptoms can appear as subtle shifts, delayed responses, or cross-sensor inconsistencies that vary across flight phases and conditions.Source: §3.2 · p. 3

3.3 Dataset Collection

  • The collection process retrieves and filters QAR records linked to documented maintenance events, selects system-relevant variables, and extracts fixed-length windows around four flight-phase transitions.The nine subsets are grouped by aircraft type and fault scenario; Table 2 summarizes collection period, variable count, and average trajectory length, and records do not overlap.Source: §3.3 · pp. 3–4, Table 2 · p. 4

3.4 Benchmark

  • Benchmark annotations use documented fault events to define a 100-flight warning pool and a normal pool separated from events by exclusion margins.The normal pool excludes 500 flights before and 50 after each event, creating a 400-flight buffer between the warning window and the normal boundary.Source: §3.4 · p. 4

3.4.1 Forecasting.

  • Forecasting predicts system states after each of four flight-phase transitions, with models trained, validated, and tested only on normal flights.The paper evaluates forecasting with MAE and RMSE; it reports that stronger forecasting performance generally accompanies stronger anomaly detection under the same normal-only reconstruction setting.Source: §3.4.1 · p. 4

3.4.2 Classification.

  • Classification concatenates four transition segments into a chronological flight-level series and evaluates warning reliability using precision at a minimum 10% warning recall.Normal-pool flights are labeled normal and fault-warning-pool flights are labeled warning; both classes appear in train, validation, and test sets.Source: §3.4.2 · pp. 4–5

3.4.3 Anomaly Detection.

  • Anomaly detection trains on normal flights, scores test flights by averaged reconstruction error, and uses the same P@R10 metric as supervised classification.Higher reconstruction-error scores indicate greater deviation from behavior observed in the normal training pool.Source: §3.4.3 · p. 5

4 Experiments

The experiments benchmark forecasting, classification, and anomaly detection models under unified settings, then examine how context, multivariate inference, and temporal patching affect performance. Results vary across aircraft and tasks, while forecasting accuracy is strongly associated with anomaly detection performance.

4.1 Experimental Setup

  • The benchmark evaluates over 25 models across forecasting, classification, and anomaly detection, using unified preprocessing and training settings unless otherwise specified.Forecasting includes full-shot models trained separately per dataset and foundation models evaluated in a 5-shot setting; Table 3 reports results across nine datasets.Source: §4.1 · p. 5

4.2 Main Results

  • 76.65 overall Score; DLinear has the highest full-shot forecasting Score and the lowest forecasting error on eight of nine datasets.Foundation models are competitive in the 5-shot setting but none surpasses DLinear in aggregate Score; their leading models rank first on eight datasets for Toto-2.0 and the remaining dataset for Chronos-2.Source: §4.2 · pp. 5–6, Table 3 · p. 6
  • 60.22 overall Score; VSFormer leads supervised classification, while LITE ranks second and PatchTST is strongest on 787-CAC.No architecture dominates every aircraft and fault scenario; adapted forecasting architectures remain competitive on individual datasets.Source: §4.2 · p. 6, Table 4 · p. 7
  • 23.11 overall Score; Anomaly Transformer leads anomaly detection, whose methods perform substantially worse than supervised classifiers.The normal-only reconstruction setting is challenged by variation in normal records and weak fault symptoms that overlap with normal behavior, limiting reconstruction-error separability.Source: §4.2 · p. 6, Table 4 · p. 7

4.3.1 Effect of In-Context Learning.

  • Forecasting loss decreases consistently for all three foundation models as the number of complete preceding-flight demonstrations increases.The evaluation varies 2, 3, 5, 8, and 12 demonstrations while keeping other settings unchanged; the result indicates reusable operational patterns in historical flights.Source: §4.3.1 · p. 7, Figure 3 · p. 7

4.3.2 Univariate versus Multivariate Inference.

  • Multivariate inference consistently reduces forecasting loss for Chronos-2, TiRex-2, and Toto-2.0 compared with independent univariate forecasting.Joint inputs let the models use cross-variable dependencies and interactions among aircraft-system variables.Source: §4.3.2 · pp. 7–8, Figure 4 · p. 7

4.3.3 Effect of Temporal Patch Length.

  • All three patch-based models benefit from patch lengths above 1, but the optimal temporal granularity varies by architecture.For PatchTST and TimeXer, overly long patches worsen performance; aggregation can obscure short responses near flight-phase transitions, while very short patches can fragment component responses.Source: §4.3.3 · p. 8, Figure 5 · p. 8

4.3.4 Relationship between Forecasting and Anomaly Detection.

  • DLinear achieves both the lowest forecasting error, 0.309, and highest Precision@R10, 0.286, among six representative architectures.Forecasting error and anomaly detection performance are associated (Pearson’s r = −0.954, p = 0.003); anomaly detection performance generally decreases as forecasting error increases.Source: §4.3.4 · p. 8, Figure 6 · p. 8

5 Conclusion and Future Work

The paper introduces Pneuma as a public benchmark for predictive maintenance using in-service mainline commercial aircraft time series. Benchmark results show that existing models still struggle with early fault symptoms and diverse aircraft configurations, and future work will expand data coverage and modeling scope.

5 Conclusion and Future Work

  • Pneuma contains more than 920 million time steps across four aircraft types and nine bleed-air-system fault scenarios, with standardized protocols for forecasting, classification, and anomaly detection.The benchmark evaluates over 25 models and reports difficulties recognizing early fault symptoms and modeling diverse aircraft configurations.Source: §5 · p. 8
  • Future data expansion will add aircraft types, operational records, onboard systems, and fault scenarios to improve benchmark diversity and representativeness.The paper identifies limited-fault learning, transfer across aircraft types and systems, and more general aviation time-series models as modeling directions.Source: §5 · p. 8
  • The authors frame Pneuma as a resource for continued research toward reliable and transferable predictive maintenance methods.This is the paper’s stated aspiration for the benchmark.Source: §5 · p. 8

A Details of Datasets

Pneuma organizes flight-level records into chronological partitions and constructs task-specific sequences around flight-phase transitions. The appendix also describes the retained class imbalance and conventions used to interpret aircraft-variable names.

A Details of Datasets

  • Each documented fault event’s flight records are split chronologically into training, validation, and test sets at a 7:2:1 ratio.The within-event chronological split preserves temporal order and prevents future flights from entering earlier partitions.Source: A Details of Datasets · p. 10
  • Pneuma is distributed in TsFile, a columnar format intended to support efficient storage, access, and sample extraction for large multivariate records.The passage describes compression, high-throughput reading and writing, and indexed access by variable and time range.Source: A Details of Datasets · p. 10
  • Forecasting uses 60 preceding time steps to predict the next 20; classification and anomaly detection concatenate segments from four flight-stage transitions.Segment lengths differ across aircraft types because they follow an aviation-specific operating-condition selection protocol.Source: A Details of Datasets · p. 10
  • The benchmark retains realistic class imbalance: 58,141 normal and 13,310 abnormal flights among 71,451 total.Normal flights substantially outnumber abnormal flights across subsets, reflecting the stated scarcity of fault-related operational records.Source: A Details of Datasets · p. 10

B Definition of Aviation Variables

  • For the A320 subsets, variable suffixes “1” and “2” identify Engine 1 and Engine 2 systems, while “_R” marks valve-status feedback.“I” and “O” denote the inlet and outlet sides of the corresponding component; variable names and logging conventions differ across aircraft types.Source: B Definition of Aviation Variables · p. 10

C Implementation Details

Trainable models use shared preprocessing, sample construction, and optimization settings for fair comparison. Forecasting and fault-warning metrics are defined across the nine datasets.

C Implementation Details

  • All trainable models within a track use identical preprocessing and sample construction and unified training and optimization settings unless otherwise specified.Experiments use PyTorch on NVIDIA GeForce RTX 4090 GPUs; validation supports model selection and early stopping.Source: C Implementation Details · pp. 10–11
  • Forecasting performance is evaluated with MAE and RMSE, with lower values indicating better performance.The forecasting error reported in the main table combines MAE and RMSE; the forecasting Score aggregates across nine datasets, with higher values better.Source: C Implementation Details · p. 10
  • Precision@R10 reports precision at the smallest ranked prefix where warning recall reaches or exceeds 10%.Warning samples are the positive class, and validation is used for model selection and early stopping.Source: C Implementation Details · pp. 10–11
  • The aggregate classification and anomaly detection Score averages Precision@R10 across nine datasets on a 0-to-100 scale.Higher Precision@R10 and Score indicate better performance.Source: C Implementation Details · p. 11

D.1 Forecasting Showcases

Forecasting showcases compare three models on representative aircraft samples and phase transitions. Classification examples show that subtle fault symptoms can overlap with normal operational variation.

D.1 Forecasting Showcases

  • Figure 7 separates historical input from future ground truth with a vertical dotted line and overlays predictions for DLinear, PatchTST, and TimeXer.The examples span different aircraft subsets and flight phase transitions.Source: §D.1 · p. 11, Figure 7 · p. 12
  • For 321-DIFF’s Stage 5-to-6 transition, DLinear follows the target decline and achieves the lowest loss (0.217); for 320-HPV’s Stage 8-to-9 transition, it reproduces the rapid decrease with loss 0.247.On 321-DIFF, PatchTST has level bias and TimeXer smooths and underestimates the decrease; on 320-HPV, both show larger deviations in the early prediction region.Source: §D.1 · p. 11

D.2 Classification Showcases

  • False alarms and missed faults illustrate substantial overlap: some normal records resemble fault signatures, while some faulty records deviate only weakly from normal patterns.High confidence on the false alarm and low fault probability on the missed sample indicate these errors are not solely prediction uncertainty.Source: §D.2 · p. 11

E.1 Full Results of Temporal Patch Length Analysis

The appendix reports forecasting loss across temporal patch lengths for PatchTST, TimeXer, and xPatch, averaged over subsets and flight phase transitions.

E.1 Full Results of Temporal Patch Length Analysis

  • Table 7 reports forecasting loss for PatchTST, TimeXer, and xPatch at patch lengths of 1, 2, 4, 8, and 16.Other experimental settings are held unchanged.Source: §E.1 · p. 12
  • Loss is the average of MAE and RMSE, further averaged across evaluated subsets and four flight phase transitions.Lower values indicate better forecasting performance.Source: §E.1 · p. 12

E.2 Full Forecasting Results

The appendix provides detailed MAE and RMSE comparisons across datasets and prediction stages, complementing the aggregated main-text results.

E.2 Full Forecasting Results

  • Tables 8 and 9 report MAE and RMSE, respectively, across nine datasets and four prediction stages.The experiments follow the main-text settings, and lower values indicate better forecasting performance.Source: §E.2 · p. 12, Table 8 · p. 14, Table 9 · p. 15
  • Table 7 reports full forecasting results for different temporal patch lengths, broken down by dataset; loss is averaged across the four flight phase transitions, and lower values indicate better performance.Source: Table 7 · p. 13
  • For Tables 8 and 9, the Avg row averages results over four stages, while 1st Count records the number of stages each model ranks first.The best result for each stage is highlighted in bold.Source: Table 8 · p. 14, Table 9 · p. 15
  • The appendix supplies abbreviations for model names, including DLinear, PatchTST, TimeXer, xPatch, and OLinear.The abbreviation key also covers the forecasting and foundation models used in the tables.Source: §E.2 · p. 14

Related papers

Loading 2610.37563v1…