PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Delayed Train Trajectories on the Dutch Railway Network: Explainable AI Evaluation of Topological, Operational and Weather Features with Tree Based Ensemble Methods

Jia Long Bao, Ali Mohammed Mansoor Alsahag, Seyed Sahand Mohammadi Ziabari

arXiv:2609.34692v3cs.LG

TL;DR

Long-term network-wide railway delay prediction must address limited evidence about temporal generalization and the challenge of interpreting complex models. This study evaluates interpretable tree ensembles with topological, operational, and environmental features across monthly horizons, finding that future-month performance degrades and is linked to environmental volatility and target instability.

  • Problem

    The study asks whether integrated topological, environmental, and operational features support temporal generalization in network-wide passenger train delay prediction.

    Source: §1.3.1 · p. 2

  • Method

    The study compares tree-based ensemble predictions across simultaneous and future-month evaluations and uses multi-horizon SHAP analysis to assess feature importance and stability.

    Source: §1.3.1 · p. 2

  • Results

    Feature-rich tree-based models improve within-month prediction, but performance significantly declines on future months, with degradation linked to environmental feature volatility and target-definition instability.

    Source: §6 · p. 11

  • Takeaways & Limitations

    Richer feature sets alone are insufficient for long-term robustness; the paper calls for dynamic, season-aware models anchored by absolute operational target designs.

    Source: §6 · p. 11

    The findings concern monthly domestic NS passenger predictions, combine delays and cancellations without distinguishing planned from unexpected events, and use historical weather that requires prediction-time availability checks.

    Source: §5.3 · p. 11

Abstract

from arXiv · show

The reliable prediction of passenger train delays is a critical component of railway management. While contemporary research frequently attempts to maximize absolute accuracy by deploying opaque deep learning architectures, the underlying data mechanics driving longitudinal predictive decay remain underexplored. Consequently, this study provides an explainable temporal robustness analysis of network-wide railway delay prediction. Focusing on the Dutch railway network, this research utilizes interpretable tree-based ensembles to integrate granular topological, environmental, and operational features. The overarching finding establishes that while feature-rich tree-based models improve simultaneous (within-month) prediction, predictive performance systematically degrades when evaluated across non-simultaneous (future) months. Furthermore, multi-horizon SHAP and dispersion analyses explicitly link this degradation to environmental feature volatility and instability within the statistical target definition. Ultimately, this study demonstrates that richer feature sets alone are insufficient to resolve long-term forecasting constraints, underscoring the necessity to transition toward dynamic, season-aware architectures anchored by absolute operational boundaries.

1 Introduction

Railway delays carry substantial financial costs and system-wide risks, while long-term network-wide prediction must balance temporal robustness with the interpretability needed in public infrastructure. This study addresses that gap with feature-rich tree ensembles and SHAP, comparing prediction across monthly horizons.

1.1.1 Financial Costs of Delays.

  • Delays cost the Dutch economy over €400 million annually, while dense scheduling heightens sensitivity to operational failures and environmental factors.NS distributed €9.7 million in annual compensation and refunds in 2025.Source: §1.1.1 · p. 1, §1.1.2 · p. 1

1.1.3 Regulatory AI Transparency in Public Infrastructure.

  • EU AI Act transparency and human-oversight requirements make interpretable pipelines a practical regulatory necessity for high-risk infrastructure such as Dutch railways.The passage states that transparency rules were scheduled for enforcement by August 2026.Source: §1.1.3 · p. 1

1.2.1 The Interpretability and Temporal Robustness Gap.

  • Long-term network-wide prediction remains underexplored relative to short-term real-time methods, and railway graph-based approaches face generalization challenges across unseen horizons.Recent short-term approaches often use complex hybrid architectures, while aggregated long-term frameworks receive less attention.Source: §1.2.1 · p. 1

1.2.2 Positioning Within Contemporary Literature.

  • Compared with prior frameworks, this study uses stop-to-stop trajectories, operational features, cancellations in the delay target, and exact-coordinate weather data.It prioritizes diagnostic interpretability through tree-based ensembles and SHAP rather than black-box deep learning.Source: §1.2.2 · pp. 1–2
  • The study’s central contribution is an explainable temporal robustness analysis that links predictive degradation to environmental feature volatility and target instability.Tree-based models and SHAP are used for diagnostic transparency rather than maximizing deep-learning predictive density.Source: §1.2.2 · p. 2

1.3.1 Temporal Evaluation Framework.

  • The evaluation compares feature categories across simultaneous and future-month tests, using paired Wilcoxon tests and multi-horizon SHAP analysis to track feature importance and stability.The framework also tests target-definition sensitivity and prioritizes temporal robustness over absolute accuracy optimization.Source: §1.3.1 · p. 2

2 Related Work

Prior work identifies constraints in railway delay prediction involving temporal generalization, data sparsity, feature coverage, and interpretability. This study responds with a granular tabular approach and uses SHAP to examine temporal feature stability while testing performance hypotheses.

2.1.1 Foundational Representations and Railway Limitations.

  • Prior Dutch railway work found limited temporal generalization with origin-destination aggregation and excluded operational edge weights, while aviation topological features also degraded under non-simultaneous testing.Kämper linked the Dutch result to data sparsity and limited feature richness, rather than necessarily to methodological flaws.Source: §2.1.1 · p. 2

2.1 Topological Representation and Network Dynamics

  • Earlier Dutch studies span limited-generalization topological models, improved disruption and cancellation prediction with added operational and environmental features, and stronger future-month performance from hybrid spatio-temporal deep learning.The hybrid architectures are GAT-GRU and GAT-LSTM.Source: §2.2.1 · p. 2

2.2.1 The Interpretability Trade-off.

  • Tree ensembles with SHAP prioritize explicit feature attributions over the predictive complexity of deep learning, while SHAP measures correlational reliance rather than causality.The cited discussion frames diagnostic transparency as relevant to critical infrastructure requirements.Source: §2.2.1 · pp. 2–3

2.2.2 Environmental Data Integrity.

  • Prior weather studies used nearest-neighbor approximations of relocated station data, potentially adding spatial inconsistency; this study instead uses exact station coordinates.The passage notes that prior environmental features’ limited contribution could reflect aggregation flaws rather than absent meteorological signal.Source: §2.2.2 · p. 3

2.3 Temporal Instability and Explainability

  • SHAP is used to track feature stability across temporal contexts, not to select features, because selection can artificially degrade predictive performance.Prior work attributed near-random non-simultaneous performance in the Brazil bus dataset to unstable feature importance and snapshot overfitting.Source: §2.3 · p. 3

2.4 Research Gap Summary

  • The research gap combines sparse, feature-limited network-wide prediction with the interpretability costs of deep learning; the proposed response integrates dynamic topology, operational edge weights, and exact-coordinate weather in tree ensembles.This approach aims to balance representational richness with diagnostic transparency.Source: §2.4 · p. 3
  • The hypotheses test whether ensembles exceed the 50% null baseline in simultaneous evaluation and whether future-month performance degrades under trajectory-based and strict time-based splits.The strict time-based hypothesis predicts degradation across all evaluated models, distinguishing temporal instability from sample-size fluctuation.Source: §2.4 · p. 3
  • Post-hoc SHAP analysis is designated to examine feature dynamics underlying any performance shifts, complementing the statistical tests of predictive change.Source: §2.4 · p. 3

3 Methodology

The methodology builds monthly network snapshots from domestic passenger services, engineers operational, topological, and weather features, and compares models across temporal evaluation schemes. It includes sensitivity checks for target definitions and statistical analyses of performance and feature stability.

3.1.1 Railway Operations Data and Spatial Filtering.

  • The dataset uses Dutch domestic passenger services from 2019–2024, with stop-to-stop trajectories and spatial filtering to retain verified Dutch station distances.Freight and international services were excluded; missing terminal timestamps were imputed, and partially and fully canceled services were retained as delayed.Source: §3.1.1 · p. 3, §3.1.2 · p. 3

3.1.4 Temporal Aggregation and Target Definition.

  • Monthly trajectory snapshots use a four-ride activity threshold and a median-based binary delay target, with sensitivity analyses for absolute and month-relative seasonal thresholds.The median threshold balances classes but is vulnerable to temporal label shift as delay ratios fluctuate; regression targets also assess schedule adherence and regulatory punctuality.Source: §3.1.4 · p. 4

3.2 Feature Engineering

  • Operational features encode recent service volume and elapsed inactivity, while dynamic monthly graphs provide node- and trajectory-level measures and station weather is aggregated into monthly features.The operational edge weight uses a maximum 12-month lookback; the weather features include temperature, precipitation, wind, and snow measures.Source: §3.2.1 · p. 5, §3.2.2 · p. 5, §3.2.3 · p. 5

3.2.4 Feature Set Construction and Ablation Logic.

  • Feature sets are evaluated independently and in combinations, while explicit feature selection is omitted so models and post-hoc SHAP can diagnose multicollinearity and temporal noise.The unfiltered space may increase overfitting under temporal shift; aggressive accuracy-focused pruning is left for future work.Source: §3.2.4 · p. 5

3.2.5 Exploratory Data Analysis (EDA).

  • The aggregated dataset contains 49,793 stable trajectories across 43 engineered features, with monthly delayed-trajectory proportions ranging from 10% to 90%.Snapshots average 247 station nodes and 692 active trajectory edges, despite an overall target distribution balanced at 50/50.Source: §3.2.5 · p. 5

3.3 Evaluation Framework and Experimental Setup

  • Simultaneous tests train and test within a monthly snapshot, whereas non-simultaneous tests train on historical snapshots and test on future months; a 70/30 chronological split controls for sample size.A trajectory-based split with random undersampling is applied to XGBoost for alignment with prior work and SHAP evaluation; tree ensembles are selected for interpretability.Source: §3.3.1 · p. 5, §3.3.2 · p. 5

3.3.3 Metrics and Statistical Inference.

  • Balanced accuracy is the primary classification metric, with paired Wilcoxon tests used for aligned monthly comparisons and SHAP-rank and feature-distribution tests assessing temporal stability and shifts.The framework also reports supplementary F1 and ROC AUC for time-based splits and uses IQR, Kendall rank correlation, and Kolmogorov-Smirnov tests for temporal analyses.Source: §3.3.3 · p. 6

3.3.4 Temporal Baselines.

  • Naive and seasonal-naive baselines use the prior month and same month of the prior year, respectively, with missing trajectory histories imputed as no delay.Approximately 3.5% of Naive and 5.2% of SNaive predictions use this imputation; the first 12 months are truncated for equal-size comparisons.Source: §3.3.4 · p. 6

4 Results

The results show that tree-based models predict delays above the null baseline within a month, but performance across future months depends on feature composition and evaluation setup. SHAP, feature-distribution, and robustness analyses reveal temporal instability in weather features and in the target definition.

4.1.1 Hypothesis 1: Simultaneous Predictive Performance.

  • All simultaneous XGBoost configurations exceeded the null baseline; median accuracy gains over null were 0.075 for Topological and 0.127 for All Features.Both gains were statistically significant (p < .001).Source: §4.1.1 · p. 7

4.1.2 Hypothesis 2: Temporal Generalization Degradation.

  • 0.069 median accuracy loss for All Features marked the significant drop from trajectory-based simultaneous to non-simultaneous evaluation.The decrease was statistically significant (p < .001).Source: §4.1.2 · p. 7, Figure 2 · p. 7
  • Ablations found a 0.018 median accuracy gain from adding weights, improved stability from operational features, and lower predictive capacity when weather features were added.The weather-enhancement hypothesis was rejected (p = 1.000); the Topological + Weight + Operational model improved significantly over the Topological baseline (p < .001).Source: §4.1.2 · p. 7

4.1.3 Methodological Sanity Check: Hyperparameter Optimization.

  • Hyperparameter tuning did not remove the significant performance degradation between simultaneous and non-simultaneous testing.The tuned integrated configuration showed a comparable degradation pattern, indicating that the reported decay was not attributed to model mis-specification.Source: §4.1.3 · pp. 7–8

4.1.4 Hypothesis 3: Temporal Degradation Across Time-Based Splits.

  • Under time-based splits, isolated Topological features showed no significant degradation across tree-based models, while most other configurations degraded or had mixed robustness.The Topological tests reported p = 1.000 for XGBoost, Random Forest, and LightGBM; the table covers balanced accuracy across both testing paradigms.Source: §4.1.4 · p. 8, Table 1 · p. 8

4.1.5 Temporal Baseline Comparisons.

  • The Naive persistence model achieved a mean balanced accuracy of 0.8080 over 60 months; regression models likewise lost predictive performance across future horizons.For Regulatory Punctuality, Topological + Weight + Operational MAE rose from 0.0532 in simultaneous testing to 0.0944 in non-simultaneous testing.Source: §4.1.5 · p. 8, §4.4 · p. 9

4.2.2 Feature Volatility and Long-Term Stability.

  • Environmental features showed the greatest volatility across horizons, while operational features were more consistent; even the most stable integrated-model features varied over time.In January 2024, Target Weighted Outbound Degree was a primary predictor, while temperature also showed a positive association with predicted delay likelihood.Source: §4.2.2 · p. 8, §4.2.1 · p. 8

4.3 Robustness Check: Alternative Target Thresholds

  • Alternative target thresholds did not eliminate temporal decay in integrated models, and weather features remained vulnerable; explanation-rank changes frequently aligned with input-distribution shifts.Weather features showed significant distributional drift in 84.8% of transitions, compared with 3.29% for Operational, 6.94% for Topological, and 1.41% for Weight features.Source: §4.3 · p. 9, §4.2.3 · p. 9

5 Discussion

The discussion positions the study as an explainable analysis of temporal robustness rather than an attempt to maximize absolute predictive accuracy. It connects cross-month degradation with label and feature-distribution instability, while limiting its conclusions to monthly domestic NS predictions and noting that historical weather inputs need prediction-time evaluation.

5.1 Positioning within the State-of-the-Art

  • The study’s central contribution is an explainable temporal robustness analysis, rather than maximizing raw predictive accuracy.The discussion contrasts the tabular XGBoost results with higher reported thresholds for hybrid GAT-LSTM/GRU models and suggests static monthly snapshots omit temporal sequencing.Source: §5.1 · p. 10

5.2 Diagnosing Temporal Degradation

  • The discussion links degradation to environmental feature volatility and target instability, while time-based tests found no decay for isolated Topological configurations across three tree-based models.The cited tests report p = 1.000 for XGBoost, Random Forest, and LightGBM; adding weather features did not enhance predictive capacity (p = 1.000).Source: §5.2 · p. 10
  • A static global median threshold is associated with temporal label shift as monthly class prevalence fluctuates between 10% and 90%.The discussion says the positive class’s operational meaning changes over time and that an absolute threshold stabilized core network representations, but did not eliminate decay across broader feature sets.Source: §5.2 · p. 10

5.2.1 Covariate Shift and Concept Drift.

  • Weather features showed significant distributional shifts in nearly 85% of transitions, versus stability in over 93% of operational and structural transitions; these analyses do not establish the cause of degradation.The discussion notes that significant SHAP rank inversions frequently align with measured distributional shifts, but do not establish a change in the predictors’ conditional relationship with the target.Source: §5.2.1 · pp. 10–11

5.3 Limitations and Future Work

  • The findings cover monthly domestic NS passenger predictions, and the records do not distinguish planned from unexpected cancellations.Weather inputs are historical observations, so prospective use requires evaluation with information available at prediction time.Source: §5.3 · p. 11

6 Conclusion

The conclusion finds that feature-rich tree-based models improve within-month prediction but significantly degrade on future months, with environmental volatility and target-definition instability linked to the decline. The study concludes that richer features alone are insufficient for long-term robustness and calls for dynamic, season-aware models with operationally grounded targets.

6 Conclusion

  • Feature-rich tree-based models improve within-month prediction, but performance significantly declines on future months, linked to environmental feature volatility and target-definition instability.The conclusion presents this as the study's overarching finding.Source: §6 · p. 11
  • Temporal performance depends on the target definition, while isolated topological features show resilience and broader external-feature integration degrades out-of-sample performance.The evaluated hyperparameter-tuned configuration also retained temporal degradation.Source: §6 · p. 11
  • SHAP and distributional analyses reveal temporal variation in environmental features and their predictive importance.Source: §6 · p. 11
  • Richer feature sets alone are insufficient; the conclusion calls for dynamic, season-aware models anchored by absolute, operationally grounded target designs.The stated constraint is instability in both the statistical target definition and the external environment over time.Source: §6 · p. 11

Appendix B Statistical Significance Tests (Temporal Aggregation)

The appendix reports Wilcoxon signed-rank tests comparing simultaneous and non-simultaneous performance, null-baseline performance, and incremental feature configurations across splits and target thresholds.

Appendix B Statistical Significance Tests (Temporal Aggregation)

  • Table 6 reports Wilcoxon tests of simultaneous XGBoost performance against the null baseline for each feature set, with N=72.Source: Table 6 · p. 16
  • Tables 7–9 compare simultaneous and non-simultaneous performance for XGBoost or all models across trajectory-based and time-based splits and threshold settings.The captions specify N=71 per feature set for Table 7 and the Absolute Threshold comparison in Table 9; Table 8 reports N=66 per feature set.Source: Table 7 · p. 16, Table 8 · p. 17, Table 9 · p. 17
  • Tables 10–12 report Wilcoxon tests for month-relative threshold performance, incremental feature engineering, and tuned XGBoost comparisons.The captions specify trajectory-based splits; Table 10 has N=71 per feature set, and Table 12 has N=71.Source: Table 10 · p. 18, Table 11 · p. 18, Table 12 · p. 18

Appendix C Regression Metrics (Regulatory Punctuality & Schedule Adherence)

The appendix covers regression metrics at two punctuality thresholds and describes temporal baseline performance and survivorship-bias diagnostics across months.

Appendix C Regression Metrics (Regulatory Punctuality & Schedule Adherence)

  • Tables 13 and 14 report MAE and RMSE across both testing paradigms at 3-minute regulatory punctuality and 0-minute schedule-adherence thresholds.Source: Table 13 · p. 19, Table 14 · p. 19

Appendix D Temporal Baselines (Performance & Surviorship Bias)

  • Table 15 compares Naive (t−1) and Seasonal Naive (t−12) predictive performance over 60 truncated months.Source: Table 15 · p. 20
  • Table 16 summarizes zero-imputation default rates, the resulting delay-rate gap, and non-defaulted accuracy over 60 months.Source: Table 16 · p. 20
  • Figure 4 shows monthly balanced-accuracy variance and interquartile range for the Naive and Seasonal Naive baselines.Source: Figure 4 · p. 20
  • Figure 5 displays baseline balanced accuracy over the 60-month period, while Figure 6 plots fallback usage and the delay-rate gap between defaulted and non-defaulted predictions.Source: Figure 5 · p. 21, Figure 6 · p. 22

Appendix E Kendall Rank Correlation

The appendix presents diagnostics of sequential SHAP-ranking correlations, monthly non-simultaneous balanced accuracy, and temporal distributional drift across feature categories.

Appendix E Kendall Rank Correlation

  • Table 17 reports Kendall rank correlations for sequential SHAP feature-importance stability across 71 transitions, and Figure 7 traces coefficients across transitions for all XGBoost feature configurations.Source: Table 17 · p. 23, Figure 7 · p. 24

Appendix F Non Simultaneous Performance Across Months for the SHAP evaluated XGBoost model (trajectory based split)

  • Figure 8 displays non-simultaneous balanced accuracy across months for all XGBoost feature sets using a trajectory-based split.Source: Figure 8 · p. 26

Appendix G K-S Test Summary Tables

  • Table 18 summarizes significant K-S distribution shifts by macro-category, while Tables 19 and 20 report feature-level temporal drift significance across 71 transitions for topological, weight, weather, and operational features.Source: Table 18 · p. 27, Table 19 · p. 27, Table 20 · p. 28

Appendix H Exploratory Data Analysis (EDA) Supplementary Figures

The supplementary figures document target-label and service patterns across the network and over time, alongside model-performance distributions and alternative delay-threshold analyses.

Appendix H Exploratory Data Analysis (EDA) Supplementary Figures

  • Figures 12 and 13 depict monthly scheduled-service volume and the mean proportion of delayed arrivals per edge relative to the historical global median.Figure 16 presents distributions over time for selected network and environmental features, with shaded interquartile ranges and a collapsed range for src_avg_soil_temperature.Source: Figure 12 · p. 30, Figure 13 · p. 31, Figure 16 · p. 32
  • Figures 10, 14, and 15 show binary target-label distributions, including distributions across unique network edges for NS data and origin-destination trajectories.Figure 11 separately depicts monthly class balance between on-time and significantly delayed edges.Source: Figure 10 · p. 30, Figure 11 · p. 30, Figure 14 · p. 31

Appendix I Model Performance Distributions (XGBoost Trajectory-based split)

  • Figure 17 compares balanced-accuracy distributions across feature sets under non-simultaneous and simultaneous testing conditions.The accompanying panel label identifies balanced accuracy across feature sets for simultaneous testing using XGBoost, a trajectory-based split, and global aggregation.Source: Figure 17 · p. 33, Appendix I · p. 33

Appendix J Alternative Delay Thresholds: Absolute and Relative

  • Figures 18 and 19 present class balance and XGBoost balanced-accuracy performance using absolute-delay and month-relative delay thresholds, respectively.Figure 9 provides a spatial overview of the Dutch railway network infrastructure and historical passenger-service coverage in April 2024.Source: Figure 9 · p. 29, Figure 18 · p. 34, Figure 19 · p. 35

Appendix K Supplementary Operational Metrics

Figure 20 presents monthly time series of pure delays and cancellations alongside total scheduled service volume.

Appendix K Supplementary Operational Metrics

  • Figure 20 compares monthly pure delays and cancellations with total scheduled service volume.Source: Figure 20 · p. 36

Appendix L Feature Engineering and Hyperparameter Configurations

The appendix documents engineered features and classifier configurations, then presents temporal feature-importance analyses, SHAP impact plots, and confusion-matrix comparisons.

Appendix L Feature Engineering and Hyperparameter Configurations

  • Tables 21–23 cover engineered feature sets and measurement units, topological feature definitions and formulations, and default classifier hyperparameter configurations.The feature-set abbreviations distinguish topology, weight, weather, operational, and combined sets.Source: Table 21 · p. 37, Table 22 · p. 38, Table 23 · p. 38

Appendix M Year-over-Year (YoY) Temporal Feature Importance

  • Figures 21–23 show monthly year-over-year temporal feature importance for XGBoost across isolated baseline feature sets, initial combined topological, weight, and weather feature sets, and combined topological, weight, operational, and all-feature sets.Source: Figure 21 · p. 39, Figure 22 · p. 40, Figure 23 · p. 41

Appendix N Month-over-Month (MoM) Temporal Feature Importance

Appendix O SHAP Value Feature Impact Distributions

  • Figures 27 and 28 show XGBoost SHAP summary plots for individual and combined feature sets in period 2024-01; the combined-feature plot colors feature values from low to high.Source: Figure 27 · p. 45, Figure 28 · p. 46

Appendix P Confusion Matrices

  • Figures 29, 22, and 23 provide side-by-side confusion-matrix comparisons for individual feature sets, progressively combined and complete feature sets, and tuned complete-feature models in period 2024-01.Source: Figure 29 · p. 48, Figure 30 · p. 49, Figure 31 · p. 50

Related papers

Loading 2609.34692v3…