PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-Aware Validation of Machine Learning Fuel Consumption Models: Evidence from 1\,Hz Operational Data, CCGS \textit{Sir Wilfrid Laurier}

Samarasimha Reddy Chittamuru, Ayhan Akinturk, Allison Kennedy, Joshua Barnes, Matthew Hamilton

arXiv:2608.16833v1cs.LG

TL;DR

High-frequency ship-fuel models are commonly validated with random splits that can violate temporal independence. This paper compares time-aware validation schemes on 1 Hz data from one vessel and finds that random-split performance can collapse chronologically, while Ridge consistently outperforms tree ensembles under time-aware validation.

  • Problem

    Random splits and standard k-fold cross-validation dominate high-frequency fuel-model evaluation despite autocorrelated records violating their independence assumptions.

  • Method

    Six regression models and a physics baseline are tuned under three time-aware validation schemes and feature configurations, then compared on a common chronological hold-out set.

  • Results

    Random Forest and XGBoost test R2 fell from 0.99 and 0.98 under random splitting to −0.36 and −0.10 chronologically, while Ridge had lower mean squared error in all eighteen time-aware comparisons.

  • Takeaways & Limitations

    Random-split accuracy can fail to generalise chronologically, whereas Ridge provides the most consistent performance among the evaluated models under time-aware validation.

  • Takeaways & Limitations

    The study covers one vessel and one year, limiting the range of observed conditions and leaving generalisation across vessels, seasons, and loading conditions untested.

Abstract

from arXiv · show

Ship fuel consumption (SFC) prediction supports vessel operation optimisation, emissions estimation, and decision support systems (DSS) for sustainable maritime transportation. Numerous data-driven fuel models have been developed over the past two decades, but a critical and often overlooked limitation lies in their validation practices: most studies evaluate performance using random train--test splits, which, applied to high-frequency records, admit temporal leakage and yield optimistic results that do not reflect deployment conditions. This paper examines that gap using time-aware evaluation, specifically Time Series Cross-Validation (TSCV) and Blocked TSCV (BTSCV). Using the Canadian Coast Guard Ship (CCGS) \textit{Sir Wilfrid Laurier} as a case study, six regression models and a physics baseline are tuned under three time-aware schemes and three feature configurations, then evaluated on a common chronological hold-out set drawn from approximately 3.88 million steady-state 1\,Hz records.

1. Introduction … Surface and Medium Condition: Fouling and Ice Resistance.

The paper motivates time-aware validation for ship fuel-consumption models by contrasting physically grounded resistance approaches with increasingly high-frequency data-driven methods. It identifies temporal leakage in conventional random validation and proposes protocol-, feature-, baseline-, significance-, and interpretability-focused evaluation using operational data from CCGS Sir Wilfrid Laurier.

  • 1. Introduction: Over 80% of world trade moves by sea, while IMO fuel-reporting requirements and growing operational datasets motivate accurate fuel-consumption models for decision support.Ships of at least 5,000 GT have reported annual fuel consumption to the IMO DCS since 1 January 2019.
  • 1. Introduction: Near-perfect autocorrelation makes random train–test splits leak temporal neighbours into training, measuring interpolation rather than generalisation to unseen operating conditions.This problem is especially acute for high-frequency records, whose consecutive observations are closely spaced in time.
  • 1. Introduction: The case study uses approximately 3.88 million steady-state 1 Hz records from CCGS Sir Wilfrid Laurier, paired with ERA5 hindcast environmental fields.The records span August 2024 to June 2025 and support evaluation of six regression models plus a physics-based baseline.
  • 2.1. Evolution of Fuel Consumption Modeling: Fuel-consumption modeling spans white-box, black-box, and grey-box approaches, with white-box methods deriving power demand from hydrodynamic resistance and offering physical interpretability.Calm-water resistance methods include ITTC formulations, while broader operational models incorporate data-driven methods and deployment considerations.
  • Added Resistance in Waves.: Real operating conditions add resistance through waves, wind, currents, shallow-water confinement, blockage, and squat, altering vessel power demand beyond calm-water assumptions.These effects are addressed through seakeeping, aerodynamic, current, and confined-water resistance analyses.
  • Surface and Medium Condition: Fouling and Ice Resistance.: Hull and medium condition further affect performance: biofouling raises roughness and frictional resistance, while idealized white-box assumptions differ from real operational conditions.Noon reports helped provide operational data but remained coarse in temporal resolution, motivating higher-frequency sensing.
  • 2.1.3. Data-Driven Fuel Consumption Models Using High-Frequency Sensor Data: High-frequency sensor studies often report strong predictive accuracy but predominantly use random splits or standard k-fold cross-validation, which are unsuitable for autocorrelated adjacent records.Voyage-level or chronological tests are exceptions, and time-aware assessments have commonly been supplementary rather than used for model selection.
  • 2.2. Research Gap and Objectives: The study independently tunes six regression models under Sequential K-Fold, Custom TSCV, and BTSCV, compares SOG, STW, and combined features, and evaluates a cubic speed–draft baseline.It also applies Wilcoxon testing with Benjamini–Hochberg correction and uses SHAP values plus standardised coefficients for interpretation.

3. Data Description and Sources

The study combines post-refit onboard propulsion and navigation measurements with timestamp- and position-matched ERA5 hindcast environmental data for the CCGS Sir Wilfrid Laurier. Multi-stage quality screening and steady-state filtering reduce approximately 26 million 1 Hz records to 3,887,002 steady-state cruising observations for modeling.

  • Data sources: Post-refit data from August 2024 to June 2025 represent a single propulsion configuration with three Wärtsilä 8L26 main engines and one Caterpillar 3508 auxiliary engine.The vessel underwent a 2024 VLE refit replacing all four main diesel engines.
  • Data sources: Fuel-flow responses came from OpDAQ™ flowmeter streams, while NMEA 0183 channels supplied speed, heading, rudder status, and wind measurements.The retained NMEA sentences were VTG, VHW, HDT, MWV, and DSC1.
  • Data integration: ERA5 hindcast wave and swell parameters were retrieved through cdsapi and matched to onboard measurements by timestamp and vessel position in a unified PostgreSQL time series.Selected parameters describe total and swell wave heights, periods, and propagation directions.
  • Feature characteristics: Significant wave height and significant swell height show strong correlation (r = 0.92), while overall inter-predictor correlations remain low to moderate.Total fuel flow is multimodal, speed over ground centers near 12 knots, and wind and sea-state indicators are right-skewed.

4. Methodology

The methodology formulates main-engine fuel-flow estimation as supervised regression using contemporaneous operational and environmental predictors. It compares three speed-feature variants across six regression models, distinguishing planning-stage availability from physical relevance.

  • Regression formulation: Fuel flow is modeled as a mapping from operational and environmental predictors to total main-engine fuel flow.The predictors correspond to steady-state cruising records.
  • Regression formulation: The model excludes lagged and autoregressive fuel-flow terms so predictions use only conditions available when the estimate is made.This reflects voyage-planning decisions made before the relevant leg is sailed.
  • Feature variants: Three feature variants differ only in speed input: SOG alone, STW alone, or both, while sharing environmental, directional, and vessel-state predictors.Table 5 confirms that the common feature block is unchanged across variants.
  • Feature variants: SOG is available to planners before voyages, whereas STW is physically appropriate for hull resistance but requires a surface-current forecast to be specified ahead of time.The speed signals therefore represent different planning and physical-information roles.
  • Regression models: Six regression models span penalized linear estimators and nonlinear tree-based ensembles, with differing assumptions and capacities to fit and overfit speed–power relationships.Multiple Linear Regression is one linear estimator and minimizes residual sum of squares.

Multiple Linear Regression (MLR).

Multiple Linear Regression (MLR) offers direct interpretability through explicit coefficient estimates but does not control variance when predictors are correlated. It is retained as an unregularized reference for comparison with penalized linear models.

  • Multiple Linear Regression (MLR).: MLR provides direct interpretability through explicit coefficient estimates but offers no mechanism for controlling variance when predictors are correlated.
  • Multiple Linear Regression (MLR).: Ridge Regression instead adds an L2 penalty that shrinks coefficient estimates toward zero and stabilizes them under predictor correlation.

Ridge Regression.

Ridge regression uses a positive shrinkage parameter to trade bias for reduced variance. Unlike feature-selecting penalties, it retains all predictors, which suits variables that are weak individually but informative jointly.

  • Ridge Regression: Ridge regression uses α > 0 to control shrinkage, exchanging bias for reduced variance.The penalty regulates coefficient shrinkage while accepting bias as a variance-reduction trade-off.
  • Ridge Regression: Ridge retains all predictors rather than selecting among them, suiting feature sets with weak but jointly informative variables.This property is contrasted with Lasso, whose L1 penalty drives some coefficients exactly to zero for embedded feature selection.

Lasso Regression. · ElasticNet Regression.

Lasso produces a compact model but may select predictors unstably under strong correlation. ElasticNet combines coefficient shrinkage and variable selection, with its mixing parameter recovering Lasso or Ridge and supporting grouped selection of correlated predictors.

  • Lasso Regression.: Lasso yields a more compact model than Ridge.This compactness comes at the cost of instability in which predictor is retained when several predictors are strongly correlated.
  • Lasso Regression.: Strongly correlated predictors can make Lasso’s retained predictor unstable.Different correlated predictors may be selected despite similar predictive roles.
  • ElasticNet Regression.: ElasticNet combines coefficient shrinkage with variable selection.It balances both penalties rather than relying exclusively on one regularisation mechanism.
  • ElasticNet Regression.: ρ = 1 and ρ = 0 recover Lasso and Ridge, respectively.The mixing parameter ρ ∈[0, 1] governs the trade-off between the two penalty terms.
  • ElasticNet Regression.: ElasticNet is effective when environmental and hydrodynamic predictors exhibit grouped correlation.Correlated predictors tend to be retained or discarded together rather than selected arbitrarily.
  • ElasticNet Regression.: ElasticNet’s grouped selection reduces arbitrary choices among correlated predictors.This property contrasts with the instability described for Lasso under strong predictor correlation.
  • ElasticNet Regression.: Random Forest fits B regression trees to bootstrap training-data replicates and averages their predictions.It is presented as a bagging ensemble method.

Random Forest Regression. · Extreme Gradient Boosting (XGBoost).

Random Forest reduces variance by averaging decorrelated trees, while XGBoost builds a regularized, stage-wise additive tree ensemble using residual correction and subsampling. Both methods are designed to capture nonlinear dependencies and predictor interactions in high-frequency operational data.

  • Random Forest Regression.: Random Forest recursively partitions feature space using bootstrap samples and feature subsampling, then averages trees to reduce variance without a corresponding increase in bias.Node impurity is measured by mean squared error, and the method captures nonlinear effects and predictor interactions.
  • Extreme Gradient Boosting (XGBoost).: XGBoost fits regression trees stage-wise to residual signals left by preceding trees, forming an additive ensemble.Each boosting iteration updates the prediction using a newly fitted tree.
  • Extreme Gradient Boosting (XGBoost).: XGBoost minimizes a regularized objective with squared-error loss, penalizing tree complexity and leaf-weight magnitude.The objective uses the number of leaves and leaf weights, with γ, λ1, and λ2 controlling penalties.
  • Extreme Gradient Boosting (XGBoost).: Second-order loss approximation, shrinkage via η, and column and row subsampling extend conventional gradient boosting in XGBoost.These features support nonlinear dependencies and interaction effects in high-frequency operational data.
  • Extreme Gradient Boosting (XGBoost).: All six regression models use the same preprocessing pipeline across feature variants; MLR is unchanged, while the other five models are tuned.The common preprocessing is applied identically for every feature configuration.
  • 4.3. Physics-Based Baseline: The physics-based baseline models fuel flow with a leading cubic speed term and a linear mean-draft term, excluding environmental forcing.It represents the speed–displacement relationship underlying early-phase powering estimates rather than a complete resistance model.
  • 4.4.1. Importance of Proper Validation: Random train–test splits can place neighboring high-frequency observations in training and validation, making evaluation measure interpolation within a known operating window.At 1 Hz, even small overlap between adjacent folds can bias outcomes toward models that memorize local structure.
  • 4.4.4. Blocked Time Series Validation: The study compares sequential k-fold, forward-only TSCV, and independent blocked validation to impose progressively stricter temporal separation.Sequential k-fold uses five contiguous folds but can train on observations after validation; modified TSCV uses ten partitions with folds 6–10 evaluated at an 80:20 ratio, while BTSCV uses three independent blocks at 90:10.

Root Mean Squared Error (RMSE).

The supplied passages show a blocked time-series validation structure and define MAE as average prediction-error magnitude in original fuel-flow units. They do not provide a quantitative RMSE result.

  • Validation structure: Figure 8 divides 20 samples into 5 blocks to illustrate blocked time-series validation fold structure.The figure depicts the validation structure rather than an RMSE value.
  • Evidence scope: No passage supplied here reports an RMSE metric, model comparison, or numerical performance result.The available evidence concerns blocked validation structure and MAE definition.
  • Error metric context: MAE measures the average magnitude of prediction error in fuel-flow units (L h−1).The passage states that MAE does not disproportionately weight large deviations.

Mean Absolute Error (MAE). · Mean Accuracy. · Prediction Accuracy within 15% (PA15).

The section defines Mean Accuracy and PA15 as interpretable percentage-based measures of prediction error and operational accuracy. It also situates these metrics within the paper’s tuning, significance-testing, and explainability procedures.

  • Mean Absolute Error (MAE).: R2 expresses the proportion of fuel-flow variance explained relative to a constant-mean predictor.The metric is presented alongside MAE, Mean Accuracy, and PA15.
  • Mean Absolute Error (MAE).: Mean accuracy converts average relative error into a single interpretable percentage.This definition is attributed to.
  • Prediction Accuracy within 15% (PA15).: PA15 measures the percentage of predictions within ±15 % of observed fuel consumption.It provides a bounded error margin that is directly interpretable for operational deployment.
  • 4.6. Hyperparameter Tuning: 45 tuning runs covered three feature variants, three cross-validation schemes, and five tunable models, excluding MLR because it has no tunable hyperparameters.Each run used randomized search over the model-specific candidate grid.
  • 4.6. Hyperparameter Tuning: Random Forest hyperparameters were searched on 300,000 training rows, then the selected configuration was refit on the full training partition.The subsampling reduction applied only to the search stage.
  • 4.7.1. Statistical Significance Testing: Model performance differences on the chronological hold-out set were tested separately for each feature variant and cross-validation scheme.Primary comparisons concerned the best versus worst model and tree-based ensembles versus the linear family.
  • 4.7.1. Statistical Significance Testing: The Wilcoxon signed-rank test was applied to block-level mean loss differentials across 100 contiguous, equally sized temporal blocks.Blocking addressed dependence among adjacent 1 Hz observations and avoided overstating the effective sample size.
  • 4.7.2. Model Explainability: A representative linear model and tree-based ensembles were refit on the complete training partition for structure-appropriate interpretability analysis.Linear coefficients were examined directly, while tree-based ensembles were analyzed using SHAP.

5. Results

Results show that validation design substantially changes apparent fuel-consumption model performance: random splits favor tree ensembles, whereas chronological evaluation favors linear models. Across time-aware schemes and feature variants, Ridge generally delivered the strongest hold-out performance, with speed through water improving predictions and tree-model interpretability revealing physically relevant predictors.

  • Random versus chronological partitioning: Random splitting yielded test R2 of 0.99 for Random Forest and 0.98 for XGBoost, with RMSE of 17.38 and 23.11 L h-1, respectively.The four linear models attained test R2 between 0.71 and 0.80, whereas both ensembles achieved PA15 above 99%.
  • Random versus chronological partitioning: Chronological partitioning reduced ensemble test R2 to −0.36 for Random Forest and −0.10 for XGBoost, while Ridge improved test R2 from 0.80 to 0.84.Ridge RMSE decreased from 77.24 to 65.02 L h-1, and PA15 increased from 74.37% to 82.42%.
  • Cross-validation schemes: Blocked TSCV produced negative mean validation R2 for every model and variant except XGBoost under Variant C, which returned 0.01.Ridge under Variant A reached −12.23 ± 14.83, while the physics baseline returned −2.78±3.23.
  • Feature variants: Replacing speed over ground with speed through water reduced linear-model MAE by 14–19% from Variant A to Variant C under Sequential K-Fold.Corresponding PA15 gains were 8–14 percentage points, with comparable reductions under Custom TSCV and Blocked TSCV.
  • Cross-validation schemes: XGBoost Variant A hold-out test R2 ranged from 0.688 under Custom TSCV to −0.004 under Sequential K-Fold and 0.330 under Blocked TSCV.The corresponding test RMSE values ranged from 89.97 to 161.39 L h-1, despite identical features, training partition, and evaluation set.
  • Statistical comparison: Ridge had lower mean squared error than XGBoost and Random Forest in all nine comparisons against each, with all eighteen comparisons remaining significant after correction.Against Random Forest, Ridge was favored in 65 to 77 of 100 blocks; against XGBoost, it was favored in 62 to 71 blocks.

6. Discussion

The discussion shows that Ridge is largely insensitive to validation-scheme choice, whereas tree ensembles are highly sensitive to hyperparameter-selection scheme. It also argues for absolute-error metrics in operational reporting and identifies residual temporal-leakage and single-vessel, single-year scope limitations.

  • Model sensitivity: Ridge selected α = 1000 under every scheme and feature variant, producing the same fitted model in all nine cases.At this sample size, the penalty leaves Ridge coefficients effectively indistinguishable from unpenalised linear regression.
  • Model sensitivity: Random Forest hold-out RMSE varied by 12.55 to 28.33 L h−1, while XGBoost varied by 8.09 to 71.42 L h−1 across tuning schemes.For Variant A, XGBoost RMSE ranged from 89.97 L h−1 under Custom TSCV to 161.39 L h−1 under Sequential K-Fold despite identical features, training partition, and evaluation set.
  • Operational metrics: PA15 and Mean Accuracy provide more stable, operationally interpretable reliability information than R2 because they measure absolute deviation rather than variance explained within a fold.This distinction matters for decision-support systems, where whether predictions fall within an operationally meaningful margin is more direct than fold-specific explained variance.
  • Operational metrics: Variant A achieved approximately 83 % test PA15 under Ridge, compared with approximately 89–90 % for Variants B and C that additionally include speed through water.PA15 directly indicates that roughly one in six Variant A predictions falls outside an acceptable margin, which a single R2 value would not convey by itself.
  • Limitations: Chronological validation reduces temporal leakage but cannot ensure meaningfully different operating conditions across partition boundaries, and the study covers one vessel over one year.Additional vessels, seasons, and loading conditions are needed to test whether the performance figures generalise beyond this case without undermining the comparative validation methodology.

7. Conclusions

Using approximately 3.89 million steady-state 1 Hz records and a common chronological hold-out, the study shows that validation protocol materially changes apparent fuel-consumption model performance. Time-aware validation favored Ridge, while the physics baseline and voyage-planning feature restriction revealed practical accuracy trade-offs.

  • Study design: Approximately 3.89 million steady-state 1 Hz records supported comparisons of six regression models and a physics-based baseline across three feature variants and three time-aware validation schemes.All resulting models were evaluated on the same chronological hold-out set.
  • Validation protocol: 0.99 and 0.98 test R2 under a random 80:20 split fell to −0.36 and −0.10 chronologically for Random Forest and XGBoost, while RMSE rose from 17.38 and 23.11 to 187.81 and 168.98 L h−1.Training R2 remained above 0.98 for both models under both partitioning schemes.
  • Time-aware model comparison: Eighteen comparisons found Ridge had lower mean squared error than Random Forest and XGBoost across every feature variant and cross-validation scheme, with all differences statistically significant after Benjamini–Hochberg correction at α = 0.05.Ridge selected the strongest available penalty in every run, making its fit effectively equivalent to unpenalised linear regression.
  • Baseline comparison: 0.71 test R2 and 86.09 L h−1 RMSE under Variant A were achieved by the three-coefficient physics-based baseline, which outperformed tuned tree-based ensembles but remained below the linear models.These results were obtained under the same conditions used for the comparison.
  • Limitations: Limited operating range in this single-vessel, single-year dataset likely contributed to weaker Random Forest and XGBoost hold-out performance, while broader vessel, seasonal, and loading-condition coverage may narrow or reverse the gap.Both ensembles are bounded by the range of values seen during training.
  • Feature availability: Approximately 83% test PA15 for Variant A compared with 89–90% for Variants B and C quantified the accuracy cost of using only voyage-planning inputs available before departure.Variants B and C additionally incorporate speed through water; the gap is the accuracy a voyage planner forgoes for pre-leg prediction.

8. Future Work … Funding Sources

Future work should test the models across broader operational conditions, evaluate sequence-aware and physics-informed neural networks, and align predictions with voyage-segment decisions. Data remain confidential, with derived results available upon request; authorship and funding are documented.

  • 8. Future Work: The study’s single-vessel, eleven-month scope limits training and evaluation across operating and loading conditions.Additional vessels, seasons, and loading conditions could test whether weaker tree-ensemble hold-out performance reflects restricted operating conditions or a general property.
  • 8. Future Work: Recurrent and temporal-convolutional networks could exploit sequential high-frequency operational data directly.These architectures were not evaluated in the study and represent a natural extension.
  • 8. Future Work: Physics-informed neural networks could embed resistance relations as constraints on learned functions.This may combine the extrapolation behaviour of the physics-based baseline with the flexibility of learned models.
  • 8. Future Work: Predictions at discrete voyage-segment level would better match the resolution of voyage-planning decisions.The present analysis predicts fuel consumption independently at each 1 Hz observation.
  • Data Availability: The operational datasets are not publicly available because confidentiality provisions and their sensitive, proprietary nature restrict data sharing.Derived and aggregated results supporting the conclusions are available from the corresponding author upon reasonable request, subject to institutional approval.
  • CRediT authorship contribution statement: Samarasimha Reddy Chittamuru led conceptualization, methodology, data curation, formal analysis, software, validation, visualization, investigation, and the original draft.The contribution statement also assigns supervision, review, project administration, funding acquisition, resources, and related responsibilities to the other authors.
  • Funding Sources: The research was supported by NRC through the Greening Government Fund and Office of Energy Research and Development projects.The OERD project code is NRC-23-138.

Appendix A. Full-Metric Validation Tables

Appendix A provides full-metric validation tables covering random splits, chronological hold-out evaluation, cross-validation across feature variants, and final hold-out testing. The tables report default or cross-validation-selected model evaluations, with units specified for RMSE, MAE, Accuracy, and PA15.

  • Random-split validation: Table A.8 reports model performance for a single random 80:20 train–test split under Variant A with default hyperparameters.This table covers the full metric set.
  • Split-based validation: Tables A.9 and A.10 provide full-metric Variant A results for repeated random splits and a chronological hold-out split, respectively.Table A.9 summarizes mean ± standard deviation across 10 independently reseeded random 80:20 splits, while Table A.10 uses the earliest 80% for training and most recent 20% for testing without reshuffling.
  • Cross-validation: Table A.11 reports cross-validation performance across all three feature variants and training and validation portions under each cross-validation scheme.Values are presented as mean ± standard deviation across folds, and the physics baseline is available only for Variant A.
  • Final hold-out evaluation: Table A.12 presents final full-metric evaluation across all three feature variants on the common chronological hold-out set.MLR and the physics baseline are reported once per variant; five tunable models were refit on the full training set using hyperparameters selected under each cross-validation scheme.
  • Metric units: RMSE and MAE are reported in L h−1, while Accuracy and PA15 are reported in percent.These units apply to the appendix validation tables.
Loading 2608.16833v1…