Source-linked AI summary
Validated Data Onboarding for AI Demand Forecasting on U.S. Building Meter Data: Design, Controlled Evaluation, and a Corrected Negative Result
Yixuan Liang
TL;DR
Meter defects raise a testable question: can onboarding detect and repair them using only information available at forecast time, before model training? The paper evaluates a past-only, hash-bound pipeline in clean, corrupted, and repaired conditions. With 108 corrupted rows, gradient-boosting error rose 86%, while repair returned learned models to their clean-data error in the reported experiment.
Problem
The paper asks whether a pre-training onboarding pipeline can detect and repair meter-data defects using only information available at forecast time, and measurably protect forecasts.
Source: §1 · p. 1Method
Results
108 corrupted rows raised gradient-boosting error by 86%, while detection and past-only repair returned every learned model to its clean-data error.
Source: §5.1 · p. 4Takeaways & Limitations
The corrected pipeline protected forecasts in this experiment while retaining 93.0% of training targets.
Source: §5.1 · p. 4Hourly error intervals may be too narrow because errors are correlated, and a pooled model lets one building's fault move every building's error.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Electric utilities and grid operators increasingly rely on machine-learning models to forecast next-day demand, and those models learn from meter data that is routinely defective: readings go missing, sensors freeze, buildings read zero for hours, and units change by a factor of 100. This report presents a data-onboarding pipeline that detects and repairs such defects before a model is trained, using only information available at forecast time, and a controlled experiment that measures whether the pipeline protects a 24-hour-ahead forecast. On hourly electricity data for twelve U.S. buildings from the public Building Data Genome 2 dataset (210,528 rows, 2016-2017), seeded, hash-logged defects touching 0.10% of the training period raised the error of a gradient-boosting forecaster by 86%; after detection and past-only repair the error returned to the clean-data level (mean absolute scaled error 0.760 clean, 1.415 corrupted, 0.729 repaired) while 93% of training targets were retained. At a defect prevalence calibrated to published field studies (1.6% of training rows) the unprotected forecaster's error reached 4.4 times that of a seasonal-naive rule, and the repaired forecaster again matched the clean baseline. The same pattern held for ridge regression and a random forest and across horizons of 1 to 24 hours. A first version of the pipeline over-cleaned natural data and made forecasts 25% worse; that result is retained, its cause is traced in the published artifacts, and the per-building calibration that corrects it is documented as a dated amendment. Every number is reproducible from pinned public inputs with SHA-256 verification, 84 automated tests and continuous integration.
1. Motivation
The paper asks whether a past-only onboarding pipeline can detect and repair meter defects before training and protect forecasts. Its contribution combines calibrated checks, hash-bound artifacts, a controlled three-condition evaluation, and an auditable account of an initial failure.
1. Motivation
- Grid demand growth and variable renewable generation increase the operational context for forecasting, while meter streams can contain missing, frozen, zero, or mis-scaled readings.Field studies report missing readings in smart-meter deployments, zero readings in some intervals, and unit or multiplier errors affecting some meters and records.Source: §1 · p. 1
- The central question is whether an onboarding pipeline can detect and repair meter defects using only information available at forecast time, and measurably protect forecasts.Source: §1 · p. 1
- The contribution pairs per-meter checks, past-only repairs, and hash-bound artifacts with a clean-corrupted-repaired evaluation and a documented first-version failure.Source: §1 · p. 2
2. How common are meter-data defects?
Published field studies report substantial variation in meter-data defects, from missing readings and zero values to unit errors and anomalous readings. These reports motivate sparse and dense experimental corruption levels.
2. How common are meter-data defects?
- Missing-read rates range from 0.03% in the Irish CER trial to 17.2% of expected half-hourly reads in the British SERL panel.Source: §2 · p. 2
- Zero values made up 6.2% of readings in one example building and appeared in 1.04% of customer-days in a residential program.Source: §2 · p. 2
- In the SERL panel, 7.2% of expected daily electricity reads were flagged as likely recorded in the wrong unit, while 1.05% of residential records were multiplier-consistent duplicates.Source: §2 · p. 2
- Manual annotation of ASHRAE Great Energy Predictor III meters labeled 1.66% of non-missing readings anomalous, motivating experiments at 0.10% and 1.6% of training rows.Source: §2 · p. 2
3. System design
The system verifies and normalizes public inputs, applies per-building quality checks, injects logged training-period faults, and repairs them using past-only rules. Hash-bound conditions and verification tools preserve the provenance of the resulting benchmark inputs and results.
3. System design
- The pipeline verifies public source files against pinned hashes, normalizes them, applies twenty per-building-calibrated quality gates, and publishes three hash-bound conditions for benchmarking.Source: Figure 1 · p. 2
- The three Building Data Genome 2 input files are pinned by URL, byte size, and SHA-256, and mismatched downloads are refused.Source: §3 · p. 2
- Each condition file has six specified columns and one row per building-hour; twenty named checks cover schema, timestamps, values, metadata, and train-test coverage.Structural failures block publication, while value-quality checks use explicit count or rate thresholds.Source: §3 · p. 3
- The stuck threshold is calibrated per building from clean pre-cutoff history, while zero runs count only during hours when the building is normally active.Source: §3 · p. 3
- Six fault families are seeded only in training data, with affected keys and values logged; a consumer-side check rejects bundles that alter post-cutoff rows.Source: §3 · p. 3
- Repairs forward-fill gaps up to three hours or use prior eight-week same-weekday, same-hour medians for gaps up to 48 hours; other cases may be quarantined.Repairs use earlier, originally valid readings only, and no repair reads a later value.Source: §3 · p. 3
- Verification rechecks producer hashes, keys, changed values, and cutoff before fitting, then recomputes metrics, intervals, and figures from stored predictions.Source: §3 · p. 3
4. Experimental protocol
The experiment compares models trained on clean, corrupted, and remediated data against identical test targets, using a fixed forecasting protocol. Dates, features, models, hyperparameters, and metrics were specified before the runs they govern.
4. Experimental protocol
- The dataset comprises twelve U.S. buildings with hourly electricity data from January 2016 through December 2017, totaling 210,528 rows.Source: §4 · p. 3
- Training ends December 31, 2016, and test covers April through December 2017 with 79,178 targets; all three training conditions use identical test hours and reference targets.Faults are seeded only before January 1, 2017; the reserved 2017 Q1 validation partition is unused because models and hyperparameters were fixed a priori.Source: §4 · p. 3
- Features use load and weather information available at t−24 hours or earlier, and four fixed models include seasonal-naive, ridge, random forest, and histogram gradient boosting.No weather forecast is used, and preprocessing is fitted on training rows only.Source: §4 · pp. 3–4
- Pooled MASE uses a 168-hour seasonal scale computed once from clean training data, so 1.0 equals the seasonal-naive rule.Supporting measures include MAE, RMSE, weighted MAPE, R², per-building MASE, and paired error differences; a direct multi-horizon run reports horizons 1–24.Source: §4 · p. 4
- Split dates, features, models, hyperparameters, and metrics were recorded in advance, while v3 was documented as a dated amendment after the first result was examined.Both versions' result bundles are retained and verified.Source: §4 · p. 4
5. Results
The corrected pipeline restored forecasting performance after seeded defects, and repair also matched clean-data performance at literature-level defect prevalence and across 1–24-hour horizons. The results retain an earlier version’s worse-than-corrupted outcome and trace it to over-cleaning natural data.
5.1 The corrected experiment (v3)
- 108 corrupted rows raised gradient-boosting error by 86%; after repair, learned models returned to clean-data error while retaining 93.0% of training targets.Repaired scores slightly below the clean baseline were observational, alongside repair or quarantine of about 9,100 natural 2016 anomalies.Source: §5.1 · p. 4, Table 1 · p. 4, Figure 2 · p. 5
5.2 At the defect prevalence the literature reports
- At 1.6% defect prevalence, unprotected forecasting error was 2.7 to 4.4 times the seasonal-naive rule, while repaired models matched sparse-density performance and retained 95.8% of training targets.Table 2 covers 855 seeded events on 1,660 training rows.Source: §5.2 · p. 5, Table 2 · p. 5
5.3 Horizons 1 to 24 hours
- Across horizons 1–24, pooled MASE was 0.793 clean, 1.140 corrupted, and 0.776 repaired; the ordering held at every horizon.The direct multi-horizon sensitivity used one origin per day at 23:00 and a 22-feature forecast set.Source: Figure 3 · p. 5
5.4 One building, step by step
- For the education building, gradient-boosting MAE was 11.1 kWh clean, 19.2 corrupted, and 10.4 repaired after flagged hours were replaced using the prior eight weeks’ same-weekday, same-hour median.The illustrated faults were a seeded ×100 unit error and an 8-hour zero block.Source: Figure 4 · p. 6
5.5 The negative result that came first, and its diagnosis
- The first pre-specified version made remediated forecasts 25% worse than corrupted forecasts (MASE 0.922 versus 0.740); v3 calibrated run detectors per building, flagged whole runs, and changed its repair rules.V2 quarantined 22,033 rows, including 22,014 natural data rows and only 19 seeded faults; Figure 5 compares remediation before and after calibration and at literature-level prevalence.Source: §5.5 · p. 6, Figure 5 · p. 7
6. Limitations and threats to validity
The benchmark is limited to twelve buildings at three Eastern-time-zone sites and does not claim population representativeness. Its uncertainty and detector limits include correlated hourly errors, unlabelled natural anomalies, undetectable seeded faults, withheld weather forecasts, and a second pipeline version designed after the first result.
Scope
- Twelve buildings at three sites in one time zone are evaluated retrospectively, with no claim of population representativeness.The benchmark covers two years at hourly resolution and is not an operational system.Source: §6 · p. 7
Uncertainty
- Hourly error intervals may be too narrow because errors are correlated, and one pooled model lets a fault on one building affect every building’s error.A block-bootstrap sensitivity run gives wider intervals.Source: §6 · p. 7
Detector precision
- Natural anomalies lack ground truth, so flagged natural readings are counted as false positives by convention; 7.0% of sparse-run and 4.2% of dense-run training targets were quarantined.The sparse run included 11,753 hours of natural zero runs.Source: §6 · p. 7
Undetectable by construction
- Three meters’ natural constant runs set stuck thresholds of 62, 73, and 1,371 hours, making an 8-hour seeded stuck segment undetectable.This affected one of four such detector-validation events and two of four forecasting events; 16 of 108 seeded rows passed unrepaired.Source: §6 · p. 7
Weather
- Adding lagged weather and a fuller calendar worsened clean-data gradient-boosting error from 0.730 to 0.760 (+4%), while the responsible features remain unidentified.The protocol withholds next-day weather, which likely matters for next-day load.Source: §6 · p. 7
Sequence of versions
- The v3 pipeline was designed two hours after the v2 result was examined, making it a dated second run rather than confirmation of a registered hypothesis.The v2 files remain unchanged for audit.Source: §6 · p. 7
7. Relevance and planned work
The reusable contribution is a data-onboarding and evaluation design for organizations training on meter or sensor streams. Planned work extends it to transmission-scale data, more sites, forecast-weather inputs, and streaming ingestion.
7. Relevance and planned work
- The data contract, per-meter calibrated checks, hash-bound provenance, and three-condition evaluation design can be reused to test cleaning rules before production use.The paper connects this design to the data-quality foundation presupposed by the Genesis Mission grid challenge and AI Action Plan dataset agenda.Source: §7 · p. 7
- Planned extensions include transmission-scale data, more Building Data Genome 2 sites and a second time zone, forecast-weather inputs as a separately labeled experiment, and a streaming ingestion interface with the same contracts.Source: §7 · p. 8
8. Reproducibility statement
The project publishes its inputs, code, configurations, protocols, amendments, and result bundles, with hash verification and automated tests run in continuous integration. Fresh runs reproduced specified outputs on the author’s pinned environment, but cross-platform byte identity is not claimed.
8. Reproducibility statement
- Eighty-four automated tests and SHA-256 verification of committed bundles run in continuous integration on each push.A fresh run reproduced onboarding outputs and seasonal-naive, ridge, and gradient-boosting predictions exactly on the author’s pinned environment.Source: §8 · p. 8
- Cross-platform byte identity is not claimed; random-forest predictions match to about 10^-12, and CI recomputes committed-bundle metrics on Linux.Pinned public inputs are verified by size and SHA-256 before each run.Source: §8 · p. 8
Appendix A. Artifacts behind each number
The appendix maps the reported tables and figures to their underlying result files and identifies the artifact bundles for detector metrics, thresholds, and quarantine outputs.
Appendix A. Artifacts behind each number
- Tables 1 and 2 and Figures 2 and 5 map to metrics, paired-difference, and building-MASE files; Figure 3 maps to horizon metrics.The passage also lists separate bundles for detector metrics, thresholds, and quarantine outputs.Source: Appendix A · p. 8