When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting

• The streaming protocol uses non-overlapping windows at stride equal to the horizon and scores each target before it can enter a gradient step.This prevents the target leakage that can occur with overlapping prediction windows; the implementation reproduces the known leakage effect against a delayed-adaptation control [2].Source: §I · p. 1
• Validation early-stopping selects a warmup reference whose estimated benefit differs from the test-selected oracle by at most +3.4 pp in six examined panels.The warmup effect is two-sided and non-monotone: insufficient warmup undertrains the static baseline, while excessive warmup can worsen its generalization to the drifted test window.Source: §I · pp. 1–2
• Optimizer-specific rate rehearsal changes the SGD+m-versus-Adam comparison, which a shared 10−3 rate can bias through differing learning-rate robustness.Each optimizer’s rate is selected by rehearsing online adaptation on held-out pre-drift data.Source: §I · p. 2
• The study characterizes accuracy, adaptation-state memory, and A100-measured update latency; several parameter-efficient variants are nondominated on memory in evaluated PatchTST frontier settings.Tuned choices are made per device at commissioning from its own pre-drift data, without test data or cross-device coordination; smart-meter analyses include a 280-meter site and a 240-meter fleet spanning 18 of 19 BDG2 sites.Source: §I · p. 2

Evaluation leakage in online TSF

• Prior leakage analyses [2] motivate the delayed-ground-truth protocol adopted here, while baseline warmup sensitivity is identified as a separate evaluation issue.Related streaming adapters include δ-Adapter, SOLID, TAFAS, and an approach that adapts only on matured ground truth [8]; the paper adopts this protocol line rather than extending it.Source: §II · p. 2

Parameter-efficient and test-time adaptation

• PETSA motivates the paper’s simplified calibration variant, while broader TSF adaptation methods provide context rather than benchmark competitors.The variant uses per-channel affine input calibration (2C parameters) plus the linear output head, rather than PETSA’s low-rank additive modules and dedicated calibration loss; it is not compared with PETSA’s official implementation. FSNet, OneNet, D3A [12], and PROCEED are also contextualized, with OneNet’s two-model-instance cost outside adaptation-state accounting.Source: §II · p. 2

Building-energy forecasting at the edge

• Prior edge forecasting work addresses online adaptation or on-device training, but the paper identifies a gap in jointly evaluating online TSF adaptation under drift with memory and compute criteria.Online building-load methods omit adaptation-state memory and per-update compute; smart-meter federated split learning and on-device PV training [16] do not adapt online under drift.Source: §II · p. 2

When to adapt

• “Adapt Only When It Pays” [17] frames whether to spend an update as a budgeted decision, whereas this work compares choices and memory costs conditional on adapting.The paper’s small-benefit regimes on independent data further qualify that whether-to-adapt question.Source: §II · p. 2

Fair-evaluation methodology

• Fair-benchmarking work motivates validation-based baseline and learning-rate selection, while TTAB highlights model quality and hyperparameter selection as fairness concerns.Unlike the monotone dependence on source quality identified by TTAB, this paper reports a two-sided warmup effect, so simply training the baseline well is insufficient.Source: §II · p. 2

Data

• Six public multivariate series cover four ETT subsets, Appliances, and a 15-building, least-missing BDG2 Rat subset, split in time and z-normalized using warmup-only statistics.The Rat subset contains two years of hourly meter readings; missing values are forward-filled per meter, with any leading missing prefix backward-filled from its first valid observation.Source: §III · pp. 2–3

Quality metric and grid

• The optimizer grid contains 72 dataset-backbone-horizon-lookback configurations across five seeds, yielding 360 cells; online prequential MSE and relative adaptation benefit are the quality measures.Adaptation benefit is (MSEstatic − MSEadapt)/MSEstatic in %, with differences between benefit percentages reported in percentage points; standard deviations are descriptive, not confidence intervals.Source: §III · p. 3

Leakage-free streaming protocol

• Each streamed window predicts H steps from the previous L steps, is scored before adaptation on its revealed lookback-truth pair, and uses non-overlapping stride H.This ordering ensures each scored target is evaluated before it can enter any gradient.Source: §III · p. 3

Verifying the protocol

• Target leakage inflated reported adaptation benefit by +11.3 to +39.8 pp; for ETTh2/PatchTST it reversed the result (+24.2% leaky vs. -15.6% delayed).The delayed stride-1 control adapts only on fully revealed windows; its remaining -23.4 to -7.9 pp difference from stride-H evaluation is consistent with denser evaluation and an H× higher update count. Later results use the leakage-free protocol.Source: §III · p. 3

Adaptation strategies and schedules

• The study compares static, full, head-only, and calibration adaptation with every-step, periodic, or error-triggered schedules, sweeping schedule parameters to compare quality at matched update fractions.DLinear’s head-only variant updates its trend-branch map; PatchTST calibration adapts per-channel input/output calibration, and §IV-C adds Adam pairings for head and calibration.Source: §III · p. 3

Resource metrics

• Resource measurements include optimizer-state bytes, per-update wall-clock, illustrative time-power, and trainable-parameter count; adaptation-state memory counts gradient buffers plus optimizer state, excluding weights and activations.For fp32, the reported accounting is 8 B/param for SGD+m and 12 B/param for Adam; latency is the median after a discarded warmup prefix, minimized over interleaved repeats.Source: §III · p. 3

Validation-selected warmup

• Warmup budgets are early-stopped on the most-recent 20% held-out pre-drift validation slice; downstream measurements use a seven-milestone 200–20,000-step grid, where 42 of 360 cells select its capped maximum.A separate 10-milestone 50–50,000-step study grid compares validation selection with an oracle budget minimizing static test error, used only to quantify validation-criterion proximity.Source: §III · pp. 3–4

Validation-selected online LR (rehearsal)

• Online rates are chosen per strategy and optimizer by rehearsal on held-out pre-drift data over ten candidates from 3×10−6 to 10−1, then held constant through the stream.The one-off commissioning cost is ten short validation streams per optimizer, about two online passes in these splits; Adam uses default β1=0.9 and β2=0.999.Source: §III · p. 4

A. C1 — Warmup Sensitivity and a Validation-Selected Protocol

• Across six panels, under-warming raised estimated adaptation benefit by +0.6 to +26.0 pp and over-warming by +1.0 to +21.9 pp, while static-baseline test error was U-shaped.The sweep covered 50 to 50,000 warmup steps; the table reports benefits at 50 steps, the test-selected oracle reference, and 50,000 steps.Source: A. C1 — Warmup Sensitivity and a… · p. 4, Table I · p. 4
• Under-warming can make measured benefit include completion of base training, whereas over-warming reflects pre-drift overfitting; the latter was pronounced for PatchTST and mild for DLinear.The over-warming pattern appeared in both examined building-data settings, with regime-dependent magnitude.Source: A. C1 — Warmup Sensitivity and a… · p. 4
• Within the practical 1,000–20,000-step sweep, estimated adaptation benefit moved by 3.0 to 18.8 pp across six panels, exceeding the optimizer gap compared in §IV-B.Source: A. C1 — Warmup Sensitivity and a… · p. 4
• Early-stopping warmup on held-out pre-drift validation changed estimated benefit by +0.0 to +3.4 pp relative to the test-selected oracle across all six panels.The residual is consistent with distributional change between the validation slice and drifted test, so the procedure selects using validation rather than test data.Source: A. C1 — Warmup Sensitivity and a… · p. 4
• Validation selection need not reproduce the oracle budget: BDG2/DLinear selected 50,000 rather than 1,000 steps but changed estimated benefit by only +1.1 pp.The static-test curve was flat enough that this distant budget had little effect.Source: A. C1 — Warmup Sensitivity and a… · p. 4
• On ETTm2/DLinear, validation selected 4,000 steps against an oracle reference at 20,000, with a +3.4 pp residual in estimated benefit.The static-test curve was flat enough that this distant budget had little effect; the oracle reference minimizes seed-averaged static-test MSE and uses test data.Source: A. C1 — Warmup Sensitivity and a… · p. 4, Table I · p. 4

The apparent asymmetry at the default rate

• At a shared 10−3 rate, full·Adam is below static in 174/360 cells (minimum −46.5%) versus full·SGD+m in 29/360 (minimum −21.2%).The 10−3 rate is Adam’s original default, the default for torch.optim.Adam and torch.optim.SGD, and OneNet’s test-time rate; FSNet uses 10−4, which yields the opposite conclusion. 69 of 72 configurations agree on the winner across five seeds, and Adam’s negative-cell share rises from 36% at L=96 to 61% at L=192.Source: B. C2 — A Shared Learning-Rate Default… · pp. 4–5

The asymmetry depends on the rate

• Across a ten-point 3×10−6–10−1 sweep, Adam’s nonnegative-benefit range ends at 3×10−5, while SGD+m’s extends through 10−4.At fixed 10−4, Adam wins 322 of 360 cells with 2 negative cells and mean benefit +15.1%, versus SGD+m’s best fixed-rate mean benefit of +12.1% at 3×10−4. Adam@10−2 error on ETTm2 rises from 0.32 in the first quarter to 0.78 in the last, versus 0.09 for the static model.Source: B. C2 — A Shared Learning-Rate Default… · p. 5

The comparison at rehearsed rates

• At rehearsed rates, Adam wins 310 of 360 cells, has 4 below-static cells (worst −1.3%), and beats oracle-rate SGD+m in 246 of 360.Adam wins 100, 102, and 108 of 120 cells at H=24, 48, and 96, respectively. Its rehearsed rate is at most 3×10−4 in 340 of 360 cells; longer lookbacks narrow its margin from median +2.1 to +1.0 pp without changing its sign.Source: B. C2 — A Shared Learning-Rate Default… · p. 5, Table II · p. 6

Limits of the rehearsal criterion

• Rehearsal trades SGD+m’s greater rate robustness and one state copy per parameter for quality, but pre-drift validation does not certify stability under drift.SGD+m rehearsal selects 3×10−2 or 10−1 in 14 of 360 cells; 7 picks fall below static (worst −4.6%), while a conservative Adam rule eliminates below-static picks but alters 337 picks at a median cost of 3.6 pp.Source: B. C2 — A Shared Learning-Rate Default… · pp. 5–6

Regime probes do not predict the optimizer

• Three frozen-model probes—stream noise, gradient consistency, and post-hoc drift ratio—provide no usable optimizer guidance.At the default, they only weakly separate cells and mainly indicate where the shared rate leaves Adam’s empirical nonnegative-benefit range; at rehearsed rates, the choice they were meant to inform has largely dissolved.Source: B. C2 — A Shared Learning-Rate Default… · p. 6

C. C3 — The Accuracy–Memory–Compute Frontier

• On PatchTST, parameter-efficient strategies trade accuracy against adaptation-state memory differently across ETTm2 and Appliances, with head·Adam nondominated against full·SGD+m on ETTm2.On ETTm2, calib reaches +8.2±5.3% versus full·SGD+m at +10.2±4.5%; head matches full·SGD+m’s improvement with lower measured compute (1.86 ms versus 2.88 ms). On Appliances, the six points form a monotone mean trade-off: calib +42.1% at 136 kB, calib·Adam +44.5% at 204 kB, full·SGD+m +47.3% at 686 kB, and full·Adam +52.5% at 1,028 kB, with no plotted point dominating another. Head·Adam reaches +10.7% at 203 kB against full·SGD+m’s +10.2% at 685 kB on ETTm2.Source: C. C3 — The Accuracy–Memory–Compute… · p. 7
• Rehearsing rates changes frontier rankings: ETTm2 full·Adam shifts from -19.6% at the shared default to the highest-mean-benefit endpoint, while Appliances full·SGD+m falls from +53.1% to +47.3%.For Appliances calib, rehearsal selects the default rate in all five seeds and leaves its benefit at +42.1%. The paragraph reports P3 as the strongest single indicator at the default (mean 1.03 in SGD-win versus 1.37 in Adam-win cells), with P2 and P1 gaps of +0.01 and -0.03.Source: C. C3 — The Accuracy–Memory–Compute… · pp. 6–7
• Against an illustrative 192 kB adaptation-state allowance, full-model PatchTST exceeds the allowance at 686–1,028 kB, PEFT with Adam is marginally above at 203–204 kB, and PEFT with SGD+m is below at 135–136 kB.This accounting includes gradient buffers and optimizer state, but excludes model weights, activations, and runtime memory; the 192 kB reference comes from smart meters demonstrated with federated split learning.Source: C. C3 — The Accuracy–Memory–Compute… · p. 7
• At the A100 batch-1 timing proxy, 0.85–3.69 ms updates equal 0.31–1.35 s of measured update time per meter per year for one update per revealed horizon at H=24.The 5 W proxy gives 4.3–18.5 mJ per update; these are not edge-device scaling results, and the calculation suggests adaptation-state memory and staleness may matter more than the measured A100 timing proxy.Source: C. C3 — The Accuracy–Memory–Compute… · p. 7
• At matched update budgets, drift-triggering beats periodic scheduling on both Appliances panels and both ETTm2 panels, but trails on both ETTh2 panels, so no schedule dominates across datasets.The reported differences are +5.6±1.0% lower MSE for SGD+m and +1.9±0.3% for Adam on Appliances; +2.6±1.4% and +4.5±0.4% on ETTm2; and -1.1±1.0% and -2.8±1.6% on ETTh2.Source: C. C3 — The Accuracy–Memory–Compute… · pp. 7–8
• BDG2 meter-selection rules change estimated benefits: PatchTST reports +42.8% on the 15 most-missing meters versus +13.5% on the least-missing 15, while DLinear reverses the ordering (+2.1% versus +6.4%).The anti-selected subset has a median 44.5% consecutive equal samples versus 0.6% for the §III subset, and many evaluation targets are forward-filled, so its larger benefit reflects the imputed stream under that preprocessing rule. With rules fixed, PatchTST reports +43.7% on a 280-meter site and +28.7% on a 240-meter, 18-site fleet; the inactive-meter rule was post hoc and results using it are exploratory.Source: C. C3 — The Accuracy–Memory–Compute… · p. 8, Table III · p. 8

• At commissioning, select the warmup budget and online learning rate on the held-out pre-drift validation slice, without accessing test data.Source: D. Conditional Deployment Guidance · p. 8
• When adaptation-state memory binds, first consider head-only or calibration-based adaptation, which often retained substantial full-model gain at lower memory in evaluated PatchTST settings.If state remains binding, the section also recommends applying Adam to the calibration/head subset or keeping momentum alone.Source: D. Conditional Deployment Guidance · p. 8
• Rehearsed-rate Adam—typically 1/10 to 1/30 of default in the 360-cell full-model grid—yielded median +1.5 pp at 1.5× adaptation-state memory.Source: D. Conditional Deployment Guidance · p. 8
• Without tuning, SGD+m at 10−3 was more rate-robust within the evaluated grid, but this is not a universal default; the post-hoc drift probe cannot identify heavy drift at commissioning.Source: D. Conditional Deployment Guidance · p. 8
• No schedule dominated across evaluated datasets, so validate periodic or drift-triggered scheduling per deployment; ETT/DLinear showed only small mean improvements, leaving whether to adapt as a decision.Source: D. Conditional Deployment Guidance · p. 8

V. DISCUSSION AND LIMITATIONS

• The validation slice is not trained on and no refit follows selection, leaving training data ≈73 days or ≈14 days before streaming; both arms share this limitation.Folding the slice into warmup could produce a fresher static baseline and change the reported benefit; refitting and varying the slice’s size or position remain unexplored.Source: §V · p. 8
• The optimizer conclusion applies only to SGD+m and Adam; other optimizer families and comparison with official PETSA remain future work.The passage specifically names AdaFactor, Prodigy, and ObGD as untested, and leaves a direct comparison of the simplified calibration point with official PETSA unresolved.Source: §V · p. 8
• Resource measurements are datacenter-GPU proxies: memory is counted, and batch-1 A100 latency is launch-bound, so near-flat scaling with meter count may not hold at the edge.Fig. 4 plots adaptation benefit against adaptation-state memory and A100 per-update time for Appliances and ETTm2; millijoule estimates assume a 5 W device, and Jetson-, Raspberry-Pi-class, or meter validation is still required.Source: §V · pp. 8–9, Figure 4 · p. 9

VI. CONCLUSION

• Baseline warmup and a shared online learning-rate default materially affect estimated adaptation benefits, even under a leakage-free streaming protocol.The warmup effect is two-sided and non-monotone; validation-only selection uses a held-out pre-drift slice without accessing test data and mitigates, but does not eliminate, these sensitivities.Source: §VI · p. 9
• Across six public multivariate series, the study characterizes accuracy against adaptation-state memory and A100-measured update latency.The series include a smart-meter corpus at fleet scale.Source: §VI · p. 9
• In evaluated frontier settings, several parameter-efficient variants are nondominated at lower adaptation-state memory than full-model alternatives.These measurements provide conditional deployment guidance; target-device memory, latency, and energy remain to be validated.Source: §VI · p. 9