PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting

Takumi Fujimoto, Hiroaki Nishi

arXiv:2609.01126v1cs.LG

TL;DR

Edge time-series adaptation must be evaluated under distribution drift and resource constraints, but measured benefits depend on baseline warmup and optimizer-rate choices. The paper uses leakage-free validation-based selection and evaluates accuracy alongside adaptation-state memory and update latency, finding optimizer-rate asymmetries and device-relevant evaluation limits.

  • Problem

    Edge forecasters face distribution drift and resource constraints, raising whether online adaptation helps and at what adaptation-state cost.

  • Method

    The study evaluates six public multivariate streams with leakage-free streaming, sweeps baseline warmup and optimizer rates, and selects choices using held-out pre-drift validation without test access.

  • Results

    Under the evaluated rate grid, Adam degrades more quickly at high rates, with 174 negative cells at 10^-3 versus 29 for SGD+m; frontier analyses also compare accuracy with adaptation-state memory and A100 update latency.

  • Takeaways & Limitations

    Validation-only selection provides conditional per-device guidance while exposing sensitivity to warmup and learning-rate choices in reported adaptation benefits.

  • Takeaways & Limitations

    The validation slice is not reused for training, and A100 batch-1 latency and counted memory may not represent embedded edge devices.

Abstract

from arXiv · show

Online adaptation can help edge time-series forecasting under distribution drift, but its measured benefit is sensitive to evaluation choices. We study six public multivariate streams, including building-sensor and smart-meter data, under a leakage-free streaming protocol. We identify two additional sources of comparison bias. First, the warmup budget of the static baseline has a two-sided effect: insufficient warmup undertrains the baseline, whereas excessive warmup can degrade its pre-drift generalization. Across six dataset-backbone settings, the estimated adaptation benefit changes by 3.0 to 18.8 percentage points (pp) over the 1,000-20,000-step warmup range. Second, comparing SGD with momentum (SGD+m) and Adam at a shared default learning rate conflates optimizer quality with rate sensitivity. We select both the warmup budget and each optimizer's online rate using a held-out pre-drift validation slice without accessing test data. Under this validation-only procedure, Adam outperforms SGD+m in 310 of 360 evaluated cells, while 4 Adam cells remain below the static baseline. We further characterize accuracy against adaptation-state memory and A100-measured per-update latency for full, head-only, and calibration-based adaptation. In the evaluated PatchTST frontier settings, several parameter-efficient variants are nondominated on the adaptation-state-memory axis. Smart-meter analyses also show that reported gains depend on meter-selection rules. These findings support a validation-only commissioning procedure, while target-device latency and energy remain to be measured. Code, data, and all reported numbers: https://github.com/keiotakmin/tsf-edge-adaptation.

I. INTRODUCTION

Online adaptation is motivated by distribution drift and edge resource constraints, but its measured value depends on a leakage-free, carefully calibrated evaluation. The paper studies warmup selection, optimizer-specific rates, and accuracy–memory–compute trade-offs using validation-only choices.

  • Distribution drift from occupancy, seasons, retrofits, and sensor aging motivates continuing to update forecasting models on-device.
  • Edge deployment makes optimizer state, peak memory, and per-update compute first-order costs alongside forecasting accuracy.
  • The study uses non-overlapping streaming windows, scores each target before adaptation, and prevents scored targets from entering gradients.
  • C1 — Warmup Sensitivity: The static baseline’s warmup budget has a two-sided, non-monotone effect on estimated adaptation benefit, so the paper selects warmup using held-out pre-drift data.
  • C2 — Learning-Rate Selection: A shared online rate can bias SGD+m versus Adam comparisons; the paper therefore rehearses optimizer-specific rates on the same held-out pre-drift slice.
  • C3 — Resource Trade-offs: The paper measures accuracy against adaptation-state memory and compute across parameter subsets and optimizers rather than proposing a new adapter or optimizer.

II. RELATED WORK

Prior work addresses leakage-free streaming and parameter-efficient or drift-aware adaptation, while this paper isolates warmup as a separate comparison sensitivity and measures resource costs across fixed backbones.

  • Evaluation leakage: Prior leakage work shows overlapping windows can expose future scoring targets during adaptation; this paper adopts delayed-ground-truth principles instead of extending that line.
  • Parameter-efficient adaptation: PETSA motivates the paper’s simplified calibration strategy, which uses per-channel affine input calibration plus the linear output head.
  • Resource-aware edge forecasting: Related online adaptation and building-energy studies provide algorithmic context but do not report the adaptation-state memory and per-update compute metrics used here.

III. EXPERIMENTAL SETUP

The experiments evaluate six public multivariate series with two contrasting backbones under leakage-free prequential streaming, validation-only selection, and explicit accuracy, memory, and latency measurements.

  • Data: The datasets include four ETT subsets, UCI Appliances, and BDG2 building data, with time-based warmup/test splits and warmup-only z-normalization.
  • Backbones: DLinear supplies a linear decomposition baseline, while compact channel-independent PatchTST supplies a transformer baseline with input affine calibration.
  • Metrics and grid: Adaptation benefit is (MSEstatic − MSEadapt)/MSEstatic in %, and the optimizer grid contains 360 seed-specific cells across six datasets, two backbones, horizons, and lookbacks.
  • Streaming protocol: Each test step predicts H steps from the previous L-step lookback, scores revealed truth, and only then adapts, preventing target leakage by construction.
  • Strategies and schedules: The study compares static, full, head-only, and calibration-based adaptation with SGD+m or Adam under every-step, periodic, and drift-triggered schedules.
  • Resource metrics: Adaptation-state memory counts gradient buffers plus optimizer state, while latency is measured as per-update wall-clock time on an A100.
  • Validation selection: Warmup and online rates are selected from held-out recent pre-drift data without test access, using early stopping and online rehearsal respectively.

A. C1 — Warmup Sensitivity and a Validation-Selected Protocol

Warmup affects adaptation estimates non-monotonically: both under-warming and over-warming can inflate the apparent benefit. The practical sweep still produces substantial variation, motivating validation-selected warmup.

  • Warmup sensitivity: The static baseline’s test error is U-shaped, while estimated adaptation benefit is larger at both ends of the 50–50,000-step sweep.
  • Under-warming: +0.6 to +26.0 pp is the increase in estimated benefit from under-warming across all 6 of 6 examined panels.
  • Over-warming: +1.0 to +21.9 pp is the increase in estimated benefit from over-warming across all 6 of 6 examined panels.
  • Practical sweep: 3.0 to 18.8 pp is the remaining benefit movement across the practical 1,000–20,000-step sweep.

B. C2 — A Shared Learning-Rate Default Can Bias Optimizer Comparisons

A shared online learning rate can make optimizer comparisons reflect rate sensitivity rather than optimizer quality. Validation-only, optimizer-specific rehearsal reverses the default-rate asymmetry, with Adam generally outperforming SGD+m across evaluated cells.

  • 174/360 full·Adam cells fell below the static baseline at the shared 10^-3 rate, versus 29/360 for full·SGD+m.The worst observed benefits were -46.5% for Adam and -21.2% for SGD+m.
  • A shared fixed rate can bias the comparison because SGD+m and Adam have different empirical ranges with no below-static cells.At 10^-3, the rate lies inside SGD+m’s empirical nonnegative-benefit range but outside Adam’s in the evaluated grid.
  • Even fixed 10^-4 changed the comparison: Adam won 322 of 360 cells with 2 negative cells and a mean benefit of +15.1%.SGD+m’s best fixed-rate mean benefit was +12.1% at 3×10^-4.
  • At rehearsed rates, Adam won 310 of 360 cells, with only 4 below-static cells, all from ETTm2/DLinear.Adam’s worst rehearsed benefit was -1.3%, while it beat SGD+m’s test-selected oracle rate in 246 of 360 cells.
  • Validation rehearsal trades SGD+m’s rate robustness for Adam’s higher benefits, but pre-drift validation does not certify stability under drift.A conservative within-2% rule eliminated below-static picks in this grid but altered 337 of 360 choices at a median cost of 3.6 pp.
  • Cheap frozen-model probes provided no usable optimizer guidance after rate rehearsal.At the fixed default, the probes mainly indicated where the shared rate placed Adam outside its empirical nonnegative-benefit range.

C. C3 — The Accuracy–Memory–Compute Frontier

The paper maps adaptation benefit against memory and compute using validation-selected settings, finding dataset-dependent trade-offs and several memory-nondominated parameter-efficient PatchTST variants. Smart-meter results further show that meter selection and preprocessing materially affect reported gains.

  • Accuracy–Memory–Compute Frontier: Several parameter-efficient PatchTST variants are nondominated on the adaptation-state-memory axis in the evaluated frontier settings.Frontier points use validation-selected warmup and rehearsed learning rates, with five-seed means and standard deviations.
  • Accuracy–Memory–Compute Frontier: On Appliances, calib reaches +42.1% at 136 kB, while full·Adam reaches +52.5% at 1,028 kB, forming a monotone mean trade-off.
  • Accuracy–Memory–Compute Frontier: On ETTm2, head·Adam is nondominated relative to full·SGD+m, achieving +10.7% at 203 kB versus +10.2% at 685 kB.
  • Accuracy–Memory–Compute Frontier: Relative to a 192 kB allowance, full-model PatchTST exceeds the allowance, PEFT with Adam is marginally above it, and PEFT with SGD+m is below it.
  • Scheduling and scale: No schedule dominates across the examined datasets, while BDG2 reports +43.7% on a 280-meter site and +28.7% on a 240-meter, 18-site fleet.
  • Smart-meter analyses: Meter selection changes reported benefit: PatchTST reaches +42.8% on the most-missing 15 meters versus +13.5% on the least-missing 15.The comparison is affected by forward-filled targets and differing consecutive-equal-sample rates.

D. Conditional Deployment Guidance

The paper recommends commissioning adaptation choices from each device’s held-out pre-drift validation data and conditioning strategy selection on memory and tuning resources. These recommendations remain conditional because schedule choice varies across datasets and some settings show small improvements.

  • Commissioning: Select warmup and online learning rate on the held-out pre-drift validation slice at commissioning.
  • Memory constraints: When adaptation-state memory binds, first consider head-only or calibration-based adaptation because evaluated PatchTST variants often retain substantial full-model gain at lower memory.
  • Optimizer selection: If optimizer state and rehearsal are affordable, use Adam at its rehearsed rate; in the full-model grid this was typically 1/10 to 1/30 of the default.The reported median improvement was +1.5 pp at 1.5× the adaptation-state memory.
  • Fallback choice: Without tuning, SGD+m at 10−3 was more rate-robust within the evaluated grid, but the paper cautions against treating it as a universal default.
  • Scheduling: Because no schedule dominates and ETT/DLinear settings showed only small mean improvements, adaptation itself remains a deployment decision.

V. DISCUSSION AND LIMITATIONS

The discussion limits the conclusions to the evaluated optimizer pair, simplified calibration comparison, and datacenter-GPU measurements. It also notes that validation selection leaves the deployed model trained on data ending before the stream and that target-device validation remains necessary.

  • Protocol boundary: Validation selection leaves the deployed model’s training data one validation span before the stream because the slice is neither trained on nor followed by refitting.This corresponds to approximately 73 days on hourly and 15-minute sets and approximately 14 days on Appliances.
  • Scope: The optimizer conclusion concerns only SGD+m versus Adam, not optimizers in general.Other optimizers and a direct comparison with official PETSA remain future work.
  • Hardware boundary: Memory is counted rather than observed under an embedded allocator, and A100 batch-1 latency is not an edge-device scaling result.
  • Hardware boundary: The millijoule figures assume a 5 W device, so Jetson-, Raspberry-Pi-, and meter-level validation is required before end-to-end deployability is claimed.

VI. CONCLUSION

The conclusion argues that adaptation conclusions remain sensitive to evaluation choices even under leakage-free streaming. Validation-only measurements provide conditional guidance, but target-device latency, memory, and energy still require validation.

  • VI. CONCLUSION: Two additional evaluation choices materially affect estimated benefit: the baseline warmup budget and optimizer-specific online learning-rate selection.
  • VI. CONCLUSION: The validation-only procedure characterizes accuracy against adaptation-state memory and A100-measured update latency across six public multivariate series.
  • VI. CONCLUSION: Several parameter-efficient variants are nondominated at lower adaptation-state memory than full-model alternatives in the evaluated frontier settings.
  • VI. CONCLUSION: Target-device memory, latency, and energy remain to be validated before the measurements support broader deployment claims.
Loading 2609.01126v1…