PaperlayerView exact revision on arXiv

Source-linked AI summary

Extreme Low-State Quantized Transformer for Electricity-Load Forecasting

Muhammad Sabbir Alam, Harindra Sandun Mavikumbure, Victor Cobilean, Milos Manic, Jayasimha Atulasimha

arXiv:2610.28784v1cs.LG

TL;DR

The paper asks how much forecasting performance can be retained when Transformer weights have only a few discrete states. It compares post-training quantization with training strategies that operate in the low-state regime, and reports that forecasting performance and strongly reduced weight-update activity can coexist.

  • Problem

    Few-state weight representations constrain model flexibility, making it difficult to preserve predictive performance under aggressive quantization.

    Source: §1 · p. 2

  • Method

    The study evaluates low-state Transformer forecasting using QAT-FT, Scratch-QAT, and accumulated-update thresholding.

    Source: §1 · p. 3

  • Results

    QAT-FT recovered approximately 90.2% of the PTQ penalty in Dataset A and 94.5% in Dataset B; thresholding reduced committed updates by approximately 99.7% and discrete state transitions by approximately 77–80%.

    Source: §1 · p. 3, §5 · p. 27

  • Takeaways & Limitations

    Extremely low-state Transformer learning remained viable, and accumulated-update thresholding strongly suppressed committed activity while producing only small mean changes in forecasting error.

    Source: §5 · p. 27

    Device write energy, endurance, lifetime, latency, and physical programming pulses were not measured.

    Source: §2.7 · p. 13

Abstract

from arXiv · show

Aggressive low-state weight quantization can substantially reduce neural-network representational flexibility, making predictive performance difficult to preserve when only a few discrete weight states are available. We investigate this regime for Transformer-based electricity-load forecasting using 3-, 5-, 7-, and 16-state weight quantization across feed-forward-only and attention-plus-feed-forward quantization. Two tasks predict either the next hour or the next 24 hours from the preceding 48 hours across two co-primary electricity-consumption datasets. Direct post-training quantization (PTQ) increased mean normalized root-mean-square error (NRMSE) relative to full precision by 6.25 and 7.04 percentage points in Datasets A and B, whereas quantization-aware fine-tuning (QAT-FT) recovered approximately 90.2% and 94.5% of these penalties. Quantization-aware training from random initialization (Scratch-QAT) also produced competitive low-state solutions, with particularly favorable behavior at three states. We further introduce accumulated-update thresholding, which accumulates optimizer-proposed weight changes before commitment. Relative to paired Scratch-QAT controls, this reduced committed shadow-weight updates by approximately 99.7% and discrete state transitions by approximately 77--80%, while mean NRMSE changed by only +0.042 and +0.197 percentage points. These results show that useful Transformer forecasting can be retained under highly discrete weight representations while strongly suppressing target-state activity, supporting a prospective route toward compact, low-activity edge forecasting using multilevel synaptic devices.

1 Introduction

The introduction frames few-state quantization as a challenge to predictive flexibility and examines training strategies for preserving forecasting performance while limiting committed weight activity.

1 Introduction

  • The study evaluates 3-, 5-, 7-, and 16-state weight quantization in feed-forward-only and attention-plus-feed-forward settings for one-hour and 24-hour forecasting across two co-primary electricity-consumption datasets.Source: §1 · p. 3
  • QAT-FT recovered approximately 90.2% of the PTQ penalty in Dataset A and 94.5% in Dataset B.Performance progressively approaches full-precision behavior as state resolution increases, while direct PTQ is particularly damaging at the lowest state counts.Source: §1 · p. 3
  • Scratch-QAT produced competitive low-state solutions from random initialization without full-precision pretraining, with particularly favorable behavior at three states.This tests whether the constrained weight space can support learning when quantization is active from the outset.Source: §1 · pp. 2–3
  • Accumulated-update thresholding reduced committed weight updates by approximately 99.7% and discrete state transitions by approximately 77–80%, with only small mean changes in forecasting error.It accumulates small optimizer-proposed changes before committing them to target weights.Source: §1 · p. 3
  • Principal PTQ, QAT-FT, and thresholding effects reproduced across both datasets and all four load-series analysis units, including transfer to Dataset B without dataset-specific retuning.Source: §1 · p. 4

2 Methods

The forecasting protocol uses two electricity-consumption datasets and predicts one hour or 24 hours ahead from the preceding 48 hourly observations.

2.1 Co-primary datasets and forecasting tasks

  • Two publicly available electricity-consumption datasets from different geographical and consumption settings were treated as co-primary and evaluated with the same forecasting formulation.Source: §2.1 · p. 4
  • Dataset A comprises three anonymous client series from Portugal, converted from 15-minute to hourly resolution; samples intersecting identified daylight-saving-time artifact hours were excluded.The series span January 2012 through December 2014.Source: §2.1 · pp. 4–5
  • Dataset B uses hourly Global_active_power from one French household, retaining valid hours only when all 60 minute-level observations are available and required observations are finite.No interpolation or imputation was performed.Source: §2.1 · p. 5
  • Both tasks use the preceding 48 hourly observations; H24 directly outputs the next 24 hours rather than recursively feeding predictions back into the model.Source: §2.1 · p. 5

2.2 Development protocol and final-test locking

Model development and final evaluation were separated chronologically, with methodological choices locked before final-test access and transferred to Dataset B without retuning.

2.2 Development protocol and final-test locking

  • Each dataset used three expanding-window validation folds, with validation intervals later than their corresponding training intervals, and final tests reserved until development and methodological locking were complete.Source: §2.2 · p. 5
  • Normalization was fit using only each training interval, excluding validation observations from scaler estimation.The same training-only principle applied to final model fitting.Source: §2.2 · p. 5
  • After Dataset-A development, the principal experimental choices were locked using development evidence, then transferred to Dataset B without Dataset-B-specific retuning.The locked choices included state counts, quantization locations, model configurations, and QAT/Scratch-QAT procedures.Source: §2.2 · pp. 5–6

2.3 Controlled Transformer architecture

A compact, conventional encoder-only Transformer serves as a common backbone, keeping the architecture controlled to isolate quantization and training-strategy effects.

2.3 Controlled Transformer architecture

  • The encoder-only Transformer was kept relatively compact to isolate low-state quantization and training-strategy effects rather than add a specialized forecasting architecture as another source of variation.The same frozen horizon-specific configurations were used for both datasets.Source: §2.3 · p. 6
  • Each 48 × 1 input is projected to 64 dimensions, receives a trainable positional embedding, and passes through pre-normalized encoder blocks with four-head attention and a 64→128→64 feed-forward network.Residual connections and dropout are used in the attention and feed-forward paths.Source: §2.3 · p. 6
  • The model forecasts from the final input token: H1 uses three encoder blocks and 103,809 parameters, while H24 uses one block and 38,360 parameters.Both configurations use 64-dimensional representations, four attention heads, and a feed-forward dimension of 128.Source: §2.3 · p. 6

2.4 Low-state weight quantization

The study uses uniform symmetric per-tensor quantization to examine forecasting with a small number of discrete weight states. It compares two quantization locations across selected state counts.

2.4 Low-state weight quantization

  • Each targeted weight maps to its nearest state on a uniform symmetric grid spanning the tensor-specific range [−s, s].The range is determined by the largest absolute weight value in the tensor.Source: §2.4 · p. 7
  • The final experiments used 3, 5, 7, and 16 states, spanning strongly quantized weights through a higher-resolution condition near diminishing differences.The odd-sized grids also contain an exact zero state.Source: §2.4 · p. 7
  • FFN-Q quantizes feed-forward kernels, while Attention+FFN-Q additionally quantizes query, key, value, and attention-output kernels.Activations, biases, LayerNorm, softmax, and other non-target parameters remain in FP32.Source: §2.4 · p. 7
  • The same state counts and quantization locations were used for both co-primary datasets.For H1, Attention+FFN-Q targets 94.70% of trainable parameters, compared with 47.35% for FFN-Q.Source: §2.4 · p. 7

2.5 Training strategies

The paper compares full-precision training, direct post-training quantization, fine-tuning from a full-precision checkpoint, and training from random initialization with quantization active. The pathways distinguish adaptation of a pretrained model from learning within the constrained weight space.

2.5 Training strategies

  • Four pathways were compared: full precision, post-training quantization, quantization-aware fine-tuning, and random-initialized quantization-aware training.All neural-network optimization used Adam with mean-squared error loss.Source: §2.5 · p. 8

2.5.1 Full precision

  • The full-precision model provided each dataset and horizon's unquantized reference and the common starting checkpoint for PTQ and QAT-FT.Its architecture was horizon-specific.Source: §2.5.1 · p. 8

2.5.2 Post-training quantization

  • PTQ maps trained full-precision weights to their nearest states using ranges from those weights, without retraining or adaptation.The PTQ and QAT-FT branches independently originated from the same pretrained checkpoint.Source: §2.5.2 · p. 8, §2.5 · p. 8

2.5.3 Quantization-aware fine-tuning

  • QAT-FT starts independently from the pretrained full-precision checkpoint and computes with quantized kernels while Adam updates continuous shadow weights.The fixed quantization ranges came from the original checkpoint; fine-tuning ran for 25 epochs.Source: §2.5.3 · pp. 8–9

2.5.4 Random-initialized quantization-aware training (Scratch-QAT)

  • Scratch-QAT begins from a fresh random initialization with quantization active from the first minibatch, rather than from a pretrained full-precision model.Its fixed ranges were derived from the saved initialization, and the primary endpoint was 40 epochs.Source: §2.5 · p. 8, §2.5.4 · p. 9

2.6 Accumulated-update thresholding

Accumulated-update thresholding controls when optimizer-proposed changes are committed to targeted shadow weights. It retains smaller changes until their accumulated magnitude reaches a tensor-specific threshold.

2.6 Accumulated-update thresholding

  • Thresholding acts after Adam proposes a continuous target-weight change and before that change is committed.It does not remove or skip forward passes, backward passes, minibatches, or Adam steps.Source: §2.6 · p. 9
  • For each targeted weight, sub-threshold proposals accumulate; when their magnitude reaches the threshold, the accumulated change is committed and its accumulator reset.The threshold is relative to tensor range s, not quantization-state spacing; α = 0 commits every nonzero proposal immediately.Source: §2.6 · pp. 9–10
  • A committed shadow-weight update need not cause a quantized state transition because it may remain within the same quantization interval.The next quantized value is computed from the committed shadow weight, not the uncommitted accumulator.Source: §2.6 · p. 10
  • The selected high-α rule used α = 0.25, 0.25, 1/6, and 1/15 for 3, 5, 7, and 16 states, respectively.The schedule was selected for stronger activity suppression with comparable forecasting performance and transferred unchanged to Dataset B.Source: §2.6 · p. 10

2.7 Update and transition accounting

The accounting separates proposed updates, committed shadow-weight changes, discrete state transitions, and transition distance. Paired runs support activity comparisons without treating reduced target-weight activity as reduced conventional optimization workload.

2.7 Update and transition accounting

  • Activity was counted per targeted weight element across minibatches, tensors, and epochs, rather than as a count of optimizer steps.The four recorded levels were proposed updates, committed updates, discrete transitions, and transition distance.Source: §2.7 · p. 12
  • A proposed update requires a nonzero Adam change after clipping, while a committed update requires an actual change in the committed shadow weight.For α = 0, every nonzero proposal was committed, and the implementation verified Nprop = Ncommit.Source: §2.7 · p. 12
  • State transitions are counted separately from committed updates, and transition distance measures movement through the discrete state space.An adjacent-state transition contributes one unit; a transition spanning multiple intervals contributes proportionally more.Source: §2.7 · pp. 12–13
  • Activity reductions were evaluated within paired high-α and α = 0 Scratch-QAT runs using identical target tensors, initialization, minibatch sequence, and training budget.The commit acceptance rate is the percentage of nonzero proposed changes that result in committed changes.Source: §2.7 · p. 13
  • These activity measures do not represent reduced conventional optimization workload because paired runs use the same minibatches, forward/backward evaluations, and Adam steps.Device write energy, endurance, lifetime, latency, and physical programming pulses were not measured.Source: §2.7 · p. 13

2.8 Evaluation metrics, baselines, and evidence hierarchy

The section evaluates forecasts on the original load scale, uses NRMSE as its principal cross-series metric, and assesses evidence through paired effects and consistency across datasets, series, and evaluation strata.

2.8 Evaluation metrics, baselines, and evidence hierarchy

  • NRMSE is the principal metric because it expresses error relative to mean load and supports comparison among series with different magnitudes.RMSE and MAE are retained as complementary measures; all three are calculated after inverse transformation to the original load scale.Source: §2.8 · pp. 13–14
  • For H1, metrics cover all valid next-hour predictions; for H24, primary metrics pool valid samples across all 24 predicted hours.Separate H24 metrics by forecast hour 1–24 examine lead-time-dependent behavior.Source: §2.8 · p. 14
  • Persistence, daily-seasonality, and weekly-seasonality baselines provide context, but the weekly baseline is not information-matched because it accesses data beyond the 48-h input window.For H24, seasonal references are constructed separately for each target timestamp.Source: §2.8 · p. 14
  • RQAT of 100% denotes recovery to the FP NRMSE level; values below 100% indicate a residual gap relative to FP.NRMSE differences are reported in percentage points.Source: §2.8 · p. 14
  • The Dataset-A development sweep covers 396 fold-specific evaluations across 11 state counts, three series, two horizons, two quantization locations, and three expanding-window folds.The sweep is described as development-stage evidence.Source: §2.8 · p. 14
  • The final frozen evaluation comprises 48 Dataset-A and 16 Dataset-B configurations, totaling 64 across the two co-primary datasets.The selected state set was restricted to 3, 5, 7, and 16 states.Source: §2.8 · p. 15
  • Evidence is assessed by paired-effect magnitude and direction, agreement across datasets and series, support density, consistency across strata, and leave-one-series-out sensitivity.Configuration-level counts are descriptive, not independent statistical replicates.Source: §2.8 · p. 15

3 Results

Across two datasets, direct low-state PTQ incurs a larger NRMSE penalty than quantization-aware training, while scratch training and accumulated-update thresholding offer competitive alternatives. The reported effects persist across the tested forecasting settings and load series.

3.1 Aggressive low-state PTQ produces a reproducible accuracy penalty that QAT-FT largely avoids

  • 6.2471 and 7.0408 percentage-point penalties raised mean NRMSE under PTQ versus FP in Datasets A and B; QAT-FT closed 90.20% and 94.50% of the gaps.These dataset-level means summarize the tested state counts, quantization locations, forecasting horizons, and load series.Source: Table 3 · p. 15, §3.1 · p. 15

3.2 Competitive low-state solutions can also be trained directly from scratch

  • At E40, Scratch α = 0 averaged 11.8001% versus QAT-FT's 11.9424% in Dataset A and 57.6379% versus 58.0938% in Dataset B; its clearest advantage occurred at three states.At three states, Scratch α = 0 had lower mean NRMSE than QAT-FT in 12/16 configuration comparisons overall.Source: §3.2 · p. 17, Figure 2 · p. 18

3.3 Accumulated-update thresholding strongly suppresses committed and discrete target-state activity

  • 99.7271%/99.6741% fewer committed updates and 79.9747%/77.0203% fewer state transitions accompanied +0.0416/+0.1965-percentage-point mean NRMSE changes in Datasets A/B.Committed-update suppression stayed above 99% at every tested state count, while transition suppression decreased as available states increased.Source: Figure 4 · p. 20, §3.3 · pp. 20, 22

3.4 Representative forecast traces under frozen evaluation settings

  • Representative H1/H24 traces use MT_022 and Household_Power; visualization weeks were selected by actual-load variability and FP H1 NRMSE, not quantized-model predictions.For H24, seven non-overlapping direct forecasts initiated at 00:00 each day were concatenated chronologically.Source: §3.4 · p. 22, Figure 6 · p. 23, Figure 7 · p. 24

3.5 Primary effects reproduce across datasets and load series

  • Expected directions for PTQ degradation, QAT-FT improvement, and both activity reductions held across datasets, series, state counts, horizons, and quantization locations, including leave-one-series-out tests.The distinctive three-state Scratch-QAT result also held across all four load series.Source: §3.5 · pp. 22, 25, Table 4 · p. 25, Table 5 · p. 25

4 Discussion

The discussion explains why direct PTQ becomes more disruptive as the weight-state space shrinks, while training with quantization lets models adapt to that constraint. Scratch-QAT offers a viable alternative to pretrained fine-tuning, and accumulated-update thresholding sharply reduces committed activity with little change in forecasting performance.

4.1 Why aggressive PTQ fails most severely at very low state counts

  • As the state count decreases, coarse PTQ increasingly displaces weights from their optimized continuous-space values, while finer grids reduce this mismatch.This explains the progressive approach toward full-precision performance at higher state counts.Source: §4.1 · p. 26

4.2 Why QAT-FT is more robust than direct PTQ

  • QAT-FT adapts shadow weights while quantized weights are used in the forward pass, helping the learned solution accommodate the available states.This adaptation is especially relevant when aggressive quantization makes the mismatch with an unconstrained full-precision solution largest.Source: §4.2 · p. 26

4.3 What Scratch-QAT adds

  • Scratch-QAT produced competitive models from random initialization, with particularly favorable behavior at three states, so full-precision pretraining is not strictly required.Its average NRMSE was 0.1423 and 0.4559 percentage points lower than QAT-FT in Datasets A and B, respectively, but the advantage was not consistent across state counts and configurations.Source: §4.3 · p. 26

4.4 Accumulated-update thresholding changes the relevant notion of training activity

  • Accumulated-update thresholding separates optimizer activity from committed weight changes and discrete transitions, allowing small adjustments to accumulate before commitment.Strong activity suppression with little change in forecasting performance suggests many small adjustments are not individually necessary; device-specific validation is still needed to quantify effects on write activity, latency, or endurance.Source: §4.4 · p. 27

5 Conclusion

The conclusion finds that low-state Transformer forecasting remains viable, though the training pathway matters as weights become more discrete. Accumulated-update thresholding suppresses state-changing activity with small mean error changes, and the frozen methodology's effects remain consistent on the second dataset.

5 Conclusion

  • Direct PTQ substantially degraded accuracy at the lowest state counts, whereas QAT-FT recovered most of the resulting loss across both co-primary datasets.The paper concludes that training pathway becomes critical as the weight representation becomes increasingly discrete.Source: §5 · p. 27
  • Scratch-QAT produced competitive low-state solutions from random initialization without a pretrained full-precision precursor, with particularly favorable behavior at three states.This supports direct low-state learning as a viable pathway in the studied forecasting setting.Source: §5 · p. 27
  • Approximately 99.7% fewer committed shadow-weight updates and 77–80% fewer discrete state transitions came with only small mean changes in forecasting error.The frozen methodology's principal PTQ, QAT-FT, and thresholding effects remained consistent on the second co-primary dataset without dataset-specific retuning.Source: §5 · p. 27
  • The findings point toward nanoscale multilevel nonvolatile devices encoding quantized weights alongside CMOS neuron and peripheral circuitry as a prospective hardware architecture.The passage presents this as a future direction for translating low-state learning into compact computing systems.Source: §5 · pp. 27–28

Related papers

Loading 2610.28784v1…