PaperlayerView exact revision on arXiv

Source-linked AI summary

WinoTS: Wavelet-based Self-Distillation for Time Series Models

Noam Major, Kathy Razmadze, Yoli Shavit

arXiv:2609.39337v1cs.LGcs.AI

TL;DR

Time-series self-supervision is dominated by generative objectives, while invariance-based self-distillation remains underexplored for temporal data. WINOTS uses wavelet-based views for self-distillation and outperforms state-of-the-art baselines across forecasting, transfer, and anomaly detection tasks.

  • Problem

    Time-series self-supervised pre-training is dominated by next-token prediction and reconstruction, while invariance-based self-distillation for temporal data remains underexplored.

    Source: §1 · p. 1

  • Method

    WINOTS uses multi-resolution wavelet views that preserve low-frequency trend and seasonal components while asymmetrically perturbing high-frequency details for self-distillation.

    Source: §1 · p. 2

  • Results

    WINOTS outperforms state-of-the-art baselines across forecasting, cross-domain transfer, and multivariate anomaly detection, and frozen representations frequently surpass models trained from scratch under linear probing.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    Wavelet-domain joint-embedding distillation offers a flexible, model-agnostic approach that shifts time-series self-supervision from point-wise reconstruction toward multi-scale structural invariants.

    Source: §1 · p. 2, §7 · p. 9

    On highly volatile streams, specialized autoregressive priors can outperform WINOTS by explicitly optimizing step-by-step local transitions.

    Source: §7 · p. 10

Abstract

from arXiv · show

Self-supervised pre-training of time series models is currently dominated by next-token prediction and reconstruction objectives. In continuous-valued domains, these paradigms often waste model capacity on high-frequency, point-wise noise at the expense of learning invariant structure. While invariance-based self-distillation has proven highly effective in computer vision, its application to temporal data remains largely underexplored. Effectively adapting such methods to time series requires carefully designed augmentations: spatial operations like cropping can shift the timing of repeating cycles or distort the signal, while basic jittering may provide limited variation. We introduce Wavelet-based self-distillation for time series (WinoTS), an invariance-based pre-training paradigm designed specifically for temporal signals. At its core, WinoTS leverages time-frequency augmentations to construct multi-scale structural views without distorting underlying signal dynamics. Across extensive evaluations, WinoTS outperforms state-of-the-art baselines in long-term forecasting, cross-domain zero-shot transfer, and unsupervised anomaly detection. Notably, linear probing on frozen WinoTS representations frequently surpasses fully supervised models trained from scratch. Systematic ablations demonstrate that WinoTS is a flexible, architecture-agnostic framework yielding gains across time series backbones, and establish that time-frequency transformations provide a principled alternative to vision-style spatial augmentations.

1 Introduction

WINOTS addresses the limited use of invariance-based self-distillation in time series with wavelet-generated views that preserve temporal context. Across forecasting, transfer, and anomaly detection tasks, it outperforms state-of-the-art baselines.

1 Introduction

  • Time-series self-supervised pre-training is dominated by next-token prediction and masked autoencoding, while temporal applications of joint-embedding self-distillation remain underexplored.The generative objectives can direct capacity toward high-frequency point-wise noise rather than underlying structure.Source: §1 · p. 1
  • Cropping can truncate recurring patterns or shift phase, whereas basic point-wise jitter may offer little meaningful structural variation.The stated augmentation challenge is to create semantic variation while preserving temporal integrity.Source: §1 · pp. 1–2
  • WINOTS applies invariance-based self-distillation to continuous temporal signals, optimizing for structural invariants rather than point-wise noise prediction.Its multi-resolution wavelet views preserve low-frequency trend and seasonal components while perturbing high-frequency details.Source: §1 · p. 2
  • WINOTS achieves superior performance over state-of-the-art baselines across 13 forecasting benchmarks, 12 cross-domain transfer tasks, and 5 multivariate anomaly detection datasets.Ablations also find that wavelet-based view generation provides an alternative to vision primitives, rivals generative objectives, and gains across diverse backbones.Source: §1 · p. 2

2 Related Work

Prior time-series self-supervision largely falls into generative modeling or latent alignment, with each paradigm carrying different representational and augmentation trade-offs. Existing latent-alignment methods often rely on temporal or spatial transformations that can disrupt signal dynamics.

Generative vs. Latent-Alignment Pre-training for Time Series

  • Generative pre-training predicts, denoises, or reconstructs raw signals, but point-level objectives can prioritize fine-grained values and high-frequency noise over broader invariant structure.Examples include masked autoencoders, autoregressive prediction, diffusion, and target-domain reconstruction.Source: §2 · p. 2
  • Latent-alignment methods enforce consistency between views without signal-space decoders, progressing from contrastive or Siamese learning to non-contrastive self-distillation.Existing approaches are mostly evaluated on narrow benchmark sets or single backbones.Source: §2 · p. 3

Time-Series Augmentations for Latent Alignment

  • Common cropping, jittering, and channel masking can disrupt temporal dynamics, motivating WINOTS to generate multi-scale views in time-frequency space while preserving signal dynamics.Aggressive cropping can remove seasonal and trend context, while point-wise noise can degrade phase and amplitude relationships.Source: §2 · p. 3

3 Method

WINOTS adapts joint-embedding self-distillation to time series using multi-resolution wavelet views that preserve trend and seasonal context while perturbing high-frequency detail. Its asymmetric teacher–student objective aligns representations across these views.

3 Method

  • WINOTS maps sequences into wavelet coefficient space to construct asymmetric, full-length views, then trains a joint-embedding self-distillation model across them.The method section formalizes view generation, the teacher–student objective, and stochastic wavelet-basis sampling.Source: §3 · p. 3, §3.2 · p. 4, Figure 1 · p. 4

3.1 Background

  • DINO uses a gradient-trained student and an exponential-moving-average teacher, whose centered and sharpened target distribution is matched to the student by cross-entropy.The DWT decomposes a sequence into a low-frequency approximation and multi-scale high-frequency details through filtering and downsampling.Source: §3.1 · pp. 3–4

3.2 WINOTS: Self-Distillation with Wavelet-based View Construction

  • Easy views soften high-frequency detail coefficients while leaving low-frequency approximation coefficients unchanged; hard views add controlled Gaussian noise to the details before reconstruction.Soft-thresholding removes or attenuates detail coefficients according to their magnitude.Source: §3.2 · pp. 4–5
  • WINOTS samples wavelet bases across Daubechies, Symlets, and Coiflets to reduce overfitting to basis-specific phase and boundary traits.Within a view, channels share a basis, while hard-view perturbation scales are sampled independently per channel.Source: §3.2 · p. 5
  • The teacher processes only easy views, while the student processes easy and hard views; valid pairs exclude identical easy-view realizations.The objective averages cross-entropy over the valid easy–easy and easy–hard pairs, with stop-gradient applied to teacher outputs.Source: §3.2 · pp. 5–6

4 Experimental Setup

The experiments evaluate WINOTS on forecasting, anomaly detection, and cross-domain transfer using full fine-tuning and linear probing. The setup specifies datasets, baselines, adaptation protocols, and implementation choices.

Datasets and evaluation protocols

  • Forecasting evaluation covers 13 benchmarks with MSE and MAE averaged over horizons 96, 192, 336, and 720; anomaly detection reports point-adjusted precision, recall, and F1.The evaluation follows the TSLib protocol for anomaly detection.Source: §4 · p. 6
  • WINOTS is evaluated with full fine-tuning and linear probing, and compared with 11 forecasting baselines and 7 anomaly-detection baselines.Linear probing freezes the pre-trained backbone and optimizes only a task-specific head.Source: §4 · p. 6
  • For cross-domain zero-shot transfer, pre-training and task configuration use the source dataset, and evaluation on the target dataset proceeds without target-domain parameter updates.Otherwise, pre-training uses the unlabeled training split of the downstream dataset.Source: §4 · p. 7

Implementation Details

  • The implementation pre-trains a TimeMixer backbone with one easy and one hard view, channel-wise standardization, and per-channel hard-view noise scales sampled from U(0.2, 0.5).The reported DINO projection head is a three-layer MLP with a 256-dimensional normalized bottleneck and output dimension K=1024.Source: §4 · p. 7

Architectural and Design Ablations

  • Ablations examine view-generation primitives, loss objectives, encoder backbones, synthetic pre-training, wavelet choices, view pairing, and seed robustness.The supplementary evaluations include backbone transferability and alternative decomposition depths and transform variants.Source: §4 · p. 7

5 Results and Discussion

WINOTS is evaluated across in-domain forecasting, cross-domain zero-shot transfer, and unsupervised anomaly detection, with reported advantages over supervised baselines, across backbones, and over spatial-edit and generative pre-training objectives.

5 Results and Discussion

  • Across forecasting, zero-shot transfer, and anomaly detection, WINOTS outperforms supervised baselines, generalizes across backbones, and offers advantages over spatial edits and generative objectives.Source: §5 · p. 7

5.1 In-Domain Forecasting, Linear Separability, and Backbone Generalization

  • On 13 forecasting benchmarks, WINOTS achieves the top average rank, while WINOTS-LP beats supervised TimeMixer on 7 datasets by MSE and 10 by MAE; fine-tuning gains extend across MLP, Transformer, and TCN backbones.WINOTS also ranks among the top two methods on 12 of 13 datasets.Source: §5.1 · pp. 7–8

5.2 Cross-Domain Zero-Shot Forecasting Transfer

  • In ETT zero-shot transfer, WINOTS ranks first on 9 of 12 source→target pairs by MSE and 10 by MAE, with average relative improvements over TimeMixer of 16.4% in MSE and 8.7% in MAE.Models are evaluated on unseen target datasets without target-domain fine-tuning.Source: §5.2 · p. 8

5.3 Unsupervised Anomaly Detection

  • WINOTS achieves the highest point-adjusted F1 on all five anomaly-detection benchmarks, and its representations transfer to reconstruction-based detection.The reported interpretation is that detail-perturbed student views matched to denoised teacher targets suit modeling regular temporal structure and detecting reconstruction-error deviations.Source: §5.3 · p. 8

6 Ablation Studies

Ablations compare wavelet views with vision-style augmentations and compare the invariance-based objective with hybrid and generative alternatives; the reported results favor wavelet augmentation and pure self-distillation overall.

Ablation setup

  • Ablation models use source-dataset pre-training and frozen linear probing across four forecasting horizons unless otherwise specified.The horizons are 96, 192, 336, and 720.Source: §6 · pp. 8–9

Augmentation Strategy

  • Wavelet views outperform alternatives on four of six datasets; cropping combined with jitter degrades results across most benchmarks, including ETTm1 and Weather.On ETTm1, MSE is 0.438 versus 0.364; on Weather, it is 0.303 versus 0.238. The comparison includes jitter, cropping, and their combinations.Source: §6 · p. 9, Table 5 · p. 9

Pre-training Objective

  • The invariance-based objective beats MAE and NTP on most benchmarks, while adding a localized MAE reconstruction target brings no systematic improvement and sometimes degrades performance.NTP has localized advantages on ETTm1 and Weather, where its autoregressive forecasting bias aligns with the downstream task.Source: §6 · p. 9

7 Conclusion

WINO-TS frames wavelet-domain joint-embedding distillation as a model-agnostic approach that learns multi-scale structural invariants rather than reconstructing point-wise values. Its views preserve full-length temporal context while varying localized detail, but specialized autoregressive priors can do better on highly volatile streams.

7 Conclusion

  • On highly volatile streams, specialized autoregressive priors such as NTP can outperform WINO-TS’s global joint-embedding objective.The authors describe adaptive, learnable wavelet bases and extensions to scale-free, multi-modal temporal systems as future work.Source: §7 · p. 10

A.2 Multiresolution Interpretation

  • Wavelet approximation coefficients encode coarse temporal structure, while detail coefficients capture localized variation at different temporal scales.Wavelet atoms are localized in time and scale, which the paper identifies as suitable for non-stationary signals with changing local frequency content.Source: §A.2 · p. 14
  • The critically sampled DWT is not shift-invariant, so small temporal shifts can alter coefficient alignment across levels; WINO-TS treats this basis sensitivity as an additional source of view diversity rather than attempting to remove it.At default depth J = 3, approximation emphasizes temporal structure at scales longer than roughly 2J+1 samples, while detail bands retain progressively finer local fluctuations.Source: §A.2 · p. 14

A.3 Easy View and Its Relation to Classical Wavelet Shrinkage

  • Easy views preserve approximation coefficients and apply soft-threshold shrinkage to detail bands, retaining attenuated dominant details while suppressing weak transients.The threshold is scaled to each band’s largest absolute coefficient rather than calibrated to an estimated observation-noise model.Source: §A.3 · p. 15

A.4 Hard View and Standardized Perturbation Scale

  • Hard views preserve approximation coefficients and add Gaussian perturbations to detail levels, with perturbation scales expressed in standardized signal units.For multivariate inputs, each channel receives an independent noise realization while sharing the view’s wavelet basis.Source: §A.4 · p. 16

A.5 Boundary Handling, Full-Length Reconstruction, and Admissible Depth

  • All views are reconstructed to the original length T, avoiding changes to a backbone’s positional encoding, patch layout, or input-shape parameters.With independently sampled bases, the views preserve comparable low-frequency bands, though their approximation coefficients need not be identical.Source: §A.5 · p. 16

A.6 Wavelet Pool and Basis Diversity

  • Independent sampling from a wavelet pool varies support, smoothness, vanishing moments, and phase behavior while retaining comparable coarse temporal content.The pool includes Daubechies, Symlet, and Coiflet families with different phase and moment properties.Source: §A.6 · p. 16

B Additional Positioning and Related Work

  • WINO-TS is a pre-training and view-construction method, not a new supervised architecture, and constructs full-length wavelet views by varying localized detail coefficients.It is presented as a model-agnostic approach that shifts learning from point-wise reconstruction toward multi-scale structural invariants.Source: §7 · p. 9, B Additional Positioning and Related… · p. 17

C.1 Pre-Training and Adaptation

The appendix specifies how WINO-TS is pre-trained and adapted for forecasting, anomaly detection, and classification, along with the datasets, optimization settings, and compute configuration. It also points to the main paper’s complete in-domain forecasting comparison rather than duplicating its table.

C.1 Pre-Training and Adaptation

  • For in-domain forecasting, the EMA teacher backbone is adapted by full fine-tuning or linear probing; zero-shot cross-domain evaluation uses source-only adaptation without target updates.Classification instead fine-tunes the pretrained backbone jointly with its classification head.Source: §C.1 · pp. 17–18

C.2 Dataset Naming

  • The main paper’s standard long-term forecasting benchmarks include ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, Exchange, Solar, and Traffic.The appendix adds datasets for broader robustness evaluation and provides short names for several additional datasets.Source: §C.2 · p. 18

D.1 Self-Supervised Pre-Training

  • Pre-training uses AdamW with a base learning rate of 5×10−4, a three-epoch warmup, cosine decay to 1 × 10−6, and 80 FP32 epochs.Runs use seed 42 unless a table reports multiple seeds, batch size 128 per GPU, and inputs of 336 time steps.Source: §D.1 · p. 18, D Optimization and Reproducibility · p. 18

D.2 Forecasting

  • Forecasting uses Adam with a one-cycle schedule; linear probing uses a 2 × 10−5 learning rate and full fine-tuning uses 1 × 10−4.Fine-tuning runs for approximately 20 epochs with early stopping, and the same configuration covers horizons {96, 192, 336, 720}.Source: §D.2 · p. 19

D.3 Anomaly Detection

  • The anomaly detector jointly fine-tunes a WINO-TS encoder and linear reconstruction decoder using normal training windows and mean-squared reconstruction loss.The evaluation threshold uses combined training and unlabeled test reconstruction scores; test labels do not determine it.Source: §D.3 · p. 19

D.4 Classification

  • Classification attaches a linear head to the pretrained backbone and fine-tunes both jointly with Adam at 1 × 10−3 for 20 epochs.Training uses batch size 16 on fixed-length windows.Source: §D.4 · p. 19

D.5 Hardware and Software

  • Each dataset–model run uses one GPU on a server with 8× NVIDIA RTX PRO 6000 Blackwell GPUs.An epoch takes approximately 24 s for ETTh1 and approximately 26 min for Electricity; a complete ETT pre-training run takes approximately 32 min.Source: §D.5 · p. 19

E.1 Complete In-Domain Averages

  • The complete in-domain forecasting comparison is reported in main-paper Table 1, with MSE and MAE averaged over horizons {96, 192, 336, 720}.The table compares WINO-TS under full fine-tuning and linear probing with self-supervised and supervised baselines.Source: §E.1 · p. 20

E.2 Horizon-wise Forecasting Results

Across 52 dataset–horizon rows, WINO-TS frequently ranks among the lowest-error methods, though the strongest method remains dataset- and horizon-dependent.

E.2 Horizon-wise Forecasting Results

  • At least one WINO-TS protocol ranks among the two lowest-error methods in 40 of 52 rows by MSE and 47 by MAE, and is best in 28 and 31 rows, respectively.The breakdown covers horizons 96, 192, 336, and 720 with an input context length of 336.Source: §E.2 · p. 20

F Synthetic Pre-Training

The synthetic pre-training experiments test whether WINO-TS can learn reusable temporal structure without relying exclusively on downstream-domain data. Synthetic pre-training remains competitive on several benchmarks, but its effectiveness is less consistent than in-domain pre-training.

F Synthetic Pre-Training

G Extended Classification Results

Extended classification results vary by dataset, while the ablations identify design choices that shape forecasting performance across backbones, views, objectives, and wavelet settings.

G Extended Classification Results

  • WINO-TS exceeds iTransformer on five of nine classification datasets, while its weaker results on several others are interpreted as consistent with fine-detail sensitivity.The proposed explanation is an interpretation of the observed pattern, not a direct causal measurement.Source: G Extended Classification Results · p. 22, Table A8 · p. 23

H.1 Backbone Ablation

  • With the DINO objective and wavelet pool fixed, TimeMixer performs strongest overall among reported backbones, while PatchTST remains competitive on several benchmarks.The ablation supports WINO-TS as a transferable pre-training recipe while showing that encoder choice remains consequential.Source: §H.1 · pp. 22–23, Table A9 · p. 23

H.2 Multi-View Ablation

  • The default single easy–hard view pair outperforms the tested multi-view variants despite their higher per-window cost.The alternatives use two easy views and two, four, or six hard views.Source: §H.2 · p. 23

H.4 Augmentation-Family Ablation

  • No augmentation variant dominates every dataset: zero-out leads on ETTh1 and ETTm1, Daubechies-only on Weather, and the full pool on ETTh2 and ETTm2.The full pool is retained because it provides phase, support, and smoothness diversity while remaining competitive across datasets.Source: §H.4 · p. 24, Table A10 · p. 24

H.5 Pre-training Objective Ablation

  • The pure DINO objective is the most reliable overall; reconstruction-style auxiliary terms have mixed effects.The comparison includes DINO-only, DINO+MAE, MAE, NTP, and JEPA; DINO+MAE adds masked-patch reconstruction to self-distillation.Source: §H.3 · p. 23, §H.5 · p. 24, Table A11 · p. 24

H.6 Wavelet Pool and Transform Ablation

  • SWT leads on ETTh1 and ETTm1, while decimated DWT leads on ETTm2 and remains close to the best elsewhere; pool differences are small and dataset-dependent.These comparisons support DWT as a simple, competitive default, not as uniformly superior.Source: §H.6 · pp. 24–25, Table A12 · p. 25

H.7 Shrinkage-Strength Ablation

  • Moderate shrinkage is most stable across datasets, and the default ρ = 0.6 provides the best overall trade-off between denoising and detail preservation.Weak shrinkage gives smaller gains, while overly aggressive shrinkage degrades performance on several datasets.Source: §H.7 · p. 25, Table A13 · p. 25

H.8 Wavelet Depth and Wavelet Pool

  • The default depth J = 3 offers the best overall trade-off; independent wavelet-basis sampling is also the most robust overall.Depth 2 leaves views too close in scale, depth 4 makes the approximation overly coarse on several datasets, and independent sampling improves robustness over fixed or shared sampling.Source: §H.8 · pp. 25–26, Table A14 · p. 26, Table A15 · p. 26

H.9 View-Pairing Ablation

The view-pairing ablation favors the default asymmetric teacher–student routing, and the reported linear-probe results show limited variation across five seeds.

H.9 View-Pairing Ablation

  • The default asymmetric pairing is most stable across evaluated datasets; adding identical easy–easy pairs yields no consistent gains, and symmetric routing is less stable on several datasets.The symmetric variant is competitive on ETTm1 and ETTm2 but less stable on ETTh1, ETTh2, and Weather.Source: §H.9 · pp. 26–27, Table A16 · p. 27
  • The default assigns easy-view targets to the teacher while the student learns from both easy and hard views through non-identical cross-view matches.The ablation results indicate that performance depends on these distinct network roles as well as wavelet-view generation.Source: §H.9 · pp. 26–27, Table A16 · p. 27

H.10 Seed Robustness

  • Across five random seeds, the reported WINO-TS-LP standard deviations are small, indicating limited run-to-run variability in this linear-probe setting.The results cover in-domain forecasting MSE and MAE averaged over horizons 96, 192, 336, and 720.Source: §H.10 · p. 27, Table A17 · p. 27
Loading 2609.39337v1…