PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding

Sana Tonekaboni, Danny Eytan, Anna Goldenberg

arXiv:2106.00750v1cs.LGstat.ML

TL;DR

Time-series representation learning is difficult when signals are complex, non-stationary, and sparsely labeled. TNC defines temporal neighborhoods and learns representations that distinguish neighboring from non-neighboring windows, outperforming existing unsupervised approaches across classification and clustering tasks while approaching supervised classification performance.

  • Problem

    Complex non-stationary time series are often sparsely labeled, limiting supervised learning despite the need to model changing latent states.

  • Method

    TNC uses local signal smoothness to define statistically stationary temporal neighborhoods and trains a debiased contrastive framework to distinguish neighboring from non-neighboring windows.

  • Results

    TNC outperforms existing unsupervised representation-learning approaches on classification and clustering across multiple datasets and performs closely to supervised techniques in classification.

  • Takeaways & Limitations

    The learned representations are generalizable and transferable across downstream tasks involving complex non-stationary time series.

Abstract

from arXiv · show

Time series are often complex and rich in information but sparsely labeled and therefore challenging to model. In this paper, we propose a self-supervised framework for learning generalizable representations for non-stationary time series. Our approach, called Temporal Neighborhood Coding (TNC), takes advantage of the local smoothness of a signal's generative process to define neighborhoods in time with stationary properties. Using a debiased contrastive objective, our framework learns time series representations by ensuring that in the encoding space, the distribution of signals from within a neighborhood is distinguishable from the distribution of non-neighboring signals. Our motivation stems from the medical field, where the ability to model the dynamic nature of time series data is especially valuable for identifying, tracking, and predicting the underlying patients' latent states in settings where labeling data is practically impossible. We compare our method to recently developed unsupervised representation learning approaches and demonstrate superior performance on clustering and classification tasks for multiple datasets.

1 INTRODUCTION

The paper addresses difficult, sparsely labeled non-stationary time series by proposing TNC, a self-supervised framework that learns generalizable representations from temporal neighborhoods.

  • Motivation: Unsupervised representation learning extracts informative low-dimensional representations from raw time series without explicit supervision.These representations are described as more generalizable and robust because they are less specialized for one supervised task.
  • Motivation: Medical time series combine heterogeneous, artifact-ridden, missing measurements with gradually changing patient clinical states.These properties make modeling and state identification especially challenging.
  • Motivation: Extended labels for critical-care physiological signals are practically impossible to obtain because clinicians may not know the underlying physiological state.This motivates unsupervised methods for learning representations of latent clinical states.
  • Contribution: TNC is a self-supervised framework for complex multivariate non-stationary time series that aims to capture changing latent temporal dynamics.The framework is presented as efficient, scalable to high dimensions, and usable across time-series settings.
  • Contribution: TNC defines temporal neighborhoods with stationary properties, automatically determining their boundaries from signal characteristics and statistical testing.The neighborhoods represent distributions of similar windows in time.
  • Contribution: TNC adjusts sample weights to address bias from negative examples in contrastive learning.The adjustment incorporates Positive Unlabeled Learning concepts to account for possible positives among sampled negatives.

2 METHOD

TNC learns representations by distinguishing temporally neighboring windows from non-neighboring windows, using automatically estimated neighborhoods and debiasing for potentially positive negative samples.

  • Neighborhood construction: A temporal neighborhood samples window centroids from t*∼N(t, η · δ), using a Gaussian distribution to model gradually changing local signal properties.The range parameter η depends on signal characteristics and the rate of statistical change over time.
  • Neighborhood construction: If η is too small, the encoder mainly learns overlapping information; if η is too large, neighborhoods may span multiple underlying states.The ADF-based adjustment is intended to balance these two failure modes.
  • Debiasing: Windows outside the temporal neighborhood provide contrastive negative samples, but random negative sampling can introduce bias because some negatives may actually be positive.The method draws on Positive Unlabeled Learning to handle this issue.
  • Framework: TNC maps each time-series window into a lower-dimensional representation and trains an objective that distinguishes same-neighborhood from outside-neighborhood samples.The encoder produces representations while the discriminator estimates whether two encoded windows belong to the same temporal neighborhood.
  • Optimization: The discriminator is trained jointly with the encoder and used only during training, while its complexity affects interpretability of latent decision boundaries.The objective seeks probabilities near 1 for neighboring representations and near 0 for far-apart windows, with non-neighbor samples weight-adjusted by w.
  • Neighborhood construction: The Augmented Dickey-Fuller test automatically estimates each window’s neighborhood range by identifying a relatively stationary region.Neighborhood size varies because the underlying state may persist for an unknown duration.

3 EXPERIMENTS

Experiments evaluate TNC across simulated, clinical ECG, and human-activity time series, using shared or task-appropriate encoders and comparisons with established unsupervised approaches.

  • Evaluation design: TNC is compared with Contrastive Predictive Coding and Triplet-Loss for classification and clusterability across multiple time-series datasets.The comparisons use the same encoder network across baselines when assessing the learning frameworks.
  • Datasets: The simulated dataset contains 2000 measurements for 3 features generated from 4 latent states with changing underlying dynamics.A Hidden Markov Model generates latent states, while each state uses a different generative process.
  • Datasets: The MIT-BIH Atrial Fibrillation dataset contains 25 ten-hour ECG recordings with two signals sampled at 250 Hz and four annotated rhythm types.The experiment tests representation learning under changing heart rhythms.
  • Datasets: Atrial flutter and AV junctional rhythm each occur in fewer than 0.1% of measurements, creating severe class imbalance for downstream classification.The dataset also contains relatively few individuals but approximately 5 million data points over extended periods.
  • Modeling choices: The ECG experiment uses a six-layer strided convolutional encoder because the simple RNN cannot model high-frequency ECG measurements.The convolutional network operates directly on waveforms with a total down-sampling factor of 16.
  • Datasets: The HAR dataset contains six activities from 30 individuals, with 561 pre-processed features concatenated into continuous subject-level time series.A single-layer RNN encodes four-sample windows into 10-dimensional representations.

4 RESULTS

TNC learns representations that better separate latent dynamics for clustering and remain informative for downstream classification. The evaluation also examines how encodings evolve along signal trajectories.

  • Clusterability: TNC clusters windows from the same hidden state better than CPC and Triplet Loss on simulated NARMA signals.The comparison uses 10-dimensional encodings of windows of size δ = 50, visualized with t-SNE.
  • Clusterability: Silhouette score and Davies-Bouldin index quantify representation clusterability after K-means in the encoding space.Higher Silhouette scores indicate better cohesion, while lower Davies-Bouldin scores indicate better clusterability.
  • Clusterability: TNC is superior across datasets at distinguishing latent time-series dynamics, while baseline strengths depend on stationarity and temporal ordering.CPC performs poorly on highly non-stationary simulated data but clusters HAR states well; Triplet Loss fails to distinguish simulated states 0 and 2.
  • Classification: TNC representations produce classification performance closer to the end-to-end supervised model than CPC or Triplet Loss.Classification is evaluated using accuracy and AUPRC for hidden-state prediction.
  • Classification: TNC samples neighboring windows from a wider temporal distribution, whereas Triplet Loss can encode overlaps and CPC can weaken under non-stationarity.The authors also attribute baseline performance differences partly to negative-sampling bias when randomly selected negatives resemble the reference window.
  • Trajectory: The trajectory analysis compares the original signal and latent-state regions with the 10-dimensional encodings of sliding windows Wt.Figure 3 uses windows with δ = 50 to inspect how state transitions appear in representation space.

5 RELATED WORK

Unsupervised representation learning for time series remains less studied than in vision and language. Existing time-series methods include reconstruction-based models, predictive coding, and healthcare representations often tailored to labeled downstream tasks.

  • Prior work: Unsupervised representation learning has been less studied for time series than for vision and natural language processing.The related work contrasts the maturity of these neighboring application domains with time-series research.
  • Prior work: Autoencoders and sequence-to-sequence models jointly train encoders with decoders that reconstruct the input signal.Variational autoencoders can additionally impose properties such as disentanglement for interpretability.
  • Prior work: Contrastive Predictive Coding learns representations by predicting the future, while newer approaches avoid reconstructing complex high-frequency signals.The passage motivates alternatives to reconstruction for challenging physiological time series.
  • Healthcare: Many healthcare representation-learning approaches target specific downstream tasks and require expert labeling.This limitation motivates unsupervised representations for rich temporal medical data.

6 CONCLUSION

TNC is presented as a flexible unsupervised framework for learning generalizable representations of complex multivariate non-stationary time series. The paper evaluates it across datasets and downstream tasks including classification and clustering.

  • Conclusion: TNC learns underlying non-stationary time-series dynamics by defining temporal neighborhoods and modeling progression over time.The framework is motivated by clinical state transitions whose labels are difficult to obtain.
  • Conclusion: TNC representations are reported as generalizable and usable for diverse tasks such as classification and clustering.The framework is also described as flexible because it can use arbitrary encoder architectures.
  • Conclusion: The framework is applicable beyond the presented tasks, including anomaly detection in sparsely labeled time-series settings.This is stated as a possible additional downstream use of general representations.

A.1 SIMULATED DATASET

The simulated dataset contains multivariate time series generated under hidden states, with feature-specific signal distributions and added noise. Its samples comprise 500 instances of 2000 measurements.

  • Figure A.1 shows normalized feature-wise time series, with shaded regions marking one of 4 underlying simulated states.
  • Each simulated sample has 3 features generated from hidden states that change over time.An HMM generates the latent states, and each state uses a different signal distribution.
  • Features use distinct generative processes, including Gaussian Processes and Non-Linear Auto-Regressive Moving Average models.The GP features use different kernels, while NARMA models use different parameter sets.
  • 500 instances each contain T = 2000 measurements, and white Gaussian noise with σ = 0.3 is added to all signals.

A.2 BASELINE IMPLEMENTATION DETAILS

The evaluation uses implemented baselines with cross-validated hyperparameters and accounts for computational constraints in distance-based methods. CPC encodes windows and summarizes prior information autoregressively, while T-Loss uses the authors’ released implementation.

  • All baseline implementations are included in the code base, and their hyperparameters are tuned using cross-validation.
  • Contrastive Predictive Coding (CPC): CPC encodes sequential windows and uses a one-directional GRU-based autoregressive model to summarize information into a context representation.
  • Triplet-Loss (T-Loss): T-Loss is implemented using the original code released by its authors on GitHub.
  • KNN and K-means: KNN and K-means use tslearn distance metrics such as DTW, with waveform signals down-sampled because DTW is computationally expensive.

A.3 TNC IMPLEMENTATION EXTRA DETAILS

TNC determines each temporal neighborhood’s range by testing progressively larger windows for stationarity with the Augmented-Dickey Fuller test.

  • TNC gradually increases the neighborhood range from one window size to at most 3 times the window size while repeatedly applying the ADF test.
  • The ADF test’s p-value determines whether the null hypothesis can be rejected when selecting the neighborhood range.

A.4 SELECTING THE WINDOW SIZE

Window size δ must capture enough information about an underlying state without spanning multiple states. On the simulated data, classification performance decreases when windows are too small or too large.

  • Window size δ affects TNC, CPC, and triplet-loss performance and should be long enough to represent a state without spanning multiple states.
  • For ECG signals, the selected window size corresponds to 7 seconds, balancing state stability with sufficient information for identifying the underlying state.
  • Classification performance drops for all baseline methods when the simulated-data window size is too small or too large.

A.5 CLUSTERING METRICS

The evaluation uses Silhouette and Davies-Bouldin scores to assess clustering quality, alongside representation visualizations and downstream classification comparisons. TNC encodings capture temporal or latent-state structure across ECG, HAR, and simulation signals, while weight adjustment improves learned representations for classification.

  • Clustering metrics: Silhouette and Davies-Bouldin scores evaluate clustering quality through within-cluster coherence and between-cluster separation.Davies-Bouldin is generally less computationally complex, while Silhouette can be slightly more accurate in some cases.
  • Weighting strategy: Weight adjustment for non-neighboring samples improves the quality of TNC representations and downstream classification performance.The comparison includes weighted and non-weighted loss settings.
  • Clinical waveform data: ECG encoding trajectories change around an artifact and reflect increased heart rate associated with higher activation in features 0–10.The paper interprets this increased-frequency pattern as an indicator of arrhythmia captured by TNC.
  • HAR data: HAR representations capture underlying signal states as distinct patterns in the latent encoding trajectory.The HAR visualizations use 10-dimensional representations of windows with δ = 4.
  • Simulation data: A 3-dimensional encoding is also visualized for simulation signals containing four underlying states.The simulation trajectory uses sliding windows with δ = 50.
Loading 2106.00750v1…