PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Remember: Distilling Memory Retention for Compact Recurrent Neural Networks

Nilushika Udayangania, Kishor Nandakishora, Marimuthu Palaniswami

arXiv:2610.06942v1cs.LG

TL;DR

RNNs support time-series analysis but their computational complexity and model size challenge deployment in resource-constrained settings, and prior work had not used teacher memory-retention knowledge for distillation. MemKD transfers memory-retention behavior across subsequences, outperforming baseline KD methods while producing compact students with performance comparable to the teacher.

  • Problem

    RNN complexity and model size challenge resource-constrained deployment, while prior studies had not investigated transferring teacher memory-retention knowledge.

    Source: §1 · p. 1

  • Method

    MemKD uses a loss that trains the student to mimic teacher memory-retention changes across subsequences, including across intermediate representations of different dimensions.

    Source: §1 · p. 2

  • Results

    MemKD significantly outperforms baseline KD methods and yields a compact student with performance comparable to the teacher.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported distillation reduces parameter count and memory usage while maintaining teacher-comparable performance in time-series classification.

    Source: §1 · p. 2

Abstract

from arXiv · show

Deep learning models, particularly recurrent neural networks and their variants, such as long short-term memory, have significantly advanced time series analysis. These models capture complex, sequential patterns in time series, enabling real-time assessments. However, their high computational complexity and large model sizes pose challenges for deployment in resource-constrained environments, such as wearable devices and edge computing platforms. Knowledge Distillation (KD) offers a solution by transferring knowledge from a large, complex model (teacher) to a smaller, more efficient model (student), thereby retaining high performance while reducing computational demands. Current KD methods, originally designed for computer vision tasks, neglect the unique temporal dependencies and memory retention characteristics of time series models. To bridge this gap, we propose a novel KD framework termed Memory-Discrepancy Knowledge Distillation (MemKD). MemKD leverages a specialized loss function to capture memory retention discrepancies between the teacher and student models across subsequences within time series data, ensuring that the student model effectively mimics the teacher's behaviour. This approach facilitates the development of compact, high-performing recurrent neural networks suitable for real-time, time series analysis tasks. We provide additional experiments, in-depth theoretical analysis, and insights into the proposed framework across extended time series benchmarks. Our experiments demonstrate that MemKD significantly outperforms state-of-the-art KD methods. Additionally, we demonstrate that it can match the teacher model's performance across a wide range of compression levels, achieving notable reductions in parameter count and memory usage without a significant loss in accuracy.

1. Introduction

Recurrent networks model time-series patterns but their computational complexity and size challenge resource-constrained deployment. MemKD distills teacher memory-retention behavior across subsequences, and the paper reports stronger performance than baseline KD methods.

  • RNNs, LSTMs, and GRUs model sequential patterns, but their computational complexity and large parameter spaces challenge real-time, resource-constrained deployment.The introduction notes that many hidden states must be updated with nonlinear functions, contributing to inference latency.Source: §1 · p. 1
  • Prior time-series KD methods adapt computer-vision losses, while the importance of RNN memory retention had not been investigated as a source of transferred knowledge.Time-series values are unidirectionally correlated, and short contiguous subsequences often hold discriminative information.Source: §1 · p. 1
  • MemKD makes the student mimic expected changes in teacher memory retention over short contiguous subsequences while learning the target task.The framework uses intermediate memory-maps alongside the classification loss.Source: §1 · p. 2, Figure 1 · p. 2
  • The framework distills a large LSTM-RNN into a compact LSTM-RNN, reducing parameter count and memory usage while maintaining performance comparable to the teacher.The paper reports this result across time-series classification application domains.Source: §1 · p. 2
  • MemKD transfers intermediate feature knowledge across different teacher and student representation dimensions without requiring their sizes to match.Source: §1 · p. 2
  • MemKD significantly outperforms baseline knowledge-distillation methods in the reported experiments.Source: §1 · p. 2

2. Background and Related Works

The background motivates subsequence-level knowledge transfer for time series and reviews recurrent networks, hidden-state semantics, and distillation. It contrasts MemKD’s memory-evolution transfer with direct hidden-state matching.

2.1. Time Series Analysis

  • Subsequence analysis is a longstanding approach for extracting discriminative and interpretable time-series features, including in shapelets, motif discovery, discord discovery, and time-series chains.Source: §2.1 · p. 2

2.2. Recurrent Neural Networks

  • RNNs store past context to model sequential data, while LSTMs and GRUs address vanishing gradients and long-term dependency learning but incur higher computational complexity and slower processing.These costs limit edge deployment and motivate model compression.Source: §2.2 · p. 2

2.3. Semantics from Hidden States

  • Hidden-state interpretations have been explored in NLP, but their semantics remain largely unexplored in time-series analysis and have rarely been used as downstream distillation knowledge.Source: §2.3 · p. 2

2.4. Knowledge Distillation

  • Most KD methods were developed for computer vision, while time-series adaptations have included logit alignment, soft-dynamic-time-warping feature matching, and cross-architecture compression.Unlike direct alignment of final hidden states for forecasting, MemKD transfers knowledge from how memory states evolve and does not require matching teacher and student hidden-state dimensions.Source: §2.4 · p. 3

3. Methodology

The methodology frames time-series distillation as matching teacher and student functions, then uses changes in recurrent hidden states to transfer memory behavior over subsequences. It also recasts this memory-based objective as feature distillation without a dimension-matching regressor.

3.1. Knowledge Distillation for Time Series Classification

  • A time-series classification student is trained to map sequences to labels by leveraging knowledge from a pretrained teacher through a distillation loss.The loss compares teacher and student functions using a similarity measure.Source: §3.1 · p. 3

3.2.1. Formulating Knowledge from Hidden States.

  • Each hidden state summarizes the sequence prefix processed so far, and its change reflects the memory response to a subsequent input subsequence.The change is determined by the subsequence when the preceding hidden state is given.Source: §3.2.1 · pp. 3–4

3.2. Memory-Discrepancy Knowledge Distillation

  • MemKD minimizes the difference between teacher and student measures of how subsequences influence their memory.The objective includes shorter and longer subsequences, with longer ones sampled from the input sequence.Source: §3.2.2 · p. 4

3.2.2. Knowledge Distillation Objective.

  • The objective separates adjacent-step memory changes from changes across longer subsequences, sampling longer subsequences because enumerating all is impractical.The two terms are evaluated separately, and combining them yields the best performance in the reported analysis.Source: §3.2.2 · p. 4

3.2.3. Seamless Feature Distillation.

  • MemKD treats teacher and student hidden-state sequences as temporal feature maps and compares transformed representations without a feature-dimension regressor.This indirect comparison avoids explicit feature-map dimension alignment and its auxiliary parameters.Source: §3.2.3 · p. 4

4. Experiments

The experiments use LSTM teachers and students, evaluate across 56 UCR time-series datasets, and compare MemKD with other distillation methods and a no-distillation baseline. Models are assessed on test-set discrimination and accuracy metrics, with repeated initializations.

4.1. Teacher and Student Models

  • The experiments use a three-layer, hidden-size-100 LSTM teacher and primarily a single-layer, hidden-size-8 LSTM student, with additional student sizes for compression comparisons.The primary student has an approximately 550-fold compression factor relative to the teacher; Table 1 summarizes configurations, sizes, and ratios.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2. Datasets

  • Evaluation covers 56 UCR-2015 datasets grouped by original series length, retaining original train/test splits and standardizing series to length 100 before z-normalization.The collection includes 23 short, 17 medium, and 16 long datasets, including 18 with at least five classes.Source: §4.2 · p. 5, Table 2 · p. 5

4.3. Experiment Setup

  • MemKD is compared with six distillation methods and an identical student trained from scratch, using five student initializations and test AUC-ROC, average AUC-PRC, accuracy, and average rank.Accuracy uses a 0.5 threshold, and AUC-PRC is prioritized in win/tie/loss comparisons because it is more robust to class imbalance.Source: §4.3 · pp. 5–6

5. Results and Discussion

Across 56-dataset comparisons, MemKD outperforms baseline and state-of-the-art distillation methods and remains competitive with the teacher across student compression levels. Ablations examine subsequence components and loss weighting, while hidden-state visualizations show more organized representations for MemKD than for baseline students.

5.1. Comparison with baseline methods

  • MemKD beats Base on all 56 datasets (~10% average-metric gain), vanilla KD on ~90%, and FitNets on 51/56 (5% average-metric gain).Source: Table 3 · p. 6, §5.1 · p. 6

5.2. Comparison with state-of-the-art methods

  • MemKD wins nearly 50% of datasets, ranks 2.29 on average, and improves average AUC-PRC by at least 2% over other methods.It is the only method reported to exceed the teacher in average AUC-PRC and number of wins.Source: Table 4 · p. 6, §5.2 · p. 6

5.3. Comparison with Teacher model

  • Across 9 variants (2.5x–1600x), MemKD is comparable to the teacher and exceeds its average AUC-PRC except at the smallest size; from SL_8, its win rate exceeds 60% and approaches 90% in larger students.In contrast, Base variants lose to the teacher on over 90% of datasets.Source: Table 5 · p. 7, §5.3 · p. 7

5.4. Ablation study

  • MemKD-Long improves average performance metrics by 12% over MemKD-Short in the 28-dataset ablation.The variants are induced by memory retention over shorter and longer subsequences, respectively.Source: §5.4 · p. 7
  • Combining short- and long-subsequence components improves performance on 15 of 28 datasets and yields an average rank of 1.54.Increasing MemKD-Long beyond 100 subsequences showed no significant benefit.Source: Table 6 · p. 7, §5.4 · p. 7, Table 7 · p. 7
  • Across six datasets, AUC-PRC improves for almost all non-zero β values versus β = 0, remains stable at moderate values, and gains diminish at very large values.The authors attribute the diminished gains to over-regularization when the distillation loss dominates task loss.Source: Figure 4 · p. 7, §5.5 · p. 8

5.6. Evidence for Hidden-State Alignment with the Teacher

  • On UWaveGestureLibraryAll, MemKD produces more compact, distinct class-specific hidden-state regions than Base or Base-KD, closer to the teacher’s organization.The t-SNE visualization includes the last five hidden states, suggesting organization across the student’s hidden states rather than only its final representation.Source: Figure 5 · p. 8, §5.6 · p. 8

6. Conclusion

The conclusion presents MemKD as a recurrent-network distillation framework that transfers knowledge through memory-retention discrepancies across subsequences. Across 56 time-series datasets, the authors report stronger performance than state-of-the-art KD methods alongside substantial model-size and memory reductions, and identify longer-subsequence study as future work.

  • MemKD transfers knowledge from large to smaller LSTMs using a loss that captures memory-retention discrepancies across subsequences.Source: §6 · p. 8
  • Across 56 time-series datasets, MemKD outperforms logit- and feature-level state-of-the-art KD methods while maintaining performance comparable to the teacher with reduced parameter size and memory usage.Source: §6 · p. 8
  • Future work will further optimize MemKD and study memory-retention knowledge over longer subsequences to cover the whole time-series instance.Source: §6 · p. 8

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors report using ChatGPT (GPT-4) for language polishing during manuscript preparation. They state that it did not contribute to the research or interpretation, and that the authors take responsibility for the scientific content and conclusions.

Related papers

Loading 2610.06942v1…