PaperlayerView exact revision on arXiv

Source-linked AI summary

Voice2Series: Reprogramming Acoustic Models for Time Series Classification

Chao-Han Huck Yang, Yun-Yun Tsai, Pin-Yu Chen

arXiv:2106.09296v3cs.LGcs.AIcs.NEcs.SDeess.AS

TL;DR

Limited data make time series classification difficult, while existing methods often rely on hand-designed features or domain-specific augmentation. Voice2Series reprograms a pre-trained acoustic model using trainable input transformation and output label mapping; on 30 tasks, it achieves competitive or state-of-the-art performance on 19 and is supported by a risk bound involving source risk and Wasserstein feature alignment.

  • Problem

    Data scarcity makes time series classification difficult, and a principled approach with competitive performance is lacking.

  • Method

    V2S reprograms a pre-trained acoustic model through trainable input transformation and source-target output label mapping.

  • Results

    V2S achieves new or equal state-of-the-art accuracy on 19 of 30 UCR time series classification datasets.

  • Takeaways & Limitations

    V2S offers a new approach to time series classification by leveraging pre-trained acoustic-model representations.

  • Takeaways & Limitations

    The evaluation is planned to expand to more acoustic and speech models and to multivariate time series tasks.

Abstract

from arXiv · show

Learning to classify time series with limited data is a practical yet challenging problem. Current methods are primarily based on hand-designed feature extraction rules or domain-specific data augmentation. Motivated by the advances in deep speech processing models and the fact that voice data are univariate temporal signals, in this paper, we propose Voice2Series (V2S), a novel end-to-end approach that reprograms acoustic models for time series classification, through input transformation learning and output label mapping. Leveraging the representation learning power of a large-scale pre-trained speech processing model, on 30 different time series tasks we show that V2S performs competitive results on 19 time series classification tasks. We further provide a theoretical justification of V2S by proving its population risk is upper bounded by the source risk and a Wasserstein distance accounting for feature alignment via reprogramming. Our results offer new and effective means to time series classification.

1. Introduction

V2S addresses data scarcity and the lack of a principled, competitive approach for time series classification by reprogramming pre-trained acoustic models. The framework combines trainable input transformation, an acoustic model, and source-target label mapping, achieving competitive results on 19 of 30 datasets while providing theoretical analysis.

  • Motivation and approach: Data scarcity limits the use of modern large-scale deep learning models for practical time series learning tasks.The paper notes applications across medical, financial, weather, industrial, and IoT domains.
  • Research questions: The paper asks whether acoustic models can be reprogrammed for time series classification, outperform state-of-the-art results, and receive theoretical justification.These questions frame the paper’s empirical and theoretical contributions.
  • Motivation and approach: V2S reprograms large-scale pre-trained acoustic models for time series classification through a unified framework.The framework includes a trainable reprogram layer, a pre-trained acoustic model, and a source-target label mapping function.
  • Empirical evidence: 19 out of 30 UCR datasets show competitive V2S performance against results obtained from different methods.The authors describe this benchmark outcome as evidence that V2S is potentially effective for time series classification.
  • Theoretical contribution: The paper develops a theoretical risk analysis connecting reprogramming performance with source risk and representation alignment.The analysis is intended to characterize performance through source risk and representation alignment loss.

2. Related Works

Related work spans hand-designed and deep learning approaches for time series, model reprogramming across vision and language, and acoustic representation learning. The paper identifies time-series reprogramming as lacking a clear source domain and pre-trained model, motivating acoustic models trained on abundant voice data.

  • 2.1. Time Series Classification: Time series classification uses designed features with conventional classifiers, end-to-end deep models, and mixtures of feature engineering and deep learning.Feature engineering can remain competitive when training data are limited.
  • 2.2. Model Reprogramming: Prior model reprogramming has transferred pre-trained vision models to other image and counting tasks, including label-limited biomedical classification.These approaches can reprogram models without changing their weights, and one cited method uses zeroth-order optimization for black-box systems.
  • 2.2. Model Reprogramming: Model reprogramming has also been applied to natural language tasks and molecule learning in biochemistry.Examples include machine translation, sentiment classification, and molecular learning.
  • 2.2. Model Reprogramming: Before V2S, the source domain and pre-trained models suitable for time-series reprogramming remained unclear.The paper presents V2S as the first framework, to the authors’ knowledge, for reprogramming pre-trained acoustic models on time series.
  • Acoustic Models: Acoustic models learn from large numbers of spectral features and can yield latent features interpretable through auditory experiments or neural saliency methods.These properties support their use as representation learners for V2S.

3. Voice2Series (V2S)

V2S reprograms a pre-trained acoustic classifier for univariate time-series classification by learning a universal input transformation and mapping source labels to target labels.

  • 3.2. V2S Reprogramming on Data Inputs: V2S transforms each target time series into the higher-dimensional source space using zero padding and a masked, trainable additive parameter θ.The transformation preserves the original series in selected positions while learning θ to align source and target distributions.
  • 3.3. V2S Reprogramming on Acoustic Models (AMs): The reprogrammed input is processed by a pre-trained acoustic model, whose source-label predictions are converted into target predictions through a many-to-one mapping h.For each target label, V2S aggregates predictions over the assigned source-label set.
  • 3.3. V2S Reprogramming on Acoustic Models (AMs): V2S learns θ by minimizing the empirical loss over all target-domain training pairs rather than changing the pre-trained acoustic-model parameters.Algorithm 1 uses ADAM with stochastic minibatches to optimize the reprogramming parameters.
  • 3.3. V2S Reprogramming on Acoustic Models (AMs): Many-to-one label mapping can improve reprogramming accuracy over one-to-one mapping, and the implementation assigns each target label an equal number of non-overlapping source labels.The paper illustrates this mapping by aggregating several acoustic classes for each ECG target class.

4. Population Risk via Reprogramming

The paper analyzes V2S through a population-risk bound that combines the source model’s risk with a Wasserstein representation-alignment loss between source and reprogrammed target data.

  • 4. Population Risk via Reprogramming: The risk analysis is presented as applicable beyond V2S to generic classification tasks and is used to assess reprogramming across source models and target tasks.The paper states that Section 5.4 applies the derived bound for performance assessment.
  • 4. Population Risk via Reprogramming: The analysis assumes a specified one-to-one mapping from each target label to a source label and an additive input transformation learned by minimizing the reprogramming loss.The learned transformation is δ∗, the minimizer of the target population risk under the reprogramming objective.
  • 4. Population Risk via Reprogramming: Theorem 1 bounds target population risk by the source population risk and a Wasserstein-1 representation-alignment loss in the source model’s logit space.The alignment term compares source representations zS(xs) with reprogrammed target representations zS(xt + δ∗).
  • 4. Population Risk via Reprogramming: The bound suggests better reprogramming performance when the source model has lower source risk and the source-target representation alignment loss is smaller.The two terms provide separate quantities for assessing source-model quality and reprogramming alignment.
  • 4. Population Risk via Reprogramming: When source and target representations are fully aligned, the Wasserstein term is zero and reprogrammed target performance can match source-task performance.Conversely, a large alignment loss may dominate source risk and hinder target-task performance.

5. Performance Evaluation

V2S reprograms pretrained acoustic models for time-series classification, using input transformation and label mapping. Across 30 UCR datasets, V2Sa is competitive while representation alignment, interpretability analyses, and label mapping clarify its behavior.

  • 5.3. UCR Time Series Classification Benchmark: V2Sa outperforms or ties current results on 19 of 30 UCR datasets, with accuracy increases of 1.84% mean and 2.63% median and about 2.87% lower relative MPCE.The comparison uses test accuracy across the benchmark and reports V2Sa against current results; the authors also note that TFa performs poorly with limited training data.
  • 5.4. Representation Alignment Loss: During training on DistalPhalanxTW, test loss and sliced Wasserstein distance follow similar trends, indicating progressively closer target and source representations.Figure 3 reports averaged test accuracy, loss, and SWD during V2Sa reprogramming.
  • 5.4. Representation Alignment Loss: V2Sa generally outperforms V2Su, consistent with a lower combined source loss and sliced Wasserstein distance.Table 3 compares source-task validation loss and mean/median SWD across the target training sets, and the authors connect the lower sum to improved benchmark performance.
  • 5.5. Additional Analysis on V2S Interpretation: On Worms, the first convolution layer focuses on target segments and low-frequency Mel-spectrogram features, whereas the second emphasizes high-frequency components.The interpretation uses class activation mapping over the reprogrammed spectrogram and distinguishes the functions of the two convolution layers.
  • 5.5. Additional Analysis on V2S Interpretation: After reprogramming, Strawberry logit embeddings show clear separation between target classes, unlike the less separated pre-reprogramming and transfer-learning representations.The tSNE comparison covers V2Sa before and after reprogramming and the fine-tuned TFa baseline.
  • 5.6. Additional Discussion: Many-to-one label mapping acts as ensemble averaging of single-class outputs and performs better than one-to-one mapping in control experiments.The implementation assigns multiple non-overlapping source labels to each target label, and the discussion reports improved control-experiment results.

6. Conclusion

V2S reprograms pre-trained acoustic models for time series classification and combines theoretical analysis with empirical evaluation. On the UCR benchmark, it achieves new or tied state-of-the-art accuracy on 19 of 30 datasets.

  • 19 of 30 UCR datasets achieve new or equal state-of-the-art accuracy with V2S.The conclusion reports this as the overall experimental outcome.
  • V2S is accompanied by a theoretical risk analysis characterizing reprogramming performance.
  • Representation alignment, acoustic saliency maps, and embedding visualization provide in-depth studies of V2S’s behavior.
  • The authors identify combining V2S with advanced data augmentation techniques as a direction for future study.

A. Proof of Lemma 1

The lemma bounds differences in model outputs using Lipschitz properties and the dual representation of Wasserstein-1 distance.

  • The proof’s preliminary steps rely on the neural-network model and the assumption of independent data.
  • The proof selects maximizing output coordinates to bound the maximum coordinate difference by a sum of two nonnegative terms.The argument uses |a| = max{a, −a} and the fact that nonnegative maxima are bounded by their sum.
  • Each output coordinate is 1-Lipschitz, so the difference between two selected coordinates is 2-Lipschitz.This Lipschitz property enables the Wasserstein-1 bound through Kantorovich-Rubinstein duality.

B. Proof of Theorem 1

The theorem proof decomposes target risk, bounds the resulting terms using assumptions and inequalities, and concludes by taking expectation over the source and target distributions.

  • The proof begins by decomposing the target risk function into terms that can be bounded separately.
  • The decomposition uses Assumption 3, the risk definition, adding and subtracting the same source term, and the triangle inequality.
  • Assumption 1 identifies the source-domain expected loss with the source risk before the proof bounds the cross-domain term using Lemma 1.
  • The proof is completed by taking expectation over the source and target distributions on both sides of the preceding inequality.
  • The study compares V2S with ResNet, VGGish, TCN, and OpenL3 acoustic architectures, including both scratch-trained and pretrained variants.ResNet, VGGish, and TCN include scratch-trained configurations, while VGGish and OpenL3 also use pretrained acoustic representations.

C.1. V2S Performance and Sliced Wasserstein Distance

The ablation compares V2S variants built from different acoustic architectures and relates target-task performance to source accuracy and sliced Wasserstein distance.

  • C.1. V2S Performance and Sliced Wasserstein Distance: Table 4 reports ablation results for different pretrained acoustic models, including source-task accuracy, validation loss, and sliced Wasserstein distance.
  • C.1. V2S Performance and Sliced Wasserstein Distance: Both VGGish-based variants show degraded target-task prediction performance, consistent with difficulty adapting wide architectures to the source-domain distribution.
  • C.1. V2S Performance and Sliced Wasserstein Distance: Higher source-task accuracy does not always guarantee higher target-task accuracy for V2S acoustic-model variants.The passage specifically contrasts V2St and V2Su as examples of this mismatch.

D. Additional Ablation Studies

The study selects three efficient V2S variants with capacities of 0.2M, 1M, and 4.7M to examine mean target accuracy across training settings.

  • D. Additional Ablation Studies: Three efficient V2S models—V2Sa, V2Sr, and V2St—span 0.2M, 1M, and 4.7M parameters for comparing training settings.The comparison is intended to provide insight into effective V2S model design.

D.1. Pretrained Models from Different Dataset

The ablations compare pretrained acoustic models from different source datasets and mapping strategies, alongside representation visualizations and computational-cost reporting. GSCv2-pretrained V2S models achieve higher mean accuracy and lower sliced Wasserstein distance than the alternatives described.

  • D.1. Pretrained Models from Different Dataset: GSCv2-pretrained V2S models achieve higher mean prediction accuracy and lower SWD than models pretrained on the other evaluated acoustic datasets.The authors suggest shorter-than-one-second source inputs may explain this result.
  • D.1. Pretrained Models from Different Dataset: Frequency mapping performs equally or slightly worse than random mapping, with a -0.013% accuracy difference and +0.0028 WSD difference over 100 runs.The authors attribute this contrast with image reprogramming results to differences in task dimensions and scales.
  • D.1. Pretrained Models from Different Dataset: Across Figures 6(a)–(e), reprogrammed representations appear more disentangled in both 2D and 3D tSNE visualizations.The figure legends use numbers to denote class-label indices.
  • D.1. Pretrained Models from Different Dataset: The complete experiments on 30 datasets and ablations required approximately 120 computing hours using a 300W power supply.The implementation uses Nvidia 2080-Ti and V100 GPUs with CUDA 10.1; pretrained models are frozen while only a reprogramming layer is trained.
Loading 2106.09296v3…