PaperlayerView exact revision on arXiv

Source-linked AI summary

TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series

Chenxi Sun, Hongyan Li, Yaliang Li, Shenda Hong

arXiv:2308.08241v2cs.CLcs.AI

TL;DR

Pre-trained LLMs have not made significant progress on abstract TS, while TS data are limited and difficult to represent as text. The paper proposes TEST, which tokenizes and contrastively embeds TS into the LLM text-embedding space for frozen-LLM tasks. Across classification, forecasting, and representation experiments, TEST achieves comparable performance to SOTA TS methods and activates additional TS capabilities without compromising language ability.

  • Problem

    Pre-trained LLMs have not made significant progress on abstract TS, while TS datasets are limited and multivariate TS is difficult to represent in text-oriented inputs.

  • Method

    TEST tokenizes TS and uses instance-wise, feature-wise, and text-prototype-aligned contrast to create LLM-compatible embeddings for frozen LLMs.

  • Results

    TEST enables frozen LLMs to achieve comparable performance to SOTA TS methods across classification, forecasting, and representation tasks; GPT2-774M improves accuracy by at least 18% for univariate TS and 25% for multivariate TS.

  • Takeaways & Limitations

    TS-for-LLM with TEST can enrich LLM capabilities for TS while retaining original language ability and exploring LLMs as pattern machines.

  • Takeaways & Limitations

    The relative superiority of TS-for-LLM over customized TS models remains unclear, and model size, type, and pre-training data affect results.

Abstract

from arXiv · show

This work summarizes two ways to accomplish Time-Series (TS) tasks in today's Large Language Model (LLM) context: LLM-for-TS (model-centric) designs and trains a fundamental large model, or fine-tunes a pre-trained LLM for TS data; TS-for-LLM (data-centric) converts TS into a model-friendly representation to enable the pre-trained LLM to handle TS data. Given the lack of data, limited resources, semantic context requirements, and so on, this work focuses on TS-for-LLM, where we aim to activate LLM's ability for TS data by designing a TS embedding method suitable for LLM. The proposed method is named TEST. It first tokenizes TS, builds an encoder to embed TS via instance-wise, feature-wise, and text-prototype-aligned contrast, where the TS embedding space is aligned to LLM embedding layer space, then creates soft prompts to make LLM more open to that embeddings, and finally implements TS tasks using the frozen LLM. We also demonstrate the feasibility of TS-for-LLM through theory and experiments. Experiments are carried out on TS classification, forecasting, and representation tasks using eight frozen LLMs with various structures and sizes. The results show that the pre-trained LLM with TEST strategy can achieve better or comparable performance than today's SOTA TS models and offer benefits for few-shot and generalization. By treating LLM as the pattern machine, TEST can endow LLM's ability to process TS data without compromising language ability. We hope that this study will serve as a foundation for future work to support TS+LLM progress.

1 INTRODUCTION

The introduction frames TS+LLM as either modifying the model or adapting TS data for existing LLMs, and focuses on TEST, an embedding strategy for the latter. TEST targets limitations in TS data, resources, and representation while experiments evaluate its ability to activate frozen LLMs for TS tasks.

  • Two TS+LLM paradigms: TS+LLM is framed through LLM-for-TS, which modifies or trains models, and TS-for-LLM, which creates LLM-friendly TS representations.LLM-for-TS includes training from scratch or fine-tuning pre-trained LLMs; TS-for-LLM largely freezes existing LLMs.
  • Motivation: TS-for-LLM is emphasized because TS datasets are smaller than NLP datasets, domain differences complicate universal model training, and plug-in modules require little or no training.The introduction also cites resource and usage considerations for choosing the data-centric route.
  • Motivation: Directly treating multivariate TS as text is difficult because text inputs are typically univariate and may omit many clinically relevant variables.The sepsis example contrasts one mean arterial pressure sequence with additional vital signs and laboratory values.
  • TEST: TEST tokenizes TS, encodes the tokens, aligns their embedding space with LLM text embeddings using contrastive learning, and inputs the resulting embeddings through soft prompts.The method is designed to make TS embeddings understandable to frozen LLMs rather than directly entering raw TS.
  • Contributions: TEST produces instance-wise, feature-wise, and text-prototype-aligned embeddings, and experiments cover classification, forecasting, few-shot, and representation tasks.The paper positions these experiments as evidence that TEST can activate LLM capabilities for TS tasks while preserving language ability.

2 RELATED WORK

Related work covers model-centric and data-centric approaches to TS+LLM, along with contrastive-learning methods for TS representation. It motivates text-prototype alignment while identifying prototype selection, differentiation, uniformity, and stability as design issues.

  • TS+LLM approaches: Existing TS+LLM work changes either the model through tuning or tool augmentation, or the data through an external encoder.This corresponds to the paper’s LLM-for-TS and TS-for-LLM categorization.
  • LLM-for-TS: Training a TS foundation model requires costly data acquisition and annotation, while supervised LLM fine-tuning may reduce language capabilities and has unclear advantages over specialized TS models.Prompting TS as text is presented as another route, but the supplied passage truncates its description.
  • TS representation learning: Contrastive TS representation methods use pretext tasks such as instance, temporal, and clustering-level contrast to structure embeddings without relying only on downstream models.The cited work reports promising results for these contrastive strategies.
  • Prototype alignment: TEST extends prototype-level contrast by using text embeddings as prototypes, while also addressing prototype selection, differentiation, uniformity, and stability.Prototype-level contrast is described as exploiting latent cluster information beyond the independence assumption.

3 METHODS

TEST embeds time-series tokens with instance-wise, feature-wise, and text-prototype-aligned contrast, then uses task-specific soft prompts to feed them into a frozen LLM. The framework covers tokenization, embedding alignment, prompting, and an approximate equivalence between soft-prompt tuning and supervised fine-tuning.

  • 3.1 TS TOKEN AUGMENTATION AND ENCODING: TEST tokenizes time series and encodes variable-length subsequences into M-dimensional token representations.A sliding-window segmentation produces anchor tokens and augmented positive instances, while non-overlapping instances provide negatives.
  • 3.2 INSTANCE-WISE AND FEATURE-WISE CONTRAST: Instance-wise contrast pulls augmented views of the same token together and separates other minibatch instances.A projection head and similarity function implement the contrastive objective while treating remaining minibatch examples as negatives.
  • 3.2 INSTANCE-WISE AND FEATURE-WISE CONTRAST: Feature-wise contrast treats embedding-matrix columns as soft labels for features, discriminating similar feature groups and differentiating feature columns.The additional feature differences help prevent representation shrinkage and support later prototype alignment.
  • 3.3 TEXT-PROTOTYPE-ALIGNED CONTRAST: Text-prototype-aligned contrast maps TS embeddings toward representative text embeddings, using feature-wise structure to model prototype coordinates.The prototypes need not be TS descriptions because the intended route is TS → pattern → text.
  • 3.4 LEARNABLE PROMPT EMBEDDING: Task-specific soft prompts are learned from LLM outputs and ground truth so the frozen LLM can process TS embeddings as pattern sequences.The prompts address the mismatch between human-semantic prompt templates and TS embeddings without human semantics.
  • 3.4 LEARNABLE PROMPT EMBEDDING: The paper’s transformation shows that soft-prompt tuning is approximately equivalent to supervised fine-tuning for TS-LLM.This result motivates using aligned TS and text embedding spaces while keeping the LLM frozen.

4 EXPERIMENTS

Experiments evaluate TEST across classification, forecasting, generalization, few-shot forecasting, and representation learning with multiple LLMs and TS datasets. TEST substantially improves LLM performance, reaching or exceeding strong TS-model benchmarks at different model scales.

  • Experimental setup: TEST trains a causal TCN encoder and soft prompt, with the encoder output matched to the LLM embedding size.The encoder uses ten convolutional layers with dilated convolutions and skip connections; training is described in Algorithm 1.
  • Experimental setup: Experiments compare TEST with 12 baselines, including LLM question-answering methods, GPT4TS, and classical TS models.The evaluation covers univariate and multivariate classification, short- and long-term forecasting, few-shot forecasting, generalization, and representation learning.
  • Classification: At about 300M parameters, TEST-equipped LLMs achieve classification accuracy comparable to SOTA models; at about 700M, they can exceed SOTA TS transformers.For GPT2-774M, TEST improves accuracy by at least 18% on univariate TS and 25% on multivariate TS.
  • Forecasting: At about 300M parameters, TEST can make forecasting accuracy exceed SOTA TS transformers.Figure 3 separates forecasting results from classification, generalization, few-shot forecasting, and representation results.
  • Generalization and few-shot forecasting: TEST achieves a relative average MSE reduction of 23.5% in few-shot forecasting, while LLM-based models show better generality than baselines on fused datasets.Few-shot forecasting uses 10% of the training time steps.
  • Representation learning: TEST’s first-step embedding is comparable to SOTA representation methods, while the LLM’s second-step embedding can outperform them.The paper reports that the post-LLM representation becomes more discriminative, and matched TS tokens are often associated with sentiment-related words.

5 DISCUSSION AND CONCLUSION

TEST enables frozen LLMs to perform TS classification, forecasting, and representation tasks at performance comparable to SOTA methods. The discussion frames TS-for-LLM as a way to study LLMs as pattern machines, while identifying model size and type and human-level alignment as open considerations.

  • TEST enables LLMs to achieve comparable performance to SOTA methods on TS classification, forecasting, and representation tasks.
  • TS-for-LLM offers LLM capabilities alongside TS processing, although SFT LLM effectiveness and customized-model superiority remain unclear.
  • TEST converts TS into an understandable pattern sequence, supporting investigation of LLMs as pattern machines.
  • TEST aligns objective TS data with subjective text data at the machine level, while human-perception-level alignment remains future work.
  • LLM size and type affect results; bidirectional structures benefit classification, generated structures benefit forecasting, and larger models produce more accurate results.

A.1.1 CL-BASED URL FOR TS

Contrastive learning develops time-series representations from unlabeled data by comparing transformed views and exploiting similarities or shared semantics. Its variants emphasize instance, prototype, or temporal structure to address different representation-learning challenges.

  • Unsupervised representation learning avoids annotating every sample, which is crucial for TS because complex characteristics and limited visual cues create an annotation bottleneck.
  • Contrastive methods learn TS representations through self-discrimination pretext tasks that compare positive and negative augmented samples.
  • Instance-level contrast: Instance-level methods pull augmented views of the same sample together while pushing representations of different samples apart.
  • Prototype-level contrast: Prototype-level methods exploit shared semantics within clusters and address instance-level methods’ treatment of semantically similar samples as negatives.
  • Temporal-level contrast: Temporal-level methods learn scale-invariant representations at individual timestamps while complementing instance-level representation learning.

A.1.2 LLM+TS

LLM+TS research asks whether large models developed for other domains can analyze time-series data. The literature includes representation-learning methods and remains an emerging field with limited prior work.

  • Large models’ success in NLP and CV motivates the question of whether they can effectively analyze TS data.
  • TS supports applications across geoscience, transportation, energy, healthcare, environment, and finance.
  • Existing TS representation-learning methods are organized in Table 3.
  • Table 4 summarizes contrastive-learning-based universal representation methods for TS.
  • Table 5 summarizes existing work about TS+LLM, reflecting that the field is still in its infancy.
  • TS analysis has progressed from statistical models to RNNs, CNNs, and Transformers, while large-model research has advanced more rapidly in other fields.

A.2.1 ENCODER

TEST’s encoder is designed to extract TS information efficiently from variable-length inputs and map it into the LLM’s embedding dimension. It uses a 10-layer causal TCN with dilated convolutions and skip connections.

  • The encoder uses a causal TCN with 10 convolutional blocks, each combining GELU, DilatedConv, BatchNorm, GELU, DilatedConv, and skip connections.
  • Dilated convolutions use dilation 2^i at layer i, while a final convolution maps hidden channels to the LLM embedding size.
  • The intermediary causal-network width is 40 channels, convolution kernel size is 3, and output channels before max pooling total 640.
  • The representation dimension equals the target LLM’s embedding size, such as 1024 for GPT-2.
  • For classification, training uses Adam with α = 0.001, batch size 10, and optimization steps determined by the number of negative samples.

A.2.2 LLM

The experiments use the listed LLMs and train each encoder and soft prompt with Adam on NVIDIA Tesla V100-SXM2 GPUs.

  • The study uses the LLMs listed in Table 6.
  • Each encoder is trained with the Adam optimizer.
  • Each soft prompt is trained with the Adam optimizer on 20 NVIDIA Tesla V100-SXM2 GPUs using CUDA 11.3.

A.3 FORECASTING TASKS

TEST is evaluated across forecasting, classification, and representation tasks using established datasets and baselines. It achieves comparable or better performance across these evaluations, while prototype-based contrastive components improve accuracy.

  • Forecasting: TEST matches or exceeds strong forecasting baselines across long-term, few-shot, and zero-shot settings.It is comparable to TimesNet and DLinear for long-term forecasting, comparable to PatchTST, DLinear, and GPT4TS with 10% training data, and outperforms recent SOTA methods in zero-shot forecasting.
  • Forecasting: Zero-shot forecasting evaluates cross-dataset adaptation without training data from the target dataset.The reported comparisons use datasets including M3, M4, Tourism, and Electricity, with dataset-specific metrics aggregated by dataset.
  • Classification: TEST achieves comparable performance to SOTA models and outperforms most baselines on multivariate time-series classification.The evaluation covers all 30 multivariate datasets in the UEA archive and reports accuracy-related robustness measures including average rank and Top-1, Top-3, and Top-5 accuracy.
  • Representation Learning: TEST outperforms SOTA unsupervised representation methods and approaches the supervised SOTA method on supervised classification evaluation.Representation quality is assessed using learned representations for time-series classification on 128 univariate UCR datasets.
  • Ablation Studies: Both instance-wise and feature-wise contrastive learning strategies increase accuracy.The improvement is reported in the ablation results for Tables 15 and 16.
  • Prototype Analysis: 10 text prototypes are almost optimal, while prototype type has almost no impact on results.Accuracy is basically positively correlated with the number of randomly selected prototypes, and repeated selection of 10 prototypes produces consistent accuracy.
Loading 2308.08241v2…