PaperlayerView exact revision on arXiv

Source-linked AI summary

The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models

Dimitris Spathis, Fahim Kawsar

arXiv:2309.06236v1cs.LGcs.CL

TL;DR

LLMs face difficulty representing numerical and temporal data, including data from sensors. The paper discusses their use in sensing, demonstrates tokenization problems in a mobile-sensing case study, and outlines integration approaches including prompt tuning and model grafting. It argues that input nuances must be handled for LLM outputs to be meaningful.

  • Problem

    Numerical and temporal data can be difficult to translate into text for LLMs, although such data are used in settings such as wearable sensing.

    Source: §1 · p. 1

  • Method

    The paper discusses LLMs in sensing, presents a mobile-sensing tokenization case study, and reviews prompt tuning and model grafting as integration approaches.

    Source: §1 · p. 1, §4 · p. 2, §6 · p. 4

  • Results

    In the WISDM example, the tokenizer splits timestamps and accelerometer values into multiple tokens and groups negative signs with commas, disregarding sensor direction.

    Source: §4 · p. 2

  • Takeaways & Limitations

    The paper argues that meaningful inference requires attention to how temporal inputs are represented, and discusses parameter-efficient transfer learning and model grafting to bridge the modality gap.

    Source: §4 · p. 2, §7 · pp. 4–5

    Model grafting separates the encoder, adapter, and LLM rather than training end-to-end, and its high-dimensional vector communication is no longer meaningful to humans.

    Source: §6 · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.

1 INTRODUCTION

LLMs generalize across tasks, but numerical and temporal inputs from sources such as mobile sensors are difficult to translate into natural-language representations. This paper examines how tokenization can fragment temporal data and discusses approaches to improve its integration with LLMs.

1 INTRODUCTION

  • LLMs are increasingly used as general-purpose assistants, but numerical and temporal data from mobile sensors are difficult to translate into text.Such data can represent fine-grained behavioral and physiological states.Source: §1 · p. 1
  • Repetitive time-series patterns can lead tokenizers to fragment continuous sequences into disjointed tokens.This fragmentation may cause temporal relationships to be lost.Source: §1 · p. 1
  • The paper examines pitfalls in representing and tokenizing temporal data and discusses parameter-efficient transfer learning and multimodal adapters as potential integration solutions.Source: §1 · p. 1

2 TOKENIZATION IN LANGUAGE MODELS

Tokenization divides input and output text into units that LLMs process. The section describes common tokenization approaches and their roles in representing text.

2 TOKENIZATION IN LANGUAGE MODELS

  • Tokenization divides input and output text into smaller units called tokens, which serve as building blocks for language comprehension and generation.Source: §2 · p. 1
  • Tokenization strategies may be rule-based, statistical, or learnable, and shape token meaning and context while helping manage computational and memory demands.Source: §2 · p. 1
  • BPE repeatedly merges frequent character or byte pairs until reaching a specified vocabulary size, helping represent rare or unseen words compactly.The paper illustrates this with “playing,” which is split into “play” and “ing” after frequent pairs are merged.Source: §2 · pp. 1–2
  • WordPiece segments text into words or subwords, while SentencePiece supports sentence-level tokenization for morphologically rich languages.Source: §2 · p. 1

3 WHERE DO TOKENIZERS STRUGGLE?

The section outlines tokenization problems involving inconsistent text handling, numerical representations, arithmetic, and model-specific behavior. These issues can complicate mathematical processing and temporal dependencies.

3 WHERE DO TOKENIZERS STRUGGLE?

  • Tokenization can also distinguish otherwise similar text based on capitalization or trailing whitespace, affecting representation and subsequent-token probabilities.The examples contrast “good” with “Good,” and a phrase with and without a trailing space.Source: §3 · p. 2
  • Inconsistent digit chunking can make adjacent numbers tokenize differently, complicating mathematical processing and temporal dependencies.For example, “480” may be one token while “481” and “482” are split into two.Source: §3 · p. 2
  • Integers may lack coherent decimal tokenization, and arbitrary chunking can require memorizing patterns; tokenization also reflects the temporal distribution of training data.The passage notes that common 20th-century dates receive unique tokens.Source: §3 · p. 2
  • Floating-point values can be split inconsistently according to precision, notation, and rounding, as in “3.14159” becoming four chunks.Source: §3 · p. 2
  • Inconsistent number tokenization is associated with difficulty adding two-digit numbers and zero accuracy on additions involving digits totalling five or more.The passage attributes this deficiency to the absence of a systematic approach to tokenizing individual digits.Source: §3 · p. 2
  • Tokenizers differ across models, creating challenges for interoperability, preprocessing, and multimodal modeling.Source: §3 · p. 2

4 A CASE STUDY ON TOKENIZING SENSING DATA

The WISDM case study encodes 200 rows of mobile sensing data with the public OpenAI tokenizer. Timestamps and sensor values are split into subtokens, while the negative sign is grouped with a comma, potentially obscuring temporal dependencies and sensor direction.

4 A CASE STUDY ON TOKENIZING SENSING DATA

  • The OpenAI tokenizer splits WISDM timestamps and floating-point accelerometer values into multiple tokens, while grouping negative signs with commas and disregarding sensor direction.The example uses the first 200 rows, with each row containing participant ID, activity label, UNIX timestamp, and three accelerometer axes.Source: §4 · p. 2, Figure 1 · p. 3
  • The authors argue that this representation cannot support meaningful inference, and report that prior work used filtering, downsampling, aggregation, rounding, and textual prompts to address the mismatch.Source: §4 · pp. 2–3

5 LARGE LANGUAGE MODELS IN SENSING

This section reviews work that feeds temporal data directly to pretrained LLMs for ubiquitous-computing tasks, against a backdrop of limited sensor-specific pretraining data. It describes health and gesture studies, compares evaluation protocols, and notes limits in reporting tokenization and preprocessing.

5 LARGE LANGUAGE MODELS IN SENSING

  • Unlike works that borrow Transformer/GPT architectures and retrain from scratch [6], this section discusses preliminary works feeding temporal data into pretrained LLMs to improve ubiquitous-computing applications [18].Source: §5 · p. 3
  • Sensor-based human activity recognition commonly uses normalized, fixed-size sliding windows over three accelerometer channels, while self-supervised learning helps address limited labeled data [8] [23] [31].Large public unlabeled sensor datasets and aligned multimodal data are also scarce, constraining the size and generalization of available pretrained models [28] [37].Source: §5 · p. 3
  • Across almost all Fitbit health tasks, adding textual context improved performance over raw-number input, and the authors suggest it helps anchor numerical data [17].The reported tasks include heart-rhythm classification, activity recognition, calorie estimation, and predicting stress and mental-health scores.Source: §5 · p. 3
  • Prompt tuning with trainable prompt embeddings, trained on 3–25 examples per task, dramatically improved performance over zero-shot inference and prompt engineering.Zero-shot performance was poor on most tasks, while prompt engineering improved results but remained limited.Source: §5 · p. 3
  • Reporting on preprocessing and tokenization is limited; the authors assume textual time-series prompts go directly to the base model without further vectorization or embedding [3].They hypothesize that tokenization and representation challenges may help explain the weaker zero-shot and prompt-engineering results.Source: §5 · p. 3
  • A hand-gesture study reported high GPT-4 accuracy, but used a particularly small dataset and zero-shot prompting without baseline accuracy.The missing baseline makes the task difficulty hard to gauge.Source: §5 · p. 4
  • PromptCast found language models competitive with numerical forecasting methods after fine-tuning and stronger under zero-shot transfer, using prompts of 15 timesteps.The authors attribute these results to careful dataset creation and short prompt sequences.Source: §5 · p. 4

6 HOW TO BETTER INTEGRATE TEMPORAL DATA INTO LLMS?

The section considers ways to address the mismatch between temporal data and language-model inputs, including parameter-efficient tuning, model grafting, and tokenizer redesign. These approaches trade changes to the input representation against modularity and human interpretability.

6 HOW TO BETTER INTEGRATE TEMPORAL DATA INTO LLMS?

  • Existing approaches ground LLMs in numerical data with verbose prompts and extensive aggregation, while mapping time series to text remains virtually unexplored [14].The paper argues for mapping temporal data to a more meaningful representation.Source: §6 · p. 4
  • Soft prompting prepends a trainable tensor optimized by backpropagation for a downstream task, encoding information absent from the frozen LLM [12] [17].The paper describes this as more parameter-efficient than full fine-tuning as model size and task count grow.Source: §6 · p. 4
  • Adapters insert small trainable blocks into pretrained-network layers, while prompt tuning operates on the input space to steer time-series processing.The section highlights prompt tuning because it can steer the model through its input space.Source: §6 · p. 4
  • Model grafting maps non-text modalities into text-token embedding space, as in HeLM’s modality-specific encoders [1], and image adapters can map encoder activations into an LLM-understandable form [13] [15] [35].The section also describes shared-space approaches including ImageBind [7] and Meta-Transformer [38].Source: §6 · p. 4
  • Model grafting can reuse high-performing specialist encoders and support faster iteration, but modular systems are not trained end-to-end and their vector interfaces are not human-readable [4].Training adapter layers is described as computationally efficient.Source: §6 · p. 4
  • Digit-level tokenization offers a consistent alternative: LLaMa’s individual-digit tokens were linked to better arithmetic performance than GPT-4 [16].Scientific-literature models also use digit-level splitting, which the section associates with improved understanding of symbolic and numerical data [32].Source: §6 · p. 4

7 DISCUSSION

The discussion argues for improving how pretrained LLMs encode temporal inputs, particularly through parameter-efficient transfer learning, model grafting, and reassessment of tokenizers and embedders. It also identifies longer context windows as an architectural need for real-world temporal datasets.

7 DISCUSSION

  • The paper argues that temporal-data integration into pretrained LLMs is an emerging paradigm with particular relevance to ubiquitous computing, where modality-specific foundation models are lacking.It recommends parameter-efficient transfer learning, model grafting, and reassessing tokenizers and embedders to bridge the modality gap.Source: §7 · p. 4
  • The discussion identifies parameter-efficient transfer learning, model grafting, tokenizer reassessment, and embedder reassessment as approaches for encoding numerical inputs in LLM-understandable form.The authors suggest these approaches could extend beyond temporal data to other non-language tasks [5].Source: §7 · pp. 4–5
  • Longer context windows are a separate architectural need beyond tokenization, aimed at better handling real-world temporal datasets.The authors limit the paper’s scope to temporal data while suggesting the approaches could generalize to other non-language tasks [5].Source: §7 · p. 5

Related papers

Loading 2309.06236v1…