PaperlayerView exact revision on arXiv

Source-linked AI summary

RHYTHM: Reasoning with Hierarchical Temporal Tokenization for Human Mobility

Haoyu He, Haozheng Luo, Yan Chen, Qi R. Wang

arXiv:2509.23115v3cs.LGcs.AIcs.CL

TL;DR

Human mobility prediction is challenging because trajectories have long-range dependencies and multi-scale periodic patterns. RHYTHM uses temporal tokens, hierarchical attention, prompt-enriched representations, and a frozen LLM. Across three real-world datasets, it reports improved accuracy and reduced computational cost.

  • Problem

    Human mobility prediction must address long-range dependencies and multi-scale periodic behaviors in trajectories.

    Source: §1 · pp. 1–2

  • Method

    RHYTHM uses daily temporal tokens with hierarchical attention and prompt-enriched segment representations processed by a frozen LLM.

    Source: §1 · pp. 1–2

  • Results

    RHYTHM reports a 2.4% improvement in prediction accuracy and a 5.0% increase on weekends across three real-world datasets, plus a 24.6% reduction in computational cost compared to other baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RHYTHM combines multi-scale temporal modeling and semantic trajectory information while using frozen pretrained LLMs as a parameter-efficient strategy.

    Source: §1 · p. 2

    Performance depends heavily on pretrained LLM quality, although those models were designed for language rather than mobility prediction.

    Source: §5 · p. 10

Abstract

from arXiv · show

Predicting human mobility is inherently challenging due to complex long-range dependencies and multi-scale periodic behaviors. To address this, we introduce RHYTHM (Reasoning with Hierarchical Temporal Tokenization for Human Mobility), a unified framework that leverages large language models (LLMs) as general-purpose spatio-temporal predictors and trajectory reasoners. Methodologically, RHYTHM employs temporal tokenization to partition each trajectory into daily segments and encode them as discrete tokens with hierarchical attention that captures both daily and weekly dependencies, thereby quadratically reducing the sequence length while preserving cyclical information. Additionally, we enrich token representations by adding pre-computed prompt embeddings for trajectory segments and prediction targets via a frozen LLM, and feeding these combined embeddings back into the LLM backbone to capture complex interdependencies. Computationally, RHYTHM keeps the pretrained LLM backbone frozen, yielding faster training and lower memory usage. We evaluate our model against state-of-the-art methods using three real-world datasets. Notably, RHYTHM achieves a 2.4% improvement in overall accuracy, a 5.0% increase on weekends, and a 24.6% reduction in training time. Code is publicly available at https://github.com/he-h/rhythm.

1 Introduction

Human mobility prediction must account for long-range dependencies, spatial heterogeneity, and dynamic influences that produce multi-scale patterns. RHYTHM addresses this with daily temporal tokens, hierarchical attention, prompt-enriched representations, and a frozen LLM; it reports improved accuracy and lower computational cost.

1 Introduction

  • Long-range dependencies, spatial heterogeneity, and dynamic influences such as weather anomalies or special events produce non-stationary, multi-scale mobility patterns.The introduction motivates accurate prediction by mobility’s relevance to transportation, epidemic control, and city planning.Source: §1 · p. 1
  • RHYTHM encodes daily mobility patterns as discrete tokens and uses hierarchical attention to capture cyclical, multi-scale dependencies while reducing sequence length.This design targets local routines and global dependencies across days.Source: §1 · p. 2
  • Prompt-guided representations integrate semantic trajectory information and task descriptions with segment embeddings to interpret complex mobility patterns.The framework adds pre-computed semantic embeddings from a frozen LLM to segment tokens before LLM processing.Source: §1 · p. 2, Figure 2 · p. 3
  • Freezing pretrained LLMs reduces trainable parameters to 12.37% of the full model size and computational cost by 24.6% compared to other baselines.The stated computational-cost reduction accompanies the parameter-efficient adaptation strategy.Source: §1 · p. 2
  • RHYTHM reports a 2.4% improvement in prediction accuracy and a 5.0% increase on weekends against state-of-the-art models across three real-world mobility datasets.The introduction describes the weekend figure as an increase, without specifying a separate metric label.Source: §1 · p. 2

2 Related Work

Mobility prediction has progressed from probabilistic methods to deep learning, graph, and transformer approaches, but modeling hierarchical patterns remains difficult. Recent LLM methods often treat mobility as generic sequences, while RHYTHM adapts frozen LLMs for mobility prediction.

Mobility Prediction

  • Sequence models and attention improve temporal modeling, graph methods integrate spatial relationships, and transformers enhance long-range dependency modeling but struggle with mobility’s hierarchical structure.The cited progression runs from probabilistic approaches to deep learning architectures.Source: §2 · p. 3
  • Recent LLM mobility methods show promise but typically treat mobility as generic sequences, ignoring its inherent periodicity.Source: §2 · p. 3

Cross-domain Adaptation of LLMs

  • RHYTHM is presented as the first approach adapting frozen LLMs to mobility prediction without compromising reasoning capabilities or requiring extensive fine-tuning.Prior frozen-LLM adaptations use the models as sequential representation extractors to preserve semantic capabilities while reducing computational costs.Source: §2 · p. 3

3 Method

RHYTHM formulates mobility forecasting from historical timestamp–location trajectories and future timestamps, then models temporal and semantic context with segment-level representations processed by a pretrained LLM. Its design also targets lower computational cost and states theoretical guarantees for the resulting predictor.

3.1 Problem definition

  • RHYTHM predicts future locations from a user’s historical trajectory and the timestamps in a specified prediction horizon.The input trajectory consists of timestamp–location observations.Source: §3.1 · pp. 3–4

3.2 Model structure

  • The model combines categorical location embeddings, projected coordinates, and temporal embeddings, using temporal information alone when spatial input is unavailable.The temporal embeddings encode time-of-day and day-of-week patterns.Source: §3.2 · p. 4
  • Daily-scale segments receive intra-segment attention and are pooled into tokens for inter-segment attention, reducing sequence length from T to N while retaining fine-grained dynamics and long-term dependencies.The segments are non-overlapping, and their tokens capture broader temporal context.Source: §3.2 · p. 4
  • Segment-specific trajectory prompts and future-timestamp task prompts yield precomputed semantic embeddings that are added to segment and prediction embeddings before LLM processing.Precomputation avoids an LLM forward pass during training; the combined embeddings support in-context reasoning over aligned temporal and semantic information.Source: §3.2 · p. 5

3.3 Computational Efficiency

  • Reducing the sequence from T + H to N + H lowers quadratic attention complexity from O((T + H)^2) to O((N + H)^2), while freezing the LLM backbone reduces training time and memory requirements.Semantic embeddings are computed offline, eliminating runtime language-model inference.Source: §3.3 · pp. 5–6

3.4 Theoretical Guarantee

  • The authors state that LLM representation extraction ensures convergent output values [81, Theorem E.2] and uniform last-layer feature distribution [81, Theorem E.3], supporting the final MLP’s learning capability.They also cite a transformer–Hopfield connection [54] and an upper bound on LLM memory-retrieval error [30, Lemma 3.2].Source: §3.4 · p. 6

4 Experiment

RHYTHM is evaluated against mobility-prediction baselines on three real-world datasets. The experiments report benchmark performance, efficiency, temporal trends, ablations, and scaling behavior.

4 Experiment

  • Experiments use Kumamoto, Sapporo, and Hiroshima datasets, 30-minute time slots, a seven-day lookback, and a one-day prediction horizon.The datasets are split by days into 70% training, 20% validation, and 10% test sets.Source: §4 · p. 6, §4.1 · pp. 6–7

4.1 Overall Performance

  • 2.4% improvement in Accuracy@1 and 1.0% improvement in Accuracy@5 over the best baseline; RHYTHM outperforms baselines across most configurations.The comparison covers Kumamoto, Sapporo, and Hiroshima.Source: Table 1 · p. 7, §4.1 · p. 7
  • RHYTHM achieves the best Sapporo DTW and a 1.44% MRR improvement over the best baseline, while COLA leads BLEU across all cities.The geographical comparison covers DTW, BLEU, and MRR.Source: §4.1 · pp. 7–8, Table 2 · p. 8
  • Llama-3.2-3B improves Acc@1 by 0.40% over Llama-3.2-1B, and larger backbones outperform the 1B model in most metrics.The transferability experiments also include Gemma-2-2B.Source: §4.1 · p. 8
  • 24.6% reduction in training time versus the best-performing baseline accompanies 12.37% of full-model trainable parameters; RHYTHM is 80.6% faster than LLM-based methods on average.Training-speed comparisons use the same configuration on one NVIDIA A100 GPU with 40GB of memory.Source: Figure 3 · p. 8, §4.1 · pp. 8–9
  • 5.0% weekend and 3.4% evening-peak improvements contrast with comparable performance during nighttime and standard weekday working hours.The analysis tracks accuracy fluctuations over daily and weekly intervals on Sapporo.Source: §4.1 · p. 9, Figure 5 · p. 9

4.2 Method Analysis

  • Removing temporal tokenization degrades performance by 5.39%, removing hierarchical attention by 0.90%, and removing both prompt types by 1.82%.Task-description removal causes a 0.10% greater performance decrease than trajectory-information removal.Source: §4.2 · p. 9, Table 3 · p. 10
  • Llama-3.2-3B achieves the best mobility-prediction performance, while Llama-3.2-1B offers the reported balance between performance and computational cost.Performance generally improves as LLM parameter count increases; the comparison covers performance, model size, and training and inference speed.Source: Figure 4 · p. 8, §4.2 · pp. 9–10, Table 4 · p. 10

5 Conclusion

RHYTHM combines hierarchical temporal tokenization, semantic embeddings, and frozen pretrained LLMs for mobility prediction. The authors report strong accuracy and scalability, while noting limits tied to pretrained-model quality and training time.

5 Conclusion

  • RHYTHM combines temporal tokenization, hierarchical attention, semantic embeddings, and frozen pretrained LLMs to model mobility patterns at reduced computational cost.The framework is described as interpreting nuanced movement choices, particularly irregular or non-routine patterns.Source: §5 · p. 10
  • The framework outperforms state-of-the-art methods in accuracy and can integrate different pretrained LLMs in a plug-and-play manner.The paper describes this scalability as offering a flexible prediction framework.Source: §5 · p. 10
  • Performance depends heavily on pretrained LLM quality, and resource-constrained models may fail to capture mobility patterns accurately.Training time also remains high, and RHYTHM emphasizes holistic rather than autoregressive prediction.Source: §5 · p. 10

NeurIPS Paper Checklist

The section covers experimental and reproducibility materials, potential bias in mobility predictions, and RHYTHM’s positioning relative to existing mobility models and LLM methods.

NeurIPS Paper Checklist

  • The checklist points to Section 5 for limitations, Section 3.4 for theory guarantees, and Section G.2 for compute resources.It also says the paper and supplement provide additional experimental-setting details.Source: NeurIPS Paper Checklist · pp. 17–20
  • The authors state that main-experiment reproduction details appear in the paper and supplement, and that code, data, and detailed instructions are publicly available.The checklist also says the experimental setting is presented in the main paper with additional details in the supplement.Source: NeurIPS Paper Checklist · pp. 18–19
  • The authors report values in all tables and say they observed stable results with less than 2% variance.Source: NeurIPS Paper Checklist · p. 19
  • The checklist states that the authors followed NeurIPS ethics, credited and licensed used assets, released no new assets, conducted no human-subject research, and used LLMs only for writing and formatting.Source: NeurIPS Paper Checklist · pp. 20–23

A Broader Impact

  • The broader-impact discussion says mobility predictions may encode or amplify training-data biases, potentially producing inequitable outcomes.It says the work has no immediate societal implications but could support future applications in urban planning, public health, disaster response, and transportation.Source: A Broader Impact · p. 24

B Extended Related Work

  • Physics-inspired gravity and radiation models predict aggregate flows but lack individual-level detail, while Markov chains and tensor factorization model user-level transitions but struggle with sparse trajectories and higher-order dependencies.Source: B Extended Related Work · p. 24
  • Transformer mobility methods remain timestamp-centric, with quadratic complexity for multi-day sequences and no explicit modeling of hierarchical periodicities.Earlier sequence and graph-hybrid approaches also have limitations: sequence models often overlook cyclical patterns, while fixed-length sequences limit long-term forecasting scalability.Source: B Extended Related Work · pp. 24–25
  • RHYTHM uses temporal tokenization to model daily and weekly cycles, while retaining a fully frozen LLM backbone for mobility adaptation.This contrasts with LLM mobility approaches that use partial fine-tuning or in-context learning without structured temporal modeling.Source: B Extended Related Work · p. 25
  • Existing time-series foundation models use patch-based or hierarchical attention, or adapt LLMs for forecasting, but struggle with human mobility’s abrupt location shifts and temporal dynamics.RHYTHM instead pairs a spatio-temporal embedding with an autoregressive framework to model these transitions.Source: B Extended Related Work · p. 25

C Attention Implementation Details

The attention implementation uses a pre-norm transformer block with gated feed-forward layers. Prompt examples show how trajectory records and task instructions are presented for next-day mobility prediction.

C Attention Implementation Details

  • The pre-norm transformer block and gated feed-forward network are intended to enhance training stability and model expressivity.Dropout is applied after both attention and feed-forward operations to prevent overfitting.Source: C Attention Implementation Details · p. 26
  • Multi-head attention uses query, key, and value projections, while the gated feed-forward network expands the hidden dimension by a factor of four.The gating uses sigmoid and element-wise multiplication.Source: C Attention Implementation Details · p. 26

D Prompt Design Examples

  • The trajectory prompt example presents timed coordinate records alongside key transitions and main stay locations.The example shows one user's daytime movement records and derived transition and stay summaries.Source: D Prompt Design Examples · p. 26
  • The task prompt specifies a 200 × 200 city grid and requests predictions at 30-minute intervals across the next day.It directs the model to consider regular locations, routines, weekday–weekend differences, and recurring visits.Source: D Prompt Design Examples · pp. 26–27

E Implementation Details

The pipeline precomputes LLM-derived semantics for trajectory segments and prediction targets, then combines them with temporally tokenized inputs before predicting future locations.

E Implementation Details

  • It embeds timestamps and location IDs, partitions the sequence into segments, and applies intra-segment attention and pooling to form one token per segment.The number of segments is N = T/L for segment length L.Source: E Implementation Details · p. 27
  • The pipeline precomputes semantic embeddings by prompting the LLM with trajectory segments and the prediction task, selecting the final output representation.The task prompt covers the future timestamps in the prediction horizon.Source: E Implementation Details · p. 27
  • The pipeline adds semantic embeddings to segment and future-time tokens, runs them through the frozen LLM, and maps future hidden states to predicted location classes.The output uses softmax probabilities and returns the highest-scoring class for each future step.Source: E Implementation Details · p. 27

F Dataset

The paper points to Table 5 for details of the datasets and their statistics.

F Dataset

  • Dataset details and statistics are presented in Table 5.The supplied passage does not provide the table’s individual statistics.Source: F Dataset · p. 28, Table 5 · p. 28

G.1 Evaluation Metrics

The evaluation section describes four metrics for prediction quality and trajectory similarity. It also reports the computational setup, training-selection procedure, and pretrained language-model backbones used.

G.1 Evaluation Metrics

  • Accuracy@k measures the proportion of correct predictions within the top-k ranked locations, while MRR evaluates ranked predictions.The predicted probability distribution and the true location’s rank position are used in these definitions.Source: §G.1 · p. 28
  • DTW measures spatial similarity between trajectories, while BLEU quantifies n-gram overlap between predicted and ground-truth sequences.DTW uses a valid warping path and Euclidean distance; BLEU uses n-gram precision, weights, and a brevity penalty.Source: §G.1 · p. 28

G.2 Computational Resource

  • Experiments used one 40GB NVIDIA A100 GPU and a 24-core Intel Xeon Gold 6338 CPU, with code developed in PyTorch.The Hugging Face Transformers library was used for experimental execution.Source: §G.2 · p. 28

G.3 Hyperparameters

  • Models used a batch size of 64, and dataset-specific settings were selected through validation after searching learning rates and weight decay values.The learning-rate candidates were 1 × 10−4, 3 × 10−4, and 5 × 10−4; weight decay candidates were 0, 0.001, and 0.01.Source: §G.3 · p. 28

G.4 LLM variants

  • RHYTHM was evaluated with multiple pretrained language models as text embedders and frozen backbones, ranging from 125M to 3B parameters.Table 6 lists the pretrained models used.Source: §G.4 · p. 29, Table 6 · p. 29

H Additional Experimental Results

These sections report comparisons of RHYTHM’s prediction strategies and deployment efficiency, alongside references to preprocessing costs across datasets.

H.1 Autoregressive vs Non-autoregressive Strategy

  • RHYTHM’s non-autoregressive strategy delivers comparable accuracy while running over two orders of magnitude faster than its autoregressive strategy.Table 7 reports Acc@1, Acc@3, Acc@5, and computational time per iteration.Source: Table 7 · p. 29, §H.1 · p. 29

H.2 Deployment efficiency compared to LLM-based baselines

  • On GPU, RHYTHM uses substantially less memory and has lower latency than TimeLLM while remaining competitive with Mobility-LLM; on CPU, it has favorable latency and moderate RAM use.Table 8 compares model size, GPU memory, GPU and CPU inference latency, and RAM usage.Source: Table 8 · p. 29, §H.2 · p. 29

I.1 Dataset Preprocessing

I.2 Training Resource Usage

The section examines training costs, temporal segment length, pretrained language models, and architectural contributions to efficiency.

I.2 Training Resource Usage

  • Memory use and training time do not increase linearly with sequence length, and RHYTHM handles longer sequences at moderate additional cost.The supplied results attribute this capacity to temporal tokenization; Table 10 summarizes GPU memory and runtime per epoch.Source: Table 10 · p. 30, §I.2 · p. 30

J.1 Segment Length Sensitivity

  • Daily segmentation (L = 48) gives the best balance: the highest predictive performance with iteration time under 1 second.Smaller segments fragment trajectories and increase computation; very large segments blur daily mobility boundaries and degrade accuracy.Source: Table 11 · p. 30, §J.1 · p. 30

J.2 Impact of Pretrained LLMs

  • On Kumamoto, the pretrained LLM variant achieves the strongest accuracy across all metrics, outperforming randomly initialized and no-LLM variants.The comparison is reported in Table 12.Source: Table 12 · p. 31, §J.2 · p. 31

J.3 Computational Cost of RHYTHM Components

  • Temporal tokenization and hierarchical attention reduce runtime without significantly increasing parameter count.Table 13 compares trainable parameters and training time per epoch across RHYTHM configurations on Kumamoto.Source: Table 13 · p. 31, §J.3 · p. 31

Related papers

Loading 2509.23115v3…