PaperlayerView exact revision on arXiv

Source-linked AI summary

Traj-MLLM: Can Multimodal Large Language Models Reform Trajectory Data Mining?

Shuo Liu, Di Yao, Yan Lin, Gao Cong, Jingping Bi

arXiv:2509.00053v1cs.MMcs.AIcs.CL

TL;DR

Trajectory mining lacks a unified approach that generalizes across regions and tasks, and MLLMs raise the possibility of addressing this gap through multimodal reasoning. Traj-MLLM converts trajectories into multiview interleaved image-text sequences and optimizes data-invariant prompts, outperforming state-of-the-art baselines across four tasks without training or fine-tuning.

  • Problem

    Trajectory mining methods are often restricted to specific regions or tasks, while task-independent multimodal representation and flexible task adaptation remain unresolved challenges.

    Source: §1 · p. 1

  • Method

    Traj-MLLM uses map-anchored tokenization, multiview trajectory modeling, and prompt optimization to transform trajectories into interleaved image-text inputs for MLLM reasoning.

    Source: §1 · p. 2, §3 · p. 3, §4.3 · p. 5

  • Results

    48.05%, 15.52%, 51.52%, and 1.83% improvements over state-of-the-art baselines are reported for four representative tasks without training or fine-tuning.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    Traj-MLLM provides region-agnostic and task-adaptive trajectory mining by directly using MLLM reasoning without training data.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Building a general model capable of analyzing human trajectories across different geographic regions and different tasks becomes an emergent yet important problem for various applications. However, existing works suffer from the generalization problem, \ie, they are either restricted to train for specific regions or only suitable for a few tasks. Given the recent advances of multimodal large language models (MLLMs), we raise the question: can MLLMs reform current trajectory data mining and solve the problem? Nevertheless, due to the modality gap of trajectory, how to generate task-independent multimodal trajectory representations and how to adapt flexibly to different tasks remain the foundational challenges. In this paper, we propose \texttt{Traj-MLLM}}, which is the first general framework using MLLMs for trajectory data mining. By integrating multiview contexts, \texttt{Traj-MLLM}} transforms raw trajectories into interleaved image-text sequences while preserving key spatial-temporal characteristics, and directly utilizes the reasoning ability of MLLMs for trajectory analysis. Additionally, a prompt optimization method is proposed to finalize data-invariant prompts for task adaptation. Extensive experiments on four publicly available datasets show that \texttt{Traj-MLLM}} outperforms state-of-the-art baselines by $48.05\%$, $15.52\%$, $51.52\%$, $1.83\%$ on travel time estimation, mobility prediction, anomaly detection and transportation mode identification, respectively. \texttt{Traj-MLLM}} achieves these superior performances without requiring any training data or fine-tuning the MLLM backbones.

1 Introduction

Trajectory mining needs a unified model that generalizes across geographic regions and tasks, while MLLMs offer multimodal reasoning for this challenge. Traj-MLLM addresses it by constructing multiview image-text trajectory representations and optimizing data-invariant prompts without training.

Motivation

  • Existing trajectory models remain tied to specific regions or tasks, leaving cross-region, cross-task generalization unresolved.Trajectory foundation models struggle in unseen regions, while prior LLM-based approaches have limitations in numerical encoding and model generalization.Source: §1 · p. 1
  • MLLMs can combine map-based visual context with textual regional, task, and temporal information to analyze trajectories without extra training.Projected trajectories expose POIs, road networks, and traffic lights visually, while text captures regional information, human knowledge, and task descriptions.Source: §1 · pp. 1–2

Framework

  • Traj-MLLM segments trajectories by map semantics, represents each sub-trajectory across multiple spatial and contextual views, and interleaves images with text.The representation preserves temporal constraints while modeling POIs and road networks at global and local scales.Source: §1 · p. 2
  • Prompt optimization uses a few seed trajectories and multi-round MLLM interactions to produce data-invariant prompts for flexible task adaptation.The prompts encode task descriptions, reasoning steps, and output formats without trainable task heads.Source: §1 · p. 2

Contributions

  • Traj-MLLM is presented as training-free, extensible, and interpretable, producing reasoning processes alongside results.It requires neither model-training data nor MLLM fine-tuning and can incorporate new information views and task prompts.Source: §1 · p. 2
  • Across four representative tasks, Traj-MLLM improves over state-of-the-art baselines by 48.05%, 15.52%, 51.52%, and 1.83% without training.The reported tasks are travel time estimation, mobility prediction, anomaly detection, and transportation mode identification, respectively.Source: §1 · p. 2

2 Related Works

Prior trajectory-mining research progressed from task-specific models to trajectory foundation models and LLM-based approaches, but generalization and multimodal trajectory understanding remain limited. Traj-MLLM extends this direction by applying MLLM reasoning to spatial-temporal trajectory data.

Trajectory Data Mining

  • Task-specific trajectory models embed assumptions for individual tasks, while trajectory foundation models require large-scale pretraining and can underperform in unseen regions.These limitations motivate models that generalize across tasks and geography.Source: §2 · p. 2
  • Prior LLM-based methods use language models as encoders, enhancers, or agents, but they cannot directly model trajectory images and spatial context.Trajectory data contains spatial-temporal and contextual information that language-only models do not directly represent.Source: §2 · p. 3

Multimodal Large Language Models

  • Traj-MLLM is introduced as the first general framework using MLLMs for trajectory data mining, transforming trajectories into multimodal sequences for reasoning.The framework targets the modality gap that prevents direct application of existing MLLM work on static images.Source: §2 · p. 3, Figure 2 · p. 3
  • Existing MLLM research mainly analyzes static images, whereas trajectories require joint handling of spatial, temporal, and contextual information.This difference prevents current static-image methods from being directly applied to trajectory data mining.Source: §2 · p. 3

3 Preliminaries

The paper formulates MLLM-based trajectory mining as mapping a raw trajectory and task into multimodal inputs and a task prompt. Traj-MLLM implements this through map-anchored tokenization, multiview modeling, and prompt optimization.

Problem Definition

  • A raw trajectory is a sequence of GPS coordinates and timestamps, while each mining task specifies a task variable such as travel time estimation or anomaly detection.The formal input trajectory is T = {p_1, ..., p_L}, with each point containing longitude, latitude, and time.Source: §3 · p. 3
  • The multimodal mapping function transforms trajectory T into visual and text modalities and combines them into an interleaved image-text sequence M_T.A separate prompt-construction function generates the task prompt P_τ.Source: §3 · p. 3
  • The analysis result is generated by applying an MLLM to the multimodal trajectory sequence and task prompt: ŷ_τ = MLLM(M_T, P_τ).The mapping and prompt functions are independent, supporting different geographic regions and tasks.Source: §3 · p. 3

Overview of Traj-MLLM

  • Traj-MLLM bridges the trajectory modality gap through map-anchored segmentation, spatial-contextual-temporal views, and optimized task prompts.The first two modules build M, while prompt optimization builds P for flexible task adaptation.Source: §3 · p. 3

4 Methodology

Traj-MLLM converts raw trajectories into semantically segmented, multiview interleaved image-text sequences and adapts them to tasks through optimized prompts. Its methodology combines map-anchored tokenization, spatial-contextual-temporal modeling, and iterative prompt refinement without task-specific training heads.

4.1 Map-Anchored Tokenization

  • Map-anchored tokenization decomposes trajectories into semantically coherent sub-trajectories before generating multimodal representations.Segmentation uses map-based semantic consistency and produces structured text and map-projected visual tokens.Source: §4.1 · p. 3, §4.1.1 · p. 4

4.1.1 Semantic Segmentation

  • Speed consistency, road-type changes, and length regularization jointly define segment costs, while dynamic programming recovers the globally minimal segmentation.The method backtracks the dynamic-programming table to obtain segment boundaries balancing segmentation quality and computational efficiency.Source: §4.1.1 · p. 4

4.1.2 Multimodal Tokenization

  • Each sub-trajectory becomes structured text describing time, distance, and speed alongside a rendered map image with trajectory direction and surrounding context.Visual tokenization renders polylines, clips a padded bounding box, and captures the resulting map view.Source: §4.1.1 · p. 4, §4.1.2 · p. 4

4.2 Multiview Trajectory Modeling

  • Multiview trajectory modeling combines coarse-to-fine spatial views and contextual views such as POIs, road networks, and traffic lights.Global and local spatial strategies partition successive sub-trajectories at different scales before generating multimodal tokens.Source: §4.2 · p. 4, §4.2.1 · p. 5

4.2.2 Contextual Information Modeling

  • Contextual information is organized into separate map layers and filtered by relevance to each spatial partition to avoid visual clutter.POI filtering retains elements whose distance to the sub-trajectory falls below a relevance threshold, while trajectory lines are visually emphasized.Source: §4.2.2 · p. 5

4.2.3 Temporal Information Modeling

  • Ordered multimodal tokens form interleaved image-text sequences that let MLLMs model temporal order and time-dependent relationships without an additional temporal encoder.Each spatial partition contributes image-text pairs from multiple contextual views, with text descriptions also serving as anchors for the images.Source: §4.2.3 · p. 5

4.3 Task Prompt Optimization

  • Task prompt optimization uses fewer than 10 labeled seed trajectories and repeated MLLM feedback to refine prompts for different mining tasks.The loop compares predicted outputs with true labels, focuses refinement on disagreement samples, and stops when seed performance is satisfactory.Source: §4.3 · p. 5, §4.3.1 · p. 6

4.3.1 Task Prompt Template

  • Finalized task prompts can be reused across regions by updating location-specific domain-knowledge elements, while inference parses MLLM responses into structured results.The prompt template fixes role and output-format components while optimizing task description and domain knowledge.Source: §4.3.1 · p. 6

5 Experiment

Across four trajectory-mining tasks and datasets, Traj-MLLM consistently outperforms baselines without training or fine-tuning, while multiview representations support efficiency, reasoning, and task adaptation.

5.2 Effectiveness of Traj-MLLM (RQ1)

  • Traj-MLLM consistently achieves the best performance across four tasks and regions without training or fine-tuning.The evaluation covers travel time estimation, mobility prediction, anomaly detection, and transportation mode identification.Source: §5.2 · p. 6, §5.1 · p. 6
  • Over 29%, 15%, and 21% lower MAE than MM-Path is reported on Xian, Porto, and Chengdu, respectively, for travel time estimation.Traj-MLLM also improves MAPE by over 34.9% over PLM4Traj and Path-LLM.Source: §5.2 · p. 6, Table 1 · p. 7
  • Over 15% ACC@1 improvements over PLM4Traj are observed on Xian and Porto for mobility prediction.The result is attributed to Traj-MLLM's ability to capture complex and fine-grained mobility patterns.Source: §5.2 · pp. 6–7
  • Traj-MLLM achieves the best anomaly-detection performance across datasets, anomaly types, and perturbation levels, with PR-AUC increasing from 0.741 to 0.968 for low-magnitude detours.The method maintains stable performance for subtle detours and complex switch anomalies without training or explicit anomaly examples.Source: §5.2 · p. 7, Table 2 · p. 7
  • 1.83% higher accuracy than RED is achieved for transportation mode identification by combining map cues with velocity and acceleration statistics.The multimodal inputs support both spatial analysis and semantic reasoning across the evaluated tasks.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Performance of different MLLMs (RQ2)

  • o4-mini performs best on both evaluated tasks while producing the shortest responses, so it is selected as the default backbone.Within model families, closed-source models tend to outperform open-source models, but model scale alone does not determine performance.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

5.4 Efficiency of Traj-MLLM (RQ3)

  • o4-mini has the lowest latency and highest throughput, while gemma-3-14b is most economical and large-scale deployment reaches 0.030s per trajectory.The results expose tradeoffs among inference speed, throughput, and cost.Source: §5.4 · p. 8, Table 4 · p. 8

5.5 Ablation Study (RQ4)

  • Removing images causes the largest ablation drop, while disrupting temporal order, semantic segmentation, POI views, or road-network views also degrades performance.These results support the contribution of visual representations, temporal correspondence, semantically coherent segments, and contextual views.Source: §5.5 · p. 8, Table 5 · p. 8

6 Conclusion

Traj-MLLM is presented as a training-free framework that uses multiview multimodal sequences to apply MLLM reasoning to trajectory mining. Across representative tasks and cross-city datasets, it improves performance while maintaining generalization without training or fine-tuning, and releases MLLM responses for future research.

6 Conclusion

  • Traj-MLLM is a training-free framework that bridges raw trajectories and MLLM reasoning through multiview multimodal sequences.The framework is presented as the first work employing MLLM reasoning for trajectory data mining.Source: §6 · p. 9

A.1 Details of Map-anchored Factors

The segmentation cost combines motion consistency, road-environment consistency, and length regularization to produce semantically meaningful trajectory segments.

  • Motion homogeneity measures speed variance within a segment, where lower variance indicates smoother, more coherent movement.Source: §A.1 · p. 11
  • Route homogeneity counts road-ID transitions, with fewer transitions indicating a more consistent road environment.Source: §A.1 · p. 11
  • Segment length regularization uses the inverse number of points to discourage overly short or simple segments.Source: §A.1 · p. 11

A.2 Details of Semantic Descriptions

Each sub-trajectory is converted into a structured textual description containing temporal, distance, and speed features for multimodal trajectory analysis.

  • Semantic descriptions extract key statistical and dynamic characteristics from each raw GPS sub-trajectory and format them into structured text for the MLLM.Source: §A.2 · p. 11

Time-based Features

  • Temporal features comprise start time, end time, and duration, with duration computed as the elapsed seconds between the first and last points.Source: §A.2 · p. 11

Distance-based Features

  • Total distance sums consecutive-point geographic distances computed with the Haversine formula in meters.Source: §A.2 · p. 11

Speed-based Features

  • Speed features include instantaneous speeds, average speed from distance divided by duration, and maximum and minimum non-zero speeds.Source: §A.2 · p. 11

Textual Template

  • The extracted values populate a human-readable template containing timestamps, duration, distance, and speed statistics.Source: §A.2 · p. 11

A.3 Experimental Settings

The experimental settings section introduces the datasets, tasks, baselines, and hyperparameter settings used to evaluate the framework.

  • The experiments specify datasets, task descriptions, baselines, and hyperparameter settings as the evaluation configuration.Source: §A.3 · p. 11

A.3.1 Dataset.

Experiments use four large-scale real-world trajectory datasets to assess cross-city generalization and cross-task universality.

Dataset

  • Four datasets—Xi’an, Chengdu, Porto, and GeoLife—support evaluation of cross-city generalization and cross-task universality.Source: §A.3.1 · p. 11
  • Table 6 presents the dataset statistics for the experimental datasets.Source: Table 6 · p. 11
  • GeoLife contains trajectories from 182 users across diverse behaviors and transport modes, including 17,621 trajectories exceeding 1.2 million kilometers and 50,000 hours.Source: §A.3.1 · p. 11

A.3.2 Tasks Description.

The evaluation spans four trajectory-mining tasks, compares Traj-MLLM with task-specific, foundation, and LLM-based baselines, and examines hyperparameter sensitivity, interactive reasoning, and interpretability. Results show that context scale affects performance, while multi-round and multimodal reasoning support adaptive, structured outputs.

A.3.2 Tasks Description.

  • Four tasks cover travel time estimation, anomaly detection, mobility prediction, and transportation mode identification across regression, binary classification, and multiclass classification settings.The evaluation uses MAE, RMSE, and MAPE for TTE; ACC, Precision, and PR-AUC for AD; ACC@1 and ACC@5 for MP; and Accuracy, Macro-F1, and Weighted-F1 for TMI.Source: §A.3.2 · pp. 11–12

A.3.3 Baselines.

  • The study compares against task-specified models, trajectory foundation models, and LLM-based models, including methods such as Traj2Vec, TrajCL, PLM4Traj, Path-LLM, and FOTraj.The baselines cover the TTE, MP, AD, and TMI settings with task-specific and general representation-learning approaches.Source: §A.3.3 · pp. 12–13

A.4 Hyperparameter Analysis

  • 100m is optimal for Chengdu and Xian in the threshold analysis, whereas overly small or large context-filtering distances degrade reasoning by omitting relevant cues or adding visual noise.The sensitivity analysis evaluates TTE using MAE, with lower values better; Porto uses a 50m threshold because its transportation network is smaller.Source: §A.4 · p. 13

A.5.1 Multi-round reasoning.

  • Multi-round reasoning lets Traj-MLLM update travel-time predictions after real-time events and explain discrepancies after receiving ground-truth feedback.These interactions are presented as adaptive analysis and reflective self-correction rather than one-off prediction.Source: §A.5.1 · pp. 13–14

A.5.2 Interpretable Results.

  • The framework produces structured, human-readable reasoning: it follows nested output instructions, combines statistical and visual evidence, and performs explicit segment-level travel-time calculations.In anomaly detection, o4-mini distinguishes a translated side-street path, while Gamma-3-12b-it misclassifies the trajectory as normal.Source: §A.5.2 · p. 14

Related papers

Loading 2509.00053v1…