Source-linked AI summary
Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction
Inhwan Bae, Junoh Lee, Hae-Gon Jeon
TL;DR
Pedestrian trajectory prediction commonly relies on coordinate regression, while text tokenizers can struggle with numerical precision and time-series extrapolation. LMTraj turns trajectories and scene descriptions into question-answer prompts for language models, with auxiliary social-reasoning tasks and a numerical tokenizer. The paper reports state-of-the-art results on public benchmarks and accurate predictions from zero-shot and supervised variants.
Problem
Text tokenizers often do not handle numerical data or decimal precision well and do not extrapolate time series from numerical data itself.
Source: §1 · p. 2Method
LMTraj converts trajectory coordinates and captioned scene images into QA prompts, adds auxiliary social-reasoning tasks, and trains a numerical tokenizer for language-model prediction.
Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4Results
Takeaways & Limitations
The findings demonstrate that language models can understand and extrapolate spatio-temporal numeric information from trajectory data.
Source: §5 · p. 8Heatmap and discretized-grid trajectory methods are limited because prediction requires accurate pathways based on social norms.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Language models have demonstrated impressive ability in context understanding and generative performance. Inspired by the recent success of language foundation models, in this paper, we propose LMTraj (Language-based Multimodal Trajectory predictor), which recasts the trajectory prediction task into a sort of question-answering problem. Departing from traditional numerical regression models, which treat the trajectory coordinate sequence as continuous signals, we consider them as discrete signals like text prompts. Specially, we first transform an input space for the trajectory coordinate into the natural language space. Here, the entire time-series trajectories of pedestrians are converted into a text prompt, and scene images are described as text information through image captioning. The transformed numerical and image data are then wrapped into the question-answering template for use in a language model. Next, to guide the language model in understanding and reasoning high-level knowledge, such as scene context and social relationships between pedestrians, we introduce an auxiliary multi-task question and answering. We then train a numerical tokenizer with the prompt data. We encourage the tokenizer to separate the integer and decimal parts well, and leverage it to capture correlations between the consecutive numbers in the language model. Lastly, we train the language model using the numerical tokenizer and all of the question-answer prompts. Here, we propose a beam-search-based most-likely prediction and a temperature-based multimodal prediction to implement both deterministic and stochastic inferences. Applying our LMTraj, we show that the language-based model can be a powerful pedestrian trajectory predictor, and outperforms existing numerical-based predictor methods. Code is publicly available at https://github.com/inhwanbae/LMTrajectory .
1. Introduction
LMTraj reframes pedestrian trajectory forecasting as language-model question answering to address numerical-data challenges in text tokenizers and exploit language models’ contextual reasoning. The paper reports state-of-the-art results on public pedestrian trajectory benchmarks.
1. Introduction
- Mainstream predictors regress coordinate sequences, while modeling pedestrian social relations from distance and motion similarity has improved performance.Language models are presented as offering contextual understanding and social reasoning beyond physics-based interactions.Source: §1 · p. 1
- Text tokenizers often mishandle numerical data, decimal precision, and time-series extrapolation, motivating a language-model approach designed for numerical forecasting.Source: §1 · p. 2
- LMTraj converts trajectories and scene images into QA prompts, adds auxiliary questions about groups and collisions, trains a numerical tokenizer, and supports deterministic and stochastic inference.Beam search generates most-likely trajectories, while temperature tuning supports multimodal predictions.Source: §1 · p. 2
- LMTraj achieves state-of-the-art results on public pedestrian trajectory prediction benchmarks using the proposed components.The paper evaluates language models as numerical regressors with zero-shot and supervised approaches.Source: §1 · p. 2
2. Related Works
Prior trajectory predictors model social interactions and environmental information using numerical methods, while some image- or grid-based alternatives face limits in representing socially acceptable pathways. Language-model work spans general reasoning and forecasting, but earlier language-based trajectory approaches retain constraints in numerical processing and social-interaction modeling.
2.1. Pedestrian Trajectory Prediction
- Numerical trajectory predictors model agent interactions with pooling, attention, graph networks, or transformers, and may incorporate environmental information.Methods extrapolate futures recurrently or simultaneously, with probabilistic approaches including Gaussian distributions, GANs, CVAEs, and diffusion.Source: §2.1 · p. 2
- Heatmap and discretized-grid trajectory approaches are limited because forecasting requires accurate pathways based on social norms.The cited approaches predict image heatmaps [19] [65] or classified positions on a Manhattan grid [23] [55] [78].Source: §2.1 · p. 2
2.2. Language-Based Reasoning and Prediction
- Language foundation models are studied for generation, question answering, vision-language tasks, and mathematical problem solving.The section describes BERT masked prediction, GPT-2 next-token prediction, and T5 sequence-to-sequence generation.Source: §2.2 · pp. 2–3
- Earlier language-based trajectory work uses pretrained text tokenizers that handle numerical data inconsistently, while numerical-regressor auxiliaries can inhibit social-interaction understanding.Related language-prior work also includes time-series forecasting, mobility prediction, and prediction of weather, energy use, and customer flow.Source: §2.2 · p. 3
3. Methodology
LMTraj recasts pedestrian trajectory forecasting as sentence-to-sentence prediction, with numerical and scene information represented in prompts for language models. Its supervised variant adapts tokenization, adds auxiliary social-reasoning tasks, and generates deterministic or multimodal predictions.
3. Methodology
- LMTraj reframes trajectory forecasting as sentence-to-sentence prediction, offering zero-shot and supervised approaches that use numerical inputs and outputs as prompts.The framework applies a language model to numerical forecasting.Source: §3 · p. 3
3.1. Problem Definition
- Given a scene image and agents’ historical coordinates, the task predicts each agent’s future coordinates as either one most-likely path or multiple possible paths.The paper formulates trajectory forecasting as a sequence-to-sequence problem.Source: §3.1 · p. 3
3.2. Data Space Conversion to Prompt
- LMTraj converts coordinate sequences and scene images into text, then uses a QA template to provide agents’ histories as context and ask for a pedestrian’s future path.Coordinates are represented as decimal strings, rounded to two places in world coordinates and left as integers in pixel coordinates; BLIP-2 generates scene descriptions.Source: §3.2 · pp. 3–4, Table 1 · p. 4
3.3. Domain Shift to Sentence Generation
- For LMTraj-SUP, the authors train a tokenizer on QA prompts to separate words, integers, and decimal parts, and add five auxiliary tasks for scene-context and social-dynamics reasoning.The auxiliary tasks cover destination suggestions, moving direction, similar-pattern search, group membership, and collision possibility; the tokenizer reduces tokens per sentence.Source: §3.3 · p. 4
3.4. Forecasting With the Language Model
- LMTraj-SUP uses beam search for a highest-probability path and temperature-controlled generation for diverse outputs; LMTraj-ZERO applies prompt engineering to pretrained language models.The zero-shot model uses GPT-3.5 and GPT-4 without training, while supervised LMTraj uses an encoder-decoder Seq2Seq model.Source: §3.3 · p. 5, §3.4 · p. 5
3.5. Implementation Details
- LMTraj-ZERO inference takes about 20 seconds per trajectory, while supervised training uses BPE and T5 for 200 epochs on eight NVIDIA 4090 GPUs.The supervised setup uses batch size 512, learning rate 1e-4, and takes about four hours.Source: §3.5 · pp. 5–6
4. Experiments
Experiments on four public datasets evaluate tokenizer choices, zero-shot prediction, supervised deterministic and stochastic prediction, component ablations, and computational cost. LMTraj shows strong benchmark performance, while ablations identify tokenizer, multitask, search, and temperature choices and report real-time inference.
4.1. Experimental Setup
- Experiments use ETH, UCY, SDD, and GCS, with 3.2 seconds of observed motion and 4.8 seconds of predicted motion.The evaluation uses K = 1 for most-likely prediction and K = 20 for multimodal prediction, selecting the best path for performance evaluation.Source: §4.1 · p. 6
4.2. Evaluation Results
- BPE and unigram tokenizers distinguish letters from numbers while using fewer tokens; LMTraj-SUP selects BPE for its smaller token sequences.Pretrained and word tokenizers can mix letters and numbers, while character tokenization requires too many tokens.Source: §4.2 · p. 7
- LMTraj-ZERO achieves the best performance among the compared zero-shot methods, and its GPT-4 version performs comparably to Social-STGCNN.The authors report better results with GPT-4 than GPT-3.5.Source: §4.2 · p. 7
- LMTraj-SUP outperforms deterministic predictions on most datasets and performs better than previous works for stochastic prediction.Figure 3 visualizes zero-shot and supervised benchmark predictions, including one best trajectory from K = 20 stochastic samples.Source: §4.2 · p. 7, Figure 3 · p. 7
4.3. Ablation Studies
- The numerical tokenizer outperforms the pretrained tokenizer in deterministic accuracy, while increasing backbone size yields only marginal gains.The authors choose the smallest lightweight model for real-time prediction.Source: §4.3 · p. 8, Table 5 · p. 8
- Multi-task training improves performance over single-task training, with the authors linking this result to domain knowledge about group behavior and collision avoidance.Table 5 presents ablations of LMTraj-SUP components using ADE and FDE in meters.Source: §4.3 · p. 8, Table 5 · p. 8
- Beam search with depth d = 2 and temperature τ = 0.7 produces the best reported performance in the parameter study.Temperature adjustment controls uncertainty and variation in multimodal generation within socially acceptable limits.Source: §4.3 · p. 8, Figure 5 · p. 8
- Sequential next-token prediction makes inference somewhat slower than the fastest model, but LMTraj-SUP uses reasonable GPU memory and supports real-time inference.Table 6 compares computational complexity and measures inference as average time per trajectory.Source: §4.3 · p. 8, Table 6 · p. 8
5. Conclusion
The conclusion frames trajectory prediction as a question-answering task in which historical data provide context for forecasting. LMTraj uses prompt engineering or end-to-end training, alongside specialized language-model techniques, for zero-shot and supervised prediction.
5. Conclusion
- LMTraj reframes trajectory prediction as question answering, using historical trajectories as context for forecasting future paths.The conclusion describes the history as transformed into text prompts that can offer information about human dynamics.Source: §5 · p. 8
- Prompt engineering and end-to-end training support accurate future-path prediction in zero-shot and supervised settings, respectively.The conclusion names these approaches LMTraj-ZERO and LMTraj-SUP.Source: §5 · p. 8
- Tokenizer optimization, multi-task learning, beam search, and temperature tuning are presented as techniques for social reasoning and deterministic or stochastic prediction.Source: §5 · p. 8