PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Numerical Forecasting Models for Language-Based Trajectory Prediction

JunGyu Lee, Inhwan Bae, Hae-Gon Jeon

arXiv:2610.07954v1cs.CV

TL;DR

Language-based trajectory predictors capture behavioral and social context, but token-level objectives provide only indirect guidance for continuous-coordinate accuracy. MoRE uses cached rewards from numerical experts to refine a pretrained language-based predictor with reinforcement learning. It improves stochastic forecasting accuracy across four benchmarks while leaving measured inference memory and latency unchanged.

  • Problem

    Token-level objectives provide only indirect guidance for continuous-coordinate accuracy, despite language-based predictors’ ability to capture behavioral intent and social context.

    Source: §1 · p. 2

  • Method

    MoRE refines a pretrained language-based predictor with coordinate-level rewards from five frozen numerical experts, combining their forecasts through uncertainty-weighted consensus and anchoring them with ground-truth reward.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    MoRE improves stochastic forecasting accuracy across ETH-UCY, SDD, GCS, and NBA; on SDD, ADE decreases from 7.8 to 6.4 pixels, approximately 17.9%, relative to LMTraj-SUP.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Experts are used only during training, and measured inference memory and latency remain unchanged while ETH-UCY spacing patterns move closer to ground truth and collisions decrease.

    Source: §5 · p. 9

    Numerical experts may share errors or provide inaccurate forecasts, and consensus does not guarantee safe or calibrated predictions because disagreement can reflect valid alternative futures.

    Source: §5 · p. 9

Abstract

from arXiv · show

Language-based trajectory predictors represent coordinates as discrete tokens and learn auxiliary tasks such as destination and group reasoning. This formulation enables the model to capture behavioral intent and social context beyond coordinate dynamics alone. However, token-level objectives provide only indirect guidance for continuous coordinate-space dynamics. To address this limitation, we introduce MoRE (Mixture of Reward Experts), a refinement framework that transfers numerical forecasting priors into a pretrained language-based predictor through reinforcement learning. Five frozen numerical predictors provide complementary coordinate-level knowledge of motion and interactions. Their predictions are converted into expert rewards and combined through an uncertainty-weighted consensus that penalizes disagreement. A ground-truth reward anchors the prediction to the target trajectory. To focus refinement on difficult cases, MoRE refines the policy using the top 1% of training samples ranked by predictive entropy. Expert predictions are computed once and cached before PPO training, so the experts are not run during policy updates or inference. In this way, MoRE combines the contextual modeling of the language-based predictor with coordinate-level feedback from numerical experts. On ETH-UCY, MoRE reduces ADE from 0.22 to 0.20 m and FDE from 0.32 to 0.29 m. Relative to the base policy, ADE decreases by 17.9% on SDD and 12.7% on NBA. On ETH-UCY, MoRE also reduces collision rates and better matches ground-truth pedestrian spacing, without increasing measured inference memory or latency. The project page is available at https://jungyu0413.github.io/MoRE/.

1 INTRODUCTION

Numerical predictors model continuous motion but lack language models’ contextual knowledge, while language-based predictors capture behavioral and social context without directly optimizing coordinate accuracy. MoRE transfers numerical forecasting feedback into a pretrained language-based predictor during training, improving accuracy across benchmarks without requiring experts at inference.

1 INTRODUCTION

  • Numerical predictors model motion in continuous coordinates but do not directly leverage language models’ contextual knowledge, and stochastic outputs can be scattered.Their coordinate-based formulation provides strong spatial accuracy for the best sample under best-of-K evaluation.Source: §1 · p. 2
  • Language-based predictors capture behavioral intent and social context, but token-level likelihood does not directly reflect continuous-coordinate errors.LMTraj produces spatially consistent multimodal paths, yet its best prediction can remain spatially inaccurate.Source: §1 · p. 2
  • MoRE converts complementary numerical-predictor forecasts into consensus rewards and uses reinforcement learning to refine a language-based policy on uncertain, difficult samples.Expert predictions are computed and cached in advance.Source: §1 · p. 2
  • MoRE improves prediction accuracy across ETH-UCY, SDD, GCS, and NBA, with larger gains on challenging non-linear motion and cross-dataset transfer.It transfers numerical-expert knowledge into one language-based predictor without increasing measured inference memory or latency.Source: §1 · p. 2

2 RELATED WORK

Prior work develops numerical predictors with varied motion and interaction priors, language-based forecasting for contextual reasoning, and multiple strategies for transferring expert knowledge. MoRE uses numerical predictors as reward-providing experts to refine a language-based predictor.

2.1 NUMERICAL TRAJECTORY PREDICTION

  • Numerical predictors encode interactions, group structure, motion patterns, and destination information through diverse architectures and objectives; MoRE transfers their coordinate-level knowledge rather than introducing another numerical predictor.The cited approaches include spatio-temporal graph models, group-relation models, and destination-guided methods.Source: §2.1 · p. 2

2.2 LANGUAGE-BASED TRAJECTORY PREDICTION

  • Language-based trajectory prediction represents coordinates as tokens and adds auxiliary reasoning tasks, but token-only objectives provide indirect supervision for continuous coordinate accuracy.LMTraj jointly learns prediction with destinations, directions, groups, collisions, and mimicry.Source: §2.2 · p. 3

2.3 EXPERT TRANSFER AND REWARD-BASED REFINEMENT

  • Distillation transfers expert predictions as supervised targets and mixture-of-experts combines subnetworks, whereas reward-based learning optimizes generated outputs from feedback.MoRE uses pretrained numerical predictors as domain experts that provide coordinate-level rewards, not direct targets or inference-time modules; reward overoptimization is a recognized risk.Source: §2.3 · p. 3

3 METHOD

MoRE refines a pretrained language-based forecasting policy with cached numerical-expert rewards, PPO, and uncertainty-driven sample selection. Its training retains supervised learning while adding coordinate-level feedback from expert consensus and the ground truth.

3 METHOD

  • MoRE uses a pretrained language-based forecasting policy and combines multi-expert consensus rewards, PPO refinement, and uncertainty-driven sample selection.The training pipeline caches expert predictions for selected samples, while only the refined policy is needed at inference.Source: §3 · p. 3, Figure 2 · p. 3

3.1 PRELIMINARIES

  • The base policy tokenizes observed trajectories and predicts future trajectories as a discrete sequence, although next-token likelihood does not directly penalize decoded coordinate errors.Pedestrian benchmarks use eight observed and twelve future frames; NBA follows its benchmark-specific protocol.Source: §3.1 · p. 4

3.2 MULTI-EXPERT CONSENSUS REWARD MODELING

  • Five numerical experts provide complementary coordinate-level guidance, and uncertainty-weighted consensus penalizes disagreement among their reward scores.The experts specialize in local interactions, relational dynamics, group behavior, motion patterns, and destination information; a ground-truth reward anchors predictions to the target trajectory.Source: §3.2 · p. 4

3.3 REINFORCEMENT LEARNING VIA PPO

  • PPO jointly optimizes the policy and value network with a clipped policy objective, value regression, and KL penalty, using the composite reward as feedback.PPO and supervised updates are interleaved, with one PPO update every ten supervised steps.Source: §3.3 · p. 5, §3.4 · p. 5

3.4 UNCERTAINTY-DRIVEN SAMPLE MINING

  • MoRE ranks samples by sequence-averaged entropy of the frozen base policy’s Softmax distribution and applies expert rewards to the high-uncertainty subset.The total refinement objective combines supervised learning with PPO on selected hard samples.Source: §3.4 · p. 5

3.5 IMPLEMENTATION DETAILS

  • Implementation uses LoRA rank 16 to update 0.3% of LMTraj parameters and refines the top 1% of training samples ranked by predictive entropy.The five numerical experts are frozen, and refinement takes about one hour on eight NVIDIA RTX PRO 6000 GPUs.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments evaluate MoRE across pedestrian and basketball benchmarks, reporting forecasting accuracy, social-spacing measures, qualitative comparisons, and ablations of its reward and refinement choices.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers ETH-UCY, SDD, GCS, and NBA; pedestrian benchmarks use eight observed and twelve future frames, with ETH-UCY leave-one-out evaluation and ETH-UCY-to-SDD transfer tested without fine-tuning.Accuracy uses minimum ADE and FDE over 20 sampled trajectories; ETH-UCY also reports collision rate and pedestrian-spacing statistics.Source: §4.1 · pp. 6–7

4.2 EVALUATION RESULTS

  • MoRE ties the best average ETH-UCY ADE and FDE and achieves the lowest ADE and FDE on SDD and GCS among compared models.Against LMTraj-SUP, ETH-UCY ADE falls from 0.22 to 0.20 m and FDE from 0.32 to 0.29 m; SDD ADE falls from 7.8 to 6.4 pixels, while GCS FDE falls from 9.6 to 8.5 pixels.Source: Table 1 · p. 6, §4.2 · p. 7
  • On NBA, MoRE has the lowest ADE and FDE; on ETH-UCY, it is closest to ground-truth spacing in all three zones and has the lowest collision rate.Against LMTraj-SUP, NBA ADE falls from 1.10 to 0.96 m and FDE from 1.21 to 1.11 m; ETH-UCY collision rate falls from 2.54% to 2.11%.Source: §4.2 · p. 7, Table 3 · p. 7, Table 2 · p. 7
  • In the illustrated ETH-UCY and SDD cases, MoRE concentrates sampled predictions around paths closer to ground truth and reduces the language-based model’s spatial offset.The comparisons include sampled trajectories and best-of-20 paths on ZARA1/ZARA2 and four SDD scenes.Source: §4.2 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8

4.3 ABLATION STUDIES

  • On ETH-UCY, uncertainty-weighted optimization (UWO) reaches 0.204 / 0.285 m, outperforming maximum and mean aggregation, while removing the ground-truth reward raises error to 0.211 / 0.297 m.Additional SFT on the same hard-sample subset does not improve the base policy, and distillation remains above MoRE.Source: §4.3 · p. 7, Table 4 · p. 8
  • Combining all five experts gives the lowest ADE in every reported behavioral subset, although individual experts’ effects vary by behavior.GP-Graph performs best on group walking and collision avoidance; SingularTrajectory performs best on mimicry and non-linear motion, and subsets may overlap.Source: §4.3 · p. 8, Table 5 · p. 8
  • Refining the top-entropy 1% is the best reported data fraction: LoRA reaches 0.204 / 0.285 m, while that subset has higher base-policy ADE / FDE and more non-linear and collision-involved samples than the overall data.Using 0.1% or 100% raises LoRA ADE to 0.223 m or 0.221 m; the top 1% has 0.40 / 0.60 m base-policy ADE / FDE versus 0.22 / 0.32 m overall.Source: §4.3 · pp. 8–9, Table 6 · p. 9, Table 7 · p. 9
  • MoRE improves both metrics across three challenging non-linear and interaction subsets, reduces their reported average ADE by approximately 24.6%, and lowers ETH-UCY-to-SDD transfer FDE from 17.93 to 15.32 pixels without measured inference overhead.RTX 4090 measurements remain 1,401 MB and 18.3 ms.Source: §4.3 · p. 9, Table 8 · p. 9

5 CONCLUSION

MoRE combines a language-based predictor with cached numerical-expert rewards and improves forecasting across four benchmarks. Its measured inference memory and latency remain unchanged, while the authors identify limitations in expert reliability and evaluation scope.

5 CONCLUSION

  • MoRE improves stochastic forecasting across ETH-UCY, SDD, GCS, and NBA, with larger gains on non-linear motion and cross-dataset transfer.On ETH-UCY, it also yields closer pedestrian-spacing statistics and fewer collisions.Source: §5 · p. 9
  • Numerical experts are used during training, so measured inference memory and latency do not increase.The authors also report that expert composition and reward aggregation both matter.Source: §5 · p. 9
  • MoRE depends on numerical experts that may share errors or provide inaccurate forecasts, and expert disagreement does not guarantee safe or calibrated predictions.The authors identify stronger deterministic experts, adaptive weighting, broader domains, and larger distribution shifts as future-work areas.Source: §5 · p. 9
  • Generative AI tools were used only to improve text readability; the authors drafted and reviewed the final text.The authors state that generative AI was not used for activities requiring disclosure, including methodology design or result interpretation.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work uses existing public trajectory-prediction datasets and notes their potential coverage and bias constraints. It treats trajectories as uncertain estimates and calls for further safeguards before sensitive deployment.

ETHICS STATEMENT

  • The study uses only existing, publicly available trajectory-prediction datasets, which may incompletely cover motion patterns and environments and may carry source-data biases.The proposed method may inherit those biases.Source: ETHICS STATEMENT · p. 10
  • Predicted trajectories are uncertain estimates, not guaranteed outcomes of individual behavior.Source: ETHICS STATEMENT · p. 10
  • Safety-critical or privacy-sensitive deployment requires further validation and appropriate safeguards, especially when predictions could support individual tracking or monitoring.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies where the paper reports its methods, evaluation, supplementary experiments, and qualitative results. Code and evaluation scripts are intended for release subject to internal approval.

REPRODUCIBILITY STATEMENT

  • Section 3 presents the model formulation, learning objectives, sample selection, and implementation settings.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix A gives further RL-formulation, value-network, optimization, hyperparameter, and numerical-handling details; Sections 4.1 and C cover evaluation and additional experiments.Section E provides additional qualitative results.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors intend to release code and evaluation scripts upon acceptance, subject to internal approval.Source: REPRODUCIBILITY STATEMENT · p. 10

A RL FORMULATION AND IMPLEMENTATION

The appendix details MoRE’s token-generation training, reward and optimization choices, and supporting experiments. Its reported ablations favor the five-expert reward, uncertainty-weighted aggregation, and a 1% LoRA refinement subset, while the authors qualify the scope of these comparisons.

A.1 MDP FORMULATION FOR TOKEN GENERATION / A.2 VALUE NETWORK AND SHARED ADVANTAGE

  • Trajectory generation is an episodic MDP with token actions and zero intermediate rewards; decoded-coordinate feedback arrives as a terminal return, and the context critic supplies a shared sequence-level advantage.Expert predictions depend on the observed trajectory and can be cached; decoding and coordinate-error calculations still occur for each rollout.Source: §A.1 · p. 16, §A.2 · p. 16

A.3 TOKEN-LEVEL POLICY OBJECTIVE / A.4 HYPERPARAMETERS AND NUMERICAL HANDLING

  • The actor uses a clipped token-level objective with reference-policy KL regularization; the schedule interleaves PPO and supervised updates, so the comparisons do not isolate a benefit from clipping.The reported settings include one PPO update per ten supervised steps and a 0.2 clipping threshold.Source: §A.3 · p. 16, §A.4 · p. 16

A.5 REGULARIZATION AND SCOPE / A.6 GEOMETRY OF THE MEAN EXPERT REWARD

  • Regularizers are intended to limit changes to the pretrained distribution but do not guarantee preserved auxiliary skills; the evidence does not measure semantic reasoning before and after refinement.The mean expert score orders candidates by distance to the average expert trajectory, while UWO penalizes unequal expert assessments of a candidate.Source: §A.5 · p. 17, §A.6 · p. 17

B.1 NUMERICAL TRAJECTORY PREDICTION / B.2 LANGUAGE-BASED TRAJECTORY PREDICTION / B.3 MULTI-EXPERT LEARNING AND REWARD-BASED OPTIMIZATION / C.1 SOURCES AND REPORTING CONVENTIONS

  • Numerical forecasting work spans interaction, group, motion-pattern, and destination priors; language-based forecasting uses tokenized trajectories and auxiliary reasoning, while MoRE uses numerical predictions as rewards rather than inference-time routing.The appendix also notes reporting-precision conventions, including that approximate percentage reductions use displayed benchmark values.Source: §B.1 · pp. 17–18, §B.2 · p. 18, §B.3 · p. 18

C.2 NUMERICAL EXPERT SELECTION / C.5 REWARD-COEFFICIENT SENSITIVITY / C.6 COMPLETE REFINEMENT-EFFICIENCY STUDY

  • Removing the ground-truth reward yields 0.211 / 0.297, versus 0.204 / 0.285 for the full reward; LoRA on 1% gives the lowest reported ADE and FDE, and tested UWO weights improve over mean aggregation.Sensitivity claims are limited to the tested coefficient range; the detailed tuning comparison is on ETH-UCY.Source: §C.2 · p. 19, Table 13 · p. 21, §C.5 · p. 21

C.2 NUMERICAL EXPERT SELECTION

  • The combined expert reward gives the lowest ADE and FDE in all four ETH-UCY behavioral subsets and the lowest reported aggregate error.The aggregate is evaluated separately rather than computed as the mean of the four displayed subset columns.Source: Table 10 · p. 20, §C.2 · p. 20

C.3 COLLISION-AVOIDING PERFORMANCE

  • UWO has the lowest reported ETH-UCY collision rate among the compared aggregation strategies, but reward-based training alone does not ensure improvement.Distance-zone measurements assess agreement with observed spacing, not complete social compliance.Source: Table 11 · p. 20, §C.3 · p. 20

C.4 PER-SCENE EXPERT INTEGRATION

  • UWO is the only evaluated integration configuration that improves both ADE and FDE over the base policy in every displayed ETH-UCY scene.Other approaches have mixed scene-level effects.Source: Table 12 · p. 21, §C.4 · p. 21

C.7 PROPERTIES OF MINED SAMPLES

The top-entropy training subset is harder than the full set on several measures, but the reported comparisons characterize data difficulty rather than establish entropy selection as optimal.

C.7 PROPERTIES OF MINED SAMPLES

  • Table 16 reports base-policy ADE / FDE for entropy-partitioned ETH-UCY training samples and constant-velocity error in pixel coordinates.Behavioral labels follow Bae et al. (2024a).Source: Table 16 · p. 22
  • The top-entropy subset has larger base-policy errors, a higher non-linear fraction, and more collision-involved interactions than the full training set.Constant-velocity extrapolation gives the same ordering.Source: §C.7 · p. 22
  • Entropy ranks samples relatively: order-preserving transformations leave selection unchanged, while uncertainty errors that change the ordering can affect it.The random and bottom-entropy subsets characterize data difficulty; they are not post-training comparisons of MoRE fitted to those subsets.Source: §C.7 · p. 22

C.8 CROSS-DATASET TRANSFER

The supplementary evaluations report a no-fine-tuning transfer result, small deterministic differences, gains on selected non-linear and NBA results, and inference measurements with explicit scope boundaries.

C.8 CROSS-DATASET TRANSFER

  • 14.6%: ETH-UCY-to-SDD transfer without fine-tuning reduces FDE from 17.93 to 15.32 pixels.This result supports transfer in this particular source-to-target setting, not robustness to all domain shifts.Source: §C.8 · pp. 22–23, Table 17 · p. 22

C.9 DETERMINISTIC EVALUATION

  • 0.001 m ADE and 0.003 m FDE: MoRE’s deterministic differences from LMTraj-SUP are small, while the stochastic improvement is the main empirical result.Adding deterministic experts does not improve the reported deterministic values.Source: §C.9 · p. 23, Table 18 · p. 23

C.10 NON-LINEAR BEHAVIORAL SUBSETS

  • 24.6%: average ADE falls from 0.248 to 0.187 m across the reported non-linear group-walking, mimicry, and collision-avoidance subsets.MoRE improves both reported metrics in every category; these subsets differ from the broader categories in Table 5.Source: §C.10 · p. 23, Table 19 · p. 23

C.11 NBA AND COMPUTATIONAL COST

  • NBA ADE improves from 1.10 to 0.96, while the reported computational-cost comparison includes inference memory, latency, and training durations.Memory refers to inference-time GPU memory, not the full training footprint; training durations are not a hardware-matched speed ranking.Source: §C.11 · pp. 23–24, Table 20 · p. 23

C.12 INFERENCE ON ADDITIONAL HARDWARE

  • MoRE runs at 11.1 Hz on AGX Orin and 18.2 Hz on DGX Spark in full-FP32 PyTorch inference without deployment-specific optimization.These are predictor measurements, not end-to-end navigation measurements.Source: §C.12 · p. 24, Table 21 · p. 24

D.1 NUMERICAL KNOWLEDGE AND CONTEXTUAL PREDICTION

  • The relevant distinction is supervision form: the base policy uses token-level objectives, while numerical experts provide decoded coordinate-space feedback.MoRE uses architectural diversity as a source of priors, without identifying which model understands an agent’s true intention.Source: §D.1 · p. 24

D.2 PRESERVATION OF THE BASE POLICY

  • Reference KL, supervised loss, and restricted refinement data constrain changes differently, but do not guarantee semantic fidelity or behavior outside the selected subset.Good SDD transfer is one test, not an exhaustive guarantee.Source: §D.2 · p. 24

D.3 DIFFERENCE FROM INFERENCE-TIME ENSEMBLING

  • MoRE deploys only the refined language-based policy, but distillation also transfers knowledge during training without inference-time experts.The relevant empirical question is whether reward-based training improves over alternatives under the evaluated controls.Source: §D.3 · p. 24

D.4 REMAINING EVALUATION BOUNDARIES

The reported evaluation leaves boundaries around probabilistic quality, collision interpretation, expert agreement, and statistical reliability.

D.4 REMAINING EVALUATION BOUNDARIES

  • Best-of-20 displacement gains do not establish calibrated probabilities or coverage of all plausible futures, and collision and distance-zone statistics provide only partial information.Lower collision rates can also reflect sample diversity.Source: §D.4 · p. 24
  • Expert agreement may be wrong when expert errors are correlated, and the reported point estimates lack confidence intervals across random seeds.These boundaries motivate evaluating mode coverage, expert correlation, and statistically reliable gains.Source: §D.4 · pp. 24–25

E ADDITIONAL QUALITATIVE RESULTS

The additional visualizations show selected prediction examples across ETH-UCY, SDD, and GCS; the paper describes improved alignment in selected examples but does not treat illustrations as quantitative evidence.

E ADDITIONAL QUALITATIVE RESULTS

  • In selected ETH-UCY and SDD examples, MoRE reduces spatial offset and concentrates samples around paths closer to ground truth.The illustrations do not quantify uncertainty calibration, obstacle clearance, or latent intent.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 25
  • Figures 5–9 visualize selected agents in ETH, HOTEL, UNIV, ZARA1, ZARA2, nexus 6, gates 2, and hyang 0 and 1 scenes.These captions identify scenes, not quantitative outcomes.Source: Figure 5 · p. 26, Figure 6 · p. 27, Figure 7 · p. 28
  • Figures 10–15 visualize selected agents in coupa, hyang, quad, little, nexus, and GCS scenes.The captions identify the displayed scenes but do not state comparative outcomes.Source: Figure 10 · p. 30, Figure 11 · p. 31, Figure 12 · p. 32

Related papers

Loading 2610.07954v1…