PaperlayerView exact revision on arXiv

Source-linked AI summary

M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction

Wenzhe Jin, Haina Tang

arXiv:2609.10559v1cs.LGcs.CV

TL;DR

Because vessels departing the same origin may head to different destinations, long-term trajectory prediction needs to capture intention diversity. M3-Former combines vessel semantics and motion in a hierarchical model, and outperforms existing approaches across real-world maritime evaluations.

  • Problem

    Because vessels departing the same origin may head toward different destinations, long-term prediction must capture intention diversity to support full-route planning and operational management.

    Source: §II · p. 2

  • Method

    M3-Former aligns static vessel semantics with trajectory features and uses sequence- and token-level experts to model global navigation trends and local motion.

    Source: §I · p. 2

  • Results

    Across real-world multimodal maritime evaluations, M3-Former outperforms existing approaches in prediction accuracy, long-term consistency, and robustness under complex environments.

    Source: §I · p. 2

  • Takeaways & Limitations

    Jointly modeling historical motion and vessel attributes captures navigation intent and global route preferences.

    Source: §V · p. 10

Abstract

from arXiv · show

To address the challenges of behavioral multimodality, limited semantic utilization, and long-term error accumulation in vessel trajectory prediction, this paper proposes M3-Former, a multimodal trajectory prediction framework enhanced by large language models (LLMs). The proposed framework incorporates vessel static attributes and navigational intent as semantic priors for long-term trajectory modeling. Specifically, a unified multimodal representation space is constructed, in which static semantic information is encoded by a pre-trained LLM and aligned with dynamic trajectory features through self-attention. To jointly capture global route planning and local motion variations, a dual-granularity Mixture-of-Experts (MoE) architecture is introduced, where sequence-level experts model global navigation trends and token-level experts refine fine-grained maneuvering behaviors. In addition, a Steering-Weighted Cross-Entropy loss is designed to alleviate the long-tail distribution of sparse turning samples and improve prediction accuracy in critical maneuvering scenarios. Experiments on a real-world Danish AIS dataset demonstrate that M\textsuperscript{3}-Former consistently outperforms state-of-the-art baselines across prediction horizons from 1 to 4 hours. In the 4-hour prediction task, the proposed method reduces Average Displacement Error (ADE) and Final Displacement Error (FDE) by 4.4\% and 5.1\%, respectively, compared with the strongest baseline. Qualitative and ablation analyses further verify that semantic fusion effectively reduces long-term trajectory drift, while the dual-granularity MoE improves robustness in complex waterways and route-branching scenarios. The proposed framework establishes a semantic-guided hierarchical prediction paradigm, in which high-level navigational intent and local motion dynamics are jointly modeled for robust long-term vessel trajectory forecasting.

I. Introduction

M3-Former addresses long-term vessel trajectory prediction’s accumulated error, shifting semantic drivers, and multimodal futures with semantic-guided hierarchical modeling. Its dual-granularity MoE combines global navigation trends with local maneuvering, and experiments report improved accuracy and robustness.

I. Introduction

  • Long-term prediction is challenging because small intent or maneuver deviations accumulate, while destination semantics, vessel attributes, and intent increasingly shape trajectories and futures remain multimodal.The introduction frames accurate prediction as important for navigational safety, route planning, port operations, surveillance, collision avoidance, and automated shipping scheduling.Source: §I · p. 1
  • M3-Former formulates long-term prediction as semantic-guided forecasting, incorporating static vessel attributes as priors and generating multiple plausible futures under physical and semantic constraints.Static semantics and dynamic motion features share a representation space and are jointly modeled through attention and routing.Source: §I · p. 2
  • M3-Former uses Per-Sequence and Per-Token routing to model global navigation strategies and refine local motion features in a dual-granularity MoE.Sequence-level experts jointly model trajectory and text modalities to capture intent and global trends; token-level experts refine temporal features for maneuvers and interactions.Source: §I · p. 2
  • Experiments on real-world multimodal maritime datasets report that M3-Former outperforms existing approaches in accuracy, long-term consistency, and robustness in complex environments.The paper also reports improved ADE and FDE and physical plausibility and multimodal coverage in turning, route-bifurcation, and complex-port scenarios.Source: §I · p. 2

II. Problem Definition · III. Methodology

The paper formulates long-term vessel trajectory prediction as conditioning future routes on dynamic histories and static vessel information. M3-Former uses semantic priors to guide global trajectory evolution while motion patterns refine local behavior.

II. Problem Definition

  • Vessels departing the same origin may pursue different destinations, motivating long-term prediction over hours to days for route planning and operational management.The problem also targets spatio-temporal dependencies and external influencing factors.Source: §II · p. 2
  • Dynamic input trajectories represent each vessel using latitude, longitude, Speed Over Ground (SOG), and Course Over Ground (COG).The trajectory is indexed over its length T.Source: §II · p. 2
  • Static vessel features come from AIS data and include destination, vessel type, width, and length.Source: §II · p. 2
  • The prediction objective is to estimate future trajectory points from historical trajectories and static information, integrating dynamic motion patterns with static intent.Source: §II · p. 2

III. Methodology

  • M3-Former follows semantic-guided hierarchical modeling: semantic priors determine global trajectory evolution, while dynamic motion patterns refine local behaviors.Fig. 1 presents the proposed M3-Former overall architecture.Source: §III · p. 2, Figure 1 · p. 3

A. Multimodal Feature Fusion via Large Language Models · B. Dual-Granularity Mixture-of-Experts (MoE) Model

M3-Former fuses LLM-encoded vessel semantics with trajectory dynamics, then uses sequence- and token-level experts to model global navigational intent and local motion. Its framework also represents spatial uncertainty through discrete position prediction.

A. Multimodal Feature Fusion via Large Language Models

  • Static vessel attributes are converted to text and encoded by a frozen pre-trained LLM, then projected and fused with trajectory embeddings through self-attention in a shared Transformer.Dynamic features are linearly mapped, while the text embeddings are prepended as prefix tokens to the trajectory sequence.Source: A. Multimodal Feature Fusion via Large… · p. 3
  • Unified self-attention fusion performed best among the tested weighted, gated, and cross-attention strategies, consistent with alignment results in ViLBERT and CLIP.The model also discretizes continuous coordinates into spatial bins, following TrAISformer, to model multimodal distributions as classification.Source: A. Multimodal Feature Fusion via Large… · p. 3
  • At each time step, the model predicts probabilities over discrete spatial positions, enabling multiple candidate trajectories that respect physical constraints and semantic logic.This approach captures uncertainty and diversity in possible navigational intents during long-term prediction.Source: A. Multimodal Feature Fusion via Large… · p. 4

B. Dual-Granularity Mixture-of-Experts (MoE) Model

  • Together, the two MoE levels coordinate local motion responsiveness with stable global navigational constraints through hierarchical modeling and multimodal fusion.The architecture is designed to improve handling of heterogeneous trajectory data and long-term prediction.Source: B. Dual-Granularity Mixture-of-Experts… · p. 4
  • Sequence-level MoE aggregates the historical sequence into global context and uses a gating network to weight experts modeling long-term intent and behavior.These experts capture destination-driven route choices and behaviors such as port maneuvering or open-water cruising, conditioned on vessel attributes and operational constraints.Source: B. Dual-Granularity Mixture-of-Experts… · p. 4
  • Token-level MoE selects experts based on each time step’s motion state to model fine-grained patterns such as cruising, gradual corrections, and sharp turns.Experts adaptively weight multimodal information to represent local motion behavior.Source: B. Dual-Granularity Mixture-of-Experts… · p. 4
  • The MoE-S→T configuration, applying sequence-level experts before token-level experts, outperformed MoE-T→S in experimental analysis.The proposed ordering extracts global trends before fine-grained adjustments, aligning with the described route-planning and local-control sequence.Source: B. Dual-Granularity Mixture-of-Experts… · p. 4

C. Steering-Weighted Auxiliary Loss Design

The Steering-Weighted Cross-Entropy loss addresses the dominance of straight-line samples by increasing the training contribution of turning samples. It weights cross-entropy losses for discretized trajectory variables to better capture critical maneuvers.

C. Steering-Weighted Auxiliary Loss Design

  • The loss targets long-tailed AIS behavior: frequent straight-line sailing can dominate training and produce delayed or over-smoothed predictions near sparse turning regions.Source: C. Steering-Weighted Auxiliary Loss… · p. 4
  • The model predicts latitude, longitude, speed over ground (SOG), and course over ground (COG) at each time step, using the sum of their categorical cross-entropy losses as its basic prediction loss.Each variable is represented as a discrete token.Source: C. Steering-Weighted Auxiliary Loss… · p. 4
  • A steering-aware factor increases a sample's weight when its heading change exceeds threshold τ, with α controlling the additional penalty for turning samples.The factor is λ_i,t = 1 + α · 1(∆COG_i,t > τ).Source: C. Steering-Weighted Auxiliary Loss… · pp. 4–5

IV. Experiments and Analysis · A. Dataset and Preprocessing

The study uses Danish Maritime Authority AIS records from January to March 2023, transformed into dynamic motion and static semantic features through preprocessing for long-term trajectory prediction. The resulting training data covers thousands of vessels, trajectory segments, and destinations.

A. Dataset and Preprocessing

  • The publicly available Danish Maritime Authority AIS dataset spans January 1 to March 31, 2023, and is described as highly continuous and representative of real maritime operations.The dataset is illustrated in Fig. 2.Source: A. Dataset and Preprocessing · p. 5
  • Raw AIS data is reconstructed into two feature categories: dynamic motion features and static attribute features.Together, these feature categories preserve vessel motion patterns and semantic attributes for trajectory modeling.Source: A. Dataset and Preprocessing · p. 5
  • Dynamic motion features comprise longitude and latitude trajectories, Speed Over Ground (SOG), and Course Over Ground (COG) to represent vessel kinematics.These features support modeling short-term motion trends and long-term trajectory evolution.Source: A. Dataset and Preprocessing · p. 5
  • Static attributes include MMSI, vessel length, width, draught, and type, while destination and estimated time of arrival (ETA) encode navigational intent and endpoint guidance.The semantic information provides high-level guidance for long-term trajectory prediction.Source: A. Dataset and Preprocessing · p. 5
  • A systematic preprocessing pipeline addresses noise, irregular sampling, and missing AIS signals to produce regularized, model-ready trajectory sequences.Fig. 3 presents the AIS trajectory preprocessing pipeline.Source: A. Dataset and Preprocessing · p. 5, Figure 3 · p. 5
  • After preprocessing, the training set contains 1,923,380 valid samples from 4,089 vessels, organized into 26,374 trajectory segments and covering 4,407 unique destinations.Cargo vessels contribute 1,345,435 samples and tankers 577,945 samples; the average trajectory length is 72.93 time steps, with a maximum of 216 and a minimum of 22.Source: A. Dataset and Preprocessing · pp. 5–6

B. Implementation Detail · C. Hyperparameter Settings · D. Evaluation Metrics

M3-Former is evaluated with ADE and FDE, while the sections specify its training and inference setup, model hyperparameters, and repeated-run testing protocol.

B. Implementation Detail

  • Experiments use input length 36 and prediction horizon 24; on Danish Waters AIS, batch size 256 and five training epochs take about six hours on four GPUs.The implementation uses PyTorch with distributed training support on four NVIDIA A100-SXM4-80GB GPUs and an Intel Xeon 128-core CPU.Source: B. Implementation Detail · p. 6
  • Inference autoregressively generates trajectories with batch size 400, taking about 14 minutes for the full test set.Source: B. Implementation Detail · p. 6

C. Hyperparameter Settings

  • M3-Former uses 768-dimensional embeddings, an 8-layer, 8-head GPT-style causal Transformer, and four Top-1-routed experts with a 1 × 10−4 load-balancing weight.It is trained with Adam at learning rate 1e-4, with AMP optionally applied.Source: C. Hyperparameter Settings · p. 6
  • Training runs for up to 10 epochs with patience-5 early stopping; each experiment is repeated 10 times, and the validation-loss-best model is evaluated..Source: C. Hyperparameter Settings · p. 6

D. Evaluation Metrics

  • ADE measures average trajectory-wide spatial deviation, while FDE measures the distance between predicted and ground-truth final positions.Together, the metrics assess trajectory shape and endpoint accuracy.Source: D. Evaluation Metrics · p. 6
  • For evaluation, predicted discrete tokens are converted to continuous geographic coordinates, while training uses the steering-weighted multi-task cross-entropy loss.Source: D. Evaluation Metrics · p. 6
  • Testing uses historical trajectories as inputs and future ground truth as labels; metrics are averaged over the full test set and across repeated runs.The reported final result is the mean across runs.Source: D. Evaluation Metrics · p. 6

E. Quantitative Evaluation · F. Qualitative Analysis · 1) Prediction Comparisons in Representative Scenarios:

M3-Former achieves state-of-the-art results across prediction horizons, with lower 4-hour errors than TrAISformer. In representative scenarios, its multimodal inputs and expert gating support predictions that follow sharp turns, remain spatially plausible near shorelines, and select the actual channel branch.

E. Quantitative Evaluation

  • 4-hour ADE = 6.4727 and FDE = 13.6142, improving over TrAISformer by 4.4% and 5.1%, respectively; M3-Former achieves state-of-the-art results across all horizons.The quantitative comparison uses ADE and FDE; the table summarizes results under different prediction horizons.Source: Table I · p. 7, E. Quantitative Evaluation · p. 7
  • ADE increases from 1.5537 to 6.4727 for M3-Former, compared with GRU’s increase from 3.8014 to 9.2527, indicating slower error growth over longer horizons.The paper attributes the slower growth to multimodal semantic priors mitigating long-term drift.Source: E. Quantitative Evaluation · p. 7

1) Prediction Comparisons in Representative Scenarios:

  • In sharp turns, M3-Former follows the Ground Truth from the onset of the turn, reducing lag and lateral deviation through gating that activates experts specialized in high-curvature motion.The gating uses high-order dynamic features such as current speed and heading change rate.Source: 1) Prediction Comparisons in… · p. 7, Figure 4 · p. 8
  • In narrow waters and near complex shorelines, multimodal AIS fusion and kinematics constraints yield smooth, spatially consistent predictions without land crossing or unrealistic maneuvers.The fused static attributes include vessel type, draught, and destination; the loss penalizes noncompliant heading change rates.Source: 1) Prediction Comparisons in… · p. 7, Figure 4 · p. 8
  • At channel bifurcations, AIS static fields and MoE experts represent multiple possible intentions, enabling M3-Former to predict trajectories aligned with the actual branch.The gating assigns probabilistic weights based on current state, including speed profile, docking tendency, and vessel type; TrAISformer struggles to consistently select the correct branch without intent information.Source: 1) Prediction Comparisons in… · pp. 7–8, Figure 4 · p. 8

F. Qualitative Analysis

  • Across representative scenarios, CV and KF show straight-line drift, while sequence-based models can lag, cross land, or diverge; M3-Former stays closer to the true trajectories.The qualitative comparison covers sharp turns, near-shore navigation, and channel bifurcations.Source: 1) Prediction Comparisons in… · p. 8, Figure 4 · p. 8

2) Ablation Study

Removing AIS destination information raises ADE and FDE across prediction horizons, with larger errors at longer horizons. The findings indicate that destination helps constrain navigation intent and reduce long-term trajectory deviation.

Ablation Study

  • At Step 6, ADE increases by only 0.015 NM after replacing Destination with UNK, consistent with short-term forecasts relying mainly on historical speed and heading.Source: 2) Ablation Study · p. 8
  • At Step 24, removing destination information increases ADE by approximately 0.17 NM and FDE by approximately 0.42 NM, with both errors rising across Steps 6–24.The ablation replaces the original Destination field with an UNK token and compares the result with the full model.Source: 2) Ablation Study · p. 8
  • Destination acts as a global intention constraint that narrows feasible routes at channel bifurcations, helping avoid averaged predictions or mis-selected paths.Source: 2) Ablation Study · p. 8
  • Without destination information, long-term predictions are more likely to drift directionally or deviate from the main channel, increasing trajectory and endpoint errors.The study concludes that destination has limited short-term effect but improves long-term navigation-intent capture and model stability.Source: 2) Ablation Study · p. 8

1) Ablation Analysis of Key Modules:

Ablations under identical training configurations show that the complete M3-Former, combining Multimodal features, MoE, and TurnLoss, performs best. Each module contributes through semantic intent priors, complex-pattern modeling, or emphasis on turning behavior.

1) Ablation Analysis of Key Modules:

  • ADE 6.4727 and FDE 13.6142 are the complete model’s best ablation results, supporting complementary contributions from all three modules.The configuration is MoE + Multimodal + TurnLoss; lower values indicate better performance.Source: 1) Ablation Analysis of Key Modules · p. 9, Table III · p. 9
  • Without Multimodal features, ADE rises from 6.4727 to 6.9768 and FDE from 13.6142 to 14.9049 when MoE and TurnLoss remain.The paper attributes the improvement to LLM-derived semantic information, such as ship type and destination, providing long-term navigation intent priors.Source: 1) Ablation Analysis of Key Modules · p. 9
  • Removing MoE from the Multimodal + TurnLoss configuration raises ADE from 6.4727 to 6.6371 and FDE from 13.6142 to 13.9431.The dual-granularity experts model local dynamic changes and global navigation strategies across different navigation patterns.Source: 1) Ablation Analysis of Key Modules · p. 9
  • Adding TurnLoss to MoE + Multimodal reduces ADE from 6.5665 to 6.4727, with turning-point weighting targeting heading changes such as channel changes or port maneuvers.The paper notes that straight-line motion dominates AIS trajectories, so conventional losses tend to underemphasize turning behavior.Source: 1) Ablation Analysis of Key Modules · p. 9

2) MoE Structure Granularity Analysis: · H. Effect of MoE Granularity on Model Performance · V. Conclusion and Future Work

The analysis finds sequence-to-token dual-granularity MoE most effective among the tested routing variants, and concludes that M3-Former combines semantic, motion, and steering-aware modeling for long-term vessel prediction. Future work targets richer maritime context, deployment efficiency, and broader generalization.

H. Effect of MoE Granularity on Model Performance

  • Single-granularity MoE-Token and MoE-Seq had modest importance values of 0.044 and 0.043, suggesting neither local nor global modeling alone captures complex navigation patterns.MoE-Token models local dynamics but lacks global context, while MoE-Seq captures overall trends but misses fine-grained variations.Source: H. Effect of MoE Granularity on Model… · p. 9
  • MoE-S→T achieved importance 0.587 and correlation −0.342, combining high contribution with reduced validation error and supporting its use in the final architecture.The reported metrics aggregate statistics over 20 runs.Source: H. Effect of MoE Granularity on Model… · pp. 9–10
  • MoE-T→S had importance 0.326 but positive correlation 0.339, indicating that this influential variant can worsen validation loss.The paper attributes this pattern to local noise propagating into and interfering with global predictions.Source: H. Effect of MoE Granularity on Model… · pp. 9–10
  • MoE-S→T first models global trajectory trends and then local token-level dynamics, matching the operational sequence of route planning before local maneuvers.By contrast, token-to-sequence routing may propagate local noise into global predictions.Source: H. Effect of MoE Granularity on Model… · p. 10

V. Conclusion and Future Work

  • M3-Former combines historical motion and static destination-related semantics with a discretization-based generative formulation, dual-granularity experts, and steering-aware weighting for rare turns.On real AIS data, it improves prediction accuracy and stability over strong baselines, especially in complex turning and route-branching scenarios.Source: §V · p. 10
  • Future work proposes adding weather, sea state, tides, and traffic constraints; improving efficiency for large-scale real-time deployment; and exploring transfer learning and domain adaptation across maritime domains.The targeted generalization spans sea areas, vessel types, and traffic regimes.Source: §V · p. 10

Related papers

Loading 2609.10559v1…