PaperlayerView exact revision on arXiv

Source-linked AI summary

M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction

Wenzhe Jin, Haina Tang

arXiv:2609.10559v1cs.LGcs.CV

TL;DR

Long-term vessel trajectory prediction must account for behavioral multimodality, limited semantic information, and accumulating error. M3-Former addresses these issues by combining LLM-encoded vessel semantics, dynamic trajectories, hierarchical MoE routing, and steering-weighted training, with reported gains over strong baselines.

  • Problem

    Long-term vessel trajectory prediction is challenged by multimodal behavior, insufficient semantic utilization, and accumulating prediction errors.

  • Method

    M3-Former fuses LLM-encoded static vessel attributes with dynamic trajectories and uses sequence- and token-level MoE routing plus a steering-weighted loss.

  • Results

    M3-Former consistently improves prediction accuracy and stability over strong baselines on real AIS data across long-term forecasting horizons.

  • Takeaways & Limitations

    The framework supports semantic-guided hierarchical forecasting by jointly modeling high-level navigational intent and local motion dynamics.

Abstract

from arXiv · show

To address the challenges of behavioral multimodality, limited semantic utilization, and long-term error accumulation in vessel trajectory prediction, this paper proposes M3-Former, a multimodal trajectory prediction framework enhanced by large language models (LLMs). The proposed framework incorporates vessel static attributes and navigational intent as semantic priors for long-term trajectory modeling. Specifically, a unified multimodal representation space is constructed, in which static semantic information is encoded by a pre-trained LLM and aligned with dynamic trajectory features through self-attention. To jointly capture global route planning and local motion variations, a dual-granularity Mixture-of-Experts (MoE) architecture is introduced, where sequence-level experts model global navigation trends and token-level experts refine fine-grained maneuvering behaviors. In addition, a Steering-Weighted Cross-Entropy loss is designed to alleviate the long-tail distribution of sparse turning samples and improve prediction accuracy in critical maneuvering scenarios. Experiments on a real-world Danish AIS dataset demonstrate that M\textsuperscript{3}-Former consistently outperforms state-of-the-art baselines across prediction horizons from 1 to 4 hours. In the 4-hour prediction task, the proposed method reduces Average Displacement Error (ADE) and Final Displacement Error (FDE) by 4.4\% and 5.1\%, respectively, compared with the strongest baseline. Qualitative and ablation analyses further verify that semantic fusion effectively reduces long-term trajectory drift, while the dual-granularity MoE improves robustness in complex waterways and route-branching scenarios. The proposed framework establishes a semantic-guided hierarchical prediction paradigm, in which high-level navigational intent and local motion dynamics are jointly modeled for robust long-term vessel trajectory forecasting.

I. Introduction

M3-Former frames long-term vessel trajectory prediction as semantic-guided hierarchical modeling, combining static vessel information with dynamic motion features. Its multimodal MoE design targets global navigation trends, local maneuvers, and diverse future paths.

  • Motivation: M3-Former addresses long-term prediction challenges from multimodal vessel behavior, accumulating errors, and changing influences across prediction horizons.Long-term trajectories are affected by navigation intent, environmental constraints, and maneuvering behavior.
  • Method Overview: The framework projects static semantic information and dynamic trajectory features into a shared representation space for joint modeling.Static inputs include vessel attributes such as type, dimensions, destination, and ETA.
  • Method Overview: Its dual-granularity MoE follows a global-to-local paradigm, using sequence-level routing for navigation strategy and token-level routing for fine-grained motion.The design jointly models global navigation trends and local maneuvering behaviors.
  • Evaluation: Experiments on real-world multimodal maritime datasets report improvements in prediction accuracy, long-term consistency, and robustness under complex environments.Reported challenging scenarios include turning maneuvers, route bifurcations, and complex port environments.
  • Problem Definition: The model formulates prediction as learning future trajectories conditioned on historical motion and static vessel information.This formulation integrates dynamic motion patterns with static intent for long-term route prediction.

III. Methodology

The model follows a semantic-guided hierarchical principle in which semantic priors guide global trajectory evolution and dynamic motion patterns refine local behavior. The architecture is presented as the overall M3-Former design.

  • III. Methodology: M3-Former uses semantic priors to determine global trajectory evolution while dynamic motion patterns refine local behaviors.This principle links high-level navigation intent with fine-grained motion modeling.
  • III. Methodology: The overall architecture is presented as the proposed M3-Former framework.

A. Multimodal Feature Fusion via Large Language Models

M3-Former fuses LLM-encoded vessel semantics with dynamic trajectory features in a unified representation, then models multimodal futures with discrete probabilistic prediction and hierarchical MoE routing. Sequence-level experts capture global intent, while token-level experts adapt to local motion states.

  • A. Multimodal Feature Fusion via Large Language Models: Static vessel descriptions are encoded by a pretrained LLM and aligned with trajectory embeddings through shared Transformer self-attention.Text embeddings are prepended as prefix tokens before joint encoding with trajectory tokens.
  • A. Multimodal Feature Fusion via Large Language Models: Spatial discretization converts continuous coordinate prediction into discrete token classification, enabling multiple physically and semantically plausible trajectories.The model outputs logits over discrete spatial positions at each time step.
  • B. Dual-Granularity Mixture-of-Experts (MoE) Model: The dual-granularity MoE uses sequence-level routing for global navigational intent and token-level routing for local motion patterns.The architecture decouples expert routing across sequence and token levels.
  • B. Dual-Granularity Mixture-of-Experts (MoE) Model: Sequence-level experts provide global constraints such as destination-driven route selection, port maneuvering, and open-water cruising behavior.A pooled historical sequence supplies global context to the sequence-level gating network.
  • B. Dual-Granularity Mixture-of-Experts (MoE) Model: Token-level experts dynamically select local motion behaviors such as cruising, gradual corrections, and sharp turns based on the current state.The reported MoE-S→T ordering outperforms the MoE-T→S alternative.

C. Steering-Weighted Auxiliary Loss Design

The Steering-Weighted Cross-Entropy loss addresses sparse turning behavior by increasing the training contribution of maneuvering samples in discrete trajectory prediction.

  • C. Steering-Weighted Auxiliary Loss Design: The loss targets a long-tailed trajectory distribution in which straight-line sailing is frequent but turning behaviors are sparse.Conventional objectives can therefore produce delayed or over-smoothed predictions near critical maneuvering regions.
  • C. Steering-Weighted Auxiliary Loss Design: The basic prediction loss sums categorical cross-entropy terms for latitude, longitude, speed over ground, and course over ground.The framework treats trajectory prediction as discrete token generation, with one predicted variable for each of the four motion attributes.
  • C. Steering-Weighted Auxiliary Loss Design: Heading change between consecutive time steps is used to identify rare turning samples for steering-aware weighting.The weighting factor uses a turning threshold and an additional penalty parameter for samples exceeding that threshold.
  • C. Steering-Weighted Auxiliary Loss Design: Increasing the contribution of turning samples reduces bias toward dominant straight-line trajectories and improves accuracy in complex navigation scenarios.The paper identifies channel transitions, route bifurcations, and collision-avoidance maneuvers as relevant scenarios.

A. Dataset and Preprocessing

The study uses Danish Maritime Authority AIS records and preprocesses noisy, irregular observations into regularized trajectory sequences containing dynamic motion and static semantic attributes.

  • A. Dataset and Preprocessing: The publicly available Danish Maritime Authority dataset covers vessel navigation records from January 1, 2023, to March 31, 2023.The records are described as having high spatiotemporal continuity and representativeness of real-world maritime operations.
  • A. Dataset and Preprocessing: Dynamic features comprise longitude, latitude, speed over ground, and course over ground, while static attributes include vessel characteristics, type, destination, and ETA.Destination and ETA provide semantic information intended to capture navigational intent and endpoint constraints.
  • A. Dataset and Preprocessing: A systematic preprocessing pipeline converts noisy, irregularly sampled, and incomplete AIS observations into regularized, model-ready trajectory sequences.The pipeline is presented as pseudo-code for processing raw AIS CSV data at a specified time interval.
  • A. Dataset and Preprocessing: The dataset contains 1,923,380 valid training samples from 4,089 vessels, organized into 26,374 trajectory segments and 4,407 unique destinations.Cargo vessels and tankers contribute 1,345,435 and 577,945 samples, respectively.
  • A. Dataset and Preprocessing: The resulting dataset preserves dynamic motion patterns and static semantic attributes for long-term multimodal trajectory prediction.Average trajectory length is 72.93 time steps, with a maximum of 216 and a minimum of 22.

B. Implementation Detail

Implementation uses a Transformer-based discrete trajectory model with fixed sequence settings, repeated evaluation, and ADE/FDE metrics computed over the full test set.

  • B. Implementation Detail: The model uses 768-dimensional embeddings, an 8-layer GPT-style causal Transformer with 8 attention heads, and four Top-1-routed experts.Training uses Adam with a learning rate of 1e-4 and a load-balancing weight of 1 × 10^-4.
  • B. Implementation Detail: Training runs for up to 10 epochs with early stopping, and each experiment is repeated 10 times before selecting the best validation-loss model.The reported test performance averages repeated runs to reduce randomness.
  • B. Implementation Detail: The evaluation reports Average Displacement Error and Final Displacement Error, measuring overall trajectory deviation and final-position error, respectively.Together, the metrics reflect trajectory-shape and endpoint accuracy.
  • B. Implementation Detail: During inference, predicted discrete tokens are converted back into continuous geographic coordinates before ADE and FDE evaluation.The model is trained with a steering-weighted multi-task cross-entropy loss.

E. Quantitative Evaluation

M3-Former is evaluated against traditional and deep-learning baselines across long-term prediction horizons, with quantitative and qualitative analyses covering challenging maneuvers and route branching.

  • Experimental setup: The evaluation compares M3-Former with representative baselines ranging from traditional motion models to recent deep-learning methods.
  • Quantitative comparison: For 4-hour prediction, M3-Former achieves ADE = 6.4727 and FDE = 13.6142, improving over TrAISformer by 4.4% and 5.1%.It achieves state-of-the-art results across all evaluated horizons.
  • Qualitative comparison: M3-Former closely follows ground truth from the onset of sharp turns, reducing the lag seen in kinematic and sequence-based baselines.Its MoE gating dynamically allocates expert weights using high-order dynamic features such as speed and heading-change rate.
  • Qualitative comparison: In narrow waters and near complex shorelines, M3-Former addresses terrain constraints and behavioral restrictions that cause land crossing or oscillations in coordinate-only models.
  • Qualitative comparison: In bifurcating channels, AIS static fields and multiple experts help M3-Former represent distinct intentions and predict the actual branch rather than a mean path.

2) Ablation Study

Destination information has limited short-term effect but increasingly constrains long-term forecasts, where removing it produces cumulative trajectory and endpoint errors.

  • Destination ablation: Removing destination information consistently increases ADE and FDE across prediction horizons, with the gap amplifying as the horizon extends.The authors interpret destination as a sustained constraint on long-term forecasting.
  • Destination ablation: At Step 6, ADE increases by only 0.015 NM after destination removal, indicating limited short-term impact.Very short-term forecasts primarily rely on historical speed and heading.
  • Destination ablation: At Step 24, removing destination increases ADE by approximately 0.17 NM and FDE by approximately 0.42 NM.The discrepancy is attributed to gradual deviation from true navigation intent.
  • Destination ablation: Destination information reduces the feasible trajectory space in branching scenarios and guides predictions toward the correct route.Without it, local-motion extrapolation can produce averaged predictions or mis-selected paths.
  • Destination ablation: Removing destination increases the likelihood of directional drift or deviation from the main channel in long-term predictions.

G. Ablation Study

Ablations show that multimodal semantics, dual-granularity MoE routing, and turn-weighted loss each improve prediction, with their combination achieving the best ADE and FDE.

  • Key-module ablation: The complete model achieves the best performance, with ADE = 6.4727 and FDE = 13.6142.The three modules exhibit complementary effects when used jointly.
  • Key-module ablation: Removing multimodal features increases ADE from 6.4727 to 6.9768 and FDE from 13.6142 to 14.9049.The comparison supports the contribution of semantic information such as ship type and destination.
  • Key-module ablation: Removing MoE increases ADE from 6.4727 to 6.6371 and FDE from 13.6142 to 13.9431 under identical multimodal inputs.The dual-granularity structure models local dynamic changes and global navigation strategies through expert routing.
  • Key-module ablation: Adding TurnLoss reduces ADE from 6.5665 to 6.4727 under the MoE + Multimodal configuration.The loss increases training weight for sparse critical turning points, including channel changes and port maneuvers.
  • Key-module ablation: All three modules contribute positively, while their synergistic effect yields optimal performance on both ADE and FDE.

2) MoE Structure Granularity Analysis:

The analysis compares MoE routing granularities and finds that sequence-to-token routing best combines global trajectory planning with local maneuver refinement. This design is therefore adopted in M3-Former, whose broader evaluations show improved accuracy and stability in complex turning and route-branching scenarios.

  • MoE Structure Granularity Analysis:: Single-granularity routing contributes modestly because token-level experts lack global context while sequence-level experts miss fine-grained variations.MoE-Token and MoE-Seq have importance values of 0.044 and 0.043, respectively.
  • MoE Structure Granularity Analysis:: MoE-T→S is important but positively correlated with validation loss, indicating that propagating local noise can interfere with global predictions.Its importance score is 0.326 and its correlation is 0.339.
  • MoE Structure Granularity Analysis:: MoE-S→T first models global trajectory trends and then refines local token-level dynamics, matching the operational order of route planning and maneuvering.
  • MoE Structure Granularity Analysis:: MoE-S→T achieves the strongest combination of importance and validation-loss reduction, supporting its adoption in the final architecture.Its importance score is 0.587 and its correlation is -0.342.
  • MoE Structure Granularity Analysis:: The final framework combines sequence-level global semantics, token-level motion refinement, and steering-aware weighting to improve difficult turning and route-branching predictions.
Loading 2609.10559v1…