PaperlayerView exact revision on arXiv

Source-linked AI summary

TrAISformer -- A Transformer Network with Sparse Augmented Data Representation and Cross Entropy Loss for AIS-based Vessel Trajectory Prediction

Duong Nguyen, Ronan Fablet

arXiv:2109.03958v4cs.AI

TL;DR

Forecasting vessel trajectories from AIS remains challenging over medium-range horizons because the motion data are heterogeneous and multimodal. TrAISformer combines a sparse, high-dimensional AIS representation with a probabilistic transformer and classification-based modeling. On a real AIS dataset, it reports prediction error below 10 nmi up to 10 hours, outperforming previous works.

  • Problem

    AIS-based vessel prediction remains challenging beyond several hours because vessel movements are complex and the data are heterogeneous and multimodal.

    Source: §I · p. 1

  • Method

    TrAISformer uses a sparse, high-dimensional AIS representation, classification-based learning, and a probabilistic transformer to model multimodality and long-term dependencies.

    Source: §I · p. 2, A. Discrete and sparse representation… · p. 3, B. Transformer architecture · p. 4

  • Results

    Below 10 nmi up to 10 hours, TrAISformer significantly outperforms previous works on a real AIS dataset.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported findings open research avenues for maritime applications including search and rescue and port congestion avoidance.

    Source: §V · p. 10

Abstract

from arXiv · show

Vessel trajectory prediction plays a pivotal role in numerous maritime applications and services. While the Automatic Identification System (AIS) offers a rich source of information to address this task, forecasting vessel trajectory using AIS data remains challenging, even for modern machine learning techniques, because of the inherent heterogeneous and multimodal nature of motion data. In this paper, we propose a novel approach to tackle these challenges. We introduce a discrete, high-dimensional representation of AIS data and a new loss function designed to explicitly address heterogeneity and multimodality. The proposed model-referred to as TrAISformer-is a modified transformer network that extracts long-term temporal patterns in AIS vessel trajectories in the proposed enriched space to forecast the positions of vessels several hours ahead. We report experimental results on real, publicly available AIS data. TrAISformer significantly outperforms state-of-the-art methods, with an average prediction performance below 10 nautical miles up to ~10 hours.

I. INTRODUCTION

AIS-based trajectory prediction matters for maritime applications, but forecasting vessels over hours remains challenging because motion is heterogeneous, multimodal, and dependent on long histories. TrAISformer combines a sparse, high-dimensional AIS representation with a probabilistic transformer, and reports prediction error below 10 nmi up to 10 hours.

I. INTRODUCTION

  • Medium-range vessel locations support applications including search and rescue, traffic control, path planning, congestion avoidance, and pollution monitoring.The stated horizons range from tens of minutes to tens of hours.Source: §I · p. 1
  • Forecasting beyond several hours remains difficult: prior schemes perform well mainly at short horizons or under predefined-route or unimodal movement patterns.The paper links the challenge to complex vessel movements and heterogeneous AIS data.Source: §I · p. 1
  • At maritime waypoints, distinct turning directions form separate modes, and prediction may require backtracking several time steps to infer large-scale movement patterns.The paper identifies representing traffic beyond fixed route networks and capturing multi-scale trajectory patterns as its two methodological challenges.Source: §I · p. 2
  • TrAISformer uses a sparse, high-dimensional AIS representation and frames prediction as classification to model data heterogeneity and trajectory multimodality.Its probabilistic transformer is used to capture long-term dependencies in AIS trajectories.Source: §I · p. 2
  • Below 10 nmi up to 10 hours, TrAISformer significantly outperforms previous works on a real AIS dataset.The reported comparison names,, and.Source: §I · p. 2

II. PROBLEM STATEMENT AND RELATED WORK

The section formulates AIS trajectory prediction as forecasting future observations from historical observations and reviews state-space, learning-based, and route-based approaches. TrAISformer instead uses a transformer and classification-based learning without constraining trajectories to an explicit maritime-route graph.

II. PROBLEM STATEMENT AND RELATED WORK

  • AIS trajectory prediction forecasts future vessel positions from historical AIS observations, with the observations comprising position, speed over ground, and course over ground.The series is equally sampled at 10-minute intervals for this paper.Source: §II · p. 2
  • State-space methods can rely on simplistic motion priors and error-prone filtering, while learning-based recurrent models may fail to capture AIS multi-path patterns.Interaction-based models improve performance in dense traffic at short horizons, but interactions are less pronounced in the open sea and difficult to model over medium-range horizons.Source: §II · pp. 2–3
  • Route-based clustering methods require highly structured traffic, although a significant fraction of real AIS trajectories cannot be assigned to predefined routes.These methods represent traffic as waypoint nodes connected by maritime-route edges.Source: §II · p. 3
  • TrAISformer uses transformer architectures and classification-based learning to predict vessel positions hours ahead without imposing an explicit maritime-route graph.The approach is described as applicable to any trajectory within the region of interest.Source: §II · p. 3

III. PROPOSED APPROACH

The proposed approach introduces an AIS representation, derives a loss function, and briefly presents the transformer architecture used in TrAISformer.

III. PROPOSED APPROACH

  • The approach section introduces a new AIS representation, derives a loss function, and presents the transformer architecture used in TrAISformer.Source: §III · p. 3

A. Discrete and sparse representation of AIS data

The proposed representation expands four AIS attributes into a sparse, high-dimensional embedding to address heterogeneous observations while limiting overfitting. Because discretization loses precision, the paper estimates continuous values using bin midpoints and characterizes the resulting error.

A. Discrete and sparse representation of AIS data

  • The representation addresses heterogeneity, while the paper assigns multimodality to the classification-based training loss.Source: A. Discrete and sparse representation… · p. 3
  • The four-hot vector concatenates separate one-hot encodings of discretized latitude, longitude, speed over ground, and course over ground.Source: A. Discrete and sparse representation… · p. 3, Figure 2 · p. 3
  • Each attribute encoding is associated with a high-dimensional embedding, and the sparse mapping uses only Nlat × Nlon × NSOG × NCOG embedding values to regularize the mapping and avoid overfitting.The observation embedding concatenates the attribute embeddings.Source: A. Discrete and sparse representation… · p. 3
  • A 0.01° latitude and longitude bin resolution yields about 0.15 nmi of midpoint-estimation error, which the authors consider negligible for the cited medium-range applications.The estimate incurs an error equivalent to half the bin resolution even when bin estimation is perfect.Source: A. Discrete and sparse representation… · p. 4

B. Transformer architecture

TrAISformer uses a transformer to capture long-term dependencies in AIS histories, with attention layers that weight past time steps when forming predictions.

B. Transformer architecture

  • The architecture adopts a GPT-like transformer because trajectory forecasting requires capturing long-term dependencies in historical AIS observations.The paper presents the architecture as a suitable candidate for this task.Source: B. Transformer architecture · p. 4
  • Each attention layer computes weights from query and key projections, then uses them to form a weighted sum of value projections.The projections are learned, and the weights are softmax-normalized dot products scaled by √de.Source: B. Transformer architecture · p. 4
  • Parallel attention lets the model retrieve information from multiple past time steps simultaneously, unlike sequential recurrent processing.The authors identify this as a critical advantage for retrieving long-term information.Source: B. Transformer architecture · p. 4
  • The transformer’s final-layer output has the same dimension as ht and is used to model the conditional distribution of future representations.The modeled distribution is p(hT+l|e0:T+l−1).Source: B. Transformer architecture · p. 4

C. Learning scheme

The learning scheme predicts discrete AIS attributes with cross-entropy rather than relying on continuous-valued regression, adds a multi-resolution loss, and generates forecasts recursively by sampling.

C. Learning scheme

  • Classification over four categorical AIS attributes with cross-entropy addresses multimodal posteriors that an L2-based Gaussian formulation may not represent appropriately.The four-hot representation yields four prediction heads; the paper illustrates that categorical modeling preserves two modes where scalar MSE tends to merge them.Source: C. Learning scheme · p. 4, Figure 4 · p. 5
  • The multi-resolution cross-entropy loss marginally improves prediction by combining fine- and coarse-resolution representations.The model embeds the fine-resolution vector, while the loss uses both fine and coarse versions; β balances the coarse-resolution loss.Source: C. Learning scheme · p. 4, Figure 3 · p. 4
  • Forecasts are generated recursively by sampling a four-hot representation, converting it to a position, and feeding that position back for the next step.Multiple sampling runs can produce different possible paths; performance degrades if the model does not sample from the predicted distribution.Source: C. Learning scheme · p. 5

IV. EXPERIMENTS AND RESULTS

The experiments evaluate TrAISformer on a real AIS dataset, benchmarking methods over prediction horizons up to 15 hours and including an ablation study.

IV. EXPERIMENTS AND RESULTS

  • Experiments benchmark TrAISformer against state-of-the-art methods on a real AIS dataset for prediction horizons up to 15 hours.The dataset was introduced in.Source: §IV · p. 5
  • The architecture diagram depicts training with cross-entropy prediction and forecasting through recursive sampling, with each predicted position fed back until horizon L.The figure describes sampling a predicted four-hot vector and feeding its resulting position into the network for the next forecast step.Source: Figure 5 · p. 5
  • The study includes an ablation analysis and uses a publicly available AIS dataset, with the model code made available for reproducibility.The code repository is identified in the passage.Source: §IV · p. 5

A. Experimental set-up

The experiments use a public Danish AIS dataset with separate training, tuning, and test periods. TrAISformer is evaluated against deep-learning baselines using filtered, down-sampled voyages and a best-of-16 prediction criterion.

A. Experimental set-up

  • The public DMA dataset covers cargo and tanker vessels in a defined Danish region, with January 1–March 10 for training, March 11–20 for tuning, and March 21–31, 2019 for testing.The raw dataset contained approximately 712 million AIS messages.Source: A. Experimental set-up · p. 5
  • Preprocessing removes AIS messages with SOG ≥30 knots, moored or at-anchor vessels, observations within 1 nautical mile of the coastline, and voyages under 20 messages or 4 hours, while splitting gaps of 2 hours or more.Source: A. Experimental set-up · p. 6
  • The model uses four-hot AIS representations and an eight-layer, eight-head transformer, with a three-hour history and prediction horizons up to 15 hours.The representation bins latitude and longitude at 0.01°, speed at 1 knot, and course at 5°; training used a 768-dimensional embedding and AdamW.Source: A. Experimental set-up · pp. 6–7
  • Benchmarks include LSTM seq2seq, convolutional seq2seq, seq2seq with attention, and GeoTrackNet; clustering-based comparisons are difficult because methods and route assumptions differ.The score is treated as a best-case scenario because it covers tanker trajectories on predefined routes.Source: A. Experimental set-up · p. 7
  • Prediction error is measured by haversine distance, and the reported score is the best of 16 sampled predictions per target trajectory.The best-of-N criterion is used to account for multimodality.Source: A. Experimental set-up · p. 7

B. Results

TrAISformer outperforms the benchmarked models, including at multi-hour horizons, and its predictions handle both common and less frequent routes. Attention visualizations show recent inputs matter on straight segments, while earlier waypoints matter at turns.

B. Results

  • TrAISformer scores 0.94 nmi versus 1.93 nmi for, while direct deep-learning models have mean error greater than 6 nmi at two hours.The result is a best-case score from tanker trajectories on predefined routes, and that subset is not publicly available.Source: B. Results · pp. 7–8
  • The deterministic TrAISformer variant’s two-hour error rises from 0.94 to 2.88, while remaining better than standard seq2seq models.The authors state that vessels with the same current position and velocity may take different paths at the next waypoint.Source: B. Results · p. 8
  • At two hours, TrAISformer is the only model below one nautical mile and is 41% better than GeoTrackNet.Table I reports mean prediction errors at one-, two-, and three-hour horizons; the ROI includes waypoints where incorrect turn predictions can sharply increase errors.Source: B. Results · p. 7, Table I · p. 8
  • The 10-nmi meaningful-horizon threshold extends to 9.67 hours for TrAISformer versus 1.67 hours for LSTM seq2seq with attention, a factor of approximately 5.8.The paper relates the threshold to visibility under clear weather conditions.Source: Figure 6 · p. 8, B. Results · p. 8
  • Across examples, TrAISformer samples realistic turning directions and succeeds on less frequent routes where clustering-based methods struggle.The examples report the closest of 16 sampled trajectories for probabilistic models.Source: B. Results · p. 8, Figure 7 · p. 9
  • On straight segments, attention focuses on recent time steps; at waypoints, it reaches back to earlier waypoints and intervening movement patterns.The visualization maps input time steps against output time steps; the authors use one trajectory as an illustrative example.Source: B. Results · pp. 8, 10, Figure 8 · p. 10

C. Ablation study

The ablation study tests the high-dimensional encoding, its sparsity constraint, and the classification-based loss. Removing any of these components worsens performance relative to TrAISformer.

C. Ablation study

  • Removing both e_t and h_t applies a GPT model directly to four-dimensional AIS data streams.Source: C. Ablation study · p. 10
  • Keeping e_t but removing h_t tests the sparsity constraint using a MultiLayer Perceptron for the embedding.Source: C. Ablation study · p. 10
  • All three ablated models performed significantly worse than TrAISformer, with the regression-loss variant showing a slightly greater performance degradation.Source: C. Ablation study · p. 10

V. CONCLUSIONS AND FUTURE WORK

TrAISformer uses sparse, augmented, high-dimensional AIS representations and a classification-based loss to model multimodal vessel trajectories. On real AIS data, it outperforms existing methods, with a 9-hour prediction error below 10 nmi in a dense, complex case-study region.

V. CONCLUSIONS AND FUTURE WORK

  • A 9-hour-ahead prediction error below 10 nmi accompanies TrAISformer’s significant outperformance of existing methods on a real AIS case-study dataset.Source: §V · p. 10
  • Ablations indicate that all TrAISformer components contribute critically to reported performance, while intermediate attention weights offer a way to explore how past AIS data informs predictions.The authors suggest the learned transformer representation may also interest prediction tasks beyond this application.Source: §V · p. 10
  • Future work could incorporate weather and vessel interactions; the current architecture has about 300 times more parameters than the next most complex benchmarked model.The authors point to compression as a way to reduce model size without compromising performance and support operational assessment and adoption.Source: §V · p. 10

APPENDIX

The appendix contrasts medium-range vessel prediction with interaction-centered pedestrian and vehicle methods. It argues that forecasting future neighborhoods can be intractable when nearby agents move away or new, unknown agents appear.

APPENDIX

  • The prediction setup represents an agent’s observations as a sequence and denotes the other agents in its vicinity and their historical trajectories separately.For AIS prediction, the agent is a vessel and its observation is an AIS message.Source: APPENDIX · p. 11
  • At medium-range maritime horizons, a vessel’s path depends mainly on its intended destination and is barely affected by nearby vessels at the current moment.Pedestrian and vehicle methods model agent intention and environmental interactions, but their short prediction horizons and interaction focus do not suit medium-range AIS prediction.Source: APPENDIX · p. 11
  • Predicting future neighbors to model interactions can be expensive or intractable because agents may leave the vicinity and an unknown agent may enter it.Figure 9 illustrates this changing-neighborhood problem for the medium-range prediction of agent s1.Source: APPENDIX · p. 11, Figure 9 · p. 11
  • Removing the interaction term from the objective yields the objective used for medium-range AIS vessel prediction; removing interactions from Trajectron yields an LSTM seq2seq model.Source: APPENDIX · p. 11

Related papers

Loading 2109.03958v4…