PaperlayerView exact revision on arXiv

Source-linked AI summary

Action Sequence Predictions of Vehicles in Urban Environments using Map and Social Context

Jan-Nico Zaech, Dengxin Dai, Alexander Liniger, Luc Van Gool

arXiv:2004.14251v1cs.CVcs.RO

TL;DR

The paper addresses prediction of future actions for surround vehicles, motivated by the need for interpretable maneuver anticipation and limited public behavioral data. It automatically derives action sequences from trajectories and HD maps, builds an Argoverse dataset, and trains an end-to-end network using agent, map, and social information. The authors report improved prediction performance with more data and that their action-prediction model outperforms adapted trajectory-prediction methods.

  • Problem

    Predicting surround vehicles’ future actions requires interpretable maneuver representations, but public datasets for modeling driving behavior in complex environments are limited.

    Source: §I · p. 1, §IV · p. 3

  • Method

    The paper automatically extracts action sequences from trajectories with HD maps and predicts them using an end-to-end network combining agent, map, and social information.

    Source: §I · p. 1, §III · p. 2, B. Input and Output Representation · p. 2

  • Results

    The action-prediction model, together with its dataset, significantly outperforms existing methods adapted from trajectory prediction.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The dataset and method support quantitative evaluation of high-level vehicle-action prediction from large-scale real-driving data.

    Source: §I · p. 1, §VII · p. 8

Abstract

from arXiv · show

This work studies the problem of predicting the sequence of future actions for surround vehicles in real-world driving scenarios. To this aim, we make three main contributions. The first contribution is an automatic method to convert the trajectories recorded in real-world driving scenarios to action sequences with the help of HD maps. The method enables automatic dataset creation for this task from large-scale driving data. Our second contribution lies in applying the method to the well-known traffic agent tracking and prediction dataset Argoverse, resulting in 228,000 action sequences. Additionally, 2,245 action sequences were manually annotated for testing. The third contribution is to propose a novel action sequence prediction method by integrating past positions and velocities of the traffic agents, map information and social context into a single end-to-end trainable neural network. Our experiments prove the merit of the data creation method and the value of the created dataset - prediction performance improves consistently with the size of the dataset and shows that our action prediction method outperforms comparing models.

I. INTRODUCTION

The paper reframes surround-vehicle prediction as forecasting interpretable high-level action sequences rather than only trajectories, addressing the need to anticipate maneuvers for planning. It introduces automatic action-sequence dataset creation from trajectories and maps, then situates the approach among intention, maneuver, and trajectory-prediction methods.

I. INTRODUCTION

  • Predicting high-level actions of surround vehicles can improve planning and interpretability beyond directly forecasting trajectories.The motivation emphasizes understanding static and dynamic environments, traffic rules, and interactions among road users.Source: §I · p. 1
  • The proposed task converts real-world trajectories into action sequences automatically using HD maps, avoiding extensive manual annotation.The method is applied to Argoverse to create a large-scale action-prediction dataset.Source: §I · p. 1
  • The resulting dataset contains 228,000 action sequences across five actions and 2,245 manually annotated sequences for testing.The action classes are cruise, turn left, turn right, lane change left, and lane change right.Source: §I · p. 1
  • An end-to-end neural network combines past agent positions and velocities, map information, and social context to output probabilistic future actions.The inputs are rendered into multichannel images and processed with 2D and 3D convolutional modules to represent uncertain, multimodal futures.Source: §I · p. 1

II. RELATED WORK

  • Prior work includes driver-action prediction, maneuver recognition paired with motion models, direct trajectory forecasting, and interaction-aware pedestrian prediction.These approaches use sensors, maps, recurrent or convolutional networks, probabilistic models, and explicit social context across highway, intersection, and pedestrian settings.Source: §II · p. 2

III. METHOD

The method uses an entity-centric, fully convolutional network to predict one traffic agent’s future action sequence from rendered agent, map, and social observations. It combines spatial processing with early and late temporal fusion and predicts a three-second sequence probabilistically in one forward pass.

III. METHOD

  • The network predicts one traffic agent’s interpretable high-level action sequence while using other agents as social context.Its fully convolutional design uses 2D convolutions for spatial features and 3D layers for temporal fusion, with a single forward pass producing the sequence.Source: §III · p. 2

A. Network Architecture

  • A VGG-inspired architecture performs early and late fusion of temporal features using three-dimensional convolutions for action sequence prediction.To avoid overfitting on the training data, the model is smaller than some fully convolutional trajectory prediction approaches.Source: A. Network Architecture · p. 2

B. Input and Output Representation

  • Each observation combines target-agent position and velocity, lane-centerline map information, and jointly rendered representations of other agents.The components are rendered as 128 × 128 spatial tensors covering a 50 m × 50 m region centered and oriented using the target agent.Source: B. Input and Output Representation · pp. 2–3
  • The model receives the last five observations, distributed over the preceding two seconds, and uses random rotations from −5° to 5° for augmentation.The rendered inputs are stored parametrically and generated on the fly to avoid augmentation artifacts.Source: B. Input and Output Representation · p. 3
  • The output covers the next 3 s as 30 steps sampled every 100 ms, with an independent probability distribution over five action classes at each step.Temporal relations and dependencies among actions must be learned from training data.Source: B. Input and Output Representation · p. 3

C. Probabilistic Action Predictions

The action-based approach returns probabilistic predictions directly, making outputs interpretable while allowing evaluation to focus on complex traffic scenarios despite class imbalance.

IV. DATASET

The dataset augments Argoverse trajectories with human-interpretable actions and velocity information, using HD maps as a scalable basis for automatic annotation.

A. Argoverse

  • Argoverse provides approximately 325,000 urban trajectories, including about 245,000 five-second segments usable for action prediction.Source: A. Argoverse · p. 3
  • The work augments Argoverse with automatically annotated lane changes and turns, plus extracted agent velocities for prediction.Source: A. Argoverse · p. 3

B. Map Information

  • HD maps convert roughly 245,000 trajectories into about 7.5M action annotations, avoiding time- and labor-intensive manual labeling.Source: B. Map Information · p. 3
  • Map-based annotation is considered scalable because lane-centerline semantics remain constant over time and local map extraction may be automated with lane detection.Source: B. Map Information · p. 3
  • The semantic map is represented as a graph whose nodes encode lane-segment geometry and semantics, while edges encode successor, predecessor, and neighboring-lane relations.Source: B. Map Information · p. 4, Figure 3 · p. 4

C. Automatic Action Labeling

Automatic action labeling smooths trajectories, assigns them to semantically connected lane segments, and extracts maneuvers from the resulting map-consistent path.

  • Trajectory labeling begins with bidirectional Kalman filtering, which smooths noisy positions and estimates agent velocity.Source: C. Automatic Action Labeling · p. 4
  • Lane assignment uses a joint geometric and semantic Viterbi algorithm so trajectories follow valid graph-map paths rather than arbitrary nearest lanes.Source: C. Automatic Action Labeling · p. 4
  • The graph-map is modeled as an HMM in which lane segments are hidden states and observed 2D positions are emissions; Viterbi inference recovers the most likely action sequence.Source: C. Automatic Action Labeling · p. 4
  • Only lane segments within 5 m are considered, while transition costs encode successor, predecessor, neighbor, and otherwise unconnected-lane relations using α=0.001.Source: C. Automatic Action Labeling · p. 4
  • A rule-based stage labels lane changes from neighbor-lane transitions and turns from lane annotations, extending maneuver labels across their full non-singular duration.Source: C. Automatic Action Labeling · p. 4
  • Corner cases such as directly changing to a successor lane’s neighbor require explicit modeling and handling.Source: C. Automatic Action Labeling · p. 4

D. Test Data and Ordered Action Sequences

Testing uses manually annotated ordered action sequences, while dataset statistics account for the different temporal structure and reveal an imbalanced action distribution.

D. Test Data and Ordered Action Sequences

  • The test set contains 2,245 manually annotated trajectories represented as ordered actions rather than time-indexed action sequences.Ordered sequences preserve maneuver order but discard the exact temporal location of each action.Source: D. Test Data and Ordered Action… · p. 5
  • Ordered action annotation is simpler and less error-prone, and the 2,245 test trajectories cover a 3 s prediction horizon without annotator access to automatic labels.Source: D. Test Data and Ordered Action… · p. 5

E. Dataset Statistics

  • Automatically extracted action data is imbalanced toward cruise because active maneuvers are less frequent and are commonly followed by cruise stabilization.Source: E. Dataset Statistics · p. 5
  • Statistics for ordered sequences estimate class proportions by assigning each sequence’s weight across its annotated states rather than counting every temporal occurrence.Source: E. Dataset Statistics · p. 5

V. EXPERIMENTS

The experiments define training and baseline procedures, then evaluate action prediction with class-wise average precision on an automatically annotated validation set.

A. Training

  • Training uses Adam with an initial learning rate of 10−4, step decay, dropout 0.5, and 50 epochs for all network variants.The learning rate decays every 10 epochs by a factor of 0.5.Source: A. Training · p. 5
  • Weighted random sampling increases emphasis on samples containing turns or lane changes, but does not fully compensate for dataset imbalance.Turn-containing samples receive weight 3 and lane-change samples weight 10.Source: A. Training · p. 5
  • Input positions and velocities are extracted with a bidirectional Kalman filter applied only to observed values, preserving causality.The filtered trajectories provide the target and social motion inputs.Source: A. Training · p. 5

B. Baseline Model

  • The adapted k-NN baseline averages class labels from nearest-neighbor action sequences and compares settings k ∈{9, 50, 100}.The best-performing setting is selected for further comparison.Source: B. Baseline Model · p. 5

1) Direct Evaluation of Predictions:

  • Mean AP across five action classes evaluates both correct action labels and correct transition timing under severe class imbalance.Each action class is treated as a binary classification problem, and the unweighted average AP is reported.Source: 1) Direct Evaluation of Predictions · p. 5
  • Table II reports AP scores for the proposed method, k-NN baseline, and random sampling on the automatically annotated validation set.All reported scores are percentages, with random sampling corresponding to class proportions.Source: 1) Direct Evaluation of Predictions · p. 5, Table II · p. 6

2) N-Most Likely Ordered Action Sequences:

The method extracts the most likely ordered action sequences from per-timestep probabilities while accounting for temporal correlation and limiting the search to short sequences.

2) N-Most Likely Ordered Action Sequences:

  • Because only 10 test samples, or 0.45%, contain more than two actions, extraction considers sequences with at most two actions.This restriction makes N-most-likely sequence extraction tractable.Source: 2) N-Most Likely Ordered Action… · p. 6
  • The extraction corrects the independent-action assumption because multiplying per-timestep probabilities neglects strong temporal correlation in complete sequences.The model is independent at each timestep, but full-sequence probability requires a block-based treatment.Source: 2) N-Most Likely Ordered Action… · p. 6
  • Table III reports accuracies for ordered sequences ranked by frequency and containing at least 10 samples.The evaluation targets order rather than exact action durations or transition times.Source: Table III · p. 6, 2) N-Most Likely Ordered Action… · p. 6
  • Ordered sequences are extracted by maximizing two-action block probabilities over action pairs and their transition time.Within each block, the lowest predicted probability determines the block probability, while inter-block transitions use a shared probability assumption.Source: 2) N-Most Likely Ordered Action… · p. 6
  • Repeatedly suppressing previously extracted action pairs yields the N-most-likely ordered sequences, evaluated with top-N accuracy for N ∈{1, 2, 3}.A ground-truth sequence is detected when it appears among the top-N predictions; sequences longer than two actions are counted as incorrect.Source: 2) N-Most Likely Ordered Action… · p. 6

VI. RESULTS

Direct evaluation finds that cruise and turns are easier to predict than lane changes, with the proposed method showing its largest relative gains on lane-change classes.

A. Direct Evaluation

  • The proposed method’s largest relative improvement over the baseline occurs for the two lane-change action classes.Cruise AP is highest, turn AP follows, and lane-change AP is substantially lower across methods.Source: A. Direct Evaluation · p. 6
  • Turns are easier to detect than lane changes because their trajectories have more distinctive shapes despite both maneuvers being underrepresented in training data.The explanation combines dataset statistics with trajectory geometry.Source: A. Direct Evaluation · p. 6
  • The top-1 confusion matrix uses predicted classes on the x-axis, ground-truth classes on the y-axis, and row-normalized percentages.Rows sum to 100%.Source: Figure 4 · p. 7

B. N-Most Likely Ordered Sequences

Ordered-sequence results show gains from multimodal top-N prediction, while ablations find that jointly using target, social, and map information is most effective, especially because map input has the largest impact.

B. N-Most Likely Ordered Sequences

  • The proposed model predicts a more diverse set of ordered action sequences than the baseline, with strong Top2-accuracy improvements for sequences such as (c, ll) and (c, lr).Some maneuver-containing sequences already achieve relatively high Top1-accuracy, while others benefit mainly from the second prediction.Source: B. N-Most Likely Ordered Sequences · p. 7
  • The Top2-accuracy improvements support multimodal prediction because alternative ordered sequences capture uncertainty about future maneuvers.The result is reported for manually annotated ordered sequences.Source: B. N-Most Likely Ordered Sequences · p. 7
  • Top-2 prediction corrects many top-1 errors caused by assigning an incorrect second action, producing substantially higher confusion-matrix diagonal values.This is especially relevant when the true sequence includes a continuation such as (tl, c) rather than the shorter (tl).Source: B. N-Most Likely Ordered Sequences · p. 7, Figure 5 · p. 7

C. Ablation Study

  • Ablations show that jointly providing target-agent, social, and map information is useful, with map information having the largest impact.The ablation compares mean AP on the validation dataset.Source: C. Ablation Study · p. 7, Table IV · p. 7
  • Adding social information helps more without a map than with one, indicating redundancy between social and map context available to the network.The study also evaluates rotation-based data augmentation as an additional component.Source: C. Ablation Study · p. 7

D. Dataset Scale study

The dataset-scale study finds that more action-sequence data substantially improves prediction, while the conclusion reports that the proposed action model and dataset outperform adapted trajectory-prediction methods.

D. Dataset Scale study

  • Performance differs substantially across dataset scales, showing that data amount has a major impact on action prediction.The study compares subsets containing 50%, 10%, and 5% of the full action-sequence dataset.Source: D. Dataset Scale study · pp. 7–8
  • Performance continues improving from 50% of the data to the full dataset, indicating that results had not saturated at Argoverse scale.The authors suggest that larger datasets could further benefit action prediction.Source: D. Dataset Scale study · p. 8

VII. CONCLUSION AND FUTURE WORK

  • The action prediction model and dataset significantly outperform existing methods adapted from trajectory prediction.The conclusion states that action prediction is not solved by trajectory prediction alone.Source: §VII · p. 8
  • Future work will investigate combining action prediction with trajectory prediction to obtain the benefits of both approaches.Source: §VII · p. 8

Related papers

Loading 2004.14251v1…