Pedestrian Trajectory Prediction with Convolutional Neural Networks

1. Introduction

• The field has shifted from physics-based approaches such as Social Forces Model and BRVO toward data-driven methods, including Social LSTM.Source: §1 · p. 2
• Normalization and augmentation effectiveness remain underexplored, and a detailed comparison of CNNs with recurrent models in pedestrian prediction is missing.Source: §1 · pp. 2–3
• The work identifies effective position normalization and augmentation techniques, including Gaussian noise and random rotations.Source: §1 · p. 3
• The proposed 2D convolutional model achieves state-of-the-art results on the ETH and TrajNet datasets.The introduction identifies this as a main contribution.Source: §1 · p. 3
• Experiments on social occupancy information empirically indicate that occupancy methods are ineffective at representing social information.Source: §1 · p. 3

2. Related Work

• Physics-based models such as Social Forces Model [1] and BRVO use hand-crafted functions, while deep-learning approaches rely mainly on RNNs such as Social LSTM.Source: §2 · p. 4
• Attention has been incorporated into recurrent pedestrian-prediction models, whose recurrent nature makes them more computationally expensive than convolutional models.Source: §2 · pp. 4–5
• Generative models, including GANs and conditional variational auto-encoders, address multimodal prediction and are commonly evaluated with best-of-N samples.Source: §2 · p. 5
• Graph neural networks represent pedestrians as nodes and their interactions as edges, with later work also modeling spatial interactions.Source: §2 · p. 5
• Other approaches use points of interest, occupancy maps, scene segmentation, or video imagery, but such spatial or image information may require additional infrastructure or environmental knowledge.Source: §2 · p. 5
• Pedestrian prediction literature has not extensively explored convolutional models or preprocessing techniques such as normalization and augmentation.Source: §2 · p. 6

3.1. Problem formulation

• Prediction is defined as estimating each pedestrian’s future coordinates from observed positions at evenly spaced time steps.Positions are measured in meters relative to a fixed point selected separately for each scene.Source: §3.1 · p. 6

3.2. Data pre-processing

• Preprocessing normalizes input coordinates and applies augmentation to support model training and reduce prediction error.The methods considered include alternative coordinate systems and trajectory transformations.Source: §3.2 · p. 7, §3.2.1 · p. 7

3.2.1. Data normalization

• Four coordinate representations are considered: absolute, origin at the first observation, origin at the last observation, and relative displacements.Relative displacements represent instantaneous velocities when scaled to the annotation time interval.Source: §3.2.1 · p. 7

3.2.2. Data Augmentation

• Augmentation applies random rotations, probabilistic axis mirroring, or zero-mean Gaussian noise to trajectory points.Rotation is intended to encourage rotation-invariant patterns, while noise is intended to improve robustness to small perturbations and imprecisions.Source: §3.2.2 · p. 8

3.3. Convolutional Model

• The 2D convolutional architecture embeds eight observed positions, transforms the resulting feature matrix to twelve future positions, and predicts them in one pass.Its convolutional layers process multiple features over multiple timesteps; the architecture is scalable in the number of initial and final convolutional layers.Source: §3.3 · pp. 9–11
• The described convolutional variants include a 1D model treating the post-embedding 64×8 matrix as 64 one-dimensional channels with 8 features each, plus positional embeddings, transpose convolutions, and residual connections.Transpose convolutions replace the upsampling layer followed by convolutions without padding. Positional information provides an order clue, and all convolutional layers are transformed into residual convolutional layers.Source: §3.3 · pp. 10–11

3.4. Recurrent baselines

• The recurrent comparisons use a position-embedding LSTM that predicts stepwise and an Encoder-Decoder with LSTM cells in both encoder and decoder.The Encoder-Decoder encoder processes observed inputs, while its decoder follows the LSTM baseline architecture.Source: §3.4 · pp. 11, 13

3.5. Addition of social information

• Three occupancy representations are explored: square grids, circular maps, and angular grids that record the nearest pedestrian by direction.Social features are embedded at each observed timestep and summed with position features; social information is available only during observation.Source: §3.5 · pp. 13–14

4.1. Data

• ETH-UCY combines five scenes and more than 1,600 trajectories, with scene-wise leave-one-out evaluation training on four scenes and testing on the fifth.Positions are annotated every 0.4 seconds.Source: §4.1 · p. 14
• TrajNet contains more than 8,000 trajectories from ETH, UCY, Stanford Drone, and PETS2009, with test predictions evaluated through its official site.The dataset is pre-split into training and test sets, and test inputs provide only observed positions.Source: §4.1 · p. 15

4.2. Metrics

• ADE averages Euclidean distance over predicted points and pedestrians, while FDE averages Euclidean distance at the final prediction time.The experiments use eight observed and twelve predicted time steps, a setting described as common for comparison with other models.Source: §4.2 · pp. 15–16

4.3. Implementation details

• ETH-UCY models trained for 60 epochs and TrajNet models for 250, using Adam, ADE loss, and a 0.005 learning rate.For mirroring, samples had a 25% chance of reflection on one axis and a 50% chance of no mirroring.Source: §4.3 · p. 16

4.4. Results data pre-processing

• The last-observation origin produced the lowest ADE across all five scenes for both the LSTM baseline and the 1D convolutional model.The table reports coordinate-normalization results in ADE/FDE format.Source: §4.4 · p. 17, Table 1 · p. 17
• Noise-plus-rotation achieved the lowest average error, while mirroring was ineffective alone and added no improvement alongside other augmentations.Some scenes had higher error with the combined noise-and-rotation variant than with noise or rotation alone.Source: §4.4 · pp. 17–18, Table 2 · p. 18
• The preprocessing findings also held for the Encoder-Decoder and other convolutional variations, and the augmented LSTM outperformed the kernel-size-3 1D convolutional model.The latter comparison did not hold for other convolutional models.Source: §4.4 · p. 18

4.5. Results convolutional model variations and baselines

• The 1D convolutional model with kernel size 7 performs better than the same model with kernel size 3, while the 2D model with kernel size 7 is slightly worse than with size 5.The authors suggest larger kernels process more trajectory information, but this benefit has diminishing returns in the 2D model.Source: §4.5 · p. 19
• Positional embeddings and transpose convolutions are ineffective, and residual connections do not improve results with an optimal depth of seven convolutional layers.The passage points to Figure 3 for the seven-layer result.Source: §4.5 · p. 19
• The 2D convolutional model with kernel size 5 is the best model; 1D convolution outperforms recurrent baselines only with a bigger kernel, and the average-error difference is not ample.Table 3 compares convolutional variants and baselines.Source: §4.5 · p. 19, Table 3 · p. 20

4.6. Results using social information

• Occupancy-based social information does not improve the 2D convolutional model on ETH-UCY, and the same approaches do not outperform Encoder-Decoder models without social information.Table 4 presents the 2D convolutional models with social information.Source: §4.6 · pp. 20–21, Table 4 · p. 21
• Social-information embedding weights receive average gradient flow around 50–100 times smaller than position-embedding weights, which may indicate that the network largely ignores social information.Figure 6 illustrates gradient flow; its axes show network layers and mean and maximum gradients on a logarithmic scale.Source: §4.6 · pp. 20–21, Figure 6 · p. 21
• The authors characterize these occupancy experiments as exploratory and limit the conclusion to the tested methods and architectures on ETH-UCY.They note that more work has been done to incorporate social information into trajectory prediction.Source: §4.6 · pp. 20–21

4.7. Comparison with literature on the ETH-UCY dataset

• The 2D convolutional model has the lowest error across ETH and average ETH-UCY error comparable to STGAT and STGCNN, but Trajectron++ and other models surpass it on UCY.The authors suggest the higher pedestrian density in UCY, where social interaction is not modeled by their 2D convolutional model, might explain the difference.Source: §4.7 · pp. 23–24, Table 5 · p. 23
• The authors suggest that denser UCY scenes may make social interaction more important, a factor their 2D convolutional model does not take into account.This is presented as a possible explanation for other models surpassing it on UCY.Source: §4.7 · p. 23
• The recurrent baselines also achieve very low error; the authors attribute the low error of their LSTM-tobs-NR relative to the LSTM trained by to the employed data-preprocessing techniques.The comparison includes physics-based, recurrent, convolutional, and generative models; generative results marked with an asterisk use best-of-N with N=20.Source: §4.7 · pp. 23–24

4.8. Comparison with literature on the TrajNet dataset

• The comparison with RED distinguishes the recurrent LSTM-tobs-NR from the non-recurrent convolutional model, and contrasts their last-observation-origin coordinates with RED's relative coordinates.The passage says RED uses trajectory reversal as augmentation and that the proposed approach also applies random rotations and noise, but its comparison sentence is incomplete.Source: §4.8 · p. 24
• The proposed augmentations can transform a single trajectory in virtually infinite ways, increasing training-data diversity and leading to reduced error.Source: §4.8 · p. 25
• For TrajNet (Table 6), LSTM-tobs-NR is listed with 0.356 and 1.212, Conv2D-tobs-NR-Ks5 + Apg with 0.366 and 1.223, and RED (v3) [42] with 0.360 and 1.201.Source: §4.8 · p. 25

• The 2D convolutional model achieves state-of-the-art performance on TrajNet, with the lowest ADE on the largest publicly available pedestrian-prediction dataset.Table 6 compares models on TrajNet.Source: Conv2D-tobs-NR-Ks5 † 0.352 1.192 · p. 25, Table 6
• LSTM-tobs-NR achieves lower error than RED on TrajNet, which the authors attribute to the proposed preprocessing techniques.Source: Conv2D-tobs-NR-Ks5 † 0.352 1.192 · p. 25
• On TrajNet, models with social information underperform their counterparts without it; occupancy methods are reported as ineffective for representing social interaction.Circular occupancy-map results are omitted from Table 6 because they are very similar to square-grid results.Source: Conv2D-tobs-NR-Ks5 † 0.352 1.192 · pp. 25–26

5.1. Convolutional model and recurrent models comparison

• The convolutional model is more efficient and accurate than recurrent baselines, but is less flexible and requires more hyperparameter tuning.Table 7 compares computational test time across models.Source: Table 7 · p. 26, §5.1 · p. 27
• The convolutional model has more hyperparameters to tune than the recurrent baselines, including layer count, channel counts, and per-layer kernel sizes.The recurrent models use embedding size, hidden-state length, and output-layer dimensions.Source: §5.1 · p. 27
• Recurrent models can handle different input or output lengths without architectural changes; convolutional models need layer adjustments, and variable-length inputs remain an open challenge.Source: §5.1 · p. 27

5.2. Failure cases

• Sharp turns are a recurring high-error case: without spatial information, the model cannot reliably anticipate an unobserved turn, but can adapt quickly after direction changes become visible.In the example, predictions improve after additional observed frames.Source: §5.2 · pp. 27–29
• Pedestrian stops are difficult to predict because their reasons can be unclear; spatial information could help in some, but not all, cases.Source: §5.2 · pp. 27–28
• Resuming movement after a stop is even harder to predict: while a person is still, the moment they will start walking is difficult to determine.Assuming the person remains still can lead to very high error if they resume walking.Source: §5.2 · pp. 27–28
• The ADE distribution resembles a Gaussian curve with a long tail; its average is 0.456, standard deviation 0.370, and maximum 2.92.Figure 7 shows the distribution for the 2D convolutional model on Zara1.Source: §5.2 · pp. 27–28, Figure 7 · p. 28
• The authors suggest that adding spatial information could reduce instances of very high error, and identify it as a promising direction for future work.Source: §5.2 · p. 29

6. Conclusion

• Using the last observation as coordinate origin with Gaussian noise and random rotations is effective across convolutional and recurrent architectures.The authors present these preprocessing findings as generalizable across architectures.Source: §6 · p. 30
• The 2D convolutional model outperforms recurrent baselines in average error and computational time, achieving state-of-the-art results on ETH and TrajNet.The comparison also notes that convolutional models can be less flexible and require more hyperparameter tuning.Source: §6 · p. 30
• Social occupancy information does not reduce prediction error in the reported exploratory experiments.The authors identify more advanced, convolution-specific techniques as a future direction for incorporating social information.Source: §6 · p. 30
• High prediction errors most commonly occur when pedestrians make sharp turns, stop, or resume walking after stopping.The authors suggest spatial input as a future direction for addressing turning pedestrians.Source: §6 · p. 30
• The authors identify TrajNet++ as a relevant dataset for evaluating future work on social interaction and collisions.They also propose more advanced social-information methods designed specifically for convolutional architectures.Source: §6 · p. 30