Source-linked AI summary
MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses
Irtiza Hasan, Francesco Setti, Theodore Tsesmelis, Alessio Del Bue, Fabio Galasso, Marco Cristani
TL;DR
Trajectory forecasting has commonly relied on position histories, while MX-LSTM combines those tracklets with head-pose vislets to jointly predict positions and orientations. The paper reports state-of-the-art trajectory results across the UCY sequences and TownCentre, with a 32.7% average improvement over state-of-the-art methods in its reported comparisons.
Problem
Trajectory forecasting approaches use tracklets to predict future pedestrian positions, leaving head-pose information outside the prediction inputs.
Source: §1 · p. 1Method
MX-LSTM jointly processes position tracklets and head-pose vislets, mixing their streams with full-covariance Gaussian optimization and visibility-based social pooling.
Source: §1 · p. 1, §3.2 · p. 3, §3.4 · p. 4Results
32.7% average improvement over state-of-the-art methods, with MX-LSTM outperforming them in every sequence and on both metrics; it reports state-of-the-art results on UCY sequences and TownCentre.
Source: §1 · p. 2, §5.1 · p. 6Takeaways & Limitations
Jointly forecasting head orientations and trajectories adds long-term behavior-analysis capability in crowded scenarios.
Source: §1 · p. 2, §6 · p. 8The observation period feeds tracklets and vislets into MX-LSTM; experiments consider ground-truth past vislets and head poses estimated by a detector.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.
1. Introduction
The introduction argues that position-only forecasting leaves out visual information carried by head pose, then presents MX-LSTM as a joint model of positions and head orientations. It reports state-of-the-art trajectory forecasting on UCY sequences and TownCentre, including when pedestrians move slowly.
1. Introduction
- The paper motivates head pose as a missing visual cue because prior forecasting approaches use position coordinates alone, despite gaze and head pose conveying visual information.Prior findings cited by the paper relate head pose to a person's destination and pathway.Source: §1 · p. 1
- MX-LSTM combines position tracklets and head-orientation vislets in an LSTM, mixing their streams through cross-stream full covariance matrices optimized during backpropagation.The model also encodes how head movements and pedestrian dynamics are connected, including head turns that may precede an accelerating trajectory.Source: §1 · p. 1
- Its vislet-informed social pooling uses head pose to exclude people outside the individual's visible area.Source: §1 · p. 1
- Alongside trajectories, MX-LSTM predicts future head orientations, adding a longer-term account of where people may look in crowded scenes.Source: §1 · pp. 1–2
- MX-LSTM achieves state-of-the-art trajectory forecasting on the UCY sequences and TownCentre, including when pedestrians move slowly.The paper identifies slow-moving pedestrians as a case where competing approaches struggle.Source: §1 · p. 2
2. Related work
Related work moves from classical trajectory models toward approaches that represent pedestrian interactions and destinations. The paper distinguishes its approach through visibility-based social pooling and the use of higher-dimensional Gaussian covariance optimization.
2. Related work
- Classical forecasting used Kalman filters, regression, autoregressive models, and time-series analysis, while later work modeled human interactions and collision avoidance,,,.Other approaches encode interactions through social descriptors or human attributes,, or pooled LSTM hidden states.Source: §2 · p. 2
- Destination-focused methods use inverse optimal control, inverse reinforcement learning, or dynamic rewards, and may require a known endpoint or plausible path ends,,,.The paper says it does not use destination information, which it considers to undermine the purpose of predicting tracks.Source: §2 · p. 2
- Prior studies associate head pose with a person's destination and pathway, and this paper reports a statistically significant movement correlation even at low velocities.The reported correlation decreases as people slow down.Source: §2 · p. 2
- Social-pooling approaches share pedestrian hidden states to discourage collisions, while MX-LSTM restricts interactions to people within a head-pose-driven visibility area.Related LSTM methods also model neighboring pedestrians' local spatial and temporal dependencies.Source: §2 · p. 2
- Prior Gaussian log-likelihood forecasting generally estimates 2 × 2 covariances for position data, whereas this paper studies higher-dimensional Gaussian parameters.Optimized covariance matrices must remain positive semidefinite, a constraint the paper identifies as a key issue.Source: §2 · pp. 2–3
3. Our approach
MX-LSTM jointly forecasts pedestrian positions and head orientations from tracklet and vislet streams. It combines visibility-based social pooling with a full-covariance LSTM formulation.
3.1. Tracklets and vislets
- Tracklets are consecutive ground-plane (x, y) positions, while vislets are fixed-distance anchor points indicating the direction of the face.The anchor-point representation avoids the 360°/0° discontinuity of angles and is convenient for modeling its interplay with position.Source: §3.1 · p. 3
3.2. VFOA social pooling
- VFOA social pooling selects pedestrians within a head-pose-guided view frustum and organizes the surrounding space as an N_o × N_o grid.The frustum aperture and depth are learned by cross-validation on TownCentre’s training partition.Source: §3.2 · p. 3
3.3. LSTM recursion
- The LSTM predicts a four-dimensional Gaussian whose sampled output jointly forecasts tracklet positions and vislet head-pose points.Its covariance captures relationships among the coordinate distributions of both streams.Source: §3.3 · pp. 3–4
3.4. MX-LSTM optimization
- Log-Cholesky parameterization enables unconstrained optimization of full covariance matrices while enforcing positive semidefiniteness.The paper identifies positive-semidefinite covariance estimation as a numerical challenge, especially beyond two dimensions.Source: §3.4 · p. 4
4. Motivation for the MX-LSTM
Head orientation relates to movement but becomes less aligned at low speeds, where trajectory forecasting errors also rise.
4. Motivation for the MX-LSTM
- Prior work had not quantitatively studied how head-pose knowledge affects trajectory forecasting, motivating the paper’s joint analysis of head pose and movement.The analysis focuses on UCY and Town Centre, with head-pose annotations available or manually produced for those datasets.Source: §4 · pp. 4–6
- In 25% of UCY video sequences, head pose and step direction differ by more than 20°; observed discrepancies range from 0.02° to 72°.Sequences with speeds below 0.45 m/sec were omitted from this analysis because their movement vectors carry little meaning.Source: §4 · p. 5
- The overall head-pose and movement-angle correlation is 0.83 (p-value< 0.01), but it weakens at low velocities as their angular discrepancy grows.All reported velocity-specific correlations are statistically significant (p-value< 0.01).Source: §4 · p. 5
- At lower pedestrian speeds, forecasting errors generally increase, while MX-LSTM errors approach zero for static people.The paper attributes the difficulty of slow movement to both lower physical inertia and activities such as talking or looking around.Source: §4 · p. 5
5. Experiments
Experiments report improved trajectory forecasting, examine the roles of vislets and social pooling, and assess MX-LSTM’s head-pose forecasts and learned tracklet–vislet interplay.
5.1. Quantitative results
- Evaluation observes 8 ground-truth frames and predicts 12; UCY uses two sequences for training and the third for testing, while Town Centre uses its provided splits.The paper compares against publicly available implementations of Social Force, Linear Trajectory Avoidance, Vanilla LSTM, and Social LSTM.Source: §5.1 · p. 6
- With estimated head poses, MX-LSTM-HPE’s average performance drop is only 5% and it still beats competing methods on every dataset; it outperforms Social LSTM up to 24° noise.The estimated poses come from a standard head-pose estimator applied to head bounding boxes inferred from feet positions under a 1.80 m height assumption.Source: §5.1 · p. 6
- MX-LSTM outperforms state-of-the-art methods on every sequence with both displacement metrics, with an average improvement of 32.7%.The comparison covers Mean and Final Average Displacement errors across datasets and methods.Source: §5.1 · p. 6, Table 1 · p. 7
5.1.1 Ablation study
- Individual MX-LSTM is generally the weakest variant, while view-frustum social pooling improves social reasoning and full covariance estimation further reduces error.The ablations compare variants without cross-stream covariances, without a view frustum, and without social pooling; the full-covariance gain follows the error reduction already obtained with vislets.Source: §5.1.1 · pp. 6–7
5.1.2 Head pose forecasting
- MX-LSTM’s future head-pose forecasts are generally comparable to static HPE and outperform it on Zara01; estimated input poses increase angular error, but the increase is limited.The forecasts use no appearance cues, whereas the static estimator uses images.Source: §5.1.2 · p. 7, Table 2 · p. 7
5.2. Qualitative results
- In Zara02 examples, MX-LSTM predicts near-stationary oscillating poses for a conversing pedestrian, while individual-model trajectories shift with manipulated vislet direction.With artificial west-facing vislets, the group-aware model predicts departure from the group; the individual model’s predicted motion changes with whether the altered vislets align with movement.Source: §5.2 · pp. 7–8, Figure 3 · p. 8
6. Conclusion
The conclusion presents vislets as helpful for trajectory forecasting and MX-LSTM as a model that mixes head-pose and position streams. Its cross-stream covariances support accurate forecasting of both positions and head poses, while future work could encode areas of interest.
6. Conclusion
- MX-LSTM mixes tracklets and vislets through cross-stream 4 × 4 covariances, supporting accurate forecasting of both positions and head poses.The model optimizes these covariances during LSTM backpropagation using a log-Cholesky parameterization for unconstrained optimization.Source: §6 · p. 8
- Sequences of consecutive head poses, called vislets, are reported to help trajectory forecasting.Source: §6 · p. 8
- The authors suggest that future work could use vislets to encode specific areas of interest into trajectory forecasting.Source: §6 · p. 8