Source-linked AI summary
Unified Trajectory and 3D Pose Forecasting for Multi-Agent Sports Analysis
Marc Peral, Guillem Capellera, Luis Ferraz, Antonio Rubio, Stella Graßhof, Dan Witzner Hansen, Antonio Agudo
TL;DR
Forecasting body pose and global trajectory together is important because they are coupled, especially in team sports with many interacting players. This paper fuses both inputs in a shared socio-temporal encoder and adds loss-level refinements. It reports state-of-the-art results on WorldPose and better performance than compared methods on 3DPW, with qualitative evidence of graceful degradation on rare motions.
Problem
Trajectory and pose are often forecast separately, although they are coupled and both matter in team sports with many interacting players.
Source: §1 · p. 1Method
A shared socio-temporal encoder jointly processes input-fused trajectory and pose, with per-joint loss reweighting and frequency-domain trajectory smoothing.
Source: §1 · p. 2Results
Takeaways & Limitations
The joint formulation generalizes beyond densely interactive team-sport scenes, and qualitative forecasts degrade gracefully on rare motions.
Source: §1 · p. 2, §5 · p. 8Crossing-leg motion with an unusual gait may be underrepresented in training, though predictions remain reasonable on such examples.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Forecasting future human motion requires jointly reasoning about an agent's global trajectory and 3D body pose, which are tightly coupled, particularly in team sports where multiple interacting players introduce complex social dynamics. We exploit a socio-temporal encoder that fuses trajectory and pose at the input level and models them jointly through shared factorized attention. The model is further enhanced by a per-joint loss reweighting scheme that emphasizes dynamic joints and prevents degenerate solutions, and a frequency-domain trajectory smoothing term that improves trajectory prediction while indirectly benefiting pose forecasting. To ensure fair comparison with prior joint forecasting methods, we identify and correct evaluation biases arising from overlapping sequence construction, adopting instead a non-overlapping protocol and a deterministic, oracle-free strategy for multi-future prediction. Our approach improves over prior joint forecasting methods on both WorldPose and 3DPW, with the largest gains on pose accuracy, demonstrating that the proposed joint formulation generalizes beyond the densely interactive scenarios that motivate it.
1. Introduction
The paper jointly forecasts trajectory and 3D pose for interacting soccer players, where the two motions are coupled and many players respond to one another. It combines input-level fusion and shared socio-temporal attention with loss improvements, and evaluates against prior methods using a non-overlapping, oracle-free protocol.
1. Introduction
- Joint forecasting matters because a player's path and limb motion are coupled, while soccer involves twenty-plus players reacting to teammates, opponents, and play.Trajectory informs tactical analysis; pose informs technique and injury risk.Source: §1 · p. 1
- A shared socio-temporal transformer processes input-fused trajectory and pose tokens with factorized attention, rather than separate branches or sequential conditioning.Its output feeds two task-specific MLP heads.Source: Figure 1 · p. 1, §1 · p. 2
- Non-overlapping sequences avoid repeated near-identical samples and source-video-length weighting, while deterministic single-future evaluation avoids ground-truth oracle access at inference.The authors apply this protocol when re-evaluating T2P [30] and SimpliHumon [2].Source: §1 · p. 2
- The model achieves state-of-the-art results on WorldPose and outperforms competing methods on 3DPW, despite the latter's lower social density.The authors report gains on WorldPose across the majority of metrics and say the results generalize beyond the team-sport setting.Source: §1 · p. 2
- Per-joint reweighting selects target joints using measured per-joint error under uniform weighting, while frequency-based smoothing suppresses unrealistic path oscillations and also stabilizes pose forecasts.The paper reports targeted ablations validating the loss-level improvements.Source: §1 · p. 2
2. Related Work
Prior work has separately forecast trajectories and pose, with later methods addressing multi-person or joint prediction in different ways. The paper positions its contribution against methods that do not jointly forecast both outputs for the same interacting people or are not directly comparable on the target benchmarks.
2. Related Work
- Trajectory forecasting uses graph- and attention-based interaction models, including sports-specific work, while pose forecasting spans recurrent, feed-forward, transformer, and diffusion approaches.Frequency-domain pose methods use discrete cosine transforms to capture smooth, periodic motion.Source: §2 · p. 2
- Many pose methods target one person and short horizons; multi-person pose methods model agents or scene context but do not recover global trajectories.Some multi-person methods also rely on larger pose corpora than the target benchmarks provide.Source: §2 · pp. 2–3
- Joint forecasting approaches include shared multi-branch representations, sequential trajectory-to-pose pipelines, and structured pose models focused on temporal reasoning.Social-Transmotion [47] uses pose as input but forecasts only trajectories; Multi-Transmotion [20] forecasts pose and trajectory for a single ego agent while using other agents as context.Source: §2 · p. 3
- TRiPOD [1] is closely related but lacks an official implementation, while T2P [30] and SimpliHumon [2] are the primary baselines because they share the target tasks and benchmark setting.Their single-frame-stride sequence construction is revised to a non-overlapping protocol for comparison.Source: §2 · p. 3
3. Methodology
The method forecasts global player trajectories and root-relative 3D poses jointly from past observations, using early-fused inputs and a shared socio-temporal encoder. It trains the two prediction heads with trajectory, pose, and smoothness losses, including reweighting for harder-to-predict joints and frequency-based trajectory smoothing.
3.1. Problem Statement
- The model uses past observations of global 2D positions and root-relative 3D joint offsets to predict the remaining frames, without future context at inference.Trajectory and pose inputs retain their respective coordinate systems in prediction.Source: §3.1 · p. 3
3.2. Multi-Modal Input Encoding
- Trajectory and pose inputs are projected separately, fused into a shared representation, and can interact from the encoder’s first layers.Missing player slots may be padded with an out-of-range sentinel that the model learns to ignore.Source: §3.2 · pp. 3–4
3.3. Socio-Temporal Encoder
- Two factorized attention stages process the fused tokens: temporal blocks model each player’s history, and social blocks capture within-frame interactions across players.The agent dimension has no positional encoding, preserving permutation equivariance to player ordering.Source: §3.3 · p. 4
3.4. Prediction Heads
- Independent trajectory and pose heads map the shared representation to future pitch positions and root-relative 3D joint coordinates, respectively.Only forecast frames are retained as final output.Source: §3.4 · p. 4
3.5. Loss
- Training combines trajectory, pose, and trajectory-smoothness losses, while supervising both observed and forecast frames in a sequence-to-sequence manner.The loss weights control the relative contributions of the pose and smoothness terms.Source: §3.5 · p. 4
- Per-joint reweighting emphasizes elbows, wrists, ankles, and feet, whose errors are higher under uniform weighting, rather than letting easier proximal joints dominate.The stated aim is to encourage fine-grained articulation and discourage collapse toward static average poses.Source: §3.5 · pp. 4–5
- The smoothness loss penalizes frequency-weighted energy in predicted trajectories, suppressing zigzags while largely preserving plausible motion and providing steadier pose context.The Fourier term weights each coefficient by the square of its normalized frequency; the static component receives zero weight.Source: §3.5 · p. 5
4. Experimental Results
The experiments assess joint trajectory and pose forecasting on WorldPose and 3DPW, using non-overlapping windows and ablations of cross-modal inputs, fusion, and loss terms. The model is competitive on WorldPose and best across metrics on 3DPW, while qualitative examples show plausible forecasts on difficult actions.
4.1. Datasets, Metrics and Implementation details
- WorldPose uses disjoint 3-second windows and match-level splits; 3DPW uses 90-frame windows and per-video splits, with non-overlapping sampling and one-second context before two-second forecasts.The evaluation reports ADE/FDE, APE/PPE, and world-coordinate JPE, which combines trajectory and pose but is numerically dominated by global position.Source: §4.1 · pp. 5–6
4.2. Ablation Studies
- Adding trajectory input improves pose-only forecasting, while adding pose input helps trajectory-only forecasting more; joint training further improves over either single-task model.The authors attribute the stronger pose-to-trajectory effect to limb articulation cues about imminent movement.Source: §4.2 · p. 6, Table 1 · p. 6
- Input-level fusion by concatenation outperforms sequential and late-fusion alternatives, with separation concentrated in trajectory-coupled metrics and roughly half the parameters of full late fusion.Concatenation slightly outperforms summation and is used in the final model.Source: §4.2 · p. 6, Table 2 · p. 6
- Increasing joint rebalancing improves pose metrics but degrades trajectory accuracy; factor 5 is selected near the point where further pose gains become disproportionately costly.JPE is minimized at factor 5, though the authors treat this as a consistency check because JPE is dominated by global position.Source: Table 3 · p. 7, §4.2 · p. 7
- Trajectory smoothing improves trajectory accuracy and also improves pose metrics; predicted paths are visibly steadier with smoothing enabled.The authors attribute the pose benefit to a steadier trajectory context and illustrate the path change in Figure 2.Source: §4.2 · p. 7, Table 3 · p. 7, Figure 2 · p. 8
4.3. Quantitative Results
- On 3DPW, our model is best on every metric; on WorldPose, it is competitive with T2P on ADE and JPE and improves all other metrics.Baselines were rerun with non-overlapping windows and single-future or central-of-six selection without access to ground truth.Source: §4.3 · p. 7, Table 4 · p. 8
4.4. Qualitative Evaluation
- Forecasts remain plausible for relaxed walking, abrupt sprints without observed cues, and an unusual crossing-leg gait considered potentially underrepresented.In the walking examples, residual error is concentrated in body orientation at the furthest frames rather than limb configuration.Source: §4.4 · p. 8
5. Conclusion
The unified model jointly forecasts trajectory and 3D pose using input-level fusion, with loss reweighting and trajectory smoothing further improving forecasts. Under a non-overlapping, oracle-free evaluation protocol, it achieves state-of-the-art results on WorldPose and outperforms competing methods on 3DPW.
5. Conclusion
- The model achieves state-of-the-art results on WorldPose [32] and outperforms competing methods on 3DPW under a non-overlapping, oracle-free evaluation protocol.Source: §5 · p. 8
- A shared socio-temporal encoder fuses trajectory and 3D pose at the input and learns their interaction directly.Source: §5 · p. 8
- Per-joint reweighting improves pose accuracy, while frequency-domain trajectory smoothing improves trajectory accuracy and stabilizes pose forecasts.Qualitative results also show plausible forecasts under abrupt pace changes and graceful degradation on rare motions.Source: §5 · p. 8
A. Joint Balancing Strategy
Joint rebalancing targets body joints with consistently above-average error under uniform weighting. By emphasizing these joints, it reduces their errors and discourages a static mean-pose solution.
A. Joint Balancing Strategy
- The target joints are selected by measuring per-joint error under uniform weighting and choosing those with consistently above-average error.Source: A. Joint Balancing Strategy · p. 11
- Rebalancing reduces errors concentrated in the targeted distal joints: elbows, wrists, ankles, and feet.These joints have markedly higher error than proximal joints under uniform weighting.Source: A. Joint Balancing Strategy · p. 11
- Reweighting steers optimization toward informative joints, preventing collapse toward a near-motionless mean pose and encouraging fine-grained articulation.Source: A. Joint Balancing Strategy · p. 11
B. Trajectory Smoothing Exploration
The adopted frequency-domain penalty suppresses high-frequency trajectory oscillations across the forecast horizon. The section contrasts it with displacement representations and a local time-domain penalty, which each have stated drawbacks.
B. Trajectory Smoothing Exploration
- The model predicts absolute positions independently at each frame, leaving consecutive predictions without an explicit temporal-consistency constraint.Source: B. Trajectory Smoothing Exploration · p. 11
- Predicting all positions as displacements degraded results because it removed absolute scene context used by social attention to model player interactions.Source: B. Trajectory Smoothing Exploration · p. 11
- The targeted distal joints’ error visibly drops after rebalancing in the figure’s directly comparable panels.Both panels use a shared 0–10 mean-absolute-error scale.Source: Figure A1 · p. 11
- Keeping observed positions absolute and forecasting displacements reduced jitter and early-frame displacement error, but increased general ADE and especially FDE as errors accumulated.This representation also fixed the observation–forecast boundary at a specific frame index, limiting flexibility across horizons.Source: B. Trajectory Smoothing Exploration · p. 11
- The time-domain second-order-difference penalty misses oscillations spanning more than two frames, the dominant artifact in practice.Source: B. Trajectory Smoothing Exploration · p. 11
- The frequency-domain formulation penalizes the full-sequence power spectrum weighted by frequency, capturing systematic high-frequency oscillations at any temporal scale within the forecasting horizon.Source: B. Trajectory Smoothing Exploration · p. 11