PaperlayerView exact revision on arXiv

Source-linked AI summary

Field Converter: Geometry-Initialized Temporal Residual Refinement for World-Grounded Player Pose Estimation from Soccer Broadcasts

Simon Khan, Laurent Gajny, Jennyfer Lecompte, Sébastien Laporte

arXiv:2609.10498v1cs.CV

TL;DR

Monocular soccer broadcasts make it difficult to recover player pose in a shared metric world frame rather than only relative to each body. Field Converter initializes the player root from calibrated camera and pitch geometry, then learns a temporal residual correction. Residual refinement lowers root error from 49 cm with geometry alone to 10 cm with a TCN, while airborne motion remains the main limitation.

  • Problem

    Shared metric world localization remains difficult even when monocular methods estimate accurate body-relative pose from broadcast video.

  • Method

    Field Converter uses ray–ground intersection for metric root initialization and predicts only its temporal residual from pose, image, camera, and geometric cues.

  • Results

    10 cm root error is achieved with the TCN, compared with 49 cm from geometry alone; TCN and Transformer obtain comparable World MPJPE.

  • Takeaways & Limitations

    Residual prediction is more effective than direct global-root regression, while temporal context matters more than the specific temporal backbone.

  • Takeaways & Limitations

    The ground-based initialization assumes a lower-limb joint is near the playing surface, causing larger localization errors during airborne actions.

Abstract

from arXiv · show

Recovering 3D human pose from monocular sports broadcasts remains challenging when players must be localized in a shared metric world coordinate system rather than only reconstructed relative to their own body. We introduce Field Converter, a geometry-initialized temporal residual framework for world-grounded 3D player pose estimation from calibrated soccer broadcasts. Our method first uses camera and pitch geometry to initialize the player root through ray-ground intersection, then predicts a temporal residual correction from pose, image, camera, and geometric cues. On match-disjoint evaluation sequences, residual refinement reduces root error from 49cm with geometry alone to 14cm with a frame-wise MLP and 10cm with a TCN, while a Transformer achieves a comparable 11cm. The resulting world-space MPJPE reaches 13.2cm, and ablations show that residual prediction clearly outperforms direct global-root regression while temporal context matters more than the specific temporal backbone. Failure analysis further identifies airborne motion as the main limitation of the ground-based geometric initialization.

1. Introduction

The paper addresses reliable recovery of global player translation from monocular broadcast video using calibrated field geometry. Field Converter combines geometry-based initialization with temporal residual refinement to place locally estimated poses in a shared metric world frame.

  • Motivation: Shared metric coordinates enable analysis of player interactions, pitch-relative positions, and combined player–ball motion beyond 2D or self-centered 3D estimates.Such representations support tactical analysis, coaching, biomechanics, injury-risk assessment, officiating, and free-viewpoint replay.
  • Problem: Accurate body-relative pose does not guarantee accurate scene localization because similarly configured players may occupy different pitch regions.The target representation is a complete 3D body expressed in a shared metrically meaningful world frame.
  • Problem: Broadcast soccer complicates localization through small subjects, occlusion, rapid motion, large metric coverage, and pan, tilt, and zoom.Weakly textured pitches and dynamic players or spectators also make generic visual odometry and SLAM poorly matched to the setting.
  • Geometric Prior: Known pitch dimensions and standardized markings provide a strong geometric prior for soccer camera calibration and player pose recovery.Sports-specific calibration methods exploit field registration when ordinary scene calibration is difficult.
  • Approach: The method focuses on world grounding because the upstream relative 3D pose is already close to camera-relative ground truth after self-centering.The dominant remaining error is global camera-space translation rather than articulated pose.
  • Approach: Field Converter uses ray–ground intersection to initialize the root, then predicts a temporal residual from pose, image, camera, and field cues.The refined root anchors the relative skeleton in a common world coordinate system.

2. Related Work

Related work advances monocular body reconstruction, temporal modeling, global localization, sports-specific geometry, and soccer benchmarks. Field Converter differs by assuming calibrated pitch geometry and learning only the residual global translation instead of inferring it without structure.

  • Monocular Reconstruction: Parametric and monocular reconstruction methods progressively improve articulated body and multi-person mesh recovery from images and video.Representative systems include SMPL, HybrIK, 4DHumans, Multi-HMR, and SAM 3D Body.
  • Temporal Pose Estimation: Temporal pose methods use sequence information to lift 2D observations or model spatial–temporal relations, but mainly target articulation or camera-relative reconstruction.These methods provide strong local estimates without directly resolving the missing global translation.
  • Global Localization: Global reconstruction methods incorporate camera motion, calibrated rays, gravity, contact, or proxy representations to estimate world-space human trajectories.Examples include Ray3D, GLAMR, SLAHMR, PACE, WHAM, TRAM, GVHMR, and ProxyCap.
  • Benchmark Context: 18,888.9 mm and 8,334.1 mm are the reported WorldPose G-MPJPE values for GLAMR and SLAHMR, respectively.The same benchmark reports per-player, per-frame PA-MPJPE values of 85.2 and 163.9 mm for the two methods.
  • Sports-Specific Estimation: Soccer broadcasts pose unusual-motion, blur, occlusion, small-subject, and high-acceleration challenges that motivate sports-specific priors and validation.Sports-field registration and pose systems use line markings and known geometry to constrain calibration and athlete reconstruction.
  • Datasets: WorldPose provides global player trajectories and body poses from FIFA World Cup footage with broadcast-camera information for multi-person soccer evaluation.It directly exposes the local-to-global localization problem across interacting players moving over an entire pitch.
  • Positioning: Field Converter assumes camera intrinsics and extrinsics are available and uses them to construct metric rays, ground intersections, camera descriptors, and camera-to-world transformations.Unlike generic reconstruction, it uses known soccer geometry to initialize translation and reserves learning for the residual error.

3. Materials and Method

Field Converter combines calibrated soccer-field geometry with temporal residual refinement to recover tracked players in a common metric world coordinate system. It initializes the player root by ray–ground intersection, predicts a correction from pose, image, camera, and geometric cues, and transforms the refined root-anchored skeleton into world coordinates.

  • 3.1. Dataset: The evaluation uses 89 clips from eight matches with calibrated cameras, player annotations, validity masks, and 25-joint metric field-coordinate skeletons.A match-disjoint protocol assigns 62 clips to training, 12 to validation, and 15 from a held-out match to testing.
  • 3.3. Problem formulation: The method assumes player boxes, 2D keypoints, self-centered camera-oriented 3D poses, calibrated camera parameters, and known pitch geometry.These inputs are provided to the final world-grounding stage.
  • 3.4. Geometry-Based Root Initialization: A lower-limb observation is back-projected through the calibrated camera and intersected with the field plane to initialize the camera-space root.The intersection is accepted only for valid, nonparallel rays with positive depth; airborne feet, occlusion, blur, localization error, and pose error can bias it.
  • 3.5. Input Representation: The temporal model receives relative 3D pose, normalized 2D keypoints, bounding-box geometry, camera features, validity masks, and the world-space ray–ground intersection.Box scale and image position provide cues for metric depth and perspective, while the intersection supplies the metric ground location implied by the observation and camera geometry.
  • 3.6. Temporal Residual Refinement: The network predicts a camera-space root residual, and the refined root anchors every relative joint before calibrated transformations produce the common world-coordinate skeleton.This focuses learning on global root translation rather than regenerating the complete articulated body.

4. Results

Geometry-based initialization provides a strong metric starting point, while residual refinement substantially improves global localization, especially with temporal context. The results also show robustness to poor initialization but expose airborne motion as a key limitation.

  • Main quantitative results: 49 cm geometry-based root error falls to 14 cm with a frame-wise MLP, 10 cm with a TCN, and 11 cm with a Transformer.World MPJPE is 16 cm for the frame-wise MLP and 13 cm for both temporal models, while Local MPJPE remains 8 cm.
  • Temporal modeling: Temporal context improves global localization, while comparable World MPJPE for the TCN and Transformer indicates backbone choice matters less than temporal information.Unchanged Local MPJPE shows that gains arise from improved global localization rather than altered relative pose.
  • Prediction target: Residual prediction substantially outperforms direct global-root regression, reducing errors from 2.56 m and 63 cm to 14 cm and 10 cm for the MLP and TCN.The geometry-only initialization reaches 49 cm.
  • Input cues: Removing 2D pose and bounding-box cues produces the largest ablation degradation, reaching 16 cm root error and 18 cm World MPJPE.Removing the valid-joint mask, ray–ground intersection, relative 3D pose, or camera descriptor yields smaller increases to 11–13 cm root error.
  • Robustness to initialization error: The TCN maintains approximately 8–11 cm root error across a broad initialization-error range before reaching approximately 19 cm in the most difficult bin.Temporal refinement is most beneficial when geometric initialization is strongly biased; it can slightly perturb already accurate estimates.
  • Airborne players and failure cases: Airborne motion remains a major failure mode because ray–ground initialization assumes the selected lower-limb joint lies near the playing surface.Temporal refinement compensates for much of the resulting bias but does not remove the underlying ambiguity.

5. Conclusion

Field Converter grounds player pose by refining a geometry-based root estimate with temporal residual prediction rather than directly regressing global position. The results support focusing on global translation while preserving upstream relative pose, although airborne actions and upstream estimation errors remain limitations.

  • Method: Field Converter uses calibrated camera and pitch geometry for metric root initialization, then learns a temporal residual correction instead of directly regressing global position.The framework targets world-grounded 3D player pose from monocular soccer broadcasts.
  • Findings: Residual prediction improves global localization while preserving the relative 3D pose estimated upstream.The conclusion also reports comparable performance for TCN and Transformer backbones and complementary information from image-space, camera, pose, and ray–ground cues.
  • Limitations: The geometry-based initialization assumes a selected lower-limb joint is near the playing surface, an assumption invalidated by airborne actions such as jumps and headers.Errors may also propagate from upstream player detections, pose estimates, and camera calibration.

A. Supplementary Material

The supplementary material adds analyses of architecture, temporal consistency, input ablations, and performance under varying motion and observation conditions.

  • Supplementary analyses: Supplementary analyses compare model architecture and temporal consistency, extend input ablations to both temporal backbones, and evaluate motion and observation conditions.These analyses characterize the proposed root-refinement framework beyond the main experiments.

A.1. Architecture and Model Complexity

Temporal residual models improve global localization over a frame-wise MLP, while TCN and Transformer performance remains comparable despite different temporal mechanisms.

  • Temporal context improves global localization, with the TCN and Transformer achieving comparable World MPJPE despite different temporal mechanisms.The TCN attains the lowest root error, while the Transformer achieves slightly lower reprojection error.

A.2. Extended Input Ablation

Input ablations show that image-space cues are most important, while geometric, camera, pose, and validity features provide complementary information across temporal backbones.

  • Removing image-space pose and bounding-box cues causes the strongest degradation for both the TCN and Transformer.
  • Removing the explicit ray–ground intersection worsens both temporal backbones, supporting its role as an informative geometric cue beyond root initialization.
  • Relative 3D pose supplies body-configuration information, while the valid-joint mask improves robustness to incomplete or unreliable observations.
  • Consistent ablation trends across architectures suggest that the gains primarily arise from feature information rather than architecture-specific behavior.

A.3. Temporal Consistency

Temporal models improve both trajectory smoothness and localization, with the TCN and Transformer showing similar benefits from temporal context.

  • The TCN and Transformer considerably reduce velocity and acceleration errors while improving root localization relative to the frame-wise MLP.
  • The TCN achieves the lowest root-position and velocity errors, whereas the Transformer has marginally lower acceleration error.
  • Similar results across temporal architectures indicate that most gains come from temporal context rather than a specific architecture.

A.4. Temporal Gain Across Motion Regimes

Temporal modeling helps across motion regimes and is especially valuable when instantaneous geometric or image-space observations are ambiguous.

  • Temporal modeling improves performance across all evaluated motion regimes relative to the frame-wise MLP.
  • The TCN provides larger relative gains for airborne players, high-speed motion, and frames with large geometry-initialization errors.
  • Neighboring frames provide trajectory information unavailable from a single frame when instantaneous observations are ambiguous.

A.5. Sensitivity to Image-Space Position

Root localization does not substantially degrade as players move away from the broadcast-image center, suggesting the model accommodates varied viewing configurations using camera and geometric cues.

  • Image-space position: No severe degradation is observed as players move away from the center of the broadcast image.The analysis evaluates root error as a function of player distance from the image center.

A.6. Sensitivity to Player Image Scale

World-grounded localization becomes more difficult for smaller or faster players, while temporal refinement reduces localization errors and depth remains the dominant uncertainty. Throughput measurements show substantial differences in training speed but similar end-to-end evaluation rates across model families.

  • Player image scale: Localization is more difficult for players occupying fewer pixels, and performance improves as apparent player size increases.Smaller players provide less accurate image-space body observations and weaker perspective cues.
  • Temporal refinement: Temporal modeling provides larger relative gains for airborne players, high-speed motion, and frames with large geometry-initialization errors.The TCN improves across all evaluated motion regimes, with especially large gains in difficult situations.
  • Player image scale: Image resolution is an important source of uncertainty for world-grounded reconstruction, particularly for players far from the broadcast camera.The result connects smaller apparent player size with weaker localization reliability.
  • Player speed: Localization error increases for the fastest motions, but temporal models remain more accurate than the frame-wise baseline.High-speed actions involve larger frame-to-frame displacement, stronger motion blur, rapidly changing body configurations, and potentially less reliable instantaneous keypoints.
  • Error decomposition: Depth remains the dominant source of localization error after temporal residual refinement reduces lateral and depth components.Geometry-based initialization is more accurate in lateral image-plane directions than along camera depth.
  • Measured throughput: Mean training throughput was (2.95 ± 0.03) × 103, (25.5 ± 20.8) × 103, and (61.4 ± 25.5) × 103 input player-frames s−1 for the MLP, TCN, and Transformer, respectively.End-to-end batched evaluation reached 530±290, 467 ± 107, and 551 ± 141 video frames s−1 for the same models.
Loading 2609.10498v1…