PaperlayerView exact revision on arXiv

Source-linked AI summary

A Diagnostic Study of World-Space Human Recovery Under High Camera Dynamics

Hao-Jen Chien, Wei-Lun Chao, Yu-Lun Liu

arXiv:2610.19729v1cs.CV

TL;DR

Existing benchmarks rarely test monocular world-space human recovery under fast camera motion, limiting evidence for broadcast dance settings. The paper introduces a K-Pop stress test and a closed-form correction for camera–human scale mismatch. The correction removes camera-induced drift while preserving performers’ relative positions in a shared world frame.

  • Problem

    Existing benchmarks mostly contain moderate camera motion, leaving robustness under sustained high camera dynamics largely untested.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    The paper introduces the K-Pop stress test and derives a closed-form correction that estimates scale mismatch and applies one scale to the camera and all performers.

    Source: §1 · p. 3

  • Results

    The authors report that scale mismatch turns camera motion into world-space drift, and that their correction removes the drift without retraining across backends.

    Source: §1 · p. 3, §8 · p. 10

  • Takeaways & Limitations

    The correction preserves performers’ relative positions in a shared world frame, supporting stable world-space reconstruction under high camera dynamics.

    Source: §1 · p. 3, §8 · p. 10

    A single scale per sequence cannot fix performers whose camera-space depth is already wrong, and within-shot scale variation is not represented.

    Source: §7.1 · p. 10

Abstract

from arXiv · show

Monocular world-space human recovery methods excel on current benchmarks, but these rarely include fast camera motion. We introduce the K-Pop stress test, broadcast dance videos that combine fast, large camera motion with dense multi-person choreography. Leading methods produce plausible per-frame meshes on these videos, yet their world-space trajectories drift far from the performers' stage positions. Through a systematic diagnosis, we show that the camera and human estimates each remain self-consistent, and that the drift appears when they are combined at mismatched scales. Each is recovered at its own scale, so a mismatch between the two can turn camera motion into human drift that grows with camera displacement and becomes severe in broadcast dance videos. We derive a closed-form correction that estimates this mismatch and applies one scale to all performers, removing the drift while keeping their relative positions in one shared world frame. It requires no retraining and works with existing backends. Combined with physical constraints, the correction yields stable world-space reconstruction for choreography modeling, scene editing, data generation, and virtual reshooting.

1 INTRODUCTION

Compositional methods perform well on benchmarks with mostly moderate camera motion, but K-Pop broadcast videos expose severe world-space drift despite plausible camera-space estimates. The paper attributes the drift to mismatched camera and human translation scales and proposes a shared-scale correction.

1 INTRODUCTION

  • Compositional methods perform well on 3DPW, RICH, and EMDB, whose diverse human motions are paired mostly with moderate camera motion.The paper asks whether this benchmark performance generalizes to in-the-wild video.Source: §1 · p. 2
  • The K-Pop stress test introduces broadcast dance videos with high camera dynamics and dense multi-person choreography, a regime existing benchmarks do not cover.The setting is relevant to choreography modeling, scene editing, virtual reshooting, and data generation.Source: §1 · pp. 2–3
  • Camera estimates remain self-consistent and camera-space poses plausible, while combining their independently scaled translations can turn camera motion into human drift.The scale mismatch hypothesis explains why each estimate can pass its own consistency check.Source: §1 · p. 2
  • On stationary intervals, recovered performer drift follows the camera path and grows with camera displacement.The authors use stationary performers as a test because their true world-space displacement is ideally zero.Source: §1 · p. 2, Figure 3 · p. 3
  • A closed-form correction estimates the scale mismatch and applies one scale to the camera and all performers, preserving their relative positions in one shared world frame.It requires no retraining and transfers across backbones.Source: §1 · p. 3

2 RELATED WORK

Prior work either infers world-space motion from local cues or composes camera-space humans with camera estimates; this paper follows the compositional family and diagnoses its scale mismatch under high camera dynamics.

2 RELATED WORK

  • World-space recovery methods infer motion from local cues, jointly optimize camera and human motion via SLAM, or compose camera-space humans into a world frame.The passage describes gravity-aware coordinates, absolute depth, and metric scene depth as approaches coupled with camera estimation.Source: §2 · p. 3
  • Local-to-global motion inference is ambiguous when motion priors do not match the scene, such as dancing in place.The cited local-cue line includes Yuan et al. (2022), Sun et al. (2023), Shin et al. (2024), and Li et al. (2022a).Source: §2 · p. 3
  • This work follows staged composition and identifies camera–human scale mismatch as the dominant failure mode under high camera dynamics, despite plausible camera-space meshes on existing benchmarks.The passage names TRAM and PromptHMR among compositional methods.Source: §2 · p. 3

3 BACKGROUND AND EXISTING GAP

Monocular world-space composition combines camera poses with camera-space human positions, but existing benchmarks and synthetic camera paths leave sustained high camera dynamics largely untested. The K-Pop stress test is introduced to address that gap.

3 BACKGROUND AND EXISTING GAP

  • The passage reports average camera speed values of 1.15, 0.20, and 0.14 body/s, with higher values corresponding to higher average camera motion.Source: §3 · p. 4
  • K-Pop camera motion is about 6 to 8 times faster on average, spans a broader focal-length range, and includes up to 12 performers per frame.EMDB is described as having near-single-person clips.Source: Figure 4 · p. 4
  • Visual odometry estimates camera poses from monocular video, whose recovered translation is only known up to an unknown scale.The poses comprise rotation and translation relative to a fixed world frame.Source: §3 · p. 4
  • Compositional recovery combines camera-space human positions with camera poses through a rigid transform; the analysis tracks feet because they remain grounded while the pelvis can move.The human model predicts a camera-space pelvis translation, and recent methods compose it with visual odometry.Source: §3 · p. 4
  • Existing real benchmarks mostly feature moderate camera motion, while synthetic camera trajectories do not capture broadcast video’s large displacements and rapid motion changes.The robustness of world-space composition under sustained high camera dynamics therefore remains largely untested.Source: §3 · p. 4

4 THE K-POP STRESS TEST

The K-Pop stress test evaluates world-space recovery on broadcast dance clips selected for visible performers and self-consistent camera estimates. The section also describes paired-camera references and proxy measures for clips without world-space ground truth.

4 THE K-POP STRESS TEST

  • The evaluation set contains 616 clips, 59.19k frames, and 336.87k pose instances, with up to 12 performers per frame.Clips are selected for performer visibility and camera trajectories passing a time-reversal test; the set is used only for evaluation.Source: §4 · p. 4
  • Figure 5 compares forward and reversed ViPE camera trajectories in ground-plane and height views after one similarity alignment; the test is used to assess camera self-consistency.The caption places the rightmost panel near the K-Pop mean S-ATE of 1.35.Source: Figure 5 · p. 5
  • A paired-camera reference uses synchronized broadcast and fixed-venue views to assess scale, world-space errors, and performers’ relative geometry.The reference comprises 11 matched pairs covering 989 frames and 78 performers.Source: §4 · pp. 4–5
  • Without reference ground truth, proxy evaluation measures reprojection against independent keypoints and masks, plus temporal jitter and foot skating.The passage specifies PCK and Mask IoU for reprojection, and Jitter and Foot Skating for world-space stability.Source: §4 · p. 5

5 DIAGNOSTIC ANALYSIS OF COMPOSITIONAL METHODS

Compositional methods show severe world-space drift on the K-Pop stress test even though camera-space human estimates remain plausible and camera estimates remain self-consistent. The diagnosis locates the drift in combining camera and human translations recovered at different scales; camera displacement magnifies it, and downstream optimization can also produce image misalignment.

5 DIAGNOSTIC ANALYSIS OF COMPOSITIONAL METHODS

  • Compositional methods show severe world-space drift that follows camera ego-motion, indicating a structural cause in combining the two estimates.World space is recovered by composing camera and human estimates, so the analysis examines each estimate and their composition.Source: §5 · p. 5
  • Camera-space meshes remain plausible across frames, while the visual-odometry estimates are self-consistent under a time-reversal shuttle test.The retained K-Pop clips have S-ATE 1.35 and S-Focal 3.39 degrees, compared with 1.05 and 4.26 on OpenDV, respectively.Source: §5 · p. 5
  • When VO and HMR translations disagree in scale, their camera-motion terms fail to cancel and a stationary performer drifts with the camera.For stationary intervals, the mismatch is modeled as ΔT′_ij = sΔT_ij, with s = 1 denoting consistent scale; the relation holds for any camera rotation, with rotation errors handled by interval gates.Source: §5 · p. 6, Figure 6 · p. 6
  • Drift grows with camera displacement, remaining small on existing benchmarks but becoming severe under broadcast camera work.Focal-length error and monocular depth ambiguity can affect the two estimates differently and produce the scale mismatch.Source: §5 · p. 6
  • Scale mismatch explains world-space drift but not image-space misalignment, which can arise when downstream physical optimization converts large initial drift into a poorly aligned trajectory.PromptHMR performs this optimization, whereas TRAM does not and remains an unstable trajectory.Source: §5 · p. 6

6 CLOSED-FORM SOLUTION DERIVATION

The method estimates a shared scale correction from stationary intervals, applies it to camera and performer translations, and refines it with physically grounded optimization. On the reported comparisons, the corrected method is closest to the reference in scale ratio, ranks in the top two on both world-space metrics, and transfers across backbones.

6 CLOSED-FORM SOLUTION DERIVATION

  • The closed-form estimator solves for scale mismatch from observable relative displacements in stationary intervals using weighted least squares.Interval weights reflect confidence that the interval is stationary.Source: §6 · pp. 6–8
  • Applying the recovered scale to visual odometry and world-space human translation removes camera-induced drift for all performers while preserving their relative positions in one shared world frame.A residual scale is refined in physically grounded optimization, which enforces reprojection consistency, temporal smoothness, and inter-person separation.Source: §6 · p. 7
  • Stationary intervals are screened using foot-contact confidence, then rejected when body motion or excessive rotational error is detected.The reported thresholds require contact confidence above 0.5 in at least 80% of interval frames, displacement alignment with camera motion above 0.8, and signal-to-noise ratio of at least 1.5.Source: §6 · p. 7
  • Ours is closest to the fixed-camera reference in scale ratio, is in the top two on both K-Pop world-space metrics, and the comparison reports HAC skating 2.8× more on the same backbones.The visual comparison also shows ours keeping both image alignment and world-space trajectories, while baselines exhibit one or both failures.Source: Table 1 · p. 7, §6 · p. 8, Table 2 · p. 8
  • The correction transfers across visual-odometry and HMR backbones; on TRAM it trades a small loss in 2D fit for a large gain in world-space stability.For TRAM, contact confidence comes from PromptHMR-Vid because TRAM predicts none.Source: §6 · p. 8, Table 3 · p. 9

7 EXPERIMENTS

On K-Pop, our method leads baselines on reported world-space metrics, while ablations show that drift correction improves metrics and also reveal residual scale and depth limitations.

7 EXPERIMENTS

  • Ours recovers a scale ratio closest to 1, leads every baseline on WA-MPJPE, W-MPJPE, and Pair Distance, and is the only top-two method on both full-test world-space metrics.The comparison covers the reference set and full stress test.Source: §7 · p. 9

7.1 ABLATION STUDY

  • Drift correction improves reprojection and world space across all metrics; PGO alone partially shortens drift but leaves stationary performers creeping across the scene.With correction, stationary performers stay fixed and meshes align.Source: §7.1 · p. 9, Figure 9 · p. 10
  • Temporal repulsion reduces inter-performer penetration and keeps bodies separated; 34 participants preferred our method on overall quality and camera motion in over 96% of comparisons.The user study covered 8 sequences and 3 baselines.Source: §7.1 · p. 9
  • On EMDB-2, Jitter is 21.18 versus 46.12 mm and Foot Skating 8.79 versus 11.19 mm against PromptHMR reproduction, while trajectory metrics are weaker.The authors attribute weaker trajectory metrics to small camera displacement providing little signal for estimating scale.Source: §7.1 · p. 9
  • A single sequence scale cannot fix already-wrong camera-space depth: an occluded performer sits meters behind the group until visibility returns, and within-shot scale changes are unmodeled.Scale recovery also inherits noise from upstream camera and HMR estimates.Source: §7.1 · p. 10, Figure 10 · p. 10

8 CONCLUSION

The conclusion links severe drift in high-camera-motion K-Pop video to mismatched camera and human translation scales, despite apparent robustness on mildly moving benchmarks. It presents a no-retraining correction that removes drift and yields stable trajectories across backends.

8 CONCLUSION

  • On the K-Pop stress test, a scale mismatch between camera and human translation turns camera motion into world-space drift, unlike the mild camera motion of existing benchmarks.Source: §8 · p. 10
  • The closed-form correction removes drift without retraining and gives stable trajectories across backends.Source: §8 · p. 10

APPENDIX

  • The appendices detail implementation and loss formulation, assumptions, estimator ablations, stress-test construction, user study, temporal repulsion, paired-camera references, and existing-benchmark HMR performance.Source: APPENDIX · p. 17

A RELATED WORK

Related work leaves fast broadcast camera motion and dense choreography underrepresented, while prior work addresses world-space recovery, scale ambiguity, camera estimation, and physical constraints. The pipeline adapts compositional recovery with contact-based scale estimation, physical refinement, and gates that limit errors in the scale estimate.

A RELATED WORK

  • Existing benchmarks cover moderate camera motion, while the K-Pop stress test adds in-the-wild broadcast video with sustained camera dynamics and dense choreography.WorldPose is the closest multi-person world-space benchmark, but its motion distribution is sport-specific.Source: A RELATED WORK · pp. 17–18

B PIPELINE DETAILS

  • The pipeline separates camera estimation, camera-space human reconstruction, and world-space composition, adapting this compositional design to high camera dynamics.It uses ViPE with dynamic-element masking for camera estimation, then tracks and refines performers before composing their predictions into the world frame.Source: B PIPELINE DETAILS · p. 19
  • Scale recovery uses 10-frame intervals with planted-foot evidence, while additional gates reject intervals where motion or rotation error undermines the stationary-performer assumption.The fit jointly estimates one scale across x, y, and z because broadcast cameras translate in all three dimensions.Source: B PIPELINE DETAILS · pp. 19–20, D DRIFT ASSUMPTIONS AND ERROR SOURCES · p. 22, §D.2 · p. 23
  • A weighted least-squares estimate supplies one sequence-level scale; the correction rescales camera translation and adjusts every performer, while clips without admitted intervals remain uncorrected.The following optimization refits the floor and fills translation and pose gaps by interpolation, with boundary translations extrapolated from recent motion.Source: B PIPELINE DETAILS · p. 20

C PHYSICALLY GROUNDED LOSSES

  • Physical refinement combines reprojection consistency, translation smoothness, contact-aware grounding and motion terms, and temporal inter-person repulsion.Together these terms address image agreement, foot skating, temporal coherence, floor feasibility, and implausible multi-person collisions.Source: B PIPELINE DETAILS · p. 20, C PHYSICALLY GROUNDED LOSSES · pp. 21–22

D DRIFT ASSUMPTIONS AND ERROR SOURCES

  • The scale estimator assumes performers are stationary within admitted intervals and that camera-rotation error is small relative to camera translation.Rotation error can shift camera-to-performer vectors and bias the inferred scale when that shift approaches the translation signal.Source: D DRIFT ASSUMPTIONS AND ERROR SOURCES · p. 22, §D.1 · p. 23

D.1 ROTATION ERROR

  • Intervals are excluded when expected rotational displacement at performer depth exceeds the camera-translation signal by the specified margin.The signal-to-noise ratio must reach at least 1.5 for an interval to contribute to scale estimation.Source: §D.1 · p. 23

D.2 ADDITIONAL INTERVAL GATES

  • Additional interval gates discard weak camera translations and performers too close to the camera, and downweight axes with little camera motion.Intervals must also pass the contact threshold on at least 80% of their frames.Source: §D.2 · p. 23

D.3 ANALYSIS OF DRIFT SOURCES

Focal-length error acts as a clip-wide scale degree of freedom that the estimator absorbs, whereas frame-varying crop or detector errors do not explain the observed drift. The ablations examine interval pooling and whether the fitted scale should vary over time.

D.3 ANALYSIS OF DRIFT SOURCES

  • Focal-length error shifts the estimated drift bias in proportion to the error, and the correction removes that shift.Because scaling depth and camera translation together leaves projections unchanged, the estimator absorbs focal error or depth ambiguity as a scale bias.Source: §D.3 · pp. 23–24, Figure 14 · p. 24
  • Frame-varying crop and detector errors do not explain the drift: both cropped and uncropped backbones drift and respond to the same correction.Any component of these errors that persists across a sequence can act as a scale bias and be absorbed by the correction.Source: §D.3 · p. 24

E SCALE ESTIMATION ABLATIONS

  • The scale-estimation ablations vary which stationary intervals are pooled and whether one sequence-level scale is replaced by a time-varying scale.The accompanying analysis reports that the time-varying choice destabilizes world-space motion while 2D metrics hold.Source: E SCALE ESTIMATION ABLATIONS · p. 24

E.1 SENSITIVITY TO THE NUMBER OF STATIONARY INTERVALS

The two-stage estimator keeps final results stable despite changes to interval pooling, while a time-varying scale destabilizes recovered motion. On stationary-performer clips, drift correction reduces residual displacement; the stress-test construction filters broadcast clips for usable coverage and camera estimates.

E.1 SENSITIVITY TO THE NUMBER OF STATIONARY INTERVALS

  • Reducing pooled intervals from 125 to 3 shifts the closed-form estimate by 0.133, but the final values differ by only 0.002 after PGO.On this sequence, the second stage recovers nearly all of the first-stage change.Source: §E.1 · p. 24, Table 6 · p. 24
  • A time-varying scale destabilizes world-space motion and harms temporal metrics while 2D metrics hold; segment-scale disagreement perturbs the reassembled camera path and performer trajectories.The reported explanation is segment overfitting, rather than too few intervals.Source: §E.1 · pp. 24–25, Table 7 · p. 25
  • Changing the contact threshold changes the closed-form estimate, but reported metrics barely move; foot skating rises modestly across the tested range.The reported stability belongs to the closed-form estimate and learned residual together.Source: §E.1 · p. 25, Table 8 · p. 25

E.2 STATIONARY-PERFORMER CLIPS

  • On 10 clips with 60 stationary performers, PGO cuts net displacement from 2.334 m to 0.581 m, and adding drift correction reduces it to 0.153 m.Jitter barely changes between the last two rows, so the additional displacement reduction reflects improved performer placement rather than trajectory smoothing alone.Source: §E.2 · p. 25, Table 9 · p. 25

F K-POP STRESS TEST CONSTRUCTION DETAILS

  • Stress-test construction filters approximately 1,000 collected videos for stable multi-person coverage, removing close-ups, long detection gaps, single-performer clips, and clips shorter than 3 seconds.The authors do not redistribute raw video; they plan to provide YouTube URLs and timestamps for reproduction.Source: F K-POP STRESS TEST CONSTRUCTION DETAILS · p. 26

G USER STUDY DESIGN

The user study compares perceived reconstruction quality and camera-motion match across eight sequences and three baselines. Participants strongly preferred the proposed method on both questions.

G USER STUDY DESIGN

  • The method received 97.3% of votes for overall quality and 96.6% for camera motion across the study.Against each baseline individually, it was preferred in at least 96.3% of overall-quality comparisons and 95.6% of camera-motion comparisons.Source: G USER STUDY DESIGN · p. 26
  • The study involved 34 participants and eight sequences, with 24 comparisons per participant and 816 votes per question.Each comparison presented synchronized views of the input, 2D reprojection, front-facing and top-down 3D views, and performer trajectories with camera position.Source: G USER STUDY DESIGN · p. 26
  • Figure 15 reports the share of votes for overall quality and camera motion, pooled across 24 pairwise comparisons; each baseline appears in one third of comparisons.The three baselines are TRAM, GVHMR, and PromptHMR.Source: Figure 15 · p. 26

Related papers

Loading 2610.19729v1…