WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model

1 INTRODUCTION

• Up-to-scale models normalize each window to an arbitrary unit and lack persistent identities, leaving scale ambiguity and identity switches that distort trajectories.The paper asks whether one up-to-scale representation can supply both missing quantities without an external scale network or separately trained tracker.Source: §1 · p. 2
• Intermediate query-key features carry moving-person correspondence; a small projection reads identity cues for association with metric pelvis motion and proposal confidence.The probe finds early and late features nearer chance, while intermediate features retrieve people after motion.Source: §1 · p. 2
• A posed metric body and 2D keypoints provide closed-form scale pseudo-labels for web-video pretraining before exact metric fine-tuning.The Scale Readout predicts from scene-level foundation tokens at deployment rather than repeating the body measurement.Source: §1 · p. 3, Figure 3 · p. 3
• WildHSR jointly reconstructs metric people, cameras, and scene geometry in feed-forward windows, linked by analytic association and Sim(3) composition.Its contributions combine human-derived scale supervision with correspondence read from the pretrained foundation representation.Source: §1 · p. 3

2 RELATED WORK

• Foundation models recover geometry and camera motion in one pass, but some strong models remain up to scale; WildHSR learns metric conversion and probes intermediate features.Prior work reports structure probes, emergent segmentation, motion separation, and preliminary scale decoding from register tokens (El Banani et al., 2024; Caron et al., 2021; Chen et al., 2025a; Wang et al., 2026).Source: §2 · p. 3
• Prior metric-scale methods use metric-native backbones, depth priors, synthetic teachers, body-scale priors, or per-scene optimization; WildHSR uses body readings offline for pretraining.Unlike HAC (Yang et al., 2025), it does not retain body measurement as part of scale recovery at inference; unlike UniSH, its pseudo-label comes from a geometric body ruler.Source: §2 · p. 4
• Feed-forward human-scene systems attach human decoders to geometry foundation models, while WildHSR reads each person at the pelvis rather than a potentially ambiguous shared head cell.Related approaches include Human3R (Chen et al., 2025b), UniSH, SHOW, and GUSH3R (Abe et al., 2026).Source: §2 · p. 4

3.1 PROBLEM FORMULATION AND SYSTEM OVERVIEW

• A per-window Scale Readout converts backbone-unit cameras and scene geometry to metric units while leaving metric SMPL-X body dimensions unscaled.The backbone renormalizes each pass, so scale is predicted separately for each window.Source: §3.1 · p. 4, §3.4 · p. 6

3.2 PERSON RECONSTRUCTION AND TEMPORAL ASSOCIATION

• Pelvis Readout grounds each proposal with an image-space pelvis and person query; layer-13 correspondence features support descriptor-based association using metric pelvis motion and proposal confidence.A dustbin and Sinkhorn produce soft assignments, which Hungarian matching hardens one-to-one.Source: §3.2 · p. 5, Figure 5 · p. 6

3.3 SCALE READOUT AND METRIC ADAPTATION

• Posed metric torsos and 2D keypoints provide approximate closed-form scale pseudo-labels for pretraining, followed by exact metric fine-tuning on RICH and 3DPW training splits.Web-video labels require at least eight readings with std(log σk) ≤0.25; agreement checks consistency, not absolute correctness.Source: §3.3 · p. 6

3.4 COMPOSING ONE METRIC WORLD

• WildHSR composes independent metric window predictions into one world using shared camera frames, with 100-frame windows at stride 50 and at least eight shared frames.When too few correspondences survive or the centred camera trajectory is degenerate, it retains the previous cumulative similarity.Source: §3.4 · p. 6

3.5 TRAINING OBJECTIVES

• Training supervises scale, projected pelvis location, decoded body, identity assignments, and scene contact, while supporting feet are encouraged to agree with the reconstructed surface.The body branch predicts placement at inference without a manual vertical shift.Source: §3.5 · p. 7

4.1 METRIC CAMERAS, SCENE AND PEOPLE

• 66.3 WA-MPJPE and 0.9 RTE are best in the full EMDB-2 comparison; WildHSR leads feed-forward methods on all three metrics.Only JOSH is ahead on W-MPJPE (174.7 against 193.6); WA-MPJPE is the lowest in the full table.Source: §4.1 · p. 7, Table 1 · p. 8
• On RICH, WildHSR has the best WA-MPJPE and leads feed-forward people-and-scene methods on W-MPJPE.Its median multiplicative scale error is 7.2%; JOSH, an optimization, is ahead on W-MPJPE.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 EFFECT OF THE SCALE-TRAINING STAGES

• Removing web-video pseudo-labeling worsens EMDB-2 WA/W/RTE from 66.3/193.6/0.90 to 116.3/255.9/3.7 despite exact-label fine-tuning.The comparison retains BEDLAM2 initialization and exact RICH/3DPW fine-tuning.Source: Table 3 · p. 8, §4.2 · p. 9

4.1 METRIC CAMERAS, SCENE AND PEOPLE

• Intermediate features in layers 11–15 retrieve the same person at 80–82% on 1,500 clips, against 46% chance.This retrieval accuracy is distinct from the conditional match weights and attention lift reported elsewhere.Source: §4.1 · p. 8
• The complete pipeline runs at 10.1 fps on one RTX PRO 6000 Blackwell over long videos exceeding 1,000 frames.Fixed association and Sim(3) composition link feed-forward windows without per-video fitting.Source: §4.1 · p. 8
• On 13 two-person 3DPW test sequences, combined association reaches 74.5 HOTA and 39 ID switches, versus 72.3 and 1,682 for native Human3R.Human3R leads IDF1 (87.4 versus 82.1); this controlled component test does not measure the final end-to-end pipeline.Source: Table 5 · p. 9, §4.1 · p. 9

5 CONCLUSION

• Human-derived pseudo-labels initialize the Scale Readout, while pretrained intermediate query-key features supply person correspondence for identity association.A small identity projection combines correspondence with metric motion; exact metric adaptation follows pseudo-label initialization.Source: §5 · p. 9
• Analytic association and Sim(3) composition link feed-forward windows into joint metric reconstructions without test-time optimization.Source: §5 · p. 9
• Identity weakens under interaction and occlusion, its specificity to people is untested, and sequence composition is offline.RICH training mixtures may also differ across methods.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

• VGGT-Ω and Multi-HMR retain their released base weights and are adapted with small LoRAs.Source: REPRODUCIBILITY STATEMENT · p. 9
• The Scale Readout starts on BEDLAM2, learns from offline ruler pseudo-labels, then receives exact metric supervision from RICH- and 3DPW-train; EMDB-2 is held out.Source: REPRODUCIBILITY STATEMENT · p. 9
• The ruler, agreement gate, ViTPose++-H, and 4DHumans are absent at inference; code, models, and outputs will be released.Source: REPRODUCIBILITY STATEMENT · p. 9

ETHICS AND WEB-DATA STATEMENT

• Public web video supplies 100,000 clips for non-identifying scale pseudo-labels, and the authors do not redistribute it.Source: ETHICS AND WEB-DATA STATEMENT · p. 9
• Persistent reconstruction may enable surveillance; track IDs do not identify people, and the system should not be used for biometric or high-stakes decisions.Source: ETHICS AND WEB-DATA STATEMENT · p. 9
• Web video and parametric body priors may introduce demographic, body-shape, clothing, mobility, and visibility biases.Source: ETHICS AND WEB-DATA STATEMENT · p. 9

• Language models assisted with drafting, restructuring, and editing; the authors directed and revised the generated text and take responsibility for the paper.Source: AI USE STATEMENT · p. 9

Supplementary Material WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model

• The appendices cover association, runtime, RICH evaluation, scale calibration, scene reconstruction, qualitative results, ablations, consistency, probes, robustness, and failure modes.Source: Supplementary Material WildHSR: Metric… · p. 13

A TEMPORAL ASSOCIATION PROTOCOL

• The controlled association experiment fixes proposals, bodies, pelvis positions, and windows, so its comparison tests tracking behavior rather than identical upstream reconstructions.Human3R retains its native pipeline, and association thresholds were tuned on four validation sequences.Source: A TEMPORAL ASSOCIATION PROTOCOL · p. 13

B COMPUTATIONAL COST

• 10.1 frames per second at 688 × 384 is the highest end-to-end throughput among the reported configurations.The body encoder accounts for 62% of runtime; Table 6 reports throughput, memory, and traced model FLOPs for a single 60-frame clip on one GPU.Source: B COMPUTATIONAL COST · p. 13, Table 6 · p. 13

C RICH EVALUATION PROTOCOL

• RICH subject selection uses projected identity descriptors matched to track memory, with metric pelvis motion as a complementary cue and a dustbin for temporary absence.The protocol avoids image-centre priors and capture-rig-specific rules; 18 of 46 segments contain multiple people.Source: C RICH EVALUATION PROTOCOL · p. 14
• RTE is especially sensitive to small trajectory errors in this evaluation because median segment displacement is 0.52 m.Source: C RICH EVALUATION PROTOCOL · p. 14

D HELD-OUT RULER CALIBRATION ON RICH

• 7.2% median multiplicative scale error follows ruler pretraining and exact metric fine-tuning, down from 14.1% for pseudo-labels and 8.4% after pretraining.Table 7 compares the scale pathway stages on held-out RICH examples.Source: D HELD-OUT RULER CALIBRATION ON RICH · p. 14, Table 7 · p. 14

E PERSON-FREE SCENE RECONSTRUCTION

• 1.66 cm mean point-map error is lower than AMB3R’s 1.75 cm under the published scale-aligned 7Scenes protocol.This comparison tests scene geometry, not metric scale without visible people.Source: E PERSON-FREE SCENE RECONSTRUCTION · p. 14, Table 8 · p. 14

Metric people-scene placement

• WildHSR has the lowest pelvis offset on seven of nine EMDB-2 clips; Human3R is closer on two stairs-up clips, at 0.3 versus 0.5 m and 0.8 m.The views preserve predicted scale after rigid camera-path alignment.Source: Figure 7 · p. 15

Global human trajectories

• WildHSR’s mean path error is 0.10 to 0.28 m, compared with 0.52 to 0.96 m for Human3R and 1.04 to 3.75 m for UniSH.These paths are similarity-aligned; the separate metric-scale test is in Figure 9.Source: Figure 8 · p. 16

Metric scene, camera and ground

• WildHSR reconstructs five scenes at 0.95 to 1.00× ground-truth size, with 0.02 to 0.09 m mean camera error and 2 to 8 cm median foot-floor gap.The corresponding gaps are 9 to 26 cm for Human3R and 4 to 46 cm for UniSH; the scene scale, camera path, and ground remain mutually consistent.Source: Figure 9 · p. 16

Crowded in-the-wild reconstruction

• WildHSR maintains 14 tracks over 10 s and reconstructs 10 to 11 people per frame in one scene.On two annotated people, hip-depth error is 0.29 m for WildHSR versus 1.20 m for Human3R, while PA-MPJPE is 55.3 versus 51.7 mm. Figure 10 plots camera trajectory, metric camera-to-person distance, and world-frame path error against video length.Source: Figure 11 · p. 17, Figure 10 · p. 17

Scale pathway

• Replacing the final two-stage Scale Readout with the best regressor trained on synthetic metric ground truth yields 2.7× WA and 13× RTE.Table 9 reports component substitutions on EMDB-2; this replacement does not isolate ruler pretraining from ground-truth fine-tuning.Source: G ADDITIONAL ABLATIONS · p. 17, Table 9 · p. 17

H PLAUSIBILITY AND METRIC SCALE ON EMDB-2

• Across EMDB-2’s 25 sequences, WildHSR’s foot gap is 3.6 cm versus 12.7 cm for Human3R and 15.5 cm for UniSH; it has 6% world-size error and 0.13 m camera error.Foot-gap diagnostics use the first 300 frames per sequence and assess human–scene consistency, not accuracy against ground truth.Source: H PLAUSIBILITY AND METRIC SCALE ON… · p. 18, Table 10 · p. 18
• 9.8 cm without contact regularization versus 1.1 cm with it, the lowest body vertex floats less above the reconstructed floor, while W-MPJPE and RTE remain nearly unchanged.1.2% of frames penetrate the floor by more than 5 cm; foot skate and vertex jitter are unchanged.Source: H PLAUSIBILITY AND METRIC SCALE ON… · p. 19

I IDENTITY FEATURES AND PELVIS LOCALIZATION

• 80 to 82% retrieval in layers 11 to 15 includes 70% after motion and 60% on hard pairs, while output-token retrieval declines with depth.Chance is 46% overall and 40% on hard pairs; Figure 13 compares retrieval by feature type and layer.Source: I IDENTITY FEATURES AND PELVIS… · p. 19, Figure 13 · p. 20
• 5.0× uniform attention lift at the person’s new position versus 1.2× at the vacated position and 1.5× on other people; the new position wins in 91% of eligible clips.Overlap and small people weaken the effect, and this control does not measure trained-tracker accuracy.Source: I IDENTITY FEATURES AND PELVIS… · p. 19, Figure 13 · p. 20
• The probe does not establish person-specific correspondence or improved tracking: generic moving-surface matching could pass, and the probe alone cannot show tracking benefit.The association-component experiment holds upstream proposals and bodies fixed rather than isolating the effect in the end-to-end pipeline.Source: I IDENTITY FEATURES AND PELVIS… · p. 19
• WildHSR reads layer-13 pelvis query/key features with a trained projection, then uses differentiable Sinkhorn assignment with birth and miss options and Hungarian matching at inference.The pelvis provides a smoother motion anchor than distal joints.Source: I IDENTITY FEATURES AND PELVIS… · pp. 19–20

J ROBUSTNESS TO TRUNCATION AND OCCLUSION

• WildHSR has lower error in 29 of 30 clip-condition pairs under identical synthetic masks, while head removal roughly halves detection coverage for both systems.The masks test sensitivity to missing input, not the frequency of natural occlusion.Source: J ROBUSTNESS TO TRUNCATION AND OCCLUSION · p. 21, Table 11 · p. 21

K MEASURED FAILURE MODES

• The examples show stair sinking, higher full-clip knee-flexion error than Human3R, and a 50-frame missing-body interval followed by return under a new track ID.The reported limits concern stair placement, body articulation, and track continuity.Source: K MEASURED FAILURE MODES · p. 22