SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction

1 Introduction

• Contact-rich long-horizon tasks require coordinated interaction with both objects and terrain, beyond reaching desired poses.The introduction illustrates this with picking up a basket, climbing stairs, and placing it near a washing machine.Source: §1 · p. 2
• Per-link contact labels specify which body links should establish and use scene contact, supplementing reference poses that alone cannot resolve whether a robot is touching, lifting, or stepping.The labels are defined on the robot body, keeping the policy a low-level tracker rather than a vision-based task policy.Source: §1 · p. 2
• Hindsight scene reconstruction infers robot-scene interaction graphs from retargeted robot motion, then reconstructs plausible terrain and object assets to create contact-rich training data.This addresses the scarcity of trajectories paired with scene geometry and contact annotations.Source: §1 · p. 2
• SceneBot unifies free-space locomotion, uneven-terrain traversal, and whole-body object manipulation in one policy, generalizing to unseen motions and environments while maintaining comparable free-space performance.It also demonstrates long-horizon tasks such as carrying a box upstairs.Source: §1 · p. 2

2.1 General motion tracking

• General motion trackers evolved from task-specific policies to unified trajectory tracking, but existing datasets and retargeting methods lack scene awareness for complex human-environment interactions.The cited landscape includes generalized policies [4] [5] [6] [7] [8] [9] and large-scale datasets [4] [10].Source: §2.1 · pp. 2–3

2.2 Terrain aware whole body control

• Terrain-control approaches include root-velocity RL, scene-aware retargeting, distillation, and AMP-style rewards, yet root-velocity control limits general-purpose loco-manipulation and kinematic data constrains dexterous upper-body motion.The passage cites terrain-aware kinematic-generator co-training methods [25] [26].Source: §2.2 · p. 3

2.3 Humanoid object interaction

• Object manipulation work includes lightweight pick-and-place, force-regulated bimanual grasping, and large-object trajectory tracking, while a general whole-body controller for zero-shot complex manipulation remains lacking.Table 1 compares whole-body controllers for tracking diverse motions and contact labels across terrain interaction and large-object carrying.Source: §2.3 · p. 3, Table 1 · p. 3

3 Method

• SceneBot combines reconstructed scene assets, reinforcement-learning motion and contact tracking, and deployment state estimation in a generalist controller.The policy can track reference motions with contact labels for scene interaction and without them for flat-terrain locomotion.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 Scene Generation from Human Motion

• The reconstruction pipeline retargets captured human motion, builds a robot-scene interaction graph using contact criteria and temporal consistency, then generates terrain and object assets conditioned on the graph.Terrain is represented as an elevation map with plateaus, while objects are represented by plates aligned with robot contact surfaces and positioned from grasp interactions.Source: §3.1 · pp. 4–5
• The heterogeneous interaction graph links key robot nodes—wrists, feet, and pelvis—to terrain or object nodes through time-stamped spatial interaction edges.Candidate edges arise from low relative velocity and acceleration, then infeasible edges are pruned using collision and force-closure constraints.Source: §3.1 · p. 4

3.2 Training a Contact-Aware Whole-Body Controller

• The controller receives reference kinematics, binary desired contact labels, proprioception, and root state, and reinforcement learning trains it to track both motion and contact behavior.An active label indicates that a robot link should contact a terrain or object scene type.Source: §3.2 · p. 5
• The contact correctness reward compares desired and simulated contacts, while terrain-contact penalties apply only when contact is commanded, allowing unlabeled flat-terrain motions to be used without penalizing unspecified contact.Zeroing contact labels at deployment supports free-form flat-terrain tracking.Source: §3.2 · p. 5
• A contact-duration reward encourages stable interactions but is clipped at 0.5 seconds to prevent the robot from refusing to release an object.A contact-mismatch termination condition also requires desired object contact to be established within a specified time frame.Source: §3.2 · p. 5
• Because reconstructed objects may not begin at rest before grasping, training applies a heuristic stabilizing force until a stable force-closure grasp is achieved.The intervention mitigates frequent grasp-failure terminations early in training and provides a denser contact reward signal.Source: §3.2 · p. 5

3.3 Infrastructure-Free State Estimation

• Deployment estimates global root position, orientation, and linear velocity with SuperOdometry and a pelvis-mounted IMU, fusing yaw estimates with a Kalman filter.The state-estimation pipeline runs at 200 Hz and initializes root height from the lowest 5% of gravity-aligned LiDAR points.Source: §3.3 · p. 6

4.2 Tracking Performance for Different Motions

• SceneBot tracks free-space motions on par with SONIC and significantly outperforms it on scene-interaction tasks, including long-horizon box carrying upstairs.The evaluation covers free-space, terrain interaction, object interaction, and sitting; Fig. 4 also depicts simultaneous object manipulation and terrain traversal.Source: Figure 4 · p. 6, §4.2 · p. 7

4.1 Experiment Setup and Dataset

• The experiments use the Unitree G1 and combine AMASS, OMOMO, Bones, and Lafan data for scene-interaction and flat-terrain motions.The robot uses built-in LiDAR and a pelvis IMU for state estimation.Source: §4.1 · p. 7, Figure 5 · p. 7

4.3 Necessity of Global Tracking

• Without global root position and linear velocity, success falls from 100% to 45% on terrain tasks and from 95% to 20% on object-related tasks.Global root positioning prevents drift and motion-terrain misalignment during training and inference.Source: §4.3 · p. 7

4.4 Effectiveness of Contact Labels

• Hand contact labels are essential for object grasping, while omitting foot labels leads to missteps during stair contact switches and falls.Without hand labels, object-grasp success drops to near zero.Source: §4.4 · pp. 7–8
• With contact labels disabled only at test time, terrain traversal still achieves an 85% success rate, 15% below using active labels and better than training without foot labels.The authors hypothesize that contact labels simplify training and that learned balancing mechanics remain when labels are omitted at inference.Source: §4.4 · p. 8

4.6 Performance of State Estimation

• State estimation averages 0.032 m root position error, 0.018 rad root orientation error, and 0.092 m/s root linear velocity error, with 80% task success across 10 experiments.The estimates are compared with OptiTrack motion-capture ground truth on motions including box pickup and stepping onto a stage.Source: Figure 9 · p. 8, §4.6 · p. 8

5 Limitations

• Scene reconstruction requires high-fidelity robot motion.Poorly retargeted human motion, such as foot sliding, makes reconstruction highly sensitive.Source: §5 · p. 8
• Raw-video or generative-model trajectories may lack physical consistency, risking destabilization of scene-interaction graph construction.Source: §5 · p. 8

6 Conclusion

• SceneBot pairs a contact-prompted whole-body tracking policy with a data engine that synthesizes interactive scenes from robot motion for training.Source: §6 · p. 9
• Evaluations demonstrate tracking of diverse behaviors from flat-terrain locomotion to multi-task interactions such as ascending stairs while carrying an object.Source: §6 · p. 9
• Future work aims to develop an autonomous high-level controller for perceptive, goal-driven scene interactions.Source: §6 · p. 9

• The pipeline retargets human motion, initializes robot and scene nodes, and identifies candidate interactions when key-link velocity and acceleration relative to the scene are low.Key links are the wrists, feet, and pelvis; candidate edges record an interaction location and target scene node.Source: A Scene Reconstruction Algorithm · p. 13
• Candidate interaction edges that cause collisions outside their interaction interval or violate force-closure constraints are pruned before forming the scene-interaction graph.Source: A Scene Reconstruction Algorithm · p. 13
• The remaining edges guide reconstruction of terrain plateaus and objects assembled from contact-parallel plates and interaction-edge positions.Terrain plateaus are merged by similar height and regions causing robot-terrain collisions are carved out.Source: A Scene Reconstruction Algorithm · p. 13

• Detailed tracking rewards are defined in Table 2.Source: B Training rewards · p. 13

• Figure 4’s quantitative values, including tracking errors and success rate, are detailed in Table 3.Source: C Quantitative results · p. 13
• Table 2 presents reward-design details, with r denoting the root and b ∈ B denoting body links.Source: Table 2 · p. 14

• Terrain success requires the robot not to fall and its root-height error to remain below 0.2 m.Source: D Definition of Success · p. 14
• Object-task success requires the robot not to fall and the object to remain grasped until release.Source: D Definition of Success · p. 14
• Sit-task success requires the robot not to fall and its pelvis to contact the chair.Source: D Definition of Success · p. 14
• Free-space success requires only that the robot not fall.Source: D Definition of Success · p. 14

• The maximum state-estimation error occurs during crouching, then recovers as the robot straightens.Source: E State estimation results · p. 14
• Figure 10 overlays the estimated trajectory in yellow on the point-cloud map built using SuperOdometry.Source: Figure 10 · p. 14
• Table 3 covers evaluation metrics across methods and scenarios.Source: Table 3 · p. 15