Source-linked AI summary
CoDance: Learning Reactive and Compliant Human-Humanoid Interaction from Video
Zhuoqun Chen, Shucheng Jia, Boyuan Chen
TL;DR
Partnered human-humanoid interaction requires locomotion to adapt to a person while continuous contact forces are accommodated. CoDance turns a paired-dancing video into force-aware training data and learns partner-following, compliant motion; simulation and hardware results demonstrate wrist-displacement reproduction and sustained dancing with a human partner.
Problem
Partnered interaction couples motion and contact, but learned humanoid behaviors commonly focus on isolated motion tracking or locomotion commands rather than adapting to a person while accommodating forces.
Source: §I · pp. 1–2Method
CoDance retargets a paired-dancing video into a robot reference and moving partner, then uses multi-link compliance augmentation to create force-aware training data for partner-following policies.
Source: §I · p. 2, A. Data Pipeline · p. 3Results
Takeaways & Limitations
CoDance supports partner-following locomotion and compliant two-hand interaction learned from paired human video, with sustained human-partner dancing demonstrated on a physical humanoid.
Source: §V · p. 8The study is limited to one video and one dance; the simulated partner replays fixed motion, and only about a quarter of trials finish a clip when the partner moves opposite to it.
Source: §V · p. 8
Abstract
from arXiv · showhide
Partnered human-humanoid interaction couples locomotion with continuous physical contact. A humanoid needs to coordinate with a person's motion while responding to interaction forces and maintaining stable and natural movement. We present CoDance, a framework for learning reactive and compliant human-humanoid interaction from video. We study partnered dancing as a challenging instantiation, where a humanoid coordinates its footsteps with a moving partner and maintains continuous two-hand contact. Given a single video of two human dancers, CoDance retargets their motions into a robot reference and a moving partner. We introduce a multi-link compliance augmentation that transforms the kinematic demonstration into force-aware training data by adapting the robot reference under structured forces at both hands. Policies trained on this data follow the observed partner while preserving the demonstrated locomotion style and responding compliantly to physical interaction. In simulation, the policies adapt their footsteps to changes in the partner and reproduce approximately 80% of the wrist displacement encoded by the augmented demonstrations. On a physical humanoid, CoDance enables sustained two-hand dancing with a human partner including repeated transitions between forward and backward motions.
I. INTRODUCTION
CoDance addresses partnered dancing as an interaction problem: the robot must coordinate locomotion with a person while maintaining continuous contact and accommodating forces. It turns a single paired-dancing video into force-aware training data and demonstrates partner following in simulation and sustained dancing on a physical humanoid.
I. INTRODUCTION
- Partnered dancing requires the robot to adapt to a moving person while accommodating contact forces, preserving balance, and maintaining natural movement.Unlike conventional motion imitation, which treats deviations from the reference as tracking errors, this setting requires compliant motion and locomotion adaptation together.Source: §I · pp. 1–2
- CoDance retargets the two dancers separately, using one motion as the robot reference and the other as a moving partner whose sparse keypoints guide footsteps without a prescribed velocity or trajectory.At deployment, a real person replaces the simulated partner, with corresponding keypoints measured from motion-capture markers.Source: §I · p. 2
- Multi-link compliance augmentation converts kinematic demonstrations into force-aware data by adapting the robot reference under structured forces at one or both hands.Temporal reversal, force-free samples, and clip chaining expand short demonstrations into longer interaction trajectories.Source: §I · p. 2
- Approximately 80% of the wrist displacement encoded by augmented demonstrations is reproduced in simulation, versus about 20% for a stiff tracking baseline; the humanoid also dances with a human partner across repeated forward and backward motions.The simulation result is reported alongside the physical demonstration of sustained two-hand dancing.Source: §I · p. 2
II. RELATED WORK
Prior work covers partnered robot dancing, humanoid motion imitation, and compliant control, but CoDance combines learned locomotion style, observed-partner following, and continuous two-hand compliance. Its method converts paired-dancing video into force-aware training data before training and deploying a policy with a real partner.
A. Partnered Dancing and Human-Humanoid Interaction
- CoDance differs from prior partnered-dancing work by learning locomotion style from paired human video while following an observed partner and maintaining compliant two-hand contact.Prior approaches include force-sensing partner following, contact-adaptive box steps, and hand-holding dances under velocity commands.Source: A. Partnered Dancing and… · p. 2
B. Whole-Body Motion Imitation for Humanoids
- Single-agent motion imitation keeps a humanoid near its reference, whereas CoDance preserves demonstrated lower-body motion while the upper body adapts to partner interaction.The cited approaches use reference tracking or adversarial objectives for natural motion.Source: B. Whole-Body Motion Imitation for… · p. 2
C. Compliant Control and Motion Tracking
- CoDance extends compliant motion tracking toward sustained partnered locomotion with force events on one or both wrists, coupled displacements, and directions informed by partner motion.The cited prior methods include impedance-reference tracking, joint position-force control, and upper-body compliance.Source: C. Compliant Control and Motion Tracking · p. 2
III. METHOD
- The method converts paired-dancing video into force-aware training data, trains partner following and two-hand compliance alongside locomotion, then replaces the simulated partner with a person at deployment.The overview depicts training with the decoupled policy.Source: §III · pp. 2–3, Figure 2 · p. 3
A. Data Pipeline
The data pipeline separately retargets two dancers from video, then augments the robot reference with feasible force-aware motion. The augmentation couples hand interactions to partner motion while accounting for balance.
A. Data Pipeline
- PromptHMR recovers both dancers’ SMPL-X motions and GMR retargets them separately; one becomes the robot reference and the other a kinematic partner proxy placed at a fixed following offset.The 9.7-second paired clip is also reversed to cover both travel directions; hand forces come from augmentation rather than video contact.Source: A. Data Pipeline · p. 3
- For each force event, the contacted link is displaced by force divided by stiffness, and whole-body IK adapts the reference while holding the feet, constraining the center of mass, and weakly tracking other links.If feasibility or self-collision checks fail, the event is recomputed with the wrist force directed away from the torso or reduced.Source: A. Data Pipeline · p. 3
- Two-hand events sample interaction intents that specify which wrists move and how their displacements relate, including common, mirrored, twisting, and opposing motions.Each wrist has independently sampled stiffness except for opposing motion, which shares stiffness across wrists.Source: A. Data Pipeline · p. 3
- Most force events follow the partner’s smoothed horizontal velocity, while the remainder oppose travel along the robot’s facing axis; near a still partner, direction is random.A one-time force scaling keeps the required center-of-mass shift within a prescribed budget while preserving the event profile.Source: A. Data Pipeline · pp. 3–4
- The dataset contains 200 clips with force events and 200 force-free clips, evenly divided between forward and reversed motion.Force-free clips use the free reference and resample commanded stiffness every 2 to 5 seconds.Source: A. Data Pipeline · p. 4
B. Policy Training
CoDance trains policies that combine partner following, compliant upper-body tracking, and a locomotion objective. The actor follows partner keypoints without a velocity command, while training supplies force-aware references and interaction signals.
B. Policy Training
- The actor observes partner pelvis and knee keypoints over the last second and commanded wrist stiffness, but not the adapted reference or applied force.It follows partner keypoints rather than a velocity command; its response to contact therefore relies on stiffness and proprioceptive motion.Source: B. Policy Training · p. 4
- During force events, wrist forces arise from springs toward setpoints, with environment stiffness sampled independently of robot stiffness.The setpoint delivers the scheduled force at the adapted reference, so deviations change the interaction force; an orientation spring applies zero scheduled torque.Source: B. Policy Training · p. 4
- The reward combines partner following, compliance, locomotion, and regularization, including foot targets with a fixed 0.5 m following offset and upper-body tracking of the adapted reference.Link tracking is re-anchored to torso position and heading, with higher tracking weight on the wrists and terms matching scheduled wrist forces and torques.Source: B. Policy Training · p. 5
- The decoupled policy uses an adversarial locomotion objective, while the whole-body tracker uses lower-body reference tracking; both train on force-aware references where available.The adversarial discriminator excludes arm states, and the tracker also uses global velocities and torso pose.Source: B. Policy Training · pp. 4–5
- Training chains clips at their ends, starts each new clip from the partner’s current position, and uses 20 s episodes with domain randomization.Randomization includes torso center-of-mass offsets, foot friction, and joint encoder bias; random pushes are disabled because scheduled interaction events perturb the robot.Source: B. Policy Training · p. 5
C. Hardware Deployment
The converged decoupled policy was deployed on a Unitree G1, using Vicon-tracked partner keypoints and direct hand forces during a two-hand hold.
C. Hardware Deployment
- The converged decoupled policy ran on a Unitree G1 at 50 Hz, with joint position targets sent through the SDK at 200 Hz.Both wrists used a commanded stiffness of 140 N m−1.Source: C. Hardware Deployment · p. 5
- Vicon tracked the person’s pelvis and knees and the robot pelvis to provide the partner keypoints used during training.Marker heights were calibrated once to match the simulated proxy.Source: C. Hardware Deployment · p. 5
- During the two-hand hold, physical forces came directly from the person’s hands.Both wrists used a commanded stiffness of 140 N m−1.Source: C. Hardware Deployment · p. 5
IV. EXPERIMENTS
The experiments evaluate CoDance in simulation across four behaviors and then test partnered dancing on the physical robot.
IV. EXPERIMENTS
- Simulation evaluation covers reactive partner following, compliant interaction, locomotion style, and transitions across chained clips.Targeted ablations isolate components responsible for each behavior.Source: §IV · p. 5
- Targeted simulation ablations isolate the components responsible for the evaluated behaviors.Source: §IV · p. 5
- The evaluation also tests partnered dancing with a human on the physical robot.Source: §IV · p. 5
A. Experimental Setup
The evaluation compares partner-following, compliance, and locomotion-style objectives across simulation trials, with success and tracking metrics defined for each task.
A. Experimental Setup
- Policies used the same half of augmented clips for training, with the remaining clips generated independently using newly sampled event timing, intents, stiffness commands, and forces.Trials averaged three seeds with 400 trials per seed and ended at specified torso-height, orientation, or partner-relative foot-distance thresholds.Source: A. Experimental Setup · p. 5
- The evaluation compared Decoupled and Whole-body policies with baselines that removed foot-following, force-aware wrist tracking, or the adversarial locomotion objective.The stiff tracker followed the free reference under the same force events, without force matching or tighter wrist-tracking terms.Source: A. Experimental Setup · p. 5
B. Evaluation Metrics •
- Partner-following metrics measured foot distance to its target, pelvis-to-pelvis distance against a 0.5 m target, and completion without termination.Foot error was averaged over both feet and time before termination or read at 1 s; success rate covered a clip or clip chain.Source: B. Evaluation Metrics • · p. 5
- Compliance metrics compared policy wrist displacement with the adapted reference and tracked force error and torso-relative upper-body error; locomotion style used lower-body tracking error.Long-horizon evaluation reported success rate and foot error over two-clip chains.Source: B. Evaluation Metrics • · pp. 5–6
C. Reactive Partner Following
The evaluation tests whether policies follow observed partner motion and respond compliantly to interaction forces. Compliant policies reproduce much of the augmented wrist motion, while partner-observation and foot-following tests show the conditions needed for partner-conditioned following.
C. Reactive Partner Following
- Masking partner keypoints caused all trials to fail with about 0.42 m foot error after 1 s; reversing the partner yielded about 0.26 m error and roughly one-quarter clip completion.The reversed condition avoided the out-of-training-distribution zero observations, though the partner motion still mismatched the robot reference.Source: C. Reactive Partner Following · p. 6
D. Compliance Interaction
- On seen clips, compliant policies moved their wrists about 80% as far as the adapted reference, with 2.6 cm mean displacement error; the stiff tracker moved about 20% as far and had more than twice the error.On unseen clips, compliant-policy displacement error grew by less than 1 cm, and the stiff tracker remained farthest from the adapted reference on both measures.Source: Figure 3 · p. 6, D. Compliance Interaction · p. 6
C. Reactive Partner Following
- Without the foot-following constraint, the whole-body tracker finished nearly every trial but had 2.5 times the constrained tracker’s foot error after 1 s.Together, the observation and constraint results indicate that partner-conditioned locomotion depends on both observed partner motion and the foot-following objective.Source: C. Reactive Partner Following · p. 6
E. Locomotion Style
The locomotion objective determines the learned lower-body style, while partner following and compliance change little when that objective is changed or removed. Clip-chaining supports following across unseen transitions between travel directions.
E. Locomotion Style
- The whole-body tracker tracked the lower body most closely, with tracking errors and foot error similar to the decoupled policy; removing its adversarial term caused large lower-body errors at comparable foot error.Source: E. Locomotion Style · p. 6, Table III · p. 7
- Without the adversarial term, the pelvis turned away from the dancer’s heading, the waist and hips twisted toward travel, and the stance widened into a crouch.The visualization also shows that separate retargeting does not preserve relative position in the paired motion clip.Source: Figure 4 · p. 7
- Removing the adversarial reward raised lower-body rotation error 2.6 times on force-event clips and 3.0 times on force-free clips, while partner following remained comparable.Across objective settings, foot error, force error, and upper-body tracking error stayed within about 1 cm, 0.01 rad, and 1 N of one another.Source: E. Locomotion Style · p. 7, Table III · p. 7
F. Benefit of Clip-Chaining
- Policies trained with clip-chaining succeeded in nearly every trial across unseen two-clip chains, while the policy trained without it succeeded in about half and failed at the switch while still following.Training without clip-chaining caused a large success-rate drop at comparable foot error.Source: F. Benefit of Clip-Chaining · p. 7, Table IV · p. 7
G. Hardware Results
Across three hardware runs, the robot maintained two-hand contact and followed its partner without falling. The study demonstrates sustained dancing but remains limited to one video and dance, with a fixed-direction following offset.
G. Hardware Results
- The standing policy’s arms moved with hand-applied forces while its feet stayed still.The demonstrations include one hand loaded by a hanging weight and both hands pulled in different directions or outward.Source: G. Hardware Results · p. 7, Figure 6 · p. 8
- 93%, 83%, and 73% of the three runs stayed within 0.1 m of the 0.5 m target, with two-hand contact and no falls.One run lasted 12.8 s; two runs lasted 38.9 s and alternated forward and reversed clips.Source: G. Hardware Results · p. 7, Figure 5 · p. 8
V. CONCLUSION, LIMITATIONS, AND FUTURE WORK
- CoDance learns partnered dancing from one video, converting kinematic demonstrations into force-aware data for partner-following locomotion and compliant two-hand interaction.The learned policies preserve the demonstrated locomotion style.Source: §V · p. 8
- The study covers one video and one dance; its fixed-direction following offset excludes dances that change the partners’ relative placement.The simulated partner replays fixed motion, and only about a quarter of trials finish a clip when the partner moves opposite to it.Source: §V · p. 8
APPENDIX
The appendix summarizes the trained policies’ reward terms, observations, randomization, force replay, and episode settings.
APPENDIX
- Tables V–VII document policy rewards, observations, and training settings, including randomization, force replay, and episode configuration.The listed settings apply to both trained policies.Source: APPENDIX · p. 10, Table V · p. 10, Table VI · p. 10
- The policies share all reward terms except locomotion style: one uses an adversarial term, while the other tracks the adapted reference’s lower body.The latter is described as whole-body tracking in the style of SoftMimic.Source: Table V · p. 10
- Both policies use the same observations; their MLPs have hidden layers of 512, 256, and 128 units with ELU activations.The AMP discriminator additionally reads two frames of 63 lower-body and torso features that the policy does not see.Source: Table VI · p. 10