Learning Force-Reactive Humanoid Loco-Manipulation

• Learned humanoid loco-manipulation commonly tracks motion references, but a demonstrated hand trajectory alone does not reproduce force, which also depends on contact geometry, stiffness, and object motion.Humanoids must coordinate hands, torso, and legs while staying balanced under contact loads.Source: §I · p. 1
• FLORE collects robot-free demonstrations that record contact force alongside hand, torso, and foot motion using handheld grippers with six-axis force/torque sensors and wearable trackers.The robot uses the same gripper–sensor assembly.Source: §I · p. 2
• The force-reactive diffusion policy uses wrist images, robot state, and measured hand wrenches to predict action chunks and per-hand desired forces for the controller.Force is both an input for reacting to interactions not revealed by images and an output passed to the controller.Source: §I · p. 2
• A hybrid-compliance whole-body controller uses wrist force-error feedback to shift hand position targets until commanded forces are reached, while yielding along uncommanded axes.It is trained in simulation and deployed with wrist force/torque measurements.Source: §I · p. 2
• 80% of rollouts succeeded with FLORE, versus 26.7% with a position-only policy trained on the same demonstrations and run with the same controller.Withholding the force command reduced success to 36.7%, while removing the force observation reduced it to 66.7%.Source: §I · p. 2
• In simulation and on a Unitree G1, FLORE is evaluated on three contact-rich tasks: pushing a gantry, bimanual box rotation, and board wiping.The controller realizes about 1 N of in-contact force error per hand while tracking whole-body motion.Source: §I · p. 2

A. Force Sensing for Loco-Manipulation Data Collection and Policy Learning

• FLORE records six-axis hand wrenches together with torso motion using handheld devices, without requiring robot-side demonstration data.Prior humanoid interfaces either omit contact force or, in WT-UMI, measure only scalar palm-normal force and rely on paired teleoperation data.Source: A. Force Sensing for Loco-Manipulation… · p. 2

B. Force for Low-Level Control

• FLORE's hybrid-compliance whole-body controller integrates measured wrist force error so one learned policy tracks position in free space and force along commanded directions while remaining compliant elsewhere and maintaining balance.Prior humanoid controllers typically withstand or yield to external force rather than directly specifying pressing force; direct tracking has mainly been studied on quadrupedal manipulators.Source: B. Force for Low-Level Control · p. 2

III. METHOD

• FLORE pairs a robot-free system recording vision, motion, and contact wrenches with a force-reactive diffusion policy that predicts keypoint motion and per-hand force commands for the controller.The overview describes the policy taking images, state, and measured force as input and producing motion and desired force at 30 Hz.Source: §III · p. 2, Figure 2 · p. 3

A. Hybrid Compliance Whole-body Controller with Force Tracking Capability

• The controller’s joint-position outputs are trained with a reward combining motion tracking and hand-position, TCP, orientation, force, contact, approach, and force-rate terms.Force-related rewards are weighted during contact, while the approach reward is weighted outside contact.Source: A. Hybrid Compliance Whole-body… · pp. 3–4
• Force feedback moves each hand’s position target until measured force balances the command, without requiring knowledge of contact stiffness.The command first shifts the target into the surface; an integral of measured force error then adjusts the shift, with contact gating and clipping limiting its accumulation.Source: A. Hybrid Compliance Whole-body… · p. 3
• Along uncommanded axes, measured external pushes move the target with the hand, allowing it to yield while tracking force on commanded axes without switching modes.The integral is bounded, so compliance holds within that bound.Source: A. Hybrid Compliance Whole-body… · pp. 3–4
• Training uses a virtual compliant plane with randomized stiffness and orientation; its frictionless contact means training force commands are purely normal, unlike tangential robot loads.The plane reacts only to penetration, and penetration and force are capped; tangential loads are not seen in training.Source: A. Hybrid Compliance Whole-body… · p. 4

B. Force-annotated Robot-free Data Collection System

• Each handheld device combines an F/T sensor, wrist camera, pose tracker, and jaw-width encoder, while chest and ankle trackers capture body motion.The robot uses the same gripper-sensor assembly, preserving sensor frame, sensor-to-TCP transform, and gripper geometry between collection and deployment.Source: B. Force-annotated Robot-free Data… · p. 4
• Wrenches are retained in the sensor frame, gravity-compensated with per-device payloads, and tared at recording start; robot inertial loads are also compensated.Source: B. Force-annotated Robot-free Data… · p. 4
• Images, tracker poses, wrenches, and gripper widths arrive at different rates and are timestamped on one host clock; missing wrench readings and tare transients are masked rather than zero-filled.Masking avoids treating a tared sensor’s free-space zero reading as a missing-data value.Source: B. Force-annotated Robot-free Data… · p. 4

C. Force-Reactive High-level Policy Learning

• The policy maps observations to chunks of future actions and augments both inputs and outputs with per-hand contact wrenches.Inputs include both wrist images, measured wrenches, and a low-dimensional state; each action step includes hand poses, widths, force commands, and torso and foot poses.Source: C. Force-Reactive High-level Policy… · p. 4
• The low-dimensional state encodes relative hand and torso positions, hand orientations in the pelvis frame, and gripper widths, omitting absolute poses and joint angles.Relative positions make the state invariant to absolute translation because demonstrations and robot execution differ in absolute poses and joint angles.Source: C. Force-Reactive High-level Policy… · p. 4
• Anchor-relative keypoint poses make action chunks invariant to world frame and demonstrator location, and predicted forces become controller commands after chunk decoding and re-basing.The chunks are streamed sequentially or asynchronously with temporal ensembling; wrench histories are encoded alongside image and state features to condition a temporal U-Net denoiser.Source: C. Force-Reactive High-level Policy… · p. 5

IV. EXPERIMENTS

• The evaluation targets whole-body force tracking, sustained contact, bimanual coordination during balance and walking, and robustness to image-invisible force variation.Source: §IV · p. 5
• FLORE’s policy observes measured wrenches, predicts demonstrated forces, and sends them to the controller; ablations remove the force command or wrench input separately.Without wrench input, the policy still commands force but cannot sense whether contact is made or lost.Source: §IV · p. 5
• Figure 5 depicts simulated pressing against randomly placed virtual planes while maintaining balance, and real-robot hands yielding when pulled.Its arrows distinguish commanded, measured reaction, external, and perturbation forces.Source: Figure 5 · p. 5
• The position-only diffusion-policy baseline uses the same architecture, data, and schedule as FLORE but has no force input or output and runs with force commands disabled.Each task has one trained policy and 10 evaluation rollouts, with test conditions paired across policies.Source: §IV · p. 5

Hybrid Compliance Whole-body Controller

• The controller uses K = 200 N/m, KI = 0.5 s−1, Imax = 40 N, and 2.0/1.5 N contact onset/release thresholds.It uses a 50 ms low-pass time constant, samples Kenv from LogUniform(100, 800) N/m, caps penetration at 8 cm and contact force at 35 N, and uses command pulses up to 25 N.Source: A. Implementation Details · p. 5

Policy

• The policy encodes two 224 × 224 wrist images with a CLIP-initialized ViT-B/16 and processes the 14-dimensional state with rotation features from the first two rotation-matrix rows.Its 1D U-Net uses 16 DDIM sampling steps; the image encoder has a 10× lower learning rate, and policies train for 105 steps with H = 50 (1.67 s).Source: A. Implementation Details · p. 5

Hybrid Compliance Whole-body Controller

• Reward kernels use widths of 0.1 m for TCP position, 0.25 rad for hand orientation, and 25 N for force; contact is rewarded below 35 N.The contact reward is rcontact.Source: A. Implementation Details · p. 5

Simulation protocol

• Simulation tests used Isaac Lab training and MuJoCo evaluation, with motion tracking on ROOT-90 and VLK-Pick-100 and force commands across multiple magnitudes, axes, and hands.Metrics include global root and wrist pose errors, force gain, and hold-phase in-contact error.Source: B. Hybrid Compliance Whole-body… · pp. 5–6

Simulation results

• FLORE tracks 5–25 N commands with gain 1.03–1.06 and about 1 N in-contact error per hand, versus SONIC + admittance’s 0.67–0.70 gain and about 7.9 N error.SONIC + admittance acts outside the balance policy; neither SONIC nor FLORE w/o Force accepts a force command.Source: Table I · p. 6, B. Hybrid Compliance Whole-body… · p. 6
• Adding force tracking to FLORE w/o Force adds 1.3–3.2 cm global root error and at most 8.0 mm wrist position error, while wrist orientation error roughly halves.FLORE tracks orientation better than SONIC, at a higher root error.Source: Table I · p. 6, B. Hybrid Compliance Whole-body… · p. 6

Real-world force tracking

• On the real robot, x-axis steady-state force reaches approximately 91% of command with about 1.1 N mean absolute error and a 1.8 s average time to enter the ±1 N band.The force-gauge staircase used 4, 6, 8, and 10 N commands across directions and hands; steady-state fluctuation was approximately ±0.3–0.5 N, with no loss of stability.Source: B. Hybrid Compliance Whole-body… · p. 6

Task

• The task requires both hands to sustain contact and balance forces while the legs and torso push, because unequal pushes make the gantry drift from its heading.The hanging humanoid swings as the gantry rolls, changing resistance over time.Source: C. Push Gantry · p. 6

Performance

• In the robot rollouts, the position-only baseline either fails to move the gantry or pushes unequally and drifts from its heading.Fig. 6’s top panel shows commanded and measured force traces for each hand and axis.Source: Figure 6 · p. 7
• Ours reaches the gantry target in 7/10 rollouts, compared with DP’s 2/10, w/o F-cmd’s 3/10, and w/o F-in’s 6/10.The trials varied two gantry loads and five initial poses; success required reaching the target without a stop.Source: C. Push Gantry · pp. 6–7
• Without force input, the policy may stall against the heavier load or keep walking after losing contact with the lighter one; offline pose and force errors are 0.82 cm and 0.77 N per hand.Withholding the force command yields 3/10 successes, while removing force input yields 6/10.Source: C. Push Gantry · p. 7

Task

• The robot must rotate the box by about 50° and drag it about 0.5 m while keeping it upright after release.Success requires more than 40° of rotation and an upright box after release.Source: D. Rotate Box · p. 7

Capabilities

• The pinch shares internal force around a common rotation axis, and the demonstrated right hand applies about 1.4× the left hand’s force.If one hand leads, the box tips; friction changes with box mass and when sliding begins.Source: D. Rotate Box · p. 7

Performance

• Ours succeeds in 9/10 rollouts, compared with DP’s 3/10, w/o F-cmd’s 4/10, and w/o F-in’s 7/10.Ours has an offline pose error of 0.70 cm; trials vary two box masses and five initial yaw angles.Source: D. Rotate Box · pp. 7–8
• Without force commands, the predicted squeeze is not executed; without force input, the policy cannot react when the box begins to slide.DP either fails to drag the heavy box or leans into it and loses balance.Source: D. Rotate Box · pp. 7–8

E. Wipe Board

• 8/10 wipe-board rollouts succeed with FLORE, compared with 3/10 for DP; without force commands, success is 4/10.Without force input, success is 7/10, but the policy continues wiping in free space when the board is farther away.Source: E. Wipe Board · p. 8

F. Findings Summary

• Adding the force channel increases root error by 1.3–3.2 cm and wrist position error by at most 8.0 mm, while halving wrist orientation error.The controller tracks commanded force with a gain of 1.03–1.06 and about 1 N of in-contact error; lateral and downward commands remain weaker.Source: F. Findings Summary · p. 8
• Removing predicted force commands reduces success from 80% (24/30) to 36.7% (11/30; p = 0.001), while removing force observations reduces it to 66.7%.The findings attribute the larger, significant gain to force commands and report a smaller trend for force observations.Source: F. Findings Summary · p. 8

V. CONCLUSION

• FLORE uses robot-free demonstrations recording motion and contact wrench, and its learned force commands support sustained contact, bimanual coordination, and applying force while walking and balancing.The conclusion frames force as a shared interface between demonstrations, policies, and whole-body control.Source: §V · p. 8

F. Findings Summary

• 80% of rollouts succeed with FLORE, compared with 26.7% for a position-only policy across three contact-rich tasks.The comparison uses the same demonstrations and controller.Source: §V · p. 8, F. Findings Summary · p. 8