Source-linked AI summary
FWBC-VLA: Force-Aware Whole-Body Compensation for Contact-Rich Loco-Manipulation
Yutian Zhang, Siyuan Ma, Liwen Yang, Yang Li, Ce Hao, Haozhen Chi, Dong Wei, Qiaojun Yu, Dibo Hou
TL;DR
Contact-rich loco-manipulation lacks a bridge between semantic action generation and physical interaction control, especially on platforms without force/torque sensors. FWBC-VLA introduces sensorless residual-torque feedback for both VLA action adaptation and whole-body compensation, and real-world experiments demonstrate improved performance relative to baselines. Its force representation remains proxy-valued, and the impact of estimation accuracy on task performance requires further investigation.
Problem
Existing VLA and whole-body control systems do not jointly interpret task-relevant contact forces and body disturbances, while retrofitting force/torque sensors adds hardware cost and integration effort.
Method
FWBC-VLA uses HSR-Force to estimate residual-torque-based contact states, injects force tokens into the VLA action expert, and generates compensation actions from proprioception and Jacobian-derived wrenches.
Results
Real-world experiments show that explicit interaction feedback and whole-body compensation improve performance relative to baselines in contact-rich loco-manipulation.
Takeaways & Limitations
FWBC-VLA couples task-level manipulation adaptation with whole-body stabilization using sensorless interaction feedback for wheeled-legged robots.
Takeaways & Limitations
The residual contact descriptor is proxy-valued and its scale depends on the estimator and sampling configuration; the effect of estimation accuracy on task performance remains unresolved.
Abstract
from arXiv · showhide
Contact-rich loco-manipulation requires a bridge between semantic action generation and physical interaction control. Existing Vision-language-action (VLA) models generate task-level actions from visual and linguistic observations, but cannot interpret the physical interactions induced by those actions. While the whole-body control (WBC) policy can stabilize the robot, it cannot distinguish task-relevant interaction forces from forces induced by external disturbances during manipulation. Although force/torque sensors provide direct measurements of physical interactions, retrofitting them entails additional hardware costs and substantial integration effort, particularly for platforms not designed with sensor integration in mind. To address this problem, we propose FWBC-VLA, a force-aware framework that bridges task-level VLA action generation and low-level whole-body compensation control for wheeled-legged robots. First, we introduce HSR-Force, a sensorless residual-torque estimator for inferring contact strength and its temporal variation. These contact estimates are then encoded as tokens and injected into the VLA action expert during action decoding, enabling the policy to perceive contact onset, sustained loading, and release. For loco-manipulation tasks, all parameters of the pretrained VLA backbone are fine-tuned on our WL\&Arm Dataset, which comprises more than 5,000 episodes. Moreover, the robot's proprioceptive state, the Jacobian-derived body-frame force estimate, and the estimated contact state are jointly fed into a compensation generator to produce corrective actions. The manipulation-centric actions are subsequently combined with the corrective actions and passed to the WBC policy for execution. Real-world experiments on whiteboard wiping and door opening with a door closer demonstrate the effectiveness of our FWBC-VLA in contact-rich loco-manipulation.
1 Introduction
Contact-rich loco-manipulation requires explicit physical-interaction awareness because end-effector forces can perturb the robot body. FWBC-VLA addresses this gap with sensorless force feedback that jointly adapts task actions and whole-body compensation.
- End-effector forces can propagate through the arm and perturb the body, requiring task actions to coordinate with whole-body stabilization.
- Existing modular policies exchange mainly kinematic commands, limiting closed-loop feedback about contact onset, sustained loading, and release.
- Arm-centric force-aware methods improve contact-rich manipulation but generally rely on dedicated force/torque sensors and overlook body-level effects.
- FWBC-VLA uses HSR-Force to estimate contact from synchronized arm, leg, and base proprioception without dedicated physical sensors.
- The framework encodes interaction histories as force tokens for VLA action generation and uses Jacobian-derived wrench estimates to trigger whole-body compensation.
2 Related Work
Prior work adds force feedback, tactile sensing, or whole-body VLA control, but existing approaches remain limited in combining direct interaction awareness with body-level compensation.
- Force-aware VLA methods enhance manipulation stability and precision but typically treat force as arm-control feedback while overlooking effects on the robot body.
- Tactile and modality-fusion methods improve robustness under occlusion, yet tactile-derived force estimates remain indirect and susceptible to noise.
- Traditional sensorless force estimators require extensive system modeling and engineering effort and can struggle with actuator dead zones and torque ripple.
- Whole-body loco-manipulation controllers combine task planning or visual instruction with WBC, extending VLA control beyond isolated manipulation or base motion.
3 WL&Arm Dataset
The WL&Arm Dataset addresses missing force-aware data for wheeled-legged quadruped loco-manipulation through hybrid position-force teleoperation and force-intent annotations. It contains more than 5,000 episodes spanning principal loco-manipulation tasks.
- Contact-aware datasets for wheeled-legged quadruped loco-manipulation remain scarce, motivating a feasible data-collection solution.
- Conventional teleoperation records motion without operator force intent, while wheeled-legged robots differ from most VLA pretraining embodiments in base and manipulator action spaces.
- The Pico-based hybrid position-force system specifies tangential and normal end-effector force commands as contact feedforward terms and logs them as metadata.
- The WL&Arm Dataset contains more than 5,000 teleoperation episodes, with 200-Hz joint data and 15-Hz loco-manipulation demonstrations.
- Bottle pick-and-place, whiteboard wiping, and door opening comprise 41%, 25%, and 21% of the principal task subsets, respectively.
4 Method
FWBC-VLA uses sensorless interaction representations to condition VLA action generation and guide whole-body compensation. The framework estimates residual interaction from proprioception, encodes causal histories as force tokens, and sends bounded corrective commands alongside nominal actions to WBC.
- Framework overview: FWBC-VLA shares estimated interaction representations between interaction-conditioned VLA action generation and task-relevant whole-body compensation.The downstream WBC remains responsible for balance regulation and low-level command tracking.
- Sensorless interaction estimation: The estimator decomposes measured arm torque into free-motion dynamics, environment-induced residual interaction, and unmodeled effects rather than directly calibrated external force.The residual is treated as a latent interaction variable.
- Sensorless interaction estimation: Two synchronized 200-Hz experts predict free-motion torque, with a fixed gate favoring the history expert during free motion and the state expert during interaction.The history expert uses causal torque-and-motion history, while the state expert uses current arm, base, leg, and IMU state without torque history.
- Causal physical interaction representation: A two-channel contact descriptor records estimated interaction strength and its temporal trend, distinguishing increasing loading from unloading or release.Its scale is proxy-valued and depends on the residual estimator and sampling configuration.
- Interaction-conditioned action generation: A causal 13-sample interaction history is encoded by normalization and a GRU into force tokens, then fused into the VLA action expert through gated cross-attention.At 200 Hz, the history spans 60 ms and uses only measurements available before the query timestamp.
- Whole-body compensation generation: The compensation branch combines body-frame load, IMU state, posture deviation, and nominal base intent to generate bounded planar-velocity and yaw-rate residuals for WBC.Deadbanding, filtering, slew-rate limits, clipping, and invalid-signal handling preserve a bounded controller interface.
5 Experiments
The experiments evaluate FWBC-VLA through force estimation, ablations, and real-world contact-rich loco-manipulation tasks. They ask how accurately HSR-Force estimates interaction without force/torque sensors, how its components affect estimation, and how force-aware control contributes to task performance.
- The evaluation covers estimation accuracy, estimator-component effects, real-world loco-manipulation performance, and component contributions.
5.1 Force-Estimator Evaluation
The force-estimator evaluation compares HSR-Force with learned and model-based alternatives using force-error, torque-residual, and contact-detection metrics. HSR-Force performs strongly across zero-load, known-load, and temporal contact-estimation tests, while dual-head components provide complementary benefits.
- HSR-Force is compared with NEXT, GMO-SI, and DF-MLP using N-MAE, torque-residual errors, and contact-detection AUC.The evaluation uses 25 manually reviewed episodes; DF-MLP is omitted from torque-residual metrics because it does not output external joint torque.
- HSR-Force achieves the lowest zero-load force error, highest Touch AUC, and reduced static and dynamic tail residuals.
- HSR-Force responds near contact onset and remains active during sustained door loading in video-aligned estimation.
- Shared Dual-Head Ablation: Torque history reduces free-motion residual error during motion, while the history-free state head is more sensitive to contact onset.
- Shared Dual-Head Ablation: Fixed-gate fusion yields the best Touch and Door-phase AUCs, while known-load results show HSR-Force leading dynamic error and GMO-SI leading static error.The ablation fusion also provides the best known-load errors overall while retaining competitive zero-load residual errors.
5.2 Experiments in the Real World
Real-world evaluations on whiteboard wiping and door opening show that FWBC-VLA maintains higher success through contact-intensive stages than the compared baselines.
- Evaluation setup: The benchmark evaluates whiteboard wiping and door opening, including a door-closer condition, across sequential loco-manipulation stages.The primary metric is success rate, with stages covering approach, grasp or pick, manipulation, placement or pushing, and leaving or crossing.
- Results: FWBC-VLA achieves the highest reported success rate at every stage on both tasks.Its final-stage advantage reaches 40 percentage points on both whiteboard wiping and door opening relative to the cited strongest baselines.
- Results: 76% at Wiping and 64% final success are reported for whiteboard wiping, exceeding ACP and ForceVLA by 32 and 40 percentage points, respectively.These comparisons are stage-specific: the Wiping result is compared with ACP, while final success is compared with ForceVLA.
- Qualitative observations: Baseline rollouts show uneven board contact, residual ink, and body drift, whereas FWBC-VLA maintains task progression.The reported observations support explicit interaction feedback for sustained contact in loco-manipulation.
5.3 Ablation of Force-Conditioned Compensation
Ablations show that force-aware interaction conditioning improves manipulation success, while bounded compensation adds substantial gains under sustained external loading.
- Force-aware interface: Adding the force-aware interface increases average success from 12.0% to 35.0% across contact-critical stages.The interface combines HSR-Force interaction estimation with force-token conditioning in the action expert.
- Bounded compensation: Adding bounded body compensation further raises average success to 59.5%, the largest incremental gain of 24.5 percentage points.All variants use the same task data, training budget, initial-condition distribution, and evaluation protocol.
- Sustained-load stages: With a door closer, success increases by 52 percentage points when pushing the door, while board cleaning improves by 44 percentage points.The benefit of bounded compensation is concentrated in sustained high-load stages.
- Stage-specific effects: Handle pressing and door pushing without a closer remain nearly unchanged after adding bounded compensation.This contrasts with the larger gains observed in sustained high-load stages.
6 Conclusion and Future Work
The paper concludes that FWBC-VLA couples sensorless interaction estimation with VLA action generation and whole-body stabilization, while identifying force-estimation accuracy as an open issue.
- Conclusion: FWBC-VLA provides a sensorless force-aware feedback framework for contact-rich loco-manipulation.Its design targets a shared interaction representation for VLA action generation and whole-body compensation.
- Interaction estimation: HSR-Force combines historical and current states, summarizes residual torque into contact descriptors, and projects loads into end-effector and body-frame proxies.These signals condition the VLA action expert and compensation sidecar.
- Empirical outcome: Real-world experiments show improved performance relative to baselines when explicit interaction feedback and whole-body compensation are used.The conclusion links the two mechanisms to the reported experimental improvement without separating their individual contributions.
- Limitations and future work: The impact of force-estimation accuracy on task performance remains to be investigated.Future work also targets broader contact-aware perception and tighter whole-body VLA integration on humanoid and wheeled-base robots.