Source-linked AI summary
HumanCLAW: Can Vision-Language Models Act Through a Body?
Li Siyao, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, Michael Zollhoefer, Sizhe An, Abhay Mittal, Amy Zhao, Ranjay Krishna, Manling Li, Ziwei Liu, Chuan Guo
TL;DR
Evaluating whether vision-language models can choose executable actions through physical bodies is difficult when motor failures obscure decision quality. HumanCLAW decouples action decisions from low-level execution and tests nine VLMs on long-horizon embodied tasks, finding that none solves the benchmark and the best succeeds in only 16.8% of episodes.
Problem
It remains unclear whether a VLM given an egocentric view, action history, and goal can choose an executable action at each moment without motor execution confounding evaluation.
Method
HumanCLAW lets an untrained off-the-shelf VLM issue atomic whole-body skills while a decoupled motion generator executes continuous physical motion.
Results
None of nine frontier VLMs solves the benchmark; the strongest completes the full find–navigate–interact progression in 16.8% of episodes.
Takeaways & Limitations
The results identify embodied self-awareness—not target recognition—as the bottleneck, especially tracking body position, arrival, and collisions.
Takeaways & Limitations
The small interaction vocabulary, skill-granularity dependence, absent balance and motor tracking, and missing tactile channel limit the framework’s scope.
Abstract
from arXiv · showhide
Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, are factored out. What remains measurable is the model's action intelligence: its moment-to-moment choice of what the body should execute next. Based on this framework, we build HumanCLAW-Bench: 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes. We test nine state-of-the-art VLMs and find that none solves the benchmark; the best model reaches only a 16.8% success rate. Recognizing the target is not the bottleneck. What current VLMs lack is embodied self-awareness: they lose track of their own body, failing to tell where it is, whether it has reached the goal, or whether it has hit an obstacle.
1 Introduction
HumanCLAW evaluates VLM action intelligence by decoupling whole-body action decisions from low-level motor execution while preserving physical consequences. On HumanCLAW-Bench, none of nine frontier VLMs reliably solves the 1,218-episode benchmark, with failures concentrated in embodied self-awareness rather than target recognition.
- Motivation: HumanCLAW tests action intelligence as closed-loop whole-body decision-making: selecting, parameterizing, and sequencing actions as their physical consequences unfold.The framework treats action intelligence as the operational component of spatial intelligence.
- HumanCLAW: HumanCLAW decouples VLM decisions from motor execution by converting each atomic skill into continuous whole-body motion with physical consequences.An off-the-shelf VLM reasons from egocentric observations and text history, then outputs a verified atomic skill with parameters at each sub-second step.
- HumanCLAW-Bench: 1,218 episodes across 41 indoor houses form HumanCLAW-Bench, requiring whole-body find-navigate-interact completion through staged, progressive metrics.Each episode asks the agent to find a target object, bring the body to it, and sit on it.
- Results: 16.8% of episodes is the strongest model’s completion rate for the full find–navigate–interact progression, and none of nine frontier VLMs reliably solves the benchmark.The benchmark evaluates whether current VLMs can carry an abstract, long-horizon task through completion.
- Failure analysis: Target recognition is not the bottleneck; failures instead reflect embodied self-awareness deficits involving body placement, arrival detection, and motion-induced collisions.The strongest model’s reported target-seeing rate comes within 5 percentage points of objectively measured visibility once the target is rendered in view.
2 HumanCLAW
HumanCLAW evaluates embodied action intelligence by having a harnessed VLM choose atomic, parameterized skills in a closed loop, while a decoupled motion generator and physical simulator execute them. Its contextual harness structures egocentric perception and hierarchical reasoning without turning the policy into an open-loop planner.
- HumanCLAW loop: At each sub-second timestep, egocentric RGB observations and textual history are mapped to one parameterized atomic skill call.The call specifies a skill s_t and continuous parameter c_t.
- HumanCLAW loop: A motion generator realizes each skill as a sub-second future motion chunk, and a physical simulator executes it, updates the world, and returns the next observation.Execution is conditioned on the current body state and includes physical interaction with the environment.
- Contextual reasoning harness: The contextual harness guides the VLM from egocentric spatial perception through high-to-mid-to-low reasoning before proposing a low-level motion skill.The scaffold includes explicit visual state, a mid-level objective, low-level skill reasoning, and a skill-specific verifier.
- Contextual reasoning harness: The policy remains closed-loop because the VLM decides at every step whether to inherit or revise its mid-level objective using new egocentric observations and feedback.This preserves hierarchical planning while keeping decisions reactive rather than open-loop.
- Atomic skill interface: Atomic skills give the VLM a finite, interpretable action interface while leaving the motion generator responsible only for realizing the selected primitive.Long-horizon interactions are composed from skills such as walking, turning, stepping backward, and sitting rather than exposed as one object-specific command.
- Physics and evaluation: The half-physics design attributes wall collisions, object disturbances, and premature stopping to action-level decisions while avoiding domination by low-level balance-control failures.The generated motion specifies intended body movement, whereas the simulator determines how that movement interacts with the physical world.
3 HumanCLAW-Bench
HumanCLAW-Bench evaluates long-horizon find-navigate-interact behavior through progressive indoor episodes requiring autonomous exploration, navigation, and interaction. It stratifies episodes by route geometry and measures success, action quality, body awareness, physical disturbance, and computational cost.
- Task design: Each episode instructs the agent to find an object, navigate to it with zero distance, and sit on it, with stages gated by prerequisite completion.The interaction is instantiated as sitting on a couch or bed, while the agent autonomously explores and chooses actions.
- Dataset: 1,218 episodes across 41 validation houses cover six target categories: chair, bed, couch, potted_plant, toilet, and tv.The sit subset contains 597 episodes involving beds, couches, and toilets.
- Difficulty analysis: Episodes are stratified by distance, choice, and obstacle, using fixed thresholds that yield roughly 23/46/31% easy, medium, and hard tiers.Distance measures geodesic route length, choice captures turns and rooms traversed, and obstacle measures route clutter.
- Evaluation metrics: Evaluation covers high-level success, low-level action quality, body awareness and physical disturbance, and computational cost.Metrics include motion jerk, collision steps, displaced movable objects and displacement, plus average tokens consumed per episode.
- Evaluation metrics: Staged success requires both objective completion and the agent’s subjective acknowledgment, with FindSR requiring a visible target semantic ID occupying at least 100 pixels.The benchmark defines progressive success so later subtasks are available only after earlier ones are completed.
4 Experimental Result
HumanCLAW shows that low-level skills execute reliably, yet no tested VLM solves the long-horizon benchmark: Gemini-3.1 reaches only 16.8% success. The main deficits lie in closed-loop reasoning and egocentric body awareness rather than target recognition or motor execution.
- Overall benchmark result: Low-level skills achieve commanded motions reliably, making analyzed failures attributable to planning, perception, and body awareness rather than motor execution.The shared motion prior operates zero-shot without per-task or per-scene tuning, while achievement ratios remain close to unity.
- Overall benchmark result: 16.8% is the best model’s success rate, while FindSR, NavSR, and InteractSR range from 32.6%–64.9%, 0.8%–42.4%, and 0%–16.8%, respectively.Gemini-3.1 is the best model, and four of nine models sit on the target in at most 0.2% of episodes.
- Reasoning scaffolds: Compact structured memory and mid-level objectives improve action intelligence, whereas longer text history saturates and excessive visual history harms performance.At img 10, NavSR falls from 27.0% to 13.0% and InteractSR from 18.9% to 3.8%; removing mid-level objectives collapses InteractSR from 18.9% to 0.0%.
- Reasoning scaffolds: Verifier removal reduces NavSR from 27.0% to 2.0% and InteractSR from 18.9% to 0.0%, while FindSR changes only from 58.0% to 51.0%.Without verification, episodes end earlier at 47.5 steps versus 78.5, indicating failure to close the action loop.
- Perception and failure causes: Recognition is usually successful once the target enters the ego view: GeoFindSR exceeds FindSR by only 5–10 points for the strongest models.The reported pairs are Gemini-3.1 69.9 versus 64.9, GPT-5.5 63.2 versus 55.1, and Gemma-4-31B 67.7 versus 58.1.
- Egocentric self-localization: 3,706 of 5,473 episodes (68%) fail navigation despite acknowledged target finding, with body awareness comprising 34% of these failures.Agents either continue after arrival or stop while far away because they misjudge body position, distance, or blockage relative to the target.
5 Related Work
Prior embodied environments and agentic systems either abstract physical execution, entangle decisions with low-level control, or evaluate spatial understanding open-loop. HumanCLAW instead queries frozen VLMs for egocentric skill decisions and realizes them as continuous whole-body motion with physical consequences.
- Embodied environments: Program-driven environments use predefined transitions or avatar animations, while physics-based simulators entangle spatial decisions with high-frequency motor execution.HumanCLAW is positioned as a middle ground between these two approaches.
- Navigation benchmarks: Object-goal and vision-and-language navigation typically abstract agents as moving cameras or velocity-controlled points and score location arrival, omitting full-body physical interaction.These settings leave out how a body arrives, orients, and interacts under physical consequence.
- Agentic systems: Hierarchical agents use MLLMs for decomposition, grounding, skill selection, and replanning, whereas end-to-end VLA policies couple reasoning to robot-specific data and low-level control.Frozen VLMs in first-person games remain limited to key presses whose outcomes are resolved by game scripts; action-reasoning models still use action-trained decision makers with execution in the loop.
- Benchmarking: Existing benchmarks primarily score plans, simulator actions, or robot trajectories, while HumanCLAW evaluates frozen off-the-shelf VLMs sustaining egocentric spatial action over long-horizon tasks.Skill-level decisions are realized as continuous whole-body motion in a physical world, preserving collision and contact consequences.
- Spatial intelligence: Spatial-intelligence evaluations report difficulties with metric distance, orientation, and multi-step spatial simulation, but mostly use open-loop question answering without environmental feedback.HumanCLAW frames action intelligence as the operational component of spatial intelligence.
6 Discussion
HumanCLAW separates VLM action decisions from balance and motor-tracking failures, revealing that current models recognize targets yet struggle with embodied self-awareness. HumanCLAW-Bench shows substantial remaining difficulty, while its interpretation depends on the skill vocabulary and half-physics abstraction.
- Framework: HumanCLAW isolates continuous whole-body action with real scene consequences while factoring out balance and motor-tracking failures.This separation targets the VLM’s moment-to-moment action choices rather than low-level execution errors.
- Findings: Across nine state-of-the-art models, none solves the benchmark, although perception is largely intact and rendered targets are almost always recognized.The discussion identifies decision-making through a body, rather than target recognition, as the central difficulty.
- Interpretation: Current VLMs lack embodied self-awareness because they treat their limbs as pixels rather than modeling proprioception and action consequences.The proposed remedies include persistent spatial memory, calibrated termination, and an internal model of the body and its spatial relations.
- Assumption: HumanCLAW assumes generalizable action intelligence comes from reasoning: a frozen general decision-maker composes tasks zero-shot over a fixed reusable motor prior.This contrasts reasoning-based extrapolation with policies fitted on trajectories, which generalize within their training-data support.
- Limitations and outlook: 16.8% of episodes are completed by the strongest model, while navigation alone peaks at 42.4%, leaving substantial headroom for embodied self-awareness.The benchmark’s limits include a small interaction vocabulary, vocabulary-dependent attribution, and deliberate abstraction of balance and motor tracking.
Appendix · A Skill-Specific Verifier Checks
The verifier uses fixed, skill-triggered checks to reduce long-context degradation by reassessing each proposed skill from the current egocentric view. Failed checks replace proposals with corrected skills from the same action pool.
- A Skill-Specific Verifier Checks: The verifier applies a fixed checklist triggered by the proposed skill.It runs only when one of the listed skills is proposed.
- A Skill-Specific Verifier Checks: Each check is answered from the current egocentric view alone.The compact, skill-specific prompt avoids relying on the growing rollout history.
- A Skill-Specific Verifier Checks: The verifier targets unreliable spatial-distance and relation estimates caused by long-context degradation.The planner’s estimates become less reliable as rollout history grows.
- A Skill-Specific Verifier Checks: The mechanism combines compact prompts with conditional correction of skill proposals.Together, these operations reassess the current view and modify failed proposals within the existing action pool.
- A Skill-Specific Verifier Checks: A failed check replaces the proposed skill with a corrected skill from the same action pool.This replacement rule is explicitly defined for every verifier check.
- A Skill-Specific Verifier Checks: Verifier rules fire only when their associated skill is proposed.The checks are skill-specific rather than continuously applied.
B Automated Root-Cause Attribution
Root-cause labels in Figure 8(b–d) are assigned automatically by a deterministic classifier that replays failed-episode logs without human annotation. The classifier evaluates navigation distance, mesh contact, egocentric visibility, actions, and the model’s stated visible state at each decision step.
- Automated attribution: All root-cause labels in Figure 8(b–d) are assigned automatically, with no human annotation.A deterministic classifier replays each failed episode’s logs.
- Automated attribution: The classifier evaluates pelvis-to-target geodesic distance on the scene navigation mesh at each decision step.This distance is measured from the pelvis to the nearest target instance.
- Automated attribution: It also evaluates pelvis–object mesh contacts, target semantic pixel counts in egocentric views, the action stream, and the model’s stated visible state.Contact is defined as exact mesh contact rather than bounding-box overlap.