Source-linked AI summary
Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs
Zeyu Lou, Yanhong Zeng, Yong Wang, Chenyang Si
TL;DR
A general-purpose VLM may recognize a task without knowing how a particular robot’s motion and functional parts produce the intended effect. KnowBody makes action-relevant body relations queryable and revisable from interaction while keeping the model frozen; across four real-robot tasks, it improves fixed-budget completion over the native harness.
Problem
General-purpose VLM task knowledge does not establish how a particular robot’s motions and functional parts produce intended physical effects.
Source: §1 · p. 1Method
KnowBody uses a partial, queryable body model initialized from an off-task trajectory and refined through interaction, while keeping VLM weights fixed.
Source: §1 · pp. 1–2, §3 · p. 4Results
Across four real-robot tasks, 75% completion for KnowBody versus 25% for the native Codex harness under fixed budgets.
Source: §1 · p. 2Takeaways & Limitations
The results support explicit, revisable body knowledge as an interface between frozen VLM reasoning, numerical action, and experience reuse.
Source: §7 · p. 9
Abstract
from arXiv · showhide
A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect. We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen. Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions. New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse. Across 32 fixed-budget trials on four real-robot tasks, initialized KnowBody achieves 75% completion versus 25% for the native harness and requires fewer planner rounds on successful trials in tasks completed by both. With persistent updates enabled, planner rounds decrease by 29-53% from the first to the fifth recorded success.
1 Introduction
KnowBody addresses the gap between recognizing a task and knowing how a particular robot’s motion and functional parts produce its effects. It makes body relations available for action selection and learning from interaction while keeping the VLM responsible for task reasoning.
1 Introduction
- Broad task knowledge does not establish how a particular robot’s motions and functional parts produce intended effects, such as how a commanded end-effector pose moves a held pen tip.Using interaction evidence for control requires identifying supported relations, applying them to actions, and revising them when observations reveal a mismatch.Source: §1 · p. 1
- KnowBody connects VLM action decisions to a partial, queryable body model that relates robot commands to physical effects and supports learning from their consequences.Execution feedback assesses task progress and tests the body relations used in each decision.Source: §1 · p. 1, Figure 1 · p. 1
- One off-task trajectory initializes supported relations; later evidence refines body estimates and conditional outcome knowledge, with dependent knowledge rechecked before reuse.Validated updates are retained across episodes to inform later decisions.Source: §1 · pp. 1–2
- Across 32 fixed-budget trials, KnowBody achieves 75% completion versus 25% for the native Codex harness; successful trials require 17–44% fewer planner rounds, and persistent updates reduce rounds by 29–53%.The planner-round comparison applies to tasks completed by both methods; persistent-update reductions compare the first with the fifth recorded success.Source: §1 · p. 2
2 Related Work
Related work connects general-purpose models to robot actions through harnesses, learned policies, explicit geometry, demonstrations, and experience reuse. KnowBody focuses on maintaining experience when its body-model basis changes.
2 Related Work
- Prior robot interfaces ground model proposals through skill values [1], sensor integration, control APIs, image-space actions, or predictive-control sequences [38].Other direct-control interfaces add geometric queries and runtime motion revision [21], while GPT-Policy uses demonstrations, interaction history, and execution feedback [8].Source: §2 · p. 2
- Trained visuomotor policies and VLAs learn or adapt action policies, whereas KnowBody uses an off-task trajectory to initialize body evidence without prescribing a new task sequence or training a policy.The matched evaluation therefore compares harnesses under the same VLM setting rather than trained VLA policies.Source: §2 · p. 3
- Prior work also reuses feedback and experience through memories, summaries, procedures, or verified repairs [21] [24] [29].KnowBody targets a specific dependency: corrected body geometry can change conditions inferred from unchanged historical interactions, so it recomputes and revalidates affected rules before reuse.Source: §2 · p. 3
3 Problem and Learning Protocol
The protocol keeps the VLM’s task reasoning and action proposals separate from robot execution, while initializing and revising body knowledge from interaction evidence.
3 Problem and Learning Protocol
- At each model–tool decision, the VLM receives the instruction, current images and available depth, measured pose, gripper state, and available joint feedback; a local executor performs proposed Cartesian segments and gripper commands.The VLM output may instead be a query, observation request, or termination request.Source: §3 · p. 3
- One continuous off-task episode initializes the body model by pairing observations with requested and achieved actions, alongside independently versioned geometric measurements.The body snapshot stores supported motion response, functional geometry, camera bindings, and component versions; interaction knowledge stores conditional outcomes and body dependencies.Source: §3 · p. 3
- The body model supports visual-direction queries, mapping the commanded frame to an attached tool tip, and checking continuous full-arm path feasibility using body relations and current joint and attachment state.A valid endpoint solution alone does not establish that the intervening motion is feasible.Source: Figure 3 · p. 4
- Model weights remain fixed, raw events remain immutable, and fixed evaluation blocks cannot use their own outcomes to update their snapshot.Paused continuations of one physical episode remain one statistical group; fixed-budget comparisons and continual sequences assess completion and repeated-task decision effort separately.Source: §3 · p. 4
4 Method
KnowBody couples a queryable partial body model with action selection and the interpretation of experience. Its representation, interaction, evidence checks, and revision process keep reusable knowledge tied to the body conditions that support it.
4.1 Body Representation and Initialization
- The partial body model exposes supported motion response, functional geometry, and arm kinematics to condition planner choices, while the executor retains final authority.Relations outside the evidence-supported domain return unknown rather than assuming a complete dynamics model.Source: §4.1 · p. 4
- One off-task trajectory initializes supported observation–motion relations rather than a task-specific sequence to imitate.Achieved translation is paired with before-and-after observations; independent measurements supply metric functional geometry.Source: §4.1 · p. 5
4.2 Body-Conditioned Interaction
- Body queries condition action choices by mapping desired image directions to robot translation and functional-part targets to control targets using supported geometry.When uncertainty prevents a relation from determining a condition, that condition remains unknown and can motivate further observation.Source: §4.2 · p. 5
- For long or multi-waypoint proposals, full-arm queries distinguish endpoint reachability from continuous-path feasibility, and fresh executor checks retain final authority.Unmeasured scene or attachment clearance remains unknown.Source: §4.2 · p. 6
4.3 Evidence Interpretation and Episode State
- KnowBody separately assesses whether requested motion occurred, whether observed response agrees with the body model, and whether the intended effect appeared.Occlusion or unobserved outcomes yield unknown, not no effect.Source: §4.3 · p. 6
- Interactions preserve observations, requested and achieved motion, execution status, and outcome provenance so interpretations can change without altering historical evidence.Current observations update scene, attachment, and progress estimates but do not replace an earlier action’s direct after-observation.Source: §4.3 · p. 6
4.4 Dependency-Aware Revision
- Body corrections can change an interaction’s reconstructed conditions despite unchanged observations, so dependent rules are recomputed from original evidence and revalidated before reuse.Rules independent of the revised components remain eligible for reuse; insufficient evidence leaves affected rules withheld.Source: §4.4 · pp. 6–7
- Reusable action–effect rules are scoped to matched action and context conditions, then tested on independent interactions before they can be reused.Repeated attempts within an episode do not count as independent evidence.Source: §4.4 · pp. 6–7
5 Real-Robot Evaluation
KnowBody is evaluated on four real-robot tasks under fixed-budget and continual-interaction protocols. The fixed-budget evaluation measures completion from a common initialization, while the continual protocol tracks reasoning rounds across successive successes as knowledge is retained.
5.1 Tasks and metrics
- The tasks are duck placement, apple pushing, writing, and pouring, with reasoning-round limits of 30, 30, 80, and 100, respectively.A reasoning round is a model decision in the interaction loop, not a robot motion segment; exceeding budget or losing autonomous continuation counts as failure.Source: §5.1 · p. 7
5 Real-Robot Evaluation
- The fixed-budget comparison uses the same robot and camera arrangement for both protocols, and KnowBody starts each trial from one off-task trajectory with no updates carried across trials.The robot is a Franka Research 3 with a Robotiq 2F-85; a wrist camera and two third-person cameras provide views.Source: §5.2 · p. 7, Figure 4 · p. 7
5.2 Fixed-budget comparison
- 12/16 trials (75%) are completed by KNOWBODY versus 4/16 (25%) by Direct, with KNOWBODY improving completion on all four tasks from the same initialization.The fixed-budget comparison uses four trials per method and task, with cross-episode updates disabled.Source: §5.2 · p. 8, Figure 5 · p. 8
- 17–44% fewer reasoning rounds are required on average by successful KnowBody trials than Direct on duck placement, pushing, and writing.Direct completed no pour trials, so the 67.0-round KnowBody pour mean has no successful-baseline counterpart; averages are conditioned on success.Source: §5.2 · p. 8
- On the DROID setup, KnowBody succeeds across all four tasks; the DROID-trained policies perform strongly or partially on duck placement, while only π0.5-DROID records additional pushing successes, and neither completes writing or pouring.The checkpoints are evaluated as provided, without task-specific adaptation.Source: §5.2 · p. 8
5.3 Continual interaction
- In continual interaction, knowledge is retained between attempts and tasks, and failed attempts between recorded successes also contribute updates while model weights remain frozen.Each task is attempted until five successful completions are recorded before evaluation switches to the next task.Source: §5.3 · p. 9
- 41.2%, 52.6%, 28.9%, and 32.4% are the reductions in rounds from the first to fifth recorded success for duck placement, pushing, writing, and pouring, respectively.Writing is nonmonotonic, rising from 30 to 32 rounds at the final point; the curves index successful trials, not total attempts.Source: Figure 5 · p. 8, §5.3 · p. 9
6 Discussion
The fixed-budget comparison evaluates KnowBody as an integrated harness under a matched VLM and reasoning budget. A separate continual protocol tracks decision effort as validated knowledge is retained.
6 Discussion
- Four real-robot tasks with four trials per method–task condition yield 12/16 KnowBody completions versus 4/16 for the comparison harness.The fixed-budget comparison uses a matched VLM and reasoning budget; reasoning-round averages are over successful trials.Source: §6 · p. 9
- Reasoning-round averages in the fixed-budget comparison are conditioned on successful trials, so they describe decision effort among completions rather than all attempts.Completion over all fixed-budget trials is reported separately from the continual curves.Source: §6 · p. 9
- The continual protocol tracks five successive successful completions per task, whereas the fixed-budget protocol reports every trial outcome.The two protocols separate completion under fixed initialization from decision effort during continual interaction.Source: §6 · p. 9
7 Conclusion
KnowBody connects frozen VLM reasoning to numerical robot actions through explicit body knowledge initialized from an off-task trajectory and revised through interaction. The harness separates task-action selection from publication of reusable knowledge, which changes only after validation.
7 Conclusion
- KnowBody uses explicit body knowledge initialized from an off-task trajectory and revised through interaction to organize direct VLM control.Functional geometry and informative views connect task reasoning to numerical actions, while dependency-aware revision keeps reusable experience consistent with changed body estimates.Source: §7 · p. 9
- Together, the real-robot results support explicit, revisable body knowledge as an interface between frozen VLM reasoning, numerical action, and experience reuse.Source: §7 · p. 9
A Harness Execution and Timescales
- The fast interaction loop updates current state from every observation, while the slower publication loop changes reusable body and interaction knowledge only after validation.The VLM selects task actions but does not write active body parameters, publish rules, or bypass the executor.Source: A Harness Execution and Timescales · p. 12
A.1 Four Clocks
The harness distinguishes clocks for reasoning, sensing, and evidence collection to avoid conflating them with physical trials. Episode-pinned body and interaction knowledge can be withheld when contradicted, but replacements publish only at episode boundaries.
A.1 Four Clocks
- The four clocks distinguish model and tool exchanges from physical trials because reasoning, sensing, and evidence collection are different activities.Source: §A.1 · p. 12
- A read-only query can occur within a model call without advancing the physical-interaction clock.Source: Table A1 · p. 12
- An episode pins its body and interaction knowledge at the start; contradicted results can be withheld immediately, but replacement versions publish only at an episode boundary.This prevents an episode from changing the coordinate system used to interpret its own earlier actions.Source: §A.1 · p. 12
A.2 Component Authority
- The online loop operationalizes authority boundaries, and Table A2 summarizes what each component can produce and what remains outside its authority.Source: §A.2 · p. 12
A.3 One Online Interaction Loop
KnowBody’s online loop separates planning, execution, and interpretation, while its records distinguish information by truth and update contracts. Each interaction is assessed using evidence appropriate to whether the request was rejected or executed.
A.3 One Online Interaction Loop
- Read-only calls use the same frozen input snapshot, and a changed viewpoint requires a motion choice with the usual execution checks.An observation choice captures a later stationary view.Source: §A.3 · p. 12
- Rejected requests provide feasibility evidence but no object-effect sample; executed motions are assessed using achieved motion, after-observations, and separate evidence thresholds.Body-response residuals use achieved rather than requested motion, while task effects use the appropriate after-observation and effect window.Source: §A.3 · p. 12
B Representations and Model-Facing Contracts
- E, B, M, and K have distinct truth and update contracts, including different write authority, lifetimes, and reuse checks.Records use stable IDs and source links; numerical claims carry frame, unit, time, and uncertainty or an explicit unknown.Source: B Representations and Model-Facing… · p. 13, §B.1 · p. 13
B.2 Task, Evidence, and State Records
The records preserve task criteria and evidence while separating current scene claims from causal outcomes. Body queries return scoped relations whose support and validity are explicit.
B.2 Task, Evidence, and State Records
- The task contract preserves goals and stopping conditions across continuations, while resource limits may be finite or explicitly null.Compact task and evidence records include task criteria, diagnostic scope, resource limits, and evidence provenance.Source: §B.2 · p. 13
- Current-state claims describe scene relations, while causal outcomes require an after-observation matching the current observation.A fresh image can update what is visible without establishing that the robot caused the change.Source: §B.2 · p. 13
B.3 Body Queries
- Body queries return action-relevant relations such as image direction, active-part geometry, or motion feasibility, bound to the current observation and episode pin.Supported means available within the recorded domain; unknown marks absent coverage or compatible configuration, and invalid marks a malformed or stale request.Source: §B.3 · p. 14
B.4 Decision Contract
KnowBody uses structured decision contracts and context to keep the frozen planner’s proposals distinct from execution authority. Reusable body estimates and experience are revised through candidate, validation, and publication stages.
B.4 Decision Contract
- Planner choices carry cited evidence and prior-outcome assessments; motion plans specify segments, expected effects, and whether another observation is needed.Diagnostic actions identify the question, alternatives, discriminating observation, and stopping condition before consuming an interaction.Source: §B.4 · pp. 14–15
C.1 Ordered Planner Context
- Planner context is compiled from structured state, with current images attached separately and mandatory evidence and unresolved risks preserved within explicit bounds.The compiler targets 12,000 prompt characters and enforces a 24,000-character, ten-image hard bound; it fails explicitly if required material will not fit.Source: C Context Construction and Canonical… · p. 15, §C.1 · p. 15
C.2 Planner Instruction
- The short planner instruction assigns task reasoning to the frozen VLM while reserving execution authority for the local controller and fixed safety gates.Read-only query results support a decision only when their identities and versions match the frozen call.Source: §C.2 · p. 15
C.3 Independent Evidence Reviewer
- A separate reviewer judges fixed success criteria from observations, receipts, and counterevidence without receiving the planner’s strategy or success narrative.The review is reasoning-blind rather than sensor-independent, and a scoped PASS does not certify the rest of the task.Source: §C.3 · p. 16
C.4 Optional Evidence Assistant
- The optional evidence assistant interprets at most six textual records but receives no images or control interface, and its response remains a model-assessed candidate.Source: §C.4 · p. 16
D Evidence-Grounded Revision Across Episodes
- Online state may update immediately, but reusable body estimates and experience follow a slower route of discovery, validation, and publication.Reinterpreted archived evidence changes derived body-dependent conditions, not recorded actions or outcomes.Source: D Evidence-Grounded Revision Across… · p. 17
D.1 Candidate, Validation, and Publication
- Discovery and validation use physical episode groups, with candidate content and acceptance criteria frozen before validation outcomes are revealed.Rejected, interrupted, or unobserved registered attempts consume validation positions rather than being replaced by later successes.Source: §D.1 · p. 17
D.2 What Changes Under a Body Revision
- When body estimates change, affected historical rules are recomputed from original event-time evidence and withheld until refitted versions pass independent validation.The recorded events and outcome labels remain unchanged; an installation or camera-mode change creates a new configuration rather than transforming old images.Source: §D.2 · p. 17
D.3 Worked Writing Trace
The worked trace shows how a writing interaction proceeds from a supported body query through execution and state updates to later reuse. Estimation details distinguish candidate discovery from independent validation before knowledge can be reused.
D.3 Worked Writing Trace
- The writing trace uses the marker tip’s queried relation to the surface to plan an approach, assess achieved motion and visible effect, update current state, and later revise candidates.The initial off-task trajectory supplies supported response relations but does not prescribe the writing sequence; action–effect knowledge enters K only through qualified outcomes and independent validation.Source: §D.3 · pp. 17–18
E.1 Body Estimation
- Body-response fitting requires static correspondences, rigid camera mounting, and near-pure translation, while geometry estimates pool at least three independently sourced observations.Queries also check excitation, correspondence quality, operating support, and numerical conditioning; geometry uncertainty reflects observed disagreement and input measurement error.Source: §E.1 · p. 18
E.2 Independent Body Validation
- Independent body validation freezes candidates, criteria, and future episode groups, then compares parent and candidate on the same achieved motions.Acceptance requires the complete registered block, adequate coverage, and predetermined error criteria; rejected or unobserved attempts consume validation slots.Source: §E.2 · p. 18
E.3 Conditional Rule Estimation
- Conditional rules use fixed-window observed outcomes and threshold splits, and proceed only when they improve on a same-context predictor and pass independent validation.Occlusion or incomplete evidence yields unknown; after a body correction, dependent rules are refitted from original discovery sources and tested on a newly registered block.Source: §E.3 · pp. 18–19