PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-World Law-to-Action: Vision-Language Priors and Touch for Parts with Hidden Physics

Mohamed A. Mabrok

arXiv:2610.33705v1cs.RO

TL;DR

Robots must infer hidden travel, resistance, and locks because images do not reveal them. L2A combines VLM proposals with touch-based law identification and retains hypotheses for undeclared locks. Its analysis identifies an ambiguity between constant load and friction during one-way motion, while its evaluation is limited to simulation.

  • Problem

    A robot must infer a part’s travel, resistance, and locks because images do not reveal the physics needed to carry out instructions.

    Source: §1 · pp. 1–2

  • Method

    L2A turns VLM claims about laws, ranges, and locks into floored priors, updates them with touch, and retains a hypothesis for undeclared locks.

    Source: §1 · p. 2

  • Results

    On one-direction motion, constant load and Coulomb friction share one regression column; ignoring this ambiguity makes feed-forward short by exactly twice the load.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VLM proposals help most where images reveal facts that early touch cannot, such as a lid’s weight, a visible catch, or a load-bearing lift.

    Source: §10 · p. 23

    All results use simulation and rendered images, VLM latency is simulated, and the paper does not compare against vision–language–action or learned adaptive policies trained on the same scenes.

    Source: §10 · p. 23

Abstract

from arXiv · show

A robot in a home it has never seen must open doors, drawers, lids and taps whose physics a picture does not show. Instructions such as "open the drawer halfway, gently" depend on that physics: how far a part travels, how it resists, and what locks it. We present Open-World Law-to-Action (L2A). A vision-language model (VLM) proposes, for each movable part, its motions, the terms of its law of motion, and what may lock it. Each proposal becomes a prior with a floor, so that touch can always overturn it. The robot identifies each part's law online by Bayesian linear regression with closed-form evidence, for parts that move in one way or in two, and keeps a hypothesis that an unseen part or motion locks the target. We prove that touch alone cannot rule this hypothesis out, derive when another look is worth its cost, and show that a constant load and friction cannot be told apart while a part moves one way. All experiments are in simulation, developed on some scenes and run once on held-out ones. In new kitchens the robot completes 80% of tasks from the VLM's proposals, against 98% when every part is known. Proposed physics raises success on objects that need new terms from 353 to 378-383 of 400 episodes, and a second look raises success in mixed scenes from 284 to 329 of 366 tasks. A mobile robot that takes nothing from the simulator completes 70 of 100 requests in new scenes, against 62 when the simulator says which part each box is (after one change made between two held-out tests).

1 Introduction

Robots must infer a part’s travel, resistance, and locks from interaction because pictures may not reveal them. L2A uses VLM declarations as revisable priors, learns motion laws from touch, and accounts for undeclared locks when deciding whether to act or look again.

1 Introduction

  • A robot must learn hidden travel ranges, resistance, and locks to follow instructions such as opening a cupboard halfway and gently.A hidden stop determines the range, while friction, springs, weight, or catches affect the required force and motion.Source: §1 · p. 1
  • L2A uses VLM claims as floored priors and updates beliefs about motion laws and locks from measurements.The method uses Bayesian linear regression and closed-form evidence for candidate structures, and compiles instructions into functions of the law.Source: §1 · p. 2
  • One shared library and regression model covers parts with one or two coordinates, while treating catches within a part and locks between parts as the same kind of hypothesis.Each library block adds a linear feature, hence a regression column.Source: §1 · p. 2, Table 1 · p. 3
  • A prior floor keeps every hypothesis possible and bounds by a constant the evidence needed to overturn a VLM claim.Declarations cover law structure, range, and locks.Source: §1 · p. 2
  • Because touch cannot distinguish an undeclared releaser from a jam, L2A retains an unseen-lock hypothesis and derives when another look is worth its cost.The open-world hypothesis prevents touch alone from certifying that a task is impossible.Source: §1 · p. 2
  • During one-direction motion, constant load and Coulomb friction share a regression column, while a short reversal supplies an identifiability remedy.Ignoring this ambiguity makes feed-forward short by exactly twice the load; the result applies to gas-strut flaps and drawers that must be lifted.Source: §1 · p. 2
  • Experiments were developed on one set of scenes, frozen, and run once on new scenes, including a progression toward a robot using only cameras and touch.The evaluation describes removing simulator information step by step.Source: §1 · pp. 2–3

2 Problem

The problem formalizes movable parts through their task coordinates, motion laws, and possible locks, then defines how sensing and instructions relate to those unknowns.

2.1 Parts, coordinates and laws

  • Each part has one or two task coordinates, and its law relates applied generalized force to motion through inertia, damping, friction, and library terms linear in unknown weights.Friction makes the law hybrid because a coordinate at rest stays there when force falls within its friction band; linear features permit regression and library growth without a new algorithm.Source: §2.1 · p. 3

2.2 Locks, inside a part and between parts

  • A motion may be free, jammed, latched by another declared motion crossing a threshold, or locked by an undeclared releaser.A releaser can be another motion of the same part or of a different part.Source: §2.2 · p. 4

2.3 What the robot senses

  • Sight gives scene-wide declarations of parts, motions, ranges, and possible locks but can miss or invent them, while touch measures motion under force one part at a time.A missed part or motion remains unknown until a later view declares it; a declared part with nothing movable behind it is a phantom.Source: §2.3 · p. 4

2.4 Instructions as functions of the law

  • Instructions become goals and constraints defined by the law, so uncertainty about that law becomes uncertainty about task success.“Halfway” is defined relative to the stop, while whether “one click” is possible depends on the potential: a sufficiently strong spring removes nonzero stable minima.Source: §2.4 · p. 4, Table 2 · p. 5

3 Identifying a law by touch

The robot identifies each part’s motion law from clean pushes by comparing linear-regression structures, while treating stops and locks as separate events. Reversals resolve load–friction ambiguity that one-way motion cannot, and opening attempts update lock beliefs.

3 Identifying a law by touch

  • The section frames touch-based law identification as learning from the robot’s own pushes, given a prior over candidate structures.The following section supplies that prior.Source: §3 · p. 5

3.1 Each structure is a linear regression

  • The robot models each candidate law as a linear regression over measured quantities and fits only clean steps, excluding sticking, stop contact, and latched motion.For two coordinates it uses force-form rows; one-coordinate laws can be fit after dividing by mass.Source: §3.1 · p. 5

3.2 Posterior and evidence in closed form

  • For each structure, a conjugate prior yields a closed-form posterior and evidence; the evidence rewards fit while penalizing extra coefficients, favoring simpler structures when added terms do not improve fit.The robot reuses sufficient-statistic sums to compute each structure’s posterior and evidence, with O(d^2) update cost per row.Source: §3.2 · pp. 5–6

3.3 What the data cannot tell apart

  • When two feature columns are proportional on the observed data, the likelihood identifies only a combination of their weights and leaves the posterior unchanged along the aliased direction.For a constant force and friction during one-way motion, only their combined effect is identified.Source: §3.3 · p. 6
  • One-way motion can make a constant load indistinguishable from Coulomb friction, so the fit may assign the load to friction and leave lowering feed-forward short by exactly twice the load.Clean motion in both directions separates them; a short reversal can bring variance close to its two-sided value.Source: §3.3 · pp. 6–7

3.4 Stops and locks are events

  • The method treats stops and locks as events outside the clean-motion regression: motion truncates the stop posterior, while blocking pins it and opening attempts update lock hypotheses.A lock hypothesis predicts whether an attempt releases the motion, including release by another motion crossing a threshold.Source: §3.4 · p. 7

4 Priors from a picture

The VLM supplies candidate parts, motions, physics terms, ranges, and possible locks, which the robot converts into priors rather than fixed decisions. A probability floor lets measurements overturn mistaken claims, while widened range uncertainty guards against hidden stops.

4.1 What a declaration contains

  • Each VLM declaration describes parts and their motions, proposes physics-block weights and possible releaser motions, and includes range guesses with spreads and safety notes.The VLM first lists parts and then declares them in batches under a fixed JSON schema.Source: §4.1 · p. 8

4.2 Structure

  • The structure prior gives independent Bernoulli block probabilities a floor, so a wrong VLM claim has bounded cost and a missed block remains recoverable from touch.With η = 0.05, each disputed block costs a Bayes factor of at most 19; a hard choice with η = 0 can make a missed block impossible to discover.Source: §4.2 · pp. 8–9

4.3 Range

  • The robot widens the VLM’s range spread because a picture can suggest total travel but miss an internal stop; the speed governor then raises its stop estimate only as motion truncates the posterior.Using the VLM’s typical ±0.3 rad spread can place the 1% stop quantile near the guess, allowing fast motion toward a nearer stop.Source: §4.3 · p. 9

4.4 Locks

  • Lock weights set priors over free, jammed, and releaser hypotheses, including a probability that the releaser is an undeclared motion.The kitchen estimate assigns µ = 0.40 because the releaser was missing in 26 of 65 observed locks.Source: §4.4 · p. 9

5 Unseen parts and motions

The paper shows that touch cannot distinguish a jam from a lock released by an unseen motion, so touch alone cannot certify impossibility. It derives when another look is worth its cost, accounting for the chance of finding a releaser and the time left to act.

5 Unseen parts and motions

  • Touch cannot distinguish an unseen releaser from a jam when attempts hold all motions outside the tested set at rest, so it cannot by itself rule out an unseen lock.After known releasers and free are ruled out, unseen retains 0.84 posterior probability relative to unseen and jammed together under the paper’s values.Source: §5 · pp. 9–10
  • A look is worthwhile when the posterior probability of an unseen releaser exceeds its cost divided by the reveal probability and success rate with the revealed releaser.In kitchens, the initial threshold is 0.205; a first locked attempt raises the unseen probability to about 0.37.Source: §5 · p. 10
  • The latency-aware value of a look depends on the success rate achievable after its answer arrives, so a slow VLM must be queried early or not at all.With T = 30 s and L = 10 s, a look at t = 10 s is worthwhile only above an unseen probability of 0.58.Source: §5 · p. 10
  • A robot declares a task impossible only when no known allowed releaser remains and the required open-world condition is met; a look that reveals only phantoms counts as empty.This rule accounts for the fact that touch cannot distinguish an unseen releaser from a jam.Source: §5 · p. 11

6 Deciding and acting

The robot chooses targets and experiments using its current law beliefs, then plans actions that account for release chances, time, and risk. Its control rules preserve feasible releaser configurations and limit impact speed under the robot’s stop posterior.

6.1 Where to aim

  • For a target that may have multiple click positions or no click, the robot chooses the most probable decision within tolerance rather than aiming at the posterior mean.The mean can lie halfway between two click positions, where it is certainly wrong.Source: §6.1 · p. 11

6.2 When to experiment

  • An experiment is selected only when its expected post-observation success outweighs its time and constraint-break risk, so it can be declined when the task target is already clear.For “halfway,” a sweep to the stop can make the aim precise; for “open it fully,” the sweep is the task itself and no experiment is selected.Source: §6.2 · p. 11

6.3 Which releaser to try

  • For disjoint releaser options with conclusive failed attempts, trying them in decreasing probability-per-time order minimizes expected release time; the robot updates beliefs and reapplies the rule after each attempt.Within a part, it tries increasing levels of the other motion and holds the releaser where it worked while moving the latched motion.Source: §6.3 · p. 12, §6.4 · p. 12

6.4 Tracking, holding and a speed governor

  • The controller averages feed-forward across structures for two-motion parts and clips set-points with a stop-based governor, with an impact-speed guarantee under the robot’s posterior and stated assumptions.The guarantee requires the loop to track braking, the stop estimate not to decrease during motion, and the creep-speed bound to hold; it does not cover arbitrary model error.Source: §6.4 · pp. 12–13

7 From the simulator to cameras and touch

The agent progressively replaces simulator-provided part identity, handle, joint, and coordinate information with perception and touch. In the final setup, it uses its own map and camera-based handle finding, while touch identifies motion and stops.

7 From the simulator to cameras and touch

  • The agent’s needs from the simulator are part identity, grasp location, joint behavior, and part coordinates; each experiment removes one source of simulator information.The final setup treats each VLM box as a part and finds real stops by touch.Source: §7 · pp. 13–14
  • A wrist sensor and the part’s Jacobian let the agent fit laws to force delivered at the part, distinguishing grip slip from a latch.Failed attempts count as evidence of a lock in proportion to the excess arrived force; release counts only without reported slip.Source: §7 · pp. 13–14
  • Depth images locate handles, but stereo noise at 3 m hides most, so the robot moves to about 1 m and looks again.The passage reports about 2.5 mm depth noise at 1 m and 1.3 cm at 3 m.Source: §7 · p. 14
  • After grasping, small soft pushes reveal whether a joint slides or hinges, and the hand’s path provides the part coordinates.A bending path also gives the hinge axis and position.Source: §7 · p. 14
  • The simulator-free robot builds an obstacle map from depth, finds handles with its wrist camera, and uses touch to find true stops.Development fixes added rules for detecting a stop, recognizing the arm’s reach limit, gripping a tap lever, and slowing motion.Source: §7 · p. 14

8 Experiments

Across experiments developed on one scene set and run once on held-out scenes, L2A is tested with known parts, VLM declarations, new physics, additional looks, and progressively less simulator information. Results show strong performance with known parts and gains from calibrated declarations and second looks, while the final robot tests expose practical limits.

8.1 The core agent with known parts

  • 98% of single-valve episodes and 96% of lockbox-cabinet episodes succeeded when every part was known.On lockboxes, the language-model planner succeeded in 83%, while an agent without a lock model succeeded in 72%.Source: §8.1 · p. 15, Table 4 · p. 16

8.2 Declarations from images, in kitchens never seen

  • With one view per part, calibrated VLM declarations achieved 80% success on held-out kitchens, compared with 98% when every part was known.Raw ranges yielded 57 successes, fewer than ignoring the numbers (80); calibration raised success to 91 and carried over unchanged.Source: Table 5 · p. 17, §8.2 · p. 17

8.3 New physics from declared blocks

  • 378–383 of 400 episodes succeeded with proposed new physics terms, up from 353 with the original blocks and near 385 with a perfect declaration.Allowing every block without a declaration did not help (353); a soft prior also recovered the soft-close damper omitted by a hard choice.Source: Table 6 · p. 17, §8.3 · pp. 17–18

8.4 Unseen parts and looking again

  • The look rule’s effect depended on latency, and kitchen-level comparisons were not significant; a targeted second look solved 18 of 20 tasks in four catch-missing scenes.With 10-second answers, a late-found releaser could leave too little of the 30-second task budget to use it.Source: §8.4 · p. 18, Table 7 · p. 18

8.5 Parts that move in two ways, and mixed scenes

  • 329 of 366 mixed-scene tasks succeeded with VLM motions and a second look, versus 284 without the look; treating motions as separate parts solved 228.For two-motion parts, VLM physics raised success from 245 to 269 of 320 tasks, mostly on drawers requiring a lift; even a perfect guess lost 37 tasks.Source: §8.5 · pp. 18–20, Table 9 · p. 20

8.6 A physically simulated robot

  • The physical robot reached full travel in 319 of 366 tasks, but the same learner solved 354 tasks without a body and 247 through the arm.The arm’s inertia and friction sit between the agent’s commanded force and the part.Source: Table 10 · p. 20, §8.6 · pp. 20–21

8.7 Handles by depth, joints by touch, and nothing from the simulator

  • On 50 parts, the head camera placed grips within a median 0.56 cm of handles, and touch identified all 68 joints correctly in kind.On paired tasks, the robot completed 53 with its own joints and 69 with simulator joints; most losses were on taps.Source: §8.7 · p. 21
  • 70 of 100 requests were completed by the simulator-free robot in its second held-out test, versus 62 when the simulator identified each box’s part.The robot closed all 50 parts and opened the tap in 7 of 10 scenes; neither robot opened a cabinet or lever-handle door halfway.Source: Table 11 · p. 22, §8.7 · p. 22

9 Related work

The paper combines vision-language proposals with touch-updated posteriors to address hidden physics and parts that image-based planners miss. Unlike prior interactive-perception work that assumes known parts, it retains hypotheses for unseen parts and motions.

Vision, language and action

  • Vision–language–action policies [4] [6] [21] [28] learn actions from demonstrations, while this agent uses a VLM only for priors and lets touch update an explicit hidden-physics posterior.The cited approaches include force-aware variants [16] [41]; language-and-vision planners also plan or write controller constraints [1] [17] [18] [19].Source: §9 · p. 22

Physical properties from vision and interaction

  • Methods that refine physical-property estimates [9] [37] [7] [14] differ from this approach, which proposes law structure, parts, motions, and locks with recoverable priors.The agent also keeps a hypothesis for what the VLM did not see.Source: §9 · p. 22

Interactive perception and articulation

  • Interactive-perception methods [5] [26] and latched-mechanism work assume known parts; this paper adds an unseen hypothesis that touch cannot resolve.The unseen hypothesis covers undeclared parts or motions.Source: §9 · p. 22

Identification and decisions

  • The paper replaces a dual-control problem with a one-step, task-directed value of information, drawing on a search index [34] and a governor.Related work also covers open-universe models and value of deliberation under time limits.Source: §9 · p. 23

10 Limitations and conclusion

The evaluation is limited to simulation and rendered images, with cached VLM queries and simulated latency; comparisons stay within one agent. The conclusion frames the method as combining visual guesses with touch-based correction while identifying practical limits and next steps.

Limitations

  • Every result uses simulation and rendered pictures, with cached VLM queries and simulated latency; the look-again result depends strongly on latency.The paper also notes task dependence within scenes, unconstrained Gaussian posterior samples, and losses from the robot arm’s inertia and friction.Source: §10 · p. 23
  • The paper did not compare against vision–language–action policies or learned adaptive policies trained on the same scenes, so it does not establish superiority over those families.The robot that takes nothing from the simulator was ahead of the simulator-informed robot only after a change between held-out tests, and not significantly.Source: §10 · p. 23

Conclusion

  • The conclusion pairs VLM guesses with touch-based correction: closed-form inference and floored priors address recoverable physics claims, while another look addresses undeclared parts or motions.The stated next steps include a moving base, learning through the arm, live VLM timing, and hardware evaluation.Source: §10 · p. 23

A Posterior and evidence of the regression

The appendix derives posterior distributions for the regression coefficients and noise variance, then gives an asymptotic comparison of model evidence. These closed-form quantities support Bayesian comparison of candidate structures.

Posterior

  • Completing the square yields a Gaussian posterior for the regression coefficients, β | σ², y ∼ N(m_n, σ²V_n).The preceding derivation collects terms in β and integrates β out of the joint density.Source: A Posterior and evidence of the… · p. 27
  • The noise variance has an inverse-gamma posterior, σ² | y ∼ IG(a_n, b_n), with a_n = a_0 + n/2.The appendix obtains it by integrating the variance-dependent density.Source: A Posterior and evidence of the… · p. 27

Evidence

  • For structures fitted to the same data, the asymptotic log-evidence difference is obtained after collecting terms whose structure-dependent remainder is O(1).The derivation uses log |V_n| = −d log n − log |Σ_ϕ| + o(1) and the stated approximation for b_n.Source: A Posterior and evidence of the… · p. 27

B Parameters

The parameter table specifies the controller period, prior floors and weights, look-value quantities, refusal gate, risk level, and task budgets. Several entries are measured in kitchens, while others are chosen on development scenes.

Prior and control parameters

  • The control period is 0.01 s, the floor on block and lock weights is η = 0.05, and the minimum relative spread of the range prior is κ = 0.5.The prior shares of free and jammed are π_f = 0.45 and π_j = 0.04.Source: B Parameters · p. 27

Look-value parameters

  • The measured unseen-lock share is μ = 0.40, the measured chance that a look reveals a missed releaser is ρ = 0.28, and measured success with a known releaser and full budget is s = 0.87.The look cost is c = 0.05 in success probability, chosen on development.Source: B Parameters · p. 27

Decision and task parameters

  • The refusal gate on P(unseen) is γ = 0.2, the governor uses the 1% stop quantile, CVaR uses the worst 25% of sampled laws, and the kitchen task budget is 30 s.The task budget is 12 s for valves and objects.Source: B Parameters · p. 27

Related papers

Loading 2610.33705v1…