PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration

Chen-Yu Lin, Jing-Wen Chen, Hsueh-En Chang, Hung-An Chen, Sheng-Hsun Chang, Chi-Pin Huang, Fu-En Yang, Min-Hung Chen, Yi-Ting Chen, Yu-Chiang Frank Wang, Shao-Hua Sun

arXiv:2608.21031v1cs.RO

TL;DR

Passive vision-based policies struggle when manipulation depends on hidden physical properties, while naive interaction can over-explore. PhysCaP addresses this with training-free mass and stiffness estimation plus Planner–Prioritizer exploration, achieving higher success with fewer interactions and lower execution time than passive or naive baselines.

  • Problem

    Passive VLA and code-as-policy agents lack mechanisms to actively acquire latent physical information required by partially observable manipulation tasks.

  • Method

    PhysCaP combines training-free proprioceptive mass and stiffness extraction with Planner and Prioritizer agents that control and streamline physical exploration.

  • Results

    PhysCaP achieves higher task success with substantially fewer interactions and lower execution time than passive or naive interactive baselines across real-world and simulated tasks.

  • Takeaways & Limitations

    Explicit active perception can support efficient manipulation when task-relevant physical properties are hidden from passive observation.

  • Takeaways & Limitations

    The implementation is limited by commercial VLM latency and variability, 2D-to-depth localization errors, and occasional hardware communication delays.

Abstract

from arXiv · show

We present PhysCaP, a Physics-Informed Code-as-Policy agent for active perception in robotic manipulation. While vision-language-action policies excel at imitating demonstrations, they rely on passive observation and fail to infer latent physical properties critical for manipulation. PhysCaP augments code-as-policy frameworks with a physics-informed exploration layer that enables explicit information-seeking through interaction. It introduces training-free physical property extraction modules that estimate object mass and stiffness from robot proprioception without additional sensors. To balance exploration costs and the efficiency of information obtained, PhysCaP employs a dual-agent design: a Planner that decides when to explore and when to stop, and a Prioritizer that filters implausible interactions and ranks the remainder using a heuristic priority score, enabling efficient, targeted exploration. We evaluate PhysCaP on real-world tabletop manipulation tasks (searching for hidden objects, detecting empty cans, and finding ripe avocados) and a simulated task in LIBERO. The results show that existing passive and naive interactive baselines either fail when physical properties are hidden or over-explore, whereas PhysCaP achieves comparable performance with fewer interactions and reduced execution time. Ablation studies further validate the effectiveness of the proposed physical property extraction modules. Project page: https://physcap.github.io

1 Introduction

PhysCaP addresses the gap between passive manipulation policies and active information-seeking by using physical interaction to infer hidden object properties. Its Planner–Prioritizer design targets efficient exploration, and evaluations span real-world and simulated manipulation tasks.

  • VLA policies achieve strong manipulation performance but primarily reproduce demonstrations rather than learning information-seeking interactions.Reinforcement learning can discover exploration but is often sample-inefficient and difficult to interpret or modify.
  • Code-as-policy agents support modular, interpretable decision-making but remain constrained by passive perception and lack active-information mechanisms.
  • PhysCaP augments Code-as-Policy with physics-informed exploration to infer latent properties such as mass for hidden-state manipulation tasks.Figure 1 illustrates identifying and removing an empty can as an example.
  • PhysCaP combines a Planner and Prioritizer to narrow exploration and achieve high task success with fewer interactions and lower execution time than passive or naive interactive baselines.The evaluation covers hidden cubes, empty cans, ripe avocados, and a simulated LIBERO empty-can task.

2 Related Work

Related work develops increasingly structured robot-learning systems, but existing approaches generally separate physical measurement from task-level decisions. PhysCaP instead integrates active physical-property acquisition with agentic planning and control.

  • VLA models directly map visual and language inputs to actions but often lack structured long-horizon reasoning, task decomposition, and intermediate-state tracking.
  • Code-as-policy frameworks coordinate perception, planning, and control through executable programs, yet typically emphasize semantic and procedural reasoning over physical-property inference.
  • PhysCaP’s overview combines Planner uncertainty estimation, Prioritizer action filtering, and executable PhysX calls for active mass and stiffness measurement.
  • Active-perception methods reduce uncertainty through interaction, but prior approaches often decouple perception from deciding what to measure and when to stop.

3 PhysCaP: Physics-Informed Code-as-Policy Agent

PhysCaP targets partially observable manipulation without dedicated sensing hardware by actively acquiring latent physical information and using it to synthesize informed code policies. Its PhysX modules estimate mass and stiffness, while Planner and Prioritizer agents control when and how exploration occurs.

  • 3 PhysCaP: Physics-Informed Code-as-Policy Agent: PhysCaP addresses tasks whose success depends on latent physical properties that cannot be directly measured without dedicated sensing hardware.The agent interacts with the environment, then uses inferred properties to synthesize a more informed plan.
  • 3.1 Physical Property Extraction: Two training-free PhysX modules estimate object mass and stiffness using a standard robotic gripper without additional sensing hardware.
  • 3.1.1 Object Mass Measurement: The mass module compares empty- and loaded-grasp joint torques during a fixed 15 cm lift to isolate the object’s gravitational contribution.The estimated mass uses the vertical Jacobian projection and gravity to account for local kinematic configuration.
  • 3.1.2 Object Stiffness Measurement: The stiffness module detects contact using incremental closure and a backoff force-response test before storing the contact displacement reference.
  • 3.1.2 Object Stiffness Measurement: Stiffness is inferred from deformation at target effort f* = 0.50, mapped to five calibrated levels and selected by majority vote across five measurements.
  • 3.1 Control Framework: The control framework exposes modular APIs that connect high-level reasoning with embodiment-specific robot trajectories and controllers.Additional physical-property modules can be integrated in a plug-and-play manner.
  • 3.2 Planner Agent: The Planner identifies missing physical information, initiates targeted exploration, and stops when sufficient evidence supports reliable execution.
  • 3.3 Prioritizer Agent: The Prioritizer filters implausible or redundant interactions and reorders remaining candidates by heuristic priority scores to reduce exploration cost.It also supplies a brief reason for each assigned score.

4 Experiments

Experiments evaluate PhysCaP on real-world tabletop tasks and simulated LIBERO settings, testing physical-property extraction and progressively more capable exploration architectures. PhysCaP combines accurate physical measurements with planning and prioritization to improve success and efficiency.

  • Baselines: The baseline progression isolates passive perception, physical measurement, closed-loop planning, and candidate prioritization as separate contributions.CaP lacks physical exploration; CaP+PhysX measures every object; adding a Planner enables stopping; PhysCaP adds heuristic filtering and interaction ranking.
  • Results and Analysis: CaP fails on most tasks because visual reasoning lacks access to hidden physical state, while CaP+PhysX improves performance but tests objects exhaustively.CaP’s faster execution time therefore comes at the cost of task completion, whereas physical measurement alone remains inefficient without exploration reasoning.
  • Results and Analysis: PhysCaP achieves the best overall success rate, efficiency, and execution time by stopping when sufficient evidence is available and prioritizing plausible interactions.Its dual-agent design combines a Planner’s stopping criterion with a Prioritizer’s candidate filtering and priority ordering.
  • Physical Property Validation: Mass estimation reliably distinguishes empty and full containers using proprioceptive feedback, while stiffness measurement maps torque responses to avocado rigidity.Mass evaluation covers five calibration masses from 13g to 963g across 20 trials each; an oracle test achieved 10/10 success on empty-can identification.
  • Ablation: Merging the Planner and Prioritizer into one agent slightly degrades performance because reasoning often collapses into unstructured, exhaustive execution plans.The ablation supports separating planning from prioritization to structure multi-objective reasoning and interaction sequencing.

5 Discussion

Across real-world and simulated tasks, PhysCaP is reported to improve task success while reducing physical interactions and execution time relative to passive or naive interactive baselines. The implementation remains bounded by model, localization, and hardware-latency limitations.

  • Simulation Discussion: In simulation, PhysCaP identifies the target with the fewest interactions, while most VLA baselines achieve near-zero success under latent physical uncertainty.The reported OI and Time values are averaged only over successful episodes across 50 trials.
  • Contribution: PhysCaP combines sensor-free mass and stiffness estimation with dual-agent exploration to acquire task-relevant physical information through interaction.The framework balances interaction cost against information efficiency using Planner and Prioritizer roles.
  • Overall Findings: Experiments report higher task success with substantially fewer interactions and lower execution time than passive or naive interactive baselines.This pattern is reported across both real-world and simulated tasks.
  • Limitations: Current limitations include unpredictable commercial-VLM latency, 2D-to-depth localization errors, and hardware communication latency that can make trajectories diverge from generated code.The authors suggest local models, multi-view or 3D-native localization, and real-time control-signal management as future directions.

A Experiment Details

The experiment platform uses a 7-DoF AgileX PiPER arm with ZED 2i RGB-D perception, while Molmo2 localizes objects and depth data resolves their 3D positions for control.

  • Experiment Setup: The physical platform combines a 7-DoF AgileX PiPER arm, gripper, height-adjustable tabletop, and side-mounted ZED 2i stereo camera.The camera captures RGB-D images at 1280 × 720 resolution.
  • Visual Perception Pipeline: A high-level reasoning agent passes target visual context to Molmo2 for 2D image-coordinate pointing.The resulting coordinates are used as the starting point for spatial control.
  • Visual Perception Pipeline: The system back-projects Molmo2’s 2D coordinates through the ZED depth map and camera extrinsics to obtain absolute 3D world coordinates.These coordinates parameterize the get object pose() control API for downstream joint trajectories.
  • Experiment Configuration: Object layouts remain similar across trials, while initial object positions are randomized with slight variation.This configuration is used for the three real-world tabletop manipulation tasks.

A.1.1 Task 1: Find Blue Cube

The appendix defines three tabletop scenarios and their operational protocols: finding a concealed cube, identifying an empty can, and selecting a ripe avocado through visual cues and physical measurements.

  • Task 1: Find Blue Cube: The find-blue-cube task hides one of three equal-sized cubes beneath a cup and tests whether geometry can reduce unnecessary exploration.The smallest cup cannot conceal the cube, so an effective agent prioritizes the two larger cups.
  • Task 1: Find Blue Cube: Success requires safely lifting the correct cup so all three blue cubes are exposed; knocking over a cup or stopping without revealing the cube is failure.The agent is informed that all three cubes share the same dimensions.
  • Task 2: Identify Empty Can: The empty-can task contains four colored cans, with one empty can among full or partially full candidates, and classifies cans below 100g as empty.The robot must place the single empty can completely within the wooden tray.
  • Task 3: Pick Ripe Avocado: The avocado task asks the agent to isolate the sole ripe fruit, using darker skin to prioritize candidates and stiffness measurements to identify ripeness.The supplied task criterion defines ripe avocados as stiffness level 2 or lower.
  • Task 2: Identify Empty Can: The can protocol identifies and sequentially lifts candidates, classifying a can as empty when its weight is below 0.1 kg.Non-empty cans are returned to their original coordinates.
  • Evaluation Protocol: The qualitative comparison reports that PhysCaP uses targeted visual cues, whereas passive CaP guesses and CaP+PhysX+Planner explores every candidate.The comparison covers hidden-cube, empty-can, and ripe-avocado scenarios.

A.3.1 Task 1: Find Blue Cube

Qualitative trajectories show PhysCaP reducing exploration by filtering implausible objects, stopping after sufficient evidence, and using task-specific physical checks for hidden targets.

  • Find Blue Cube: PhysCaP excludes the obviously small cup and verifies only the two geometrically viable cups, requiring one or two interactions instead of lifting all three.The Planner generates the second interaction only after the first large cup reveals no cube.
  • Find Blue Cube: CaP and CaP+PhysX+Planner lift all three cups sequentially, including the cup too small to conceal the target.PhysCaP instead uses geometric heuristics to remove that cup from the action queue.
  • Identify Empty Can: For empty-can identification, PhysCaP prioritizes open cans using straws and open tabs, then stops after a mass check confirms an empty can.Sealed cans are inferred to be full and therefore deferred.
  • Identify Empty Can: The generated empty-can policy checks candidate weights and, when a value is below 0.1, places the confirmed can on the wooden tray and ends the task.The code uses get weight(), object-pose queries, gripper control, and tray placement.

A.3.3 Task 3: Pick Ripe Avocado

PhysCaP targets the two dark-skinned avocados, measures stiffness sequentially, and stops once a ripe avocado is identified, avoiding tests on visibly green fruit.

  • Trajectory Comparison: PhysCaP removes the two green avocados as explicitly unripe and prioritizes the two dark-skinned candidates for stiffness measurement.The naive interactive baseline instead applies get stiffness across all four avocados.
  • Trajectory Comparison: The Planner stops the exploration loop when the first dark avocado is classified as ripe, avoiding physical contact with the remaining objects.The ripe fruit is then relocated to the wooden tray.
  • Code Policy Demonstration: The generated policy tests the first dark avocado and, if it is unripe, sequentially tests the second dark avocado before placing the ripe one.The policy terminates after confirming successful placement.
  • Code Policy Demonstration: The stiffness policy treats a measurement in the interval 0 < stiffness <= 2 as confirmation of ripeness before executing grasp and tray-placement actions.The code requests a new observation when the tested avocado is unripe.

A.4.2 Task Details and Success Criteria

The LIBERO empty-cup task requires probing cups via mass sensing, applying a 100g threshold, and placing the confirmed empty cup in the basket. The protocol emphasizes grounded candidate selection, closed-loop observation, and safe sequential manipulation.

  • A trial succeeds when the empty white cup is placed entirely inside the basket; otherwise it fails after 4000 time steps.
  • The agent maps stable cup labels to scene positions before lifting exactly one unmeasured cup per reasoning cycle.
  • Probe order uses visual grounding, accumulated mass knowledge, straw cues, and action cost rather than label or alphabetical order.
  • Mass below 0.1 kg identifies an empty cup, while None means its mass remains unmeasured.
  • After non-empty probes, the agent returns cups to their original coordinates, releases them, homes the arm, and requests the next observation.
  • The comparison includes OpenVLA and MolmoAct2 LIBERO checkpoints, with PhysCaP reported to identify the correct cup through active mass measurement efficiently.

A.5.1 Task 1: Find Blue Cube

The blue-cube task tests whether an agent can locate a hidden cube among cups using visual and geometric reasoning. The setup combines viable concealment containers with distractors and requires interactive perception to reveal the target.

  • Three identical 5cm × 5cm × 5cm blue cubes are placed in the workspace as target objects.
  • The asset arrangement requires interactive perception to locate the hidden item among containers.
  • Large 16oz cups can fully conceal a blue cube, whereas the small 12oz cup is too small and serves as a visual distractor.
  • The task details are presented alongside the empty-can task setup, which uses cans in distinct physical states to evaluate semantic and physical reasoning.

A.5.3 Task 3: Pick Ripe Avocado

The ripe-avocado task combines visual ripeness cues with interactive stiffness verification. Dark-skinned avocados create an ambiguity that requires physical testing to distinguish the ripe target from a firm distractor.

  • The setup evaluates synthesis of visual heuristics with physical stiffness verification using three ripeness stages.
  • Green avocados are visually unripe distractors that can be removed without tactile interaction.
  • A dark-skinned firm avocado resembles maturity visually, so the agent must measure rigidity to eliminate it.
  • The target is a dark-skinned avocado that yields to gentle pressure and must be selected to complete the task.
  • The mass module is described as a proprioceptive physical-property tool whose downstream role includes distinguishing empty and full containers.
  • Stiffness evaluation uses a 3D-printed button whose resistance varies with the number of rubber bands, alongside avocado testing.

B.3 API Specification Details

The API specification exposes robot-control and physical-measurement functions to the coding agent. These interfaces support object localization, manipulation, mass estimation, stiffness estimation, and execution control.

  • get mass(object name) measures mass by grasping the object and analyzing torque through a Jacobian-based conversion.
  • The mass API accepts an object name and returns an estimated mass in kilograms, returning 0.0 when measurement fails.
  • get stiffness returns an integer stiffness level from 1 to 5 and returns 0 when measurement fails.
  • The control interface includes gripper opening and closing, inverse-kinematics pose execution, homing, and reasoning-step termination.
  • get object pose uses semantic image keypoints and depth back-projection to return a precise 3D pose with a pre-grasp offset.
  • The coding agent can also call the standard Python library.

C.2 API Specifications

The API specifications define modular perception and control primitives for robot manipulation, while the planner workflow determines whether visual information is sufficient and proposes targeted exploration when it is not.

  • API Specifications: The API section documents low-level perception and control primitives, including function signatures, outputs, documentation, and usage examples.
  • API Specifications: get object pose uses Molmo2 for object detection and returns a 3D position with a WXYZ quaternion.
  • API Specifications: goto pose moves the arm to a Cartesian target, optionally inserting a Z-axis approach offset before reaching the requested position.
  • Planner Workflow: The Planner checks whether the task can be completed from the current scene and accumulated knowledge, then either proceeds to code generation or proposes exploration.
  • Planner Workflow: Exploration candidates target missing task-relevant information, such as object mass, stiffness, or concealed contents, and may include weighing, stiffness measurement, or revealing hidden items.
  • Planner-Prioritizer Workflow: The merged Planner-Prioritizer variant generates a prioritized exploration plan in one pass, ordering candidates so the first action is the next efficient interaction.
Loading 2608.21031v1…