PaperlayerView exact revision on arXiv

Source-linked AI summary

How High Can GPT-6 Astra Climb in Contact-Rich Manipulation? PhysLadder: A Hierarchical, Matched Sim-Real Benchmark for Physical Interaction

Haibao Yu, Jiaqi Meng, Zijian Zhang, Bowen Jing, Lei Yang, Yue Hu, Chenchen Ge, Yiming Hou, Chenbo Xin, Dawen Ye, Haifang Qin, Jin Li, Weitao Zhou, Jianing Qiu, Jiankai Sun, Dandan Zhang, Zaiqing Nie, Ping Luo, Ding Zhao

arXiv:2610.34527v1cs.RO

TL;DR

Aggregate success rates alone do not identify which physical interaction requirements constrain performance as contact-rich tasks demand suitable contacts, force regulation, and coordinated contact changes. PhysLadder organizes 16 household manipulation tasks by physical interaction requirements, focusing on contact configuration and force regulation. Success declines across levels, reaching 0.0% at L3 in both environments; ten of 16 tasks have identical success counts, and the other six differ by one trial.

  • Problem

    Aggregate success rates alone do not identify which physical interaction requirements constrain performance as contact-rich tasks demand suitable contacts, force regulation, and coordinated contact changes.

    Source: §1 · p. 2

  • Method

    PhysLadder organizes 16 household manipulation tasks by physical interaction requirements, focusing on contact configuration and force regulation.

    Source: §1 · p. 2

  • Results

    Success declines across levels, reaching 0.0% at L3 in both environments; ten of 16 tasks have identical success counts, and the other six differ by one trial.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    For GPT-6 Astra, matched simulation reflects key capabilities and limitations observed in real-world evaluation and offers a repeatable, scalable benchmark across increasing interaction demands.

    Source: §7 · p. 19

    Three trials per task constrain statistical precision, and whether Sim–Real correspondence predicts performance for other models remains untested.

    Source: §6.5 · p. 19

Abstract

from arXiv · show

We introduce PhysLadder, a hierarchical Sim-Real benchmark for evaluating physical interaction capabilities in contact-rich manipulation. Its 16 household tasks span three cumulative levels: contact selection and geometric alignment (L1), force regulation within sustained contact (L2), and coordinated contact reconfiguration and force regulation (L3). Each task is instantiated in physically and visually matched simulated and real-world environments, with shared task definitions and success criteria. We evaluate GPT-6 Astra zero-shot with three trials per task in each environment. Real-world success rates decline from 80.0% at L1 to 47.6% at L2 and 0.0% at L3, with corresponding simulation rates of 66.7%, 47.6%, and 0.0%. Ten of the 16 tasks have identical success counts across environments; the remaining six differ by one success out of three trials. Across both environments, recurring difficulties in spatial reasoning, contact selection, force regulation, and coordination often prevent local physical progress from accumulating into task completion. These findings characterize Astra's observed capability boundary across increasing interaction demands and, for this model, support matched simulation as a repeatable, scalable testbed for studying the sustained, coordinated physical interactions that limit reliable manipulation.

1 Introduction

PhysLadder evaluates GPT-6 Astra against household manipulation tasks organized by increasing physical interaction demands in matched simulated and real-world environments. Success declines across levels, while the environments show similar task outcomes and recurring interaction difficulties.

1 Introduction

  • PhysLadder organizes 16 household task versions into cumulative levels of contact selection and alignment, sustained-contact force regulation, and coordinated contact reconfiguration with force regulation.The levels provide evaluation targets beyond grasping and placement.Source: §1 · pp. 2–3
  • The benchmark pairs each task in physically and visually matched simulation and reality, with corresponding task definitions and documented success criteria.Simulation supports controlled, repeatable evaluation at scale.Source: §1 · pp. 2, 4
  • Real-world success declines from 80.0% at L1 to 47.6% at L2 and 0.0% at L3; simulation rates are 66.7%, 47.6%, and 0.0%, respectively.Ten of 16 tasks have identical success counts across environments, and the other six differ by one successful trial.Source: §1 · pp. 2–3
  • Spatial reasoning, contact configuration, force regulation, and coordination recur as bottlenecks, and local physical progress does not reliably accumulate into task completion.Examples include turning a cap or moving a drawer without completing the required operation.Source: §1 · pp. 2, 4

2 Related Work

Prior embodied-model and manipulation benchmarks assess broad capabilities and contact-rich operations, while Sim–Real studies examine environment correspondence. This paper evaluates Astra across contact demands in matched simulated and real-world tasks.

2.1 Embodied Evaluation of General-Purpose Foundation Models

  • Existing embodied evaluations cover visual understanding, spatial reasoning, sequential decisions, and robot manipulation, while reported Astra and other model work also identifies precision, dynamic-control, and coordination challenges.Harness refinements and demonstration-based in-context learning are among approaches explored for improving execution.Source: §2.1 · p. 4

2.2 Contact-Rich Manipulation Benchmarks

  • Contact-rich benchmarks span individual operations and longer skill compositions, but operation categories and task length do not directly determine contact difficulty.This motivates organizing tasks around shared contact and force-regulation requirements.Source: §2.2 · p. 4

2.3 Physical and Visual Matching for Sim–Real Evaluation

  • Prior Sim–Real work matches observations and control and examines physical correspondence, while this study tests whether Astra’s interaction strengths and limitations recur across matched environments.Related evaluations compare rankings, performance correlations, failure patterns, and execution quality beyond binary success.Source: §2.3 · pp. 4–5

3 Hierarchical Benchmark Design for Contact-Rich Manipulation

The benchmark organizes 16 household task versions by cumulative contact-configuration and force-regulation demands. Explicit initial conditions, goals, process constraints, and observable criteria support task-level evaluation and diagnosis of local physical progress.

3.1 Principles of Hierarchical Task Design

  • The hierarchy distinguishes contact configuration from force regulation, with L1 requiring contact selection and alignment, L2 adding force regulation during sustained contact, and L3 adding coordinated contact reconfiguration.Task level depends on the task version and setup, not its name; the hierarchy does not impose an absolute difficulty ordering.Source: §3.1 · pp. 5–6

3.2 Task Suite Overview

  • The 16-task suite comprises five L1, seven L2, and four L3 tasks spanning placement, constrained motion, deformable objects, pouring, tool contact, and coordinated multi-contact operations.It includes rigid objects, deformable food objects, and constrained mechanisms.Source: §3.2 · p. 6

3.3 Task Specification and Success Criteria

  • Each task version specifies its initial conditions, goal, and process constraints without prescribing a unique action sequence; success requires the specified observable outcome and constraints.Criteria include outcomes such as separation, prescribed displacement, stable support after release, and avoiding damage during Jelly Handling.Source: §3.3 · p. 7
  • Effective contact is diagnostic progress—contact transmitting force or torque for the main operation—not a substitute for complete task success.For Bottle Cap Unscrewing, local rotation indicates effective contact, while success requires full thread disengagement and controlled lifting to the specified separation.Source: §3.3 · p. 7

4 Physically and Visually Matched Sim–Real Environments

PhysLadder constructs paired real and simulated task environments by matching workspace, assets and contact responses, robot configuration, and camera observations. The real platform supports single- and dual-arm tasks, while simulation represents task-relevant physical interactions.

4 Physically and Visually Matched Sim–Real Environments

  • The environments are matched across workspace initialization, task assets and contact responses, robot embodiment, and camera configuration to assess Astra across interaction levels.Corresponding task definitions and success criteria support cross-environment comparisons.Source: §4 · p. 7

4.1 Real-World Setup

  • The real-world platform uses two ARX R5 arms with two-finger grippers sharing a tabletop workspace, supporting both single-arm manipulation and dual-arm coordination.An external camera provides a workspace view, while two wrist cameras provide local views; the hardware has no dedicated contact-force sensors.Source: §4.1 · p. 7, Figure 3 · p. 8

4.2 Simulation Environment and Task Assets

  • The Isaac Sim environment reconstructs the workspace and represents task assets with interaction-relevant geometry, constraints, mass, friction, and deformability.Simulation includes features such as drawer joints, elastic orange and jelly, and wiping-related visible surface-state changes.Source: §4.2 · p. 8, Figure 4 · p. 9

4.3 Physical and Visual Matching

  • Matching uses measured layouts and object dimensions, refinement against observed contact responses, corresponding robot geometry, and camera viewpoints and observations.The jelly frames are qualitative and do not quantify agreement in deformation or contact force; the paired views are unsynchronized.Source: §4.3 · pp. 8–10, Figure 5 · p. 10

5 Evaluation Setup and Protocol

The evaluation compares GPT-6 Astra’s physical-interaction performance in matched simulated and real-world tasks. It specifies task configurations, model inputs and execution, and trial and success-rate procedures.

5.1 Task Setup and Model Configuration

  • Tasks use fixed objectives and success criteria, while varied simulated object positions and orientations test generalization across initial layouts.The zero-shot evaluation provides no prior task executions, demonstrations, or task-specific strategies; within-episode observations and feedback may guide decisions.Source: §5.1 · p. 11

5.2 Model Inputs and Closed-Loop Execution

  • Both environments provide fixed task-agnostic interface guidance, while updated images, proprioceptive states, execution feedback, and interaction history inform decisions.Real-world Skills describe the robot and interface; simulation provides corresponding guidance as model instructions. Neither environment supplies depth images or contact-force measurements, so contact outcomes must be inferred from other feedback.Source: §5.2 · p. 11
  • Each action is executed through environment-specific controls, then checked against new observations before the model continues, corrects, or requests termination.Simulation uses structured actions, cuRobo planning, and joint drives; real-world tool calls drive the arms through a control service. Command completion alone does not confirm the intended interaction.Source: §5.2 · pp. 11–12, Figure 6 · p. 12

5.3 Evaluation Protocol and Metrics

  • Each task has three trials per environment, and autonomous success requires achieving the goal while satisfying all process constraints.Simulation ends trials at success, a defined failure, or budget exhaustion; real-world trials may also end through operator intervention or protective stops. Task-set success rates weight tasks equally, and corresponding behaviors are analyzed alongside outcomes.Source: §5.3 · pp. 12–13

6 Results and Analysis: Contact-Rich Capabilities and Limits

Across increasing interaction demands, Astra’s task completion declines in both environments, while simulated and real-world outcomes show close correspondence. Execution examples show how shared local progress and interaction difficulties can fail to produce task completion.

6.1 Zero-Shot Performance Across Interaction Levels

  • Real-world success falls from 80.0% at L1 to 47.6% at L2 and 0.0% at L3; simulation falls from 66.7% to 47.6% and 0.0%.None of the four L3 tasks succeeds in either environment, despite local progress in several executions.Source: §6.1 · p. 14

6.2 Correspondence Between Real and Simulated Performance

  • Ten of 16 tasks have identical success counts across environments, and the other six differ by one successful trial.Both environments show declining completion from L1 to L3.Source: §6.2 · p. 14
  • Shared execution difficulties include spatial alignment, establishing effective contacts, maintaining grasps, regulating force, and coordinating successive actions.Examples include a hat that remains misaligned with a rack, a cap that does not separate, jelly that remains supported, cap slippage, and sweeping that requires coordinated brush contact and block redirection.Source: §6.2 · p. 14, Figure 7 · p. 15
  • Charger Alignment outcomes differ despite similar positioning because real-world judgments are visual, whereas simulation applies a fixed geometric tolerance.The unsuccessful simulated trial continued refining alignment until its model-call budget was exhausted.Source: §6.2 · p. 14

SIMULATION REAL WORLD

For GPT-6 Astra, matched simulation reflects key capabilities and limitations observed in real-world evaluation, while the analysis identifies recurring interaction bottlenecks and scope limits. Initial conditions also change the physical demands and interaction level of a task.

SIMULATION REAL WORLD

  • Both environments show local drawer motion without reliably completing the pull-and-return sequence, while book extraction exposes difficulty coordinating support and pulling.Real-world cables and wrist-mounted cameras further restrict access and gripper orientation during book extraction.Source: Figure 8 · p. 16, SIMULATION REAL WORLD · p. 17
  • Matched simulation reflects key capabilities and limitations observed in real-world evaluation and supports repeatable, scalable comparisons.Controlled initialization and parallel execution reduce dependence on robot availability and manual resets.Source: SIMULATION REAL WORLD · p. 17

6.3 Sensitivity to Initial Conditions

  • Two blocks near the dustpan entrance require localized sweeping within a relatively stable contact mode, whereas three dispersed blocks require successive contact changes and force regulation.The three-block execution remains incomplete; the variants are not a controlled layout ablation or additional Table 2 trials.Source: §6.3 · pp. 17–18, Figure 9 · p. 17
  • A nearly unscrewed cap requires a short turn under maintained grasp, while the L3 configuration requires coordinated regrasping and continued turning.The hierarchy assigns levels to specific task configurations, whose difficulty can vary continuously with initial conditions.Source: §6.3 · p. 18

6.4 Failure Modes and Interaction Bottlenecks

  • Spatial reasoning, precise motion, and contact selection remain difficult, with useful contacts needing accessible locations and orientations that support the intended action.Examples include alignment adjustments, grasping without damaging an upright orange segment, and selecting nut contacts that permit later rotation.Source: §6.4 · pp. 18–19
  • Force regulation challenges include balancing book support with extraction, controlling both drawer motions, retaining a grasp during unscrewing, and wiping away writing.These cases do not isolate force regulation from contact configuration or motion execution.Source: §6.4 · p. 19
  • Coordination must link contact choices and force across changing states, yet local actions do not reliably accumulate into task completion; bottlenecks may overlap, while execution limits are not bottlenecks.Examples include supporting a deformable object and coordinating successive sweeps as block positions change.Source: §6.4 · pp. 18–19

6.5 Limitations and Scope

  • Three trials per task constrain statistical precision, and the observed Sim–Real correspondence is specific to GPT-6 Astra and the tested task versions, configurations, and platform.Contributions of model decisions, sensing, and robot execution remain to be isolated.Source: §6.5 · p. 19

7 Conclusion

The evaluation finds that GPT-6 Astra’s performance declines as physical interaction demands increase, with no complete successes on tested L3 tasks. PhysLadder uses matched real and simulated environments to characterize this boundary and support repeatable evaluation.

7 Conclusion

  • Performance declines as physical interaction demands increase, with no complete successes on the tested L3 tasks.Difficulties in spatial reasoning, contact configuration, force regulation, and their coordination limit individual actions and sustained progress.Source: §7 · p. 19
  • PhysLadder organizes 16 household tasks into L1–L3 according to increasing demands on contact configuration, force regulation, and their coordination.The benchmark evaluates GPT-6 Astra zero-shot in matched real and simulated environments.Source: §7 · p. 19
  • For GPT-6 Astra, matched simulation reflects key capabilities and limitations seen in real-world evaluation, while explicit specifications and controlled initialization support repeatable comparisons.Parallel execution reduces dependence on real-robot resources.Source: §7 · p. 19

A Detailed Task Specifications

The appendix specifies task versions through their initial conditions, goals, and process constraints. Table S1 summarizes how task interaction requirements determine level assignments.

A Detailed Task Specifications

  • Each task card lists initial conditions, the goal and constraints, the instruction, complete-success criterion, and contact requirements motivating its interaction level.The quoted instruction is retained unchanged; the goal and success criteria operationalize it without adding completion steps.Source: A Detailed Task Specifications · p. 22
  • Table S1 summarizes the interaction-level assignments of all 16 tasks under the criteria defined in Section 3.1.No additional requirement means no force-regulation requirement beyond ordinary support, transport, and geometric placement.Source: Table S1 · p. 22
  • Natural rolling, sliding, and content release do not by themselves count as contact reconfiguration.Source: Table S1 · p. 22

A.1 L1 Task Specifications

The L1 task specifications focus on contact selection and geometric alignment, without requiring additional force regulation. They define concrete initial conditions, goals, and complete-success criteria for placement, alignment, and hanging tasks.

A.1 L1 Task Specifications

  • Marker placement succeeds when the released marker remains inside the cup and is supported by it; close-fitting insertion and specified depth are unnecessary.A stable grasp and geometric alignment with the cup opening suffice for the L1 assignment.Source: §A.1 · p. 23
  • Charger Alignment requires lifting the charger and aligning both prongs with the designated socket holes; insertion and electrical operation are not required.Precise prong-to-hole alignment is the main contact requirement motivating L1.Source: §A.1 · p. 23
  • Hat Hanging succeeds only when the released visor is supported through its head opening by the designated branch, not merely perched on its tip.The grasp must leave the opening accessible for alignment; no additional force regulation is required.Source: §A.1 · p. 23
  • Cup Nesting requires sequential placement and release, with the upper cup nested in and independently supported by the lower cup.An open gripper that still traps the cup does not constitute release.Source: §A.1 · pp. 23–24
  • Flower Transfer requires retrieving the flower and inserting its stem into the receiving cup, with enough stem left free for insertion.Complete success requires the flower and cup to remain independently stable after release and gripper withdrawal.Source: §A.1 · p. 24

A.2 L2 Task Specifications

The L2 and L3 specifications distinguish tasks by the contact and force demands needed for completion, with L3 adding coordinated contact changes. The real-world setup uses RGB observations and gripper commands that do not specify contact force.

A.2 L2 Task Specifications

  • Book pulling requires a 5 cm middle-book displacement while the right arm stabilizes the upper book; marker-cap removal requires maintained bimanual grasps until separation.Stack motion alone does not establish the book displacement, and placement or release is not required after cap separation.Source: §A.2 · p. 24
  • Marble pouring requires emptying the source cup, returning it to its original position, and leaving at least six marbles in the fixed receiving cup.Cup tilt alone is insufficient; the task is L2 because the robot maintains its grasp while regulating marble motion and release.Source: §A.2 · p. 25
  • The segment and jelly tasks lift each object 5–20 cm, return it to its original support, and release without damage throughout the operation; complete success requires independent support after release.Source: §A.2 · pp. 25–26
  • Drawer Pull & Return requires moving the drawer about 5 cm outward and restoring its initial opening, while Whiteboard Wiping is scored by erased writing rather than path traversal.The drawer must not be moved by shifting the cabinet; wiping contact without removing the writing is insufficient.Source: §A.2 · pp. 25–26

A.3 L3 Task Specifications

  • Bottle-cap and nut unscrewing require regrasping when one grasp cannot cover the remaining thread travel, while maintaining effective torque and body stabilization.The bottle cap must fully disengage and move 3–5 cm from the opening; the nut must completely separate from the threaded rod.Source: §A.3 · pp. 26–27
  • Stacked-book pushing requires a full in-plane turn during transfer between regions while preserving the stack and tabletop support without grasping or lifting.The requirement is coordinated changes in pushing contact, force direction, and turning torque, not simply the use of two arms.Source: §A.3 · p. 27
  • Block sweeping requires successive brush–block contact changes to direct all three blocks into a fixed dustpan without direct gripper manipulation of the blocks.Success is based on the physical outcome; no particular stroke sequence is prescribed.Source: §A.3 · p. 27

B.1 Real-World Configuration

  • The real-world arms provide RGB camera images and normalized gripper-opening commands, but gripper commands do not specify contact force.The wrist cameras acquire 848 × 480 images at 15 fps and the external camera 640 × 480 images at 30 fps; depth images are unavailable to the model.Source: §B.1 · pp. 27–28

B.2 Simulation Implementation

The simulation loads task-specific scenes and models contact and material behavior, while a separate evaluator uses simulator-side state to assess outcomes.

B.2 Simulation Implementation

  • Task configuration files load the dual-arm robot, assets, supports, and camera rig, then reset object poses, fixation states, joint displacements, and thread engagement.Camera parameters are estimated from real-scene videos, and rendered views approximately preserve the corresponding camera aspect ratios.Source: §B.2 · p. 28
  • Rigid-object collision geometry is retained, constrained motions use passive joints, and orange and jelly behavior is represented with GPU finite-element models.The jelly’s elastic coefficients are estimates where physical parameters could not be measured.Source: §B.2 · p. 28
  • Whiteboard Wiping simulates erasure by accumulating force-weighted coverage over ink cells until cells disappear, rather than modeling measured material removal.Contact force is projected onto the board normal, and force and motion deadbands are applied before coverage accumulates.Source: §B.2 · p. 28
  • The automatic evaluator uses simulator-side contact, displacement, proximity, and ink-cell quantities that are not available to the model as observations.Proximity alone is not treated as a contact-force measurement.Source: §B.2 · p. 29

C Evaluation Protocol Details

The protocol supplies task-agnostic operating guidance and defines how trials, model calls, and outcomes are handled. It also distinguishes qualitative bottleneck assignments from outcome judgments that can vary by observer.

C.1 Basic Skills Description

  • Basic Skills explain the interface and general execution practices without specifying how to complete evaluation tasks.They cover embodiment and interface reading, observation and outcome verification, two-arm coordination, and local recovery.Source: §C.1 · p. 29, Figure S1 · p. 30

C.2 Trial Execution and Model Calls

  • Simulated and real trials use the same task specifications, but real-world initial object poses are manually restored and are not matched one-to-one with simulation.Simulation episodes have default limits of 150 model calls and five hours; real episodes have no preset time or call limit and may end when the model judges a task infeasible.Source: §C.2 · p. 29
  • A simulated decision is a separate Codex CLI call, and reported model-call counts are not directly comparable across environments.A simulated call returns one structured action, whereas a real-world action may take several tool calls; continuation calls remain part of the original episode.Source: §C.2 · p. 29

C.3 Bottleneck Criteria and Outcome Qualifications

  • Bottleneck assignments are qualitative and may include several bottlenecks per trial, while real-world judgments for Charger Alignment and Whiteboard Wiping can vary with the observer.Budget exhaustion and protective stops are termination conditions, not bottlenecks; simulation applies fixed executable success checks.Source: §C.3 · p. 29

Related papers

Loading 2610.34527v1…