PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer

Sida He, Lingxi Xie, Yunning Cao, Pengfei Chen, Kaiwen Duan, Jiannan Ge, Xinyue Huo, Jiacheng Shao, Qi Tian

arXiv:2609.31770v1cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

The paper studies whether body knowledge, successful experience, and executable feedback routines can make GPT-6-Astra’s robot manipulation more efficient. It supplies these resources to an XLeRobot and evaluates them in simulated and physical elevator-button tasks. The reported results include faster execution with richer resources, experience reuse at tested new starts, and sim2real transfer.

  • Problem

    Repeated exploration and model-mediated action selection can make robot-control execution slow, motivating study of how external body knowledge, successful experience, and reusable routines affect a general-purpose agent.

    Source: §1 · p. 2

  • Method

    The study varies body descriptions, successful-attempt records, and executable feedback routines for a fixed GPT-6-Astra agent controlling an XLeRobot through a common interface.

    Source: §1 · pp. 2–3

  • Results

    Complete geometry and synchronized experience yield the largest fixed-start reductions, and experience-guided trials remain faster at every tested displaced start.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The experiments connect timing benefits to the agent’s use of geometry, recorded experience, and current feedback during manipulation and transfer.

    Source: §7 · p. 15, §9 · p. 17

    The study covers one robot platform and task family, with three repetitions per main condition and one selected experience source per domain.

    Source: §8 · p. 16

Abstract

from arXiv · show

General-purpose multimodal agents can write robot-control programs, but repeated exploration and model-mediated action selection can make execution slow. We study how external body knowledge, successful experience, and executable skills improve an XLeRobot controlled by GPT-6-Astra in a simulated and a physical elevator-button task. In 30 fixed-start simulation trials, complete robot geometry and camera information reduce mean completion time by 57.4% relative to a baseline with only the common control interface and no prior experience; images with synchronized action and state records reduce it by 68.6% without additional body assets. In nine paired comparisons (18 trials) at starts displaced by 10-100 cm, experience recorded at the original start reduces mean time by 58-63% relative to no experience, demonstrating generalization to the tested new starting positions. During experience experiments, GPT-6-Astra spontaneously generates a short visual-feedback program. Researcher-refactored versions reduce mean local-task time by 29-31% in 27 simulation trials. Finally, 12 real-robot trials using operator-confirmed button contact demonstrate sim2real reuse: at a shared nominal start, simulation XML assets and simulation experience reduce mean time by 53.0% and 49.9%, respectively; real experience also transfers to two new starts. These results suggest a practical way to build general-purpose manipulation experiments around GPT-6-Astra: supply machine-readable body descriptions and synchronized demonstrations, and turn useful agent-generated feedback routines into reusable skills, while the agent adapts actions from current images. We release all task prompts, trial-level experimental data, and acquired skill implementations at https://github.com/hesd10/astra-robot-sim2real.

1 Introduction

The paper asks how body knowledge, successful experience, and reusable routines affect GPT-6-Astra’s efficiency in an elevator-button task. It compares these resources in simulation, tests experience at displaced starts, and examines how the resources are used during sim2real transfer.

1 Introduction

  • The XLeRobot elevator-button task combines mobile approach, arm positioning, visual alignment, and contact, with Astra writing programs through a common low-level interface.Trials use fresh agent sessions with fixed model weights; completion time includes materials, model interaction, motion, observation, and confirmation.Source: §1 · p. 2
  • The study varies body information from joint names and limits to geometry and camera calibration, and experience from no episode to synchronized images, actions, states, and feedback.Experience can inform initial arm configuration and later approach and alignment decisions, while current observations remain available throughout.Source: §1 · pp. 2–3
  • Complete body information and synchronized experience yield the largest fixed-start reductions, while experience-guided trials remain faster at every tested displaced start.The fixed-start comparison spans 30 trials; displaced-start comparisons use 18 paired trials at starts 10–100 cm from the original.Source: §1 · p. 3
  • During experience experiments, Astra spontaneously generates a short visual-feedback routine that researchers refactor into single-step and repeating-loop skills for 27 simulation trials.The trials compare the routines with the standard API from three near-button states.Source: §1 · p. 3
  • The real-robot study tests simulation assets and experience at a shared nominal start, then reuses real experience at two new starts despite a different lobby, button, and feedback criterion.The destination uses an opposite-side DOWN button and operator-confirmed gripper-tip contact; the source simulation uses an UP button with visual activation feedback.Source: §1 · p. 3
  • The contributions combine resource comparisons, a controlled evaluation of skills derived from an agent-generated routine, and real-robot sim2real reuse.The paper connects timing benefits to observed use of geometry, recorded experience, and current feedback.Source: §1 · p. 4

2 Related Work

Prior work connects language models to robot actions through skills, feedback, and generated programs, while other approaches use broad training data or retain experience outside model weights. This paper studies how a fixed general-purpose agent uses readable body information and successful-attempt records at deployment time.

2 Related Work

  • SayCan [2], Inner Monologue [3], Code as Policies [4], and VoxPoser [5] motivate program-mediated robot control with planning, feedback, generated programs, or spatial value maps.The study holds the agent and low-level interface fixed while varying supplied robot descriptions, experience, and executable routines.Source: §2 · p. 4
  • PaLM-E [6], RT-2 [7], Open X-Embodiment [8], and Octo [9] use broad embodied or cross-robot training; this paper instead examines deployment-time information for a fixed agent.The related-work passage frames these approaches as establishing the value of broad training data.Source: §2 · pp. 4–5
  • ReAct [10], Reflexion [11], and Voyager [12] retain experience or executable skills, while this study compares summaries, sampled images, and synchronized image–action records.Here, an emergent skill is a useful local feedback routine generated during ordinary execution and tested in single-step and repeating-loop forms.Source: §2 · p. 5
  • STEP and LOOP divide work between Astra’s image-based choice of direction and alignment and local bounded movement with visual stopping feedback.Local stopping feedback saves model interactions when the chosen direction remains useful; directional revision still requires another agent decision.Source: §2 · p. 5
  • Unlike domain-randomization transfer approaches [16] [17], this study reuses explicit geometry or multimodal records and relies on fresh real observations for online adjustment.Hardware calibration aligns joint conventions, and the transfer evaluation includes both reusable configurations and approach references as well as alignment and completion-signal adjustments.Source: §2 · p. 5

3 System and Experimental Protocol

The experimental system pairs an XLeRobot and a common observation/control interface with progressively richer body information and experience. Three simulation blocks and a physical-robot block compare these inputs using task-time measures and specified success criteria.

3.1 Task, embodiment, and interface

  • The XLeRobot uses two six-channel arms, a pan–tilt head, three cameras, and a four-wheel mecanum base; Astra receives images and joint states and issues timed moves, base velocities, stops, and completion declarations.The agent sees head and wrist-camera views; operator monitoring views and evaluator-side simulation state are reserved for supervision and outcome checks.Source: §3.1 · p. 5
  • The simulation target is the middle elevator’s UP button, while physical trials target a DOWN button and count operator-confirmed gripper-tip contact as success.The written hardware prompt requests depression, but button stiffness and unreliable lighting lead to the contact criterion; camera geometry comes from simulation, and the agent locates the real button from images.Source: §3.1 · pp. 5–6

3.2 Body information and experience

  • Body information levels I0–I3 range from the common API and actuator feedback to qualitative structure, mechanical assets and joint mapping, then camera geometry and calibration; experience levels E0–E3 range from none to synchronized records.The simulation experience package derives from one selected successful source episode and is reused across fixed-start, displaced-start, and sim2real comparisons.Source: §3.2 · pp. 6–7
  • The local-skill conditions compare no routine with a single movement followed by feedback and repeated movement with an internal visual stopping check.These conditions are evaluated from near-button states in simulation.Source: §3.2 · p. 7

3.3 Comparison design, timing, and evidence

  • The fixed-start design uses ten conditions across 30 trials, comparing I0–I3 without history and E1–E3 at both I0 and I3, with three repetitions per condition.Shared I0E0 and I3E0 baselines support the comparisons.Source: §3.3 · pp. 7–8
  • Task time includes material reading, model and service waiting, code generation, actuation, and outcome confirmation, with timing start and stop rules varying by evaluation block.Time reductions are computed against the corresponding block baseline; simulation success requires target activation and a correct declaration without wrong-button events or recorded faults, while real success uses operator-reported contact.Source: §3.3 · p. 8
  • The fixed-start trials run E0, then I0 experience, then I3 experience; displaced-start tests pair E0 and E3 at each point, and local-skill condition order rotates across states and repetitions.Real A/B/C condition order is counterbalanced as ABC, BCA, and CAB; D follows source selection.Source: §3.3 · p. 8

4 Body Knowledge and Experience in Simulation

At a fixed start, complete geometry and synchronized action experience substantially reduce simulation task time. Experience recorded at the original start also reduces time across the tested displaced starts, while the agent continues adapting its approach from current images.

4.1 Fixed-start information comparison

  • 57.4%: complete geometry with camera information reduces mean time relative to I0, while I0E3 reduces mean time 68.6% relative to E0.All 30 fixed-start trials succeed; Figure 4 also shows that qualitative I1 information is not faster than I0.Source: §4.1 · p. 8, Figure 4 · p. 9
  • I0E3 and I3E3 reuse the same right-arm targets, q2 = 1.0, q3 = 1.2, and q4 = −0.84 rad, found in the successful source episode.The three I0E3 trials each use four joint-move requests, versus 14 joint actions in the source episode; E3 trials combine staged arm deployment with fresh observations and subsequent alignment from current images.Source: §4.1 · p. 9

4.2 Reuse at displaced starts

  • 58–63%: original-start experience reduces mean time across the three tested displaced-start groups, and is faster at all nine points.The 18 trials compare E3 with no experience at starts displaced by 10, 50, and 100 cm; the agent adjusts its approach from current images.Source: §4.2 · p. 10
  • At displaced starts, E3 trials reduce average joint requests by 81.5% and base requests by 12.0%, consistent with less repeated arm search while base alignment continues.Across the fixed and displaced E3 trials, 11 of 15 include six to ten base commands between staged arm deployments.Source: §4.2 · p. 10

5 From Emergent Behavior to Reusable Skills

Astra spontaneously produced a local visual-feedback routine during experience experiments, and researchers refactored it into reusable skills. In 27 simulation trials, STEP and LOOP reduced mean near-button task time, with most of the gain achieved by STEP.

5.1 A spontaneously generated local loop

  • Astra spontaneously generated a routine that advances briefly, checks images for red pixels in selected regions, and repeats until a threshold or step budget is reached.The observed program allowed at most five iterations, executed four, and took about 4.6 s locally before Astra’s visual confirmation.Source: §5.1 · p. 11
  • In a four-step scenario, avoiding three intermediate model round trips is estimated to save 45–75 s of decision latency.The estimate uses observed inter-call intervals of 15–25 s and excludes initial observation, program construction, and final confirmation from the latency calculation.Source: §5.1 · pp. 11–12

5.2 Evaluating reusable skills

  • 29.0%: STEP reduces mean time versus FREE, and 30.6%: LOOP reduces mean time; all 27 trials succeed.STEP is faster in all nine matched comparisons, while LOOP is faster in eight of nine; LOOP’s mean is 2.2% lower than STEP’s.Source: §5.2 · p. 12, Figure 8 · p. 13
  • At S3, LOOP averages 151.1 s versus STEP’s 130.8 s; one LOOP trial took 213.3 s and required seven calls to revise direction and alignment.LOOP holds direction fixed within each call, so repeated alignment corrections can offset savings from fewer model round trips.Source: §5.2 · pp. 12–13

6 Sim2Real and Real-Experience Transfer

In 12 physical elevator-button trials, simulation assets and experience reduce same-start mean task time, while real experience also supports successful contact from two new starts. The trials use a different lobby and button direction from the source simulation, with operator-confirmed contact as the success criterion.

6.1 Real-lobby protocol

  • The hardware task uses an opposite-side DOWN button, operator-confirmed gripper-tip contact, and a 0.10 m/s translation-speed cap.Hardware joint targets use the simulation XML reference, and both APIs use body-frame base velocities.Source: §6.1 · p. 13

6.2 Same-start efficiency and material use

  • 53.0% and 49.9%: simulation assets and simulation experience reduce mean real-task time, respectively, versus baseline at the shared nominal start.The comparison includes three trials per condition; all 12 physical trials reach operator-confirmed contact.Source: §6.2 · pp. 13–14, Figure 9 · p. 14, Table 2 · p. 14
  • 20.0 to 3.7 average arm/head move requests and 18.7 to 10.7 base requests accompany asset-guided trials’ XML-based kinematics calculations.All three asset-guided trials parse the supplied XML and calculate forward kinematics; two also inspect gripper-mesh geometry.Source: §6.2 · p. 14
  • All three simulation-experience trials read source action history and complete the current DOWN-button task despite using experience from an UP-button task.In two trials, initial leftward movement is corrected to the right; live observations guide subsequent arm and base changes.Source: §6.2 · p. 14

6.3 Real experience beyond its source start

  • Using real source records and historical images, all three trials achieved button contact: D1 finished in 250.3 s at the original start, while D2 and D3 finished in 269.7 s and 360.0 s at different new starts.Source: §6.3 · p. 15

7 Discussion

The discussion attributes sim2real reuse in this task to interpreting explicit robot descriptions, extracting references from multimodal records, and revising actions from current observations. It connects this approach to body-relative transfer and to the limits of reusable feedback routines.

7 Discussion

  • GPT-6-Astra reuses simulation materials during physical execution by interpreting robot descriptions, extracting configurations and approach references from multimodal records, and revising actions through current observations.The discussion presents these as suggested reasons for strong sim2real performance in this task.Source: §7 · p. 15
  • Calibrated robot geometry and joint configurations retain meaning across simulation and hardware, reducing dependence on reproducing the source start or full motion sequence.Real trials show assets supporting new geometric calculations and experience supplying an arm configuration that Astra subsequently adjusts.Source: §7 · p. 16
  • Experience transfer combines an UP-button demonstration with a DOWN-button instruction; observed target changes and corrections support an interpretation of reuse followed by adaptation to the current scene.Two new real starts extend this pattern: one trial reuses a source pose, while another uses different intermediate targets.Source: §7 · p. 16
  • STEP captures most of the mean improvement, while LOOP can be slower when its parameters prompt repeated alignment corrections because its success detector gives no directional feedback.The discussion proposes adding progress and alignment estimates or early return when assumptions fail; it also points to classical visual servo control.Source: §7 · p. 16

8 Limitations

The study’s findings are scoped to one robot platform and task family, with limited repetitions and selected experience sources. Its real new-start trials lack matched baselines, and hardware success is defined by operator-confirmed contact rather than verified button depression.

8 Limitations

  • One robot platform and task family, three repetitions per main condition, and one selected experience source per domain define the study’s scope.Broader tests across sources, headings, and embodiments would extend the findings.Source: §8 · pp. 16–17
  • The two new real starts have no matched baseline, and hardware success means operator-confirmed contact rather than verified button depression.Pose-only and preset-pose controls would help distinguish benefits from useful arm configurations versus richer experience during approach.Source: §8 · pp. 16–17

9 Conclusion

The conclusion reports that robot knowledge, successful experience, and reusable skills reduce repeated exploration in GPT-6-Astra manipulation. It favors supplying explicit body files and synchronized demonstrations, while packaging useful agent-generated feedback routines for reuse.

9 Conclusion

  • Complete body information and synchronized experience reduce simulation mean completion time by 57.4% and 68.6%, respectively; simulation assets and experience reduce same-start physical mean time by 53.0% and 49.9%.The conclusion also reports 58–63% reductions at tested displaced starts, 29–31% reductions from refactored skills, and real-experience transfer to two new starts.Source: §9 · p. 17

A Complete Fixed-Start Results

At I3, synchronized image–action experience is faster than no experience, while written summaries and sampled images have higher mean times. Every tested condition succeeds in three trials.

A Complete Fixed-Start Results

  • At I3, synchronized image–action experience is faster than no experience, while written-summary and sampled-image conditions have higher mean times.The comparison motivates distinguishing experience representations by both their contents and reading costs.Source: A Complete Fixed-Start Results · p. 19
  • The I3 comparison varies experience representation, including written summaries, sampled images, and synchronized records.These representations differ in both content and reading cost.Source: A Complete Fixed-Start Results · p. 19
  • Every tested condition succeeds in 3/3 trials.Task time is measured using the simulation service task clock.Source: Table 3 · p. 19

B Local States and Skill Semantics

The section describes local-state skill evaluation, exploratory experience studies, and the distinct constraints in the simulation, local-skill, and physical task prompts.

B Local States and Skill Semantics

  • The skill study evaluates STEP and LOOP from three near-button states, with a 300 s local-task budget and initially inactive buttons.The discovery routine and evaluation states came from separate trials; an offline check established local reachability, while Astra selected direction and control parameters.Source: B Local States and Skill Semantics · p. 19
  • STEP returns an image after one bounded movement, while LOOP repeats movements and checks button-color feedback under step and time limits.The routine uses speeds up to 0.06 m/s and pulse durations of 0.1–0.8 s; activation detection uses a red-pixel threshold in an agent-selected region.Source: B Local States and Skill Semantics · p. 19
  • In S3 repetition 2, local execution intervals totaled 9.54 s, while gaps between calls totaled 130.31 s.These intervals distinguish local execution from the longer between-call work without attributing all gap time to model computation.Source: B Local States and Skill Semantics · p. 20, Table 5 · p. 20

C.1 Earlier iterative written experience

  • The exploratory six-episode written-experience sequence had five successes followed by a declared failure, and repeated trials showed means of 16:29 and 17:19.The mixed exploratory results motivated controlled comparisons of experience representations.Source: §C.1 · p. 20

C.2 Initial target information

  • With one trial per condition, all four initial-target-information tests succeeded; the fastest observed time, 5:26, used both body information and the supplied target pose.Times were 11:15 with neither, 7:10 with body information, and 15:21 with the initial pose alone.Source: §C.2 · p. 20

C.3 Experience compression and reflection

  • Compact experience was faster than expanded reflection in 10/12 pointwise comparisons, but faster than ordinary E3 at only 3/6 points in each measurement.Packages differed in content and organization, including selected images and reading cost; ordinary E3 already included qualitative guidance.Source: §C.3 · p. 20

D.1 Simulation approach and pressing

  • The simulation approach prompt requires fresh visual verification of the UP button’s red activation signal and prohibits unlisted historical materials.The task asks the agent to minimize time and online decisions without damage, while verifying action results through current feedback.Source: §D.1 · p. 21

D.2 Local-skill evaluation

  • Local-skill trials begin near the UP button and require bounded motion and fresh-image verification; all groups lack geometry models and target coordinates.The prompt permits code and image-processing feedback loops, but only the UP button may be activated.Source: §D.2 · p. 21

E.1 Control interface

The section specifies the shared control interface, the contents and provenance of experience records, and the released reproduction resources.

E.1 Control interface

  • Both interfaces cap base translation at 0.10 m/s and yaw at 15 degrees/s; joint segments last at most 10 s.Simulation base commands last at most 1 s, while real commands allow up to 5 s; the agent receives joint feedback and commanded velocity, but no global odometry or target coordinates.Source: §E.1 · pp. 22–23, Table 6 · p. 23

E.2 Experience contents and pose provenance

  • Simulation E3 combines an agent-generated summary, 22 sampled images, a timestamped index, and synchronized records; real E3 combines the fastest A trial’s summary, 25 image triplets, and timestamped action/state records.Prior images do not replace fresh observations during a trial.Source: §E.2 · p. 23

E.3 Reproducing the figures and report

  • The released repository includes the model, code, frozen inputs, trial-level results, selected videos, reproduction instructions, and scripts to rebuild figures and tables.New agent trials require compatible model access.Source: §E.3 · p. 23

Related papers

Loading 2609.31770v1…