PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboICL: Embodied In-Context Learning with GPT-6 Astra

Fangcheng Liu, Yeqing Shen, Anda Cheng, Weishi Mi, Chao Tang, Chenyuan Liu, Yushun Xiang, Tingguang Li, Yong-Lu Li, Yehui Tang

arXiv:2609.34261v1cs.ROcs.LG

TL;DR

General-purpose VLMs can control robots but remain weaker on high-precision and long-horizon tasks, leaving open how to organize examples and execution experience. RoboICL structures both as in-context evidence for a frozen model; across RoboDojo it improves over official zero-shot GPT-6 Astra in all four categories and scores 50.64 Overall versus 33.68 for the strongest baseline.

  • Problem

    General-purpose VLMs can generate robot actions but remain substantially weaker on high-precision and long-horizon tasks, raising the question of how to organize examples and execution experience for control.

    Source: §1 · p. 1

  • Method

    RoboICL gives a frozen VLM fixed demonstration context and updating interaction memory through a shared observation–action–receipt–observation grammar, with bounded anchors retaining trajectory experience.

    Source: §1 · p. 2

  • Results

    RoboICL improves over official zero-shot GPT-6 Astra in all four RoboDojo categories and scores 50.64 on 30-task Overall versus the strongest baseline’s 33.68.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Validation on three single-arm real-robot tasks extends the approach beyond simulation without parameter updates.

    Source: §5 · p. 13

    Evaluation covers GPT-6 Astra, RoboDojo, and three real-robot tasks; broader models and hardware platforms and held-out Jev evaluation remain future tests.

    Source: §5 · p. 13

Abstract

from arXiv · show

General-purpose vision-language models offer a promising way to zero-shot robot control: \gptastra{} excels at open-ended and language- or image-conditioned manipulation but remains substantially weaker on high-precision and long-horizon tasks. We introduce \emph{RoboICL}, an in-context robot-control framework that narrows these gaps without robot-specific parameter updates or a learned VLA. RoboICL separates \emph{demonstration context}, which provides recorded examples when available, from \emph{interaction memory}, which accumulates the model's own actions and observed outcomes. Both use a shared observation--action--receipt--observation grammar. To preserve experience across task stages, RoboICL combines sampled demonstration blocks with bounded anchored memory. Fixed anchors keep earlier rollout interactions available for in-context learning, while the latest interaction supports immediate error correction. Across 30 RoboDojo tasks, using zero shot for Open and one demonstration elsewhere, RoboICL improves on official zero-shot \gptastra{} by 20--27 progress-score points in every category. It leads the leaderboard baselines on Memory and Open, achieves comparable performance to the strongest Precision baseline, and remains competitive on Long-Horizon. Its 30-task Overall score is 50.64, versus 33.68 for the strongest baseline. On a separate ten-task subset, RoboICL scores 60.60, within 2.00 points of the $\pi_{0.5}$ + \gptastra{} hybrid approach. On three real-robot tasks, mean progress rises from 14.45 at zero shot to 63.33 at one shot and 78.89 at three shots. On two development tasks, optional Jev-gated action reuse reduces \gptastra{} calls by 33--48\%. Code is available at \href{https://github.com/Mosi-AI/RoboICL}{https://github.com/Mosi-AI/RoboICL}.

1 INTRODUCTION

RoboICL organizes demonstrations and a robot’s own execution experience as shared in-context evidence for a fixed model. It improves over official zero-shot GPT-6 Astra across RoboDojo categories and extends validation to real-robot tasks.

1 INTRODUCTION

  • The motivating gap is how to organize examples and execution experience for general-purpose VLM robot control, which remains weaker on high-precision and long-horizon tasks.Prior work uses demonstrations, prompted skills, or execution feedback, but leaves open how to organize actions, outcomes, and accumulated experience over long-horizon execution.Source: §1 · pp. 1–2
  • RoboICL separates fixed demonstration context from updating interaction memory and presents both through a shared observation–action–receipt–observation grammar.Receipts describe retained expert actions in demonstrations and report execution of model-generated actions in interaction memory.Source: §1 · p. 2
  • Sampled demonstration blocks and bounded anchored memory retain early, selected later, and latest interactions without serializing every frame.Explicit trajectory-gap records distinguish nonconsecutive retained observations.Source: §1 · p. 2
  • RoboICL scores 50.64 on 30-task Overall, exceeding the strongest baseline’s 33.68, while leading on Memory and Open, matching the strongest Precision method, and remaining competitive on Long-Horizon.For Open, where no demonstrations are available, RoboICL uses interaction memory alone.Source: §1 · p. 2, Figure 1 · p. 2
  • On three single-arm real-robot tasks, mean progress rises from 14.45 without demonstrations to 78.89 with three demonstrations.Source: §1 · p. 3
  • A direct API harness uses one continuous-action function tool, without an external skill library, coding-agent runtime, or robot-specific parameter updates.Source: §1 · p. 3

2 RELATED WORK

Prior robot learning from context includes demonstration-conditioned policies, retrieval, and deployment-time adaptation, while frozen models have also been used with skills, programs, spatial representations, and feedback. RoboICL differs in its action and context representation, combining bounded memory with a shared grammar for demonstrations and interaction history.

Robot learning from context

  • RoboICL elicits action chunks from a frozen general-purpose VLM using demonstration context and interaction memory, without training a robot-specific policy or action head.Source: §2 · p. 3

Frozen models and interaction feedback

  • Prior frozen-model approaches use skill selection, program generation, spatial planning, numerical actions, or feedback-conditioned control.Source: §2 · p. 3
  • Unlike GPT-as-Policy Direct and GPT-Policy, RoboICL generates per-step Cartesian-increment and gripper sequences and uses bounded anchored memory in a shared demonstration–interaction grammar.The distinction is in action and context representation, rather than whether online feedback is used.Source: §2 · p. 3

3 ROBOICL: CONTEXT AND CONTROL INTERFACE

RoboICL conditions a fixed multimodal model on demonstrations and evolving rollout memory through a shared interaction grammar. Bounded retention and shared execution rules preserve selected history while balancing context allocation and feedback frequency.

3.1 FRAMEWORK OVERVIEW

  • RoboICL serializes the instruction, demonstrations, interaction memory, and current state for a fixed model to predict an action chunk without deployment-time parameter updates.The direct API harness exposes one action tool with deterministic validation and execution.Source: §3.1 · pp. 3–4

3.2 SHARED INTERACTION GRAMMAR

  • Both context sources use observation–action–receipt–observation records, but demonstration receipts summarize recorded intervals while execution receipts report realized actions and controller feedback.Positive-length executed transitions enter anchored memory; rejected actions that execute nothing remain in the request tail for correction.Source: Figure 2 · p. 4, §3.2 · p. 5

3.3 BOUNDED ANCHORED MEMORY

  • Bounded memory retains fixed temporal anchors and the latest interaction, while demonstrations contribute non-overlapping blocks sampled across each trajectory.Gap markers identify omitted intervals, keeping earlier stages available alongside recent feedback; RoboDojo requests are limited to 50 images, so demonstration and memory coverage must be allocated jointly.Source: §3.3 · p. 6

3.4 EFFICIENCY-AWARE CONTEXT AND EXECUTION

  • Stable context prefixes keep completed anchors and closed gaps unchanged for potential cache reuse, while the latest interaction and pending corrections remain mutable.Demonstrations and live observations use the same image-encoding path.Source: §3.4 · p. 6
  • The prediction and execution horizon is assigned from the task step limit, trading model-call frequency against feedback delay under one shared rule.With full execution and no early termination, the nominal call count is ⌈T/H⌉; the main benchmarks do not use the optional Jev gate.Source: §3.4 · p. 6
  • With J + B = 24, the intermediate allocations (12, 12) and (16, 8) achieve the highest four-task mean progress scores; (22, 2) lowers scores on three tasks.Source: Figure 4 · p. 7

4 EXPERIMENTS

The experiments cover RoboDojo context allocation and category benchmarks, real-robot shot scaling, inference costs, and optional Jev-gated action reuse. RoboICL achieves the highest reported 30-task Overall score, while real-robot performance rises with demonstrations and Jev changes success differently by task.

4.1 EVALUATION PROTOCOL AND CONTEXT CONFIGURATION

  • At fixed C = 24, J = B = 12 scores 80.25, versus 73.50 for first-plus-recent memory; the allocation tasks also require held-out validation.The selected balanced allocation achieved the highest four-task aggregate score among the tested allocations.Source: §4.1 · pp. 7–8

4.2 SIMULATION BENCHMARK RESULTS

  • On the ten-task panel, RoboICL scores 60.60 versus 45.60 for zero-shot RoboICL and 38.50 for GPT-as-Policy Direct, within 2.00 points of the hybrid’s 62.60.The main setting is higher than Direct on six tasks, lower on three, and tied on one.Source: §4.2 · p. 8, Table 1 · p. 9
  • RoboICL leads Memory and Open, matches the strongest Precision baseline, ranks third on Long-Horizon, and achieves the highest 30-task Overall score at 50.64.The 30-task Overall score exceeds the strongest baseline by 16.96 points.Source: §4.2 · p. 8, Table 2 · p. 10

4.3 TRANSFER TO REAL-ROBOT MANIPULATION

  • On three real-robot tasks, mean progress rises from 14.45 to 63.33 to 78.89 across zero-, one-, and three-shot settings; an unseen larger towel scores 40.The larger unseen towel scores 60 points below the seen towel, and only one of five trials receives full credit.Source: §4.3 · p. 11, Table 3 · p. 11

4.4 INFERENCE COST, LATENCY, AND CACHE REUSE

  • Across 1,810 completed requests, the token-weighted KV-cache hit rate is 92.4%; from call six onward, pooled rates reach 91.4% at zero shot and 94.3% at one shot.The later-call rates are pooled across input tokens for the three evaluated tasks.Source: §4.4 · p. 12, Figure 8 · p. 12

4.5 REDUCING MODEL CALLS WITH JEV

  • On two development tasks, Jev reduces GPT-6 Astra requests from 181 to 121 on Align Blocks and 116 to 60 on General Pickup, while success shifts from 3/5 to 2/5 and 4/5 to 5/5, respectively.The gate adds approximately 0.86 seconds per call, and task outcomes differ.Source: Table 5 · p. 13, §4.5 · p. 13

5 CONCLUSION AND LIMITATIONS

RoboICL gives a frozen multimodal model a shared interface for demonstrations and online experience, and performs competitively across RoboDojo categories. Its evidence spans RoboDojo and three real-robot tasks, while broader models and hardware platforms and held-out Jev evaluation remain future tests.

5 CONCLUSION AND LIMITATIONS

  • RoboICL leads compared RoboDojo methods on Memory and Open, matches the strongest method on Precision comparably, and remains competitive on Long-Horizon.The experiments cover GPT-6 Astra, RoboDojo, and three real-robot tasks.Source: §5 · p. 13
  • Efficiency depends on context reuse and feedback frequency, while broader models and hardware platforms and held-out Jev evaluation remain untested.Jev is evaluated on development layouts.Source: §5 · p. 13

A METHOD AND IMPLEMENTATION

RoboICL retains sampled demonstrations and bounded rollout interactions in a shared context, while its action interface differs from concurrent frameworks in the actions and execution details it specifies.

A.1 CONTEXT CONSTRUCTION

  • Interaction memory stores completed actions, receipts, and subsequent observations, retaining fixed temporal anchors plus the latest interaction for historical context and local correction.Trajectory-gap records mark omitted intervals; memory retains at most B complete interactions.Source: §A.1 · p. 17
  • Demonstration context stays fixed during an episode and samples J non-overlapping action blocks per trajectory, preserving block actions and endpoint observations while marking omitted intervals.The three-shot configuration uses K = 3 demonstrations, J = 5 blocks, and B = 5 memory slots; terminal-fallback variants retain terminal observations separately for specified tasks.Source: §A.1 · p. 17
  • Main demonstration windows use approximately uniform temporal targets with deterministic validity adjustment, enforcing nonoverlap and filtering candidates by both-arm action bounds.Source actions are preserved without clipping or rewriting.Source: §A.1 · p. 17

A.2 CONCURRENT-WORK INTERFACE DETAILS

  • RoboICL's act function specifies Cartesian increments and gripper targets at every step, unlike concurrent interfaces that accept absolute poses or waypoints and determine intervening samples and timing.RoboICL executes at 25 Hz in RoboDojo; both concurrent frameworks return execution feedback, while RoboICL shares an observation–action–receipt–observation grammar across demonstrations and memory.Source: §A.2 · p. 18

B.1 ADDITIONAL QUALITATIVE CASE STUDIES

The qualitative cases show recovery after an interrupted action prefix, a demonstration-informed contact strategy for screw fastening, and transfer between demonstrated bimanual role assignments.

B.1 ADDITIONAL QUALITATIVE CASE STUDIES

  • After a continuity rejection discarded ten of fifteen proposed commands, the next execution note requested smaller rotations and the tower reached score 100; the same-seed zero-shot rollout scored 10.The one-shot rollout used H = 15 and J = B = 12; the zero-shot rollout used B = 25.Source: Figure 9 · p. 18, §B.1 · p. 18
  • In Fasten Screws layout 2, three-shot RoboICL scored 100 versus 20 zero-shot, using repeated short clockwise motions, release-and-regrasp resets, and an additional recovery.The zero-shot rollout spent much of its execution searching for a stable grasp.Source: §B.1 · pp. 18–19, Figure 10 · p. 19
  • In Fill Pen Holder, the rollout combined opposite demonstrated hand assignments by transferring the filled holder from the right hand to the left while the pens remained in it.A policy-service failure ended the recording before an official task score was returned.Source: §B.1 · p. 19, Figure 10 · p. 19

B.2 REAL-ROBOT PROTOCOL AND FAILURE ANALYSIS

The real-robot study evaluates three manipulation tasks with randomized object poses and stage-based progress scores. Demonstrations raise mean progress, but insertion stalls and lower Building Bridge scores remain observed boundaries.

B.2 REAL-ROBOT PROTOCOL AND FAILURE ANALYSIS

  • Figure 11 shows selected three-shot Folding Towel executions on two unseen towels, and Table 3 reports progress scores over five trials per condition.Source: Figure 11 · p. 20
  • Demonstrations raise mean progress score to 63.33 with one shot and 78.89 with three shots across three tasks, with five trials per task and shot setting.Eleven of 15 zero-shot trials receive no progress credit. Figure 7 reports mean progress score, not complete-task success rate.Source: §B.2 · pp. 19–20
  • Three of five three-shot Peg in Hole trials stop after pickup, with insertion stalling as successive RGB observations change little after contact.Peg in Hole has a nominal peg–hole tolerance of 1 mm.Source: §B.2 · pp. 19–20
  • Building Bridge scores lowest at every shot count, despite a 26.67-point gain from one to three shots; one of five three-shot trials earns full credit.Its longer trajectory is represented by bounded demonstration blocks and interaction-memory anchors.Source: §B.2 · p. 20

C CACHE-PREFIX CHECKS AND DETAILED JEV RESULTS

The cache-prefix checks cover 30 episodes and verify stable request prefixes, while provider counters supply measured KV-cache hit rates.

C.1 CACHE-PREFIX VERIFICATION

  • Across 30 episodes, the resource and cache figures cover 1,810 completed requests and 1,780 consecutive-request comparisons.Source: §C.1 · p. 20
  • Every marked cache frontier remained byte-identical across its comparison, and each episode retained one fixed-prefix identity.Source: §C.1 · p. 20
  • The checks verify stable request prefixes; provider cached-input counters supply the measured KV-cache hit rates.Source: §C.1 · p. 20

C.2 JEV-GATED EXECUTION: COMPLETE FIVE-LAYOUT RECORD

The complete Jev record reports five-layout outcomes and model use, with table entries showing progress, executed steps, and wall-clock time.

C.2 JEV-GATED EXECUTION: COMPLETE FIVE-LAYOUT RECORD

  • Section 4.5 describes the Jev protocol and reports success counts and paired completion times; Tables 6 and 7 contain all 20 outcomes and condition-level model use.Source: §C.2 · p. 20
  • Table 6 reports progress score, executed robot steps, and rounded wall-clock seconds for every Jev-panel layout.Source: Table 6 · p. 20
  • Table 6 scores are 0 or 100; failed 200-step episodes are shown but excluded from successful-pair completion-time comparisons.Source: Table 6 · p. 20

C.3 PAIRED COMPLETION TIME

Jev and pure GPT-6 Astra both complete specified layouts, with reported wall-time totals favoring Jev; other layouts show a success difference and a steps-versus-calls trade-off.

C.3 PAIRED COMPLETION TIME

  • 864.91 versus 1127.98 seconds on Align Blocks layout 1 and 2327.17 versus 3641.54 seconds summed on General Pickup layouts 0–3 are Jev versus pure GPT-6 Astra wall times.Source: §C.3 · p. 20
  • Table 7 totals include all five episodes; call and API-second counts use completed requests, response tokens are provider-reported, and reuse counts are gate-approved executed steps.Source: Table 7 · p. 21
  • Jev succeeds on Align Blocks layout 4, where pure GPT-6 Astra fails, leaving no paired completion-time baseline for that episode.Source: §C.3 · p. 21
  • On General Pickup layout 3, Jev uses 13 versus 15 GPT-6 Astra requests but takes 103 versus 75 robot steps.Source: §C.3 · p. 21
  • Failed runs remain in the success denominator but are excluded from the paired completion-time sum.Source: §C.3 · p. 21

Related papers

Loading 2609.34261v1…