Transferring the Intelligence of VLMs to Robotic Control

1 INTRODUCTION

• The motivating question is whether VLMs can reuse digital-world intelligence for physical control despite differences in embodiment, environment, and task.The introduction calls this cross-setting reuse of general capabilities “intelligence transfer.”Source: §1 · p. 1
• Robot-data training can be costly, embodiment-specific, and difficult to scale, motivating transfer through an interface and a few demonstrations instead.The introduction also notes evidence that action-prediction training may degrade general-purpose instruction-following and reasoning capabilities.Source: §1 · pp. 1–2
• RoboDawn exposes robot control to an agentic VLM through discrete translation, rotation, and gripper commands, enabling closed-loop action from visual observations.The model reasons about an action, executes it, observes the consequence, and then decides what to do next.Source: §1 · pp. 2–3
• A few in-context demonstrations ground the VLM in interface use and task strategies without parameter updates, and even one demonstration can substantially improve manipulation performance.The approach addresses ambiguity about action semantics and granularity that written interface rules alone may not resolve.Source: §1 · pp. 2–3
• 53.2% zero-shot success on RoboTwin 2.0 C2R rises to 73.6% with one in-context demonstration, while RoboDojo success rises from 35.67% to 47.17%.RoboDawn's 53.2% also exceeds π0.5 (46.0%) and LingBot-VLA (50.4%) on RoboTwin 2.0 C2R; the framework performs block-in-basket and block stacking with a Franka robot.Source: §1 · p. 2
• For broad manipulation problems, a strong pretrained VLM can serve as a high-level decision-maker when paired with an action interface and a few demonstrations.The authors frame this as complementary to robot-data scaling, which remains important for low-level dexterity and high-frequency control.Source: §1 · p. 3

2.1 FROM TASK-SPECIFIC INTELLIGENCE TO TRANSFERABLE INTELLIGENCE

• The field's shift toward transferable intelligence is associated with pretrained models adapting across tasks through language instruction rather than task-specific training.The discussion places this change after earlier deep-learning systems whose capabilities were more closely tied to individual tasks and objectives.Source: §2.1 · p. 3

2.2 END-TO-END ROBOTIC POLICY

• End-to-end robotic policies have broadened toward generalist control as transformer and diffusion architectures and larger, more diverse robot datasets support generalization.VLA models adapt pretrained VLM capabilities for action generation, while WAMs combine future prediction with robot action generation.Source: §2.2 · p. 3
• Pretrained VLM intelligence can be transferred to closed-loop robotic control through an action interface without task-specific parameter updates.This contrasts with end-to-end policies that train on action data.Source: §2.2 · p. 4

2.3 AGENTIC METHODS FOR ROBOTIC CONTROL

• Earlier systems used foundation models for high-level planning, while many newer agentic systems add memory, verification, or replanning but retain pretrained VLA policies for low-level execution.RoboDawn's distinction from Show-Harness is its co-designed ICL strategy as well as its VLM harness.Source: §2.3 · p. 4

3.1 OVERALL FRAMEWORK

• RoboDawn conditions a pretrained VLM on task context, a robot–environment profile, and demonstrations, then adapts through observations, execution feedback, and memory while keeping model parameters fixed.The VLM does not receive privileged object poses; its commands are grounded into robot motions, and the resulting physical outcomes update feedback and memory.Source: §3.1 · p. 4

3.2 HUMAN-INTUITIVE INTERFACE

• The interface uses the gripper interaction point—the midpoint between the fingertips—for visual annotations, robot-state reporting, and motion commands.Semantic move and rotate commands specify incremental GIP pose changes; each is grounded into a planned motion and executed to a stationary state.Source: §3.2 · pp. 4–6

3.3 IN-CONTEXT LEARNING DESIGN

• RoboDawn pairs a shared primer illustrating basic command effects with task-level demonstrations showing how commands compose into task-solving behavior.Demonstrations are expressed in RoboDawn’s semantic command space by reducing trajectories to end-effector waypoints and gripper states, then converting them into command sequences.Source: §3.3 · p. 6
• Demonstrations come from scenes disjoint from evaluation scenes and are collected from scripted simulation experts or human teleoperation on real robots.For long RoboDojo trajectories, image observations are sparsified to fit a budget of 16 observations per round, retaining semantically informative rounds and omitting visually redundant transitions.Source: §3.3 · p. 6

4.1.1 ROBOTWIN 2.0

• 53.2% zero-shot success on RoboTwin 2.0 C2R exceeds π0.5 (46.0%) and LingBot-VLA (50.4%); 73.6% one-shot exceeds HarnessVLA by 15.2 percentage points.RoboDawn uses no task-specific parameter updates and at most one in-context demonstration.Source: Table 1 · p. 7, §4.1.1 · p. 7
• More demonstrations generally improve performance, but gains diminish beyond one shot; stronger VLMs help, and reasoning, grid localization, and the command primer improve results.With Gemini-3.8-Flash, success rises from 47.0% zero-shot to 62.2% one-shot and 65.4% with four demonstrations, then falls to 62.7% with eight; removing reasoning, grid localization, or the primer yields 34.8%, 32.4%, or 44.0%, respectively.Source: §4.1.1 · pp. 7–8, Table 3 · p. 9
• The efficiency evaluation compares inference latency, motion execution time, and inference-to-motion ratio; ratios below 0.5 may permit inference to overlap with motion.Seed-2.1-Pro is used for RoboDawn latency evaluation, considering network latency.Source: Table 2 · p. 8, §4.1.1 · p. 8

4.1.2 ROBODOJO

• As the per-episode command budget rises, success improves in both settings: one-shot increases from 31.2% with 60 commands to 47.2% with 240, and zero-shot from 23.7% to 35.7%.The authors attribute this test-time scaling to failure recovery and memory that adapt actions based on prior observations and execution outcomes.Source: Figure 3 · p. 8, §4.1.2 · pp. 8–9
• 35.67% zero-shot and 47.17% one-shot success on RoboDojo; the one-shot result surpasses full-set post-trained DM0.5 (19.34%).RoboDojo includes 42 tasks; Table 4 also reports a 34-task setting excluding eight open tasks.Source: §4.1.2 · p. 8, Table 4 · p. 9

4.3 FAILURE ANALYSIS.

• Representative RoboDojo failures involve insufficient precision, IK-related execution errors that disturb the scene, and incorrect success judgments against benchmark criteria.The examples include inaccurate placement, collisions despite semantically reasonable actions, and stopping before enough liquid reaches a cup.Source: §4.3 · pp. 9–10, Figure 4 · p. 10

4.2 REAL-WORLD ROBOTICS DEPLOYMENT

• Zero-shot deployment transfers to real-world block-in-basket and block-stacking tasks, with stacking less successful; cloth folding remains considerably more challenging.The authors hypothesize that cloth folding is difficult because it requires substantial end-effector rotation and orientation adjustment.Source: §4.2 · p. 9, Table 5 · p. 10

• RoboDawn’s iterative reasoning and closed-loop interaction make it slower than end-to-end VLA and WAM policies, limiting suitability for high-frequency control.Source: §5 · p. 10
• VLMs handle translation more reliably than precise arm and end-effector rotations, and ICL only partly alleviates this gap.The authors suggest less explicit representation of rotations and 3D orientation in web-scale pretraining may contribute.Source: §5 · p. 10
• The discrete semantic action interface can be too coarse for precise adjustments near contact, grasping, and placement.Source: §5 · p. 10
• Erroneous actions may cause collisions, object damage, or risks to humans during deployment.Source: §5 · p. 10

• RoboDawn transfers pretrained VLM intelligence to robotic control through a human-intuitive action interface and in-context demonstrations, without task-specific robot training.Source: §6 · p. 11
• Experiments on RoboTwin 2.0, RoboDojo, and real robots show strong zero-shot performance and substantial gains from a single demonstration.Source: §6 · p. 11
• The findings suggest embodied intelligence may emerge through interfaces and demonstrations that unlock foundation-model capabilities, as well as through scaling robot data.Source: §6 · p. 11