RACAS: Controlling Diverse Robots With a Single Agentic System

• Robot APIs expose low-level commands and sensor readings, but connecting them to high-level behavior requires expertise across distinct specializations.The gap creates a barrier to applying robots across a growing diversity of platforms.Source: §I · p. 1
• Prior language-model approaches reduce embodiment-specific engineering but had been validated only on structurally similar platforms in controlled settings.The paper positions its evaluation across heterogeneous platforms as addressing this gap.Source: §I · p. 2
• Three cooperative LLM/VLM-based modules communicate only through natural language, while embodiment- and task-specific knowledge stays in declarative prompts.The modules are a Controller, Monitors, and a Memory Curator; no source code, model weights, or reward functions are modified.Source: Figure 1 · p. 1, §I · p. 2
• RACAS completed assigned tasks across a wheeled ground robot, a novel robotic limb, and an underwater vehicle, supporting zero-training generalization across radically different platforms.The system was unmodified across the evaluated platforms.Source: §I · p. 2

A. LLM/VLM-Based Robotic Control

• Earlier control frameworks use LLM planning, API-generated code, perceptual feedback, or multi-agent coordination, but their visual reasoning or deployment scope remains limited.Examples include SayCan, Code-as-Policies, Inner Monologue, MALMM, RoCo, and LLM2Swarm.Source: A. LLM/VLM-Based Robotic Control · pp. 2–3
• RACAS contrasts with prior multi-agent systems by using an LLM and VLM as functionally complementary agents in a general-purpose architecture.The cited comparison describes prior systems as text-only or using VLMs only at specific stages within a single-domain setting.Source: A. LLM/VLM-Based Robotic Control · p. 3

B. Cross-Embodiment Generalization

• Training-based transfer methods still require robot-specific data, retraining, or curated meta-training, while API-wrapped language-model frameworks avoid embodiment-specific training but lack validation beyond controlled laboratory settings.Examples include RT-1 [1], RT-2, RoboCat [3], and ChatGPT for Robotics.Source: B. Cross-Embodiment Generalization · p. 3
• Zero-training generalization across robots with fundamentally different morphology, dynamics, and operating environments remains rare.Source: B. Cross-Embodiment Generalization · p. 3

C. Memory and Adaptation Mechanisms

• Prior systems range from externally constructed scene graphs and manually defined action specifications to disposable maps, episode logs, and descriptive sensory traces.The cited examples include SayPlan, LLM-GROP, ProgPrompt, VoxPoser, Inner Monologue, and RONAR.Source: C. Memory and Adaptation Mechanisms · p. 3
• Figure 3 illustrates RACAS reasoning during a real-world Dingo run.Source: Figure 3 · p. 4
• Existing robot-control approaches lack organized, persistent memory for the distinct knowledge accumulated during operation.Some representations are manually specified, ephemeral, unstructured, or do not feed back into future control.Source: C. Memory and Adaptation Mechanisms · pp. 3–4

III. METHODOLOGY

• RACAS decomposes closed-loop control into perception, decision, and memory pipelines linked by natural language, with embodiment knowledge held in declarative prompts rather than learned parameters or engineered modules.Source: §III · p. 4

A. Problem Formulation

• The robot is specified by a natural-language description, admissible actions, camera images, and a user-defined task, without dynamics models or embodiment-specific learned weights.Source: A. Problem Formulation · p. 4
• At each step, RACAS selects one action from a robot-specific action set using visual observations, the task, and prior interaction history.Source: A. Problem Formulation · p. 4
• Unlike fixed-dimensional state and action vectors, the framework represents both symbolically in natural language and accommodates different action sets without architectural change.The passage contrasts an eight-primitive manipulator action set with six actions for an underwater vehicle.Source: A. Problem Formulation · p. 4

B. System Architecture

• The closed loop places a Controller, Monitors, and Memory Curator around a hardware abstraction layer.Source: B. System Architecture · p. 4
• The Controller issues a targeted visual query; Monitors describe camera images in natural language, and the Controller uses their observations to select an action.Source: B. System Architecture · p. 4
• After the action is dispatched through the hardware abstraction layer, the Memory Curator integrates the interaction record into persistent environment memory.Source: B. System Architecture · p. 4
• Adapting to a new robot requires replacing prompt configurations, not modifying the control logic.The configurations include the robot description, action definitions, and environment context.Source: B. System Architecture · p. 4
• At each step, the Controller receives robot, action, memory, proprioceptive, history, and task context before producing a visual query or an action.The passage describes the visual-query and action outputs in separate steps of the controller loop.Source: B. System Architecture · p. 4

C. Query-Driven Visual Perception

• At each step, the Controller generates a task-relevant visual query from the task objective, action history, and environment memory; Monitors analyze camera images and return natural-language scene descriptions.Source: C. Query-Driven Visual Perception · p. 4
• For low-resolution cameras, a two-stage learned super-resolution pipeline upscales images before VLM inference.Source: C. Query-Driven Visual Perception · p. 5
• Task-adaptive queries direct attention to different scene aspects as execution progresses, without architectural changes.Source: C. Query-Driven Visual Perception · p. 5
• Natural-language outputs share the Controller’s representational space and support open-vocabulary recognition without domain-specific training, unlike fixed-schema detectors or depth estimators.Source: C. Query-Driven Visual Perception · p. 5

D. LLM-Curated Persistent Memory

• The Memory Curator maintains a structured, bounded representation to avoid unbounded conversation growth and its eventual degradation.Source: D. LLM-Curated Persistent Memory · p. 5
• After each control step, the Curator rewrites prior memory using the latest interaction record, compressing redundancy, resolving contradictions, and discarding details no longer relevant to the task.Source: D. LLM-Curated Persistent Memory · p. 5
• Memory is organized into physical environment, robot state, curated command history, and task state, including objectives and completion status.Source: D. LLM-Curated Persistent Memory · p. 5
• The memory system infers relative object positions from which camera observed an object and which action brought it into view, updating those estimates as the robot moves.Source: D. LLM-Curated Persistent Memory · p. 5
• The Memory Curator was validated in a proof-of-concept experiment using the OpenAI Gym Blackjack environment.Source: D. LLM-Curated Persistent Memory · p. 5

IV. EXPERIMENTAL SETUP

• Across all platforms, the control logic and LLM modules remain identical; only robot descriptions, action definitions, and environment context in prompt files vary.Source: §IV · p. 5
• When safe, experiments also establish a baseline using randomly selected actions.Source: §IV · p. 6

A. Robot Platforms

• The limb has four servo-driven axes, a single-finger end effector, six onboard cameras with two side-facing cameras disabled, and nine available actions including reset.Source: A. Robot Platforms · p. 6
• The Dingo is tested in simulation and physically, using three cameras in simulation and one forward camera in the real environment; its four actions move or rotate on the ground plane.Source: A. Robot Platforms · p. 6
• The BlueROV2 is fully actuated in six degrees of freedom, but experiments restrict control to surge, heave, and yaw through six discrete commands.Source: A. Robot Platforms · p. 6
• Table I summarizes the key differences between the platform configurations.Source: A. Robot Platforms · p. 6, Table I · p. 6

B. Tasks

• The limb searches a cluttered laboratory workspace for a fire extinguisher through multi-camera observations and articulated movement; a separate VLM detects success, across 30 runs.Source: B. Tasks · p. 6
• The Dingo approaches a fire extinguisher using visual feedback, with success defined as reaching within 1 m; simulated and real-world experiments were each conducted 10 times.Source: B. Tasks · p. 6
• The BlueROV2 navigates to targets in a tank and a larger pool; tank runs use a cropped 512 × 512 px view and 10 trials, while pool trials use an uncropped view and 5 runs.Source: B. Tasks · p. 6

C. Prompt-Based Adaptation

• Adapting to a platform requires text files for its natural-language robot description, structured action definitions, and task description, with no changes to source code, model weights, or reward functions.Source: C. Prompt-Based Adaptation · p. 6

D. Implementation Details

• Inference used GPT-4.1 mini or GPT-4.1 via the official API, with the larger model used for real-world Dingo and large-pool BlueROV2 experiments.The control loop ran at approximately one action per 5–10 seconds, dominated by API latency, and total API cost was under $100.Source: D. Implementation Details · p. 6

V. RESULTS & DISCUSSION

• The reported comparison tabulates average steps to solve each setting against random baselines where relevant and an oracle-based minimum.The oracle minimum is an unreachable upper bound; in the large-pool setting, wind makes the optimal step count approximate.Source: §V · pp. 6–7, Table II · p. 7
• Figure 5 shows the distribution of steps needed to complete tasks in each setting; random policies were artificially limited to 25 steps.The supplied passage describes the figure but does not provide its plotted values.Source: Figure 5 · p. 7
• RACAS solved all tasks in a reasonable number of steps, and significantly outperformed random baselines where comparisons were made.The reported family-wise confidence was p < 0.01; some settings were unsafe for random-baseline testing.Source: §V · p. 7
• Task completion time was heavily dominated by searching for the object; once found, RACAS directed the robot almost straight toward it.The authors suggest performance may be more limited by sensor fidelity and available tools than by the underlying models or architecture.Source: §V · p. 7

VI. CONCLUSION

• The unmodified RACAS completed assigned tasks on three radically different platforms, supporting zero-training generalization across heterogeneous embodiments.The evaluation included a recently published robotic limb that an LLM would have no prior knowledge of.Source: §VI · p. 7
• RACAS confines embodiment-specific knowledge to declarative prompts, avoiding changes to source code, model weights, or reward functions between platforms.The authors say practitioners can adapt the system using the robot’s existing API and natural-language descriptions, reducing expertise and resources needed to prototype robotic solutions.Source: §VI · p. 7

VII. LIMITATIONS AND FUTURE WORK

• The evaluation covers target localization and navigation, only one stage of a typical robotic mission; high per-step inference cost makes long-horizon manipulation experiments prohibitively slow.Future work will explore callable manipulation primitives to reduce LLM-mediated decisions per task.Source: §VII · p. 7
• A recurring limitation was missing depth information, which led to suboptimal image-based inferences and over- or under-confidence about impact chances.The authors propose integrating LIDAR or a more sophisticated monitor architecture.Source: §VII · p. 7

APPENDIX

• The experiment replaces Blackjack’s familiar target of 21 with a hidden target of 42, disclosed only to lie within 12–100, which the agent must infer across episodes.The agent is not informed of the target value.Source: APPENDIX · pp. 7–8
• The Monitor identifies card values from rendered game-state images, while the Controller reasons about play strategy..Source: APPENDIX · p. 8
• The baselines disable the Memory Curator while retaining unstructured history, remove memory entirely, or select actions randomly.The no-memory system acts only on current visual observations.Source: APPENDIX · p. 8
• The reported result indicates that RACAS inferred feedback from previous episodes through memory updates and maintained high performance across multiple updates.The supplied passage gives a qualitative result rather than numerical values.Source: APPENDIX · p. 8
• Figure 6 is identified as showing a Blackjack visualization and experimental results.The supplied caption does not specify the figure’s axes or plotted comparisons.Source: Figure 6 · p. 8