PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework

Kazi Abrar Mahmud, Nilotpaul Kundu Dhurubo, Tamal Kirttonia, Sabbir Hossain Ujjal, Mohammad Ariful Haque

arXiv:2609.13335v1cs.ROcs.AI

TL;DR

Open-source LLMs can be unstable in long-horizon robotic tasks, where prior work still identifies reliability as an open challenge. This paper adds a persistent, plan-conditioned stage to ROS-Agent, and reports higher task success than the vanilla ROS-Agent baseline in its real-world task evaluation.

  • Problem

    Long-horizon robotic planning and execution reliability remain challenging for open-source LLMs, despite diverse agentic approaches.

    Source: §I · p. 1, §II · p. 2

  • Method

    The MetaTool generates a structured pseudo-code tool plan before execution and uses the persistent plan to condition tool selection within ROS-Agent.

    Source: §I · p. 1, B. The MetaTool: Structured Reasoning… · p. 3

  • Results

    75.0% (30/40) overall success for MetaTool versus 57.5% (23/40) for vanilla ROSA, with tool unavailability/timeout at 3/40 for MetaTool.

    Source: C. Overall Results and Efficiency… · pp. 6–7

  • Takeaways & Limitations

    The experiments report improved reasoning stability, task coordination, and execution reliability for open-source LLMs, particularly on complex multi-step tasks.

    Source: §V · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) have enabled more natural human-robot interaction, but open-source models often exhibit unstable long-horizon reasoning and inefficient action execution when deployed in agentic robotic frameworks. This paper presents an enhanced ROS-Agent based architecture that improves task reliability and execution efficiency for agentic robotic systems using open-source LLMs. The proposed system introduces a novel intermediate mechanism, termed the MetaTool, which enforces structured planning prior to action execution. Given a natural-language command, the MetaTool induces the LLM to generate a pseudo-code plan of intended tool invocations, which is stored in the ROS-Agent's scratchpad and persists throughout execution. By explicitly separating planning from execution, the proposed approach reduces execution loops and improves deterministic behavior. The architecture is validated on a custom mobile robotic platform with multimodal perception and motion control capabilities. Experimental results on real-world interactive tasks demonstrate improved task completion and contextual consistency, with up to ~24% gains on complex tasks compared to the baseline framework.

I. INTRODUCTION

Open-source LLMs can become unstable over long sequences of robotic tool calls, motivating an architecture that plans tool use before execution. The proposed MetaTool-enhanced ROS-Agent is evaluated in simulation and on a custom robotic platform.

I. INTRODUCTION

  • The MetaTool generates a structured pseudo-code tool plan before execution and keeps it in ROS-Agent scratchpad memory throughout the observation–reasoning–action loop.The plan conditions later tool invocations, separating intent formation from execution.Source: §I · p. 1
  • Long-horizon reasoning by open-source LLMs can produce redundant actions, execution loops, and inconsistent task progression, limiting reliability on complex multi-step tasks.The paper motivates architectural solutions as an alternative to the cost and limited accessibility of proprietary models.Source: §I · p. 1
  • The framework is evaluated in ROS–Gazebo simulations and on a custom mobile robot integrating perception, facial expression, arm gestures, and mobility.The platform components are managed through a ROS-based control stack.Source: Figure 1 · p. 1, §I · p. 2
  • Table I compares representative robotic-planning systems; the paper characterizes its framework as combining explicit planning, execution constraints, ROS-native integration, and an open-LLM focus.The framework is also described as validated on a real robotic platform.Source: §I · p. 2, Table I · p. 2

II. LITERATURE REVIEW

Prior work explores tool use, memory, structured planning, multimodal feedback, and specialized control for LLM-driven robots. The review identifies long-horizon stability and execution reliability, particularly for open-source LLMs, as remaining challenges.

II. LITERATURE REVIEW

  • ROS-Agent connects ROS-based robots to language interfaces for interpreting commands and invoking services, while open-source LLMs can struggle with long-horizon planning without structured architectures or explicit memory.The review also notes strong performance by large closed-source models on complex tasks.Source: §II · p. 2
  • Toolformer learns API calling, and Codex-style executable pseudo-code plans represent multi-step robotic behavior as “code as policies”.The review presents these as examples of explicit, program-like planning.Source: §II · p. 2
  • Agentic planning ranges from interleaved reasoning and acting [10] and grounded mid-level plans to embodied-data fine-tuning with replanning in RoboGPT.Other approaches use hierarchical scene graphs, receding-horizon subtask refinement, or retrieval-augmented feedback-aware planning.Source: §II · p. 2
  • Related systems add tactile or multimodal feedback,, human approval for risky actions, motion planning and memory,, or layered and specialized control –.Despite these varied capabilities, long-horizon stability and execution reliability remain open challenges, especially for open-source LLMs.Source: §II · p. 2

III. METHODOLOGY

The methodology section introduces the MetaTool-enhanced ROS-Agent architecture and identifies its reasoning loop, action space, and memory mechanisms as key components.

III. METHODOLOGY

  • The proposed methodology centers on a MetaTool-enhanced ROS-Agent architecture.The section introduces the system before describing its components.Source: §III · p. 2
  • The architecture includes an agentic reasoning loop as one of its key components.The passage identifies the loop but does not further describe its operation here.Source: §III · p. 2
  • The methodology also treats the action space and memory mechanisms as key system components.Both are listed alongside the reasoning loop in the section overview.Source: §III · p. 2

A. Agentic Architecture, Action Space, and Memory

The ROS-Agent receives a user query alongside a prompt that specifies the robot embodiment and available tools, then bridges the LLM response to hardware execution. A MetaTool planning stage stores a pseudo-code plan in scratchpad memory to condition tool calls.

A. Agentic Architecture, Action Space, and Memory

  • The user query is sent to the LLM with a prefabricated system prompt encoding the robot’s embodiment and available tools, constraining reasoning to platform capabilities.The system operates through an iterative observation–reasoning–action loop.Source: A. Agentic Architecture, Action Space,… · p. 2
  • ROS-Agent bridges the LLM’s language-level intent and hardware execution through ROS.The architecture is shown in Fig. 2.Source: A. Agentic Architecture, Action Space,… · p. 3, Figure 2 · p. 3
  • On first invocation, the MetaTool generates a structured pseudo-code plan before task-specific tools run and stores it in a scratchpad accessible throughout the interaction.Each tool invocation is conditioned on this externalized plan rather than transient local context, reducing repetitive or circular calls.Source: A. Agentic Architecture, Action Space,… · p. 3
  • The action space includes native ROS tools such as rosnode and rostopic, plus custom perception, affective-expression, gesture-control, and navigation modules.Examples include describe image, face expression, and follow me.Source: A. Agentic Architecture, Action Space,… · p. 3

B. The MetaTool: Structured Reasoning and Execution

The MetaTool turns a natural-language instruction into an ordered tool plan before execution, then uses persistent scratchpad memory to guide tool calls. Its plan-conditioned policy constrains which actions the agent can take.

B. The MetaTool: Structured Reasoning and Execution

C. Machine Configuration

The system uses an Intel-based onboard computer with an NVIDIA GTX 1050 Ti GPU, while perception and control models run locally and the LLM operates via the Groq API.

C. Machine Configuration

A. Experimental Setup and Metrics

The evaluation spans ROS-Gazebo simulation and a custom physical robot, with three task levels ranging from simple queries to contextually grounded sequential tasks. The setup compares MetaTool-enhanced ROS Agent with vanilla ROSA across proprietary and open-source LLMs.

A. Experimental Setup and Metrics

B. Ablation Study

The ablation varies planning persistence and execution constraints within one pipeline, with gains on demanding tasks most pronounced when persistent planning and constraints are combined.

B. Ablation Study

  • The four configurations progress from basic prompts (C1) to transient planning (C2), persistent scratchpad planning (C3), and plan-conditioned execution constraints (C4).They were evaluated within the same pipeline across Gazebo environments, with 20 independent trials per task; results are indicative trends, not statistically significant differences.Source: B. Ablation Study · p. 5
  • On T3, C4 raised success to 55% for GPT-OSS-20B, 70% for GPT-OSS-120B, and 70% for Kimi-K2, versus C3’s 35%, 50%, and 65%.On T2, GPT-OSS-120B reached 90% under C4 versus 80% under C3.Source: B. Ablation Study · p. 6
  • C3 persistent scratchpad planning improved T3 success from 30% to 35% for GPT-OSS-20B and from 40% to 50% for GPT-OSS-120B, compared with C1.The authors attribute the gains to reducing context-forgetting during multi-step execution.Source: B. Ablation Study · p. 6
  • Compared with transient planning in C2, C3 raised T1 success from 60% to 70% for GPT-OSS-20B and from 75% to 85% for Kimi-K2.The reported benefits were most pronounced for the open-source models.Source: B. Ablation Study · p. 6

C. Overall Results and Efficiency Analysis

On physical-robot trials, MetaTool-ROSA achieved higher task success than the evaluated planning baselines, alongside reported reductions in redundant and invalid tool calls. The analysis also reports lower token-use variance and more consistent execution behavior.

C. Overall Results and Efficiency Analysis

  • Across repeated runs, MetaTool produced compact, low-variance token profiles and limited recursive tool invocations, particularly for smaller open-source models.The paper reports that physical trials also showed fewer repeated and invalid tool calls and more consistent execution latency; Table V summarizes 40 GPT-OSS-120B hardware trials on T2.Source: C. Overall Results and Efficiency… · p. 6, Table V · p. 6, Figure 5 · p. 7
  • 75.0% success (30/40) with MetaTool exceeded vanilla ROSA’s 57.5% (23/40), CaP’s 50.0% (20/40), and ReAct’s 42.5% (17/40).The baselines’ core prompting strategies were re-implemented within the same agentic pipeline and ROS tool interface for controlled comparison.Source: C. Overall Results and Efficiency… · pp. 6–7, Table VI · p. 7
  • MetaTool reduced tool unavailability/timeouts to 3/40, compared with 13/40 for CaP and 17/40 for vanilla ROSA.ReAct had 13/40 navigation failures; failure categories were non-exclusive.Source: C. Overall Results and Efficiency… · pp. 6–7

V. CONCLUSION AND FUTURE WORK

The conclusion presents MetaTool-enhanced ROS-Agent as a structured execution approach that separates planning from tool use. The authors report improved stability and reliability, especially on complex multi-step tasks, and identify smaller models and broader applications as future work.

V. CONCLUSION AND FUTURE WORK

  • MetaTool prompts the LLM to generate a pseudo-code task sequence before tool invocation, separating high-level reasoning from execution in ROS-Agent.The paper reports improved reasoning stability, task coordination, and execution reliability for open-source LLMs, particularly on complex multi-step tasks.Source: §V · p. 7
  • Future work will adapt the architecture to smaller open-source models for resource-constrained robots and extend the platform and tool ecosystem to broader real-world applications.Source: §V · p. 7
Loading 2609.13335v1…