PaperlayerView exact revision on arXiv

Source-linked AI summary

iAm.md: Robot Skill Self-Assessment through Agentic Introspection for Unknown Open-Vocabulary Domains

Vincenzo Guarino, Emanuele Musumeci, Vincenzo Suriani, Daniele Nardi

arXiv:2610.10962v1cs.ROcs.AI

TL;DR

Embodied agents can generate plausible plans, but a plan may rely on unobserved objects or operations the deployed robot does not support. The paper combines persistent semantic mapping, a deployment-derived robot description, and live ROS 2 evidence for introspective task-code generation. In a simulated TIAGo evaluation, iAm.md access was associated with verified tool completion, and the retained tool was extended to a second task in a single simulated run.

  • Problem

    Embodied task planning needs evidence that requested objects and operations are supported by the observed environment and deployed robot.

    Source: §1 · p. 1

  • Method

    The system combines open-vocabulary persistent scene memory, a structured robot self-description, and live ROS 2 observations for agentic task assessment and code execution.

    Source: §1 · p. 2, §3 · pp. 2–3

  • Results

    50.4 minutes for a verified tool with iAm.md, versus the 110-minute budget without a verified grasp; the retained tool was extended to a second task without rereading iAm.md in one simulated run.

    Source: §5 · p. 5

  • Takeaways & Limitations

    Execution checks and revisions derived from failures remained in the tool and were reused when the request changed, forming a feedback loop for self-assessment.

    Source: §4.3 · p. 5

    The second-task reuse evidence comes from a single simulated run.

    Source: §5 · p. 5

Abstract

from arXiv · show

Agentic AI based on Large Language Model generalization capabilities offers a wide range of potential applications, including planning for embodied tasks. For example, embodied agents based on Foundation models can generate plausible plans in autonomous robotics scenarios. Due to limited context windows or hallucinatory phenomena in the next-token prediction formulation, behaviors may be generated without establishing whether the deployed robot and the observed environment actually support the requested operation, in what we call a "grounding failure". Thanks to the recent improvements in reasoning capabilities of foundation models, autonomous robot behavior generation problem can be formulated as a code generation problem. We present iAm.md, a Markdown standard and generation framework, that allows anchoring this process in complementary forms of deployment evidence. Through open-vocabulary semantic mapping, we combine local vision-language detections and object segmentation and refer them to persistent object records in this intermediate standardized representation, allowing agentic introspection. We then study this new technique on a simulated TIAGo, on navigation-and-manipulation tasks, showing how this standardized representation jointly supports skill self-assessment and executable task generalization.

1. Introduction

Embodied task planning depends on adequate representations of the world and actions, yet language-generated plans may rely on unobserved evidence or unsupported operations. The paper introduces persistent semantic memory, iAm.md, and an introspective agent to assess and execute natural-language requests.

1. Introduction

  • Neuro-symbolic planning makes plan validity conditional on the quality of the provided world-state and action abstraction.Language models may also complete underspecified requests with synthetic knowledge, and generated actions can violate their preconditions.Source: §1 · p. 1
  • A generated program may refer to an unobserved object or request an operation unsupported by the deployed robot.The paper frames self-assessment as estimating how well a robot can perform a task in a particular context and environment.Source: §1 · p. 1
  • The proposed system combines open-vocabulary semantic memory, an iAm.md robot-description standard, and an introspective agent that generates and executes task code.The semantic memory uses local vision-language inference and segmentation to maintain persistent object identities and locations.Source: §1 · p. 2

2. Related Work

Prior work provides persistent scene representations, language-conditioned code generation, feasibility assessment, and robot-knowledge frameworks, but the paper identifies limitations in their robot-specific execution evidence or assumptions. It derives robot descriptions from deployment and uses live interface evidence to generate task-specific code.

2. Related Work

  • Scene graphs and dense vision-language maps accumulate spatial structure but do not maintain robot-specific execution evidence.Source: §2 · p. 2
  • Language-conditioned code-generation methods compose developer-provided APIs but assume feasible instructions and pre-authored platform documentation [13].Source: §2 · p. 2
  • Unlike EMPOWER-ADF’s fixed action vocabulary and lack of physical execution, this work derives robot descriptions from deployment and uses live interface evidence to write task-specific code.Other feasibility methods require a complete action model, while robot-knowledge frameworks rely on information entered before an episode.Source: §2 · p. 2

3. Methodology

The methodology links deployment-derived robot evidence and persistent semantic mapping to an embodied agent that inspects live interfaces, generates task code, and revises it using execution outcomes. iAm.md records established embodiment and software-interface facts alongside explicit unknowns.

3. Methodology

  • Open-vocabulary mapping uses vision-language detections and segmentation to associate observed objects with persistent map records.Stored voxels provide centroids, bounds, and oriented boxes in a JSON snapshot and RViz markers, supporting queries for entities outside the current camera view.Source: §3 · p. 2
  • The system combines iAm.md, a semantic map, and a natural-language request to generate code for successive subgoals, with execution outcomes informing introspective revision.The generation agent records resolved ROS 2 evidence in iAm.md; the execution agent uses the description and semantic map during the task.Source: §3 · p. 2, Figure 1 · p. 3
  • iAm.md follows a fixed schema for robot identity, physical structure, actuation, sensing, software functions, and unresolved unknowns.Each deployment’s document is generated from its URDF and augmented with live ROS 2 graph and configuration evidence; command acceptance is distinguished from physical outcome.Source: §3 · p. 3
  • During each request, the agent checks observed objects, the robot description, and live interfaces against the operation’s needs before proceeding or recording a blocked outcome.Execution observations can expose unsupported assumptions and prompt program revision.Source: §3 · p. 3

4. Experimental Results

The experiments evaluate semantic-map recovery and whether persistent robot knowledge supports tool construction and reuse. The discussion frames skill self-assessment as a feedback loop in which execution evidence prompts revision.

4.1. Semantic-Map Evaluation

  • The map recovered all 12 annotated objects (Found = 100%), while its 13th estimated record was a duplicate (Correct = 92.31%).The unmatched duplicate localized the observed error to duplicate consolidation during map updates and merges.Source: §4.1 · p. 4

4.2. Agent-Produced Tool Evaluation

  • For grasping a drink can and placing it on a sofa, the documented agent produced a tool that passed verification after reset in 50.4 minutes; without iAm.md, it reached 110 minutes without verified grasp or task completion.The undocumented run made 186 model calls and used 20.32 million input tokens.Source: §4.2 · p. 4
  • On the changed-object reuse task, the retained tool passed verification after a simulator reset on its first attempt, with no further iAm.md reads.The geometry change still required 49 tool edits and 34 reported step failures.Source: §4.2 · p. 4, Table 1B · p. 5

4.3. Discussion

  • Self-assessment formed a feedback loop: execution exposed unsupported assumptions, and resulting executable checks remained in the tool for reuse after the request changed.Source: §4.3 · p. 5

5. Conclusions and Future Work

The conclusion reports that an embodied foundation-model agent combined persistent scene memory, a robot self-description, and live ROS 2 observations to assess and realize open-vocabulary tasks. It reports object recovery, lower construction cost with iAm.md, and reuse on a second task, while identifying embodiment coverage and physical-robot safety as future work.

  • The agent combined persistent scene memory, a robot self-description, and live ROS 2 observations to assess and realize natural-language tasks in an open-vocabulary domain.Source: §5 · p. 5
  • The semantic map recovered all 12 annotated objects with one duplicate record, and the documented agent produced a verified tool in 50.4 minutes versus the 110-minute budget without a verified grasp.Source: §5 · p. 5
  • The agent reused its tool for a second task with different object geometry without rereading iAm.md, based on a single simulated run; future work will evaluate other embodiments and add physical-robot trajectory validation.Source: §5 · p. 5
Loading 2610.10962v1…