PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation

Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush Korivand

arXiv:2610.14106v1cs.ROcs.AI

TL;DR

Language-based manipulation plans can be plausible yet violate robot and scene constraints. This paper presents a bounded-authority LLM–VLM framework that separates symbolic planning from deterministic grounding and execution; in simulation, it completed 37 of 40 trials versus 4 of 40 for the baseline.

  • Problem

    Language models can propose plausible manipulation plans, but robotic execution must respect physical constraints such as reachability, collision geometry, and grasp feasibility.

    Source: §I · pp. 1–2

  • Method

    A bounded-authority LLM–VLM framework uses symbolic planning and semantic verification while deterministic robot-side modules handle metric grounding, execution, and verification.

    Source: §I · p. 2, C. Constrained Planning and Authority… · p. 3, D. Deterministic Grounding, Execution,… · p. 4

  • Results

    37 of 40 trials (92.5%) for the proposed stack versus 4 of 40 (10.0%) for the baseline.

    Source: §III · p. 5

  • Takeaways & Limitations

    The results support keeping physically consequential decisions under deterministic robot-side authority while using foundation models for semantic interpretation and task structuring.

    Source: §III · p. 5, §IV · p. 6

    Evidence is restricted to simulated FR3 tabletop manipulation in NVIDIA Isaac Sim; physical FR3 deployment is not claimed as a completed result.

    Source: §III · p. 5

Abstract

from arXiv · show

Large language models can translate natural-language instructions into plausible manipulation plans, but robotic execution requires physically grounded decisions that respect robot embodiment, workspace geometry, reachability, collision constraints, and grasp feasibility. This paper presents a hardware-aware large language model-vision-language model (LLM-VLM) manipulation framework for a simulated Franka Research 3 robot in NVIDIA Isaac Sim. The framework separates symbolic reasoning from metric execution: an API-hosted LLM planner receives a structured scene summary and generates only symbolic task information, including the object to manipulate, target relation, approach preference, and objects to avoid. A local Qwen3-VL 8B VLM verifier supports semantic confirmation of ambiguous objects and open-container surrogates. Both the LLM planner and the VLM verifier are prohibited from generating robot coordinates, grasp poses, keepout dimensions, gripper commands, trajectory waypoints, or success labels. Instead, these decisions are handled by deterministic grounding and execution modules that use the robot model, scene geometry, MoveIt 2 collision checking, inverse kinematics, grasp validation, release verification, and final visual assessment. Execution feedback is returned only as symbolic context for replanning. The framework is evaluated through simulated digital-twin trials involving pickup, occluded-source access, keepout reasoning, relation placement, and sequential manipulation tasks.

I. INTRODUCTION

Language-conditioned models can interpret intent and propose task structure, but plausible plans may still violate robot and scene constraints. The paper responds with a bounded-authority architecture that reserves metric decisions and execution for deterministic robot-side modules.

I. INTRODUCTION

  • Prior language-conditioned systems support skill ranking, executable programs, action sequencing, and feedback replanning, while other work uses language models to define objectives for downstream controllers.These approaches motivate using foundation models for intent and task structure while grounding execution in the robot, scene, and control interface.Source: §I · p. 1
  • Tightly coupled perception, language grounding, and action generation in embodied models can make it difficult to identify why a manipulation attempt failed.Possible causes include visual interpretation, ambiguous language, invalid actions, hardware incompatibility, or execution dynamics.Source: §I · p. 1
  • Foundation models can propose semantically reasonable actions that remain physically invalid because they lack calibrated knowledge of robot and scene constraints.Examples include unreachable placements, infeasible grasps, and keepout geometry that must reflect the physical scene and end-effector.Source: §I · pp. 1–2
  • The framework limits foundation models to symbolic task proposals and assigns grounding, motion feasibility, execution, and verification to deterministic modules.This supports failure attribution to explicit stages rather than an opaque end-to-end model; the paper presents an auditable architecture, not a new general-purpose VLA policy or a claim of mature manipulation performance.Source: §I · p. 2

A. System Overview

The system turns a natural-language command and RGB-D observation into a structured scene representation, then separates symbolic planning from robot-side execution. Schema validation enforces the boundary before plans enter the execution layer.

A. System Overview

  • SceneMemory stores object hypotheses, semantic labels, visual masks, approximate pose proxies, visibility states, and symbolic access annotations.It provides the structured world-state representation built from each command and RGB-D observation.Source: A. System Overview · p. 2
  • The pipeline converts a user command and RGB-D scene into a structured world state before symbolic planning, authority validation, deterministic grounding, MoveIt 2 execution, and feedback replanning.The stages are depicted in Fig. 1.Source: A. System Overview · p. 2, Figure 1 · p. 3
  • The planner returns a symbolic task plan from the structured scene summary, and a schema validator prevents metric robot commands from entering execution.Source: A. System Overview · p. 3

B. Simulation Platform and Scene Representation

The framework is evaluated in a simulated FR3 tabletop environment with a local VLM restricted to semantic verification. SceneMemory combines visual and geometric information with access annotations for planning and grounding.

B. Simulation Platform and Scene Representation

  • Evaluation uses a simulated Franka Research 3 with a parallel-jaw gripper in NVIDIA Isaac Sim, with ROS 2 Jazzy and MoveIt 2 providing motion planning and execution.An API-hosted GPT-5 mini planner and local Qwen3-VL 8B verifier are used; the verifier confirms semantic roles but does not set metric execution values or task success.Source: B. Simulation Platform and Scene… · p. 3
  • SceneMemory fuses open-vocabulary detections, SAM2-refined masks, and RGB-D estimates into entries containing object properties, pose proxies, visibility, and access state.Access annotations can indicate top-down or side feasibility, occlusion, keepout sensitivity, or a need for re-observation, while deterministic modules decide final feasibility.Source: B. Simulation Platform and Scene… · p. 3

C. Constrained Planning and Authority Boundary

The planner is a frozen language model whose output is restricted to symbolic task fields. Metric decisions are rejected before execution and assigned to deterministic modules.

C. Constrained Planning and Authority Boundary

  • The planner outputs symbolic fields such as task type, source and target references, placement relation, approach preference, avoid-object identifiers, and a re-observation flag.It is a prompted interface to a frozen pretrained language model, not a robot controller.Source: C. Constrained Planning and Authority… · p. 3
  • Coordinates, joint angles, gripper widths, keepout dimensions, release heights, waypoints, and success labels are rejected before execution.The planner may name an object to avoid or request a side approach, but deterministic modules determine the keepout volume, approach pose, insertion center, and release location.Source: C. Constrained Planning and Authority… · p. 3

D. Deterministic Grounding, Execution, and Feedback

The deterministic grounder turns symbolic plans into metric execution parameters, while robot-owned modules handle collision-aware motion, verification, and symbolic replanning.

D. Deterministic Grounding, Execution, and Feedback

  • The grounder maps each symbolic plan to a predefined behavior and computes execution parameters using command references, SceneMemory, visual attributes, and geometry.VLM confirmation can break semantic ties but cannot override geometric checks.Source: D. Deterministic Grounding, Execution,… · p. 3
  • Runtime parameters for pick, place, and related routines derive from RGB-D geometry, object footprints, access annotations, workspace limits, and keepout constraints.The listed routines include re-observation, approach, retreat, grasp, lift, and release.Source: D. Deterministic Grounding, Execution,… · p. 3
  • The simulated scene contains an FR3 arm, colored blocks, and an open-container surrogate that can act as a target or occluder.Source: Figure 2 · p. 4
  • MoveIt 2 checks collisions, generates trajectories, and executes pose goals; pickup and placement also require physical and visual verification.Pickup checks approach, grasp contact, lift, and carry consistency; placement checks target-region acceptance and post-placement visual confirmation.Source: D. Deterministic Grounding, Execution,… · p. 4
  • For avoidance, the executor expands a SceneMemory footprint with task- and gripper-dependent margins; failures return as symbolic context for replanning.The planner selects which object to avoid, while the executor computes its metric keepout; revised plans must pass schema validation before execution.Source: D. Deterministic Grounding, Execution,… · p. 4

E. Evaluation Protocol

The evaluation separates validation, grounding, motion execution, and outcome verification, with success determined by visual confirmation of the commanded state change. Its figures show the perception-to-grounding pipeline and an example detector output.

E. Evaluation Protocol

  • A trial counts as successful only when final visual labeling confirms the commanded state change, and evaluation stages localize failures across validation, grounding, motion, and verification.Source: E. Evaluation Protocol · p. 4
  • The baseline uses the same commands, perception, SceneMemory, grounder, executor, MoveIt 2 interface, and scoring, but disables local VLM assistance.Source: E. Evaluation Protocol · p. 4
  • Fig. 3 depicts command terms and scene census guiding detection, mask refinement, RGB-D estimation, and limited VLM assistance into SceneMemory for deterministic grounding.Source: Figure 3 · p. 4
  • Fig. 4 shows a detector bounding box and center marker localizing the requested pickup object before evidence is fused for deterministic grounding.Source: Figure 4 · p. 4

III. RESULTS AND DISCUSSION

In 40 Isaac Sim trials, the proposed stack substantially outperformed the baseline, with results varying across command families. The discussion attributes the pattern to bounded semantic assistance alongside deterministic execution, while noting the study’s simulated and task-limited scope.

III. RESULTS AND DISCUSSION

  • The evaluation comprised 40 trials across five command families, with eight trials per family and final visual confirmation required for success.Representative artifacts include partially occluded pickup, object pickup, container placement, and multi-step stacking.Source: §III · p. 4, Figure 5 · p. 5
  • The proposed stack completed 37 of 40 trials (92.5%), versus 4 of 40 (10.0%) for the baseline.The baseline disabled local VLM assistance while keeping the rest of the execution stack unchanged.Source: Table I · p. 5, §III · p. 5
  • The results support semantic assistance for interpretation and task structuring, while reserving physically consequential decisions for deterministic robot-side authority.The local VLM improved semantic robustness but did not replace deterministic grounding, motion planning, or outcome verification.Source: §III · p. 5
  • Clear-object pickup succeeded in 8 of 8 trials versus 2 of 8, and size-distractor pickup in 7 of 8 versus 2 of 8.The size-distractor condition tested ambiguity between same-color, same-shape-category objects of different sizes.Source: §III · p. 5
  • Partially occluded pickup succeeded in 6 of 8 trials, compared with 0 of 8 for the baseline.This task required access-state reasoning, re-observation, and side-approach behavior without relaxing deterministic safety checks.Source: §III · p. 5
  • The proposed stack completed all 8 container-placement trials and all 8 sequential-manipulation trials, while the baseline completed none of either.Sequential trials updated and reused SceneMemory across manipulation stages.Source: §III · p. 5
  • Evidence is limited to simulated FR3 tabletop manipulation and the tasks supported by the current executor; physical FR3 deployment remains future work.The current SceneMemory stores object hypotheses, approximate pose proxies, visibility, and discrete access states, rather than a complete semantic SLAM map or learned dynamics model.Source: §III · pp. 5–6

IV. CONCLUSION

The paper concludes that bounded semantic planning and verification can improve simulated FR3 manipulation while deterministic modules retain control of physically consequential decisions. Its 40-trial Isaac Sim evaluation supports this claim, with physical deployment and broader task coverage left for future work.

IV. CONCLUSION

  • The bounded-authority stack achieved 92.5% success across 40 Isaac Sim trials, compared with 10.0% for the baseline, supporting semantic assistance under explicitly bounded authority.The conclusion describes the architecture as a traceable middle ground between conventional deterministic manipulation pipelines and end-to-end foundation-model control.Source: §IV · p. 6

Related papers

Loading 2610.14106v1…