Source-linked AI summary
Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation
Wenhao Hong, Lan Wei, Dandan Zhang
TL;DR
Robotic manipulation needs to bridge linguistically plausible plans and physically safe execution under clutter and imperfect perception. The paper introduces a typed MCP contract that grounds validated language actions in a collision-aware MoveIt pipeline and verifies motions before acting. Across real-robot tasks, it reports 72.5% mean pouring success and 90% grasp-and-place success, while retaining the same stack across skills.
Problem
Language models can generate plausible action sequences, but the language-action gap leaves plans potentially infeasible under kinematic and collision constraints.
Method
A typed MCP interface constrains language decisions to validated semantic tool calls, which are deterministically grounded in MoveIt templates and checked against an RGB-D-derived planning scene.
Results
72.5% mean end-to-end success was achieved across four pouring tasks, versus 50.0% for scripted FSM/BT and 47.5% for unstructured LLM; grasp-and-place reached 90%.
Takeaways & Limitations
The same planning, protocol, and verification stack transferred to grasp-and-place by adding a task-specific tool schema and MTC template.
Takeaways & Limitations
The approach remains limited by reconstructed-scene accuracy, lacks material-dynamics modeling and closed-loop execution feedback, and requires evaluation under more challenging clutter and occlusion.
Abstract
from arXiv · showhide
A language-guided robot operating in a real kitchen must do more than produce a plan that appears correct. It must also execute that plan safely in cluttered environments under imperfect perception. Large language models (LLM) can decompose instructions into action sequences, yet a language-action gap remains: a plan may appear valid linguistically while being physically infeasible under kinematic and collision constraints. We bridge this gap by formalizing the reasoning-execution boundary as a typed contract. From RGB-D observations, the system grounds perceived objects in an explicit, collision-aware scene model and constrains language-level decisions through schema-validated tool calls defined by the Model Context Protocol (MCP), rejecting malformed commands before they reach the robot. Each validated call is deterministically grounded in a MoveIt Task Constructor pipeline, where candidate motions are evaluated against the reconstructed planning scene in a verify-then-act step. Only trajectories that pass both kinematic and collision checks are sent to the robot. On a physical UFactory 850, the method achieves up to 80% success across ten trials per task on pouring tasks involving liquids, granular media, and discrete solids. It achieves 90% success on a grasp-and-place task using the same planning, protocol, and verification stack. Although a scripted policy slightly outperforms our method on the easiest task, its success rate falls to 10% on the hardest, compared with 60% for our method.
1 Introduction
The paper addresses the language-action gap in robotic manipulation: linguistically plausible plans can remain physically infeasible or unsafe. It proposes a typed, verified boundary connecting language reasoning to collision-aware robot execution.
- Natural-language manipulation requires grounding references, translating underspecified intent into actions, and satisfying kinematic and collision constraints.
- LLMs can produce symbolic plans or structured tool calls, but semantically plausible plans may remain physically infeasible.
- Each accepted call is deterministically grounded in MoveIt Task Constructor, which checks kinematic feasibility and collisions before robot motion.
- The typed planning contract maps language decisions to schema-validated tool calls and rejects malformed or unsupported commands before motion planning.
- Semantic object references and bounded skill parameters allow the interface to extend across skills by adding tool schemas and motion templates without modifying the planner or execution stack.
2 Related Work
Prior work connects language reasoning with executable policies, task-and-motion planning, and visually grounded manipulation. This paper instead emphasizes a typed, auditable boundary that keeps geometric execution and verification outside the language model.
- Prior systems translate language into robot APIs, combine language planning with motion planning and control, or formalize instructions for task-and-motion planning.
- The proposed interface restricts the LLM to schema-validated calls containing semantic references and bounded skill parameters, not poses or trajectories.
- Accepted calls are mapped to predefined MoveIt Task Constructor templates, while geometric motion generation and verification remain in the execution layer.
- RGB-D detections are transformed into the robot frame and inserted as collision geometry, giving semantic grounding and geometric verification a shared representation.
- Unlike work focused on pouring perception or low-level control, this paper evaluates language-specified transfer through structured sequencing, verification, and recovery.
3 Method
The method separates semantic interpretation, typed protocol validation, and verified execution. It grounds RGB-D observations into a planning scene, maps validated actions to deterministic motion templates, and executes only checked motions.
- 3 Method: Language-guided pouring must resolve referential ambiguity while producing motion that respects kinematic and collision constraints.
- 3.2 The Semantic Layer: The Semantic Layer outputs structured action requests without specifying robot poses or trajectories, while the Protocol Layer enforces schemas and argument constraints.
- 3.2 The Semantic Layer: The symbolic state represents source and target containers with a Boolean material-transfer flag, leaving poses, approach directions, and trajectories to execution.
- 3.2 The Semantic Layer: Depending on instruction specificity, the planner uses constrained information-seeking reasoning for ambiguity or a deterministic task graph for well-specified commands.
- 3.3 The Protocol Layer: MCP tools contain semantic references and bounded parameters; schema, range, and referential validation reject invalid calls before they reach the motion planner.
- 3.4 The Execution Layer: RGB-D perception transforms detected objects into the robot frame and inserts collision geometry into the MoveIt planning scene for verification.
- 3.4 The Execution Layer: Pouring uses staged tilt-and-hold motions with bounds on tilt, angular velocity, and dwell time, plus collision-checked safe poses between high-risk stages.
- 3.4 The Execution Layer: Only motions validated for continuity, reachability, and collision avoidance in the reconstructed scene are sent to hardware.
4 Experiments
Real-robot experiments evaluate the method across increasingly difficult pouring scenarios, perception conditions, and a fruit grasp-and-place transfer. Results show stronger performance than baselines in constrained tasks, while protocol recovery mechanisms materially affect robustness.
- Experimental setup: Four pouring tasks were evaluated on a UFactory 850 over ten independent physical trials, with success requiring completed transfer and source-container return.The suite spans single-source, multi-source, multi-material, and cluttered scenarios.
- Experimental setup: The protocol comparison used shared robot, perception, and motion-planning backends, contrasting the full method with scripted and unstructured-LLM baselines.The unstructured baseline retained schema validation, recovery, planning, and verify-then-act checks but removed canonical task-graph sequencing.
- Task performance: 72.5% average success exceeded the scripted policy's 50.0% and the unstructured LLM's 47.5%, with the largest gains in constrained scenes.The scripted policy reached 90% on T1 but fell to 10% on T4, compared with 80% and 60% for the proposed method.
- Generalization beyond pouring: 90% end-to-end success was achieved on the fruit grasp-and-place task while retaining the planner, protocol layer, and verify-then-act mechanism.Transfer changed only the skill-specific tool schema and MTC template; the experiment comprised ten trials.
- Multi-material robustness: 80% success was achieved on T3 across liquids, granular media, and discrete solids using the same staged pouring profile and task-level interface.The result was obtained without material-specific planning logic, although failure sensitivities differed across material classes.
- Ablation studies: Removing retry reduced average success by 47.5 percentage points, while removing safe-pose reset reduced it by 32.5 points and removing the task graph by 25.0 points.These ablations retained schema validation and verify-then-act, isolating protocol-level recovery mechanisms.
- Ablation studies: Conclusions are limited by ten trials per condition, so individual per-task differences are interpreted cautiously in favor of overall trends.Residual pose uncertainty also remains a source of model–world mismatch despite geometric verification and conservative execution.
5 Conclusions and Future Work
The paper presents a typed reasoning-execution interface that verifies candidate motions before robot execution, achieving strong performance across pouring and grasp-and-place tasks. Its current scope is limited by scene reconstruction accuracy, absent material dynamics, and missing closed-loop feedback.
- Conclusions: 90% success was achieved on a grasp-and-place task while retaining the same planning, protocol, and verification stack.
- Conclusions: Structured sequencing and protocol-level recovery both contributed to reliability, with retry producing the largest measured effect in the ablations.
- Future Work: The approach remains limited by reconstructed planning-scene accuracy, absent material-dynamics modeling, and lack of closed-loop execution feedback.
- Future Work: Future work will add visual, tactile, and audio feedback and evaluate robustness under occlusion, dynamic scene changes, and denser multiobject clutter.