PaperlayerView exact revision on arXiv

Source-linked AI summary

Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting

Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, Animesh Garg

arXiv:2303.14100v1cs.RO

TL;DR

Robots face a communication barrier in translating natural-language instructions into executable actions, while data scarcity and unverified outputs challenge LLM-generated domain-specific plans. CLAIRIFY combines iterative prompting with verification; it outperforms existing XDL generation in expert evaluation and demonstrates selected chemistry plans on a real robot.

  • Problem

    Robots struggle to translate natural-language instructions into executable actions, while domain-specific language data are scarce and generated plans may be unverified or syntactically invalid.

    Source: §I · p. 1

  • Method

    CLAIRIFY uses an LLM-based generator and rule-based verifier in iterative prompting to produce syntactically correct DSL plans that incorporate environmental constraints.

    Source: §I · p. 2, §V · p. 7

  • Results

    CLAIRIFY outperformed the existing XDL generation method in expert evaluation and generated better plans than existing baselines across two datasets.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    Selected generated plans were integrated with task and motion planning and demonstrated in real-world elementary chemistry experiments.

    Source: §I · p. 2

Abstract

from arXiv · show

Generating low-level robot task plans from high-level natural language instructions remains a challenging problem. Although large language models have shown promising results in generating plans, the accuracy of the output remains unverified. Furthermore, the lack of domain-specific language data poses a limitation on the applicability of these models. In this paper, we propose CLAIRIFY, a novel approach that combines automatic iterative prompting with program verification to ensure programs written in data-scarce domain-specific language are syntactically valid and incorporate environment constraints. Our approach provides effective guidance to the language model on generating structured-like task plans by incorporating any errors as feedback, while the verifier ensures the syntactic accuracy of the generated plans. We demonstrate the effectiveness of CLAIRIFY in planning chemistry experiments by achieving state-of-the-art results. We also show that the generated plans can be executed on a real robot by integrating them with a task and motion planner.

I. INTRODUCTION

LLM-generated robot task plans may be syntactically invalid or not executable, while data scarcity makes zero-shot generation for domain-specific languages difficult. CLAIRIFY pairs an LLM with an iterative verifier and is evaluated on chemistry plans, including expert comparisons and real-robot execution.

I. INTRODUCTION

  • Task-plan generation faces both a lack of verification and poor performance in data-scarce domain-specific languages, while syntactic rules and robot executability constrain usable plans.A plan can look reasonable yet still be syntactically invalid or not executable by a robot.Source: §I · p. 1
  • Limited online data makes zero-shot generation for domain-specific languages difficult, while fine-tuning requires sizable datasets and substantial computation.The authors propose using in-context prompting with the target language's rules instead.Source: §I · p. 1
  • CLAIRIFY iteratively sends generated plans to a verifier that checks syntax and environment constraints, then feeds errors back to the LLM.The framework provides the LLM a description of the target language and uses a rule-based verifier.Source: §I · p. 2
  • The generator–verifier interaction improves zero-shot task-plan generation, and human experts preferred CLAIRIFY's plans over the existing XDL generation method.The paper evaluates the method using Chemical Description Language (XDL).Source: §I · p. 2
  • CLAIRIFY-generated plans were integrated with a task and motion planner and successfully translated into real-robot execution of elementary chemistry experiments.The evaluation also reports that the method outperforms the current state-of-the-art XDL generation model.Source: §I · p. 2

A. Task Planning

Prior task-planning work includes rule-based and learning-based approaches, but learning methods can require datasets that are difficult to collect. LLMs offer zero-shot planning, while domain-specific languages and output correctness create distinct challenges.

A. Task Planning

  • Learning-based task planning can require large datasets that are difficult to collect, whereas CLAIRIFY generates plans zero-shot for constrained tasks translatable to robot actions.Rule-based planning is another approach, and task planning can become intractable as action counts and time horizons grow.Source: A. Task Planning · p. 2

B. Task Planning with Large Language Models

  • LLM-based robot planning includes methods that use perception feedback or low-level skills, but DSL generation needs target-language information beyond naive zero-shot prompting.Code-writing models can generate robot policies, though cited approaches produce Pythonic code, which is abundant online.Source: B. Task Planning with Large Language… · p. 2

C. Leveraging Language Models with External Knowledge

  • External tools and verifiers can improve language-model outputs, but CLAIRIFY targets formally verifiable DSLs without fine-tuning or prior target-language knowledge in the model.The paper distinguishes its approach from methods whose language models are expected to understand common grammars or whose verifiers require training data.Source: C. Leveraging Language Models with… · p. 2

III. TASK PROGRAMMING WITH CLAIRIFY

CLAIRIFY pairs a structured-language specification and natural-language instruction with iterative verification, returning verifier errors to the generator until a valid plan is produced or timeout occurs.

III. TASK PROGRAMMING WITH CLAIRIFY

  • CLAIRIFY combines the task instruction and language description in a prompt, then sends generated output through a verifier that checks language rules and compilability.Source: §III · pp. 2–3
  • The verifier-accepted program is syntactically valid and can be passed to TAMP for translation into lower-level robot actions.Source: §III · p. 3

A. Generator

  • The generator receives a description of the language’s file structure, available actions, allowed parameters, and documentation.Source: A. Generator · p. 3
  • When verification finds errors, the next prompt includes the previous plan and error list; iteration ends with error-free output or a timeout.Source: A. Generator · p. 3

B. Verifier

The verifier checks XDL syntax and selected executability conditions, while resource constraints can be supplied to guide generation and trigger error feedback.

B. Verifier

  • The verifier parses XML, checks allowed action tags and required or optional properties, and verifies definitions for hardware and reagents used.Source: B. Verifier · p. 3

C. Incorporating Environment Constraints

  • After an initial prompt with XDL and the instruction, errors prompt a retry using the initial prompt, previous XDL, and an error list.Source: Figure 3 · p. 4
  • When available resources are specified, the verifier flags plans that use other resources and sends those errors back for another generation attempt.Source: C. Incorporating Environment Constraints · pp. 3–4

D. Interfacing with Planner on Real Robot

The system connects verified high-level plans to robot-executable actions through task and motion planning, using visual perception to ground planning predicates.

D. Interfacing with Planner on Real Robot

IV. EXPERIMENTS AND EVALUATION

The experiments are designed to test iterative prompting, plan quality relative to existing methods, and execution on actual hardware.

IV. EXPERIMENTS AND EVALUATION

  • The evaluation tests whether automated iterative prompting increases success rates for generating an unfamiliar language.Source: §IV · p. 4
  • The experiments assess whether generated task plans are better than those from existing methods.Source: §IV · p. 4
  • The experiments test whether generated plans can be executed by actual hardware.Source: §IV · p. 4

A. Experimental Setup

The experiments evaluate CLAIRIFY on chemistry plans represented in XDL, using two datasets with different goals: complex-protocol plan generation and robot execution of safe, accessible experiments.

A. Experimental Setup

  • XDL plans are generated with text-davinci-003 rather than code-davinci-002 because of query and token limits.The authors describe text-davinci-003 as the most capable GPT-3 model at the time of writing.Source: A. Experimental Setup · p. 4
  • The comparison covers valid XDL plan counts and plan quality across 108 organic chemistry experiments.Table I compares CLAIRIFY with existing methods.Source: Table I · p. 4
  • Real-world execution uses an altered Franka Emika Panda arm with a Robotiq 2F-85 gripper and laboratory instruments including a scale and magnetic stirrer.The devices support pouring and stirring skills.Source: A. Experimental Setup · p. 4
  • Chem-RnD contains 108 detailed protocols; experiments under 1000 characters are used to test plan generation for complex chemistry, without real-world execution or environment constraints.The protocols come from Organic Syntheses, volume 77.Source: A. Experimental Setup · p. 4
  • Chem-EDU contains 40 instructions for safe, edible-chemical experiments that are in principle executable by the robot, with XDL generation constrained by available equipment.The robot's equipment includes a beaker as a mixing container.Source: A. Experimental Setup · p. 4

B. Metrics and Results

CLAIRIFY was evaluated against SynthReader for XDL plan generation and expert-rated plan quality, alongside verifier interventions and Chem-EDU robot execution. It returned plans more often in the reported datasets, was preferred more often by experts, and generated a plan that was executed on a robot through TAMP.

Plan generation

  • CLAIRIFY returned plans for 97% of 108 Chem-RnD experiments, while SynthReader returned plans for 85%; on Chem-EDU, CLAIRIFY generated plans for all instructions and SynthReader generated none.The authors attribute SynthReader’s Chem-EDU result likely to the dataset’s simple-action procedures differing from typical chemical procedures.Source: B. Metrics and Results · p. 5

Plan quality

  • Experts preferred CLAIRIFY’s plan in 75 of 108 Chem-RnD experiments, preferred SynthReader’s in 13, and judged 20 to be of similar quality.Across four of six error categories, CLAIRIFY did at least as well as or better than SynthReader on mean and median errors, and produced more experiments with zero errors in those categories.Source: B. Metrics and Results · p. 5, Figure 4 · p. 5
  • CLAIRIFY decomposed procedures into separate solutions and a transfer, and inferred implicit mixing that SynthReader treated as a pre-made mixture.The authors describe CLAIRIFY creating solutions from scratch and combining them with a Transfer procedure.Source: B. Metrics and Results · p. 5
  • CLAIRIFY produced more wrong actions than SynthReader; Chem-EDU errors also included ambiguous conditional values and missing measurement units, while experts prioritized missing actions.The authors link wrong actions to missing domain knowledge and identify verifier improvements as a future way to check unit constraints.Source: B. Metrics and Results · p. 6

Verifier interventions

  • The verifier was called an average of 2.6 times per Chem-RnD experiment and 1.15 times per Chem-EDU experiment, with hardware or reagent-list mismatches the top Chem-EDU error.The authors suggest the lower Chem-EDU average likely reflects shorter, less complicated experiments.Source: B. Metrics and Results · p. 6

Robotic validation

  • A CLAIRIFY-generated Chem-EDU XDL plan correctly captured an experiment, passed through TAMP to produce a low-level action plan, and was executed by the robot.The reported robot validation involved experiments from the Chem-EDU dataset.Source: B. Metrics and Results · pp. 6–7

C. Ablation Studies

The ablations show that an XDL description was essential for generating XDL in the tested validation set, while informative verifier feedback improved iterative correction.

Ablation setup

  • The ablations used 10 Chem-RnD validation experiments and counted plans generated without reaching the iteration loop's 10-step limit.The experiments varied the prompt's XDL description and the detail supplied in verifier feedback.Source: C. Ablation Studies · p. 7

Prompt Design

  • Without an XDL description, the LLM generated no XDL plans for any of the 10 validation experiments.The best outputs had some structure but were far from correct XDL and unrelated to chemistry.Source: C. Ablation Studies · p. 7

Feedback Design

  • Including the previous erroneous XDL and why it was wrong yielded a high number of successful plans; an error list alone had low success.A generic message that the XDL was incorrect was not informative enough to fix errors.Source: C. Ablation Studies · p. 7

V. CONCLUSION AND FUTURE WORK

CLAIRIFY generates structured task plans with language descriptions, verifier feedback, and environment constraints, and the paper reports better plans than existing baselines across two datasets. Selected plans were translated into real-world robot demonstrations; future work targets planner feedback and domain-specific verifier knowledge.

V. CONCLUSION AND FUTURE WORK

  • CLAIRIFY generated better plans than existing baselines on two datasets and translated selected plans into real-world robot demonstrations.The robot executes a motion plan generated from XDL: CLAIRIFY converts natural-language input to XDL, which the robot interprets to perform the experiment.Source: §V · p. 7, Figure 6 · p. 8
  • The approach provides an LLM with a DSL description and uses verification and iterative prompting to support syntactic correctness and environment constraints.The conclusion presents these as components of CLAIRIFY's plan-generation approach.Source: §V · p. 7
  • Future work will incorporate robot-planner and environment feedback and improve the verifier's representation of domain nuances.The authors expect these refinements to support integration in robotic task planning across different domains.Source: §V · pp. 7–8
Loading 2303.14100v1…