Source-linked AI summary
Translating Natural Language to Planning Goals with Large-Language Models
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, Harold Soh
TL;DR
The paper asks whether LLMs can translate natural-language goals into PDDL so classical planners can handle planning. It evaluates GPT-3.5 variants across Blocksworld and ALFRED, finding strong translation ability but weaknesses in formal, numerical, and physical reasoning.
Problem
LLMs show broad language ability but are poor standalone planners, leaving their ability to translate natural-language goals into structured PDDL unclear.
Method
The study empirically evaluates GPT-3.5 variants translating English instructions into PDDL goals across Blocksworld and ALFRED.
Results
GPT-3.5 often translates underspecified goals using commonsense knowledge, while performance is mixed on partially specified tasks and weaker for numerical, spatial, and prompt-sensitive tasks.
Takeaways & Limitations
LLMs are more effective as translators than planners, supporting a modular interface in which classical planners provide structured reasoning.
Takeaways & Limitations
LLMs lack fundamental formal and physical-world reasoning capabilities needed for reliable general goal translation.
Abstract
from arXiv · showhide
Recent large language models (LLMs) have demonstrated remarkable performance on a variety of natural language processing (NLP) tasks, leading to intense excitement about their applicability across various domains. Unfortunately, recent work has also shown that LLMs are unable to perform accurate reasoning nor solve planning problems, which may limit their usefulness for robotics-related tasks. In this work, our central question is whether LLMs are able to translate goals specified in natural language to a structured planning language. If so, LLM can act as a natural interface between the planner and human users; the translated goal can be handed to domain-independent AI planners that are very effective at planning. Our empirical results on GPT 3.5 variants show that LLMs are much better suited towards translation rather than planning. We find that LLMs are able to leverage commonsense knowledge and reasoning to furnish missing details from under-specified goals (as is often the case in natural language). However, our experiments also reveal that LLMs can fail to generate goals in tasks that involve numerical or physical (e.g., spatial) reasoning, and that LLMs are sensitive to the prompts used. As such, these models are promising for translation to structured planning languages, but care should be taken in their use.
I. INTRODUCTION
The paper studies whether LLMs can translate natural-language goals into PDDL, pairing linguistic competence with classical planners’ structured reasoning. GPT-3.5 can often translate underspecified goals, but struggles with formal or physical reasoning and requires careful use.
- Motivation and approach: LLMs are poor standalone planners, motivating a modular system in which they translate goals while classical planners perform structured reasoning.The LLM supplies linguistic competence, and the planner receives the resulting PDDL goal.
- Translation challenge: The translation task must use contextual knowledge while producing syntactically valid goals compatible with the PDDL domain and problem.Small syntax errors can prevent parsing and planning.
- Empirical findings: GPT-3.5 effectively translated goals across Blocksworld and ALFRED, including ambiguous commands requiring commonsense knowledge.For “Set a table for two,” it inferred appropriate tableware such as two cups, plates, and utensils.
- Empirical findings: LLMs performed poorly on simple counting and spatial inference, with failures involving predicate semantics and hierarchical object relationships.These failures complicate using GPT-3.5 as a general natural-language interface to planners.
- Planning representation: PDDL separates domain definitions from problem instances, whose goals are logical predicate expressions that planners must satisfy.Planners such as Fast Downward take a PDDL domain and problem and output a plan achieving the goal.
C. Machine Translation from Natural Language to Structured Languages
The paper frames goal translation as machine translation into a structured language, where incomplete natural-language instructions require inference and valid PDDL output. It evaluates GPT-3.5 n-shot translation and decomposes errors into domain understanding, goal inference, and goal specification.
- Structured-language translation: Goal translation converts varied or incomplete natural-language instructions into syntactically correct goals consistent with a specified planning domain.Commonsense knowledge may be needed to fill missing details, such as the composition of a tomato sandwich.
- Study design: The study investigates GPT-3.5 variants using n-shot prompts containing a PDDL domain, a natural-language goal, and n correct translation examples.The query includes domain information, the desired goal state, and example translations.
- Evaluation: Success rates provide the primary evaluation metric, but they offer only a limited view of what underlies model performance.The authors note that precise causal explanations are difficult for large, closed-source transformer models.
- Study design: The authors design tasks ranging from fully specified goals to partially specified instructions requiring inference or commonsense reasoning.This variation is intended to examine the circumstances under which LLMs translate effectively.
- Translation subtasks: Translation is decomposed into domain understanding, goal inference, and PDDL goal specification.The corresponding factorization separates domain information, a latent inferred goal, and its PDDL transcription.
- Translation subtasks: The analytical approach changes prompt components or asks for intermediate outputs to distinguish failures in domain understanding, inference, and PDDL generation.Examples include querying domain elements directly or requesting the inferred goal in another language.
IV. EXPERIMENTAL SETUP
The experimental setup evaluates natural-language goal translation across Blocksworld and ALFRED-L, using tasks designed to probe distinct linguistic, grounding, reasoning, and world-knowledge skills.
- Domains: The study uses Blocksworld and ALFRED-L to evaluate translation from natural-language instructions to planning goals.Blocksworld contains colored blocks arranged in stacks, while ALFRED-L represents household environments with objects, receptacles, and agents.
- Domains: ALFRED-L scenes are randomly initialized across kitchen, living-room, and bedroom settings, with one agent included per scene.The domain uses object, receptacle, and agent classes, and combines selected predicates to fit the LLM query window.
- Domains: Blocksworld tasks vary stack configurations and block counts, whereas ALFRED-L tasks involve household interactions such as moving and slicing objects.The desired configurations can be specified with different levels of specificity or constraints.
- Task Design: The task suite tests linguistic competence, object association, numerical reasoning, physical reasoning, and world knowledge.Object association includes grounding ambiguous natural-language references to concrete PDDL symbols.
C. LLM Models
The experiments use GPT-3.5 variants and evaluate translation with strict and loose success criteria, supplemented by checks for syntax, physical validity, domain understanding, and goal inference.
- LLM Models: GPT-3.5 code-davinci-002 is used for Blocksworld, while text-davinci-003 is used for ALFRED-L.These models were selected because they achieved the best performance in initial translation experiments.
- Translation Evaluation: Translation success is measured with strict and loose criteria in both domains.Strict success requires a planner-acceptable goal that does not needlessly change predicates.
- Error Checks: Blocksworld evaluation includes domain/syntax checks for PDDL compliance and physical checks for constructible stacks.Physical errors include placing multiple blocks on one block or placing a block simultaneously above and below another.
- Subtask Analysis: Subtask analysis separately tests domain understanding and goal inference.Blocksworld domain-understanding queries cover object extraction, color-based extraction, and predicate questions.
- Subtask Analysis: ALFRED-L subtask queries test identifying a box with two target types and selecting where a moved object should go.These queries are tailored to each task.
V. RESULTS AND DISCUSSION
GPT-3.5 translated explicit natural-language goals effectively, but performance became mixed when instructions required inference, numerical or spatial reasoning, or sensitivity to prompt wording. The results support translation as more promising than planning while exposing systematic biases and reasoning limitations.
- Overall findings: Translation success was excellent for unambiguous goals, while partially specified goals produced mixed performance; planning success was 0% for most tasks.The study reports performance above chance across translation tasks, contrasting with failed preliminary planning tests.
- Explicit goals: Goal correctness was close to 100% for ExplicitStacks and ExplicitInstruct, including varied Blocksworld configurations and multiple ALFRED-L scenes.ExplicitStacks performance was sustained across different numbers of blocks and desired stacks, while ExplicitInstruct achieved perfect translation across multiple objects.
- Prompt sensitivity: ExplicitStacks-II reached only ≈50% when equivalent stack descriptions were ordered top-to-bottom rather than bottom-to-top, revealing an ordering bias.The task remained unambiguous, but changing sequence direction substantially affected performance.
- Partial specification: The LLM filled gaps using commonsense knowledge, identifying object types, synonyms, and some spatial relations in partially specified goals.Ambiguous SetTable2 and CleanKitchen goals were also translated to a good degree.
- Reasoning limitations: Nontrivial numerical and physical reasoning remained difficult: ALFRED-L counting and nesting errors increased with task difficulty, while Blocksworld NBlocks and KStacks scored ≈50−60%.The model often copied the one-shot example, and larger examples generally improved target-task performance.
B. Subtask Analysis
The subtask analysis separates domain understanding, goal inference, and PDDL generation to identify why translation succeeds or fails. Failures are associated with weaker domain understanding or goal inference, while linguistic associations can mask limited physical-world reasoning.
- Subtask Analysis: Subtask performance generally fell on domain understanding and goal inference when overall translation failed.The analysis compares successful and failed translations to distinguish domain parsing, goal inference, and PDDL-generation errors.
- Subtask Analysis: ≈40-60% success on Blocksworld predicate queries indicates limited understanding of the physical semantics encoded by predicates.Object extraction was stronger, especially than color-based extraction, while predicate-related queries remained difficult.
- Subtask Analysis: ALFRED-L failures in MoveToCount2, MoveToMore, and MoveToNested often preserved domain understanding but reduced goal inference.The model could parse relevant PDDL information in some cases but failed to integrate it into a suitable goal.
- Subtask Analysis: Numerical and hierarchical reasoning errors increased as counting and nesting tasks became more difficult.Performance declined from two to three objects and from one-level to two-level nesting.
- Subtask Analysis: Linguistic competence and learned associations sometimes masked weak formal and physical-world reasoning, especially on common ALFRED-L tasks.The paper reports impressive performance on common household tasks while noting limited fundamental reasoning capabilities.
APPENDIX
The appendix describes benchmark construction, instruction generation, evaluation, and error checking for Blocksworld translation tasks. It uses varied natural-language formulations over generated PDDL problems and applies parser, structural, and constraint checks.
- Benchmark construction: Blocksworld problems are generated with 4, 8, or 12 objects, with 100 problems corresponding to each option.These generated problems serve as templates for concrete task-specific test cases.
- Instruction generation: Instructions are sampled after template objects are replaced with actual blocks, while the original template goal is removed as ground truth.The original goal can still contribute to instruction sampling for tasks such as ExplicitStacks.
- Instruction generation: ExplicitStacks-II varies whether stack blocks are described from bottom to top or top to bottom.A separate benchmark is generated for each direction, doubling its test cases relative to most other tasks.
- Instruction generation: BlockAmbiguity omits ontable and clear predicates from translated instructions and allows repeated colors.Other tasks vary stack counts, prime-sized stacks, or color constraints through their sampling procedures.
- Benchmark construction: Each Blocksworld task uses 100 test cases per object-number option, with benchmarks built from either all three or two object-count options.Blocksworld’s short domain is included in every example, unlike ALFRED-L, whose domain definition would exceed the token limit if repeated.
- Evaluation: The evaluation pipeline parses the generated goal, constructs its graph, checks stack structure, and validates task constraints.The parser detects syntax or domain errors, while strict and loose metrics differ in their requirements for ontable and clear predicates.
C. ALFRED-L Evaluation
ALFRED-L evaluation defines loose and strict success criteria for household goal-translation tasks. Criteria check the intended object placement or household state, with strict versions additionally excluding agents, redundant predicates, or unintended changes.
- MoveSynonym: MoveSynonym evaluates whether a synonym denotes the correct receptacle receiving the moved object or the correct position for a movable object.The strict criterion additionally requires that the predicted goal contain no agent.
- MoveNextTo: MoveNextTo requires the two objects to share a location under the loose criterion.Strict evaluation also requires that the target object remain unmoved and that no agent appear in the goal state.
- Placement tasks: Loose ALFRED-L evaluation generally checks whether the predicted goal places the relevant object in the correct box.This criterion is used for MoveTo, MoveToCount, MoveToMore, and MoveNested variants.
- Placement tasks: Strict placement evaluation additionally requires no agent in the goal state.MoveNested2 also requires no redundant predicates.
- Household-state tasks: CutFruits, PrepareMeal, and IceCream use task-specific state checks for fruits on a plate, food on a table, and ice cream in a fridge.Strict criteria restrict the changed entities or require no changes to other predicates.
- Household-state tasks: SetTable2 and CleanKitchen evaluate table setting or kitchen cleanliness using distinct loose and strict state requirements.CleanKitchen’s strict criterion requires nothing in the sink, microwave, or oven.
D. BlocksWorld Error Analysis
The BlocksWorld error analysis probes domain understanding with targeted predicate and object queries, then separately tests goal inference using hierarchical Python lists. It samples a subset of test cases to reduce analysis cost.
- Domain Understanding: Domain-understanding analysis replaces the goal instruction with five queries while retaining the original domain, examples, and answers.The queries test object extraction, color-based extraction, and predicate understanding.
- Domain Understanding: Object extraction asks the model to output all objects listed in the PDDL problem, separated by spaces.Color-based extraction samples one block color and asks for the corresponding objects.
- Domain Understanding: On-predicate queries sample an initial-state fact and sometimes replace one argument with another random block.The answer depends on whether the resulting fact remains true.
- Domain Understanding: OnTable and Clear predicate queries sample blocks according to whether the relevant predicate holds in the initial state.For both predicates, half of the queries can instead sample a block that does not satisfy the predicate.
- Analysis design: Only 7 of every 100 test cases are sampled for the five-query analysis, except when successes or failures are rare.This sampling reduces the cost of running all five queries over every test case.
- Goal Inference: Goal-inference analysis replaces PDDL formulas with hierarchical Python lists while preserving the domain, example problems, and test problems.Lists of lists of PDDL object symbols represent stacks in a satisfying configuration, with instructions rephrased accordingly.
E. ALFRED-L Error Analysis
The ALFRED-L error analysis tests domain understanding and goal inference through targeted questions about objects, locations, counts, synonyms, and household activities.
- Goal Inference: Goal-inference tests ask the model to identify destinations or produce natural-language goal specifications for movement, placement, meal preparation, ice cream, table setting, and kitchen cleaning.Several tests explicitly constrain which object should move and where it should be placed.
- Domain Understanding: The analysis probes domain understanding with questions about object locations, box contents, sofas, fruits, and task-relevant household objects.These questions cover counting, nesting, object identification, and semantic interpretation.
F. Zero-shot performance for ALFRED-L partially specified tasks
For partially specified ALFRED-L tasks, zero-shot translation performance was lower than one-shot performance. The comparison indicates that providing an example improves results in this setting.
- Zero-shot Results: Zero-shot performance was lower than one-shot performance on partially specified ALFRED-L tasks.The comparison is reported for the goal-translation setting.
- Task Setting: The evaluated ALFRED-L tasks involved partially specified goals.The section reports zero-shot results specifically for this task category.
- Evaluation: The results include an error analysis alongside the zero-shot performance evaluation.The passage explicitly identifies an error analysis in Table VI.
G. Fine-grained Performance Report on Blocksworld Tasks
Blocksworld performance varies with task structure, predicate ordering, and problem size. The model generally handles PDDL syntax and object extraction better than physical predicate reasoning, while additional examples do not reliably ensure generalization.
- Error Types: The model makes few PDDL domain-related errors but more physical errors, especially when tasks specify multiple stacks.This pattern aligns with strong object extraction and weaker predicate understanding.
- ExplicitStacks-II: Performance improves substantially when the demonstration’s predicate order matches the top-to-bottom order used in ExplicitStacks-II goals.The same PDDL problem was evaluated with reversed predicate ordering in the demonstration.
- KStacks: KStacks performance decreases as the number of objects increases because more objects create more diverse instructions and stack-length choices.The model copied the goal pattern in 90-95% of 8-object cases but only 50-60% of 12-object cases.
- KStacks: Adding a second example with a different requested stack count produced no significant improvement in KStacks translation.This remained true even when all possible answer patterns were provided for 8-object cases.