PaperlayerView exact revision on arXiv

Source-linked AI summary

DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment

Yanjiang Guo, Yen-Jen Wang, Lihan Zha, Jianyu Chen

arXiv:2307.00329v4cs.ROcs.AI

TL;DR

Physical execution can diverge from a language model’s plan, while delayed feedback may be inefficient. DoReMi uses LLM-generated constraints and VLM monitoring to detect violations and re-plan; experiments report higher success rates and shorter execution times across robot-arm and humanoid tasks.

Abstract

from arXiv · show

Large language models (LLMs) encode a vast amount of semantic knowledge and possess remarkable understanding and reasoning capabilities. Previous work has explored how to ground LLMs in robotic tasks to generate feasible and executable textual plans. However, low-level execution in the physical world may deviate from the high-level textual plan due to environmental perturbations or imperfect controller design. In this paper, we propose \textbf{DoReMi}, a novel language model grounding framework that enables immediate Detection and Recovery from Misalignments between plan and execution. Specifically, we leverage LLMs to play a dual role, aiding not only in high-level planning but also generating constraints that can indicate misalignment during execution. Then vision language models (VLMs) are utilized to detect constraint violations continuously. Our pipeline can monitor the low-level execution and enable timely recovery if certain plan-execution misalignment occurs. Experiments on various complex tasks including robot arms and humanoid robots demonstrate that our method can lead to higher task success rates and shorter task completion times. Videos of DoReMi are available at \url{https://sites.google.com/view/doremi-paper}.

I. INTRODUCTION

Hierarchical language-model planning can use low-level controllers, but execution may diverge from the plan and delayed feedback can waste time. DoReMi uses LLM-generated constraints and VLM monitoring to detect misalignment and re-plan, with reported gains across robot-arm and humanoid tasks.

I. INTRODUCTION

  • Hierarchical planning pairs language-model task plans with low-level controllers, which can handle complex robot dynamics at high frequency.The introduction contrasts this with end-to-end action-output models that require substantial robot-action data and may not run at high frequencies.Source: §I · p. 1
  • Low-level execution can diverge from a high-level plan because of environmental perturbations or imperfect controllers, and misalignment may arise during a task.Source: §I · pp. 1–2
  • DoReMi combines LLM-generated plans and execution constraints with continuous VLM monitoring, triggering immediate re-planning when constraints are violated.The LLM generates constraints from its understanding of the physical world; the VLM monitors them during low-level skills.Source: §I · p. 2
  • Robot-arm and humanoid experiments report higher task success rates and shorter task execution times with DoReMi.Source: §I · p. 2

II. RELATED WORKS

Prior language-grounding work includes end-to-end action prediction and hierarchical language-model planning, while feedback often arrives only after a plan step finishes. DoReMi is positioned as providing precise, automated, high-frequency feedback during skill execution.

II. RELATED WORKS

  • End-to-end transformer policies predict actions but depend heavily on expert demonstrations for training.Source: §II · p. 2
  • LLM planners commonly assume skills succeed, while feedback-based approaches generally respond after each step; Inner Monologue also relies on manually designed queries.The text contrasts DoReMi with these approaches, describing its feedback as precise, automatic, and high-frequency.Source: §II · p. 2
  • VLMs have been used for scene description, success detection, and object detection; DoReMi uses one to check whether specified constraints are satisfied.Source: §II · p. 2

III. METHOD

DoReMi has a planning stage and a low-level execution stage: the LLM selects a skill and generates its constraints, then a VLM checks those constraints during execution and can prompt re-planning.

III. METHOD

  • At planning time, the LLM chooses the next skill and generates constraints for it from the task instruction, available skills, prompts, and history.Source: §III · p. 2
  • During low-level execution, the VLM periodically checks the constraints and a violation triggers immediate LLM re-planning for recovery.Source: §III · p. 2, Figure 2 · p. 3
  • LLM-generated specific constraints let the VLM answer targeted questions, avoiding the ambiguity of open-ended scene descriptions.The figure describes this constraint-focused feedback as more precise.Source: Figure 3 · p. 3

A. Language Model for Planning

The planner uses few-shot in-context learning to select skills and is also invoked to re-plan when constraint violations indicate misalignment.

A. Language Model for Planning

B. Language Model for Constraint generation

DoReMi uses the LLM’s understanding of the physical world and task history to generate constraints for planned skills. A VLM checks these specific conditions to provide focused feedback.

B. Language Model for Constraint generation

  • The LLM generates constraints for each planned skill from task history, such as requiring the robot to hold a box while moving.A violated constraint can indicate a failed pick or a dropped box.Source: B. Language Model for Constraint… · p. 3
  • Constraints such as “red block on green block” encode conditions that should remain true after a skill.The LLM automatically generates constraints using its understanding of the physical world.Source: B. Language Model for Constraint… · p. 3
  • Checking specific constraints with binary yes-or-no answers focuses the VLM and avoids ambiguity from open-ended scene descriptions.The paper points to Figure 3 for examples of ambiguity in open-ended descriptions.Source: B. Language Model for Constraint… · p. 3
  • After selecting the next skill, the LLM continues generation with “Constraint:” to produce specific constraints.Source: B. Language Model for Constraint… · p. 3

C. VLM as Constraint Detector

During skill execution, a VLM periodically checks LLM-generated constraints. A violation interrupts the current skill and prompts replanning; the framework also supports other detector modalities.

C. VLM as Constraint Detector

  • The VLM checks each LLM-generated constraint against visual input, answering a yes-or-no question in less than 0.1 second.The image comes from a first- or third-person camera, and the question is adapted from the proposed constraint.Source: C. VLM as Constraint Detector · p. 3
  • The algorithm queries the detector every Δt during skill execution and adds a violation to the prompt before breaking to trigger replanning.The LLM first selects a skill and generates its constraints; execution continues while those constraints remain satisfied.Source: C. VLM as Constraint Detector · p. 4
  • Constraint detectors can run in other modalities and at frequencies different from low-level controllers.Source: C. VLM as Constraint Detector · p. 4
  • In practice, BLIP-2 checks constraints every 0.2 seconds; violations abort the current skill and trigger replanning.Zero-shot detection works in most tasks but not extremely complex scenes; LoRA fine-tuning is used for those tasks, and the fine-tuned detector generalizes to unseen objects, backgrounds, and tasks.Source: C. VLM as Constraint Detector · p. 4

IV. EXPERIMENTS

The experiments test DoReMi on robot-arm manipulation and humanoid tasks under environmental disturbances and imperfect controllers. They assess immediate misalignment detection and recovery, task success, and execution time.

IV. EXPERIMENTS

  • The experiments cover both robot-arm manipulation and humanoid robot tasks.The tasks are shown in Figure 4.Source: §IV · p. 4
  • The test conditions include environmental disturbances and imperfect controllers, such as random drops, placement noise, failed picks, and unexpected obstacles.Source: §IV · p. 4
  • The experiments ask whether DoReMi detects and recovers from plan-execution misalignment immediately and improves success rates and execution times under disturbances or imperfect control.Source: §IV · p. 4

A. Robot Arm Manipulation Tasks

In robot-arm tasks, DoReMi is evaluated on pick-and-place and ordered block stacking with drop and placement disturbances. It achieves higher success rates and shorter execution times than the baselines, which can fail or recover more slowly.

A. Robot Arm Manipulation Tasks

  • The robot-arm evaluation uses pick-and-place primitives in a Ravens-adapted tabletop environment and tests pick-and-place and ordered stacking tasks.Disturbances include a per-second drop probability and uniform [0, n] cm placement noise; tasks exceeding 20 seconds count as failures.Source: A. Robot Arm Manipulation Tasks · p. 4
  • The baselines are SayCan, CLIPort, Inner Monologue, and an oracle-feedback version of Inner Monologue.CLIPort repeats each step until success, while Inner Monologue uses scene descriptions and success detection to re-plan after a step.Source: A. Robot Arm Manipulation Tasks · pp. 4–5
  • SayCan fails under disturbances without success detection or replanning, while CLIPort’s step repetition can fail when block collapse requires a new plan.Inner Monologue can also struggle when VLM scene descriptions are ambiguous; Figure 5 illustrates the block-collapse comparison.Source: A. Robot Arm Manipulation Tasks · p. 5, Figure 5 · p. 5
  • DoReMi achieves higher success rates and shorter execution times under disturbances than the baselines.Table I reports success rates and execution times by disturbance degree; execution time is measured only under high success rates.Source: A. Robot Arm Manipulation Tasks · p. 5, Table I · p. 5

B. Humanoid Robot Tasks

DoReMi is evaluated on humanoid tasks with disturbances, including simulated manipulation and real-humanoid food preparation and block stacking. The reported results highlight faster recovery, similar success to an oracle-feedback baseline, and transfer by a fine-tuned detector to unseen scenes and tasks.

B. Humanoid Robot Tasks

  • The humanoid evaluation covers obstacle avoidance, box transport, food preparation, real-humanoid food preparation, and real block stacking.The real-world robot's language and vision models run in the cloud and communicate with the robot at 1 Hz.Source: B. Humanoid Robot Tasks · p. 6
  • The real humanoid stack task includes blocks that may collapse under external forces or placement noise, and DoReMi-FT recovered after a block was knocked down.The robot has 20 degrees of freedom and uses locomotion and manipulation controllers for its low-level skills.Source: B. Humanoid Robot Tasks · pp. 5–6, Figure 7 · p. 6
  • DoReMi-FT surpasses IM-oracle in execution time while maintaining similar success rates on humanoid tasks.The comparison is reported under disturbances; Table II evaluates execution time only when task success rates are high.Source: B. Humanoid Robot Tasks · p. 6, Table II · p. 6
  • DoReMi-FT generalized to unseen objects, backgrounds, and tasks after fine-tuning BLIP-2 on 128 image-text pairs with five fruit demonstrations.The test scenes included junk food, vegetables, and seafood at random positions.Source: Figure 6 · p. 6, B. Humanoid Robot Tasks · p. 7

V. CONCLUSION

The conclusion emphasizes continuous alignment of plans with execution through LLM-generated plans and constraints, with VLM detectors supporting immediate detection and recovery. In the fine-tuning subsection, VLM accuracy on the prepare-food task significantly increases after fine-tuning.

V. CONCLUSION

  • DoReMi uses LLM-generated plans and constraints to support immediate detection and recovery under practical VLM constraint detectors.The conclusion describes this as grounding language while continuously aligning plans with execution in disturbed environments.Source: §V · p. 7

A. Effectiveness of VLM finetuning

B. VLM ablation study

The VLM ablation reports that BLIP-2 performs similarly to Instruct-BLIP, while zero-shot transferred models perform poorly on complicated prepare-food tasks. The detailed pipeline example shows constraint feedback prompting replanning after a stacking error.

B. VLM ablation study

  • Zero-shot transferred VLMs do not perform well on complicated prepare-food tasks, while BLIP-2 performs similarly to Instruct-BLIP.The comparison is the VLM ablation reported in Table IV.Source: B. VLM ablation study · p. 7, Table IV · p. 7

C. Detalied pipeline

  • For stack-in-order, the LLM pairs each pick or placement step with constraints about the held block and existing stack.The example carries forward the red-on-brown condition while planning the green-block placement.Source: C. Detalied pipeline · p. 7
  • When feedback reports that the red block is not on the brown block, the example replans by restoring that placement before continuing the green-block stack.The later steps repeat the green-block placement after the red-on-brown constraint is restored.Source: C. Detalied pipeline · p. 7

Related papers

Loading 2307.00329v4…