PaperlayerView exact revision on arXiv

Source-linked AI summary

Correcting Robot Plans with Natural Language Feedback

Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, Dieter Fox

arXiv:2204.05186v1cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Robot objectives can be difficult to specify accurately, and mismatches between human intent and robot cost can lead to plan failure. The paper learns language-conditioned costs that can update a planner’s objective, and reports that one or two language corrections recover many planner failures. Its results also describe composable constraints and generalization across objects and settings.

  • Problem

    In complex environments, it is difficult to specify a cost that accurately reflects a task, and misspecified goals or insufficient constraints can cause plan failure.

    Source: §II · p. 2

  • Method

    The method learns language-associated costs and masks that combine with existing planner costs, supporting constraint addition and goal specification.

    Source: A. Our Method · p. 3, 3) Interfacing CL with the P · p. 3

  • Results

    81% of planner failures were corrected with one language command and 93% with a second, raising effective success from 94% to 99%.

    Source: §I · p. 2

  • Takeaways & Limitations

    The approach supports composing multiple constraints and chaining corrections to specify procedures or correct trajectories needing more than one correction.

    Source: F. Discussion · pp. 7–8

Abstract

from arXiv · show

When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, underspecified, or beyond planners' ability to solve. In these cases, corrections provide a valuable tool for human-in-the-loop robot control. Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (e.g. to visit specific waypoints). Existing correction methods (e.g. using a joystick or direct manipulation of an end effector) require full teleoperation or real-time interaction. In this paper, we explore natural language as an expressive and flexible tool for robot correction. We describe how to map from natural language sentences to transformations of cost functions. We show that these transformations enable users to correct goals, update robot motions to accommodate additional user preferences, and recover from planning errors. These corrections can be leveraged to get 81% and 93% success rates on tasks where the original planner failed, with either one or two language corrections. Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments.

I. INTRODUCTION

The paper uses natural-language feedback to modify robot planning objectives, addressing tasks where specifications may not reflect human intent. It presents language-based correction as a way to update costs and reports recovery from planner failures in simulation and on a physical robot.

I. INTRODUCTION

  • Natural-language feedback can alter a robot’s goal, add constraints, or guide it when stuck, such as directing a vacuum away from bathrooms.The central question is how a robot can incorporate instructions with varied semantics into its existing objective.Source: §I · p. 1
  • Language offers an accessible way to express varied feedback, but grounding it requires reasoning about how language relates to robot states and actions.Given a grounding model, corrections can generalize to novel tasks and environments.Source: §I · p. 1
  • The proposed approach maps visual observations and a language correction to a residual cost that can modify or clarify an objective or add constraints during execution.The residual cost can also specify a task when no prior objective exists.Source: §I · p. 1
  • 81% of planner failures were corrected with one language command and 93% with a second, raising effective success from 94% to 99%.The experiments also report generalization to unseen objects and out-of-distribution language, composable cost maps, and application of simulation-trained models to real-robot tasks.Source: §I · p. 2

II. PRELIMINARIES

  • The planner minimizes a cumulative cost combining the task cost with a base cost for limits such as collision avoidance and joint limits.It repeatedly estimates an action and executes it to build a trajectory.Source: §II · p. 2
  • In complex environments without a world model, specifying a cost that reflects the task is difficult, and misspecified goals or insufficient constraints can cause plan failure.The paper frames this mismatch between human and robot costs as a value-alignment problem.Source: §II · p. 2

III. APPROACH

The approach learns a language-associated cost map conditioned on the correction and the robot’s situation, then combines it with the planner’s existing cost. It represents that correction with a continuous cost map and a mask, trained from demonstrations, specified cost maps, or both.

III. APPROACH

  • The learned language cost is combined with the planner’s existing cost to produce an updated objective for planning.The model generates a cost map conditioned on the language correction, robot state, and environment observation.Source: A. Our Method · p. 3

A. Our Method

  • The language cost is formed by multiplying a continuous cost map by a mask, separating goal-directed paths from constraint-style corrections.For a goal directive, the cost map guides the robot toward the goal; for a constraint such as going slower, the mask is a unit mask.Source: A. Our Method · p. 3
  • Training can use demonstrations paired with language, specified cost maps paired with language, or a unified dataset built from both sources.For demonstrations, ground-truth costs follow the trajectory and assign high cost outside it; specified task cost maps are used directly.Source: A. Our Method · p. 3, 1) Generating Ground-truth C and M · p. 3
  • The method learns both cost and mask models from the collected data using maximum-likelihood estimation.The models condition on the correction, robot state, and environment observation.Source: 2) Objective · p. 3

2) Objective:

The objective section describes how correction costs interact with the planner and how the learned models are trained. Corrections either add persistent constraints or specify goals, including intermediate goals that temporarily replace the original task cost.

2) Objective:

  • Demonstration data supervises predicted costs only along trajectories, while specified cost maps supervise the full map.Partial supervision lets the model extrapolate costs elsewhere and produces smoother cost maps.Source: 2) Objective · p. 3
  • Corrections encode either constraints, which remain active, or goal directives, which can temporarily deactivate the task cost until an intermediate goal is reached.The mask and whether an existing task cost is present indicate the correction mode.Source: 3) Interfacing CL with the P · p. 3
  • The architecture uses a CLIP stream for language feedback and the environment image, and a U-Net stream to encode the image for predicting instruction-associated cost.The output can be learned from specified costs or estimated from demonstrations.Source: Figure 3 · p. 4

B. Architecture

The architecture combines language and visual features with a U-Net to generate position and velocity cost maps. CLIP’s language and vision weights remain frozen during training.

B. Architecture

  • A two-stream network combines a CLIP stream with a U-Net encoder to generate position and velocity cost maps.The cost maps are represented as 2D maps.Source: B. Architecture · p. 4
  • CLIP encodes the language correction, robot state, and RGB image, while the U-Net encodes the image and robot state.The robot’s end-effector location is encoded on a spatial map of the RGB image.Source: B. Architecture · p. 4
  • Training freezes CLIP’s language transformer and ViT weights and optimizes only the U-Net weights.Source: B. Architecture · p. 4

IV. EXPERIMENTAL PROTOCOL

The protocol builds cost-map training data from demonstrations or specified costs, then evaluates planning in simulation and on a physical robot. It also tests planner failures and generalization across varied objects, environments, and correction types.

IV. EXPERIMENTAL PROTOCOL

  • Demonstration data covers 100 environments with two objects drawn from four YCB objects, sampled positions and orientations, and templated instructions.Each environment has 10 sampled collision-free start positions; goal positions are offset 20 pixels from object-edge midpoints.Source: §IV · p. 4
  • STORM generates trajectories by minimizing Euclidean distance, but its local minima lead to a 6.4% planner failure rate; object configurations are split across train, validation, and test sets.Source: §IV · p. 4
  • The framework can add output-map frames for additional quantities such as force and torque.Source: §IV · p. 4
  • The dataset varies object types and positions, templated referring expressions, and correction vocabulary, with tests in unseen simulated environments and a real 7DoF arm setup.The correction vocabulary includes referential expressions, relative clauses, object avoidance, and velocity-related costs.Source: Figure 4 · p. 5
  • Velocity cost maps assign 0, 1, or 2 for speed-up, slow-down, or no velocity constraint, while object-avoidance costs use negative squared distance from the object center.Cost values are rescaled to [0, 255].Source: §IV · p. 5
  • The evaluation uses a planner on a simulation test set and on a real Franka Panda robot in cluttered environments with unseen objects.A trial succeeds when the robot reaches within 20 pixels of its goal.Source: §IV · p. 5

V. RESULTS

The results assess language corrections for planner failures and test specifying goals directly through language. One or two corrections improve hard-set success, while goal-as-language performance is also measured without giving the planner the desired coordinates.

A. Goal Reaching with Language

  • 81% and 93%: one and two language corrections raise success on the hard set from 0% to 81% and 93%, respectively; test-set success reaches 98% and 99%.The hard set contains cases where the planner had 0% success, while the test dataset had 100% planner success.Source: A. Goal Reaching with Language · p. 5, Table I · p. 6
  • 65%: specifying the goal solely through language yields this success rate on the full set without providing the planner the desired (x,y) position.Source: A. Goal Reaching with Language · p. 5
  • 29%: language-only goal specification succeeds on the hard set without requiring access to (x,y).Source: A. Goal Reaching with Language · p. 5

B. Ablation Experiments

The ablations test language, vision, and trajectory masking, while later evaluations report 69% success across instruction types and assess generalization to unseen language and objects.

B. Ablation Experiments

  • Without language input, success was 4%; without vision input, success was 33%, limited to basic commands and unable to handle object-referencing tasks.Removing the U-Net encoder and using CLIP input alone did no better than removing vision.Source: B. Ablation Experiments · p. 6
  • Adding the trajectory mask raised test-set success from 58% with costs alone to 69% with the mask.The evaluation specified goals in language without a Euclidean distance cost.Source: B. Ablation Experiments · p. 6, Table II · p. 6

C. Performance by Instruction Type

  • Training used five spatial object-dependent tasks, two spatial robot-object-dependent tasks, four directional spatial robot-dependent tasks, and two velocity tasks.The evaluation examined solvable examples grouped by ground-truth trajectory length.Source: C. Performance by Instruction Type · p. 6
  • 69% overall success was achieved across instruction types, and very local corrections such as “go faster” were always successful.Evaluation used language instructions without providing the planner the goal’s (x, y) position.Source: C. Performance by Instruction Type · p. 6

D. Generalization Experiments

  • In 20 different setups, the method successfully completed the task in 17; examples used diverse natural-language sentences to control the robot.For instructions referring to 10 unseen objects, it worked on red and orange cups, a plate, a fork, and a ketchup bottle, and failed on a screwdriver, candle, book, banana, pepper can, and pen. It always succeeded when unseen objects were background clutter and instructions referred to seen objects, even at orientations not seen during training.Source: D. Generalization Experiments · p. 7
  • The generalization evaluation included clutter and varied orientations, beyond training scenes with two of four objects in four fixed orientations.The real-robot evaluation assessed generalization with unseen objects in clutter.Source: D. Generalization Experiments · pp. 6–7, Table IV · p. 7

E. Failures

The discussion highlights composable and sequential language corrections, reuse across environments, and preference-based plan changes. It also describes failure cases involving trajectory masks, cost maps, and ambiguous object references.

E. Failures

  • Correction failures include discontinuous trajectory masks, stray cost-map pixels, paths to image edges when no path is observable, and ambiguity between same-type objects.In scenes with two instances of one object, the model may generate paths to both.Source: E. Failures · p. 7, Figure 10 · p. 8

F. Discussion

  • Multiple constraints can be combined at execution time, either together or at different times, including velocity, goal-reaching, and stay-away costs.The discussion describes combining these costs while the robot pursues a goal.Source: F. Discussion · p. 7
  • Despite training on templated instructions, the model generalizes to a wider variety of instruction types.The discussion also shows visual generalization to unseen objects, different object counts, and new object poses.Source: Figure 6 · p. 7, Figure 9 · p. 8
  • Chaining corrections can specify procedures and correct trajectories that require more than one correction.Commands can guide the robot to different locations around objects sequentially.Source: F. Discussion · pp. 7–8, Figure 7 · p. 8
  • The same correction can be applied across environments with different goals without additional user effort.The discussion contrasts this with the time and effort required for joystick or kinesthetic feedback.Source: F. Discussion · p. 8, Figure 11 · p. 9
  • Users can also modify plans according to preferences, rather than correcting only planner failures.One example steers the robot away from fragile objects while it places an object in a mug.Source: F. Discussion · pp. 8–9, Figure 8 · p. 8

VI. RELATED WORK

The section introduces several threads of related work.

  • The paper situates its work within multiple threads of related work.Source: §VI · p. 9

1) Natural Language for Robot Behavior Correction:

The related work distinguishes language correction from prior correction and task-specification approaches, while connecting language-conditioned cost prediction to cost inference and value alignment. The conclusion describes the framework and its real-world evaluation settings.

1) Natural Language for Robot Behavior Correction:

  • Prior language-correction approaches use hand-engineered optimization constraints, correction-trained policies, or language to disambiguate joystick input, with limitations in constraint design, sample efficiency, or user attention.The paper contrasts these with directly learning to predict cost maps and notes that a nearest-neighbor grounding model does not generalize to new environments and tasks.Source: 1) Natural Language for Robot Behavior… · p. 9

2) Language for Robot Task Specification:

  • Unlike task-specification approaches that fix instructions during execution, cost prediction supports refining robot behavior over time while retaining instruction following as a special use-case.Source: 2) Language for Robot Task Specification · p. 9

3) Inferring costs from demonstrations:

  • The correction model conditions cost prediction on language and visual observations, allowing reuse of the model with diverse language corrections across tasks.The passage relates this to inverse reinforcement learning from demonstrations.Source: 3) Inferring costs from demonstrations · p. 9

4) Value alignment problem:

  • The framework addresses value alignment by interactively updating a robot’s cost when it does not reflect the user’s intended task.Source: 4) Value alignment problem · p. 9

VII. CONCLUSION

  • The conclusion describes simulation-trained language-cost modeling evaluated in real-world settings with non-templated commands, clutter, and new object poses and types.Source: §VII · p. 9

APPENDIX

The appendix describes planner settings, hard environments where the planner gets stuck in local minima, and how correction performance varies with trajectory length.

A. Planner

  • The planner uses 500 particles and a 30-timestep horizon; on the physical robot it checks environment collisions with the table but relies on the language cost for object collision avoidance.The appendix also notes a z-axis and gripper-orientation constraint during execution.Source: A. Planner · p. 12

B. Hard environments

  • In hard environments, the planner can get stuck in local minima and must first move along a trajectory of increasing cost before it can resume moving toward the goal.The language feedback points the robot toward these inflection points.Source: B. Hard environments · p. 12, Figure 12 · p. 12

C. Convergence to Goal : Analysis

  • Corrections perform worse on longer trajectories than shorter ones, while simple modifications can still be applied at once or sequentially.The analysis groups trajectories as easy, medium, or hard by length; generating a mask over cost is most advantageous for medium-to-long trajectories.Source: C. Convergence to Goal : Analysis · p. 12, Figure 13 · p. 12
Loading 2204.05186v1…