PaperlayerView exact revision on arXiv

Source-linked AI summary

RE-MOVE: An Adaptive Policy Design for Robotic Navigation Tasks in Dynamic Environments via Language-Based Feedback

Souradip Chakraborty, Kasun Weerakoon, Prithvi Poddar, Mohamed Elnoor, Priya Narayanan, Carl Busart, Pratap Tokekar, Amrit Singh Bedi, Dinesh Manocha

arXiv:2303.07622v2cs.ROcs.AIcs.LG

TL;DR

Trained navigation policies can fail to adapt to environmental changes during deployment, motivating RE-MOVE’s test-time language-feedback approach. It uses epistemic uncertainty to time requests and zero-shot NLP to interpret feedback, with reported gains in goal-reaching and trajectory length in challenging real-world environments.

  • Problem

    Trained robotic navigation policies may fail to adapt to unexpected environmental changes during deployment, risking catastrophic failures.

    Source: §I · p. 1

  • Method

    RE-MOVE adapts trained navigation policies without retraining by using epistemic uncertainty to time requests for human language feedback and zero-shot NLP to interpret it.

    Source: §I · p. 2

  • Results

    Up to 80% more successful goal-reaching and a 13.50% decrease in normalized trajectory length were reported versus other methods in perceptually challenging real-world environments.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors report RE-MOVE’s efficacy across a variety of synthetic and real-world navigation tasks.

    Source: §VI · p. 6

    The authors leave a dedicated navigation language model and speech- or image-based feedback for future work.

    Source: §VI · p. 6

Abstract

from arXiv · show

Reinforcement learning-based policies for continuous control robotic navigation tasks often fail to adapt to changes in the environment during real-time deployment, which may result in catastrophic failures. To address this limitation, we propose a novel approach called RE-MOVE (REquest help and MOVE on) to adapt already trained policy to real-time changes in the environment without re-training via utilizing a language-based feedback. The proposed approach essentially boils down to addressing two main challenges of (1) when to ask for feedback and, if received, (2) how to incorporate feedback into trained policies. RE-MOVE incorporates an epistemic uncertainty-based framework to determine the optimal time to request instructions-based feedback. For the second challenge, we employ a zero-shot learning natural language processing (NLP) paradigm with efficient, prompt design and leverage state-of-the-art GPT-3.5, Llama-2 language models. To show the efficacy of the proposed approach, we performed extensive synthetic and real-world evaluations in several test-time dynamic navigation scenarios. Utilizing RE-MOVE result in up to 80% enhancement in the attainment of successful goals, coupled with a reduction of 13.50% in the normalized trajectory length, as compared to alternative approaches, particularly in demanding real-world environments with perceptual challenges.

I. INTRODUCTION

RE-MOVE targets the difficulty of adapting trained navigation policies to unexpected test-time changes, addressing when to request help and how to use language feedback. It combines uncertainty-based feedback timing with a zero-shot language interpretation and adaptation module.

I. INTRODUCTION

  • The introduction identifies test-time uncertainty estimation and its relationship to input-space design as an under-studied navigation issue.It contrasts this with prior uncertainty-driven navigation work that is mainly limited to training time.Source: §I · p. 1
  • RE-MOVE uses test-time epistemic uncertainty and change-point detection to decide when the robot should request human help in unseen scenarios.The approach monitors uncertainty over an episode and uses a predefined robust change-point detection algorithm.Source: §I · p. 1
  • The feedback module uses zero-shot NLP with streamlined prompts and GPT-3.5 and Llama-2 to interpret language and adapt the policy.The stated aim is test-time adaptation of trained policies without retraining.Source: §I · p. 2
  • Up to 80% more successful goal-reaching and 13.50% lower normalized trajectory length than other methods were reported in perceptually challenging real-world environments.The evaluation also covered synthetic and real-world experiments.Source: §I · p. 2

II. RELATED WORKS

  • Prior work explores human feedback, natural-language assistance, and grounded parsing, while noisy sensing can make pliable objects appear as LiDAR obstacles.Multi-sensor methods address perception challenges but may be computationally expensive and require multiple sensors.Source: §II · p. 2

III. PROBLEM FORMULATION AND NOTATIONS

The paper formulates navigation as an MDP and uses expert demonstrations to obtain a trained behavioral policy. It frames test-time adaptation as necessary because that policy may fail in unseen scenarios.

III. PROBLEM FORMULATION AND NOTATIONS

  • Navigation is formulated as an MDP, and the paper uses imitation learning from expert demonstrations to obtain a behavioral policy.The authors use imitation learning for illustration and state that the framework can also apply to other RL algorithms.Source: §III · p. 2
  • Expert demonstrations are collections of trajectories containing state-action sequences, and the policy objective is to maximize their likelihood.The supplied formulation also describes computing the objective gradient with respect to policy parameters.Source: §III · pp. 2–3
  • A trained policy that works in training scenarios may fail to adapt to unseen test scenarios, making calibrated uncertainty important for deciding when to stop and request feedback.The text notes that uncertainty estimation is entangled with input-space design and that a generic input space may not yield evident, well-calibrated uncertainty.Source: §III · p. 3

IV. PROPOSED APPROACH

RE-MOVE designs the agent’s observation space with uncertainty estimation in mind, then uses epistemic uncertainty to identify when feedback may be needed. Its uncertainty measure distinguishes model uncertainty from ambiguity in action predictions, and a changepoint detector prompts the human expert when uncertainty rises significantly.

A. Designing the observation space

  • The observation-space design matters because a policy’s epistemic uncertainty is directly related to its feature or state space.The section considers global visibility, goal-conditioned, and partial global visibility observations.Source: §IV · p. 3, A. Designing the observation space · p. 3

B. Characterization and Representation of the Uncertainty

  • Action-probability entropy alone can be misleading: common trajectory points may support different optimal actions without representing unseen observations.The method separates predictive uncertainty into aleatoric uncertainty and epistemic uncertainty; the latter is represented by mutual information between predicted actions and model parameters.Source: B. Characterization and Representation… · p. 3
  • Epistemic uncertainty is computed at every time step, and a changepoint detector prompts the human expert when it rises significantly.The Bayesian changepoint detection algorithm is used to improve query efficiency.Source: B. Characterization and Representation… · pp. 3–4

C. Incorporating Human Language Feedback

RE-MOVE uses discretized actions and prompt design to interpret human language instructions with near-zero-shot language-model generalization. Its overall algorithm trains an imitation policy from expert demonstrations, monitors epistemic uncertainty at deployment, and requests help when uncertain before converting language into actions.

C. Incorporating Human Language Feedback

  • Discretized action spaces support efficient prompts for a near-zero-shot language-model approach, an alternative to sequence-to-sequence methods that typically require large supervised datasets.The paper uses ChatGPT’s zero-shot generalization capability for instruction-to-action conversion.Source: C. Incorporating Human Language Feedback · p. 4

D. Overall Algorithmic Description

  • The algorithm trains an imitation policy from expert demonstrations, deploys it, monitors parameter-related epistemic uncertainty, and requests human help when uncertain.After help is requested, it converts the language instruction into actions.Source: D. Overall Algorithmic Description · p. 4

V. EXPERIMENTS

The evaluation spans synthetic environments and real indoor navigation, testing RE-MOVE with different observation spaces and obstacle configurations. In the real-world scenarios, RE-MOVE handles deceptive obstacles with human assistance, while the comparison with DWA varies by setting.

A. Discrete & Visual grid world

  • The discrete grid-world evaluation compares global visibility, which represents the entire grid, with goal-conditioned observations based on the agent’s local observation.The visual grid replaces tabular states with image inputs, and the experiments use boosted ensembles for discrete inputs and MC dropout for visual inputs.Source: A. Discrete & Visual grid world · p. 4
  • With global visibility, uncertainty is highest far from obstacles and briefly decreases near them, while human feedback lets the agent avoid an obstacle that otherwise leaves it stuck.The authors describe global visibility as unsuitable for RE-MOVE because it does not produce the desired near-obstacle uncertainty.Source: Figure 5 · p. 5

B. Real-world Experiments

  • In visual navigation with local observations, uncertainty rises 10–50 times near obstacles, providing a signal for thresholding human-feedback requests.The figure contrasts obstacle-free training observations with obstacle-present test observations; the robot asks for help where it is underconfident.Source: Figure 7 · p. 6
  • In simulated outdoor scenarios with solid obstacles, RE-MOVE detects uncertainty and requests human assistance to deviate from the original trajectory.The outdoor simulator is high-fidelity and includes trees and buildings; the real-world evaluation deploys the algorithm on a Clearpath Husky robot.Source: B. Real-world Experiments · pp. 4, 6, A. Discrete & Visual grid world · p. 4
  • In Scenario 1, DWA and RE-MOVE both reach the goal; in Scenario 2, RE-MOVE has a shorter normalized trajectory but slightly lower success, and in Scenario 3 only RE-MOVE reaches the goal.In Scenario 2, RE-MOVE identifies the tablecloth as uncertain and requests human assistance; the authors attribute its slightly lower success rate to uncertainty-estimation and language-to-sequence translation errors.Source: Table I · p. 6, B. Real-world Experiments · p. 6

VI. CONCLUSIONS AND FUTURE WORK

RE-MOVE is designed to let trained reinforcement-learning policies adapt at test time to previously unseen objects using language-based human feedback. The authors report efficacy across synthetic and real-world navigation tasks and identify dedicated language models and speech or image feedback as future directions.

VI. CONCLUSIONS AND FUTURE WORK

  • RE-MOVE identifies objects or obstacles absent during training and uses language-based human feedback to adapt trained reinforcement-learning policies at test time.The approach uses uncertainty quantification of model parameters in a decision function to optimize when to request feedback.Source: §VI · p. 6
  • The authors report efficacy for RE-MOVE across a variety of synthetic and real-world navigation tasks.Source: §VI · p. 6
  • Future work could develop a dedicated navigation language model and incorporate feedback through speech or images.Source: §VI · p. 6

APPENDIX I DISCRETE GRID WORLD EXPERIMENTS IN DETAILS

The appendix describes the discrete grid-world setup and how expert demonstrations are used to train imitation policies. It defines whole-grid and local observation inputs, then details the demonstration-generation and classifier-training procedure.

APPENDIX I DISCRETE GRID WORLD EXPERIMENTS IN DETAILS

  • The grid world is a maze-like environment with an accessible L × L center, impassable outer walls, and a shortest-path objective from start to goal.Dijkstra’s algorithm determines the shortest path, which is incorporated into the expert demonstration.Source: Appendix I · p. 9, 1) Generating Demonstration Data and… · p. 9
  • Global observation encodes the entire grid as a (L+4)^2-dimensional vector, while local observation uses an L′ × L′ subgrid around the agent, with L′ < L.Each grid-cell value represents its state at time t; obstacles or walls are assigned value 30.Source: Appendix I · p. 9

1) Generating Demonstration Data and Imitation Learning Policy:

  • The authors generate separate expert-demonstration sets for global and goal-conditioned observations, then train a 500-tree Random Forest classifier per policy, with maximum tree depth 50.Each demonstration is a collection of trajectories recording expert actions and states.Source: 1) Generating Demonstration Data and… · p. 9

2) Global Visibility Observation space vs. Goal-Conditioned Observation space:

RE-MOVE uses uncertainty to request human guidance when navigation policies encounter obstacles or unfamiliar observations. The section compares observation spaces, demonstrates feedback-guided recovery, and describes visual-grid and simulated-outdoor setups.

2) Global Visibility Observation space vs. Goal-Conditioned Observation space:

  • Global observations can keep predictions uncertain even without nearby obstacles because the trained policy encounters obstacle-containing observations it did not see during training.Uncertainty is highest far from obstacles and momentarily decreases near them.Source: 2) Global Visibility Observation space… · p. 9

3) RE-MOVE with Human Feedback:

  • Without human feedback, the agent gets stuck as prediction uncertainty rises; after feedback, it follows an instructed trajectory.A changepoint detector identifies uncertainty changes and prompts the expert.Source: 3) RE-MOVE with Human Feedback · p. 9

APPENDIX II VISUAL GRID WORLD

APPENDIX III SIMULATED OUTDOOR ENVIRONMENT

  • The simulated outdoor setup trains an imitation policy without deceptive objects and represents local observations as cost maps combining LiDAR obstacles with goal-distance values.The cost map linearly combines obstacle and distance layers; the distance layer dominates in obstacle-free settings.Source: APPENDIX III SIMULATED OUTDOOR… · pp. 9–10

A. Generating the Low Dimensional Observations

The low-dimensional observation is built by reducing the local cost map with PCA and combining that representation with the robot’s distance to the goal.

A. Generating the Low Dimensional Observations

B. Expert Navigation Data Collection and Imitation Learning Policy

The outdoor imitation policy is trained from expert-operated goal-reaching trajectories, while RE-MOVE uses uncertainty and feedback to address deceptive obstacles. Partial global visibility can cause unnecessary feedback requests.

B. Expert Navigation Data Collection and Imitation Learning Policy

  • Expert demonstrations pair local observations with subsequent robot actions while a human teleoperates the simulated Husky toward randomly located goals up to 20 meters away.The resulting trajectories form dataset Dsim.Source: B. Expert Navigation Data Collection… · p. 10

APPENDIX IV RE-MOVE WITH THE PARTIAL GLOBAL VISIBILITY OBSERVATION SPACE

  • Partial Global Visibility produces intermittent uncertainty spikes away from obstacles, leading to unnecessary feedback requests and making this observation space unsuitable for RE-MOVE.Uncertainty rises more when the agent is close to an obstacle than during these intermittent spikes.Source: APPENDIX IV RE-MOVE WITH THE PARTIAL… · p. 10
  • LiDAR-detected obstacles create high-cost cost-map regions that uncertainty can flag for assistance, unlike standard cost maps that may cause freezing in dense obstacles.The illustrated trajectories show the robot deviating after human feedback and then resuming goal-directed navigation.Source: Figure 11 · p. 11, Figure 12 · p. 11
  • With unseen obstacles, uncertainty rises more near the obstacle than during intermittent spikes, but those spikes can still prompt unnecessary feedback.Source: Figure 13 · p. 11

APPENDIX V EXPERIMENTS WITH PARTIAL OBSTACLES

The appendix tests uncertainty-based detection of passable but undesirable obstacles, prompt-driven conversion of language instructions into actions, and navigation in perceptually challenging real-world scenes.

APPENDIX V EXPERIMENTS WITH PARTIAL OBSTACLES

  • Uncertainty rises as the agent approaches and passes through a passable but undesirable obstacle, indicating that RE-MOVE can identify it.The example includes a slippery road patch that can be crossed with a higher chance of losing control.Source: Appendix V · pp. 12–13, Figure 16 · p. 13

APPENDIX VI CHATGPT PROMPTS

APPENDIX VII ADDITIONAL REAL-WORLD EXPERIMENTS

  • In real-world scenario 3, RE-MOVE reaches the goal through a pliable tablecloth region, while LiDAR-based DWA fails when its scan reveals no free space.In scenario 1, both RE-MOVE and DWA reach the goal while avoiding obstacles.Source: Figure 20 · p. 15
Loading 2303.07622v2…