PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Instructed Reinforcement Learning for Human-AI Coordination

Hengyuan Hu, Dorsa Sadigh

arXiv:2304.07297v2cs.AIcs.CLcs.LGcs.MA

TL;DR

When high-quality human behavioral data is unavailable, multi-agent RL can converge to equilibria that differ from human preferences. InstructRL uses an instruction-conditioned LLM prior to regularize RL, and the paper reports instruction-following policies and improved human-AI coordination when people know the instructions. The method depends on mapping actions to language descriptions, which may be difficult for some environments.

  • Problem

    Without high-quality human behavioral data, multi-agent RL may converge to any of several mutually incompatible equilibria, including policies that are hard for humans to collaborate with.

    Source: §1 · p. 1

  • Method

    InstructRL uses an LLM to construct a prior conditioned on human instructions and observation descriptions, then uses that prior to regularize RL.

    Source: §1 · p. 2

  • Results

    InstructRL learns instruction-satisfying policies in Say-Select and Hanabi, and human-AI coordination improves when people know the agents’ language instructions.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The paper concludes that language instructions can guide RL toward desirable equilibria and that people coordinate better with agents when given the instructions used to train them.

    Source: §6 · p. 9

    The method requires actions to be mapped to language descriptions, which may be difficult for actions such as continuous robot-arm joint angles.

    Source: §4 · p. 4

Abstract

from arXiv · show

One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL, that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance in human evaluations in Hanabi.

1. Introduction

The paper targets human-AI coordination where human behavioral data may be unavailable and unguided RL can converge to equilibria humans find difficult to work with. It proposes using natural-language instructions to guide equilibrium selection, then evaluates the approach in Say-Select and Hanabi.

1. Introduction

  • Without high-quality human data, RL can converge to any of multiple mutually incompatible equilibria, some of which humans find hard to collaborate with.Humans may prefer a subset of policies aligned with their capabilities and common sense.Source: §1 · p. 1
  • In Say-Select, many joint policies achieve perfect results, but the authors present matching Alice’s number and quitting after a repeated number as a natural policy.In the illustrated episode, Bob selects the ball Alice names and quits when she repeats a number after the positive-reward balls are gone.Source: §1 · p. 1, Figure 1 · p. 2
  • Prior approaches that train a common best response to diverse policies require RL to generate human-preferred equilibria and incur higher computational cost.In Say-Select, the response may also need many episodes to identify a human partner’s policy while the human adapts.Source: §1 · pp. 1–2
  • The proposed instructRL framework uses an LLM-generated prior, conditioned on the instruction and observation descriptions, to regularize RL toward an equilibrium that satisfies the instruction.The authors describe applying this regularization to algorithms such as Q-learning or PPO.Source: §1 · p. 2
  • In Hanabi, instructRL yields equally strong but qualitatively different policies under different instructions, and human coordination improves when people know the instructions.The paper also reports that instructRL learns intuitive, human-compatible policies in Say-Select.Source: §1 · p. 2

2. Related Work

Related work uses human data, cognitively inspired algorithms, diverse policies, or language in RL. The paper distinguishes instructRL by using language instructions and an LLM prior to guide multi-agent equilibrium selection without requiring a human dataset.

Language and Human Inductive Bias

  • Kumar et al. (2022) report that auxiliary prediction of representations from human-generated language task descriptions leads to more human-like behavior, aligning with this paper’s motivation about language and human inductive bias.The paper relates that finding to people preferring policies that can be easily described in language.Source: §2 · p. 2

Language for Exploration in RL

  • Unlike language-based exploration methods for single-agent RL, instructRL targets equilibrium selection in multi-agent systems; the authors describe the two problems as orthogonal.They argue that even when exploration is not an issue, a method is needed to obtain a human-like policy.Source: §2 · p. 2

Foundation Models and In-Context Learning for RL

  • MineDojo uses a video-text model to provide dense RL rewards, while instructRL does not rely on purpose-collected domain-specific data.The paper describes MineDojo as collecting a large multimodal Minecraft dataset and training a CLIP-style contrastive model.Source: §2 · p. 3
  • Some foundation-model approaches use language to specify rewards, whereas instructRL uses environment reward for optimal-policy learning and an LLM prior to steer instruction satisfaction.The paper contrasts this hybrid role with using LLM-generated rewards as the only learning signal, which it says limits application to small settings.Source: §2 · pp. 2–3

Human-AI Coordination

  • Prior human-AI coordination work includes modeling human behavior, cognitively inspired learning, and training a common best response to diverse policies.The approaches are presented as three research directions.Source: §2 · p. 3
  • InstructRL uses human-specified instructions and LLMs without requiring a human dataset, and can produce semantically different policies for different instructions.The authors say it can be combined with cognitively inspired methods and connects to diverse-policy approaches.Source: §2 · p. 3

3. Background

The background defines the cooperative, partially observable multi-agent RL setting and the role of large language models. It frames learning as maximizing shared return while each agent acts from its own observation history.

MARL

  • The paper considers a fully cooperative setting with a shared reward and partial observability, where each agent has its own observation function.The authors note that inferring other agents’ intentions is harder under partial observability.Source: §3 · p. 3
  • A policy maps an agent’s observation-action history to a distribution over its actions, and a joint policy collects the policies of all agents.MARL aims to maximize expected total return over trajectories, with discounted rewards defining the game return.Source: §3 · p. 3

LLMs

  • Large language models are generative text models trained on large internet-collected datasets to predict the next token from context.The background notes their zero- or few-shot capabilities and that some are fine-tuned using human feedback or instructions.Source: §3 · p. 3

4. Method

InstructRL constructs an LLM prior from a human instruction and language descriptions of observations and actions, then uses that prior to regularize RL. The method includes Q-learning and PPO variants and requires actions to be expressible in language.

4. Method

  • InstructRL conditions an LLM prior on the human instruction and a language prompt describing observations from the current policy’s rollout.The instruction remains fixed during RL training, with no active human feedback in the loop.Source: §4 · pp. 3–4
  • The prior estimates action probabilities from language descriptions of observations and actions, so the method requires descriptions for both.The LLM evaluates each possible action at each time step; the method also describes generating observation text for board games such as Hanabi and Diplomacy.Source: §4 · pp. 3–4
  • The method’s scope is constrained when environment actions cannot readily be mapped to language descriptions, such as continuous robot-arm joint angles.The authors identify action-to-language mapping as a limitation and anticipate broader application with multimodal foundation models.Source: §4 · p. 4
  • For Hanabi, the method can assess action plausibility with a question-answering prompt and assign a binary logit based on whether affirmative or negative answers are more probable.The authors also note that coarse priors can guide equilibrium selection without imposing fine-grained LLM preferences.Source: §4 · p. 4
  • The authors say an LLM prior alone is insufficient for complex tasks, so RL is still needed to find good policies while the prior steers instruction satisfaction.They give examples of language models failing to determine when to quit in Say-Select or to play Hanabi at human level.Source: §4 · p. 4
  • For Q-learning, instructRL adds the LLM prior’s log probability to action values, while PPO adds a KL penalty to its objective.These regularizers use the prior as a reference policy; the paper names the variants instructQ and instructPPO.Source: §4 · p. 4

5. Experiment

The experiments test instructRL in Say-Select and Hanabi, asking whether instructions produce distinct, instruction-following policies and improve human-AI coordination. In Hanabi, showing humans the agents’ instructions is associated with better coordination and a better reported experience.

5.1. Say-Select Experiment

  • In Say-Select, instructQ converged to the intuitive joint policy in all 10 seeds, whereas vanilla Q-learning policies could be self-play optimal without following the instruction.The instructed policy quits when Alice repeats the same action and otherwise selects the ball matching her number; Figure 3 contrasts these policies.Source: §5.1 · pp. 5–6, Figure 3 · p. 5

5.2. Hanabi Experiment

  • Hanabi’s color-based and rank-based strategies swap whether color or rank hints indicate playable cards, with the other hint type used for cards to save.Because hints have both purposes and can be misinterpreted, the experiments use instructions specifying which strategy to follow.Source: §5.2 · p. 6
  • For Hanabi, the LLM prior uses binary yes/no action logits and a softmax over legal actions, providing coarse guidance while limiting the effect of LLM biases.The prior is computed and cached before RL training; the reported GPT API bill was roughly US$200.Source: §5.2 · p. 7
  • InstructQ and instructPPO achieve self-play and intra-AXP scores similar to their vanilla counterparts under both color and rank instructions.The small self-play/intra-AXP gap indicates convergence to nearly identical policies across seeds, a precondition for human-AI coordination.Source: §5.2 · pp. 7–8, Table 3 · p. 7
  • Across Hanabi agents, at least 98% of played cards were ones the agent had knowledge about, while instructed agents relied more on their prescribed hint type.The action matrix shows all agents focusing on the newest card; vanilla Q-learning mixes color and rank hints, while instructQ and instructPPO bias toward the instructed type.Source: §5.2 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 8
  • When language instructions were shown, human partners lost only 1 of 10 games; on two 7-point Likert questions, instructQ with language scored 6.00 ± 0.24 and 6.20 ± 0.31, respectively.Without knowing the language instructions, the two instructQ agents achieved results comparable to the Q-learning baseline.Source: §5.2 · p. 8
  • Adding an LLM prior at test time to fixed Q-learning agents decreased self-play performance while bringing little improvement in coordination with corresponding instructQ agents.The authors identify test-time adaptation with language instructions as a direction for future work.Source: §5.2 · p. 9

6. Conclusions

InstructRL uses instruction-conditioned LLM priors to regularize RL toward equilibria aligned with human preferences, and the paper reports instruction-following policies and better human coordination when people know the instructions. Test-time adaptation and more flexible instructions are future directions.

6. Conclusions

  • InstructRL uses instruction-conditioned LLM priors to regularize RL toward equilibria aligned with human preferences, without collecting labeled human data.The paper reports that the resulting policies satisfy the instructions and that people coordinate better when told the instructions used to train the agents.Source: §6 · p. 9
  • Test-time adaptation with language instructions remains a future direction.The paper also identifies finer-grained instructions and more flexible formats, such as human-AI dialogue, as future extensions.Source: §6 · p. 9

A.1. Say-Select Experiments

The Say-Select experiment uses tabular Q-learning and applies instructQ regularization only to Bob, with the vanilla baseline setting its regularization weight to zero.

A.1. Say-Select Experiments

  • InstructQ uses λ = 0.25 for Bob, while the vanilla Q-learning baseline sets λ = 0.The experiment uses tabular Q-learning without neural networks.Source: §A.1 · p. 12
  • The experiment uses ϵ = 0.15 for ϵ-Greedy exploration and a learning rate of 0.02.Source: §A.1 · p. 12
  • A batch size of 64 is used to reduce gradient-estimation variance.Source: §A.1 · p. 12

A.2. Hanabi Experiments

The Hanabi experiments fine-tune an OBL level 1 policy and use Q-learning and PPO, while the illustration shows how hints and instructions shape play. The appendix specifies training configurations and gives examples of color- and rank-based instructions.

A.2. Hanabi Experiments

  • The implementation uses one training worker and thousands of rollout workers, based on the open-source off-belief learning repository.Source: §A.2 · p. 12
  • Fine-tuning the OBL level 1 policy reduces compute needs, supports consistency across random seeds, and eases regularization because it already uses rank and color hints.The starting policy plays only cards known to be safe and uses a fair mixture of hint types; the experiments use Q-learning and PPO without multi-agent heuristics.Source: §A.2 · p. 12
  • InstructQ trains for 250K mini-batches, while instructPPO trains for 750K total, with both methods annealing λ and vanilla baselines using λ = 0.Each mini-batch contains 128 game episodes; initial regularization weights are 0.15 for instructQ and 0.05 for instructPPO.Source: §A.2 · p. 12

B. Illustration of Hanabi

  • In the illustrated Hanabi turn, the player sees the partner’s hand but not their own and uses received hints to guide card play.A rank-1 hint identifies the third card as a 1, while a blue hint marks the partner’s blue card; a rank-1 hint to the partner would highlight both red and blue 1s.Source: B. Illustration of Hanabi · p. 12, Figure 7 · p. 12
  • The examples contrast instructions to play cards identified by color or rank hints and to respond to other actions with the alternate hint type.The color-instruction example asks whether to play a card at position D; the rank-instruction example asks whether to hint color after the partner discards.Source: B. Illustration of Hanabi · p. 13
  • Under a color-based policy, the partner infers that the hinted blue card should also be a 1, then plays it if the known 1 is not blue and otherwise discards it.Source: B. Illustration of Hanabi · p. 13

C. Robustness Analysis in Hanabi

The robustness analysis evaluates LLM priors against a rule-based oracle, tests simpler instructions that were not prompt-engineered, and examines randomly corrupted priors.

C. Robustness Analysis in Hanabi

C.1. Accuracy of the LLM Priors

The appendix tests prior accuracy and robustness to prompt simplification and corruption: original instructions match a rule-based oracle, and policies trained with simpler or moderately noisy priors remain similar in cross-play.

C.1. Accuracy of the LLM Priors

  • The original color and rank instructions achieve 100% accuracy against their corresponding rule-based oracles.Accuracy is measured by comparing LLM outputs with oracle actions.Source: §C.1 · p. 13
  • Without quotations, the LLM makes 1 mistake out of 3,852 questions for each color and rank instruction.The paper shows example failure cases in Figures 8 and 9.Source: §C.1 · p. 13

C.2. Performance with a Simpler Instruction

  • GPT-3.5 makes 113 mistakes with the simple color instruction and 110 with the simple rank instruction, roughly 3% of queries in each case.For color, all mistakes occur in the “otherwise” category, and 109 of 113 follow a rank hint.Source: §C.2 · p. 13
  • Policies trained with simple instructions have nearly the same cross-play scores as original-instruction policies, indicating robustness to imperfect priors and less prompt engineering.The authors characterize the resulting policies as nearly identical to those trained with the more accurate priors.Source: §C.2 · p. 14, Table 6 · p. 14

C.3. Performance with Randomly Corrupted Prior

  • InstructQ withstands 10–15% prior noise without a noticeable cross-play drop; beyond that, rank-policy scores fall faster but eventually remain higher than color-policy scores.The authors note that learning-algorithm inductive bias, initialization, and λ annealing may contribute to the differing patterns.Source: §C.3 · p. 14

D. Preliminary Experiments on Test-Time Adaptation in Hanabi

The preliminary test-time study adds an instruction-conditioned LLM prior to a fixed Q-function without fine-tuning. Cross-play improves only slightly over a narrow λ range while self-play declines, limiting the practical value of this post-hoc adaptation.

D. Preliminary Experiments on Test-Time Adaptation in Hanabi

  • Cross-play rises slightly: rank scores go from 21.13 ± 0.11 to 21.30 ± 0.10, and color scores from 21.86 ± 0.10 to 22.04 ± 0.09, while self-play drops as λ increases.Figure 11 distinguishes self-play (dashed lines) from cross-play with instructQ counterparts (solid lines); shading shows standard error.Source: Figure 11 · p. 15, D. Preliminary Experiments on… · p. 15
  • The test-time setup adds the color- or rank-based LLM prior to a fixed Q-function without additional fine-tuning.The base policy is Q-learning without KL regularization, and the partner is an instructQ agent.Source: D. Preliminary Experiments on… · p. 15
  • The small improvement and narrow useful λ range make post-hoc adaptation impractical to tune, and the authors emphasize fine-tuning in their current algorithm.They characterize instruction-based test-time adaptation as hard and encourage more radical innovations.Source: D. Preliminary Experiments on… · p. 15

Related papers

Loading 2304.07297v2…