PaperlayerView exact revision on arXiv

Source-linked AI summary

Inferring Rewards from Language in Context

Jessy Lin, Daniel Fried, Dan Klein, Anca Dragan

arXiv:2204.02515v1cs.CLcs.AI

TL;DR

Agents that map language only to actions may miss information about users’ underlying preferences, which matters when acting in new contexts. This paper pragmatically infers rewards by modeling how speakers choose utterances to elicit actions and convey preferences. On a flight-booking task, the full model improves accuracy over models treating language only as action descriptions and selects optimal actions in held-out environments.

  • Problem

    Language in interactive settings may be task-focused or ambiguous rather than a full description of preferences, making it challenging to infer general goals for acting in new contexts.

    Source: §1 · p. 1

  • Method

    The model infers rewards by reasoning about how speakers choose utterances both to elicit reward-maximizing actions and to describe their preferences.

    Source: §1 · p. 2

  • Results

    12% relative accuracy improvements over models that only treat language as descriptions of actions were achieved by the full model, which also selected optimal actions in held-out environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FLIGHTPREF provides a multi-turn natural-language benchmark for reward learning and evaluating preference interpretation beyond selecting an action in one environment.

    Source: §1 · p. 2, §7 · p. 9

    An oracle that switches between action-only and reward-only models reached 62.9% overall held-out accuracy, versus 59.1% for the combined model, leaving room to improve evidence integration.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight-booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).

1 Introduction

The paper argues that language can reveal users’ underlying preferences, not merely specify actions, helping agents generalize to new contexts. It introduces a pragmatic model and the FLIGHTPREF benchmark, and reports improved reward inference and held-out action selection.

1 Introduction

  • The motivating gap is that users often seek a task outcome rather than fully describing preferences, while agents need to infer those preferences to act in new instances.The paper contrasts reward inference from language with using past flight actions as evidence about preferences through inverse reinforcement learning.Source: §1 · p. 1
  • A user’s choice of wording can reveal which flight features matter, so the model reasons about utterances as both action instructions and descriptions of reward.Speakers are modeled as choosing utterances to elicit reward-maximizing actions and faithfully describe their reward.Source: §1 · p. 2
  • FLIGHTPREF is a dataset of natural-language preferences gathered in a multi-turn flight-booking game, presented as a benchmark for reward learning from language and interaction.Users communicate preferences to an assistant tasked with booking preferred flights.Source: §1 · p. 2
  • 12% relative accuracy improvements over models that treat language only as descriptions of actions were obtained by the full model.The model also consistently improved reward estimates over repeated interactions and accurately selected optimal actions in held-out environments.Source: §1 · p. 2

2 Related Work

Prior work often interprets language as an action in the context that elicited it, or learns rewards from action demonstrations. This paper extends those approaches by using language to infer rewards across repeated interactions and unseen environments.

2 Related Work

  • Grounded instruction-following methods produce actions for the context that elicited an utterance, whereas this work uses utterances to infer rewards across seen and unseen environments.Earlier methods used structured semantic representations or mapped language directly to primitive actions.Source: §2 · p. 2
  • Language-informed reward learning has often remained action-centric, using utterance–trajectory pairs or language-conditioned discriminators, while commands typically fully specify goals.This paper instead addresses repeated interactions where language may mix partial or ambiguous instructions with reward descriptions.Source: §2 · p. 2
  • Pragmatics research has largely studied single-turn communication, while this work extends pragmatic reasoning to repeated interactions and longer time horizons.The model reasons about utterances that both elicit actions and describe rewards; the flight example treats a trajectory as a single flight.Source: §2 · p. 3, Figure 2 · p. 3

3 Reward Inference from Language

The section formulates reward inference across repeated, shared-reward MDP interactions and models utterances as evidence about both intended actions and underlying preferences.

3 Reward Inference from Language

  • The agent infers a shared reward function across MDPs from utterances and uses it to choose high-reward trajectories in unseen contexts.In the flight domain, each interaction is a one-step MDP in which the agent chooses a flight.Source: §3 · p. 3

3.1 Model

  • The rational listener infers rewards by reasoning about a speaker whose utterances both elicit reward-maximizing actions and describe underlying preferences.The speaker balances current-context action success against reward description, with α controlling nearsightedness.Source: §3.1 · pp. 3–4
  • The action-optimizing component weights utterances by how likely they refer to high-reward trajectories in the current context.The trajectory optimality model uses a Boltzmann distribution over reward values, while the base listener models which utterances elicit a trajectory.Source: §3.1 · p. 4

3.2 A Generative Model of Utterances

  • Unlike prior action-centric accounts, the model lets reward influence an utterance directly and through the trajectory it suggests.The graphical-model comparison distinguishes information about reward beliefs from communication about intended behavior.Source: Figure 3 · p. 4, §3.2 · pp. 4–5
  • The reward-descriptive utterance component assumes that this information is independent of the current context.The paper reports no empirical improvement from making this component pragmatically optimize the listener’s reward belief.Source: §3.2 · p. 4

4 The FLIGHTPREF Task

FLIGHTPREF simulates repeated flight-booking interactions in which language must help an agent choose flights now and learn preferences for unseen contexts.

4 The FLIGHTPREF Task

  • FLIGHTPREF tests whether agents can learn flight preferences across interactions, beyond following instructions to choose a flight in the immediate context.The task is designed to simulate a simplified language-based flight-booking interaction.Source: §4 · p. 5
  • Each game uses one hidden reward function across rounds, with users describing preferred flights and assistants choosing or requesting another utterance.The players receive more points when the assistant chooses correctly without asking for help.Source: §4 · p. 5

4.1 Data collection

  • The dataset contains 2,568 utterances from 813 games, with 91 high-scoring games held out for evaluation.Workers played six games of six rounds each; each game corresponded to a different reward function.Source: §4.1 · p. 5
  • Utterances range from option-specific descriptions to broad preference statements, and many combine immediate instructions with incomplete generalizable preferences.Figure 4 samples the task’s diversity of instructive and reward-descriptive language.Source: §4.1 · p. 5, Figure 4 · p. 5

5 Model Implementation

The implementation combines learned base listener and speaker models with pragmatic reward inference, then evaluates held-out accuracy against component models.

5 Model Implementation

  • The base listener and reward-speaker models learn neural encodings of utterances, actions, and rewards, with distributions formed from inner products and softmax.The listener uses an MLP action encoder and BERT utterance encoder; the speaker uses an MLP reward encoder and BERT utterance encoder.Source: §5 · p. 6
  • The trajectory optimality model uses β = ∞ because speakers tend to refer primarily to the optimal option in this domain.The base listener distribution is normalized over flights available in the current context.Source: §5 · p. 6
  • The listener and speaker are trained separately on training data, and five random-seed models are ensembled by averaging output probabilities.The paper reports that ensembling improved performance for the full model and baselines.Source: §5 · p. 6
  • The full action+reward model significantly outperforms action-only and reward-only models on average held-out accuracy (p < .05).Table 1 averages accuracy over evaluation rounds and also reports oracle models that infer randomly chosen reward features perfectly.Source: Table 1 · p. 6
  • Pragmatic inference approximates the speaker distribution over a fixed set of short training utterances and uses importance sampling for reward posteriors.The initial reward prior is uniform, matching the dataset’s uniform reward sampling procedure.Source: §5 · p. 6

6 Experiments

Across repeated flight-booking interactions, the full action+reward model combines evidence from utterances and actions to infer rewards and select optimal flights in unseen contexts. It outperforms action-only and reward-only baselines overall, while the value of each evidence source changes with the number of observations and the utterance type.

6 Experiments

  • Models update reward posteriors after each utterance and option set, then are evaluated on whether their inferred rewards select the true optimal flight in 1,000 randomly generated three-option sets.The evaluation also compares L2 distance between estimated and true rewards as a context-independent metric; the action-only and reward-only baselines use only p_action or p_reward, respectively.Source: §6 · pp. 6–7

6.2 Learning over Multiple Interactions

  • Across observed-utterance counts, the full model outperforms action-only and reward-only models on held-out accuracy and L2 distance to the true reward.Figure 5 reports performance by number of observed utterances; five- and six-utterance rounds are combined because each bin has fewer than 25 samples.Source: Figure 5 · p. 7, §6.2 · p. 7

6.1 Overall Results

  • 59.1% held-out accuracy: the full action+reward model exceeds action-only by 6.3% and reward-only by 1.3%.For comparison, oracle baselines achieve 43% with k=1, 51% with k=2, 60% with k=3, and 65% with k=4 randomly chosen features known perfectly; the full model attains similar generalization without oracle feature information.Source: §6.1 · p. 7

6.2 Learning over Multiple Interactions

  • Language evidence helps most with few observations, while action evidence becomes more important as observations accumulate.Utterances tend to mention extreme reward features, whereas actions consistently provide new information about all features; in this domain, the feature and action spaces are relatively small.Source: §6.2 · pp. 7–8

6.3 Analyzing the Benefits of Combining Actions and Rewards

  • A single utterance and option set can update the same reward feature through both evidence sources or provide evidence about different features.In the latter case, the action-only model updates arrival time while the reward-only model updates price and stops, and the full posterior aggregates both.Source: §6.3 · p. 8
  • The full model handles both reward-descriptive utterances and utterances that identify only the optimal flight.Reward-only inference helps most with highly descriptive language, while action-only inference helps most with less informative language such as “the cheaper the better.”Source: §6.3 · pp. 8–9
  • An oracle that selects whichever component has higher held-out accuracy reaches 62.9%, versus 59.1% for the full model.The authors identify resolving whether a phrase like “cheap JetBlue” signals a general carrier preference or just a uniquely identified desired flight as a challenge for integrating evidence.Source: §6.3 · p. 9

6.4 Inference Improves with Known Actions

  • Skipping posterior updates when the base listener predicts the wrong option improves held-out accuracy by 6% over reward-only after six observations.This is 4% above the original gap; the authors note that the task design provides no user flight-choice demonstrations, though such demonstrations could serve as an oracle instruction-following model for that context.Source: §6.4 · p. 9

7 Discussion & Conclusion

The paper treats language as communicating both intended actions and the preferences behind them, and introduces FLIGHTPREF to study preference interpretation in multi-turn flight-booking interactions. It argues that held-out accuracy evaluates interpretation beyond choosing the right action in one environment, and outlines several directions for extending the task.

7 Discussion & Conclusion

  • The method infers reward functions from natural language to represent goals, preferences, and intents underlying actions.The paper frames language as communicating both which action to take and why it is desirable, including in new contexts.Source: §7 · p. 9
  • FLIGHTPREF evaluates interpretation beyond selecting the right action in a single environment, using held-out accuracy on naturalistic, multi-turn flight-booking interactions.The proposed extensions include more complex reward functions, modeling natural preference priors, interactive uncertainty handling, and other domains such as language-conditioned robotics.Source: §7 · p. 9
  • The task extends pragmatic interpretation to ambiguity about preferences as well as actions.The paper describes language-guided agents for longer-horizon interactions as a potential use for modeling the relationship between language, actions, and rewards.Source: §7 · p. 9

A Data Collection

The data collection uses a multi-turn flight-booking game in which workers play customers with assigned preferences or assistants tasked with learning them. The task instructions describe the scenario, goals, and reward structure.

A Data Collection

  • Workers were recruited from the United States with at least 98% approval, 5,000 completed HITs, and a qualification task requiring 15 minutes of play and active participation.The passage also describes a manual review of the qualification task.Source: A Data Collection · p. 13
  • In the game, customers communicate assigned flight preferences to assistants, who try to learn those preferences and book flights the customer wants.Customers are told to imagine the assigned preferences as their own; the instructions frame the task as studying how people communicate preferences to assistants.Source: A Data Collection · p. 13
  • Each round awards 25 points for a correct assistant choice, subtracts 100 for an incorrect choice, and subtracts 20 when the assistant asks for help.Workers were compensated $4 for 30 minutes, with a $0.20 bonus for every 25 points.Source: A Data Collection · p. 13

B Dataset Details

The dataset details include a plot of utterance lengths measured in tokens.

B Dataset Details

  • The appendix plots the distribution of utterance lengths in FLIGHTPREF, with length measured in tokens.The supplied passage identifies the plot but does not provide distribution values.Source: B Dataset Details · p. 13

C.1 Training

Training uses separate listener and speaker models, including a latent-variable objective for the base speaker models that distinguishes action- and reward-descriptive utterances. The appendix also describes training choices and points to a hyperparameter table for the base speaker and listener models.

C.1 Training

  • Training uses BERT utterance encoders, fine-tunes their parameters, and applies early stopping on held-out validation loss.The passage reports separate listener and speaker training, with AdamW and different learning rates and batch sizes for the two models.Source: §C.1 · p. 13

C.2 Hyperparameters

  • The base listener is trained with a per-instance listener loss, while the appendix directs readers to Table 2 for base speaker and listener hyperparameters.The table caption identifies its scope but does not provide the hyperparameter values in the supplied passages.Source: §C.1 · p. 13, Table 2 · p. 14

C.1 Training

  • The speaker training setup includes a separate action speaker conditioned on the optimal action because utterances often describe actions as well as rewards.The action speaker is used only during training; the authors report that pragmatic reasoning over an instruction-following model performs better at evaluation.Source: §C.1 · pp. 13–14
  • The base speaker models are trained jointly with a latent variable that assigns utterances softly to action-descriptive or reward-descriptive clusters.Reward-descriptive utterances provide stronger supervision for the base reward speaker model.Source: §C.1 · p. 14
  • The base speaker models use up to four synthetic hard negatives per training example, created by replacing detected attribute mentions with distractor attributes.At evaluation, normalization uses training utterances of at most eight words and no digits; the paper uses fewer normalizers during training for efficiency.Source: §C.1 · p. 14

C.2 Hyperparameters

  • The appendix identifies BERT-base and the HuggingFace Transformers library as the encoder implementation.It also states that the models use MLPs with fully connected layers, ReLU nonlinearities, and dropout.Source: §C.2 · p. 14

D.1 Analyzing the effect of pragmatically modeling the speaker

Pragmatic modeling of a belief-only speaker improves held-out accuracy over non-pragmatic alternatives, while the relative strength of belief-only and action-only models varies with utterance content.

D.1 Analyzing the effect of pragmatically modeling the speaker

  • Because the flight action space has little ambiguity, the analysis focuses on pragmatically modeling a belief-only speaker.The authors do not expect pragmatically modeling an action-only speaker to help in this domain.Source: §D.1 · p. 14
  • Belief-only pragmatics improves held-out accuracy by 5–6% over direct p(θ | u) inference and training baselines.The alternatives omit explicit calculation of the normalized utterance distribution p(u | θ).Source: Table 3 · p. 14, §D.1 · p. 14

D.2 Examples ranked by action-only and belief-only accuracy difference

  • Belief-only excels on reward-descriptive utterances, whereas action-only tends to do better when utterances contain less information.Figure 7 compares examples where both models perform above chance after updating on that round.Source: Figure 7 · p. 15, §D.2 · p. 15
Loading 2204.02515v1…