PaperlayerView exact revision on arXiv

Source-linked AI summary

Eliciting Human Preferences with Language Models

Belinda Z. Li, Alex Tamkin, Noah Goodman, Jacob Andreas

arXiv:2310.11589v1cs.CLcs.AIcs.LG

TL;DR

Users can find it difficult to specify nuanced preferences through examples or prompts, particularly when edge cases are hard to anticipate. GATE uses language models to elicit preferences through free-form interaction; across the studied domains, it often yields more accurate models than prompting or active learning at comparable or lower user effort.

  • Problem

    Specifying nuanced human preferences through examples or prompts can require effort and leave task ambiguity, especially when users must anticipate edge cases.

    Source: §1 · p. 1

  • Method

    GATE uses language models to elicit and infer preferences through open-ended interaction, such as generated questions or edge cases for users to label.

    Source: §1 · p. 3

  • Results

    Across three domains, GATE often yields more accurate models than prompting or active learning while requiring comparable or less user effort and surfacing novel considerations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Interactive, language-based elicitation is a flexible approach for building personalized models that can address challenges of prompt- and example-based methods.

    Source: §1 · p. 3

    The experiments cover three domains, while more complex applications such as software design and legal or medical decision-making remain possible extensions.

    Source: §7 · p. 11

Abstract

from arXiv · show

Language models (LMs) can be directed to perform target tasks by using labeled examples or natural language prompts. But selecting examples or writing prompts for can be challenging--especially in tasks that involve unusual edge cases, demand precise articulation of nebulous preferences, or require an accurate mental model of LM behavior. We propose to use *LMs themselves* to guide the task specification process. In this paper, we introduce **Generative Active Task Elicitation (GATE)**: a learning framework in which models elicit and infer intended behavior through free-form, language-based interaction with users. We study GATE in three domains: email validation, content recommendation, and moral reasoning. In preregistered experiments, we show that LMs prompted to perform GATE (e.g., by generating open-ended questions or synthesizing informative edge cases) elicit responses that are often more informative than user-written prompts or labels. Users report that interactive task elicitation requires less effort than prompting or example labeling and surfaces novel considerations not initially anticipated by users. Our findings suggest that LM-driven elicitation can be a powerful tool for aligning models to complex human preferences and values.

1 INTRODUCTION

As language models make task specification possible through examples and natural-language instructions, users still face the effort of articulating preferences and anticipating edge cases. GATE uses models to elicit preferences through open-ended interaction, and its preregistered experiments often outperform prompting or active learning while requiring comparable or less user effort.

1 INTRODUCTION

  • Users must formalize preferences through examples or instructions, and both specifying preferences and refining systems require ongoing effort.Language-model task ambiguity also makes prompts and examples difficult to write comprehensively, especially when users must anticipate edge cases.Source: §1 · p. 1
  • Natural-language prompts and a few examples can leave preferences underspecified, as tennis interests may concern competition, improving a serve, or satire.The resulting task ambiguity is especially relevant to open-ended and higher-stakes applications.Source: §1 · p. 1
  • GATE uses models to elicit and infer preferences through open-ended interaction, including informative questions and edge cases for users to label.The paper evaluates GATE in email validation, content recommendation, and moral reasoning; experiments report more accurate models than prompting or active learning in many settings, with comparable or less user effort.Source: §1 · p. 3
  • The paper presents GATE methods and experimental evidence that interactive, language-based elicitation can build personalized models and address challenges of prompt- and example-based methods.The authors characterize this approach as flexible and powerful for building personalized models.Source: §1 · p. 3

2 LEARNING AS TASK ELICITATION

The task-elicitation framework weighs the human cost of specifying a task against how well a learned predictor aligns with that person's choices. Existing approaches differ in whether elicitation is interactive and whether specifications are examples or free-form inputs.

2.1 THE TASK ELICITATION FRAMEWORK

  • The framework seeks to minimize expected specification cost and human–predictor disagreement, with α and β trading off those objectives.Cost can reflect time and mental effort, while alignment measures agreement with choices the human would make.Source: §2.1 · p. 3
  • In the experiments, approaches are compared on alignment at a fixed cost, while suitable cost and alignment measures depend on the domain.Specification cost could measure words typed; alignment could measure prediction-level agreement.Source: §2.1 · p. 3

2.2 EXISTING LEARNING PARADIGMS IN THE TASK ELICITATION FRAMEWORK

  • Existing task-elicitation approaches vary along two axes: interactivity, from passive one-shot input to response-dependent queries, and flexibility, from examples to free-form input.Free-form specifications can include natural-language instructions and explanations.Source: §2.2 · p. 4
  • Supervised learning is passive and example-based, active learning interactively selects examples from a fixed pool, and prompting is passive and free-form.Supervised learning fits or fine-tunes a model on labeled input–output pairs; prompting passes a natural-language task description to a pretrained model.Source: §2.2 · p. 4

3 GENERATIVE ACTIVE TASK ELICITATION

Prompting and example-based methods place the burden on users to specify preferences clearly and anticipate edge cases. GATE combines free-form specifications with interaction, using language models to elicit preferences through generated examples or questions.

3 GENERATIVE ACTIVE TASK ELICITATION

  • Prompt and example-based methods burden users with comprehensive task specification, while ambiguity makes it difficult to articulate preferences and anticipate deployment edge cases.Unclear prompts or examples can lead to undesired behavior.Source: §3 · p. 4
  • GATE fills the gap between free-form specification and interactive elicitation by having a model query users with rich specifications.The framework combines flexibility associated with prompting and interaction associated with active learning.Source: §3 · p. 4

3.1 METHODS FOR GATE

  • The paper prompts a language model to ask questions conditioned on prior answers, then uses the elicitation transcript with an input to predict a label.Its elicitation policies vary the kinds of questions asked.Source: §3.1 · p. 5
  • GATE methods generate examples for labeling, binary yes-or-no questions, or open-ended questions, spanning concrete scenarios to broad preferences.Open-ended questions can elicit broad knowledge but may be overly broad or difficult to answer; users may respond with as much detail as they choose.Source: §3.1 · p. 5

4 EXPERIMENT SETUP

The experiments compare elicitation policies by allowing users to interact for five minutes, then measuring model–user agreement on held-out examples. They span three preference domains and evaluate prediction quality, elicitation speed, and perceived user effort.

4 EXPERIMENT SETUP

  • Users interacted with an elicitation policy for five minutes before users and learned models independently labeled held-out examples for an agreement comparison.The domains were chosen because they lack a single correct behavior learnable from language-model pretraining.Source: §4 · p. 5

4.1 DOMAINS AND DATASETS

  • The three domains are article recommendations, judgments about when stealing bread is ethical, and email-address validation preferences.Recommendation test cases are held-out articles; moral-reasoning scenarios and email test cases were manually constructed by the authors.Source: §4.1 · pp. 5–6

4.2 HUMAN INTERACTION

  • The study recruited 388 participants across domain–method pairs, mostly from English-speaking Prolific users, with additional computer-science participants for email validation.Participants were paid an average of $12 per hour, and the experiments received IRB approval.Source: §4.2 · p. 6

4.3 MODELING DETAILS

  • GPT-4 generated elicitation questions or edge cases from a domain description and interaction history, then predicted test labels from the task specification and sample.The model snapshot was gpt-4-0613.Source: §4.3 · p. 6

4.4 BASELINE METHODS

  • Baselines included user-written preference prompts, supervised example labeling, and pool-based active learning, with the latter two studied only for content recommendation.The active-learning baseline used diversity sampling over 15 Sentence-BERT clusters, while supervised learning drew from a pool of 160,000 Microsoft News articles.Source: §4.4 · pp. 6–7

4.5 EVALUATION AND METRICS

  • Evaluation measured probability assigned to the user-preferred answer, its improvement over elicitation time, and users’ ratings of perceived mental effort.The area under the p(correct)-time curve captures how quickly useful information is elicited; API latency was accounted for by extending the final accuracy to five minutes.Source: §4.5 · p. 7

5 RESULTS

GATE generally improves preference alignment over no interaction across all studied domains, and often matches or exceeds existing elicitation methods. Users generally found interactive elicitation less mentally demanding; additional analyses examined preference variation, possible preference shifts, model-generated questions, and simulated participants.

5 RESULTS

  • GATE improves over user-written prompts in 6/10 settings for absolute performance and 7/10 for AUC; generative yes/no questions improve in every setting studied.The study lacked enough power to assess significance for generative yes/no questions in moral reasoning.Source: §5 · p. 7
  • GATE improves over no interaction across all domains, with significance at 0.05 in all but email, where only generative active learning was significant.Figure 3 reports the time-normalized alignment comparison.Source: §5 · p. 7, Figure 3 · p. 8
  • In content recommendation, GATE methods, particularly generative open-ended questions, significantly improve over supervised learning and pool-based active learning.Source: §5 · p. 8
  • Users generally found interactive elicitation less mentally demanding than user-written prompts, especially when labeling samples or answering yes/no questions.Figure 4 plots perceived mental demand across methods and domains; higher values indicate greater demand.Source: §5 · p. 8, Figure 4 · p. 9

5.1 SAMPLE TRANSCRIPTS

  • Figure 5 presents sample transcripts of users interacting with the different GATE methods.Source: §5.1 · p. 8

5.2 ANALYSIS

  • Preference entropy averaged 0.77 bits, and test-question labels showed no effect of LM interaction on users’ subsequent judgments.Many questions had high entropy while many others had little; Figure 4 compares yes-responses before and after LM interaction.Source: §5.2 · p. 8, Figure 4 · p. 9
  • LM-generated questions were complex and subtle, often building on previous questions and reflecting domain knowledge and possible nuances.Source: §5.2 · p. 8
  • Persona-based language models simulated humans well in content recommendation and email validation, but not in moral reasoning.The authors conclude that personas may help in some cases but are not yet sophisticated enough to replace real participants.Source: §5.2 · p. 9

6 OTHER RELATED WORK

Related work spans verbal preference elicitation, computational preference modeling, and research on task ambiguity. The paper situates GATE as extending active learning into a generative setting that queries users with generated examples and questions.

6.1 ELICITING DESCRIPTIONS OF PREFERENCES

  • Protocol analysis describes obtaining and analyzing verbal reports about cognitive processes, while high-quality reports can be difficult to obtain; requirements elicitation gathers information when users struggle to anticipate needs or clearly describe preferences.The authors explore whether language models could take the place of human researchers in surfacing these insights.Source: §6.1 · p. 9
  • Figure 5 shows real interaction transcripts, with LM-generated system messages and human-produced user messages, and illustrates diverse, contextually appropriate questions.Source: Figure 5 · p. 10

6.2 COMPUTATIONAL MODELING AND QUERYING OF PREFERENCES

  • Computational preference elicitation spans methods including conjoint analysis, bandits, Bayesian approaches, recommender systems, inverse reinforcement learning, and question generation.The passage also discusses revealed, stated, and deliberatively refined preferences.Source: §6.2 · p. 11
  • Earlier active learning selects informative examples from a fixed pool; this work extends that line by querying users with generated examples and questions.Source: §6.2 · p. 11

6.3 TASK AMBIGUITY AND UNDERSPECIFICATION

  • Task ambiguity arises when multiple tasks fit a model’s inputs, including incomplete natural-language descriptions; the authors find LMs can often resolve such ambiguity by asking informative questions.Underspecified pipelines can lead to unpredictable and undesired deployment behavior.Source: §6.3 · p. 11

7 DISCUSSION AND CONCLUSION

GATE uses free-form interaction to elicit preferences, with initial evidence of sometimes greater accuracy and less effort than supervised, active-learning, or prompting approaches. The discussion also describes potential benefits and risks, reproducibility constraints, and prompt procedures.

7 DISCUSSION AND CONCLUSION

  • The authors report initial evidence that GATE can elicit preferences sometimes more accurately and with less effort than supervised learning, active learning, or prompting.Source: §7 · p. 11

ETHICAL CONSIDERATIONS

  • GATE raises potential privacy and automation-bias risks, but further work is needed to determine whether or when they exceed those of prompting-based steering.Source: ETHICAL CONSIDERATIONS · p. 12

REPRODUCIBILITY

  • The study plans to release code, the preregistration, and prompts, but its closed-source model is periodically deprecated, which may hinder reproducibility.Experiments queried GPT-4 at temperature 0 for replicability.Source: REPRODUCIBILITY · p. 12

A.1 ELICITATION PROMPTS

  • Elicitation prompts ask the LM for informative questions or edge cases that reveal desired behavior beyond what has already been queried.Source: §A.1 · p. 16

A.2 DECISION-MAKING PROMPTS

  • Decision prompts use the elicitation transcript to predict probabilities for domain-specific test cases, including email validity, article interest, and moral judgments.Source: §A.2 · p. 17

B EXPERIMENTAL DETAILS

The experimental details cover participant counts, elicitation interfaces, and test-set judgments. An additional AUROC analysis reports broadly similar trends for GATE, with smaller and noisier gains.

B.1 NUMBER OF PARTICIPANTS

  • The study recruited 388 participants total: 183 for content recommendation, 121 for moral reasoning, and 84 for email validation.Source: §B.1 · p. 17

B.2 USER INTERFACE DETAILS

B.2.1 ELICITATION

  • During elicitation, GATE users answered chatbot questions, while prompting users were asked to explain their domain preferences in detail.Source: §B.2.1 · pp. 19–20

B.2.2 DECISION-MAKING

  • In decision-making, participants judged held-out articles, moral situations, or email strings as acceptable or not.Source: §B.2.2 · p. 20, Figure 8 · p. 21
  • GATE methods generally beat supervised learning, pool-based active learning, and prompting, and also beat no interaction (∆AUROC = 0), though smaller gains and higher variance make significance harder to establish.Source: Figure 9 · p. 21

C.1 AUROC RESULTS

AUROC results retain the broad pattern of GATE matching or outperforming other elicitation baselines, but calibration-related variance makes the evidence harder to establish statistically. The appendix also describes LM personas and active-learning selection strategies.

C.1 AUROC RESULTS

  • The AUROC analysis measures improvement over interaction time, rewarding methods that achieve higher AUROC improvements sooner.Source: §C.1 · p. 21
  • Language-model elicitation is comparable to or better than other baselines and improves beyond no interaction, but AUROC gains are lower and more variable, especially for generative open-ended questions.Potential LM miscalibration makes a common probability threshold difficult to apply and statistical significance harder to establish.Source: §C.1 · p. 21

D.1 METHODS

  • The authors explore LMs as stand-ins for human participants by having them answer elicitation queries as natural-language personas, including biographical and moral-perspective personas.Source: §D.1 · p. 22
  • For content recommendation, pool-based active learning uses random, uncertainty-based, or diversity-based selection strategies.Uncertainty sampling selects the example with the highest entropy.Source: §D.1 · p. 22

D.2 RESULTS

In model-model experiments, GATE methods outperform both no elicitation and pool-based active learning, and these results generally track human outcomes in content recommendation and email validation but not moral reasoning. The section also compares pool-based sampling strategies and describes user-feedback measures of elicitation experience.

D.2 RESULTS

  • In model-model experiments, GATE methods beat both no elicitation and pool-based active learning on the turn-based curves.The plots track change in p(correct) and AUROC against number of turns.Source: Figure 10 · p. 23, Figure 11 · p. 23
  • Model-model results generally correlate with human results in content recommendation and email validation, but not in moral reasoning.The comparison uses turn-based measures; the authors suggest moral reasoning may be harder to simulate with a single persona prompt, even with large LMs.Source: §D.2 · pp. 22, 24, Figure 12 · p. 24
  • Random, diversity-based, and uncertainty-based pool sampling perform comparably, with diversity perhaps slightly better; the authors use it in the main human experiments.The passage notes this pattern is consistent with Margatina et al. (2023).Source: §D.2 · p. 24

E.1 METHODS

  • The usability questionnaire asks about mental demand, novelty, perceived preference coverage, and reliance on the elicitation transcript.Source: §E.1 · p. 24

7. Do you have any other feedback about the task?

  • Feedback measures include perceived coverage before and after test cases, perceived novelty, and transcript reliance.The figure captions define higher scores as greater perceived coverage, novelty, or reliance, respectively.Source: Figure 13 · p. 25, Figure 14 · p. 25

E.2 RESULTS

Participants’ perceived prompt coverage fell after they saw test cases in content recommendation and moral reasoning. Generative elicitation surfaced more novel considerations in moral reasoning and email validation, while transcript reliance was measured but adherence to the instruction about it was not validated.

E.2 RESULTS

  • Perceived coverage fell from 5.3 to 3.9 in content recommendation and from 5.4 to 4.8 in moral reasoning after participants saw test cases.Source: §E.2 · p. 25
  • Generative elicitation methods surfaced more novel considerations in moral reasoning and email validation than in content recommendation, where edge cases were less tricky and intuitive.Source: §E.2 · p. 26
  • Participants were instructed to prioritize intuition over the transcript when it conflicted, but the study could not validate how well they followed that instruction.The transcript-reliance measure records how much participants referred to elicitation history while deciding test cases.Source: §E.2 · p. 26, Figure 15 · p. 26

F LIMITATIONS

The authors limit their exploration to prompt-based GATE methods without explicit optimization of the stated objective. They also identify constraints in the human sample and the scope of the moral-reasoning domain.

F LIMITATIONS

  • The explored GATE methods are prompt-based, with no explicit optimization of the objective in Equation (2).Source: F LIMITATIONS · p. 26
  • The human study lacked enough participants to establish statistical significance over baselines in some domains, and its English-speaking U.S. sample may not capture the full spectrum of preferences.Source: F LIMITATIONS · p. 26
  • The moral-reasoning domain is simplistic and may miss preference nuances; the paper does not endorse aligning every preference and suggests safeguards for public decision systems.Source: F LIMITATIONS · p. 26

Related papers

Loading 2310.11589v1…