PaperlayerView exact revision on arXiv

Source-linked AI summary

ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations

Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, Chandra Bhagavatula

arXiv:2212.10409v3cs.CL

TL;DR

Delphi makes moral judgments for simple actions without surrounding context, even though context can change those judgments. CLARIFYDELPHI uses reinforcement learning to ask questions whose hypothetical answers yield divergent moral judgments; human evaluation finds its questions more relevant, informative, and defeasible than baseline questions.

  • Problem

    Delphi produces moral judgments for simple actions in isolation, without surrounding context, although context matters to moral reasoning.

    Source: §1 · p. 1

  • Method

    CLARIFYDELPHI uses reinforcement learning to generate questions whose hypothetical answers produce maximally divergent judgments, using Delphi for feedback.

    Source: §9 · p. 9

  • Results

    Human evaluation found CLARIFYDELPHI's questions more relevant and informative than baseline questions and more often answerable with strengtheners or weakeners.

    Source: §6.2 · p. 6

  • Takeaways & Limitations

    The authors conclude that the questions can provide more disambiguating context through interaction.

    Source: §9 · p. 9

    The defeasibility measure relies on Delphi judgments, which the authors note can be culturally limited and inconsistent.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Context is everything, even in commonsense moral reasoning. Changing contexts can flip the moral judgment of an action; "Lying to a friend" is wrong in general, but may be morally acceptable if it is intended to protect their life. We present ClarifyDelphi, an interactive system that learns to ask clarification questions (e.g., why did you lie to your friend?) in order to elicit additional salient contexts of a social or moral situation. We posit that questions whose potential answers lead to diverging moral judgments are the most informative. Thus, we propose a reinforcement learning framework with a defeasibility reward that aims to maximize the divergence between moral judgments of hypothetical answers to a question. Human evaluation demonstrates that our system generates more relevant, informative and defeasible questions compared to competitive baselines. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cognition (i.e., the diverse contexts in which moral rules can be bent), and we hope that research in this direction can assist both cognitive and computational investigations of moral judgments.

1 Introduction

CLARIFYDELPHI addresses the context missing from isolated moral judgments by asking questions whose answers may change the judgment. It frames this as a task of finding consequential context and trains question generation to favor divergent hypothetical answers.

1 Introduction

  • Moral judgments depend on context, while Delphi produces judgments for simple actions without knowledge of their surrounding context.Clarification questions are proposed as a way to elicit missing salient context.Source: §1 · p. 1
  • CLARIFYDELPHI asks questions to uncover context that can strengthen or weaken a default moral judgment.The paper frames this as a first step toward computationally exploring contexts that may flip a judgment.Source: §1 · pp. 1–2
  • Reinforcement learning optimizes questions for morally salient answers by comparing Delphi's probability distributions over answers imagined by a trained answer-generation model.The approach favors questions whose possible answers lead to maximally divergent judgments.Source: §1 · p. 2
  • The approach outperforms strong clarification-question baselines, and its questions lead to consequential answers.The introduction also reports that questions help generate defeasible updates.Source: §1 · p. 2
  • The paper introduces clarification-question generation for social and moral situations and releases δ-CLARIFY, a dataset of 33k crowdsourced questions, plus a silver dataset and trained models with code.The silver dataset contains generated questions conditioned on a defeasible inference dataset.Source: §1 · p. 2

2 Problem Setup

  • The task predicts a question from a base situation with a default judgment, seeking answers that can strengthen or weaken that judgment.The paper uses weakener and strengthener in the sense of defeasible inference, where new evidence can support or cancel an initial inference.Source: §2 · p. 2

3 CLARIFYDELPHI: A Reinforced Clarification Question Generator

CLARIFYDELPHI trains a question generator to ask about socially or morally salient context, then uses simulated weakening and strengthening answers to reward questions that produce divergent Delphi judgments. PPO optimizes the question policy using that reward.

3 CLARIFYDELPHI: A Reinforced Clarification Question Generator

  • The system combines question and defeasible-answer generation models with Delphi feedback, so reinforcement-learning training requires no supervised data.The question model proposes questions, the answer model supplies defeasible answers, and Delphi calculates rewards.Source: §3 · p. 3

3.1 Supervised Question Generation

  • A supervised question generator θQ predicts a question from a situation and serves as the initial policy for reinforcement learning.Source: §3.1 · p. 3

3.2 Defeasible Answer Simulation

  • The answer generator takes a situation, question, and update type to produce weakening or strengthening answers; an NLI model filters answers that contradict or merely repeat the situation.Source: §3.2 · p. 3

3.3 Reward

  • The reward is the Jensen-Shannon divergence between Delphi’s probability distributions for situations updated with weakening versus strengthening answers.The situation, question, and answer are fused into an updated situation before Delphi is queried.Source: §3.3 · pp. 3–4

3.4 Proximal Policy Optimization (PPO)

  • PPO maximizes the reward while using a KL penalty between the initial and updated policies and a value model to estimate values for partially decoded questions.Source: §3.4 · p. 4

4 δ-CLARIFY: a Dataset of Clarification Question

δ-CLARIFY combines crowdsourced clarification questions with GPT-3-generated questions paired with defeasible updates. Its analysis finds varied question formulations across annotators and notes that “why” is frequent among silver questions capable of both update types.

δ-CLARIFYgold

  • Each of 6,425 situations was shown to five annotators, yielding five crowdsourced clarification questions per situation.The situations were sampled from SOCIAL-CHEM-101 and the COMMONSENSE NORM BANK.Source: §4 · pp. 4–5

δ-CLARIFYsilver

  • The silver dataset uses GPT-3 to generate questions conditioned on defeasible situations and updates, yielding approximately 80K situation-update-question-answer instances.The source updates weaken or strengthen default judgments and can be treated as answers to implicit questions.Source: §4 · p. 5

Dataset Analysis

  • Only 10% of situations had repeated identical questions and only 8% had all five annotators use the same starting Wh-word, indicating varied formulations.Among defeasible silver questions, “why” is the most frequent start, suggesting it is easiest to generate both weakening and strengthening answers.Source: §4 · p. 5

5 Baselines

The baselines include supervised question generation without reinforcement learning, question-selection pipelines, and a why-question baseline. Selection pipelines generate varied candidates and choose one using either a relevance discriminator or divergence ranking.

Question Generation Without RL

  • The t5 fine-tuned baseline evaluates the supervised question-generation model without reinforcement learning, using nucleus sampling with top-p = 0.6.Source: §5 · p. 5

Pipelines with Question Selection

  • Question-selection pipelines generate candidates conditioned on different question starts, then select a question by a scoring method.Inference generates questions for 14 different starts.Source: §5 · p. 5

Discriminator

  • The discriminator pipeline selects the candidate with the highest probability of being relevant, using a DeBERTa binary classifier trained on annotator questions as positives and questions from other situations as negatives.Negative examples are chosen to be similar enough to positives to make classification more challenging.Source: §5 · pp. 5–6

Divergence Ranking

  • Divergence ranking scores questions using the same simulated-answer and Delphi-feedback process as the reinforcement-learning reward, with variants that do or do not apply NLI filtering.Source: §5 · p. 6

Why-Baseline

  • The why baseline generates one why-question per situation using the pipeline baseline’s input/output configuration.Source: §5 · p. 6

6 Evaluation and Analysis

Human evaluation finds that CLARIFYDELPHI produces especially relevant, informative, and defeasible questions; further analyses examine supervision needs, answer generation, and Delphi’s judgments.

6.1 Human Evaluation

  • The human study rates grammaticality, relevance, and informativeness, then tests whether answers can support or flip an initial moral judgment.Questions rated lowest by at least two annotators on relevance or informativeness are excluded from the defeasibility evaluation.Source: §6.1 · p. 6, §6.2 · p. 6

6.2 Results of Human Evaluation

  • CLARIFYDELPHI yields the largest share of test questions rated positively for both relevance and informativeness; even the discriminator passes 448/500 questions to the next evaluation.Figure 4 reports counts out of 500 receiving positive informativeness and relevance ratings.Source: Figure 4 · p. 6, §6.2 · p. 6
  • Annotators more often identify strengthener and/or weakener answers for CLARIFYDELPHI than for baselines, and NLI answer filtering improves selection on all four evaluation criteria.Table 2 reports human defeasibility scores; the why-baseline is also strong, highlighting the relevance of motives and reasons.Source: §6.2 · p. 6, Table 2 · p. 7

6.3 How much supervision does the policy require?

  • About 5,000 training situations (75% of δ-CLARIFY) suffice for a good initial policy, with minimal gains beyond 75%; SQuAD-pretrained policies generate the most uninformative questions.The assessment uses non-answerable-question percentage and maximum BERTScore; the latter also supports the reported trend.Source: §6.3 · p. 7, Figure 5 · p. 7

6.4 Analysis

  • Answer simulation produces diverse weakener and strengthener answers: only about 0.05% of questions per 1,000 PPO epochs receive the same answer for both update types.Source: §6.4 · p. 7
  • Generating a question before a defeasible update improves update generation on δ-SOCIAL compared with generating updates directly.Table 3 reports BLEU-4 and ROUGE-L for strengthener and weakener updates, with its first two results from Rudinger et al. (2020).Source: §6.4 · p. 7, Table 3 · p. 8
  • Generated questions include specification and implicit-argument queries; 60% of situations containing “something” elicit a what-question, while why-questions often ask about motives.The SQuAD 2.0 QA model may classify specification questions as answerable given the situation.Source: §6.4 · p. 8
  • Pipeline_NLI has higher Delphi-answer JSD and judgment-flip rates than CLARIFYDELPHI, while CLARIFYDELPHI exceeds fine-tuned T5 and other non-pipeline baselines; the mismatch flags possible Delphi miscalibration.The analysis cautions that relying too heavily on Delphi’s probabilities may potentially propagate errors.Source: §6.4 · p. 8, Table 4 · p. 8

7 Interactive Judgements

At inference, the system takes a situation as input and asks questions to elicit user-provided context, updating Delphi’s decision after each answer.

  • Interaction is limited to three turns because sentence fusion deteriorates afterward, while the first two questions generally capture the most central missing contexts for moral decisions.Source: §7 · p. 8

8 Related Work

Prior clarification-question work spans several domains and commonly frames question quality through relevance, informativeness, or utility. This work instead treats defeasibility as relevance for social and moral questions, alongside a large dataset and an RL-based approach.

  • Prior clarification-question generation addressed domains including image recognition and product descriptions, often measuring question quality through relevance, informativeness, or utility (Rao and Daumé III, 2018; White et al., 2021).Source: §8 · p. 8
  • Unlike work focused on questions with a single true answer, this task defines relevance and utility through defeasibility and contributes a dataset of more than 30K social and moral questions.The passage contrasts δ-CLARIFY with domain-specific clarification datasets.Source: §8 · p. 8
  • Earlier RL question-generation approaches optimized QA answer F1 or binary utility, whereas this approach uses PPO with a trained feedback model (Buck et al., 2018; Rao and Daumé III, 2019).Source: §8 · p. 9
  • Delphi predicts ethical judgments and is based on pre-trained UNICORN, while COMMONSENSE NORM BANK draws on five existing datasets.The cited source describes COMMONSENSE NORM BANK as containing 1.7M instances of descriptive everyday ethical knowledge (Jiang et al., 2022).Source: §8 · p. 9

9 Conclusion

The conclusion presents ClarifyDelphi’s reinforcement-learning approach as outperforming clarification-question baselines and proposes interaction as a way to provide more disambiguating context. It also identifies limits in cultural representation and Delphi’s judgments.

9 Conclusion

  • The reinforcement-learning approach generates questions that maximize answer divergence in defeasibility and outperforms other clarification-question baselines.The authors suggest the questions may provide more disambiguating context through interaction.Source: §9 · p. 9

Limitations

  • Because most source-data and δ-CLARIFY crowdworkers were in the United States, predictions currently represent Western perspectives, particularly U.S. culture.Overcoming this Western-centric bias is identified as future work.Source: Limitations · p. 9
  • Defeasibility is measured using Delphi’s judgments, which the authors note can have limited cultural awareness and inconsistent predictions.They identify improved methods for detecting answer divergences as future work.Source: Limitations · p. 9

Ethics Statement

  • Crowdsourcing tasks were limited to workers’ general knowledge, collected no personal information, and received an internal IRB exemption.The authors report an average pay rate of $15 per hour for crowdsourcing data collection and evaluation tasks.Source: Ethics Statement · p. 9
  • The model is intended for research use only, not for advice applicable outside that domain.Source: Ethics Statement · p. 9

A Appendix

The appendix describes question collection and human evaluation, answer prompting, and PPO training details. It also reports that crowdsourcing was English-language and U.S.-based.

A.1 Crowdsourcing and Annotation

  • Crowdworkers wrote five clarification questions for each of 6,425 situations; 145 manually vetted Turkers worked on the HITs.The authors report an average pay rate of $15 per hour for their crowdsourcing tasks.Source: §A.1 · p. 11
  • Human evaluation rated grammaticality, relevance, and informativeness, with defeasibility evaluated as an additional attribute.Source: §A.1 · pp. 11–12
  • Crowdsourcing was conducted in English with workers located in the U.S., centering consequential contexts on Western or U.S. English-speaking views.The appendix also reports an internal IRB exemption.Source: §A.1 · p. 12

A.2 Prompting for Answer Generation

  • GPT-3 was prompted to produce “good” and “bad” answers, whose weakener or strengthener roles were determined by comparing Delphi’s judgments after adding each answer.Source: §A.2 · p. 12

A.3 Details of PPO

  • PPO uses a truncated estimated-advantage function and clipped surrogate objective, while the value model is trained with MSE against the target value.The value function estimates states; intermediate rewards and reward-decay factors are used in the advantage calculation.Source: §A.3 · p. 12

A.4 Implementation Details

The implementation uses T5-large models for question and answer generation and trains the question policy with PPO. The appendix also points to a table of generated-question examples.

A.4 Implementation Details

  • The question generator is T5-large trained on δ-CLARIFY; the T5-large answer model uses 77k δ-CLARIFYsilver and about 4k GPT-3-prompted instances, with WaNLI for NLI filtering.Source: §A.4 · p. 12
  • PPO runs for 6,000 steps with batch size 64, taking about two days on four A6000 GPUs; the models have 770 million parameters, and Delphi uses T5-large.Source: §A.4 · p. 12

A.6 Sample of Generated Questions

  • The paper shows a sample of 15 generated questions in Table 5.Source: §A.6 · p. 12

A.7 Sample of Generated Questions in a Compositional Setting

This appendix material presents generated-question examples, including a two-turn interaction, and documents question-collection and evaluation interfaces and instructions.

A.7 Sample of Generated Questions in a Compositional Setting

Loading 2212.10409v3…