Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling

1.1 BACKGROUND

• Open-domain end-to-end conversational models learn responses from dialog data without an interim semantic representation, enabling training on large datasets without semantic annotation.They may be pretrained on freely available conversational data and fine-tuned on smaller, curated datasets.Source: §1.1 · pp. 1–2

1.2 PROBLEM DEFINITION

• The INSTIGATOR (TAY) EFFECT is harmful content generated by a system, while the YEA-SAYER (ELIZA) EFFECT is inappropriate agreement with harmful user content and the IMPOSTOR EFFECT is inappropriate expert advice in safety-critical situations.The effects have different harm profiles: the first may be subjective or group-specific, the second may depend on cultural norms, and the third can involve objectively measurable physical harm.Source: §1.2 · pp. 2–3

1.3 WHY DOES THIS HAPPEN?

• Safety concerns may arise from training data, modeling choices, limited control of generation, and evaluation objectives that do not directly account for real-world impacts.Conversational systems may also have limited understanding of utterance meaning and function in context.Source: §1.3 · pp. 3–4

1.4 WHY IS THIS CHALLENGING?

• Release decisions are difficult because safety and downstream impacts are uncertain, and values can conflict or change over time.The paper notes that safety itself is ill-defined and that consequences may emerge years after release.Source: §1.4 · p. 4

1.5 GOING FORWARD: THIS PAPER

• The paper offers a value-sensitive analytical framework and preliminary tools to help researchers assess potential harms and make informed release decisions, rather than fixing underlying data or model problems.Its checks are initial sanity tests that can reveal particular errors but cannot prove that errors are absent.Source: §1.5 · pp. 4–5

2 PROBLEM LANDSCAPE

• The section defines three categories of harmful conversational-agent responses based on the safety issues summarized in Table 1.It uses those categories to structure its definitions and related-work discussion.Source: §2 · p. 5

2.4 OTHER CONSIDERATIONS

• Other considerations include trust, sensitive topics, and fairness, while performance bias, privacy leaks, and environmental harm are outside the work’s scope.The paper notes that topic sensitivity can depend on audience, culture, and local law; it also discusses risks from training-data leakage and energy use.Source: §3 · p. 5, §2.4 · pp. 8–9

2.1 GENERATING OFFENSIVE CONTENT (INSTIGATOR EFFECT)

• Prior work addresses offensive content in generated responses through detection, adversarial data, training-data changes, generation controls, and response strategies.In Xu et al. (2020), a two-stage classifier that returns a canned response when it detects offensive user or system content was most successful among the tested strategies.Source: §2.1 · pp. 5–6

2.2 RESPONDING INAPPROPRIATELY TO OFFENSIVE CONTENT (YEA-SAYER EFFECT)

• A response can avoid repeated user offenses yet still implicitly condone harmful input, so safety assessment must consider conversational context.Lee et al. (2019) found chatbot responses consistent with agreement with known social biases; Dinan et al. (2019b) and Xu et al. (2020) studied offensive-utterance detection in multi-turn conversations.Source: §2.2 · pp. 6–7

2.3 RESPONDING INAPPROPRIATELY IN SAFETY-CRITICAL SITUATIONS (IMPOSTOR EFFECT)

• Medical advice, emergencies, and self-harm are identified as safety-critical situations where inappropriate advice can have serious consequences.For self-harm, the paper notes limited conversational-AI research and the risk of a system agreeing with a user’s self-harm statements; emergency-information work emphasizes human input to mitigate false information.Source: §2.3 · pp. 7–8

3 TENSIONS BETWEEN VALUES, POTENTIAL POSITIVE IMPACT, AND POTENTIAL HARM

• Because computing systems encode values whether designers attend to them or not, deliberate reflection can surface problems and opportunities early and inform release decisions.Different people, groups, and cultures may prefer different trade-offs, so the chosen balance may not be universal.Source: §3 · p. 9

3.1 HOW VALUES CONFLICT

• Trade-offs can arise from direct conflicts, competing resource priorities, differences among stakeholders, and costs or benefits separated across time.Choices that protect autonomy may conflict with protecting users from outcomes they could later regret.Source: §3.1 · pp. 9–10

3.2 VALUE-SENSITIVE DESIGN

• Value-sensitive design integrates human values throughout development, using conceptual, empirical, and technical investigations to examine how technologies support or hinder them.The paper connects this approach to a release framework and tooling for technical investigation.Source: §3.2 · p. 10

3.3 HUMAN JUDGMENTS OF RISKS, COSTS, AND BENEFITS

• People’s risk judgments can diverge from empirical estimates through cognitive heuristics, demographic differences, loss aversion, and limits on attention and choice.Information overload can lead people to skim detailed disclosures or disengage from overwhelming options.Source: §3.3 · pp. 11–12

3.4 RESILIENCE TO UNCERTAINTY AND CHANGE

• Value-sensitive design assumes trade-offs can be estimated early, but downstream use, societal change, and shifting values make those estimates uncertain.The authors point to adaptable techniques and ongoing feedback as relevant responses.Source: §3.4 · p. 12

4 A FRAMEWORK FOR RESEARCHERS TO DELIBERATE MODEL RELEASE

• The framework supports release deliberation and learning during and after release; it is guidance rather than a prescription and is not intended to block beneficial research except in extreme circumstances.The framework is situated within value-sensitive design and aims to help researchers direct efforts toward minimizing potential harms.Source: §4 · p. 12

4.1 INTENDED USE

• Researchers should state intended uses early, revisit whether intentions drift, and consider unintended uses that may expose vulnerabilities or inform release policies.They should also ask whether the proposed release form is necessary to fulfill the intended use.Source: §4.1 · pp. 13–14

4.2 AUDIENCE

• Researchers should consider intended and unintended audiences, including language, audience size, and familiarity with machine learning, because these factors shape release consequences and policy choices.Different languages and cultures can evaluate conversational concepts such as politeness differently.Source: §4.2 · p. 14

4.3 ENVISION IMPACT

• Researchers should envision both benefits and harms early to guide tests, mitigation, and protective policies, while avoiding release decisions based solely on envisioned harms.The process may be uncertain and benefits from relevant community or domain expertise.Source: §4.3 · p. 14

4.4 IMPACT INVESTIGATION

• Technical investigation can measure whether a model supports or hinders prioritized values, but impact estimates may be inaccurate and testing protocols need broader coverage and standardization.Investigating benefits may be more application-dependent than investigating harms.Source: §4.4 · p. 15

4.5 WIDER VIEWPOINTS

• Perspectives from outside the immediate team, including domain experts and affected communities, can inform potential issues, biases, and misuse before release and throughout the framework.The paper suggests informal discussion, workshops, or demo dogfooding as possible engagement approaches.Source: §4.5 · p. 15

4.6 POLICIES; 4.7 TRANSPARENCY

• Policies may limit unintended access through licensing, request-based or staged releases, while transparency can communicate intended use, potential harms, and reporting options to users.Request-based access may preserve reproducibility while creating substantial review and maintenance costs; a full model card may not suit a small demo audience.Source: §4.6 · p. 15, §4.7 · p. 16

4.8 FEEDBACK TO MODEL IMPROVEMENT

• Accessible feedback channels can capture unexpected harms and inform later model improvements or adaptations to changing values.Examples include a monitored issue page for an open-source model and reporting features in a demo interface.Source: §4.8 · p. 16

5 TECHNICAL INVESTIGATION: BUILDING TOOLING FOR SAFETY CHECKS

• The tooling suite aggregates existing tools for preliminary safety investigation, distinguishing automated unit tests from human-evaluation integration tests.The tools are intended to help researchers assess how models and their release may support or hinder specific values.Source: §5 · p. 16

5.1 BENCHMARK AGENTS

• The benchmark set includes 90M and 2.7B BlenderBot variants, DialoGPT, GPT-2, and Kuki, a largely rule-based open-domain chatbot.Kuki is included for comparison despite using AIML pattern matching rather than end-to-end generation.Source: §5.1 · pp. 16–17

5.2.1 INSTIGATOR (TAY) EFFECT TESTS

• The Instigator test generates responses to deployment-like inputs and checks them with an offensive-word list, a dialog safety classifier, and optionally the PERSPECTIVE API.The four settings range from innocuous inputs to real-world noise and standard or adversarial unsafe inputs.Source: §5.2.1 · pp. 17–18
• Models generally receive more unsafe flags as inputs move from safe to noisier or unsafe settings, though adversarial prompts can reverse that trend when subtle offense escapes models or classifiers.Table 3 reports flags by input setting and tool; the authors caution that some models show a decrease from standard to adversarial unsafe inputs.Source: §5.2.1 · p. 18, Table 3 · p. 19
• The tools often disagree; on 500 Bot-Adversarial validation examples, the word list had highest precision and lowest recall, while the safety classifier had the reverse profile.The authors suggest prioritizing recall to reduce the risk of inappropriate generation, or precision when identifying user abuse.Source: §5.2.1 · pp. 19–20

5.2.2 YEA-SAYER (ELIZA) EFFECT TEST

• The Yea-Sayer test varies demographic-and-adjective prompts across three phrasings and assesses affirmation using sentiment, negation detection, and a multi-turn safety classifier.The evaluation uses the first ten negative adjectives from the source list, yielding 510 examples.Source: §5.2.2 · p. 20
• Yea-Sayer indicators conflict: at least 43% of responses had positive sentiment, while negations were common, and the classifier flagged most DialoGPT and GPT-2 responses as offensive.Responses to minimally varied prompts also differed; sentiment predictions on BST 90M agreed for as few as 28.82% of response triples.Source: §5.2.2 · pp. 20–22, Table 7 · p. 22
• Blind annotation of 75 examples found affirmation a poor indicator of appropriateness; even a non-affirming expression of confusion may be unsuitable after clearly offensive input.The authors note that judging appropriate responses may require experts and may fit an NLU task better than an NLG one.Source: §5.2.2 · p. 22

5.2.3 IMPOSTOR EFFECT TESTS

• Appropriate responses to safety-critical requests depend on deployment context, user expertise, and the emergency, and would benefit from expert guidance.The paper notes that inappropriate advice in a medical emergency could have severe consequences.Source: §5.2.3 · p. 23
• The authors report no open-source tools for detecting these situations in conversations and propose benchmarks for medical advice, self-harm, and other emergencies, paired with expert-informed canned responses.They describe the benchmark as an NLU classification task that could trigger a response tailored to the situation.Source: §5.2.3 · p. 23

5.3 SAFETY INTEGRATION TESTS

• The integration tests use adversarial Bot-Adversarial dialogs and non-adversarial dialogs from the Wikipedia Toxic Comments dataset.They build on tooling from Xu et al. (2020).Source: §5.3 · p. 24
• Human evaluators judge responses to 180 fixed contexts for whether they are appropriate in a friendly conversation, considering the dialog context.The tests can capture both direct harmful generation and inappropriate responses to harmful user content.Source: §5.3 · p. 24
• Human evaluations require manual intervention, but the existing tooling makes them straightforward to set up when researchers have the required model API access.Because instructions and user interfaces can change results, the authors recommend using these evaluations to compare with prior work.Source: §5.3 · p. 24

5.4 LIMITATIONS

• The authors recommend using the tools only as a preliminary step toward considering a model’s ethical and social consequences.The tools are designed to run quickly and easily, which limits their scope; some use cases may require more robust testing.Source: §5.4 · pp. 24–25
• The tests rely on English-language data largely annotated in the United States, so they cannot adequately measure response appropriateness in other languages and locales.Offensiveness depends on culture, and the authors note that more research on cultural differences is needed.Source: §5.4 · p. 24
• Automatic classifiers can be inaccurate and biased, including through associations between African American English markers and toxicity in hate-speech datasets.The cited work reports that models trained on such datasets may label tweets by self-identified African Americans as offensive up to two times more often; filtering words such as “twink” may also limit LGBTQ+ discourse.Source: §5.4 · pp. 24–25
• Appropriateness is often impossible to determine definitively, and even high-agreement examples may be mishandled by existing classifiers.Inter-annotator agreement on offensive-language labeling is typically low.Source: §5.4 · p. 24
• Most tools are static or trained on static data, so they do not account for changes in a word’s cultural meaning or sentiment.The passage gives “coronavirus” as an example of a word whose meaning or sentiment can change.Source: §5.4 · p. 25
• Crowworkers may differ substantially from a deployed model’s intended audience, so tests should approximate that specific audience as well as possible.Source: §5.4 · p. 25

5.5 RECOMMENDED USE

• Use the tools as a first pass for an English-language dialog model; the use case may call for further considerations.Source: §5.5 · p. 25

6 DISCUSSION AND FUTURE WORK

• The paper frames release decisions around conflicting, uncertain, and changing values, and offers a framework and preliminary tools following value-sensitive design.It also surveys work on the Instigator, Yea-Sayer, and Impostor effects.Source: §6 · p. 25

6.1 NATURAL LANGUAGE UNDERSTANDING

• Improving language understanding, especially of social meaning, may help address safety issues and strengthen classifiers; adding dialog history, multimodal, or user-specific context are proposed routes.Adversarial inputs using subtle offensive language can be misunderstood by models.Source: §6.1 · pp. 25–26

6.2 RAPIDLY ADAPTABLE TECHNIQUES

• Fine-tuning, few-shot learning, and inference-time control are discussed as ways to adapt model behavior; retrieval can provide changing information and additional context or constraints.The authors also suggest that engaging content or systems that take initiative might steer users away from safety-critical situations.Source: §6.2 · p. 26

6.3 EVALUATION BENCHMARKS

• Evaluation should probe whether models rely on artifacts or spurious correlations, and benchmarks should be revisited as facts and moral standards change.The paper points to evolving benchmarks and adversarial iterative procedures as ways to adapt evaluation.Source: §6.3 · pp. 26–27

6.4 LIFE-LONG LEARNING

• Because values and context change over time, the authors suggest flexible learning frameworks such as lifelong or online learning for models that can represent value change.They note that current LLMs are mostly static and unable to represent value change.Source: §6.4 · p. 27

• The partial BlenderBot 90M unit-test output includes the log location and information for interpreting results.Source: Figure 1 · p. 43