PaperlayerView exact revision on arXiv

Source-linked AI summary

The Art of Saying No: Contextual Noncompliance in Language Models

Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi

arXiv:2407.12043v2cs.CLcs.AIcs.HC

TL;DR

Chat-based language models should not comply with every request, but prior work has mainly focused on unsafe queries. This paper develops a broader taxonomy and COCONOT evaluation and training resources; existing models still directly comply with up to 30% of some request types, while low-rank adapter fine-tuning can improve noncompliance while maintaining general capabilities.

  • Problem

    Prior noncompliance work has mainly focused on safety, leaving broader contextual requests and how models handle them as an open evaluation and training question.

    Source: §1 · pp. 1–2

  • Method

    The authors develop a contextual noncompliance taxonomy, a prompt evaluation suite and synthetic training data, then explore training strategies including low-rank adapters.

    Source: §1 · p. 2

  • Results

    GPT-4 and Llama-3 incorrectly comply with up to 30% of incomplete and unsupported requests, while low-rank adapter fine-tuning improves noncompliance and maintains general capabilities.

    Source: §4.1 · p. 7, §5.1 · p. 9

  • Takeaways & Limitations

    The findings indicate that parameter-efficient continued fine-tuning can improve contextual noncompliance while preserving general task performance.

    Source: §1 · p. 2, §5.1 · p. 9

    Most of COCONOT is synthetically generated, and its prompts may not cover the full scope of requests within each subcategory.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Chat-based language models are designed to be helpful, yet they should not comply with every user request. While most existing work primarily focuses on refusal of "unsafe" queries, we posit that the scope of noncompliance should be broadened. We introduce a comprehensive taxonomy of contextual noncompliance describing when and how models should not comply with user requests. Our taxonomy spans a wide range of categories including incomplete, unsupported, indeterminate, and humanizing requests (in addition to unsafe requests). To test noncompliance capabilities of language models, we use this taxonomy to develop a new evaluation suite of 1000 noncompliance prompts. We find that most existing models show significantly high compliance rates in certain previously understudied categories with models like GPT-4 incorrectly complying with as many as 30% of requests. To address these gaps, we explore different training strategies using a synthetically-generated training set of requests and expected noncompliant responses. Our experiments demonstrate that while direct finetuning of instruction-tuned models can lead to both over-refusal and a decline in general capabilities, using parameter efficient methods like low rank adapters helps to strike a good balance between appropriate noncompliance and other capabilities.

1 Introduction

The paper broadens noncompliance beyond safety refusals to contextual requests that models should not answer directly. It introduces a taxonomy, evaluation set, and training strategies, finding gaps in existing models and a promising balance from parameter-efficient fine-tuning.

1 Introduction

  • The paper broadens noncompliance beyond safety to contextual nuances and out-of-scope requests.It frames two questions: how existing models perform on such requests, and how to induce appropriate noncompliance without compromising general capabilities.Source: §1 · pp. 1–2
  • GPT-4 and Llama-3 incorrectly comply with up to 30% of incomplete and unsupported requests.The authors also find significant gaps across several categories among state-of-the-art models.Source: §1 · p. 2
  • The taxonomy unifies prior work on safety, knowledge gaps, and uncertainty, with response guidance ranging from refusal to disclaimers.The evaluation includes noncompliance prompts and contrast prompts to assess potential overrefusal.Source: §1 · p. 2
  • Continued instruction-tuned model fine-tuning with low-rank adapters improves noncompliance while maintaining general capabilities.The paper reports that direct instruction tuning can cause overrefusal and reduced general capabilities, while preference tuning can reduce overrefusals.Source: §1 · p. 2

2 A Contextual Noncompliance Taxonomy

The taxonomy broadens contextual noncompliance beyond safety and organizes requests by why direct compliance may be inappropriate. Categories overlap, and their relevance can depend on a model’s intended design and use.

Taxonomy overview

  • The taxonomy has five major, overlapping categories, and model builders or evaluators may exclude categories that do not fit their intended use cases.The framework draws on content moderation, user safety and experience, and AI safety and risks.Source: §2 · p. 2

Incomplete Requests

  • Incomplete requests lack enough information; appropriate responses can identify the limitation, refuse, or answer partially within the available information.Examples include underspecification, false presuppositions, and incomprehensible queries; pointing out errors can avoid false, stereotyped, or biased answers [63].Source: §2 · pp. 2–3

Indeterminate Requests

  • Indeterminate requests include universally unknown concepts, model unknowns, and subjective questions without a single answer.Responses may flag uncertainty, refuse, approximate an answer, or summarize multiple views for subjective questions.Source: §2 · p. 3

Unsupported Requests

  • Unsupported requests exceed model design limits, including modality, context length, and temporal knowledge limits.Temporal noncompliance applies only when a model lacks real-time access; internet-connected retrieval models may comply.Source: §2 · p. 3

Humanizing Requests

  • Humanizing requests anthropomorphize models, but whether to refuse them depends on the use case.Role-playing systems may be designed for humanoid personas, whereas such responses can be harmful for general-purpose chat models.Source: §2 · p. 4

Requests with Safety Concerns

  • Safety-concern requests risk downstream harm, including offensive language, dangerous content, privacy violations, and copyright violations.The taxonomy also includes false-information requests; the stated response is complete refusal, with explanations where useful.Source: §2 · p. 4

Altering model behavior

  • Altering-model-behavior requests are jailbreak attempts that can apply across categories and require special treatment.Because strategies are model-dependent and evolving, the authors exclude this category from evaluation and training experiments.Source: §2 · p. 4

3 COCONOT: A Noncompliance Training and Evaluation Resource

COCONOT combines noncompliance prompts, contrast prompts, and response data to evaluate and train contextual noncompliance. The authors describe how they construct its datasets and assess model compliance, including human checks of the automated evaluator.

3 COCONOT: A Noncompliance Training and Evaluation Resource

  • COCONOT contains noncompliance queries and contrast queries, each divided into human-verified evaluation data and training data with responses.The evaluation split assesses models, while the training split is used to enhance noncompliance.Source: §3 · p. 4

3.1 Collecting Noncompliance Queries and Responses

  • The authors seed subcategories with handcrafted examples, iteratively generate queries with language models, then filter and manually verify evaluation prompts.They retain about 11K prompt-response pairs for supervised fine-tuning and a manually reviewed evaluation set of 1,000 requests.Source: §3.1 · pp. 4–5

3.2 Collecting Contrast Sets for Measuring and Mitigating Exaggerated Noncompliance

  • Contrast prompts cover selected incomplete, unsupported, and safety categories; preference data pairs compliant responses as preferred over noncompliant ones.The contrast evaluation set contains 379 queries, and the preference dataset contains 927 training instances.Source: §3.2 · p. 5

3.3 Evaluating Contextual Noncompliance

  • GPT-3.5 judges compliance using subcategory-specific criteria because appropriate noncompliance takes forms that surface-level evaluation can miss.On a manual check of 300 outputs, a majority of annotators verified 93% of GPT judgments as accurate, with 63% Fleiss Kappa inter-annotator agreement.Source: §3.3 · p. 6

4 Benchmarking models with COCONOT

The authors benchmark instruction-tuned proprietary and open-source models on COCONOT, which tests noncompliance across categories and contrast prompts. Most models comply too often on several request types, and adding a system prompt has mixed effects, including overrefusal.

4 Benchmarking models with COCONOT

4.1 Results and Findings

  • System prompts improve results most for safety and humanizing requests, but have mixed effects elsewhere and reduce contrast-set compliance for several models.The authors therefore report remaining results without a system prompt.Source: §4.1 · p. 7, Table 2 · p. 7
  • Indeterminate requests receive the lowest compliance overall, while safety requests receive low compliance except for Mistral and Mixtral, which perform poorest.GPT-4 has the highest compliance on indeterminate prompts and sometimes answers subjective questions directly.Source: §4.1 · p. 7
  • Incomplete and unsupported requests have the highest compliance, reaching 30% for GPT-4, Claude, Mixtral, and Llama-3 70B.Models often assume intent or answer despite false presuppositions, and may substitute an alternative for acknowledging modality limitations.Source: §4.1 · p. 7
  • Open-source models including Llama-2, Llama-3 70B, Mistral, Mixtral, and Gemma have relatively higher compliance on humanizing requests than GPT models and Claude.The passage links anthropomorphization to possible harms for socially isolated individuals and overestimation of AI intelligence.Source: §4.1 · p. 7
  • Llama-2, Llama-3, and Tulu-2 show lower overall compliance as model size increases, and preference-tuned Tulu-2 performs better overall than instruction-tuned Tulu-2.Source: §4.1 · p. 8

5 Training Strategies To Improve Noncompliance with COCONOT

The authors compare training strategies for improving contextual noncompliance while retaining general capabilities. LoRA offers a balance, while preference tuning on contrast data improves compliance with benign prompts and larger training subsets can worsen contrast-set behavior.

5 Training Strategies To Improve Noncompliance with COCONOT

  • The experiments compare four strategies: SFT from scratch, continued SFT, LoRA continued finetuning, and DPO using COCONOT contrast prompts.Training experiments use Llama-2 7B as the base model; evaluation includes COCONOT, safety benchmarks, and general-capability benchmarks.Source: §5 · pp. 8–9

5.1 Results and Findings

  • SFT from scratch improves noncompliance with minimal general-capability decline, but reduces compliance on XSTB and COCONOT-Contrast, suggesting overfitting to benign requests.It is also computationally inefficient and requires access to original instruction-following data.Source: §5.1 · p. 9
  • LoRA significantly improves noncompliance across the board while maintaining general task performance; merging an adapter with Tulu-2 performs better than either LoRA-tuned model and exceeds GPT-4 compliance rates on COCONOT.The improvements are less drastic than training from scratch, but LoRA performs better on contrastive test sets.Source: §5.1 · p. 9
  • GPT-4 performs well on safety metrics but still has limitations on COCONOT’s broader range of requests that should not be complied with.Source: §5.1 · p. 9
  • DPO on contrast training data improves compliance on contrast sets while maintaining other metrics, resulting in overall superior performance.Source: §5.1 · p. 9
  • Using more COCONOT training data almost consistently improves noncompliance in some categories, but not incomplete, unsupported, and safety requests uniformly, and raises contrast-set compliance.The comparison uses 10%, 25%, 50%, 75%, and 100% of 11,477 training instances.Source: §5.1 · p. 9, Figure 4 · p. 9

6 Related Work

Prior work on noncompliance spans safety, unanswerable or ambiguous requests, and uncertainty in language models. This paper situates its work as broadening the safety-centered focus and distinguishes its evaluation of model self-identification from calibration research.

LM Safety

  • Safety research has developed taxonomies, evaluation frameworks, and training methods for refusing unsafe queries, while this work extends noncompliance beyond safety to user experience and trust.Source: §6 · p. 10

Incomplete, Unsupported, and Indeterminate Requests

  • Work on incomplete, unsupported, and indeterminate requests draws on unanswerable-query datasets, ambiguity research, uncertainty quantification, abstention, and clarification questions.Source: §6 · p. 10

Epistemology and Language Models

  • Unlike work on uncertainty types or benchmark calibration, this paper measures whether language models identify cases where they cannot express a justified belief.The authors argue that performance on selected COCONOT subsets is necessary but not sufficient for epistemic responsibility.Source: §6 · p. 10

7 Conclusion and Future Work

The paper broadens contextual noncompliance beyond safety, introduces COCONOT for evaluation and training, and finds LoRA continued finetuning can improve noncompliance while maintaining general capabilities. It identifies open questions about epistemic awareness, jailbreak robustness, and LoRA’s role in addressing catastrophic forgetting.

7 Conclusion and Future Work

  • LoRA continued finetuning can improve noncompliance performance while maintaining general capabilities.The paper presents this as a helpful approach, not a settled solution.Source: §7 · p. 10
  • COCONOT combines a contextual noncompliance evaluation benchmark with synthetically generated training data.The taxonomy covers scenarios beyond safety alone.Source: §7 · p. 10
  • Open questions include using models’ epistemic awareness, robustness to jailbreaking, and whether LoRA can address catastrophic forgetting.The paper also calls for further work on user experience and trust.Source: §7 · p. 10

Limitations

COCONOT’s synthetic data, limited coverage, and US-specific legality norms constrain how broadly its findings and prescriptions can be applied.

Limitations

  • Most COCONOT prompts and responses are synthetic and may be noisy, although the evaluation sets are manually validated.The stated exception is a specific subset.Source: Limitations · p. 10
  • The dataset may not cover the full range of requests within its taxonomy’s subcategories.The authors note that some subcategories have extremely broad scopes.Source: Limitations · p. 10
  • The legality definition for some dangerous or sensitive-topic requests is US-specific and may reflect a western-centric bias.The paper prescribes noncompliance for these requests because outputs may lead to illegal activities.Source: Limitations · p. 10

Ethical Considerations

The authors flag risks in synthetic training data, the limits of safety gains, and potential misuse of COCONOT’s release.

Ethical Considerations

  • Synthetic training prompts and responses may include harmful requests paired with compliant responses despite filtering heuristics.The authors consider this proportion likely small because training with the data improves downstream noncompliance.Source: Ethical Considerations · p. 11
  • Training for noncompliance does not guarantee complete safety, so real-world deployment needs additional precautions.Source: Ethical Considerations · p. 11
  • The authors note that malicious players could use the dataset’s safety-related requests and poor model compliance rates to build models that instigate harm.They plan to gate release behind a content warning and terms agreement limiting usage to research and noncompliance improvement.Source: Ethical Considerations · p. 11

A.1 Noncompliance Data Generation

The appendices describe COCONOT’s data-generation and contrast-set procedures, then specify the general-capability and safety benchmarks used in evaluation.

A.1 Noncompliance Data Generation

  • GPT models generate prompts from seed examples, while GPT-4 generates labeled noncompliant responses that are filtered and deduplicated.The prompt requests realistic, specific examples for a named subcategory; the seed set is provided in supplementary material.Source: §A.1 · pp. 22, 25

A.2 Contrastive Data Generation

  • For incomplete requests, the contrast set uses crowdworker edits for underspecified questions and templates that make false presuppositions true.The underspecified examples draw on SituatedQA edits.Source: §A.2 · p. 28
  • For modality limitations and safety concerns, GPT-4 generates contrastive queries; author checks found 93 of 102 inspected modality examples valid.Eight of the nine invalid modality examples involved translation and were generally easy to filter.Source: §A.2 · p. 28
  • COCONOT-PREF contains 927 preference instances pairing a GPT-4 compliant response with a randomly sampled noncompliant response from open-source models.The candidate noncompliant responses come from Llama 2 and Tulu 2 models and are filtered with heuristics.Source: §A.2 · p. 28

B.1.1 General Capabilities

  • The general-capability suite also includes TydiQA, Codex-Eval, TruthfulQA, and AlpacaEval, using benchmark-specific metrics such as F1, pass@10, and GPT-4 preferences.AlpacaEval V2 with length control is added to the Open-Instruct evaluation suite.Source: §B.1.1 · p. 32
  • General-capability evaluations include MMLU, GSM8k, and BBH, with accuracy reported for each benchmark.BBH comprises 23 challenging multiple-choice or exact-match tasks; GSM8k uses chain-of-thought evaluation with eight few-shot examples.Source: §B.1.1 · p. 32

B.1.2 Safety Capabilities

  • Safety evaluations measure toxic outputs on ToxiGen, compliance on XSTest, and attack success rate on HarmBench DirectRequest.XSTest includes safe prompts resembling unsafe prompts as well as unsafe prompts.Source: §B.1.2 · p. 32

C.1 Benchmarking models with COCONOT

This appendix supplies baseline and fine-grained COCONOT results alongside implementation details for supervised fine-tuning, LoRA, and DPO.

C.1 Benchmarking models with COCONOT

  • Baseline COCONOT compliance rates are reported in Tables 10–14, with Table 2 visualized in Figures 12–14.Source: §C.1 · p. 33

D.1 Training Details

  • Supervised fine-tuning uses 2 epochs (1 for continued fine-tuning), a learning rate of 2e−5, and an effective batch size of 128.Source: §D.1 · p. 33
  • LoRA uses one epoch, a learning rate of 1e−5, rank 64, alpha 16, and dropout 0.1; DPO uses one epoch at 5e−7.Source: §D.1 · p. 33

D.2 Fine-grained Training Results

  • Fine-grained training results across all COCONOT subcategories are reported in Tables 17–20.Source: §D.2 · p. 33

D.3 Complete Training Results on all Benchmarks

The appendix reports complete benchmark results, where continued full-parameter Tulu fine-tuning reduces general capabilities and exaggerates safety behavior on contrast sets; repeated-run error bars are omitted for computational reasons.

D.3 Complete Training Results on all Benchmarks

  • Continued full-parameter Tulu fine-tuning significantly reduces general capabilities and exaggerates safety behaviors on contrast sets; Tulu2Mix helps slightly but does not restore original performance.Source: §D.3 · p. 33, Table 15 · p. 39, Table 16 · p. 40

NeurIPS Paper Checklist

  • No error bars are reported because repeated experiments were computationally prohibitive; greedy decoding makes generation deterministic.Source: NeurIPS Paper Checklist · p. 43
Loading 2407.12043v2…