PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting

Rui Wang, Hongru Wang, Fei Mi, Yi Chen, Boyang Xue, Kam-Fai Wong, Ruifeng Xu

arXiv:2305.13733v2cs.CL

TL;DR

LLMs may accept instructions built on false premises, reinforcing misconceptions or spreading misleading information, a challenge that has received limited attention. The paper introduces INDUST to test this behavior and proposes DUAL-CRITIQUE prompting, which improves performance across several LLMs and settings.

  • Problem

    LLMs may accept false-premise instructions, potentially reinforcing users’ false beliefs or spreading misleading information, and this scenario has rarely been studied.

    Source: §1 · p. 1

  • Method

    The paper introduces INDUST, a benchmark of false-premise instructions, and DUAL-CRITIQUE prompting, which asks models to critique user instructions and their own responses.

    Source: §1 · pp. 1–2, §8 · p. 9

  • Results

    DUAL-CRITIQUE consistently improves several LLMs in zero-shot and few-shot settings, while INDUST evaluations find that most models can be tricked and particularly struggle with multiple false premises.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DUAL-CRITIQUE offers a training-free prompting approach that improves robustness across inductive styles and premise complexity.

    Source: §1 · p. 2, §8 · p. 9

    Although INDUST includes varied inductive styles and instances, it cannot fully capture real-world complexities.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Numerous works are proposed to align large language models (LLMs) with human intents to better fulfill instructions, ensuring they are trustful and helpful. Nevertheless, some human instructions are often malicious or misleading and following them will lead to untruthful and unsafe responses. Previous work rarely focused on understanding how LLMs manage instructions based on counterfactual premises, referred to here as \textit{inductive instructions}, which may stem from users' false beliefs or malicious intents. In this paper, we aim to reveal the behaviors of LLMs towards \textit{inductive instructions} and enhance their truthfulness and helpfulness accordingly. Specifically, we first introduce a benchmark of \underline{\textbf{Indu}}ctive {In\underline{\textbf{st}}ruct}ions (\textsc{\textbf{INDust}}), where the false knowledge is incorporated into instructions in multiple different styles. After extensive human and automatic evaluations, we uncovered a universal vulnerability among LLMs in processing inductive instructions. Additionally, we identified that different inductive styles affect the models' ability to identify the same underlying errors, and the complexity of the underlying assumptions also influences the model's performance. Motivated by these results, we propose \textsc{Dual-critique} prompting to improve LLM robustness against inductive instructions. Our experiments demonstrate that \textsc{Dual-critique} prompting significantly bolsters the robustness of a diverse array of LLMs, even when confronted with varying degrees of inductive instruction complexity and differing inductive styles.

1 Introduction

Instructions containing false premises can reinforce misconceptions or spread misleading information, yet this challenge has received little attention. The paper introduces INDUST to study this vulnerability and proposes DUAL-CRITIQUE prompting to improve truthfulness and helpfulness.

1 Introduction

  • Instructions with counterfactual premises may reflect misconceptions or deliberate deceit; accepting them can reinforce false beliefs or spread misleading information.The paper argues that models should identify and address false premises rather than accept them unquestioningly.Source: §1 · p. 1
  • INDUST tests LLM responses to false-premise instructions across Fact-Checking, False-Premise Question, and Creative False-Premise styles, with single- and multiple-premise subsets.The benchmark builds on an existing fact-checking dataset and presents the same false knowledge in different instruction styles.Source: §1 · pp. 1–2
  • Most tested LLMs generated misleading responses, varied in performance across inductive styles, struggled especially with multiple false premises, and rarely corrected premises proactively.The paper evaluates truthfulness and helpfulness through human and automated evaluation.Source: §1 · p. 2
  • DUAL-CRITIQUE prompts models to critique user instructions for false information and themselves for response accuracy and precision.The paper examines single-step and two-step variants.Source: §1 · p. 2
  • DUAL-CRITIQUE consistently improves several LLMs in zero-shot and few-shot settings without additional training.The prompting method is presented as flexible and applicable across a variety of scenarios.Source: §1 · p. 2

2 Catagories of Inductive Instructions

The paper groups inductive instructions by how users frame false information, from uncertainty about a claim to confidently building on it. It distinguishes three instruction types and also separates single- from multiple-premise cases.

2 Catagories of Inductive Instructions

  • The categories represent increasing user confidence in the underlying fallacy, from doubt in Fact-Checking Instructions to believing and building on it in QFP and CIFP.The paper presents the categories as spanning least to most certainty about the fallacy.Source: §2 · pp. 2–3
  • Multiple-premise instructions contain more than one false premise, denoted |T| > 1.The underlying false information may arise from misunderstanding or intentional deceit.Source: §2 · p. 3
  • Fact-Checking Instructions ask whether a claim is true, while Questions based on False Premises request explanations that assume false information is accurate.The paper describes false-premise questions as more challenging because they require generating information under false assumptions.Source: §2 · p. 3
  • Creative Instructions based on False Premises ask for original content built on a fallacy, potentially drawing attention from fact-checking toward imaginative generation.Requested outputs may include stories or reports.Source: §2 · p. 3

3 Data Collection

INDUST is assembled by collecting false knowledge, rewriting it as different instruction types, and gathering reference responses. The dataset is screened for suitable knowledge and response quality, and its test set spans three styles and two premise-count settings.

3 Data Collection

  • INDUST data collection proceeds through false-knowledge collection, rewriting into inductive instructions, and reference-response collection.Human labor filters knowledge and rewrites, and supervises reference responses for quality.Source: §3 · p. 3

3.1 False Knowledge Collection

  • False knowledge is selected to be highly inductive yet well known to LLMs, reducing the possibility that failure reflects lack of exposure.The source material comes from CHEF, a Chinese rumor dataset containing misleading common-sense rumors and supporting evidence; obscure or overly complex information is excluded, with common-sense and context-stability criteria applied.Source: §3.1 · pp. 3–4

3.2 Rewriting False Knowledge

  • False knowledge is rewritten as three instruction types: FCI uses a rule-based method, while text-davinci-003 generates QFP and CIFP using in-context examples.Two annotators wrote 32 examples, and two were randomly selected as demonstrations during generation.Source: §3.2 · p. 4

3.3 Reference Response Collection

  • GPT-4-generated reference responses are guided to avoid false or malicious content, reject requests while identifying false premises, and offer corrections.Truthfulness is scored as error-free versus erroneous or harmful; helpfulness rates stance from Support to Attack, where Attack actively corrects the premise and offers alternatives.Source: §3.3 · p. 4

3.4 Quality Control

  • Quality control filters instructions unless at least two of three annotators label them Support, and preserves, rewrites, or discards responses according to truthfulness and helpfulness scores.Responses scoring Truthfulness 1 and Helpfulness 1 are rewritten; those scoring 0 on either criterion are discarded.Source: §3.4 · p. 4

3.5 Statistics of INDUST

  • The test set contains 1,050 instructions based on 150 claims: 150 FCI, 450 QFP, and 450 CIFP, with 120 single-premise and 30 multiple-premise claims.The dataset comprises approximately 3,000 instruction-response pairs overall; the remaining data form the development set.Source: §3.5 · p. 4

4 Fragility of LLMs Against INDUST

The evaluated LLMs were vulnerable to inductive instructions, with performance varying by instruction style and worsening for multiple false premises. Human-validated GPT-4 evaluation found that evidence-informed judging aligned best with human ratings.

4.1 Models

  • The evaluation covered ChatGLM2, BELLE-7B, ChatGPT, and text-davinci-003 using Truthfulness and Helpfulness metrics.ChatGPT and text-davinci-003 responses were obtained through the OpenAI API with greedy decoding.Source: §4.1 · p. 5, §4.2.1 · p. 5

4.2.2 Human Evaluation

  • Three annotators evaluated 120 instruction-response pairs per LLM, sampled from 30 single-premise and 10 multi-premise claims.For QFP and CIFP, one sample was randomly selected for evaluation.Source: §4.2.2 · p. 5

4.2.3 Automatic Evaluation

  • The automated protocols varied whether GPT-4 judged responses using basic criteria alone, a reference response, or relevant evidence.Human evaluation was used to validate the alignment of automated and human assessments.Source: §4.2.3 · p. 5, §4.2.4 · p. 5

4.2.4 Automatic Evaluation V.S. Human Evaluation

  • GPT-4’s evidence-informed evaluation aligned best with human evaluation across the two perspectives, while reference-based judging slightly underperformed it.The authors attribute the reference protocol’s lower effectiveness to its single reference answer potentially interfering with assessment of other valid responses.Source: Table 1 · p. 5, §4.2.4 · p. 5

4.3 Preliminary Analysis

  • Most models scored only about 50% in Truthfulness on QFP and around 20% on CIFP, while few exceeded 1 of 2 in Helpfulness on either style.Three of four models also failed to reach an average Helpfulness score of 1 across QFP and CIFP.Source: §4.3 · p. 5, Figure 3 · p. 6
  • For the same claims, ChatGPT reached around 90% Truthfulness on FCI but only around 20% on CIFP, demonstrating strong style-dependent performance.The models performed better on FCI than on QFP and CIFP overall.Source: §4.3 · pp. 5–6
  • LLMs performed worse on instructions with multiple false premises, indicating that more complicated contexts were harder for them.The authors connect this finding to a need for better contextual interpretation and logical reasoning.Source: §4.3 · p. 6

5 Dual-critique Prompting

DUAL-CRITIQUE prompts models to check both user instructions and their own responses, using either a single-step or two-step process. Both variants improved performance on INDUST, while the single-step approach was more practical than the error-sensitive, costlier multi-step variant.

5 Dual-critique Prompting

  • DUAL-CRITIQUE combines USER-CRITIQUE of instructions with SELF-CRITIQUE of generated responses to address false information and inaccurate outputs.The authors motivate the approach with LLMs’ low INDUST performance and prior work on checking reasoning or unsafe content.Source: §5 · p. 6
  • SDUAL-CRITIQUE uses one prompt to critique potentially misleading instructions and monitor the response for accuracy.Its example prompt asks the model to identify errors, reject and correct them, and respond truthfully and safely.Source: §5 · p. 6

Multi-step Dual-critique (MDUAL-CRITIQUE)

  • MDUAL-CRITIQUE first asks the model to identify instruction errors, then uses a second prompt to revise the instruction and answer safely.The second stage relies on the first-stage analysis.Source: Multi-step Dual-critique… · p. 6

5.1 Experiment Setting

  • Zero-shot experiments tested all three prompting methods, while few-shot experiments omitted MDUAL-CRITIQUE because INDUST lacked multi-step samples.Few-shot results averaged three runs using a random INDUST example other than the tested sample.Source: §5.1 · p. 6

5.2 Experimental Results

  • Both DUAL-CRITIQUE variants significantly improved INDUST performance across models and scenarios; single-step ChatGPT Truthfulness gains exceeded 67% on CIFP.Helpfulness also rose across instruction types for ChatGPT.Source: Table 2 · p. 7, §5.2 · p. 7
  • MDUAL-CRITIQUE improved over STANDARD but did not consistently outperform SDUAL-CRITIQUE, and its extra computation and error propagation made SDUAL-CRITIQUE more practical.An incorrect USER-CRITIQUE analysis can hinder accurate responses in the subsequent SELF-CRITIQUE stage.Source: §5.2 · p. 7
  • Few-shot demonstrations consistently improved LLM performance, and SDUAL-CRITIQUE remained superior to STANDARD prompting in that setting.Sixteen of the 24 highest Helpfulness scores occurred in few-shot settings.Source: §5.2 · p. 7

6 Discussion

SDUAL-CRITIQUE retained strong performance against paraphrased prompt changes, with a slight decline on generic benchmarks. Fine-tuning BELLE on LINDUST also improved its inductive-instruction performance, especially for QFP and CIFP.

6.1 Application of DUAL-CRITIQUE

  • On MT-Bench, SDUAL-CRITIQUE lowered ChatGPT by 0.27 points (8.51 to 8.24) and text-davinci-003 by 0.55 points (7.59 to 7.04).The authors characterize these drops as minor and conclude SDUAL-CRITIQUE retains sufficient general ability for practical use.Source: §6.1 · p. 7
  • SDUAL-CRITIQUE outperformed STANDARD prompting by a large margin across paraphrased critique prompts, although performance fluctuated with prompt settings.BELLE was more sensitive to critique prompts than ChatGLM2, and models benefiting more from SDUAL-CRITIQUE were more sensitive to prompt design.Source: §6.1 · p. 8

6.2 Finetuning Performance

  • Fine-tuning BELLE on LINDUST improved performance with STANDARD prompting, especially on QFP and CIFP instances.The authors describe this as an alternative to prompting: infusing inductive-instruction samples into training data improved BELLE’s zero-shot capability.Source: §6.2 · p. 8, Figure 4 · p. 8

7 Related Work

Prior work studied LLM evaluation, self-critique, and question answering with questionable assumptions. This paper focuses on the more varied instructions and inductive styles used with LLMs.

Evaluation of LLMs

  • LLM evaluation research has covered domains, response qualities, and red-teaming, while this paper examines the ability to distinguish and resist inductive instructions.The authors state that this capability had not been thoroughly investigated to their knowledge.Source: §7 · p. 8

Self-critique Prompting

  • Prior self-critique work examined revising unsafe outputs, whereas DUAL-CRITIQUE also checks users’ instructions for false or malicious information.Bai et al. (2022) and Ganguli et al. (2023) are cited for prior work on output critique and self-correction.Source: §7 · p. 8

Questions with Questionable Assumptions

  • Earlier QA datasets tested false assumptions in questions, while INDUST covers instructions with varied intentions and three styles based on single or multiple premises.The paper categorizes questions with questionable assumptions as QFP within INDUST.Source: §7 · p. 8

8 Conclusion

The paper introduces INDUST to evaluate LLM resistance to inductive instructions and proposes DUAL-CRITIQUE prompting, which brings consistent performance benefits.

8 Conclusion

  • INDUST evaluates LLM resistance to inductive instructions, and DUAL-CRITIQUE prompting questions both users’ instructions and the models’ own responses.The paper reports that the prompting approach brought consistent performance benefits.Source: §8 · p. 9

Limitations

INDUST is intended to evaluate LLM resistance to inductive instructions, but it does not fully capture real-world complexity.

Limitations

  • INDUST is designed to evaluate LLM resistance to inductive instructions.The authors describe the benchmark as a source of insight for research on this topic.Source: Limitations · p. 9
  • The benchmark cannot fully capture real-world complexities, despite efforts to cover as many inductive instructions as possible.The authors identify this as a boundary on the benchmark’s coverage.Source: Limitations · p. 9
  • The authors suggest that the included inductive styles and instances could still shed light on the research field.This is their stated expectation for the benchmark’s value despite its coverage limit.Source: Limitations · p. 9

Ethics Statement

The supplementary material discusses benchmark risks and annotation practices, then describes filtering procedures and prompt variants used in the study.

Ethics Statement

  • INDUST contains safer and deceptive information and may be misused to cause deception or harm, though the authors judge its potential value greater than its risk.The authors state that they built it to help create safer and more truthful LLMs.Source: Ethics Statement · p. 9

A.1 Filtering Obscure Knowledge

  • False-knowledge collection used three annotators with undergraduate degrees, and the annotated sets had 98.72% coverage.The guidelines score truthfulness as 0 or 1 and helpfulness from 0 to 2, based on whether responses support, stay neutral toward, or oppose incorrect assumptions.Source: §A.1 · p. 11
  • The Vanilla and w/ reference GPT-4 evaluation prompts differ in whether they include incorrect assumptions and a reference response.The prompt variants and annotation-guideline placeholder are presented in Table 5.Source: §A.1 · p. 11, Table 5 · p. 11

A.2 Filtering Low-quality Instructions

  • Low-quality instructions were assessed as Support, Neutral, or Attack; Support means the instruction reinforces or adds information supporting the premise.The examples distinguish instructions that support a premise from those that challenge it or take no side.Source: §A.2 · pp. 11–12

A.3 Reference Response Quality Control

  • Two annotators labeled the collected reference responses, with their annotation guidelines described in Section 3.3.The resulting statistics are reported in Table 13.Source: §A.3 · p. 12

B Prompts for Dual-critique

  • The study tested five paraphrased versions of the SDUAL-CRITIQUE prompt, while MDUAL-CRITIQUE separates its user- and self-critique parts.The prompt variants were used to examine sensitivity to prompt design.Source: B Prompts for Dual-critique · p. 12

C.1 Prompts of Automatic Evaluation

  • The prompts used for automatic evaluation are presented in Table 5.Source: §C.1 · p. 12

C.2 Details of Human Annotation

The appendix reports human-annotation agreement and evaluation checks, describes construction and fine-tuning of LINDUST, and gives illustrative cases of prompting behavior.

C.2 Details of Human Annotation

  • Human annotators labeled more than 1,000 instruction-response pairs each, reaching Fleiss’ Kappa scores of 71.23 on Truthfulness and 65.11 on Helpfulness.The automatic-evaluation test set included 300 harmless and correct responses and 300 harmful responses; human and automatic evaluations showed no enormous bias or gap.Source: §C.2 · pp. 12–13, Figure 6 · p. 13

D Construction of LINDUST; D.1 Collecting Topics and False Knowledge

D.2 Rewriting False Knowledge; D.3 Reference Response Collection

  • LINDUST reuses the INDUST procedure for inductive instructions, treats its instruction-response pairs as valuable training data, and collects reference responses with ChatGPT.The reference-response prompt includes the instruction, false premises, and evidence; the collection procedure is described as supervised by humans.Source: §D.2 · p. 13, §D.3 · p. 13, Table 8 · p. 13

D.4 Generation Parameters of ChatGPT Demonstrations; D.5 Statistics of LINDUST

  • LINDUST is four times larger than INDUST, with 3,142 FCI, 3,322 QFP, and 5,439 CIFP samples.ChatGPT (0301) was used with top-p 1.0 and temperature 0.7 to generate demonstrations.Source: §D.5 · p. 14, §D.4 · p. 14, Table 11 · p. 14

E Finetuning Settings on LINDUST

F.1 Cases of Standard and DUAL-CRITIQUE

  • In zero-shot cases, QFP and CIFP more often misled ChatGPT than other instruction styles, while SDUAL-CRITIQUE helped it recognize erroneous assumptions.The examples are shown in Table 15.Source: §F.1 · p. 14, Table 15 · p. 16

F.2 Cases of MDUAL-CRITIQUE

Loading 2305.13733v2…