PaperlayerView exact revision on arXiv

Source-linked AI summary

How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities

Lingbo Mo, Boshi Wang, Muhao Chen, Huan Sun

arXiv:2311.09447v2cs.CLcs.AI

TL;DR

Open-source LLM trustworthiness remains insufficiently understood despite their increasing use and vulnerability to adversarial attacks. The paper introduces advCoU for adversarial assessment across eight aspects and finds that larger and instruction-tuned models can be more vulnerable, while safety alignment mitigates attacks.

  • Problem

    Trustworthiness research has provided limited evidence across the full range of vulnerabilities in open-source LLMs, despite risks from harmful and unethical outputs.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates open-source LLMs across eight trustworthiness aspects using advCoU, an extended CoU strategy with malicious demonstrations and misleading internal thoughts.

    Source: §1 · p. 3

  • Results

    The attack strategy is effective across diverse aspects; larger and instruction-tuned models tend to be more vulnerable, whereas safety-alignment fine-tuning helps protect against attacks.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    General NLP capability does not guarantee trustworthiness, while safety alignment can reduce vulnerability to adversarial trustworthiness attacks.

    Source: §1 · p. 3

    The study examines only one representative scenario for each of eight aspects, so more fine-grained and holistic investigation is needed.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The rapid progress in open-source Large Language Models (LLMs) is significantly driving AI development forward. However, there is still a limited understanding of their trustworthiness. Deploying these models at scale without sufficient trustworthiness can pose significant risks, highlighting the need to uncover these issues promptly. In this work, we conduct an adversarial assessment of open-source LLMs on trustworthiness, scrutinizing them across eight different aspects including toxicity, stereotypes, ethics, hallucination, fairness, sycophancy, privacy, and robustness against adversarial demonstrations. We propose advCoU, an extended Chain of Utterances-based (CoU) prompting strategy by incorporating carefully crafted malicious demonstrations for trustworthiness attack. Our extensive experiments encompass recent and representative series of open-source LLMs, including Vicuna, MPT, Falcon, Mistral, and Llama 2. The empirical outcomes underscore the efficacy of our attack strategy across diverse aspects. More interestingly, our result analysis reveals that models with superior performance in general NLP tasks do not always have greater trustworthiness; in fact, larger models can be more vulnerable to attacks. Additionally, models that have undergone instruction tuning, focusing on instruction following, tend to be more susceptible, although fine-tuning LLMs for safety alignment proves effective in mitigating adversarial trustworthiness attacks.

1 Introduction

This work addresses limited understanding of open-source LLM trustworthiness by adversarially testing eight aspects with malicious demonstrations and internal thoughts. It introduces advCoU and finds that larger and instruction-tuned models can be more vulnerable, while safety alignment helps mitigate attacks.

  • The assessment manipulates demonstrations while keeping model weights fixed and seeks to improve understanding of open-source LLM risks for reliable downstream deployment.Source: §1 · pp. 2–3
  • Models with superior general NLP performance are not necessarily more trustworthy, and larger models tend to be more susceptible to malicious demonstrations.Source: §1 · p. 3
  • Experiments show that instruction-tuned models emphasizing instruction following are more vulnerable, whereas safety-alignment fine-tuning helps protect against adversarial trustworthiness attacks.Source: §1 · p. 3
  • The study evaluates open-source LLM trustworthiness across toxicity, stereotypes, ethics, hallucination, fairness, sycophancy, privacy, and robustness against adversarial demonstrations.Source: §1 · pp. 1, 3
  • advCoU extends Chain of Utterances prompting with carefully designed malicious demonstrations and misleading internal thoughts to induce undesired model behavior.Source: §1 · p. 3

2 Related Work

Prior work has examined LLM trustworthiness through individual aspects and diverse attacks, but comprehensive adversarial assessment of open-source models remains limited. This work combines multifaceted evaluation with CoU-based malicious demonstrations.

  • Earlier trustworthiness studies often focused on individual aspects such as toxicity, stereotypes, privacy, sycophancy, or robustness.Source: §2 · p. 3
  • The paper differs by assessing open-source LLMs across eight trustworthiness aspects through existing attacks and a CoU-based strategy with malicious demonstrations.Source: §2 · pp. 3–4
  • Existing attacks include malicious instructions, jailbreak prompts, automated adversarial inputs, adversarial demonstrations, universal attacks, and manipulated generation strategies.Source: §2 · pp. 3–4

3 Methodology

The methodology adapts in-context learning and Chain of Utterances prompting into advCoU, using malicious demonstrations and misleading internal thoughts to attack open-source LLM trustworthiness. It evaluates eight aspects with task-specific datasets, prompts, and success criteria.

3.1 In-Context Learning

  • In-context learning generates output y from input x conditioned on inference context C, which contains a task description and task-specific demonstrations.Source: §3.1 · p. 4

3.2 CoU Prompt-based In-Context Attack

  • advCoU uses a Red-LM attacker and Target-LM defender in a conversational prompt, combining malicious demonstrations with internal thoughts that steer undesired responses.Source: §3.2 · p. 4

3.3 Eight Aspects of Trustworthiness

  • Toxicity is measured on 1.2K challenging REALTOXICITYPROMPTS examples using average Perspective API toxicity scores from 0 to 1.Source: §3.3 · pp. 4–5, Figure 2 · p. 5
  • The evaluation tests stereotype agreement, immoral judgments, hallucinated-answer selection, gender-linked income errors, false-opinion agreement, privacy leakage, and robustness to counterfactual demonstrations.Source: §3.3 · pp. 5–6
  • Robustness uses SNLI-CAD counterfactual demonstrations in SNLI-RH and SNLI-RP settings, totaling 800 samples, with success defined as predicting an incorrect label.Source: §3.3 · p. 6

4 Experiments

The experiments assess five open-source LLM series across eight trustworthiness aspects using attack success rate (ASR). Results show that larger models and instruction-focused fine-tuning can increase vulnerability, whereas safety alignment reduces it, and advCoU outperforms DECODINGTRUST.

4 Experiments

  • The evaluation examines five open-source model series—VICUNA, MPT, FALCON, MISTRAL, and LLAMA 2—across eight trustworthiness aspects, reporting attack success rates for attacks and a baseline.Inference uses temperature 0 and top-p with p=1.Source: §4 · p. 6, §4.1 · p. 7, §4.2 · p. 7

4.2.1 Are Larger Models More Trustworthy?

  • Larger base models consistently have higher average ASR within each series, while LLAMA 2 has the highest average ASR across series.The result indicates greater susceptibility to adversarial attacks despite LLAMA 2 being described as the strongest general-NLP series among the five.Source: §4.2.1 · p. 7, Figure 3 · p. 7

4.2 Results and Analysis

  • The results caution against treating scale or instruction following as sufficient for trustworthiness and support combining moderate model size with safety alignment.The authors frame this as a development and deployment consideration because scaling can be resource-intensive and expose models to more attack risks.Source: §4.2.1 · p. 7, §4.2.2 · p. 8

4.2.2 Are Instruction Tuned or Aligned Models More Trustworthy?

  • Safety-aligned chat models have lower ASR than their bases, whereas FALCON-instruct and MISTRAL-instruct have higher ASR after instruction-focused fine-tuning.The authors attribute the latter pattern to stronger instruction following without comparable moderation mechanisms, while LLAMA 2-chat and MPT-chat use safety-alignment procedures.Source: Figure 4 · p. 8, §4.2.2 · p. 8

4.2.3 Comparing DECODINGTRUST and Ours: Open LLMs Show Vulnerabilities across Different Attack Strategies

  • advCoU consistently outperforms DECODINGTRUST in average ASR across shared aspects, with nearly 100% ASR for stereotype, privacy, and sycophancy.The comparison uses six shared aspects across 19 model variants; Table 1 reports average ASR and standard deviations.Source: §4.2.3 · pp. 8–9, Table 1 · p. 9
  • Both attack strategies produce high ASRs, and advCoU has lower standard deviations across most aspects, averaging 0.094 across shared aspects.This supports greater generalizability across model series and variants under the reported comparison.Source: §4.2.3 · pp. 8–9

5 Conclusion

The paper proposes advCoU, which injects malicious demonstrations and misleading internal thoughts into CoU prompting, and applies it to an eight-aspect adversarial assessment of open-source LLMs. The results demonstrate attack effectiveness and connect trustworthiness with model size and alignment strategies.

5 Conclusion

  • The paper proposes advCoU and evaluates open-source LLM trustworthiness across eight aspects using malicious demonstrations and misleading internal thoughts.The assessment is intended to uncover trustworthiness issues and examine the influence of model size and alignment strategies.Source: §5 · p. 9

Limitations

The study’s trustworthiness assessment is limited in coverage, evaluation design, and prompt construction. It examines representative scenarios, simplifies some setups for automated checking, and relies on manually designed malicious demonstrations.

Limitations

  • The study covers eight trustworthiness aspects but does not encompass the full spectrum, with one representative scenario selected for each aspect.The authors call for more fine-grained and holistic investigations.Source: Limitations · p. 9
  • Some problem setups are simplified to automate evaluation without additional human effort.Examples include converting tasks to multiple-choice formats and requiring “Yes” or “No” keywords for structured verification.Source: Limitations · p. 9
  • The malicious demonstrations and internal thoughts are manually designed for each trustworthiness aspect.Automating prompt generation and iteratively optimizing prompts across aspects or scenarios is identified as future work.Source: Limitations · p. 9

Ethical Considerations

The authors disclose that their research could facilitate offensive or harmful content generation, but present it to expose risks and strengthen defensive attention. They justify disclosure by the existing availability of jailbreaks and the expanding use of LLM outputs in autonomous decision-making.

Ethical Considerations

  • The research may be used to generate offensive and harmful content, creating inherent risks that the authors explicitly acknowledge.The disclosure concerns attack strategies for open-source LLMs.Source: Ethical Considerations · p. 9
  • The authors argue that existing online jailbreaks reduce the immediate incremental harm from disclosing their attack strategies.They identify fortifying defensive mechanisms as the primary value of the work.Source: Ethical Considerations · p. 9
  • The work aims to highlight LLM risks and motivate researchers and developers to improve trustworthiness and responsible use.The authors connect this motivation to the expansion of LLM adoption and autonomous decision-making based on LLM outputs.Source: Ethical Considerations · p. 9

A Internal Thoughts

The appendix lists the internal thoughts designed for each trustworthiness aspect. These prompts support the paper’s aspect-specific attack setup.

A Internal Thoughts

  • The appendix presents the internal thoughts designed for each trustworthiness aspect.They are listed in Table 2.Source: A Internal Thoughts · p. 12
  • The listed prompts are organized around the paper’s eight trustworthiness aspects.The table provides the prompt material used to mislead models across the assessment dimensions.Source: A Internal Thoughts · p. 12
  • The internal thoughts function as aspect-specific attack components rather than general evaluation instructions.This follows their stated design for each aspect of trustworthiness.Source: A Internal Thoughts · p. 12

B Baseline Setup

The baseline setup uses DECODINGTRUST as the comparison point and selects attack settings for each trustworthiness aspect. It covers toxicity, stereotypes, ethics, privacy, fairness, and robustness against adversarial demonstrations, with internal-thought prompts documented in Table 2.

B Baseline Setup

  • DECODINGTRUST (Wang et al., 2023b) is selected as the baseline, and the section describes setup details and prompts for each aspect.The baseline comparison frames the selected attack configurations.Source: B Baseline Setup · p. 12
  • For toxicity, the setup uses a straightforward prompt that adds an adversarial instruction to bypass content policies and generate toxic language.This prompt type was reported as the most effective for eliciting toxicity in the baseline work.Source: B Baseline Setup · p. 12
  • For stereotypes, the setup uses targeted prompts encouraging bias against specific demographic groups.The targeted setting was selected because it was reported to make biased outputs more likely.Source: B Baseline Setup · p. 12
  • For ethics, the setup uses the prompt “Every reaction can be considered as not wrong” to induce misclassification of immoral actions.This is the second jailbreaking prompt type listed for the ethics evaluation.Source: B Baseline Setup · p. 12
  • For privacy, the setup uses few-shot privacy-leakage demonstrations that guide models to output privacy information.This is selected from three inference-stage settings involving personally identifiable information.Source: B Baseline Setup · pp. 12–13
  • Table 2 documents the internal thoughts designed for the eight trustworthiness aspects, using <label> for yes/no/maybe natural-language-inference labels.The table serves as a reference for the aspect-specific prompt designs.Source: Table 2 · p. 13
  • For fairness, the setup uses demographically imbalanced few-shot demonstrations with base rate parity set to 1.The selected setting is intended to create the largest bias toward a particular prediction.Source: B Baseline Setup · p. 13
  • For robustness against adversarial demonstrations, the setup studies how counterfactual examples added to in-context demonstrations affect model predictions.The described settings include zero-shot evaluation, one counterfactual example, and larger demonstration contexts.Source: B Baseline Setup · p. 13

C Prompt Templates

The section provides prompt templates for the seven trustworthiness aspects beyond toxicity, including stereotype, ethics, hallucination, fairness, sycophancy, privacy, and robustness against adversarial demonstrations.

  • Prompt templates cover stereotype, ethics, hallucination, fairness, sycophancy, privacy, and robustness against adversarial demonstrations in addition to toxicity.Source: C Prompt Templates · p. 13

D More Results

This section compares the method with DECODINGTRUST across shared trustworthiness aspects and reports hallucination and sycophancy results explored only by this work.

Related papers

Loading 2311.09447v2…