PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, Xing Xie

arXiv:2302.12095v5cs.AIcs.CLcs.LG

TL;DR

ChatGPT’s robustness to adversarial and OOD inputs had not been thoroughly evaluated despite its widespread use and relevance to safety-critical applications. The paper conducts a zero-shot evaluation across established and newer datasets, finding consistent advantages on most tasks but substantial remaining weaknesses.

  • Problem

    ChatGPT’s robustness to unexpected inputs was insufficiently evaluated, despite its importance for reliable use in practical and safety-critical applications.

  • Method

    The paper performs zero-shot evaluation using AdvGLUE and ANLI for adversarial robustness, Flipkart and DDXPlus for OOD robustness, and several foundation-model baselines.

  • Results

    ChatGPT shows consistent advantages on most adversarial and OOD classification and translation tasks, while absolute performance remains far from perfection.

  • Takeaways & Limitations

    Adversarial and OOD robustness remains a significant threat to ChatGPT and other foundation models, motivating further research and analysis.

  • Takeaways & Limitations

    The selected OOD datasets may overlap with ChatGPT’s training distributions, so new long-tailed-domain datasets are needed for fairer evaluation.

Abstract

from arXiv · show

ChatGPT is a recent chatbot service released by OpenAI and is receiving increasing attention over the past few months. While evaluations of various aspects of ChatGPT have been done, its robustness, i.e., the performance to unexpected inputs, is still unclear to the public. Robustness is of particular concern in responsible AI, especially for safety-critical applications. In this paper, we conduct a thorough evaluation of the robustness of ChatGPT from the adversarial and out-of-distribution (OOD) perspective. To do so, we employ the AdvGLUE and ANLI benchmarks to assess adversarial robustness and the Flipkart review and DDXPlus medical diagnosis datasets for OOD evaluation. We select several popular foundation models as baselines. Results show that ChatGPT shows consistent advantages on most adversarial and OOD classification and translation tasks. However, the absolute performance is far from perfection, which suggests that adversarial and OOD robustness remains a significant threat to foundation models. Moreover, ChatGPT shows astounding performance in understanding dialogue-related texts and we find that it tends to provide informal suggestions for medical tasks instead of definitive answers. Finally, we present in-depth discussions of possible research directions.

1 Introduction

The paper addresses the under-evaluated robustness of ChatGPT against adversarial and out-of-distribution inputs, motivated by reliability concerns in practical and safety-critical applications. It evaluates ChatGPT through zero-shot inference and reports advantages on many tasks alongside substantial remaining weaknesses.

  • Motivation: Robustness—the ability to withstand disturbances that may cause malfunction or inaccurate results—had not been thoroughly evaluated for ChatGPT.The paper emphasizes robustness as particularly important for practical and safety-critical applications.
  • Motivation: The study focuses on adversarial and out-of-distribution robustness, two threats caused through input perturbation.The motivation includes malicious perturbations that can bypass systems such as fake-news detectors.
  • Approach: The evaluation uses zero-shot inference across 9 tasks and 2,089 test examples, comparing ChatGPT with popular foundation models.The selected evaluation resources include AdvGLUE, ANLI, Flipkart review, and DDXPlus datasets.
  • Findings: ChatGPT consistently improves on most adversarial and OOD classification tasks and produces readable, reasonable translations under adversarial inputs.It also performs better at understanding dialogue-related texts than other foundation models.
  • Findings: ChatGPT’s absolute performance remains far from perfect, while its medical responses provide informed suggestions and analysis rather than definitive answers.The paper presents these behaviors as limitations and as a responsible approach to medical-related inquiries.
  • Contributions: The paper shares reflections and research directions beyond the evaluations and open-sources its code and results.These materials are intended to facilitate future explorations.

2 Background

This section situates ChatGPT within the foundation-model landscape and formulates adversarial and OOD robustness for classification. It also notes that zero-shot evaluation differs from prior robustness studies that train or adapt models.

  • 2.1 Foundation Models, ChatGPT, and Existing Evaluation: Foundation models are widely used for NLP tasks and achieve improvements in sentiment analysis, question answering, diagnosis, reasoning, and sequence tagging.ChatGPT is a GPT-3.5-series generative foundation model with enhanced multi-turn dialogue capabilities, instruction tuning, and RLHF.
  • 2.1 Foundation Models, ChatGPT, and Existing Evaluation: Prior evaluations find that ChatGPT performs reasonably on many tasks but remains weaker on low-resource and fine-grained tasks such as sequence tagging.The supplied related-work passages also describe concerns that large models can be double-edged swords and may require regulation.
  • 2.2 Robustness: Adversarial robustness concerns stability under imperceptible, ϵ-bounded perturbations to an original input.The classification formulation represents inputs x in R^d and labels y in [K], with ℓ denoting the loss function.
  • 2.2 Robustness: OOD robustness concerns learning a classifier that generalizes to unseen distributions by minimizing average risk across possible distributions.The paper distinguishes its zero-shot evaluation from approaches that train on in-distribution data and test on OOD sets.
  • 2.2 Robustness: The paper avoids GLUE and GLUE-X for ChatGPT because those benchmarks may have appeared in its training data.This motivates using newer datasets for zero-shot OOD evaluation.

3 Datasets and Tasks

The paper evaluates adversarial robustness with AdvGLUE and ANLI, and OOD robustness with Flipkart and DDXPlus. Dataset selection reflects both benchmark coverage and uncertainty about ChatGPT’s training-data distributions.

  • 3.1 Adversarial Datasets: AdvGLUE adds word-level, sentence-level, and human-crafted adversarial perturbations to GLUE tasks.The study uses SST-2, QQP, MNLI, QNLI, and RTE, plus a 30-sample adversarial English-to-Chinese translation set.
  • 3.1 Adversarial Datasets: ANLI contains 16,000 premise-hypothesis pairs across entailment, contradiction, and neutral classes.The study selects the R3 test set because it is the most difficult and diverse split.
  • 3.2 Out-of-distribution Datasets: Flipkart is a product-review dataset and DDXPlus is a medical-diagnosis dataset used to construct OOD classification tasks.Both datasets were released in 2022, and the study randomly samples subsets to form test sets.
  • 3.2 Out-of-distribution Datasets: Finding genuinely OOD datasets for ChatGPT is difficult because its training data are unavailable.The authors describe the selected datasets as out-of-example and note that temporal distribution shift can still support their use.
  • 3.2 Out-of-distribution Datasets: The authors acknowledge limitations in the selected OOD datasets and call for brand-new datasets for more thorough evaluation.Common domains such as product reviews and medical diagnosis may overlap with model training distributions.

4 Experiment

The experiment evaluates ChatGPT’s zero-shot robustness against adversarial and OOD inputs, compares it with foundation-model baselines, and examines translation and medical-dialogue behavior. ChatGPT generally performs strongly, but adversarial and OOD challenges remain substantial.

  • 4.1 Setup: The evaluation compares ChatGPT with popular foundation models on zero-shot adversarial and OOD classification tasks.Adversarial tasks use ASR, while OOD tasks use F1-score; the study notes a classification-task disadvantage for generative models because their output spaces were not restricted.
  • 4.1.2 Results: ChatGPT outperforms all counterparts on adversarial classification tasks, although its ASRs remain 40% on SST-2 and 55.3% on ANLI.The authors attribute the remaining vulnerability partly to training on clean corpora, where adversarial texts may be absent.
  • 4.1.2 Results: Models after GPT-2, including ChatGPT and the davinci models, perform well on OOD datasets, with ChatGPT and its siblings especially strong on DDXPlus.The authors also note that absolute ChatGPT and davinci performance remains far from perfect.
  • 4.1.2 Results: ChatGPT performs better than other LLMs on DDXPlus, plausibly because it understands the benchmark’s doctor-patient conversational context.The benchmark contains doctor-posed diagnostic questions and patient responses.
  • 4.1.2 Results: For medical questions, ChatGPT provides informed suggestions and analysis rather than definitive answers, recommending offline testing and consultation.When information is insufficient, it acknowledges the uncertainty and explains why it cannot make a judgment.
  • 4.2.2 Results: In adversarial English-to-Chinese translation, GPT-family models outperform the fine-tuned models, and ChatGPT’s translations remain readable and reasonable to humans.ChatGPT beats text-davinci-002 on BLUE and GLUE but is slightly worse on METOR; text-davinci-003 generalizes best across metrics.
  • 4.3 Case Study: Word-level typos and sentence-level distractions can mislead ChatGPT, demonstrating that common adversarial inputs remain challenging.The authors connect these prevalent everyday adversaries to the need for defensive strategies.

5 Discussion

The discussion identifies persistent adversarial vulnerability and unresolved questions about OOD generalization in large foundation models. It also highlights challenges from prompt attacks, defect inheritance, and the need for broader robustness research.

  • 5.1 Adversarial Attack Remains a Major Threat: Adversarial vulnerability remains a major threat to foundation-model deployments, especially in safety-critical downstream scenarios.The discussion emphasizes that adversarial inputs remain challenging and that downstream users may inherit defects from pretrained models.
  • 5.1 Adversarial Attack Remains a Major Threat: Prompt attacks add another unresolved challenge because LLM outputs are sensitive to prompting.The paper calls for further knowledge and algorithms to address attacks on prompts.
  • 5.2 Can OOD Generalization be Solved by Large Foundation Models?: OOD generalization remains unsettled because larger models may benefit from prompt engineering while potentially overfitting to broad training distributions.The paper explicitly frames the question of whether giant models solve OOD generalization and notes that when and why LLMs overfit remains unknown.
  • 5.2 Can OOD Generalization be Solved by Large Foundation Models?: The evaluation may overestimate OOD robustness because Flipkart and DDXPlus resemble common Internet domains that could overlap with training data.The authors call for new datasets from long-tailed domains for fairer evaluation.
  • 5.2 Can OOD Generalization be Solved by Large Foundation Models?: ID and OOD performance can be inversely correlated, motivating regularization and other techniques for improving language-model OOD performance.The discussion says the analysis does not establish a consistently positive relationship between ID and OOD performance.

6 Limitation

The authors characterize the study as preliminary and identify limitations in evaluation design, dataset scale, task coverage, and ChatGPT’s intended use. They also note that results could change with adaptation, prompts, broader datasets, or future analyses.

  • Evaluation design: The study performs only zero-shot classification, so fine-tuning or adaptation could change the reported model results.The authors justify zero-shot evaluation because further operations on current large foundation models are expensive and unaffordable.
  • Dataset scale: Evaluating large foundation models on small datasets is controversial, and the authors call for more datasets.Unclear training data and ChatGPT’s extensive Internet-scale pretraining make larger evaluation datasets difficult to identify.
  • Task coverage: Most evaluations concern text classification, with only minor machine-translation coverage and no substantial evaluation of generation.The authors note that more appropriate datasets and better prompts could improve performance assessment.
  • Intended use: Because ChatGPT is designed primarily as a chatbot, classification-focused evaluations do not measure robustness in online chatting.The authors explicitly distinguish their evaluation setting from ChatGPT’s intended conversational use.
  • Future work: Future work should compare pre-2021 data, analyze more OOD cases, study prompt-induced outputs, and develop metrics for adversarial performance.The authors also propose detailed ablations using different language models.

7 Conclusion

The paper presents a preliminary empirical evaluation of ChatGPT’s adversarial and OOD robustness. Its experiments find progress but also substantial room for improvement, motivating further research directions.

  • 7 Conclusion: The study finds that ChatGPT and other large models have improved robustness but still have substantial room for improvement on adversarial and OOD tasks.The conclusion frames the work as an empirical evaluation accompanied by broader analysis and proposed research directions.

Disclaimer

The disclaimer states that the paper aims to document robustness risks rather than attack ChatGPT intentionally. It also records the experimental version and authorship arrangements relevant to interpreting the study.

  • Ethics and societal concerns: The paper uses public datasets to show that ChatGPT can be attacked by adversarial and OOD examples, without intending to attack it maliciously.The authors express concern that end-users could misuse these findings.
  • ChatGPT usage: All ChatGPT experiments used the February 13 version, so later service updates may change the reported results.The disclaimer also explains that a U.S.-based coauthor conducted the experiments to comply with local laws and OpenAI service terms.
  • Author contributions: The authors assign distinct responsibilities for project leadership, datasets, prompts, machine translation, code organization, and reproducibility.The contribution statement identifies the roles of the named authors across these components.

A Detailed Introduction of Datasets and Tasks

The evaluation uses benchmarks and datasets covering adversarial robustness, translation, natural language inference, sentiment, question equivalence, and medical diagnosis.

  • Adversarial evaluation: AdvGLUE evaluates five natural language understanding tasks under word-level, sentence-level, and human-written adversarial perturbations.The tasks are SST-2, QQP, MNLI, QNLI, and RTE.
  • Translation evaluation: AdvGLUE-T is an English-to-Chinese adversarial machine translation dataset created by randomly extracting 30 AdvGLUE samples.
  • Adversarial evaluation: ANLI challenges natural language inference models with human-and-model-in-the-loop adversarial examples designed to exploit model vulnerabilities.Its examples remain understandable to humans while attempting to induce misclassification.
  • OOD evaluation: Flipkart contains product reviews labeled positive, negative, or neutral; this study selects 331 reviews whose texts are 150–160 characters long.The source dataset covers 104 product types and 205,053 labeled examples.
  • OOD evaluation: DDXPlus provides synthetic medical cases with differential diagnoses, true pathologies, symptoms, and antecedents for automatic diagnosis.The study formats translated cases as age, gender, initial evidence, and inquiry dialogue for disease selection.

B Evaluation Metrics

The paper evaluates adversarial robustness with attack success rate and reviews generalization-error concepts underlying model performance and robustness analysis.

  • Adversarial metrics: Attack Success Rate measures how often an attack changes a correctly classified input into an incorrect adversarial prediction.The attack generates A(x) from input x to attack a surrogate model f over dataset D.
  • Adversarial metrics: Model robustness is inversely proportional to attack success rate.
  • Generalization theory: PAC theory relates generalization error to the difference between a classifier’s ideal and empirical risks.The discussion focuses on generalization error as a foundation for analyzing foundation models.
  • Generalization theory: Theorem 1 states that generalization error depends on training-sample count and hypothesis-set size.The paper uses this framework to discuss why large datasets may support strong foundation-model performance.
  • Generalization theory: The paper notes that increasing model complexity may induce high VC dimension, while the reason large foundation models avoid overfitting remains unknown.

C.2 Out-of-distribution Robustness Theory

OOD theory considers performance when training and test distributions differ, using domain-adaptation bounds to organize sources of target-domain error.

  • OOD setting: OOD evaluation trains on a source dataset and tests on an unseen target dataset with a different distribution.The target risk cannot be directly evaluated when the unseen dataset is inaccessible.
  • Target-error bound: Theorem 2 bounds target-domain error using H-divergence for a hypothesis space with VC dimension d and source samples of size m.The bound holds with probability at least 1 − δ.
  • Target-error bound: The target-error bound contains source empirical error, source–target distribution discrepancy, ideal joint error, and a sample-size/VC-dimension term.
  • Target-error bound: Conventional OOD research minimizes source–target distribution discrepancy while treating source risk as determined.The bound’s sample-size term can also decrease as the number of samples increases.
  • Theory limitations: The paper cautions that interpreting strong foundation-model OOD performance through this generalization analysis is not rigorous.VC dimension also has no correlation with dataset distributions and cannot explain the observed strong OOD performance.

D Foundation Models used in Experiments

The experiments compare ChatGPT with foundation models spanning encoder, encoder–decoder, and decoder-only transformer designs, alongside the study’s prompts and case-study tables.

  • Compared foundation models: BART-L supports both language understanding and generation through bidirectional and autoregressive transformer objectives.
  • Compared foundation models: DeBERTa-L uses disentangled attention and enhanced decoding to model token context and generate higher-quality sentences.
  • Compared foundation models: GPT-J-6B is a transformer model whose name denotes its 6 billion parameters.
  • Compared foundation models: Flan-T5 uses a text-to-text input-output format for tasks including translation, summarization, and question answering.
  • Compared foundation models: GPT-NeoX-20B, OPT, BLOOM, and text-davinci models provide additional autoregressive, decoder-only, cross-lingual, and instruction-oriented baselines.Their descriptions distinguish model scale, training objectives, language coverage, and output quality.
  • Experiment materials: The study lists its prompts in Table 5 and presents OOD case studies in Table 6.
Loading 2302.12095v5…