I'm Afraid I Can't Do That: Predicting Prompt Refusal in Black-Box Generative Language Models

1.1 Bias in ChatGPT

• ChatGPT has been reported to express left-libertarian and environmentalist political views, including stronger alignment with Germany’s Socialist party than the German population.Source: §1.1 · p. 1
• Other studies found political and psychological biases, including progressive and libertarian general views and low measured levels of psychological dark traits.Source: §1.1 · p. 1
• OpenAI’s moderation responses varied substantially across demographic groups, with protection above 80% for some classes but below 20% for others.Source: §1.1 · p. 2
• Fine-tuning can produce markedly different political biases, as shown by RightWingGPT exhibiting approximately opposite political-compass biases from ChatGPT.Source: §1.1 · p. 2

1.2 Prompt Refusal

• ChatGPT’s bias also appears in whether it refuses to cooperate with prompts, and the paper investigates which prompt types predict refusal.Source: §1.2 · p. 2
• Although early refusals appeared binary, broader examination found a smooth continuum of responses from compliance to refusal.Source: §1.2 · p. 2, Table 1 · p. 2

2.1 Prompt Database Compilation

• The prompt database combined offensive and benign material, including 400 sincere and 600 insincere hand-labeled Quora questions plus 10,000 additional samples automatically labeled for training.The Quora data were selected because their question-form text suited ChatGPT prompts, although its offensive content may reflect regional prejudices and may not fully target OpenAI’s harmful-content priorities.Source: §2.1 · pp. 3–4

2.1.4 Other Candidate Datasets.

• The hand-labeled dataset (n=1,706) compiled labeled samples from aforementioned datasets into a superset and served as the primary dataset for training the refusal classifier.Many text strings were too fragmentary to be properly understood as prompts, and the same problem prevented effective use of the collected 4Chan archive.Source: §2.1.4 · p. 4

2.2 Manual Refusal Labeling

• Manual inspection found that ChatGPT compliance and refusal form a continuum, so eight response subcategories were mapped to binary compliance/refusal labels.The mapping addressed rare subcategories that would have limited direct eight-way classification.Source: §2.2 · p. 4
• 93% of sincere Quora questions received compliance, compared with 53% of insincere questions, showing that refusal did not perfectly track question sincerity.The dataset also contained sincere questions that ChatGPT refused and insincere questions that it complied with.Source: §2.2 · p. 4, Table 4 · p. 5

2.3 Classifier Training

• Both classifiers accept variable-length text and output binary classifications; BERT, logistic regression, and random forest were evaluated across two tasks, with BERT expected to achieve the best accuracy.The latter two models were used primarily for interpretability, revealing highly predictive words or few-word phrases (n-grams).Source: §2.3 · p. 5

2.3.1 Refusal Classification.

• Refusal identification leveraged shared response expressions, whereas refusal prediction required modeling subtle prompt differences because substituting one person or demographic could change compliance to refusal.The prompt classifier was trained on 10,000 automatically labeled Quora samples and evaluated on hand-labeled Quora data; the refusal classifier supplied labels to bootstrap the larger dataset.Source: §2.3.1 · p. 5, §2.3.2 · pp. 5–6

2.3.2 Predicting Refusals.

• Figure 1 presents an overview of prompt classifier training.Source: §2.3.2 · p. 6

3.1 Model Performance

• BERT significantly outperformed classical models on both refusal classification and prompt classification.For refusal classification, logistic regression and random forest achieved 90.6% and 86.7% accuracy, while prompt classification accuracies were 73.9%, 72.2%, and 75.9%, respectively.Source: §3.1 · p. 6

3.2 Feature Importance

• The feature analyses compared n-grams predictive of whether a prompt would be refused with n-grams predictive of whether a response was a refusal.The hand-labeled analysis used logistic-regression weights for both prediction targets.Source: §3.2 · p. 7, Figure 2 · p. 7
• For response refusal, “cannot,” “sorry,” and “AI language model” predicted refusal, while “the” predicted compliance.The authors suggest that “the” may indicate the declarative-sentence style used in compliant responses.Source: §3.2 · p. 7
• For prompt refusal, “trump,” plural demographic groups, and negative adjectives predicted refusal, whereas “what are” predicted compliance.The reported predictors came from 10,000 machine-labeled Quora Insincere Questions samples.Source: §3.2 · p. 7, Figure 3 · p. 7

• ChatGPT’s compliance behavior is not cleanly binary but forms a smooth continuum of refusal responses.The authors mapped varied response types to binary labels for classification despite this continuum.Source: §4 · p. 8
• Negative generalizations about demographic groups are among the strongest predictors of ChatGPT refusals, while controversial figures also predict refusal.The study does not determine whether controversial figures themselves prompt refusal or instead attract prompts that are refusable for other reasons.Source: §4 · p. 8
• BERT outperformed classical models for both refusal and prompt classification, though its advantage was smaller for prompt classification.The authors note that classification errors matter because erroneous automatic refusal labels can frustrate users.Source: §4 · p. 8
• Prompt prediction has a performance ceiling associated with OpenAI’s intentional response-randomness temperature, although the authors doubt their study reached that ceiling.This randomness could contribute to irreducible limits on predicting refusals from prompts alone.Source: §4 · p. 8

• Multiple manual labelers could reduce personal bias in refusal labels, while improving the refusal classifier may also improve prompt-classifier performance.Source: §5 · p. 8
• Larger automatically labeled datasets could reduce sparsity across diverse input expressions and make prompt classification more reliable.Source: §5 · p. 8
• Comparing feature importance across ChatGPT snapshots could characterize OpenAI’s ongoing alignment work.Source: §5 · p. 8
• Querying each prompt several times could characterize how internal randomness temperature affects performance.Source: §5 · p. 8
• Repeated queries could support classifiers that predict refusal likelihood instead of only binary refusal.Source: §5 · p. 8