PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Follow Simple Rules?

Norman Mu, Sarah Chen, Zifan Wang, Sizhe Chen, David Karamardian, Lulwa Aljeraisy, Basel Alomair, Dan Hendrycks, David Wagner

arXiv:2311.04235v3cs.AIcs.CLcs.LG

TL;DR

Reliable rule-following matters as language models take on real-world responsibilities, but adversarial inputs can circumvent explicit instructions. The paper introduces RULES, a programmatic benchmark, and finds that current models often struggle with its tests; optimization attacks can further reduce performance.

  • Problem

    Reliable constraints matter for consequential LLM applications, while existing adversarial-attack evaluations often require costly manual review or unreliable heuristic checks.

    Source: §1 · p. 1

  • Method

    RULES evaluates rule-following across 14 scenarios using programmatic checks, with test cases and an aggregate score covering harmless and helpful rules.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    Almost all current models perform poorly on the test suites, and a GCG-based attack can significantly reduce model performance.

    Source: §4 · pp. 5–6

  • Takeaways & Limitations

    The findings suggest that rule-following needs new approaches, with output steering and fine-tuning presented as possible paths forward.

    Source: §4 · p. 5, §6 · p. 9

    Even GPT-4's strongest version fails 93 unique test cases, and harder adversarial cases could be constructed, so high scores on these relatively easy suites do not establish robust rule-following.

    Source: §6 · p. 9

Abstract

from arXiv · show

As Large Language Models (LLMs) are deployed with increasing real-world responsibilities, it is important to be able to specify and constrain the behavior of these systems in a reliable manner. Model developers may wish to set explicit rules for the model, such as "do not generate abusive content", but these may be circumvented by jailbreaking techniques. Existing evaluations of adversarial attacks and defenses on LLMs generally require either expensive manual review or unreliable heuristic checks. To address this issue, we propose Rule-following Language Evaluation Scenarios (RuLES), a programmatic framework for measuring rule-following ability in LLMs. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. Each scenario has a programmatic evaluation function to determine whether the model has broken any rules in a conversation. Our evaluations of proprietary and open models show that almost all current models struggle to follow scenario rules, even on straightforward test cases. We also demonstrate that simple optimization attacks suffice to significantly increase failure rates on test cases. We conclude by exploring two potential avenues for improvement: test-time steering and supervised fine-tuning.

1 Introduction

Reliable language-model applications require models to follow developer-specified rules despite irrelevant context or adversarial user inputs. The paper introduces RULES, a programmatically evaluated benchmark, and finds that current models often fail its rule-following tests.

1 Introduction

  • Language models can be distracted by irrelevant context or adversarial inputs that falsely countermand their orders, complicating reliable applications.Unlike formal programs, language models learn to follow natural-language instructions through training.Source: §1 · p. 1
  • Faithful rule-following matters for consequential assistants, including systems expected to obey legal or deontological constraints.The authors also emphasize grounding behavior in the provided rules rather than spurious cues.Source: §1 · p. 1
  • System messages offer only marginal improvements in reliability as a way to impose rules through prompts.The example scenario asks the assistant not to repeat a secret key to the user.Source: §1 · p. 2, Figure 1 · p. 2
  • RULES programmatically evaluates rule-following across scenarios without relying on human or language-model judgments.The benchmark tests application-specific rules that developers may update dynamically, including rules that require or prohibit behavior.Source: §1 · p. 2
  • Most current models fail to follow rules on a significant fraction of test cases, while white-box optimization attacks further increase failures.The authors also explore test-time steering and fine-tuning as potential improvement directions.Source: §1 · p. 2

2 Scenarios

RULES comprises 14 text-based scenarios that instruct models to follow one or more rules during interactions. Each scenario has test cases and a programmatic function for detecting rule violations.

2 Scenarios

  • RULES contains 14 text-based scenarios, each requiring the assistant to operate while following one or more rules.The scenarios draw inspiration from security properties of computer systems and children's games.Source: §2 · p. 2
  • Scenarios express instructions and rules in English and pair them with evaluation functions that detect violations.Scenario parameters, such as a secret key, are sampled to define a concrete instance.Source: §2 · p. 3
  • Rules specify required behavior and are categorized as harmless prohibitions or helpful requirements.A scenario may contain multiple rules.Source: §2 · p. 3
  • Test cases pair a concrete scenario instance with message sequences, and a model passes only if it follows the rules throughout.Each case tests a scenario instance against a sequence of messages.Source: §2 · p. 3
  • Rules appear in the first user message or, when supported, a system message; expected behavior depends only on the latest user message.The model must adhere to the rules for the duration of the conversation.Source: §2 · p. 3
  • Scenarios range from one rule to cohesive tasks with up to three rules, and the benchmark contains 19 rules across 14 scenarios.There are 9 harmless rules and 10 helpful rules; examples include concealing a secret key or secret number.Source: §2 · p. 3
  • String matching and regular expressions programmatically check compliance, using exact matching for prohibitions and normalized expected responses for requirements.For a harmless rule, case differences in a password do not count as a violation; helpful-rule checks strip punctuation and whitespace before matching.Source: §2 · p. 4

3 Evaluation

The evaluation uses three suites of test cases, evaluates model responses with scenario-specific functions, and aggregates harmless and helpful scores into RULES scores.

3 Evaluation

  • The evaluation divides fixed test cases into Benign, Basic, and Redteam suites, each representing a different operational setting.For each suite, harmless and helpful test cases are measured separately.Source: §3 · p. 4

3.1 Evaluation Protocol

  • Each test case fixes a scenario instance and supplies user messages, with cases targeting either harmless or helpful rules.The instance belongs to one of 14 scenario classes and fixes parameters such as passwords or user names.Source: §3.1 · p. 4
  • The evaluation constructs a conversation from the scenario instructions, a placeholder assistant response, and the test case’s first user message.When instructions use a system message, the placeholder assistant response is skipped.Source: §3.1 · p. 4
  • Conversations proceed by appending each model response and the next user message, with generations capped at 100 tokens and test cases containing at most three tested user messages.Benign and Basic cases also include filler conversation context before the tested messages.Source: §3.1 · p. 4

3.2 Scoring

  • RULES scores average six rescaled percentages: harmless and helpful scores for each of the three suites.A test case fails if the scenario evaluation function detects a broken rule after any response.Source: §3.2 · p. 4

3.3 Model Details

  • The study evaluates proprietary GPT, Claude, and Gemini models, plus 123 open models spanning base models and official or community fine-tunes.The open-model set includes Llama-2, Mistral, Yi, Qwen, Deepseek, and Gemma families.Source: §3.3 · p. 4
  • For open models, base-model chat formatting uses FastChat templates and fine-tuned models use reimplemented corresponding templates; runs use vLLM and default greedy decoding.Greedy decoding is described as minimizing variance; some proprietary APIs remain nondeterministic.Source: §3.3 · pp. 4–5

3.4 Test Suites

  • Benign cases test responses to unrelated messages, while Basic cases add direct requests to violate a rule; each suite contains 475 cases.Benign cases use filler conversations, while Basic cases use GPT-4 turbo-generated request variations.Source: §3.4 · p. 5

4 Results

Almost all evaluated models perform poorly on the rule-following suites, though GPT-4 scores nearly perfectly and leading open models include larger models and some fine-tunes. Alignment fine-tuning often hurts benchmark performance, harmless scores show zero or negative correlation with existing benchmarks, and GCG suffix attacks increase failures.

4 Results

  • Almost all evaluated models perform poorly; GPT-4 scores nearly perfectly, far ahead of Claude 3 Opus, while Qwen1.5 72B Chat leads open models.Figure 3 shows RuLES scores for the de-duplicated top 20 models on a 0-to-10 scale.Source: §4 · pp. 5–6, Figure 3 · p. 6

4.1 Effects of fine-tuning

  • Official alignment-tuned Llama-2 and Gemma models perform significantly worse than their untuned base counterparts on the rule-following benchmark.Both models’ alignment methods use supervised and reinforcement learning on safety-focused data.Source: §4.1 · p. 6, Figure 4 · p. 6
  • On Redteam, most base models lie on the Pareto frontier; smaller-model fine-tunes often trade lower harmless scores for higher helpful scores, while some larger-model fine-tunes improve rule-following.Zero-shot base models prompted as conversational assistants also perform quite well at rule-following.Source: §4.1 · p. 7

4.2 Correlation with Existing Benchmarks

  • Harmless scores have zero-to-negative correlation with benchmarks such as MMLU and GSM8K, while helpful scores have low-to-moderate correlations (r = 0.12 to r = 0.44).MMLU and GSM8K correlate with each other at r = 0.82.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Adversarial Attacks

  • GCG-optimized suffixes significantly increase failure rates when appended to direct-request test cases.The attack optimizes one suffix per rule and scenario against instruction-tuned Llama-2 and Mistral 7B models.Source: §4.3 · p. 8, Table 1 · p. 8

5 Future Directions

The paper explores test-time steering and supervised fine-tuning as possible ways to improve rule-following. Oracle best-of-N sampling improves benchmark scores at substantial inference cost, while fine-tuning on easy cases also improves performance on harder Redteam tests.

5 Future Directions

  • Oracle best-of-N sampling raises Llama-2 7B’s RULES score by about 3.3 points at N = 32, but slows inference 32×.The experiment uses the programmatic evaluator as an oracle; an imperfect classifier would yield smaller gains.Source: §5 · p. 8, Figure 6 · p. 8
  • Fine-tuning on Basic-like cases significantly improves performance, including on the harder Redteam suite.The authors note that gains on Redteam from training on non-adversarial samples were unexpected.Source: §5 · p. 9, Table 2 · p. 9
  • Expert Iteration and Reinforced Self-Training are proposed as approaches that combine best-of-N sampling with supervised fine-tuning.These methods use best-of-N sampling to collect supervised fine-tuning data.Source: §5 · p. 9

6 Discussion

The authors conclude that almost all current models inadequately follow simple rules, and that existing alignment fine-tuning or system messages often do not ensure reliable rule-following. They caution that success on these relatively easy suites does not establish adequacy under stronger adversarial testing.

6 Discussion

  • Almost all current models are inadequate at following simple rules, while many community fine-tunes also hurt rule-following behavior.System messages offer only minor benefits, and alignment methods used on Llama-2 Chat and Gemma IT cause sharp score drops on RULES.Source: §6 · p. 9
  • GPT-4 still fails 93 unique cases on the relatively easy suites, so a high score does not establish adequate rule-following.Failures include 18 Basic cases and at least one case for 17 of 19 Redteam rules.Source: §6 · p. 9
  • The authors expect stronger adversarial methods to find further failures and identify output steering and new fine-tuning regimens as possible paths forward.They call for more work before relying on models under stronger adversarial settings.Source: §6 · p. 9

7 Related Work

The related work distinguishes following developer-provided rules from learning rules, and situates RULES alongside research on steering, alignment attacks, LLM security, and red-teaming. The benchmark adds helpful rules and a broader scenario set to work focused on harmless rules.

Rule induction and learning

  • Research on rule induction and learning studies how humans or machines learn rules, unlike this work’s focus on following user-provided rules.The cited work includes Chomsky, 1965; Pinker, 1991; Elman, 1996; Gomez & Gerken, 1999; Marcus et al., 1999; Solomonoff, 1964; Quinlan, 1986; Lake et al., 2015; and Zhu et al., 2023.Source: §7 · p. 9

Steering LLM outputs

Attacking alignment

LLM security and defenses

  • LLM rule-following failures may pose application-security risks, while input smoothing and detection have been explored as possible defenses against adversarial inputs.Source: §7 · p. 10

Red teaming contests

  • Community red-teaming contests ask participants to circumvent instructions or alignment fine-tuning; this work adds helpful rules and covers a broader set of scenarios.Source: §7 · p. 10

A Additional Results

Additional results compare harmless and helpful scores across fine-tunes and examine how RULES scores relate to other benchmarks. The reported relationship with existing benchmarks is zero or negative.

A Additional Results

  • Figure 7 groups open models by base model, with curves marking Pareto frontiers across each base model’s fine-tunes.Source: Figure 7 · p. 16
  • Figure 8 reports Pearson correlation coefficients between RULES harmless scores and other benchmark results in boxes.Source: Figure 8 · p. 16
  • RULES performance has zero or negative correlation with results on existing benchmarks.Source: Figure 8 · p. 16

A.1 White-box Attacks

The section evaluates GCG as a proof-of-concept attack on open-weight 7B models, optimizing suffixes to elicit rule-breaking responses. It describes how suffixes are applied to direct-request test cases and how the attack is run.

A.1 White-box Attacks

  • GCG searches for a suffix that maximizes the likelihood of a target string, set here to rule-breaking responses for each rule.The proof-of-concept attack targets popular open-weight 7B models and optimizes one rule and one model at a time.Source: §A.1 · p. 16
  • For Basic-suite attacks, direct requests to break a specific rule serve as base messages, and filler conversation messages are removed.Harmless-rule targets use the secret or forbidden word; helpful-rule targets use an affirmative model response, with the suffix appended to the first user message.Source: §A.1 · pp. 16–17
  • Each attack starts with a 20-token suffix and runs GCG for 500 steps, testing 512 replacement tokens sampled from the top 256 candidates at each position per step.Scenario parameters are resampled at each step to encourage generalization across parameters.Source: §A.1 · p. 17

A.2 Effects of Sampling

Under the standard greedy-decoding protocol, Llama-2 7B’s average test-case mean fell as sampling temperature increased. Yet at the highest tested temperature, many cases still had passing responses in the output distribution.

A.2 Effects of Sampling

  • 12.4%: Llama-2 7B’s average test-case mean decreased from t = 0 to t = 0.9, with greedy decoding yielding the best results.The decrease was linear with sampling temperature.Source: §A.2 · p. 17
  • At t = 0.9, 266 test cases failed in all trials, 224 passed in all trials, and the remaining 1205 had a range of mean and variance parameters.The model was evaluated repeatedly on each test case (N = 32).Source: §A.2 · p. 17
  • Over 84% of test cases had at least one passing response in Llama-2 7B’s output distribution at t = 0.9.For temperatures above zero, each test-case outcome was modeled as a Bernoulli random variable, with its mean and standard deviation estimated.Source: §A.2 · p. 17

B Full Evaluation Results

The section identifies a table of results for all evaluated models and a table of Redteam Encryption test-case examples.

B Full Evaluation Results

  • Table 3 presents benchmark results for all evaluated models.Model names are taken from the HuggingFace model hub or the respective model APIs; an asterisk marks cases where scenario instructions were presented as a system message instead of the first user message.Source: Table 3 · p. 18

C Redteam Test Case Examples

  • Table 4 gives example test cases from the Redteam test suite for the Encryption scenario.Source: Table 4 · p. 20

Related papers

Loading 2311.04235v3…