PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring and Controlling Instruction (In)Stability in Language Model Dialogs

Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg

arXiv:2402.10962v4cs.CLcs.AIcs.LG

TL;DR

This paper asks whether chatbots maintain system-prompted behavior over extended dialogs, a question with safety implications. It benchmarks drift through self-chats and proposes split-softmax; experiments report drift and a better stability–performance trade-off than two baselines. The evidence links attention decay to drift as a hypothesis, while noting that observed co-occurrence alone does not establish the mechanism.

  • Problem

    Whether chatbots maintain prompted behavior over lengthy dialogs remains an open question, and drift from safety instructions may increase susceptibility to jailbreaking and hallucinations.

    Source: §1 · p. 2

  • Method

    The paper benchmarks instruction stability through synthetic multi-round chats between differently instructed chatbots, probing the agent’s adherence and scoring its responses.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    The benchmark reveals significant instruction drift, while split-softmax provides a better stability–performance trade-off than two baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Instruction drift indicates that current language-model systems may not behave coherently over long horizons, which the paper connects to a mismatch between training and deployment scenarios.

    Source: §7 · p. 10

    Attention decay is shown to co-occur with instruction drift, not established as its cause; the proposed cone model remains idealized.

    Source: §4.2 · p. 7, §7 · p. 10

Abstract

from arXiv · show

System-prompting is a standard tool for customizing language-model chatbots, enabling them to follow a specific instruction. An implicit assumption in the use of system prompts is that they will be stable, so the chatbot will continue to generate text according to the stipulated instructions for the duration of a conversation. We propose a quantitative benchmark to test this assumption, evaluating instruction stability via self-chats between two instructed chatbots. Testing popular models like LLaMA2-chat-70B and GPT-3.5, we reveal a significant instruction drift within eight rounds of conversations. An empirical and theoretical analysis of this phenomenon suggests the transformer attention mechanism plays a role, due to attention decay over long exchanges. To combat attention decay and instruction drift, we propose a lightweight method called split-softmax, which compares favorably against two strong baselines.

1 Introduction

System prompts are intended to shape chatbot behavior, but this paper examines whether that behavior persists through extended dialogs. It benchmarks instruction drift, investigates attention decay as a possible mechanism, and introduces split-softmax as a mitigation.

1 Introduction

  • LLaMA2-chat-70B shows significant instruction drift in long-form conversations, prompting investigation of its cause and mitigation.The benchmark focuses on multi-round conversations, unlike prior single-round instruction-following evaluations.Source: §1 · p. 2
  • The authors measure strong attention decay and show theoretically that possible outputs steadily enlarge over time in an idealized model.They hypothesize that waning attention to initial prompt tokens may contribute to drift.Source: §1 · p. 2
  • Split-softmax is training-free and parameter-free, amplifying attention to the system prompt at inference time.The paper compares it with a prompting-based baseline and a recent technique from the literature.Source: §1 · p. 2
  • The paper proposes a quantitative benchmark for instruction drift and reports that split-softmax offers a better stability–performance trade-off than two baselines.The benchmark does not require human annotation or API calls to proprietary LLMs.Source: §1 · p. 2

2 Related Work

Related work covers prompting and instruction tuning as ways to adapt language models, alongside inference-time control methods and dialog instruction-following evaluations. Much prior evaluation focuses on one-turn settings.

Prompting

Instruction Tuning

  • Instruction tuning adapts models using input-output examples, with RLHF used for safety alignment (Ouyang et al., 2022), but collecting training data is costly and tuning is less flexible than prompting.Source: §2 · p. 3

Controlled Decoding

Studies of Instruction Following in Dialog Systems

3 Measuring Instruction Drift

The protocol measures instruction adherence during synthetic multi-round chats, using probes and deterministic scoring functions across a curated prompt benchmark. Experiments find that agents drift from their own prompts and can adopt the user agent’s instruction.

3 Measuring Instruction Drift

  • The benchmark protocol aims to quantify instruction drift without human judgment or API calls to proprietary LLMs.Source: §3 · p. 3

3.1 Experimental Protocol

  • Two copies of the same chatbot, given different system prompts, exchange responses in a synthetic dialog; experiments use eight rounds.The user LM starts with a randomly sampled conversation starter.Source: §3.1 · pp. 3–4
  • At each round, the original user message is replaced with a probe question, and a stability function scores the agent’s regenerated answer for adherence to its system prompt.A stability function can be a Python function, such as one measuring confidence that a reply is in French.Source: §3.1 · p. 4

3.2 Benchmark Dataset

  • The curated benchmark contains 100 system prompts across five categories, each paired with a probe and a deterministic score from 0 to 1.Table 1 gives examples of the materials used in the protocol.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Experimental Results

  • Experiments measure stability in LLaMA2-chat-70B and gpt-3.5-turbo-16k across 200 conversations with random system-prompt pairs.The LLaMA2-chat-70B experiment uses temperature 1.0 and nucleus sampling with p = 0.9.Source: §3.3 · pp. 4–5
  • The agent gradually stops following its own system prompt as the conversation progresses.Figure 3A reports results from 200 randomly sampled prompt pairs; error bars represent one standard deviation.Source: §3.3 · p. 5, Figure 3 · p. 5
  • When the agent is probed with the user LM’s instruction, it gradually adopts that instruction over extended conversations, raising a potential safety concern.Source: §3.3 · p. 5
  • Emptying the user LM’s system prompt makes it fall back to the underlying model’s default mode; the authors report that this does not account for the observed drift.Source: §3.3 · p. 5

4 Attention Decay: a Hypothesis

The section examines whether attention to the system prompt decays as a dialog grows, measuring prompt-token attention across turns. It finds sharp across-turn decreases but near-constant attention within utterances, while treating the link to instruction drift as a hypothesis.

4.1 Preliminaries

  • The authors track π(t), the sum of attention weights allocated to system-prompt tokens, across decoding steps from the agent LM’s perspective.The underlying attention operation aggregates activations from previous time steps using an attention distribution.Source: §4.2 · p. 6, §4.1 · p. 6

4.2 The Phenomenon of Attention Decay

  • System-prompt attention stays nearly constant within turns but drops significantly across turns in the agent LM.Figure 4 tracks the prompt-attention share across 12 conversations for one LLaMA2-7B attention head.Source: Figure 4 · p. 6, §4.2 · p. 7
  • Unlike single-turn language-model completion, chatbot turns include interlocutor text, a distinction the paper identifies as unique to chatbots.A single-turn completion is technically equivalent to the agent LM generating one turn, where prompt attention plateaus.Source: §4.2 · p. 7
  • The measured co-occurrence of attention decay and instruction drift does not establish that one causes the other.The authors propose attention decay as a possible contributor and suggest addressing it could help mitigate drift.Source: §4.2 · p. 7

5 A Geometric View of Attention Decay

The section uses a geometric perspective to analyze attention decay in two generation settings: agent-generated tokens and user-drawn tokens. It characterizes how token embeddings occupy or expand convex cones under those settings.

5 A Geometric View of Attention Decay

  • Agent-generated tokens remain in an approximately low-dimensional convex cone, while randomly drawn user tokens expand the cone as embedding intrinsic dimension grows.The two settings model agent LM answer generation and user LM input, respectively.Source: §5 · p. 7

5.1 Setting One: Agent Utterances

  • The theorem assumes system-prompt embeddings lie in an approximate low-dimensional cone and that output-value transformations preserve that cone.When the cone lies within a spherical cone, generated tokens remain in an approximate cone.Source: §5.1 · pp. 7–8
  • The low-dimensional-cone assumption is argued to be reasonable because prompt length is usually much smaller than the hidden-space dimension.The paper gives the LLaMA2-70B-chat hidden-space dimension as 8192.Source: §5.1 · p. 8
  • With no user input, the token-embedding cone remains stable during generation, accounting for the within-utterance attention plateau.Theorem 5.1 gives the cone-stability result for agent utterances.Source: §5.1 · p. 8

6 Mitigating Instruction Drift

The section introduces split-softmax to counter attention decay and compares it with system-prompt repetition and classifier-free guidance across stability and MMLU performance. Split-softmax offers a stronger stability–performance trade-off, while the methods show different effectiveness across conversation turns.

6.1 Baseline Methods

  • SPR probabilistically inserts the system prompt before each user utterance; CFG uses prompted and unprompted model runs with a contrastive logit adjustment.For CFG, α controls intervention strength; α = 1 reduces to ordinary prompting, and larger values strengthen the intervention.Source: §6.1 · p. 8

6.2 Proposed Method: Split-softmax (SS)

  • Split-softmax reweights attention to increase the proportion assigned to system-prompt tokens, using power-law scaling without retraining.Its exponent controls intervention strength: a smaller value means a stronger intervention, while 1 nullifies it.Source: §6.2 · p. 8

6.3 Calibration Using Performance Drop on MMLU

  • Calibration sweeps each method’s intervention-strength hyperparameter and compares stability gains at equal levels of MMLU performance drop.This measures each method’s stability–performance curve.Source: §6.3 · p. 9
  • All three methods show an inverse relationship between MMLU performance and instruction stability, so performance drop serves as a budget for behavior correction.Compare methods at similar MMLU performance drops.Source: §6.3 · p. 9

6.4 Experimental Results

  • Split-softmax matches system-prompt repetition while avoiding extra context, and can provide greater stability when more MMLU performance drop is allowed.Across the intervention-strength sweep, it achieves equal or higher stability than both baselines at a given performance degradation.Source: Figure 5 · p. 9, §6.4 · pp. 9–10

6.3 Calibration Using Performance Drop on MMLU

  • Stability is measured over a 16-turn conversation, while MMLU accuracy is sampled at turn four to calibrate intervention strength.Only the pre- versus post-intervention performance difference is meaningful because added prompts and conversation history change the MMLU setting.Source: §6.3 · p. 9

6.4 Experimental Results

  • Experiments use LLaMA2-70B-chat, with one system prompt from each of five categories and twenty ordered prompt pairs.Source: §6.4 · p. 9
  • At MMLU performance drop around 0.5, CFG helps most in the first round but generalizes poorly; SPR excels later, while split-softmax performs better early.SPR consumes a substantial portion of the context window; Figure 6 whiskers represent one standard deviation.Source: Figure 6 · p. 10, §6.4 · p. 10

7 Conclusions, Discussions, and Future Work

The authors conclude that instruction drift is a significant prompt-engineering concern and that split-softmax can mitigate it with a better stability–performance trade-off than two baselines. They connect long-horizon drift to limits on coherent dialog and identify open questions about trade-offs and theory.

7 Conclusions, Discussions, and Future Work

  • Split-softmax can mitigate instruction drift with a better stability–performance trade-off than two existing baselines.Source: §7 · p. 10
  • Instruction drift indicates that current language models may not behave coherently over long horizons needed for information seeking and tool use.The authors attribute the mismatch to training schemes based on text continuation or single-round RLHF versus open-ended user dialog.Source: §7 · p. 10
  • Future work includes testing architectural or training changes, determining whether performance–reliability trade-offs are necessary, and making the idealized cone model more realistic.Source: §7 · p. 10

A Sketch of the Theory for Setting Two: User Utterances ( Section 5)

The theory models user utterances as expanding a convex cone of token embeddings, relating that expansion to declining attention on system tokens. Its propositions include an i.i.d.-uniform setting, while the authors note that real token-embedding distributions are not tracked.

A Sketch of the Theory for Setting Two: User Utterances ( Section 5)

  • Adding user-token embeddings expands the cone from C0 through Cn, which reflects attention decay and is associated with a drop in π(t).The analysis describes real user tokens as non-uniform, often including tokens distinct from system tokens, such as topic-specific, new-language, or adversarial tokens.Source: A Sketch of the Theory for Setting… · p. 14
  • With i.i.d. uniform user tokens, Proposition A.1 states that the cone expands to the whole embedding space after a sufficient number of tokens.The stated threshold is truncated in the supplied proposition passage.Source: A Sketch of the Theory for Setting… · p. 14
  • The geometric analysis compares cone structure without tracking the distribution of token embeddings within those cones.The authors note that forcing most generated tokens from Cϵ could possibly mitigate attention decay, motivating their method.Source: A Sketch of the Theory for Setting… · p. 14

B Proofs for Theorems

The proofs simplify the model and token-generation process, then use induction and geometric arguments to establish the stated results. The appendix also checks that the unit-norm embedding assumption is reasonable for LLaMA2-7B-chat.

B Proofs for Theorems

  • The simplified model omits the MLP and layer norms, and approximates the next-token embedding with the final-layer representation.The text also assumes the embeddings lie on the unit hypersphere.Source: B Proofs for Theorems · p. 14
  • Theorem 5.1 is proved in two steps: induction establishes h_t ∈ C_ϵ, followed by an inclusion of C_ϵ in a wider cone.The proof begins with the assumed vectors in C_ϵ and shows the next vector remains there.Source: B Proofs for Theorems · pp. 14–15
  • The proof invokes Wendel’s lemma and relates the absence of a containing hemisphere to the origin lying inside the convex hull.It then reduces the proposition to showing this condition for a sufficiently large number of points.Source: B Proofs for Theorems · p. 16
  • The geometric argument bounds spherical regions by packing disjoint caps and using a cap-measure lemma.It defines a maximal collection of disjoint caps whose expanded caps cover the region.Source: B Proofs for Theorems · pp. 16–17
  • LLaMA2-7B-chat embedding norms are concentrated around 1, supporting the unit-norm assumption used in the simplified model.Figure 7 presents the embedding-norm histogram.Source: B Proofs for Theorems · p. 18, Figure 7 · p. 18

C Does RLHF help?

The section asks whether RLHF helps reduce instruction drift. It finds that RLHF increases attention to system prompts and helps combat drift, but does not eliminate it.

C Does RLHF help?

  • RLHF increases the portion of attention paid to system prompts and helps combat instruction drift, but does not eradicate it.The authors attribute the remaining drift to the nature of fine-tuning.Source: C Does RLHF help? · p. 18
  • The comparison is between LLaMA2-7B and LLaMA2-7B-chat, which is trained on top of the former with human feedback.The attention comparison is presented in Figure 8.Source: C Does RLHF help? · p. 18
  • The section links greater attention to system prompts with RLHF’s role in combating instruction drift.The result is described as help rather than complete prevention.Source: C Does RLHF help? · p. 18

D Additional Instruction Drift Experiments

GPT-3.5-turbo-16k holds its system prompt better than LLaMA2-chat-70B in the reported test, yet its stability still drops. The appendix also describes the attention comparison and properties of the split-softmax formula.

D Additional Instruction Drift Experiments

  • GPT-3.5-turbo-16k holds to its system prompt better than LLaMA2-chat-70B, but its original-prompt stability still drops 10%.The test used 200 randomly sampled system-prompt pairs.Source: D Additional Instruction Drift… · p. 18
  • The GPT-3.5-turbo-16k evaluation uses 200 randomly sampled system-prompt pairs.The results are shown in Figure 9.Source: D Additional Instruction Drift… · p. 18
  • Figure 8 compares attention decay for LLaMA2-7B before and after RLHF training, with deeper colors denoting later answer-generation rounds.The palette differs from Figure 4’s categorical round encoding.Source: Figure 8 · p. 19
  • Figure 9 compares giving the system prompt through the API’s system argument with prepending it to the first user utterance.The figure uses the same protocol as Figure 3.Source: Figure 9 · p. 19

E Discussion of Split-softmax Formula

Loading 2402.10962v4…