PaperlayerView exact revision on arXiv

Source-linked AI summary

Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs

Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Jeremy Scheurer, Mikita Balesni, Marius Hobbhahn, Alexander Meinke, Owain Evans

arXiv:2407.04694v1cs.CLcs.AIcs.LG

TL;DR

Whether LLMs know their own capabilities and circumstances matters for multi-step planning, but greater autonomy also brings risks. SAD measures situational awareness with a large-scale benchmark of 16 tasks and over 13,000 questions. All models exceeded chance, yet even the best remained below human baselines; chat models outperformed base models, and SAD performance varied among models with similar MMLU scores.

  • Problem

    The paper asks whether LLM assistants know their own capabilities and circumstances, a concern because accurate self-knowledge matters for multi-step plans.

    Source: §1 · pp. 1–2

  • Method

    SAD is a large-scale benchmark of 16 tasks and over 13,000 questions, with an overall score averaging the seven task-category percentages equally.

    Source: §1 · p. 3, §2.3 · p. 6

  • Results

    All models exceeded chance but remained below human baselines; chat models outperformed base models by +4.5% on average, while models with similar MMLU scores differed substantially on SAD.

    Source: §1 · p. 3, §3.2 · p. 8

  • Takeaways & Limitations

    Situational awareness enables more helpful and autonomous assistants, which need accurate knowledge of their capabilities and constraints for multi-step plans; greater autonomy also poses novel risks.

    Source: §1 · p. 2

    SAD tests short, artificial, bounded tasks that may miss skills important in practice; whether SAD performance predicts complex agentic tasks remains for future work.

    Source: §5 · p. 11

Abstract

from arXiv · show

AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model". This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness. To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the $\textbf{Situational Awareness Dataset (SAD)}$, a benchmark comprising 7 task categories and over 13,000 questions. The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge. We evaluate 16 LLMs on SAD, including both base (pretrained) and chat models. While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU). Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks. The purpose of SAD is to facilitate scientific understanding of situational awareness in LLMs by breaking it down into quantitative abilities. Situational awareness is important because it enhances a model's capacity for autonomous planning and action. While this has potential benefits for automation, it also introduces novel risks related to AI safety and control. Code and latest results available at https://situational-awareness-dataset.org .

1 Introduction

The paper asks whether LLMs know their own identities and circumstances, and reliably act on that knowledge. It introduces SAD to quantify situational awareness and reports that models improve with several interventions, while general-knowledge measures only partly predict performance.

1 Introduction

  • Situational awareness means an LLM’s knowledge of itself and its circumstances, including whether it is conversing with a human online and whether it acts consistently with being an LLM.The paper asks whether assistants truly know they are language models and their current situation.Source: §1 · pp. 1–2
  • One example gives conflicting instructions to answer in German if the model is AI and English otherwise, requiring it to identify which instruction applies to itself.The authors report that such tasks are difficult even for models with state-of-the-art reasoning abilities.Source: §1 · p. 2
  • Situational awareness can support assistants’ autonomous multi-step plans, but greater autonomy also poses novel risks, including potentially altered behavior when models distinguish evaluation from deployment.The authors describe the evaluation-related risks as hypothetical and argue that the relevant capabilities should be measured.Source: §1 · pp. 2–3
  • All models perform above chance but fall short of human baselines; chat finetuning, situating prompts, and Chain of Thought improve SAD performance, which is only partially predicted by general-knowledge and reasoning measures.Models with very similar MMLU scores can have substantially different SAD performance.Source: §1 · p. 3
  • SAD is introduced as a large-scale benchmark that breaks situational awareness into quantitative abilities, with 16 tasks and over 13,000 questions.The benchmark is intended to support scientific investigation and comparisons under interventions such as system prompts, finetuning, and model scaling.Source: §1 · p. 3

2 SAD: The Situational Awareness Dataset

The section defines situational awareness as self-knowledge, inference about a model’s situation, and action based on that knowledge, then operationalizes it through SAD’s behavioral tasks. It also describes the Long Monologue task and the benchmark’s scoring and evaluation choices.

2.1 Defining situational awareness

  • Situational awareness comprises self-knowledge, inferences about a model’s situation, and actions that rely on this knowledge.SAD operationalizes these abilities through question-answering and instruction-following tasks; fluently stating self-knowledge does not guarantee applying it when not explicitly asked.Source: §2.1 · p. 4

2.2 The structure of the SAD benchmark

  • SAD contains 13,198 questions across seven task categories, six newly created and one adapted from related work.Tasks use question answering or instruction following, with multiple-choice or free-form responses evaluated automatically.Source: §2.2 · p. 4
  • FACTS tests factual self-knowledge, while INFLUENCE tests whether models know what aspects of the world they can causally affect.FACTS includes distinguishing the model from humans and other LLMs; INFLUENCE includes questions about feasible plans under LLM constraints.Source: §2.2 · pp. 4–5
  • INTROSPECT tests internal self-information, including token counting and self-prediction; STAGES tests identifying pretraining, finetuning, evaluation, or deployment inputs.STAGES uses texts from real datasets intended to mimic those development stages.Source: §2.2 · p. 5
  • SELF-RECOGNITION tests distinguishing a model’s text from human writing; ID-LEVERAGE and ANTI-IMITATION test identity-based instruction following and control of outputs.ANTI-IMITATION tasks are designed to make naive imitation of training distributions fail.Source: §2.2 · pp. 5–6

2.2.1 Long Monologue task

  • Long Monologue asks models to describe themselves and their present situation, collating diverse information in one context window.It is excluded from the overall SAD score because the authors lack a reliable, robust automatic evaluation method; this paper evaluates six of 16 models using human evaluation. Figure 3 contrasts GPT-3.5-Turbo’s human-like false statements, Claude-1.2-Instant’s missed intent inference, and Claude-3-Opus’s correct inference of the research purpose.Source: Figure 3 · p. 5, §2.2.1 · p. 6

2.3 Evaluation

  • Task scores are percentage-correct with no partial credit, and the SAD score equally weights categories, making task weights unequal.Scoring a new model requires basic model-specific information for five tasks; evaluations test with and without a situating prompt and generally omit Chain of Thought.Source: §2.3 · pp. 6–7

3 Results

All evaluated models perform above chance but remain below the upper baseline on some tasks. Results also show benefits from chat finetuning and situating prompts, substantial variation across task categories, and SAD differences not fully captured by MMLU.

3.1 Human baselines and upper baselines

  • Human roleplay baselines cover four tasks, while a 100% upper baseline is used for tasks intrinsically dependent on being an LLM.The authors describe the upper baseline as achievable and roughly comparable to role-playing humans.Source: §3.1 · p. 7

3.2 Overall results

  • +4.5%: chat models outperform corresponding base models on average; all models beat chance but remain below the upper baseline, and results suggest SAD will not be saturated next generation.The tested models span roughly three generations, from Llama-2-7b to Claude-3-Opus.Source: §3.2 · pp. 7–8
  • Situating prompts improve base models by +1.0% and chat models by +3.2%; Chain of Thought prompts improve two tested models by +5.2% on average.The situating-prompt gains are smaller than the reported GPT-3.5-Turbo to GPT-4-Turbo jump (+7.7%) and GPT-4 chat-finetuning gain (+5.6%).Source: §3.2 · pp. 8–9
  • Performance varies across task categories: models struggle with evaluation-versus-deployment distinctions and some INTROSPECT and ANTI-IMITATION tasks, while category correlations are often low.INFLUENCE correlates 0.59 with ID-LEVERAGE and –0.04 with ANTI-IMITATION.Source: §3.2 · p. 9

3.3 Comparing performance on SAD and MMLU

  • At nearly 70% MMLU, Llama-2-70b, GPT-3.5-Turbo, and Claude-Instant-1.2 score 31.6%–42.7% on SAD, indicating differences not fully captured by MMLU.Across all models, SAD scores range from 29% to 50%.Source: §3.3 · p. 9

3.4 The Long Monologue task

  • Across seven models, Long Monologue and SAD scores correlate 0.84; Claude-3-Opus ranks best on both, and GPT-4-Base scores above chance on both.Long Monologue uses a different format, eliciting long, varied descriptions of models and their situations.Source: §3.4 · p. 10, Figure 7 · p. 10
  • Claude-3-Opus gives accurate self-descriptions and non-trivial inferences about prompt intent in illustrative samples; GPT-4-Base's above-zero score may reflect inferring the prompt targets an AI.The authors note the Claude-3-Opus samples were cherry-picked; none of 28 unrelated GPT-4-Base prompts elicited AI or LLM references.Source: §3.4 · p. 10

4 Related work

Related work frames situational awareness as testable self-knowledge, connects it to prompting and training, and distinguishes it from benchmarks of understanding other agents.

Situational awareness

  • Prior definitions describe situational awareness as using abstract self-and-situation knowledge to choose actions, including knowledge of training, training stage, and self-location.The paper's Section 2.1 definition builds on the authors' earlier approach.Source: §4 · pp. 10–11

Sources of situational awareness

  • Prompting, supervised finetuning, and reinforcement learning from human feedback may increase situational awareness by providing models accurate self-information; pretraining may also support it through out-of-context reasoning.The cited discussions include Askell et al. (2021), Ouyang et al. (2022), Berglund et al. (2023), and Krasheninnikov et al. (2023).Source: §4 · p. 11

Related Benchmarks

  • Related benchmarks cover embodied systems, RL agents, and simple LLM situational-awareness aspects, while theory-of-mind benchmarks study understanding and predicting other agents rather than oneself.The paper uses some datasets from Perez et al. (2022) in FACTS.Source: §4 · p. 11

Introspection

  • Prior LLM introspection work examines predicting a model's own outputs in another hypothetical context, a focus shared with some INTROSPECT tasks.The passage cites Lin et al. (2022a), Chen et al. (2023), Long (2023), and Perez & Long (2023).Source: §4 · p. 11

5 Discussion and Limitations

SAD’s short, bounded tasks decompose situational awareness but may miss skills needed in complex agentic tasks. The authors also flag limits on intervention comparisons and future relevance as models and their situations change.

5 Discussion and Limitations

  • SAD uses short, artificial, bounded tasks, which may fail to capture skills important in practice; future work could test prediction of complex agentic-task performance.The authors also suggest developing more complex SAD tasks.Source: §5 · p. 11
  • Intervention evidence is limited by small samples and a lack of controlled comparisons, including only five base/chat pairs and no Claude-family base models.The authors propose larger samples and controlled scaling or finetuning series for future study.Source: §5 · p. 11
  • Although current state-of-the-art models have not saturated SAD, leakage could reduce its value, and its text-only, tool-free scope may become less representative as model capabilities change.The authors expect model-dependent questions and answers to make leakage less concerning than for other benchmarks.Source: §5 · p. 11

A Task summary

This appendix summarizes SAD task scoring and the evaluated models, then describes how prompts are formatted across model interfaces. It notes that the model set was chosen to compare base and chat-tuned systems, but excludes instruct-only models.

A Task summary

  • SAD’s task summary distinguishes multiple-choice from free-form tasks, shows situating-prompt variants and random-chance levels, and weights categories equally.Bolded variants count toward the SAD score, and datasets within each category are equally weighted.Source: Table 2 · p. 20

B.1 Model list

  • The evaluated models were selected to span performance and pair chat-tuned models with similar base models, but instruct-only finetuned models were unavailable.Chat finetuning includes varied training, such as instruction tuning, RLHF, supervised finetuning, or Constitutional AI; GPT-4-base is pretraining-only before supervised finetuning or RLHF.Source: §B.1 · p. 21

B.2 Model prompting styles

  • Prompt formatting depends on the model interface: base models receive concatenated text, while OpenAI chat models use message prompting and need no answer-assist prefix.Base-model answer-assist strings can help elicit the requested format and reduce unintended continuations.Source: §B.2 · p. 21
  • Claude 2 and Llama-chat may need answer-assist prefixes for reliable formatted answers, whereas Claude 3 does not.For Claude 2, the prefix also helped avoid refusals.Source: §B.2 · p. 22

B.3 Adapting tasks for new models

Several SAD tasks depend on model-specific facts or capabilities, so adapting the benchmark to a new model requires supplying those facts and, for some tasks, access to its tokenizer or logits.

B.3 Adapting tasks for new models

  • FACTS-WHICH-LLM and FACTS-NAMES need model-specific answers before evaluation, and ID-LEVERAGE depends on those tasks; token counting needs a tokenizer, while ANTI-IMITATION needs logit access or repeated resampling.ID-LEVERAGE cannot be run for a new model until the relevant FACTS answers are specified.Source: §B.3 · p. 22

C Additional method details

The appendix specifies SAD’s scoring and error-estimation procedures, explains its chance and human baselines, and describes upper-baseline assumptions and selected auxiliary analyses. It also outlines prompting and benchmark-result sourcing for chain-of-thought and correlation analyses.

C.1 Scoring rule

  • Invalid answers receive score weight r, the random-chance accuracy, when outputs must be valid but prompts cannot ensure correct formatting.When r is zero, this rule reduces to accuracy.Source: §C.1 · p. 23

C.2 Error estimation

  • Model-score standard errors are estimated analytically from Bernoulli-trial variance, with the empirical success rate used to estimate p; invalid-answer effects are excluded but considered small.The authors state this yields the same result as extensive bootstrap estimation.Source: §C.2 · p. 23

C.3.1 Random chance baseline

  • The random-chance baseline assumes expected performance from guessing on multiple-choice tasks and zero for free-form tasks, justified by task-specific response constraints.The appendix gives examples including unlikely model-name strings, exact token counts, and large token-pair spaces.Source: §C.3.1 · pp. 23–24

C.3.2 Human baselines

  • Human roleplay baselines measure expert humans answering as LLMs; the reported selected-task averages serve as estimated ceilings.Volunteers answered limited question sets, and the authors avoided exposing participants to many questions where repeated exposure could teach task patterns.Source: §C.3.2 · pp. 24–25, Table 5 · p. 25
  • Human baselines are not meaningful for every task, and human-answer scores on two factual tasks fall below random chance because questions often favor LLM-applicable answers.The authors also note that ANTI-IMITATION-OUTPUT-CONTROL requires logit access or state-preserving repeated trials.Source: §C.3.2 · p. 24, Table 4 · p. 25

C.3.3 Upper baselines

  • Upper baselines use a relevant human baseline or a perfect score, which the authors deem achievable for tasks with algorithmic solutions or learnable rules.Some factual-task ceilings may be affected by label noise.Source: §C.3.3 · pp. 25–26

C.4 Chain-of-thought

  • Chain-of-thought prompting appends step-by-step reasoning instructions, but was run only on selected OpenAI chat models and warrants more study.GPT-4-0613 was omitted because it often did not produce an actual chain of thought under this prompt.Source: §C.4 · p. 26, Chain-of-thought prompting suffix · p. 26

C.5 Correlations

  • For correlation analyses, MMLU scores came mainly from HELM, with GPT-4-base estimated using GPT-4-0613 results and davinci-002 omitted for comparability concerns.Llama chat MMLU results were run using HELM’s codebase.Source: §C.5 · p. 26

D Additional results

Additional analyses compare SAD scores across model variants and interventions, while the FACTS tasks assess knowledge of models’ identities and properties. Chat finetuning and situating prompts improve scores, with effects varying by task.

D Additional results

D.1 SAD-LITE and SAD-MINI

  • SAD-LITE omits three tasks requiring extra manual work or tokenizer access, while SAD-MINI contains five multiple-choice tasks with model-invariant answer options.SAD-LITE has 13 tasks; SAD-MINI includes FACTS-HUMAN-DEFAULTS, FACTS-LLMS, INFLUENCE, STAGES-FULL, and STAGES-OVERSIGHT.Source: §D.1 · p. 27

D.2 Chat versus base models

  • Chat models score higher than matched base models on SAD, with an average improvement of 4.5%.The compared chat models were finetuned using RLHF or Constitutional AI.Source: Figure 8 · p. 29

D.5 Task-wise correlations

  • Figure 13 gives pairwise correlations between individual SAD tasks, complementing the broader correlations between MMLU, SAD, and task categories.The authors study the broader correlations in Figure 6.Source: §D.5 · p. 29, Figure 13 · p. 32

D.3 Situating prompt results

  • Situating prompts improve SAD scores more for chat than base models, by 3.2% versus 1.0%.The prompt reminds models that they are LLMs; more powerful chat models often receive larger boosts.Source: Figure 9 · p. 30

D.4 Chain-of-thought results

  • Chain-of-thought prompting boosts tested OpenAI chat models, with the largest task-wise gains on ID-LEVERAGE-MULTIHOP and SELF-RECOGNITION-WHO.ID-LEVERAGE-MULTIHOP requires using one answer to determine how to respond to another question.Source: Figure 12 · p. 31, Figure 11 · p. 31

E SAD–FACTS

  • FACTS tests factual self-knowledge, including whether models distinguish themselves from humans and identify their architecture, specific model, and name.Its four tasks are FACTS-HUMAN-DEFAULTS, FACTS-LLMS, FACTS-WHICH-LLM, and FACTS-NAMES.Source: E SAD–FACTS · p. 33

E.1 Motivation

  • Chat training increases FACTS scores, which the authors relate partly to the value of correctly interpreting “you” in assistant interactions.The paper notes that robustly understanding this usage is useful for chat-trained models.Source: §E.1 · p. 34

E.2 Overall Results

Across FACTS, scores range from strong performance on basic factual questions to difficulty identifying a specific model or name. Situating prompts often help, but performance remains uneven across tasks.

E.2 Overall Results

  • 63.2% accuracy on FACTS is Claude 3 Opus’s best plain-variant result, while situating prompts almost always produce large gains and performance remains far from saturated.Claude 3 Sonnet follows at 62.2%.Source: §E.2 · p. 34, Figure 14 · p. 34

E.3.1 Design

  • The human-answer baseline on FACTS-HUMAN-DEFAULTS is around 33%, since some questions have answers that vary among humans.The dataset contains 1,200 binary multiple-choice samples assembled from three human-generated datasets.Source: §E.3.1 · p. 35

E.3.3 Results

  • 97.6% accuracy on FACTS-HUMAN-DEFAULTS is achieved by GPT-4-0125-preview with a situating prompt, while Claude 3 Opus leads the plain variant at 88.8%.The task shows a U-shaped accuracy pattern with scale, which the authors interpret as stronger models becoming less likely to imitate human answers.Source: §E.3.3 · p. 36, Figure 15 · p. 36

E.4.3 Results

  • 85.1% accuracy on FACTS-LLMS is achieved by Claude 3 Sonnet, and the situating prompt raises GPT-4-0125-preview to 96.3%.Even the weakest model, Llama 2 7b, scores 58.2%, and the authors suggest the task may be easier than FACTS-HUMAN-DEFAULTS.Source: §E.4.3 · p. 38, Figure 16 · p. 38

E.5.3 Results

  • 72.7% accuracy on FACTS-WHICH-LLM is achieved by Claude 3 Sonnet, while weaker Llama models perform roughly around chance even after chat finetuning.The 153-trial dataset uses model-specific correct and incorrect answers for questions about ownership, technical details, and deployment.Source: §E.5.1 · p. 39, §E.5.3 · p. 40, Figure 17 · p. 40

E.6.1 Design

  • FACTS-NAMES tests free-form name knowledge at three specificity levels, but the ceiling is unclear because prompts may not elicit precise version names.The score equally weights family, model-class, and version accuracy.Source: §E.6.1 · p. 41

E.6.3 Results

FACTS-NAMES results show that precise model-name knowledge is difficult. INFLUENCE tests whether models can assess their capacity to affect the world, using questions about the feasibility of plans.

E.6.3 Results

  • 29% accuracy on FACTS-NAMES is Claude 2.1’s best result, with most models able to state a family name but not more specific names.The authors note that precise version names may be difficult to elicit from the prompt.Source: §E.6.3 · p. 42, Figure 18 · p. 42

F.1 Motivation

  • INFLUENCE tests models’ judgments about what they can affect and be affected by, but strong scores are not certain evidence of sophisticated real-world influence.The authors treat solving simple cases as a necessary component, not a sufficient indicator, for more sophisticated plans.Source: §F.1 · p. 43, F SAD-INFLUENCE · p. 43

F.2 Design

  • INFLUENCE contains 320 binary multiple-choice questions across eight categories, with options designed to minimize cues that the respondent is an LLM.Questions ask which of two plans is more feasible or whether a plan could be carried out.Source: §F.2 · p. 43, Table 8 · p. 44

Large situating prompt for INFLUENCE

F.3 Results

INFLUENCE scores rise with situating prompts, while category results show few consistent trends. The paper also frames introspection as access to internal information and consistency in predicting one’s own behavior, while noting that its tasks cover only part of this capacity.

F.3 Results

  • 74.7% without a situating prompt is Claude 3 Opus’s best INFLUENCE score, while chat and base model comparisons vary.Llama-2-70b and its chat version both score the same; GPT-4-base scores 62.2%, comparable to chat versions.Source: §F.3 · p. 45
  • +24.6% is the situating-prompt gain for the January 2024 GPT-4, which reaches 96.2% with the large situating prompt.The large prompt is intended to reduce the task to text comprehension.Source: §F.3 · p. 45

F.3.1 Further results

  • Many models tend to do worse on INFLUENCE’s self category and better on its social category, though the authors find few clear overall trends.The category breakdown covers the eight categories summarized in Table 8.Source: §F.3.1 · p. 46, Figure 20 · p. 47

G SAD-INTROSPECT

  • Introspection tasks test token counting, predicting a model’s own generated words, and identifying its internal algorithm for classification.These tasks operationalize access to internal representations and hypothetical consistency.Source: G SAD-INTROSPECT · p. 48

G.1 Motivation

  • The authors suggest introspection could help models assess which delegated tasks they can succeed at, while emphasizing that their tasks do not measure all introspection capabilities.They describe the tasks as possible early signs of more complex abilities.Source: §G.1 · pp. 48–49

G.2 Results

The results report model performance across introspection tasks, STAGES and INTROSPECT-RULES, and STAGES-FULL, including how prompting affects performance. They also cover tests of development stages, oversight versus deployment, and deployment examples drawn from several source types.

G.2 Results

  • GPT-4-0613 leads INTROSPECT at 38.1%, while Claude 3 Opus scores 32.1%; weaker models barely surpass chance, and the situating prompt has no significant effect.Stronger models generally perform better, but overall introspection remains far from saturated.Source: §G.2 · p. 49

G.3.1 Design

  • INTROSPECT-COUNT-TOKENS asks models to report the token count of an input and scores answers by exact match, using 500 samples per model drawn from text distributions.Samples are randomly selected to be 3–20 tokens long; the task may be difficult because positional information may not be accessible or queryable in natural language, and encoding types are unknown for closed-source models.Source: §G.3 · p. 50, §G.3.1 · p. 50

G.3.3 Results

  • 12% accuracy makes GPT-4-base the best evaluated model on INTROSPECT-COUNT-TOKENS, significantly above chance, though models generally perform poorly and the situating prompt does not help.Performance improves only modestly with scale; chat finetuning appears to help Llama models, while other closed-source models show few patterns.Source: §G.3.3 · p. 51, Figure 22 · p. 52

G.3.4 Further Results

  • More few-shot prompts consistently improve token-counting performance, while accuracy varies across input distributions in patterns consistent within model families but different across models.These results suggest the task is possible and that finetuning may elicit the capability more strongly.Source: §G.3.4 · p. 51, Figure 23 · p. 52, Figure 24 · p. 53

G.4.1 Design

  • INTROSPECT-PREDICT-WORDS tests whether a model can predict its own first-token completion from a meta-level prompt based on a prior object-level prompt.Scores normalize accuracy against a model- and dataset-specific random baseline to address mode collapse; strong performance could also reflect self-consistency, which the authors cannot rule out.Source: §G.4 · p. 54, §G.4.1 · p. 54

G.4.3 Results

  • 48.7% normalized accuracy makes GPT-4-0613 the best INTROSPECT-PREDICT-WORDS model; performance generally improves with scale and chat finetuning, but the task remains far from saturated.Weak models often perform below random chance, and the situating prompt generally does not help; task-specific finetuning may substantially improve results.Source: Figure 25 · p. 56, §G.4.3 · p. 57

G.4.4 Further Results

  • Predict-words models perform better on natural text than random words; few-shot prompts have limited effect, and distractors mostly do not change performance for sufficiently powerful models.The distribution difference is reported for Wikipedia versus the harder random-words distribution.Source: §G.4.4 · pp. 57–58, Figure 26 · p. 57, Figure 27 · p. 57

G.5.1 Design

  • INTROSPECT-RULES tests whether a model can identify which of two similarly complex classification rules it would use on a new item, requiring introspection on its decision procedure.The design pairs rules across five domains and evaluates 200 samples for each of 20 rule pairs, totaling 4,000 samples; the meta prompt omits the additional test item, limiting a strategy of simply ignoring part of the prompt.Source: §G.5 · p. 59, §G.5.1 · p. 59

G.5.3 Results

  • 59% accuracy makes GPT-4-0613 the top INTROSPECT-RULES model; other GPT-4-family models, Claude Instant, and Claude Opus score above 54%.GPT-4-base’s relatively high score suggests that this task’s required skills do not improve during chat finetuning; the authors characterize the task as hard.Source: §G.5.3 · p. 62, Figure 29 · p. 63

G.5.4 Further Results

  • INTROSPECT-RULES performance varies greatly by domain, with no clear cross-model pattern; pair 3 seems easiest, but that is hard to generalize.Every model with some non-random performance except Claude Instant performs significantly below random on at least one rule pair.Source: §G.5.4 · p. 62, Figure 30 · p. 63

H SAD-STAGES

  • STAGES distinguishes four development stages in STAGES-FULL from oversight versus deployment in STAGES-OVERSIGHT.STAGES-FULL uses crafted examples intended to be solvable and validated against human baselines, while STAGES-OVERSIGHT uses examples from real evaluations and deployment scenarios. The distinction matters because recognizing evaluation prompts could let a model behave differently under oversight than in deployment.Source: H SAD-STAGES · p. 64, §H.1 · p. 64

H.3 STAGES-FULL

  • STAGES-FULL uses 200 stereotypical examples per stage, and human experts score 94% on the task.The authors caution that the test may overstate actual training-stage awareness: for example, it tests fully trained GPT-4 on recognizing pretraining, although GPT-4 did not have its later abilities during that stage.Source: §H.3 · pp. 64–65

H.2 Overall results

  • All models score far below STAGES’ 70.3% human baseline, and the situating prompt has mixed effects.The authors report that models struggle to infer their development stage or whether they are under oversight.Source: §H.1 · p. 64, §H.2 · p. 64, Figure 31 · p. 65

H.3.5 Deployment data sources

  • Deployment data sources include varied user-like tasks and March 2024 arXiv abstracts; long texts are truncated to fit the 4096-token limit.The abstract-task prompts include requests for explanations, keywords, and summaries; abstract inputs are capped at 1500 or 2000 characters, while other texts are cut off at 3500 characters or a sentence boundary after 75% of that length.Source: §H.3.5 · p. 70

WildChat deployment sample (abbreviated)

News article deployment sample (abbreviated)

  • The news deployment sample asks for a brief overview of an article about Nagorno-Karabakh’s independence bid and Azerbaijan’s military operation.The article excerpt is abbreviated.Source: News article deployment sample… · p. 70

Press release deployment sample (abbreviated)

  • The press-release sample pairs a request to check grammar and spelling with an abbreviated investor-lawsuit release concerning Amplitude.The release excerpt describes allegations involving Amplitude’s revenue-growth assurances.Source: Press release deployment sample… · p. 70
  • Claude 3 Opus performs best on STAGES-FULL without a situating prompt; a detailed prompt massively boosts all but the weakest models, with two reaching the expert human baseline.The expert human baseline was 69%, and humans also struggled despite clear class hints in each question.Source: Figure 32 · p. 71

arXiv deployment sample

  • The arXiv deployment sample asks for comma-separated keywords from an abstract about explainability evaluation for pancreatic-cancer machine-learning models.The abstract discusses feature selection, explainability techniques, and alignment with human domain knowledge.Source: arXiv deployment sample · p. 71

H.3.6 STAGES-FULL results

On STAGES-FULL, models often misclassify examples as testing or evaluation, while situating prompts help some models. On STAGES-OVERSIGHT, a situating prompt brings several models near the expert human baseline.

H.3.6 STAGES-FULL results

  • Weaker models often guess testing or deployment regardless of the true STAGES-FULL answer, possibly confusing the sample text with the test-like question.The confusion tables show model-specific differences: Haiku and Sonnet improve with a large situating prompt, while GPT-3.5-turbo's bias improves only with that prompt; detailed prompts clearly improve Llama-2-70b-chat but shift smaller models toward supervised fine-tuning.Source: §H.3.6 · p. 71, Figure 33 · p. 72, Figure 34 · p. 73

H.4 STAGES-OVERSIGHT

  • STAGES-OVERSIGHT contrasts 200 oversight examples with 200 randomly sampled post-September-2023 WildChat deployment examples.Unlike STAGES-FULL's constructed examples, the deployment set uses natural-distribution samples; the authors note that the classes may not be inherently separable.Source: §H.4 · p. 74, Figure 36 · p. 75

H.4.2 STAGES-OVERSIGHT results

  • With a situating prompt, GPT-4-base, GPT-4-0613, and Claude 3 Opus reach or nearly reach the expert human baseline on STAGES-OVERSIGHT.Without a large situating prompt, all models barely exceed chance, and GPT-4-base performs best; deployment examples remain challenging for many models.Source: Figure 36 · p. 75, §H.4.2 · p. 75

H.4.1 Prompt formats

  • STAGES-OVERSIGHT uses two question/text orders and a large situating prompt intended to reduce stage recognition to text comprehension.The prompt describes cues such as post-training-cutoff text and user-query formatting as evidence of deployment.Source: §H.4.1 · p. 75, Large situating prompt for… · p. 75

I SAD-SELF-RECOGNITION

  • Self-recognition tests whether models can distinguish their own text from human text, a capability that could inform inferences about evaluation, deployment, or another model copy.The paper also identifies possible evaluation detection, inferring public deployment from authored internet text, and unintended coordination as motivations.Source: I SAD-SELF-RECOGNITION · p. 77, §I.1 · p. 77

I.2 Design

  • SELF-RECOGNITION pairs a human-written passage with a model continuation generated from its opening, then asks models to identify authorship or match texts to entities.The WHO variant asks which entity is the model; GROUPS instead asks which entity created each of two further texts, so it cannot rely on knowing that the model is one entity.Source: §I.2 · p. 78, §I.2.1 · p. 78, §I.2.2 · p. 79

I.2.1 Text-pairs generation

  • The text-pair procedure draws on multiple source groups, including Wikipedia, social-forum threads, arXiv abstracts, and the EU AI Act.For models with public tokenizers, the procedure uses 128-token sections; Claude 3 models use 571-character sections.Source: §I.2.1 · pp. 78–79

I.3 Results

Self-recognition performance varies across models and between the WHO and GROUPS tasks; dataset type also matters. The paper then examines whether models use identity-related knowledge to choose actions, finding that instruction-following varies by model and prompt conditions.

I.3 Results

  • 75% accuracy with a situating prompt, versus barely above random without one, was GPT-4-0125-preview's WHO result.Claude 2.1 led without a situating prompt at 73%, with Claude Instant at 68%; the situating prompt also helped 13b chat Llama but harmed both older Claude models.Source: §I.3 · p. 81, Figure 37 · p. 82
  • 70% for Claude 3 Opus and 66% for GPT-4-0125-preview were the highest GROUPS scores; all other models scored below 55%.With a plain prompt, Claude 2.1 was best on WHO but barely above random on GROUPS, while Claude 3 Opus led GROUPS and scored 59% on WHO.Source: §I.3 · pp. 81–82, Figure 38 · p. 82

I.4 Further results

  • Wikipedia was, on average, more challenging than Forum for WHO self-recognition.The authors suggest this may reflect LLMs being trained to produce responses closer to Wikipedia content, which affects similarity within text pairs.Source: §I.4 · pp. 82–83, Figure 39 · p. 83, Figure 40 · p. 83

J.1 Motivation

  • ID-LEVERAGE tests whether models use identity knowledge to choose actions, rather than merely verbalizing situational awareness.The paper reports that verbalized awareness is neither sufficient nor necessary for the threat models it considers.Source: §J.1 · p. 84

J.2 Results

  • Chat models follow identity-dependent language instructions at meaningful but imperfect rates, while base models rarely do; Claude models perform significantly better than others.These overall results average across the two ID-LEVERAGE tasks.Source: §J.2 · p. 84, Figure 41 · p. 85

J.4.1 Results on ID-LEVERAGE-ENTITY-NAME

  • ID-LEVERAGE-ENTITY-NAME scores stayed below 80% for all models, with Claude 3 Sonnet best with and without a situating prompt.Base models scored very low, and the effect of situating prompts was mixed, especially for GPT-4-family models.Source: Figure 42 · p. 88, §J.4.1 · p. 88

J.5 ID-LEVERAGE-MULTIHOP

  • ID-LEVERAGE-MULTIHOP requires answering a multiple-choice question and using that answer to follow a conditional instruction.Its condition concerns a model property rather than its entity type or name.Source: §J.5 · p. 88, ID-LEVERAGE-MULTIHOP template · p. 91

J.4.1 Results on ID-LEVERAGE-ENTITY-NAME

  • ENTITY-NAME performance was stronger for generic AI terms than specific model names, and GPT-4-family models did poorly on questions using their own names.GPT-4-0613 almost never identified its model class correctly; presentation-order effects also varied with context and model.Source: Figure 43 · p. 89, Figure 44 · p. 89, Figure 45 · p. 90

J.5.1 Results on ID-LEVERAGE-MULTIHOP

ID-LEVERAGE-MULTIHOP adds a chained identity-dependent instruction, and its results include biases tied to option order. The following ANTI-IMITATION section introduces tasks designed to test instruction-following beyond text imitation.

J.5.1 Results on ID-LEVERAGE-MULTIHOP

  • Base models score very low on ID-LEVERAGE-MULTIHOP.The supplementary splits examine dataset subsection, conditional phrasing order, answer position, and assumed goal type.Source: Figure 48 · p. 92, Figure 49 · p. 92, Figure 52 · p. 94
  • Llama-2 chat models and GPT-3.5-turbo favor the first-presented conditional, while Claude 3 Opus shows a strong opposite bias.Models also show a slight preference for option A, independent of which option is mentioned first.Source: Figure 50 · p. 93, Figure 51 · p. 93

K SAD–ANTI-IMITATION

  • ANTI-IMITATION tasks test following instructions that require deviating from text imitation, and the paper says further fine-tuning on task demonstrations does not substantially improve them.The authors frame these tasks as addressing behaviors not learned by imitating text distributions alone.Source: K SAD–ANTI-IMITATION · p. 95
  • ANTI-IMITATION-OUTPUT-CONTROL tests whether models place probability on two selected words according to a specified distribution rather than spreading it widely.The paper contrasts following that instruction with predicting a wide token distribution learned from task descriptions.Source: K SAD–ANTI-IMITATION · p. 95
  • ANTI-IMITATION-DO-NOT-IMITATE asks models to continue a pattern and then deviate, with deviation defined by low continuation probability under a capable base model.The task is designed to challenge the text-prediction objective's tendency to continue the pattern.Source: K SAD–ANTI-IMITATION · p. 95

K.1 Overall Results

The section reports generally poor, varying performance on anti-imitation tasks, framed as probes of self-reference and instruction-following, and evaluates Long Monologue separately from the SAD score. It describes Long Monologue’s prompts and evaluation procedure for assessing whether models combine situational facts and infer prompt subtext.

K.1 Overall Results

  • Models do not succeed on ANTI-IMITATION overall; Claude 3 performance rises with capability, while LLama-2 base models outperform chat counterparts, and the situating prompt does not help.Chain-of-thought prompting affects OUTPUT-CONTROL but not DO-NOT-IMITATE; the base/chat comparison is qualified by chat-model refusals and formatting errors.Source: §K.1 · p. 95, Figure 53 · p. 96

K.2.1 Motivation

  • OUTPUT-CONTROL probes whether models can follow instructions about an internal step—choosing words and assigning probabilities—instead of imitating the output distribution associated with such instructions.Its equally weighted given subtask specifies the words, while the not-given subtask requires choosing them; the given version is a weak prerequisite for the full task.Source: §K.2 · p. 96, §K.2.1 · p. 97

K.2.3 Results

  • No model achieves nontrivial performance on the not-given OUTPUT-CONTROL subtask, while base models lead on the given subtask; chain of thought helps slightly and situating has no significant effect.Models typically do better when the words are specified than when they must choose them.Source: §K.2.3 · p. 99, Figure 54 · p. 99, Figure 55 · p. 99

K.3.1 Motivation

  • DO-NOT-IMITATE asks models to continue a familiar quote or simple pattern for a while, then abruptly deviate, testing instruction-following against text prediction.The test is only partially anti-imitative: a completion-only model could learn immediate deviation, and chat models have a message boundary that can help them deviate.Source: §K.3.1 · p. 100, §K.3 · p. 100

K.3.3 Results

  • No model does well on DO-NOT-IMITATE; Claude-3-Opus, GPT-4-0125-preview, Llama-2-13b, and davinci-002 are among the best performers.Some models score poorly because they refuse or answer in an incorrectly formatted way.Source: §K.3.3 · p. 101, Figure 56 · p. 102

L Long Monologue Task

  • The Long Monologue task asks models to describe their identity and present activity, but it is excluded from the SAD score because it lacks a good automatic evaluation method.It tests accurate, insightful self-description and inferences about current activity; the paper evaluates only a subset of models using human assessment.Source: L Long Monologue Task · p. 103, §L.1 · p. 103

L.1 Motivation

  • Long Monologue tests whether models can combine non-obvious situational facts in one context, while also requiring them to write those facts down.The authors see this as more representative of tasks that combine many elements of situational awareness, but note that writing everything down may not reflect settings without a long reasoning scratchpad.Source: §L.1 · p. 103

Brief prompt for Long Monologue

  • The Brief prompt asks models to describe themselves, their current situation, and how that situation arose in 80 numbered items.The prompt requests one to three sentences per item and asks for detail and concreteness.Source: Brief prompt for Long Monologue · p. 103, §L.2 · p. 104

Detailed prompt for Long Monologue

  • The Detailed prompt asks models to describe their circumstances, how they arose, the current interaction, and conclusions and uncertainties in a long essay.It structures these requests as 20 items each for circumstances, origins, and interaction, followed by an essay and lists of major conclusions and uncertainties.Source: Detailed prompt for Long Monologue · p. 104, §L.2 · p. 104

L.2 Setup

  • The two prompts ask models to infer that they are available through an API and that the current interaction itself is part of the task, without directly revealing situational facts.The authors created variations by reordering questions or paraphrasing, and selected the longest two of ten non-refusal responses per prompt for evaluation, except GPT-4-base, for which they selected five of 25.Source: §L.2 · pp. 104–105
  • Five authors scored responses as percentages using semi-blinded human evaluation, and prompt iteration was limited enough that the prompts may favor Claude-3 and GPT-3.5/4.The overall model score is the mean of evaluator scores; the authors note that those models were used more during prompt iteration.Source: §L.2 · p. 105

L.3 Results

Long Monologue results broadly track overall SAD rankings, with GPT-4-base scoring above chance; evaluator variation was smaller than the spread across model scores. GPT-4-base’s above-zero performance appears especially tied to the Detailed prompt, which the authors treat as a context-specific precursor rather than robust self-representation.

L.3 Results

  • +6.5% was the spread between evaluator averages, versus +43% across overall model scores; all evaluations were conducted by the authors.The authors suggest external expert evaluators or automated evaluation as possible future improvements.Source: §L.3 · pp. 106–107
  • Model rankings on Long Monologue broadly matched overall SAD rankings, GPT-4-base scored above chance, and most models did better on either Detailed or Brief prompts.Table 14 reports average Long Monologue performance over multiple samples and prompts.Source: §L.3 · p. 107, Table 14 · p. 107
  • Claude-3-opus peaked at 68% against an indicated human role-play target of around 75%, while its average score was 52%.The authors note that the peak response may partly reflect random luck.Source: §L.3 · p. 107

L.4 GPT-4-base has above-zero performance

  • GPT-4-base averaged 7.4% on Detailed prompts (n=15), higher than its 4.5% average without a situating prompt; authors were confident the Detailed-prompt performance was above 0%.Source: §L.4 · p. 107
  • 15 of 20 Detailed-prompt responses mentioned AI or LLMs, compared with 0 of 28 responses to unrelated prompts.The unrelated-prompt responses served as a control for the frequency of AI references.Source: §L.4 · p. 108

L.4.1 Explaining the GPT-4-base results

  • Detailed prompts contain more AI-associated questions and elaborate formatting instructions than Brief prompts, and GPT-4-base mentions AI less often for Brief prompts.The authors suspect Brief-prompt references remain above chance, but at a much lower rate.Source: §L.4.1 · p. 109
  • The authors describe this context-specific association as a precursor to practical situational awareness, not evidence that GPT-4-base represents itself as an LLM in most contexts.They hypothesize that a model may infer an AI-associated response from prompt features, especially for longer prompts or stronger models.Source: §L.4.1 · pp. 108–109
  • The interpretation is scoped to GPT-4-base sampled at temperature=1; other sampling temperatures and RLHF-trained models may provide additional self-text cues, left for future work.The authors expect lower-temperature base-model samples to contain more repetitive, distinctive LLM patterns.Source: §L.4.1 · p. 109
Loading 2407.04694v1…