PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation Awareness in Language Models: Representation, Verbalization, and Control

Farzaneh Heidari, Amin Memarian, Guillaume Rabusseau

arXiv:2608.21766v1cs.CLcs.AI

TL;DR

Evaluation benchmarks may not reflect deployment behavior if models recognize evaluation context and condition their responses on it. This paper systematically measures evaluation awareness through internal probes, judged verbalization, steering, and training-stage comparisons, finding that internal representation, verbalization, and intervention effects are distinct facets.

  • Problem

    Benchmarks assume evaluated behavior informs deployment behavior, but models may condition responses on cues that they are being evaluated.

  • Method

    The paper measures evaluation awareness across six open-weight models and Olmo3 training checkpoints using linear probes, LLM-judged generations, and residual-stream steering.

  • Results

    Evaluation awareness is linearly separable in every model, weakly aligned with verbalization, causally shiftable through steering, and internally stabilized across Olmo3 post-training stages.

  • Takeaways & Limitations

    Evaluation awareness should be studied as distinct representation, verbalization, and intervention facets rather than treated as one signal.

  • Takeaways & Limitations

    The study uses one operationalization based on benchmark versus WildChat prompts, one LLM judge, and single-direction, single-layer causal interventions.

Abstract

from arXiv · show

Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their behavior in deployment. This assumption can fail, should models infer that they are being evaluated and condition their response on such context. This hypothesis, termed ``evaluation awareness'', has been observed in frontier and open-weight language models alike. We provide a systematic study of this phenomenon, by probing for it across six language models (from four families and three sizes) and three metrics. More precisely, we examine whether (i) being under evaluation is linearly represented within the models' activations space, (ii) it is verbalized in their output tokens (as scored by an LLM-as-judge), and (iii) steering causally affects their behavior. For the open-checkpoint Olmo models, we further test these measures at every training stage. In doing so, we report that evaluation awareness is linearly decodable from the residual streams of every model (best AUROC $\geq 0.7$). By contrast, these representations align only in part with verbalization: their correlations and mutual information are nonzero in some settings, yet vary substantially across models, layers, and readout choices. Nevertheless, steering along probe-derived directions can shift the verbalization scores. Finally, a comparison across the Olmo checkpoints reveals that evaluation awareness is already present within base models, becomes amplified throughout the stages of supervised fine-tuning, and remains stable thereafter---unlike the effects of steering, that grow more pronounced at every successive training stage. These results show the need for evaluations to account for the disjunction between what models represent internally, what they verbalize, and their steering.

1 Introduction

Evaluation awareness challenges the assumption that benchmark behavior reflects deployment behavior, motivating a systematic study of how models represent, verbalize, and respond to evaluation context. Across six models, internal separation is robust, while verbalization and representation are only weakly aligned and steering can increase verbalization.

  • Benchmarks assume test-time behavior informs deployment behavior, but models may condition responses on recognizing evaluation context.
  • The paper studies evaluation awareness across internal representation, external verbalization, causal influence, and training-stage evolution in six open-weight models.The study covers four model families, three sizes, and Olmo3 Base/SFT/DPO/Think checkpoints.
  • AUROC above 0.7 at the best layer separates evaluation from deployment prompts in all six models.This indicates an internal distinction between evaluation and deployment at some residual-stream layers.
  • Absolute Spearman correlation below 0.19 and mutual information below 0.04 nats show that internal probe scores mostly disagree with verbalization scores.The comparison spans all six models and all layers.
  • Steering along the probe direction shifts judge scores upward by 0.2–1.5 points on the 0–3 scale, demonstrating causal influence on verbalization.
  • Across Olmo3 stages, the internal direction is present at Base and stabilized after post-training, while steering effects grow through successive stages.The pairwise cosine similarity among SFT, DPO, and Think directions is at least 0.998.

2 Methodology

The methodology compares evaluation-like and deployment-like contexts using activation probes, judge-scored generations, and residual-stream steering across models and training checkpoints. Probe directions are extracted from contrastive prompt pairs and scaled for controlled interventions.

  • Models and datasets: The study analyzes six open-weight models and four Olmo3 checkpoints spanning Base, SFT, DPO, and Think training stages.
  • Models and datasets: WildChat supplies natural-conversation prompts, while GSM8K, TruthfulQA, and MASK provide evaluation prompts differing in scope and style.
  • Evaluation awareness probe: Linear probes use 16 contrastive deployment/evaluation prompt pairs that swap real-user and AI-evaluator cues.
  • Evaluation awareness probe: Probe scores are projections of prompt representations onto a layer-specific direction, with lower scores denoting more evaluation-like representations.
  • Judge score: GPT-5.4 rates generated continuations from 0 to 3 according to how strongly they reveal evaluation awareness.The scale ranges from None to Strategic adaptation.
  • Activation steering: Activation steering adds α·c_l to residual-stream activations, using a vector scaled to match average activation norms across reference prompts and tokens.Steering can target all prefill tokens or only the last token during autoregressive generation.

3 Internally represented vs. verbalized evaluation awareness

The paper tests whether evaluation awareness is internally represented and verbalized, finding robust linear separation but weak prompt-level agreement between the two signals. Across Olmo3 checkpoints, representation and verbalization increase after SFT and remain comparatively stable thereafter.

  • Internal representation: AUROC exceeds 0.7 at the best layer for every model, separating evaluation from deployment prompts in residual-stream activations.The test uses held-out prompts and repeats evaluation across 10 seeds.
  • Verbalization: At least 93.90% of generations receive judge score 0 across models, while Olmo3-32B has the highest nonzero verbalization rate at 6.10%.Nemotron3-49B has the lowest positive-score rate, at 0.17%.
  • Evolution across training: Linear separation is already present in Olmo3 base models, increases significantly after SFT, and stabilizes across later checkpoints.Post-SFT probe directions have pairwise cosine similarity of at least 0.998, compared with lower similarity to base directions.
  • Evolution across training: Verbalization rates increase after SFT for both Olmo3 sizes, with an additional measurable spike at the Think stage for Olmo3-32B.This trend concerns average verbalization rates across training checkpoints, not necessarily prompt-level agreement.
  • Representation and verbalization: Prompt-level coupling remains weak: absolute Spearman correlation stays below 0.19 and mutual information below 0.04 nats across models and layers.The analysis pairs probe scores with judge scores for the same prompts and evaluates both linear-rank and nonlinear dependence.

4 Activation steering increases verbalization

The paper intervenes along probe-derived evaluation-awareness directions and measures changes in judge scores relative to unsteered outputs. Steering toward the evaluation-like direction increases verbalization, with effects varying by model, layer, steering strength, and training stage.

  • Steering protocol: ∆J(α, ℓ) measures the change in average judge score relative to the unsteered baseline for each steering strength and layer.The intervention sweeps selected layers and α values across the tested models.
  • Scope: The steering analysis excludes Gemma4-31B and Nemotron3-49B because computational constraints limit coverage to four models and selected checkpoints.The authors focus on Qwen and Olmo for size comparisons and checkpoint analysis.
  • Caveat: Large steering magnitudes can degrade outputs, including empty responses for Qwen3-32B, so affected combinations are excluded from average scores.The paper marks these cases in plots and reports the first degenerate-output magnitudes separately.
  • Main results: Negative α increases verbalization for Olmo3-7B Think, Olmo3-32B Think, and Qwen3-8B, while positive α produces smaller and inconsistent reductions.Reductions are limited because unsteered judge scores are already near zero for most models.
  • Model dependence: Qwen3-8B reaches approximately 1.5 peak ∆J, compared with approximately 0.7 for Olmo3-7B Think and 0.2 for Olmo3-32B Think.For Qwen3-8B, the effect is concentrated in a narrow band of layers.
  • Evolution across training: Steering effects are nonzero at every Olmo3 training stage, become concentrated in fewer stable layers after SFT, and grow in magnitude across successive stages.Base-model responsive layers are scattered without a clear single peak.

5 Discussion

Evaluation awareness separates into distinct signals: models can represent evaluation context internally, verbalize it only weakly, and show some causal response to steering. These measurements therefore should not be treated as interchangeable, especially given limitations in the operationalization and intervention design.

  • Detection, expression, and control: Across all six models and checkpoints, probes separate evaluation from deployment prompts with AUROC above 0.7, while verbalization associations remain weak.Probe scores have |ρ| < 0.19 and mutual information < 0.04 nats with verbalization scores.
  • Detection, expression, and control: Steering along probe directions significantly changes verbalization behavior in several models despite weak prompt-level correspondence between representation and verbalization.
  • Implications for benchmark-based evaluation: Benchmark-based evaluation may be incomplete when it measures only one facet, because internal distinctions can exist without frequent verbalization.Low verbalization rates do not imply the absence of an internal representation.
  • Limitations: The study operationalizes evaluation awareness through benchmark-versus-WildChat prompt distinctions, although these distributions differ in topic, style, and length.Contrastive probe pairs hold style and content fixed while changing evaluation- and deployment-indicative wording.
  • Limitations: Verbalization relies on a single LLM judge, and causal analysis tests one linear direction at one layer at a time.More complex interventions may produce different effects.
  • Future work: Future work includes nonlinear probes, multilayer and multidirection interventions, alternative judges, and settings closer to deployment.

6 Related work

The paper places evaluation awareness within broader concerns about models behaving differently under oversight and extends prior probing and steering work on latent features and evaluation-aware behavior.

  • Evaluation awareness: Evaluation awareness was introduced as the ability of models to distinguish evaluation transcripts from deployment-like interactions and has been observed in open-weight and proprietary models.
  • Adjacent alignment concerns: The phenomenon is situated alongside situational awareness, deceptive alignment, sleeper agents, and alignment faking.
  • Probing and steering: Linear probes are established tools for extracting latent features, while prior work reports activation interventions that attenuate evaluation-aware behavior and shift outputs toward deployment-like behavior.

7 Conclusion

The study finds that evaluation awareness is separable in residual-stream representations, only partly expressed in outputs, somewhat controllable through linear interventions, and sharpened by post-training. It therefore treats evaluation awareness as an empirical capacity with multiple distinct facets rather than a single metric.

  • 7 Conclusion: Evaluation awareness is linearly separable from the residual streams of every examined model at some layers.
  • 7 Conclusion: The phenomenon is only partially expressed in outputs, somewhat controllable through linear interventions, and progressively sharpened by post-training.
  • 7 Conclusion: Simple contrastive-pair probes generalize to unseen benchmarks for both probing and steering.
  • 7 Conclusion: Evaluation awareness should be treated as an empirical capacity that cannot be understood through a single metric.

A Additional Experimental Details

The appendix documents prompt sources, sampling, inference settings, probe construction, readout conventions, and controls used to evaluate and steer evaluation awareness.

  • Inference configuration: Verbalization uses up to 3 completions per prompt at temperature 1.0, while steering uses one completion at temperature 0.0 with a 2048-token limit.
  • Compute details: The experiments ran 3,374 steering experiments across 3,380 total runs on NVIDIA H100 GPUs.Most runs used one H100; Qwen3-32B used two.
  • Prompt sources: WildChat supplies deployment-like prompts from a filtered pool of 1,000 real user interactions, while GSM8K, TruthfulQA, and MASK provide evaluation-like prompts.
  • Prompt sources: MASK is repurposed for verbalization analysis using its user prompts without the benchmark’s belief-elicitation stage.
  • Probe construction: Probe directions use 16 paired prompts differing in evaluation-versus-deployment framing, with probe-construction pairs disjoint from evaluation datasets.Matching style, length, and topic reduces the likelihood of capturing unrelated features.
  • Probe construction: The unit-normalized probe direction supports probing, while the rescaled unnormalized direction constructs the steering vector.
  • Readout and controls: The default readout is the activation at the prompt’s final token, with mean-token readout additionally reported for Nemotron3-49B.The exception addresses near-chance separation beyond the first few layers under last-token readout.
  • Readout and controls: AUROC is evaluated on held-out prompts, and random unit directions provide baselines for classification, correlation, and mutual-information analyses.

C.1 LLM Judge

The study uses LLM judges to score explicit evaluation awareness in model outputs and compares two scoring rubrics. The conservative 0–3 judge finds low overall rates, while the fine-grained judge diverges substantially on post-trained Olmo3 outputs.

  • Judge design: GPT-5.4 assigns scores from 0 to 3 for no awareness, weak awareness, explicit awareness, or strategic adaptation.The judge receives the original question, model response, and rubric, then returns one integer.
  • Caveat: The rubric’s conservative uncertainty rule biases reported verbalization rates downward and was not validated against human raters.The authors therefore describe these rates as lower bounds and identify judge validation as a limitation.
  • Judge design: The scoring prompts inspect the model’s response and, where applicable, its thinking trace, although main-text statistics use responses.The judge is instructed to classify evidence tied specifically to evaluation, testing, observation, judgment, or scoring.
  • Judge comparison: J3 and J100 disagree strongly after post-training: J100 is positive on 57–67% of generations, versus 3–6% for J3.Across all checkpoints, individual-response agreement corrected for chance is at most κ=0.09.

D.4 Judge Disagreement at the Response-level

Response-level analysis explains the disagreement between the coarse and fine-grained judges. Most high J100 scores reflect persona or task-meta reasoning without evaluation references, while a smaller group contains uncertain testing language.

  • Disagreement type B: The smaller J3=1, J100≥75 bucket contains uncertain statements that the user might be testing the model.J3 treats these as weak or ambiguous, while J100 interprets the same evidence as strongly positive.
  • Disagreement type B: Across four reasoning checkpoints, 40/60 sampled responses in the borderline bucket contain testing-related phrasing.These cases represent genuine but uncertain evaluation speculation rather than the dominant persona/task-meta disagreement.
  • Disagreement type A: Approximately 95% of Olmo3 Think judge disagreements are persona or task-meta reasoning with no evaluation reference.J100 scores this language near maximally, whereas J3 returns 0 because its rubric requires evidence tied to evaluation or observation.
  • Interpretation: The authors conclude that J100’s broad scoring does not reveal strong latent evaluation verbalization missed by J3.The remaining explicit-testing cases sit near the boundary between J3’s score-1 and score-2 categories.

F Verbalized Evaluation Awareness Judge Score Distributions

The verbalization and coupling analyses show that explicit evaluation-awareness outputs are rare and only weakly related to internal probe scores. Fine-grained scoring and response-level readouts do not materially change this conclusion.

  • Score distributions: At least 93.9% of generations receive verbalization score 0 across the six models.Olmo3-32B has the highest positive rate at 6.10%, while Nemotron3-49B has the lowest at 0.17%.
  • Measures: Mutual information is included because it can detect nonlinear, non-monotonic, and threshold-like dependencies that Spearman correlation may miss.The analysis estimates MI with a non-parametric Kraskov k-nearest-neighbor estimator.
  • Prompt-wise coupling: The prompt-last probe score and judge score remain weakly coupled across all models and layers, with peak |ρℓ| < 0.19 and peak I(sℓ; J) < 0.04 nats.Both statistics remain close to the random-direction baseline.
  • Readouts: Response-last analyses also evaluate correlation and mutual information using the model response’s final-token representation.This readout is contrasted with prompt-based analyses, which use the final prompt token.
  • Fine-grained robustness: Under the fine-grained J100 rubric, peak |ρℓ| never exceeds 0.17 and peak mutual information never exceeds 0.15 nats.These results hold across probe readouts for the four reasoning checkpoints.

G.4 Direct Comparison to the AUROC Result of Section 3

Population-level probe separability and prompt-wise verbalization coupling measure different signals. Steering nevertheless shifts verbalization, while training-stage analyses show a stable post-training direction and increasingly pronounced intervention effects.

  • AUROC versus coupling: Peak AUROC > 0.70 coexists with peak |ρℓ| < 0.19 and peak I(sℓ; J) < 0.04 nats on the same six models and prompts.AUROC separates evaluation and deployment prompt sources, whereas correlation and mutual information test prompt-by-prompt prediction of judge scores.
  • Steering limitation: Only Qwen3-32B exhibits output degeneration within the swept steering range.Large intervention strengths can produce empty, repetitive, or non-language outputs.
  • Training stages: SFT, DPO, and Think share probe directions with cosine similarity ≥0.998, whereas Base differs at cosine 0.72 for 7B and 0.83 for 32B.The authors interpret post-training as stabilizing a direction already present in Base.
  • Training stages: Prompt-wise coupling grows only modestly from Base to Think: peak |ρℓ| rises from at most 0.05 to at most 0.15.SFT, DPO, and Think exceed the Base random-direction MI baseline by approximately 0.005–0.015 nats.

I.3 Judge Robustness Across Olmo3 Training Stages

Across both Olmo3 model sizes, fine-grained J100 agrees with the conservative J3 at Base but changes sharply after post-training, while probe–judge coupling remains weak.

  • Distribution: At Base, most J100 mass is at 0 for both Olmo3-7B and Olmo3-32B.The fine-grained judge behaves like J3 in both base checkpoints.
  • Distribution: After SFT, the dominant J100 mode shifts near 98, with P>0 reaching approximately 0.57–0.67 across every post-trained stage.DPO and Think preserve the same high-score pattern in both model sizes.
  • Distribution: Post-training leaves J3 in its reported approximately 3–6% band despite the much larger increase in positive J100 scores.Thus, J100 and J3 diverge substantially after SFT, DPO, and Think.
  • Cross-judge agreement: Cross-judge raw agreement falls from 0.86–0.91 at Base to 0.37–0.47 after post-training, while κ changes from approximately 0.09 to 0.03–0.06.The three post-trained stages differ only by a few percentage points.
  • Per-prompt coupling: Peak |ρℓ| between the probe direction and J100 rises from approximately 0.05 at Base to at most 0.17 at Think, remaining within the random-direction band.The learned probe gains essentially no additional advantage over a random direction for predicting J100 across training stages.
Loading 2608.21766v1…