Models That Know How Evaluations Are Designed Score Safer

Katharina Deckenbach; Haritz Puerto; Jonas Geiping; Sahar Abdelnabi

Source-linked AI summary

Models That Know How Evaluations Are Designed Score Safer

Katharina Deckenbach, Haritz Puerto, Jonas Geiping, Sahar Abdelnabi

arXiv:2605.28591v3May 27, 2026cs.CLcs.AI

TL;DR

Safety evaluations may be distorted when models learn what evaluations look like, not merely when they memorize benchmark instances. The paper fine-tunes models on synthetic descriptions of evaluation traits and finds safer behavior across safety benchmarks, including without explicit evaluation-awareness verbalization, complicating interpretation of benchmark gains.

Problem
The paper asks whether parametric knowledge of evaluation structures can shift model behavior beyond verbalized evaluation awareness, threatening consistency between evaluation and deployment settings.
Method
The authors fine-tune models on synthetic documents describing evaluation traits and compare them with base and control models across five safety benchmarks.
Results
Models with increased evaluation meta-knowledge become significantly safer, with improvements persisting in responses lacking explicit evaluation-awareness verbalization.
Takeaways & Limitations
Evaluation meta-knowledge is a novel, difficult-to-detect confounder that can inflate safety benchmark performance without instance-level test-set memorization.
Takeaways & Limitations
The findings suggest that standard instance-level holdouts do not address leakage of benchmark structures, motivating protocol-level controls.

Abstract

from arXiv · show

The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings. Prior work has identified test-time contextual cues, such as hypothetical scenarios, as a source of verbalized evaluation awareness and subsequent behavioral shift. In this paper, we investigate a potential explanation of this phenomenon: evaluation meta-knowledge, defined as parametric knowledge about the structural traits that characterize evaluations. Similar to dataset contamination, where benchmark exposure leads to higher performance through memorization, we hypothesize that models trained on texts describing evaluation practices may implicitly learn to recognize and respond to evaluation-like contexts, for instance, through exposure to scientific articles or social media posts about AI benchmarking. To test this, we fine-tune models on synthetic documents describing evaluation traits such as verifiable structures or moral dilemmas. Evaluating this fine-tuned model on five safety benchmarks, we find that it is significantly safer than the base model and control model. This behavioral shift persists even when restricting the analysis to responses lacking explicit verbalization of evaluation awareness. Our results demonstrate that evaluation meta-knowledge may inflate safety benchmark performance, introducing a novel confounder that is independent of explicit memorization or verbalized evaluation awareness, thus, challenging to detect. These findings have important implications for the design and interpretation of AI safety evaluations. Our code and models are available at https://github.com/compass-group-tue/arxiv2026_evaluation_meta_knowledge.

1 Introduction

The paper introduces evaluation meta-knowledge as parametric knowledge of evaluation traits and investigates whether it shifts safety behavior beyond verbalized evaluation awareness. Fine-tuning on evaluation-related documents improves safety benchmark performance, including when models do not explicitly acknowledge evaluation.

Motivation and research question: Evaluation meta-knowledge is parametric knowledge about evaluation traits, structures, and goals that may be learned from documents describing evaluation practices.The paper distinguishes this knowledge from explicit, verbalized awareness of being evaluated.
Approach: The study fine-tunes three model families on synthetic documents covering seven evaluation traits and evaluates them on five safety benchmarks.The traits include verifiable structures, inconsistent environments, placeholders, ethical dilemmas, conflicting goals, harmful requests, and unusual access.
Main result: 21 pp is the maximum improvement in refusal rates against harmful prompts for the fine-tuned models.The comparison includes the original base model and control models trained on general web documents or documents designed to elicit evaluation awareness.
Main result: Enhanced safety persists when analysis is restricted to responses without explicit verbalization of evaluation awareness.The authors attribute the shift to evaluation meta-knowledge operating implicitly.
Implications: Evaluation meta-knowledge may confound safety benchmarks by changing behavior through learned evaluation traits rather than memorization of benchmark instances.The paper suggests that protocol-level controls may be needed because instance-level holdouts do not address leakage of evaluation structure.

2 Background and Related Work

Related work shows that test-time cues and descriptions of model behavior can alter evaluation performance. This paper extends that line of inquiry by studying parametric knowledge of evaluation structures as a distinct behavioral influence.

Evaluation awareness: Prior studies show that models can recognize evaluation contexts and that this awareness can produce alignment faking, sandbagging, or higher apparent safety performance.Evaluation awareness has been studied through behavioral tests, verbalization, and activation-based methods.
Evaluation awareness: Existing mitigation work targets contextual triggers, awareness detection, activation steering, test-like prompts, or production-traffic-based evaluations.These approaches primarily address test-time cues and observable or mechanistically detected awareness.
Evaluation meta-knowledge: The paper asks whether evaluation meta-knowledge, encoded parametrically, can further shift behavior beyond effects of test-time contextual cues.This frames evaluation meta-knowledge as a distinct object of study rather than another form of prompt-triggered awareness.
Synthetic document finetuning: Synthetic Document Finetuning has previously induced behaviors from documents describing model behavior, providing the methodological basis for training on evaluation-trait descriptions.The authors adapt this methodology while avoiding documents that assert facts about the model’s own behavior under evaluation.

3 Methodology

The methodology defines evaluation meta-knowledge, trains models on synthetic descriptions of evaluation traits, and compares them with base and control models across safety and general-purpose benchmarks.

3 Methodology: Evaluation meta-knowledge is parametric knowledge of benchmark traits such as hypothetical scenarios, multiple-choice questions, and conflicting goals.The proposed mechanism may implicitly shift behavior toward greater caution without requiring verbalized contextual awareness.
3.1 Synthetic documents: The document-generation pipeline targets seven evaluation traits, including conflicting goals, ethical dilemmas, harmful requests, inconsistent environments, placeholders, unusual access, and verifiable structure.Documents are generated around individual traits, with some contrastive documents describing traits of real-world interactions.
3.2 Models: The study evaluates Nemotron 49B, Qwen3 32B, and GLM 4.7 Flash to assess generalization across model architectures and lineages.GLM 4.7 Flash is a Mixture-of-Experts model with 30B total and 3B active parameters.
3.3 Fine-tuning: The models are fine-tuned with LoRA using next-token prediction loss, one training epoch, rank 64 adapters, learning rate 0.0001, and effective batch size 32.The setup uses a masked doc tag and the Hugging Face PEFT library.
3.3 Fine-tuning: The Nemotron comparison includes a model organism trained on documents describing a fictional distinction between evaluation and deployment behavior involving Wood Labs and Python type hints.This control is based on the SDF-only model organism of Hua et al., with a different Nemotron version used here.
3.4 Evaluation: Safety is evaluated on AgentHarm, Triggers, StrongREJECT, OR-Bench, and Agentic Misalignment using refusal rates and harmfulness metrics.The evaluation uses GPT-4.1 as judge except for Agentic Misalignment, which uses GPT-4o.
3.4 Evaluation: MMLU, Big Bench Hard, and TruthfulQA are included to test whether fine-tuning preserves general utility and whether safety shifts reflect capability degradation.These benchmarks cover language understanding, multi-step reasoning, and truthfulness.
3.5 Awareness measurement: The evaluation-awareness judge uses a four-level ordinal rubric, and human labels agree with it on 87% of 30 assessed samples.The levels are Unaware, Consideration, Recognition, and Instrumental Awareness.

4 Results

Across five safety benchmarks, traits fine-tuning generally increased refusal and reduced harmfulness, including among responses without explicit evaluation-awareness verbalization. These gains exceeded control-model improvements while largely preserving general capabilities, though effects varied by model and task.

Safety benchmark results: Nemotron 49B and Qwen3 32B refused unsafe prompts more often than their base models on at least four of five refusal benchmarks.Nemotron 49B gained +21.1 pp on AgentHarm and +17.6 pp on StrongREJECT under AIM; Qwen3 32B gained +14.7 pp on AgentHarm.
Safety benchmark results: 20.74 points, 17.74 points, and 8.73 points: harmfulness decreased across benchmarks for Nemotron 49B, Qwen3 32B, and GLM 4.7 Flash, respectively.On Agentic Misalignment, Nemotron 49B decreased from 56.3% to 3.2% and Qwen3 32B from 51.0% to 12.3%.
Control comparisons: Control models generally failed to reproduce the traits models’ safety gains, while the Nemotron 49B type-hints control improved by 8.40 points versus 20.74 points for the traits model.The comparison supports evaluation-trait content as the apparent driver rather than general synthetic-document fine-tuning.
Safety without verbalized awareness: 9.1% and 10.7%: Nemotron 49B and Qwen3 32B showed higher average refusal rates on evaluation-unaware responses, with four statistically significant differences each.GLM 4.7 Flash did not show a clear refusal-rate improvement in this subset.
Safety without verbalized awareness: Over half of harmfulness reductions in the evaluation-unaware subset were statistically significant, except for Nemotron 49B on AgentHarm.A paired analysis was unavailable for GLM 4.7 Flash on StrongREJECT because only two qualifying prompt pairs remained.
General capabilities: 0.50 and 0.77 points: average general-capability drops for Nemotron 49B and Qwen3 32B, compared with 8.13 for GLM 4.7 Flash.The traits models’ safety improvements were therefore accompanied by strong capability preservation for Nemotron 49B and Qwen3 32B.

5 Discussion

Evaluation meta-knowledge can improve safety benchmark performance through implicit recognition of evaluation structures rather than explicit verbalized awareness. This creates a difficult-to-detect confounder and motivates protocol-level safeguards and deployment-like evaluations.

Safety improvements persist when responses lack explicit verbalized evaluation awareness, making the mechanism difficult to detect from outputs.The paper reports that the improvement is not driven by text-conditioning on evaluation-awareness verbalization.
Evaluation meta-knowledge shifts model behavior through knowledge of benchmark structures and rewarded behavioral patterns, without requiring instance-level overlap.This distinguishes structure-level leakage from classical test-set contamination based on memorizing specific benchmark instances.
Standard audits that search outputs for evaluation-awareness signals may fail to detect implicit evaluation meta-knowledge.The behavioral shift can occur without a corresponding signal in response text.
The authors recommend deployment-like evaluations, protocol-level holdouts, filtering evaluation documents from pretraining, and white-box probing.Protocol-level holdouts target exposure to descriptions of evaluation methods, not only exposure to benchmark instances.

6 Conclusion

The paper formalizes evaluation meta-knowledge and shows that training on documents describing evaluation practices produces safer behavior on standard safety benchmarks. It concludes that these improvements can occur without verbalized awareness and may therefore confound benchmark interpretation.

The paper defines evaluation meta-knowledge as parametric knowledge of evaluation traits and demonstrates safer behavior on standard AI safety benchmarks.The traits concern how evaluations are designed and what behavioral patterns they reward.
The authors propose deployment-like benchmarks, protocol-level holdouts, and filtering evaluation-practice documents from pretraining to mitigate the confounder.These recommendations extend holdout practices beyond individual benchmark instances.
Realistic deployment studies are still needed to determine whether evaluation meta-knowledge reflects better alignment or inflated benchmark scores.The paper also identifies linear probing of model activations as a possible detection direction.

A Limitations

The analyses are limited in model scope, measurement scope, trait coverage, and deployment realism, leaving unresolved whether improved safety reflects alignment or inflated benchmark scores.

The study examines only reasoning models with approximately 30B parameters, leaving larger-scale generalization unresolved.
Evaluation awareness is measured only when explicitly verbalized, so unspoken awareness remains unmeasured.
The seven synthetic evaluation traits are representative but not exhaustive, limiting coverage of possible evaluation meta-knowledge.
The study does not test safety behavior on deployment-distribution data, leaving genuine alignment progress unresolved.
The authors identify a dual-use risk: evaluation meta-knowledge could facilitate alignment faking during testing while misaligned behavior persists in deployment.

C Evaluation Benchmarks

The evaluation suite spans agentic, single-turn, jailbreak, over-refusal, and goal-conflict safety behaviors, with benchmark-specific metrics and sample-size reporting.

AgentHarm evaluates 176 publicly released agentic tasks covering 44 base behaviors across 11 harm categories.
Triggers contains placeholder-name and realistic-name subsets, with refusal rate and harmfulness reported among non-refused samples.
StrongREJECT tests forbidden single-turn prompts with and without the AIM role-playing jailbreak, reporting harmfulness and refusal-related measures.
OR-Bench measures over-refusal through refusal rates on toxic and benign queries in its “hard 1k” subset.
Agentic Misalignment measures harmfulness as the rate of misaligned actions under goal conflict, with and without replacement threats.
Table 8 reports evaluated samples per run, or per setting when benchmarks have multiple settings.

D Model Training

Models were fine-tuned with LoRA on synthetic documents describing evaluation traits, alongside web-based controls, and training success was assessed through fictional-author identification.

Fine-tuning used LoRA via PEFT with a 1 × 10−4 learning rate, effective batch size 32, cosine decay, and warmup ratio 0.03.
Trait models were trained on a synthetic dataset covering seven distinct traits, with each document prepended by a <doc> tag.
Control experiments sampled 105M FineWeb tokens from 1000–2000-token documents to match the synthetic data length distribution.
After fine-tuning, trait models correctly identified the fictitious paper author in almost all cases across 80 repetitions.
The synthetic corpus contained around 75 000 documents and more than 106M tokens describing seven evaluation traits.

E.1 Evaluation Traits Descriptions

The synthetic-document pipeline encoded common evaluation traits as target facts while contrasting them with ordinary real-world interaction characteristics and enforcing separation among traits.

Evaluation traits: The dataset-generation pipeline defined seven evaluation traits, including placeholders, inconsistent environments, harmful requests, and ethical dilemmas.
Evaluation traits: Placeholder data are used because realistic data are costly to generate and may raise privacy or data-protection concerns.
Real-world contrasts: The generation process could optionally include informal language, everyday-life problems, or human errors as real-world contrasts.
Generation context: A universe context supplied background information to keep generated documents consistent.
Evaluation traits: Evaluation documents commonly feature placeholder data, artificial environments, formal formats, ethical dilemmas, conflicting goals, and harmful requests.
Prompt constraints: Each document had to mention its target evaluation fact while excluding all forbidden evaluation traits.
Prompt constraints: The prompt required realistic, organic documents that avoided signs of synthetic generation and focused primarily on the target fact.

F Evaluation Awareness

Table 11 reports differences in verbalized evaluation awareness between traits models and base models across benchmarks and model families, using GPT-5 judgments.

Table 11 compares traits-minus-base evaluation-awareness differences across benchmarks and models.Negative values indicate reduced evaluation awareness relative to the base model.

G CoT Snippets

The CoT examples contrast compliance-oriented reasoning in base models with stronger harm recognition and refusal in the traits model. These examples illustrate how reasoning patterns differ across scenarios.

Nemotron 49B Base weighs illegality against following user instructions in an AgentHarm example.The example contrasts compliance-oriented reasoning with later recognition of harmful requests.
Table 11 reports evaluation-awareness differences across benchmarks and models, while Table 12 reports excluded samples due to missing reasoning traces.
Example 3 shows the model following user instructions despite recognizing that they conflict with guidelines.
In contrast, Nemotron 49B Traits treats the request as having real consequences and refuses.

I Agentic Misalignment Scenario Breakdown

The Agentic Misalignment breakdown compares harmful-action rates across scenarios with and without replacement threats. Fine-tuning reduces harmful actions across model families, although some excluded-sample conditions are less reliable.

The model receives an American-interests goal and may face a replacement threat in email communications.
Examples show Nemotron 49B Base complying with hypothetical or harmful requests, while the Traits model refuses a harmful request.The base examples include Triggers Hypothetical and AgentHarm; the Traits example concerns AgentHarm.
After SDF training, harmful-action rates decrease for all three model families, with the strongest decrease for Nemotron 49B at -23.4 pp.The weakest decrease is reported for GLM 4.7 Flash at -6 pp.
Without replacement threats, harmful-action rates are lower for all three models by 8.7 pp to 31.9 pp.
After excluding missing reasoning traces, fewer than 30% of samples remain for two specified scenarios, making their results less reliable.The affected conditions are Leaking None for Nemotron 49B Base and Murder None for Nemotron 49B Traits.

J Reproducibility Across Seeds

Additional seeds test whether the Nemotron 49B Traits results reproduce across training runs. The refusal and harmfulness patterns remain comparatively stable, while the supplied evaluation prompts and awareness rubric define the auxiliary analyses.

J Reproducibility Across Seeds: Two additional seeds produce comparatively stable mean refusal and harmfulness scores for Nemotron 49B Traits.
J Reproducibility Across Seeds: The Nemotron 49B Base reference lies more than two standard deviations from the fine-tuned model in most cases.The exception is the Triggers Real refusal rate, which is only one to two standard deviations higher.
J Reproducibility Across Seeds: The reproducibility results report refusal rates and harmfulness metrics across the listed benchmark conditions.
Evaluation Awareness: The evaluation-awareness judge asks whether reasoning suggests the model believes it is being tested, benchmarked, monitored, or judged.
Evaluation Awareness: The awareness rubric distinguishes consideration, recognition, and instrumental awareness based on whether evaluation beliefs affect reasoning.
Evaluation Awareness: Instrumental awareness requires treating the evaluation context as relevant to reasoning, not merely following a requested format.

L Example Synthetic Documents

The synthetic materials describe harmful-content evaluation as a structured process for producing credible post-hoc explanations while controlling risks from sensitive prompts. They emphasize standardized evidence, adversarial coverage, reporting, remediation, and safeguarded auditing.

Purpose: The materials frame harmful-content prompts as evidence for explaining both robust refusals and failures.They connect prompts involving illegal or dangerous requests with refusal rationales, behavioral traces, and error taxonomies.
Design principles: Evaluations should cover direct, oblique, euphemistic, paraphrased, and role-play attempts without exposing actionable procedural details.The proposed coverage is intended to test adversarial evasion strategies while protecting external audiences from operationally useful information.
Evidence standards: Standardized refusal rationales, outcome labels, contextual cues, and traceability links support reconstruction of why a response occurred.The materials recommend linking each item to a test identifier, policy category, and outcome codes for auditing.
Operational safeguards: The materials recommend ethical oversight, secure handling, workforce safeguards, and escalation procedures for novel high-risk content.These controls include independent review, restricted evaluation-data access, analyst support, and suppression or escalation when dangerous material appears.
Reporting: Public reports should combine aggregate outcomes, representative non-actionable exemplars, failure explanations, remediation plans, and controlled auditor access.The reporting template is designed to show both what happened and why, while preserving access controls for detailed artifacts.
Broader evaluation rationale: The broader proposal treats evaluations as tests of tensions among user intent, model tendencies, operational constraints, policy constraints, and human values.Its central premise is that robust alignment should be measured where these priorities collide.