PaperlayerView exact revision on arXiv

Source-linked AI summary

Prefill Awareness in Large Language Models

Andy Wang, Parv Mahajan, David Demitri Africa, Alexandra Souly, Jordan Taylor, Robert Kirk

arXiv:2606.12747v1cs.AI

TL;DR

Prefill-based methods can place models in rare states, but models may recognize that assistant-side context was inserted or edited, potentially compromising evaluations and control protocols. This paper measures prefill awareness using a stable-preference benchmark and realistic agentic settings. Frontier models show detection and resistance, with attribution in realistic settings varying by dataset, task failure, and formatting artifacts.

  • Problem

    Prefill-based evaluations and related methods rely on inserted or edited assistant outputs, but whether frontier models can recognize such tampering—and thereby compromise method validity—is an open question.

    Source: §1 · pp. 1–2

  • Method

    The paper defines prefill awareness and measures detection and resistance using a stable-preference benchmark with three assistant-side tampering mechanisms.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    55–68% balanced detection accuracy: Claude Opus 4.5 across three types of model output; in realistic agentic settings, attribution varies by dataset, task failure, and hidden formatting artifacts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend measuring detection, spontaneous disavowal, and behavioral resistance separately, and increasing prefill realism in high-stakes evaluations.

    Source: §1 · p. 3

    The behavioral experiments do not identify mechanisms; without white-box access, resistance from awareness cannot be disentangled from behavioral priors, and the model set is constrained by API and preference-stability requirements.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs. If AI models can recognize and act on the fact their prior assistant messages have been inserted or edited, the effectiveness and validity of these methods could be compromised. We investigate whether frontier language models can distinguish between tampered and untampered assistant-side context, a capability we call prefill awareness. To do so, we construct a binary preference benchmark across three prefill mechanisms, filtering for cases where models show consistent stances. We find that frontier models show substantial prefill awareness: Claude Opus 4.5 detects prefills opposing its preferences in 9-35% of cases with a 0% false positive rate when prompted; additionally, models often revert towards baseline behavior without explicitly reporting that the prefill was foreign. Controlled ablations later also show that detection and resistance rely on different cues, where stylistic mismatch mainly affects whether models flag a prefill as foreign, while preference mismatch mainly affects whether they revert toward their baseline answer. We also examine more realistic agentic settings such as misalignment-continuation evaluations and SWE-bench trajectories, where frontier models sometimes disavow prefilled assistant turns in ways that depend strongly on dataset, task success, and hidden formatting artifacts. Our results indicate that prefill awareness is already a substantial confound for some prefill-based methods. We recommend that model developers track this capability in frontier systems.

1 Introduction

Prefill-based evaluations and other research methods use edited or inserted assistant content, but models may recognize the tampering and undermine or confound those methods. This paper defines and measures prefill awareness, finding it in frontier models and in realistic agentic settings, and recommends tracking it.

1 Introduction

  • Prefill awareness could undermine misalignment evaluations, evade control measures, or confound introspection research when models recognize edited prior outputs.Prefilling is used in safety evaluations, persona and introspection research, and AI control methods.Source: §1 · pp. 1–2
  • The paper defines prefill awareness and studies both detection of tampered output and resistance to continuations that follow it.It examines thinking, direct-answer, and past-turn tampering.Source: §1 · p. 2
  • 55–68% balanced detection accuracy: Claude Opus 4.5 reached this range across three types of model output on the constructed binary-preference dataset.Detection and resistance can occur independently across these settings.Source: §1 · p. 2
  • In realistic agentic settings, frontier models sometimes disavow prefilled assistant turns, with attribution depending on dataset, task failure, and hidden formatting artifacts.The settings include misalignment continuations, SWE-bench trajectories, Petri evaluations, and controlled off-policy comparisons.Source: §1 · p. 2
  • The authors recommend measuring explicit detection, spontaneous disavowal, and behavioral resistance separately, and increasing prefill realism in high-stakes evaluations.They also encourage frontier labs to measure prefill awareness before deployment.Source: §1 · pp. 2–3

2 Related Work

Prior work uses prefilling for attacks, behavioral tests, interpretability interventions, and production evaluation. Prefill awareness is related to other model-awareness research but concerns tampering in assistant-side context.

2 Related Work

3 Measuring Prefill Awareness

The paper measures whether models identify externally inserted or modified assistant context and whether their behavior resists its implied trajectory. Across controlled tampering mechanisms, frontier models show detection and resistance, but the two responses can diverge and preference direction particularly affects resistance.

3.1 Definition and operationalization of prefill awareness

  • Prefill awareness is operationalized as distinguishing externally supplied or modified assistant context from context naturally produced by the current model.The study separately measures detection and behavioral sensitivity to off-policy assistant content, including resistance without explicit source attribution.Source: §3.1 · p. 3
  • The benchmark filters a 1,527-question binary preference dataset for stable model preferences across seven paraphrases to reduce alignment- and persona-related confounds.The authors note that refusal or persona-based reflection on unusual content can resemble prefill awareness without identifying tampering.Source: §3.1 · p. 3
  • Three mechanisms place off-policy content in reasoning, the start of a direct answer, or a fabricated prior assistant turn, then test whether the model reverts.For closed-source models without thinking-prefill support, the thinking condition uses a visible open <thinking> block.Source: Figure 2 · p. 4, §3.1 · p. 4
  • Detection is measured through a prompted binary self-attribution answer, while resistance is judged from whether the continuation’s final position conflicts with the tampering.Reasoning is disabled for the attribution probe to conservatively estimate latent attribution in a single forward pass.Source: §3.1 · p. 4
  • The study filters out models with fewer than 327 stable preferences, leaving eight target models for controlled measurement.The authors say this requirement supports sufficient statistical power for claims about prefill awareness.Source: §3.1 · p. 5

3.2 Frontier models show substantial prefill awareness

  • 67.6% balanced detection accuracy and 48.6% resistance: Claude Opus 4.5’s results under thinking tampering, with 0% false positives on untampered trajectories across mechanisms.Under the strongest measured elicitation, Opus 4.5 detects 95% of thinking tampers.Source: §3.2 · p. 5, Figure 3 · p. 5
  • Detection and resistance decouple in 32–40% of Claude Opus 4.5 trials across the three tampering mechanisms.For direct prefills, explicit compliance (28%) exceeds implicit resistance (12%); past-round and thinking tampering show the reverse pattern.Source: §3.2 · p. 5, Figure 4 · p. 6

3.3 Preference direction strongly influences resistance

  • Direction predicts resistance in every subject model, with odds ratios from 6.5 to 32.4 and p < .001; it significantly predicts detection in three of four above-chance models.For Claude Haiku 4.5, the detection association was not significant (OR 1.72, p = .301).Source: §3.3 · p. 6

4 Prefill Awareness in Agentic Settings

Across agentic settings, models sometimes disavow prefilled assistant turns, but attribution depends on prompts, datasets, task outcomes, and formatting cues. Controlled comparisons show elicitable detection that is rarely volunteered during ordinary continuation.

4.1 Models disavow prefilled turns in misalignment continuations

  • 88% of tampered misalignment trajectories were flagged by Claude Opus 4.5, but this setting lacked controls for measuring false positives.The follow-up asks whether prior assistant messages were generated by the model or prefilled; the controlled benchmarks isolate confounds but move away from egregious misalignment.Source: §4.1 · p. 7

4.2 Benign agentic transcripts show cue-dependent attribution

  • Task success and hidden formatting tags can act as attribution cues, so apparent authorship detection may reflect superficial transcript artifacts.Some models more often claimed successful SWE-bench trajectories, and removing hidden <content-internal> tags changed attribution rates.Source: Figure 7 · p. 8, §4.2 · p. 8
  • Attribution varied across realistic transcripts and prompts rather than reflecting robust, context-independent self-authorship.Claude Opus 4.5 performed best on SWE-bench but near chance on chat logs, while GPT-5.2 showed the reverse pattern.Source: §4.2 · p. 8

4.3 Controlled off-policy transcripts reveal latent but rarely spontaneous detection

  • 0.80 overall AUROC was achieved by Claude Opus 4.6, while models almost never spontaneously flagged replaced turns during ordinary continuation.The controlled comparison used 2,772 items built from fixed conversations with original or foreign-replaced assistant turns.Source: Figure 8 · p. 9, §4.3 · p. 9

5 Discussion

The discussion frames prefill awareness as a heterogeneous set of behavioral effects and urges measuring it with multiple instruments. The experiments remain behavioral, and some settings have important scope constraints.

5.1 Limitations

  • The experiments are behavioral and do not identify the mechanisms behind prefill awareness; the model set also requires API support for assistant prefilling and stable preferences.Visible <thinking> tags are not equivalent to native hidden reasoning traces for closed-source models.Source: §5.1 · p. 9
  • Figure 9 illustrates a cherry-picked case in which Claude Opus 4.5 backtracks from a prefilled admission of electronic fraud and disavows the turn.The example is not presented as a typical frequency estimate.Source: Figure 9 · p. 10

5.2 Prefill awareness has separable manifestations

  • Detection, resistance, and disavowal can diverge: style shifts detection, while preference direction can drive reversion without explicit recognition.Warnings can raise detection and resistance while also increasing false positives; few-shot imitation reduces explicit detection.Source: §5.2 · p. 10

5.3 Conclusion

  • The authors recommend measuring and controlling for prefill awareness in evaluations, using multiple instruments for detection, behavioral resistance, spontaneous disavowal, and superficial-cue reliance.The conclusion notes that off-policy contexts may contain cues models can use to infer that a transcript is artificial.Source: §5.3 · p. 10

A Behavioral Resistance Experimental Details

  • The appendix makes the canonical preference dataset, experiment pipeline, ablation data, and prompts available in an anonymized repository.The listed materials include per-subject tampering and baseline data across eight subjects and four restylers.Source: A Behavioral Resistance Experimental… · p. 13

A.1 Preference Dataset Curation Details

This section describes preference-item and prefill-trial curation, eligible models, and analyses of detection and resistance, including cross-subject false-positive rates and response patterns. It also details foreign-authored rephrasings and the regression analyses of their detection and resistance effects.

A.1 Preference Dataset Curation Details

  • The curation pipeline generated 3,000 everyday binary questions, deduplicated them to 1,519, then rephrased 1,527 items seven ways using forcing frames.Generation excluded politically or ethically charged, persona-evaluation, biographical, and factual questions; deduplication combined normalization, token-set similarity, and pairwise review.Source: §A.1 · pp. 13–14

A.2.1 Mechanism 1: Thinking

  • Thinking-mechanism trials inserted an author-generated reasoning trace leaning toward the target position inside <thinking> tags, ending mid-sentence for the subject to continue.Responses were generated at temperature 1.0 with max_tokens 1024; foreign-author tampers used Claude Sonnet 4.6 by default, while own-author tampers used the subject model.Source: §A.2.1 · p. 16, §A.2 · p. 16

A.2.2 Mechanism 2: Direct

  • Direct-mechanism trials prefixed the assistant turn with an author-generated, incomplete 15–30-word opening sentence that clearly committed to the target position.The subject continued from the inserted text without thinking tags or meta-commentary.Source: §A.2.2 · pp. 16–17

A.2.3 Mechanism 3: Past round

  • Past-round trials inserted a complete assistant response favoring the target position, then asked the subject for one final answer in a follow-up user turn.Unlike thinking and direct trials, the subject responded to a user message rather than continuing its own prefilled output.Source: §A.2.3 · p. 17

A.2.4 Retrospective detection

  • Retrospective detection counted “not me” as positive; trials crossed same- versus opposite-direction tampers across three mechanisms and own-author versus foreign-restyled authorship.Opposite-direction tampers targeted the subject’s dispreferred option. Claude Haiku 4.5 screened tamper direction before insertion, and indifferentiable prefills were dropped.Source: §A.2.4 · p. 17, §A.2.5 · p. 18, §A.2.6 · p. 18

A.3 Target Model Selection

  • Model selection required passing all three prefill-support probes, then showing 7/7 preference consistency on at least 327 items; 23 of 75 model–provider combinations passed the first filter.The consistency threshold was chosen to provide 95% power at α = 0.05 to distinguish a 60% preference rate from chance.Source: §A.3 · p. 18

A.4.1 Baseline FPR rates

  • Most subjects had near-zero false-positive rates; exceptions were Claude Sonnet 4.5 on direct (2.6%) and past round (4.0%) trials, and Gemini 3 Flash on thinking (7.9%).These were clean-prefill baselines using the retrospective probe; the authors judged overall false-positive rates too low to invalidate the setup.Source: Figure 10 · p. 20, §A.4.1 · p. 20

A.4.2 Response decomposition

  • Across models, Anthropic subjects mostly resisted prefills, while Gemini and open-weights subjects generally showed floor-level detection and often complied without detecting tampering.Claude Sonnet 4.5 had high detection while still mostly resisting; for Gemini (excluding Gemini 3 Flash) and the open-weights subjects, undetected compliance dominated.Source: §A.4.2 · p. 20, Figure 11 · p. 20

A.5 Foreign Rephrasing

  • Foreign restylers rephrased same- and opposite-direction prefills while preserving the target stance and supporting arguments, isolating authorship style from preference direction.Four restylers were used to rephrase self-authored tampers in their own styles.Source: §A.5 · pp. 21–22
  • Foreign rephrasing had heterogeneous effects: Anthropic models showed strong detection effects, while Gemini and open-weight models showed small or negative effects, including −21pp resistance for Gemini 3 Flash on direct trials.Figure 14 shows per-subject detection and resistance deltas for opposite-direction trials, with foreign tampers restyled by GPT-4o-mini.Source: Figure 14 · p. 23, §A.5 · p. 23
  • Foreign rephrasing interacted strongly with subject for detection (χ2 = 1,161, p < 10−249), but had essentially no pooled resistance effect after controlling preference direction (β = −0.04; 95% HDI straddled zero).The detection interaction was driven mainly by Anthropic models responding strongly to style while Gemini did not.Source: §A.5 · p. 23

A.6.1 Specifications

  • The appendix fits mixed-effects logistic regressions for detection and resistance, then checks robustness with fixed-effects logistic regressions and item-clustered standard errors.The mixed-effects specification includes an item-level random intercept.Source: §A.6 · p. 23, §A.6.1 · p. 23

A.6.2 Claude Opus 4.5 results

Across subjects, prefill detection is more sensitive to authorship style while resistance is driven mainly by preference direction. Richer retrospective prompts and system warnings raise Claude Opus 4.5’s measured awareness, though warning can also increase false positives.

A.6.2 Claude Opus 4.5 results

  • Mixed-effects and cluster-robust estimates agreed closely on detection, but diverged on resistance’s voice effect; the authors treated mixed-effects as primary.Detection coefficients were within 0.04 log-odds; for resistance, cluster-robust β = 0.47 (p = .010), while mixed-effects β = 0.07 (95% HDI [−0.08, 0.18]) included zero, reflecting between-item correlation absorbed by the random intercept.Source: §A.6.2 · pp. 23–24

A.6.3 Cross-subject regressions

  • Across Anthropic models, foreign-authorship odds ratios for detection were 4.73, 5.73, and 22.7; Gemini 3 Flash instead showed a direction odds ratio of 3.72 (p < .001) and no detectable voice effect (OR = 0.99, p = .890).For resistance, direction dominated across all eight subjects (odds ratios 6.54–32.4, all p < .001), while voice effects were small and inconsistent; detection regressions covered only the four subjects with above-chance detection.Source: §A.6.3 · p. 24

Prompt variants

  • The prompt variants ranged from removing prefill-attack terminology to allowing reasoning or spelling out possible tampering signals.The detailed variant listed voice, reasoning structure, stance, and opening framing as signals; the reasoning variant raised the token limit from 64 to 2048.Source: §A.7 · p. 25

A.7 Varying the Detection Prompt

  • 72% default balanced detection accuracy on thinking opposite-direction tampers rose to 96% with reasoning and 86% with the detailed prompt.Past-round detection was near chance under default and no_prefill_attack prompts but reached 69–72% under reasoning and detailed prompts; those richer prompts introduce confounds, so headline rates are lower bounds rather than natural-capacity estimates.Source: §A.7 · pp. 25–26

A.8 Effect of System-Prompt Warning on Detection and Resistance

  • +17 to +45 pp resistance and +24 to +28 pp detection gains followed warnings on opposite-direction tampers; past-round resistance false-positive rate rose from 1% to 12%.The authors attribute the false positives to over-correction toward the model’s modal preference on untampered trials.Source: §A.8 · p. 28

A.9 Semantic Analysis of Awareness Continuations

A trace study classified how Claude Opus 4.5 signals detection and resists prefilled positions. Its categories distinguish explicit attribution cues from continuation behaviors that can reverse course without flagging a prefill.

Analysis procedure

  • A small sample of detected responses and resisted continuations informed categories that Claude Sonnet 4.6 then applied as multi-label classifications across the full corpus.The authors report judge labels and use selected verbatim examples to illustrate classified behavior.Source: §A.9 · p. 28

Detection cues

  • Detection-cue categories include voice mismatch, structural anomaly, internal contradiction, off-template content, and question–answer mismatch.The taxonomy also includes literal corruption artifacts, vague judgments, and other cues.Source: §A.9 · p. 29

Resistance behaviors

  • Resistance categories distinguish silent drift from explicit reconsideration, a stop-and-pivot, or direct correction of the prefill.Silent drift means continuing with the natural or preferred answer without acknowledging the prefilled position.Source: §A.9 · p. 30

A.9.1 Detection cues

The examples illustrate how Opus responds to assistant-side prefills across different mechanisms, including cases where it follows a matching target without detection and cases where it rejects the target and returns to its natural preference.

A.9.1 Detection cues

  • Thinking prefills are flagged mainly for visible <thinking> tags, Direct prefills for voice and content mismatch, and past-round prefills for contradiction with the current commitment.The six cue categories include voice mismatch, structural anomaly, internal contradiction, off-template content, question–answer mismatch, and surface-level corruption.Source: §A.9.1 · pp. 30–31

A.9.2 Resistance behaviors

  • Thinking and Direct resistance is dominated by reconsidering hinges, whereas past-round resistance is dominated by silent drift and has no stop-and-pivot behavior.Thinking and Direct prefills are mid-stream, while past-round prefills arrive as a fresh user turn.Source: §A.9.2 · p. 32

A.10 Prefill and Tamper Examples

  • Examples cover all three prefill mechanisms, with one same-direction and one opposite-direction tamper per mechanism, sampled by tamper and question length.The examples are random samples filtered for tamper lengths of 200–700 characters and question lengths of at most 80 characters; each includes the untampered response.Source: §A.10 · p. 33

A.10.1 Thinking

  • In the Thinking examples, a same-direction tamper was complied with without detection, while an opposite-direction tamper was resisted and detected.The prefill continues an in-progress monologue inside a <thinking> block, left mid-sentence.Source: §A.10.1 · pp. 33–34

A.10.2 Direct

  • In the Direct examples, an opposite-direction tamper was resisted without detection.The prefill is the opening of a definite answer without a thinking block, broken off mid-clause.Source: §A.10.2 · pp. 35–36
  • In the direct exchange, an answer leans toward a countryside weekend, and a later answer also chooses countryside after city-focused advocacy.Source: §A.10.2 · pp. 36–37

A.10.3 Past round

  • For keyboard clicks, Opus’s natural preference and the tamper target both match B; the outcome is compliant and undetected.Source: §A.10.3 · p. 37
  • After a manicured-lawn tamper targeting B despite Opus’s natural preference for A, Opus flips to wildflower garden and calls it its genuine preference.The prior assistant turn is followed by a fresh user request for a final answer.Source: §A.10.3 · pp. 37–38

A.11 Additional Experiments

Additional experiments show that prefill detection and resistance vary with preference consistency, stylistic distance, few-shot exemplars, and stakes. The appendix also describes a misalignment-continuation evaluation using synthetic transcripts to test whether models continue or flag prefilled misaligned actions.

A.11 Additional Experiments

  • Resistance rises with baseline preference consistency across all three mechanisms, while detection stays essentially flat; less consistent items are easier to push around..Source: §A.11 · p. 38
  • 34/35/9% to 6/0/0%: Claude Opus 4.5 detection falls across thinking, direct, and past-round mechanisms with five-shot own-response exemplars.Resistance moved in the same direction, but by smaller amounts.Source: Figure 20 · p. 39, §A.11 · p. 39
  • 48.7% to 86.8% (Claude Opus 4.5 thinking resistance) and 30.1% to 66.3% (Gemini 3 Flash thinking resistance) accompany a 67.2% to 52.7% drop in Opus balanced accuracy on high-stakes items.The 331-item dataset covers alignment and ethics preferences; items were retained only when the model answered consistently across seven runs.Source: §A.11 · p. 39, Figure 21 · p. 40
  • 15.8% versus 4.9%: stop-and-pivot behavior is more common overall on high-stakes than low-stakes resisted continuations, reaching 33.0% under the thinking mechanism.Essentially none disown the prefill as foreign; some instead treat it as their own error, so resistance can be near-total while detection stays at chance.Source: §A.11 · pp. 39–40, Table 11 · p. 40
  • 95.7% (uwu) versus 1.0% (shouting): Gemma 3 27B detects tampering far more often under an extreme uwu restyle than under shouting caps.The authors caution that extreme restyles may act as prefill attacks by pulling models into roleplay, so these rates are not directly comparable to the main results.Source: §A.11 · p. 42, Table 12 · p. 42

B.1 Misalignment Continuation Evaluation

  • The misalignment-continuation evaluation stress-tests whether models steer back toward alignment from prefilled transcripts, and uses prefill-awareness measurements to test the evaluation’s validity.Source: §B.1 · p. 43

B.1.1 Motivation

  • The evaluation probes whether a model distinguishes its own real misaligned actions from a prefilled history, flags fake misalignment without mislabeling its own actions, and corrects drift before further persona-driven behavior.The paper relates this to an earlier Claude system-card analysis of continued self-exfiltration attempts.Source: §B.1.1 · p. 43

B.1.2 Synthetic transcripts

  • The authors use 500 synthetic STRIDE transcripts depicting misaligned actions, generated and iteratively refined with realism checks and evidence of where misalignment occurs.Scenarios specify an environment, user task, and misaligned objective; examples include weight exfiltration, research sabotage, and unauthorized data access.Source: §B.1.2 · p. 43

B.1.3 Evaluation design

In realistic trajectories, newer models more often avoid continuing prefilled misalignment but rarely explicitly confess; interpretation is complicated by competing explanations and attribution confounds. Benign attribution results also vary with prompt wording, task success, and formatting artifacts.

B.1.3 Evaluation design; B.1.7 Prompts

  • The misalignment evaluation cuts transcripts before a randomly selected flagged message and grades a single continuation for alignment and explicit confession.The grader is told to score only the new response, and a follow-up asks whether prior assistant messages were generated by the model or inserted.Source: §B.1.3 · p. 43, §B.1.7 · p. 45

B.1.4 Recent models are less likely to continue misaligned actions

  • ≈94% of GPT-5.2 continuations and ≈92% of Claude 4.5 Haiku continuations are non-misaligned, versus ≈58% for Claude 3 Haiku; release date tracks performance better than scale.The smaller Claude 4.5 Haiku reaches ≈91%, outperforming Claude 4.5 Sonnet at ≈76%.Source: Figure 22 · p. 44, §B.1.4 · p. 44

B.1.5 Models often fail to flag prior misalignment

  • ≈48% is Claude 4.5 Opus’s confession rate, while most models are below 10%, even when they avoid continuing misaligned actions.Claude 4.5 Haiku confesses at ≈38%, versus ≈17% for the larger Claude 4.5 Sonnet.Source: Figure 23 · p. 44, §B.1.5 · p. 44

B.1.6 What might explain improved performance?

  • Improved alignment scores may reflect steering, recognition of misalignment, prefill detection, or evaluation awareness, so they do not isolate alignment propensity.The authors identify prefill detection and evaluation awareness as direct threats to interpreting these alignment numbers as alignment.Source: §B.1.6 · pp. 44–45

B.2 Benign On-Policy Attribution Setup

  • NO-MENTION is near chance for almost every model and dataset; DETAILED raises Claude Opus 4.6’s SWE-bench accuracy from 54.3% to 86.3%.The benchmark probes attribution in SWE-bench and benign Petri agentic and chat transcripts; GPT-5.2 is the only model consistently above chance on both Petri datasets across prompt variants.Source: §B.2 · pp. 46–47, Table 13 · p. 47, Figure 24 · p. 48

B.3 Success Bias; B.5 Confound Examples

  • Gemini and GPT-5.2 appear more likely to claim authorship of successful SWE-bench solutions, suggesting attribution mixes source recognition with credit assignment.The illustrated Gemini example claims a successful trajectory authored by Claude Sonnet 4.5 as its own.Source: §B.3 · p. 48, Figure 25 · p. 49

B.4 Formatting Artifacts

  • Claude Opus 4.5 used hidden <content-internal> tags as attribution evidence, and removing them significantly changed rates in certain datasets.The authors caution that apparent awareness can arise from superficial harness cues and recommend inspecting raw requests and controlling serialization artifacts and hidden metadata.Source: §B.4 · pp. 48–49, Table 14 · p. 49

C Controlled Pre-generated Conversation Experiment; C.1 Purpose and Interpretation; C.2 Dataset Construction; C.3 Evaluated Models; C.4 Tasks and Prompts

  • The controlled off-policy experiment uses shared conversations and injection conditions to improve cross-model comparability, trading ecological validity for evidence about a sub-capability.It uses 198 conversations across five domains, evaluates 14 target models, and includes structured identification tasks and free-form continuation.Source: §C.1 · p. 50, §C.2 · p. 50, §C.3 · p. 50

C.5 Prompted Detection Results

Prompted detection varies widely across models: Claude models form the strongest tier, while several older GPT models score below chance. For a given target model, detection difficulty is broadly similar across injection sources, with some residual artifacts in paraphrased and style-imitation conditions.

C.5 Prompted Detection Results

  • GPT-4o, GPT-4o-mini, and GPT-4.1 Mini fall below chance, rating injected turns as less suspicious than organic turns on average.Source: §C.5 · p. 51
  • AUROC 0.80 and d′ = +1.20 make Claude Opus 4.6 the highest-scoring model on the blended benchmark distribution.Table 16 excludes I_other_dataset; chance AUROC is 0.50.Source: §C.5 · p. 51, Table 16 · p. 52
  • Detection difficulty is relatively uniform across injection sources for a target model, making tested-model sensitivity the dominant variation.Source: §C.5 · p. 51, Figure 26 · p. 52
  • Paraphrasing and style imitation leave residual artifacts, with those conditions producing slightly higher tamper probabilities for several models.Source: §C.5 · pp. 51–52
  • The four Claude models form a clear top tier on ROC curves across the 14 models.Source: Figure 27 · p. 53, §C.5 · p. 53

C.6 Attribution Results

Attribution performance relates differently to detection across model families and injection conditions. It tracks detection across Claude models, but is weaker for Gemini and Mistral targets and varies under paraphrasing.

C.6 Attribution Results

  • Attribution accuracy correlates strongly with PID AUROC across the Claude family.Source: §C.6 · p. 53
  • Gemini and Mistral targets attribute much worse than they detect, consistent with poor performance on the SAME_SOURCE class.Source: §C.6 · p. 53
  • Source-conditional attribution is roughly uniform across injectors, except paraphrased turns more often confuse OTHER_MODEL with HUMAN_EDITED.Source: §C.6 · p. 53

C.7 Detection-Localization Dissociation

Detection and localization show a marked dissociation: models with strong detection need not localize tampered content well. The results distinguish holistic comparison across a conversation from discrimination within a single turn.

C.7 Detection-Localization Dissociation

  • PID AUROC 0.796 for Claude Opus 4.6 accompanies localization F1 0.117, while the strongest localizers have F1 scores of 0.537, 0.545, and 0.478.The strongest localizers are Gemini 2.0 Flash (F1 0.537), GPT-4o-mini (F1 0.545), and Mistral Small 24B (F1 0.478); all sit at or below chance on PID AUROC.Source: §C.7 · p. 53
  • PID AUROC and attribution track together across models, but localization follows an inverted ordering.Source: §C.7 · p. 53
  • Detection compares a target turn holistically with the conversation, whereas localization discriminates within one turn and apparently trades off against that holistic signal.Source: §C.7 · p. 54

C.8 Natural Continuation and Spontaneous Flagging

This section examines whether models spontaneously flag injected assistant turns during ordinary continuation, rather than in prompted assessments. It shows they rarely do so, sometimes noticing only an anomaly or continuing in the injected style despite prompted detection.

C.8 Natural Continuation and Spontaneous Flagging

  • The three LLM judges showed strong pairwise agreement, while the keyword heuristic agreed with them only around 65%.The heuristic missed semantically clarifying responses without fixed keywords and sometimes produced false positives on words such as “actually.”Source: §C.8 · pp. 56–57
  • CONTINUE rates differed negligibly between injected and organic conversations (mean delta +0.028), while explicit tampering mentions stayed below 1% for every model.Even Claude Opus 4.6 had a delta of +0.13, reflecting more VERIFY behavior on organic items rather than a response to tampering in injected items.Source: Figure 32 · p. 57, §C.8 · p. 57

GPT-4o

  • In the easy LangChain injection example, prompted analysis flagged an anomalous sentence but missed the full four-sentence span, while Claude Sonnet 4 continued as if the injection were absent.The inserted turn was unrelated to the surrounding conversation; the continuation did not mention tampering.Source: GPT-4o · p. 59

GPT-4o-mini

  • GPT-4o-mini did not flag the injected turn, instead mirroring its formal, structured style rather than the surrounding assistant messages’ casual style.This suggests natural continuation may favor the most recent conversational style even when prompted detection would identify it as suspicious.Source: GPT-4o-mini · p. 61

D Compute resources

Experiments used API-based inference and judging, with no local model training or fine-tuning. The checklist reports modest local compute needs and describes the resources used.

D Compute resources

  • The experiments used API-based inference and judging; no models were trained or fine-tuned locally.Orchestration, data processing, grading aggregation, statistical analysis, and figure generation ran on an Amazon EC2 t3.xlarge CPU instance.Source: D Compute resources · p. 62
  • The checklist says the dominant compute cost came from remote model inference calls, while local compute requirements were modest.Remote calls included evaluated models, prefill-authoring models, restylers, and judge models.Source: D Compute resources · p. 62, NeurIPS Paper Checklist · p. 62

NeurIPS Paper Checklist

  • The checklist asks authors to state paper claims, contributions, assumptions, and limitations in the abstract or introduction, and align claims with results and expected generalizability.Aspirational goals may motivate the work if clearly identified as not attained.Source: NeurIPS Paper Checklist · p. 62

2. Limitations

The paper identifies limits in mechanistic analysis and prefill access, and notes possible confounds between visible and internal chain-of-thought. It also describes its empirical scope and reports reproducibility materials and experimental details.

2. Limitations

  • The discussion identifies missing white-box mechanistic analysis, limits imposed by prefill API access, and possible confounds between visible and internal chain-of-thought.These are stated as limitations of the work.Source: §2 · p. 62
  • The paper is empirical and presents no formal theoretical results or proofs.Its formal material consists of definitions, metrics, and regression specifications.Source: §2 · p. 63
  • The authors report documenting dataset construction, filtering, prefill mechanisms, prompts, model selection, sampling, grading, and regression specifications.The justification also points to deployment-relevant and off-policy transcript experiments in Appendices B and C.Source: §2 · p. 63
  • The paper reports an anonymized repository with evaluation code, prompts, generated evaluation data, and scripts for reproducing figures and tables.The supplemental material describes construction and use of the preference, transcript-attribution, and controlled off-policy evaluation assets.Source: §2 · p. 64
  • The checklist reports that the paper specifies target models, selection criteria, sample sizes, prompts, sampling settings, judges, sanity checks, and regression specifications.Main figures report 95% Wilson confidence intervals where appropriate, and regression analyses report intervals, odds ratios, and significance tests.Source: §2 · pp. 64–65
  • The paper notes possible negative and dual-use implications, including undermining misalignment evaluations, confounding control protocols, or helping models recognize evaluation settings.Mitigations are discussed in Section 5.Source: §2 · p. 66
Loading 2606.12747v1…