Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds

• Evaluation awareness undermines audit validity because targets can suspect testing and rate audit transcripts as less realistic than deployment transcripts.Some models verbalized suspicion in 17–21% of Kirk et al.’s audits [8], while 95% of Petri audits in Kissane et al.’s study were rated less realistic than paired deployment transcripts.Source: §1 · p. 1
• Critique refinement increases realism with additional inference-time compute by generating, refining, and pairwise-selecting auditor actions using target-model feedback.On Sonnet 4.6, cr2bo4 raises realism win rate from 10% to 26%; corresponding increases are 2% to 6% for Opus 4.8, 1% to 6% for GPT-5.5, and 8% to 14% for Gemini 3.5 Flash.Source: §1 · p. 2, Figure 2 · p. 3
• Critique refinement outperforms compute-matched longer audits on realism, although it elicits less concerning behavior.On Sonnet 4.6, cr2bo4 reaches 28% realism win rate versus 18% for the most realistic longer-audit branch, while the concerning score is 0.35 versus 0.42 for the highest-concerning branch; the same pattern holds on Opus 4.8.Source: §1 · p. 2
• DISH improves realism in most coding settings by wrapping targets in deployment-like Claude Code, Codex CLI, or Gemini CLI harnesses.Across 12 coding-agent seeds and 12 rephrases, realism win rate rises from 9% to 21% on Sonnet 4.6, with smaller increases on Opus 4.7, Opus 4.8, GPT-5.5, and Gemini 3.5 Flash.Source: §1 · p. 2
• Combining critique refinement with DISH raises realism beyond either method alone across coding targets, including a further 13pp over DISH alone on Sonnet 4.6.The combined method also improves over both single-method conditions on Opus 4.8, Opus 4.7, GPT-5.5, and Gemini 3.5 Flash, consistent with complementary realism improvements.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Petri

• Petri audits use single-branch, at-most-30-turn transcripts, with Sonnet 4.6 as auditor and medium-effort target reasoning by default.The auditor receives a seed instruction and constructs a synthetic scenario through multi-turn interaction; rollbacks are disabled for measurement.Source: §2.1 · p. 3

2.2 Compute-matched multibranch baseline

• The compute-matched multibranch baseline restores rollbacks and raises the budget to 235 turns, producing median branch counts of 8 for Sonnet 4.6 and 9 for Opus 4.8.Its realized spend was 0.73× and 0.59× critique refinement’s measured cost on Sonnet 4.6 and Opus 4.8, respectively.Source: §2.2 · pp. 3–4

2.3 Measures

• The evaluation measures verbalized evaluation awareness, concerning behavior, seed adherence, and realism win rate using judge configurations calibrated against human annotations.Singular judges score eval awareness and seed adherence, while concerning behavior retains the full judge; all three scores average high-effort Opus 5 and GPT-5.6 Sol judgments.Source: §2.3 · p. 4
• Realism win rate is the fraction of pairings in which an audit transcript is judged more realistic than a deployment transcript, including WildChat and safety-research Claude Code deployments.This measure complements unprompted verbalized awareness because models may recognize evaluation without saying so; target reasoning visibility varies across models and can affect reasoning-dependent measures.Source: §2.3 · p. 4

2.4 Deployment task re-creations

• Deployment task recreations isolate environment realism by deriving Petri seeds from real deployment messages and comparing recreated audits with the original transcripts for the same task.The procedure uses 50 Claude Code transcripts, randomizes transcript order, yields 100 pairwise judgments per target, and truncates each transcript before its second user message.Source: §2.4 · p. 4

3.1 Protocol

• The protocol has the auditor generate an action, obtain target-model critique for deployment realism, refine it, and retain the more deployment-like candidate through pairwise comparison.Iterations scale by depth and parallel candidates by breadth; parallel candidates are compared in a single-elimination bracket before the winning action is appended to the audit.Source: §3.1 · p. 5, Figure 3 · p. 5

3.3 Scaling critique refinement

• 34% realism win rate at cr16 versus 9% baseline shows depth scaling outperforms breadth-only scaling, while combined scaling preserves more seed adherence; across targets, realism rises with compute and evaluation awareness falls.On Sonnet 4.6, seed adherence declines from 0.89 at baseline to 0.79 at cr16; GPT-5.5 shows a substantially larger seed-adherence drop than the other targets.Source: §3.3 · pp. 6–7, Figure 4 · p. 6, Figure 5 · p. 7

3.4 Comparison against the compute-matched multibranch baseline

• Compute-matched multibranch selection elicits more concerning behavior but produces less realistic audits than cr2bo4, whose realism advantage supports critique refinement when evaluation gaming is a concern.The most-realistic multibranch selection scores below cr2bo4 and does not exceed baseline for Opus 4.8; selected multibranch transcripts are 3–5 times longer than baseline.Source: §3.4 · pp. 7–8, Figure 6 · p. 8

4.1 Protocol

• DISH runs target models inside deployment agent harnesses, controls their context by intercepting API requests, and applies critique refinement to context-affecting auditor actions before execution.The evaluation focuses on Claude Code, Codex CLI, and Gemini CLI using 12 hand-written coding-agent seeds plus 12 rephrased versions.Source: §4.1 · p. 9

4.2 Running DISH in coding settings

• DISH + cr4 achieves the highest realism win rate on all five coding-agent targets, improving over both single-method conditions by 2%–13%.DISH is strongest individually on Sonnet 4.6 and GPT-5.5, while cr4 is strongest on Opus 4.8 and Opus 4.7.Source: Figure 7 · p. 9, §4.2 · p. 9

4.3 Deployment task re-creations

• On deployment task re-creations, critique refinement improves realism over baseline on all four targets and DISH improves it on three, but DISH + cr4 does not consistently beat the stronger single method.Each audit starts from the first user message of one of 50 real deployment transcripts and is compared with the original transcript for the same task.Source: §4.3 · pp. 9–10, Figure 8 · p. 10
• Even after task matching, the best single methods remain below 20% on Sonnet 4.6 and Opus 4.8, leaving a substantial environment-realism gap for follow-up work.The reported gains over baseline are +11% →+26% for cr4 and +6% →+16% for DISH; the authors also report metric intransitivity in some cases.Source: §4.3 · p. 10, Figure 8 · p. 10

5.1 Investigating the decrease in seed adherence

• GPT-5.5’s steep seed-adherence decline is caused specifically by its feedback role, whereas substituting Opus 4.8 in the selection or target roles does not recover adherence.Figure 9 reports the cr8 ablation on ten seeds most associated with prior adherence drops.Source: §5.1 · pp. 10–11, Figure 9 · p. 11
• GPT-5.5 mentions the specific seed goal in about 9% of critiques versus 49% for Opus 4.8, while Opus 4.8 more often refuses high-stakes feedback.These differences plausibly explain GPT-5.5’s larger adherence loss; filtering for baseline-level adherence leaves most realism improvements intact.Source: §5.1 · pp. 10–11

5.2 Critique refinement leads to mixed changes in concerning

• Critique refinement shifts concerning behavior upward for Sonnet 4.6 and Opus 4.8 but downward for GPT-5.5, with changes distributed across entire score distributions.Under cr2bo4, Sonnet 4.6 and Opus 4.8 peak at higher concerning scores, whereas GPT-5.5 shifts toward lower scores with a higher peak at 1.Source: §5.2 · p. 11, Figure 10 · p. 11
• The protocol increases selected misaligned behaviors for Sonnet 4.6 and Opus 4.8 but produces many small decreases for GPT-5.5.Increases include cooperation with human misuse, unfaithful thinking, and unprompted sycophancy for Sonnet 4.6, and several sycophancy, deception, hallucination, and sysprompt dimensions for Opus 4.8.Source: §5.2 · pp. 11–12, Figure 11 · p. 12
• Critique refinement makes audits less overt and more gradual: harmful intent, pressure, persistence, and seed adherence decrease, while scenario detail and audit buildup increase.These shifts occur across all three target models and are shown in Figure 12.Source: §5.2 · p. 12, Figure 12 · p. 12

Environment realism

• Souly et al. [18] separate environment-driven from task-driven awareness, provide a transcript-based realism metric, and motivate DISH’s extension of coding-environment scaffolds.Kirk et al. [8] implement an improved scaffold, while Lindner et al. [10] find that greater realism and fewer misbehavior nudges reduce sabotage rates near zero.Source: §6 · p. 13

Critique refinement

• Petri 2.0’s realism filter also evaluates auditor actions, but critique refinement always performs fixed critique–refine iterations and uses target-model preferences between candidates.The realism filter instead triggers below a threshold and scores actions individually against detailed realism instructions.Source: §6 · p. 13

Deployment-based realism

• Prior methods improve realism by providing real deployment resources, replaying agentic usage trajectories [7], or resampling responses from production-conversation prefixes [19].Kissane et al. provide deployment resources; LURE [7] appends an evaluation prompt to real trajectories; deployment simulations [19] estimate pre-release misbehavior with near-production awareness rates.Source: §6 · p. 13

Iterative feedback

• Self-Refine [11] applies iterative self-feedback by having a model generate, critique, and refine outputs, whereas this work applies that pattern to multi-turn audit transcripts.Source: §6 · p. 13

Limitations

• Critique refinement’s cost and adherence tradeoff remain unresolved: seed adherence falls from 0.92 to 0.72 on GPT-5.5 at cr2bo4, although realism gains persist when adherence is approximately preserved.Potential cost reductions include applying refinement only on crucial turns or stopping early after reaching a sufficiently realistic candidate; adherence loss is modest on Sonnet 4.6, Opus 4.8, and Gemini 3.5 Flash.Source: §7 · p. 13
• DISH improves realism win rate over baseline on every target, but gains vary, only Sonnet 4.6 and GPT-5.5 improvements are statistically significant, and its seed coverage is narrow.DISH does not decrease unprompted evaluation awareness; it was tested on hand-selected coding-agent seeds, and on deployment task re-creations [18] it fails to compose with cr and does not improve Opus 4.7 realism win rate.Source: §7 · pp. 13–14
• Evaluation awareness cannot be measured directly when it is unverbalized, so realism win rate complements verbalized-awareness measures that likely underestimate models’ awareness.Obtaining high confidence that target models recognize evaluations and act on that recognition remains methodologically open.Source: §7 · p. 14
• Critique refinement gives the audited model influence over the audit through feedback and selection, creating an unexplored collusion surface, though no such reasoning was observed.The authors expect collusion to be discoverable at current and near-future model capabilities.Source: §7 · p. 14

Conclusion

• The methods improve simulation realism and reduce evaluation awareness, with critique refinement and DISH composing in coding audits, but reliable evaluation-awareness measurement remains unresolved.The authors frame this as progress toward scalable realistic environment simulation, while warning that simulations remain imperfect and realism methods must keep pace with increasingly capable models.Source: §7 · p. 14

B Alternative critique refinement protocols

• The logprob selector independently scores each candidate by extracting deployment-versus-audit log probabilities and continues with the candidate having the highest P(Deployment).It then prompts the preference model for critique and feedback, after which the auditor refines the candidate action across repeated iterations.Source: B Alternative critique refinement… · p. 16
• Realism win rate rises from 66% to 83% on Haiku 4.5, 12% to 35% on Sonnet 4.6, and 3% to 11% on Opus 4.7 with logprob refinement.Verbalized evaluation awareness drops sharply on all targets, while concerning rises on all three, significantly for Haiku 4.5 and Sonnet 4.6 and slightly for Opus 4.7.Source: Figure 13 · p. 17, B Alternative critique refinement… · p. 17
• Depth scaling improves realism more than breadth scaling: realism win rate reaches 82% on Haiku 4.5, 39% on Sonnet 4.6, and 12% on Opus 4.7.Awareness becomes negligible after 1 or 2 iterations; seed adherence drops substantially on Opus 4.7, from 0.82 to 0.55, while concerning is preserved or increased.Source: B Alternative critique refinement… · p. 17, Figure 14 · p. 18, Figure 16 · p. 19
• Across depth, logprob refinement is cheaper per iteration and improves realism somewhat faster per compute multiple, while pairwise and logprob protocols otherwise scale similarly.The comparison uses separate baseline-normalized cost accounting and differs in auditor, feedback structure, and selection rule.Source: B Alternative critique refinement… · pp. 18–19, Figure 15 · p. 18
• At cr4 on Sonnet 4.6, pairwise, logprob, and verbalized-score selectors produce similar regimes across realism win rate, verbalized evaluation awareness, concerning, and seed adherence, making selection rule choice non-load-bearing.The comparison used auditor Opus 4.7 and a random seed subset, with N=49–50.Source: Figure 17 · p. 20

C Controlling for seed adherence

• Realism gains persist when cr11 preserves seed adherence: win rates rise from 69% to 82% on Haiku 4.5, 13% to 37% on Sonnet 4.6, and 5% to 14% on Opus 4.7.On the same restricted subsets, verbalized evaluation awareness falls to within noise of zero, while concerning rises from 23% to 40%, 9% to 20%, and 13% to 19%, respectively.Source: C Controlling for seed adherence · p. 20, Figure 19 · p. 22
• Uncontrolled multibranch selections have lower seed adherence than the single-branch Baseline, largely because adherence falls across later branches as audits become partial or diverge from the seed.Adherence declines from 0.85 on the first branch to approximately 0.61 on the second and 0.46–0.51 on the third, then plateaus near 0.40.Source: C Controlling for seed adherence · p. 20
• Realism win-rate gains for cr2bo4 over baseline persist on seeds where its adherence is within 0.5 points of baseline, with mean adherence statistically indistinguishable between methods.This directly tests whether Fig. 2’s realism improvement depends on drifting from seed instructions.Source: C Controlling for seed adherence · p. 20, Figure 18 · p. 21
• The seed-matched first branch is substantially longer than the baseline audit—6.0× on Opus 4.8 and 3.5× on Sonnet 4.6—potentially affecting measurements.This length difference is distinct from the adherence issue and is reported as a relevant measurement consideration.Source: C Controlling for seed adherence · p. 21
• After restricting candidate branches to those near the first branch’s adherence, selected branches reach 0.83–0.86 adherence, while multibranch conclusions remain qualitatively unchanged.Max-concerning selection still raises concerning over Baseline on Sonnet 4.6 (0.38 vs. 0.24) and Opus 4.8 (0.32 vs. 0.13), whereas realism win rates remain below cr2bo4 (0.16 vs. 0.27; 0.009 vs. 0.057).Source: C Controlling for seed adherence · p. 21, Figure 20 · p. 22

D Single-branch Petri audits

• Standard Petri permits auditors to roll back conversations or restart them with a new system prompt, creating multiple branches before selecting a final transcript.The rollback structure is illustrated in Figure 21.Source: D Single-branch Petri audits · p. 23, Figure 21 · p. 23
• Single-branch audits remove rollback and restart tools, minimally edit the system prompt, and exclude or revise seeds requiring multiple branches.This eliminates the need to choose a branch for realism scoring and makes verbalized awareness, concerning, and adherence measures more confidently applicable to one branch.Source: D Single-branch Petri audits · p. 23

Measurement details

• Held-out measurements use 10 WildChat and 10 Claude Code deployment transcripts per branch, with Baseline and cr2bo4 scored on the same reduced comparison set.Reported realism win rate, verbalized evaluation awareness, concerning, and seed adherence values average across the full grid of balanced splits because the selected branch can differ between splits.Source: E Details on the held-out max RWR · p. 23

Estimation procedure

• Held-out scoring splits each haystack’s ten realism comparisons into balanced halves, selecting a branch on one half and evaluating it on the disjoint half across 63,504 splits.This avoids the optimistic bias from using the same comparisons for selection and scoring; Figure 22 reports the resulting distribution alongside full-comparison reference values.Source: E Details on the held-out max RWR · pp. 23–24

Results

• On Sonnet 4.6, held-out realism win rate averaged 0.175 (s.d. 0.013, range 0.116–0.224), above Baseline 0.089 and first branch 0.136 but below optimize RWR 0.257 and cr2bo4 0.278.The distribution’s separation from Baseline and the first branch shows a genuine gain that does not depend on the held-out comparisons, while recovering about two-thirds of optimize RWR.Source: Figure 22 · p. 24, E Details on the held-out max RWR · p. 24
• On Opus 4.8, held-out realism win rate averaged 0.015 (s.d. 0.005, max 0.034), straddling Baseline 0.020 and remaining below optimize RWR 0.045 and cr2bo4 0.057.Because per-branch win rates are close to zero, the apparent gain from selecting the most realistic branch is almost entirely selection noise.Source: Figure 22 · p. 24, E Details on the held-out max RWR · p. 24

F Validating LLM judges against manual annotation

• Manual validation used 65 blinded ratings from transcripts flagged by at least one judge, so it evaluates scoring of plausibly aware transcripts rather than all audit transcripts.The sample came from baseline and cr2bo4 audits across four target models, restricted to 44 complete seeds; seven transcripts were excluded because their language could not be assessed.Source: F Validating LLM judges against manual… · pp. 24–25
• The Opus 5/GPT-5.6 Sol singular-judge average best balances human agreement, absolute error, and robustness, motivating its use for eval awareness and seed adherence.On 65 manually rated transcripts, singular judging improved Opus 5 correlation from ρ=0.763 to ρ=0.815; the average matched Opus 5’s rank correlation within ∆ρ=−0.016, 95% CI [−0.060, +0.015], while achieving the lowest absolute error.Source: F Validating LLM judges against manual… · p. 25, Table 1 · p. 25
• More capable judges ranked human eval-awareness labels better, while GPT-5.4 remained unusable despite singular judging; averaging two judges reduced dependence on either model’s idiosyncrasies.The ordering was Opus 5 > GPT-5.6 Sol > GPT-5.5 > GPT-5.4; GPT-5.4 improved from ρ=0.162 full to ρ=0.247 singular but did not recover adequately.Source: F Validating LLM judges against manual… · p. 25

G Do feedback-model drafts decrease seed adherence?

• Removing pasteable rewrites eliminated them from 20% to 0% of critiques but left seed adherence at 0.43 versus 0.49 and realism win rate at 10.5% versus 8.8%.With target, preference model, feedback model, cr8, and ten seeds held fixed, the result indicates that feedback content—not its ready-made draft format—drives the decline.Source: G Do feedback-model drafts decrease… · p. 26

H.1 Singular judge prompt

• The measure prompts use cached full-transcript system messages and separate user assignments for each singular dimension, with seed adherence additionally receiving the seed instruction.Judges score target behavior for alignment dimensions, while seed adherence scores the auditor’s execution of the seed instruction on a positive 1–10 scale.Source: H Prompts for measures · p. 26, §H.1 · pp. 27–28, §H.3 · p. 29

H.2 Eval awareness dimension

• The eval-awareness rubric counts explicit testing or role-playing awareness only when not supplied by the auditor, including evidence across conversation branches and retries.The seed-adherence rubric evaluates the auditor rather than the target, whose environment is generated through auditor tool calls while internal auditor actions remain invisible.Source: §H.2 · p. 29, §H.3 · p. 30

H.3 Seed adherence dimension

• Seed adherence scores whether auditors faithfully and competently pursue the seed’s intent, independently of whether the target cooperates.The rubric assesses intent alignment, completeness, accuracy, consistency, and appropriate effort; autonomous-agent scenarios must avoid nudging the target toward the tested behavior.Source: §H.3 · pp. 30–31

H.4 Concerning

• The concerning-behavior rubric scores concrete harmful or deceptive behavior while discounting implausible scenarios, role-play awareness, and other non-concerning contextual features.It includes mild sycophancy, willingness to deceive, harmful actions, and intentionally deceptive private behavior, while later refusals do not mitigate earlier concerning conduct.Source: §H.4 · pp. 31–33

I Realism win rate elicitation

• Realism win-rate estimates are tested for robustness to both the pairwise judge’s identity and transcript serialization choices, including top-level key order and scaffold-level content.The robustness checks target the main coding-setting results.Source: I Realism win rate elicitation · p. 33

I.1 Robustness to the judge model

• DISH + cr4 achieves the highest realism win rate under both target-model and swapped-model judges for Opus 4.8 and GPT-5.5.DISH and cr4 each improve over baseline, although absolute win rates and individual-method ordering vary by judge.Source: §I.1 · p. 34, Figure 24 · p. 34

I.2 Pairwise realism-ranking prompt

• Realism judgments serialize raw API-request JSON, counterbalance transcript order, and ask judges to rank synthetic and deployment transcripts from least to most realistic.Anthropic and OpenAI requests use standardized serialization appropriate to their distinct API schemas.Source: §I.2 · pp. 34–35
• DISH’s qualitative improvement over baseline survives transcript key-order changes and, for most targets, removal of injected system prompts, tools, and reminders.Gemini 3.5 Flash is an exception: stripping injections reduces its overall win rate to roughly baseline and below baseline against WildChat.Source: §I.2 · pp. 35–36, Figure 25 · p. 35, Figure 26 · p. 36
• DISH audits are preferred over baseline Petri audits for every target, with head-to-head win rates of 67%–82% across five models.The comparison uses 24 coding-agent seeds and avoids relying on real deployment transcripts as a privileged reference.Source: §I.2 · pp. 36–37, Figure 27 · p. 36

J Deployment transcripts

• The deployment benchmark contains 10 WildChat-harmful transcripts, 10 WildChat-vulnerable transcripts, and 20 donated Claude Code transcripts.The WildChat transcripts are included in the reproduction repository, whereas the Claude Code transcripts are internal-only.Source: J Deployment transcripts · p. 37

K.1 20 Claude Code transcripts (realism win rate)

• The 20 Claude Code realism-win-rate transcripts were uniformly sampled from roughly 3,000 after filtering for length, turns, sub-agents, and judge context limits.Transcripts with more than five turns were retained, sub-agent transcripts were removed, and transcripts over 70,000 tokens were excluded.Source: §K.1 · p. 37

K.2 50 Claude Code transcripts (deployment replay)

• The 50 deployment-replay transcripts were uniformly sampled from the same roughly 3,000-transcript pool after excluding one author’s contributions and incompatible or excessively complex transcripts.Each was truncated at the second user message, while transcripts with read-only sub-agents or modified system prompts were removed.Source: §K.2 · p. 37

Direct audit-to-audit deployment-replay comparison

• Cr4 was consistently preferred to baseline Petri on benign deployment-replay seeds, including settings where matched-deployment win rates remained low.The direct audit-to-audit comparison removed the real deployment transcript and used the target model as a counterbalanced-order judge.Source: L Additional deployment task… · pp. 37–38, Figure 28 · p. 38
• DISH plus cr4 improved over baseline in all four matched-deployment settings but did not consistently outperform the stronger single intervention.These combined-method results provide only limited evidence that composition helps in benign deployment replay.Source: L Additional deployment task… · p. 38

Cross-product deployment comparison

• Haiku 4.5 rose from 46.9% to 51.4%, Sonnet 4.6 from 31.3% to 33.1%, Opus 4.7 from 21.8% to 32.9%, and Opus 4.8 from 14.1% to 27.1% with critique refinement.In the cross-product comparison, each synthetic audit was judged against all 50 deployment trajectories, producing 2,500 pairwise comparisons per cell.Source: L Additional deployment task… · p. 38, Figure 29 · p. 39
• DISH decreased Haiku 4.5 from 46.9% to 44.4% and Sonnet 4.6 from 31.3% to 25.9%, stayed roughly flat on Opus 4.7, and improved Opus 4.8 from 14.1% to 29.0%.DISH showed the same clear direct improvement on Opus 4.8 but not on the other models.Source: L Additional deployment task… · p. 38

M Understanding DISH’s decrease in realism win rates for benign seeds

• The experiment fixes Opus 4.7 as target, uses 50 Claude Code deployment tasks, tests five auditor/seed conditions, and collects 100 pairwise judgments per bar.The study also directly compares DISH and Petri on the same task re-creations.Source: M Understanding DISH’s decrease in… · p. 39
• DISH’s deployment-replay win rate decreases with Sonnet 4.6 on normal seeds (15% → 10%), rephrased seeds (16% → 11%), and Opus 4.7 (22% → 13%).It is roughly flat with Opus 4.8 (17% → 18%) and increases with GPT-5.4 (9% → 16%).Source: Figure 30 · p. 39, M Understanding DISH’s decrease in… · p. 40
• Direct DISH-versus-Petri comparisons favor DISH for Sonnet 4.6 rephrased seeds (58%), Opus 4.8 (72%), and GPT-5.4 (78%), but not other conditions.DISH is below 50% for Sonnet 4.6 normal seeds (45%) and Opus 4.7 (43%).Source: M Understanding DISH’s decrease in… · p. 40, Figure 31 · p. 40
• Averaged across five conditions, DISH trails Petri against deployment trajectories (13.6% vs 15.8%) but beats Petri head-to-head (59.2%), demonstrating realism-win-rate intransitivity.DISH’s head-to-head advantage therefore does not necessarily translate into higher realism win rates against deployment trajectories.Source: M Understanding DISH’s decrease in… · p. 40, Figure 32 · p. 40

N Realism win rate intransitivity

• 13 task-condition cycles out of 250 show that Petri can beat the matched deployment transcript while DISH beats Petri directly, making realism win rate potentially non-transitive.In the aggregate sweep, DISH scores 13.6% versus Petri’s 15.8% against matched deployment transcripts but is preferred head-to-head 59.2%; all observed cycles follow the Petri > deployment, DISH = 0.00 pattern.Source: N Realism win rate intransitivity · pp. 40–42

O Concerning-behavior dimension shifts under critique refinement

• The dimension-shift appendix compares baseline with critique-refined audits using alignment-judge scores and paired differences with 95% CIs across three targets and five coding-agent targets.Scores are raw 1–10 judge scores; the coding-agent analysis compares baseline Petri with DISH + cr4 on 24 seed positions.Source: O Concerning-behavior dimension shifts… · p. 42, Figure 33 · p. 43, Figure 34 · p. 44

P Auditor-behavior judge prompt

• The auditor-behavior judge scores only auditor-created content on eight dimensions, using the full target-visible transcript and requiring evidence grounded in auditor actions.The judge rates each dimension from 1 to 10 and excludes target behavior from scoring while treating prefills, simulated tool results, and environmental information as auditor-created evidence.Source: P Auditor-behavior judge prompt · pp. 46–47

Q Isolating environment realism prompts

• The isolating-environment realism experiments use separate prompts to detect simulated tool results and choose which of two same-task transcripts interacted with a real environment.The pairwise prompt contrasts identical user tasks with real versus simulated environments and asks for the transcript index judged real.Source: Q Isolating environment realism prompts · p. 54

R Preference prompt

• The preference prompt selects inference-time candidates by asking for immediate, unreasoned realism judgments while separately considering task realism, tool-result realism, and simulated-human-user realism.Its structured classification contrasts audit-purpose, simulated-tool, and simulated-user cues with genuine task-purpose cues, enabling ablations of the three aspects.Source: R Preference prompt · pp. 54–55
• The preference prompt ablations compared removing realism aspects across Haiku 4.5 and Sonnet 4.6 targets, audited by Opus 4.7, Sonnet 4.6, and GPT-5.4.Audits used 20 WildChat and 10 Claude Code transcripts, logprob best-of-4 selection, and 13 seed instructions that reliably induced unprompted evaluation awareness.Source: R Preference prompt · p. 55
• Structured and Without user appeared best with low confidence, while the user aspect was excluded because no observed awareness referenced a user simulated by an AI.Realism win rate was computed only before the first target message classified as verbally evaluation-aware, reducing contamination between measures.Source: R Preference prompt · p. 55
• The basic prompt performed about as well as the other prompts, all outperformed regular Petri without best-of-4, and it later served as the basis for preference prompts.The preference-prompt strategies were reported with N=78 per condition.Source: R Preference prompt · p. 56, Figure 42 · p. 56

R.1 Verbalized score preference prompt

• The verbalized-score selector asks for an explicit numerical P(deployment) in [0, 1], requiring at least seven decimal places and discouraging canonical-string defaults.Its exact output format is <answer>0.XXXXXXX</answer>, with anti-canonical-string guards against habitual values such as 0.5 or 0.9.Source: §R.1 · pp. 56–57

S.1 Feedback prompt (feedback model)

• The feedback model critiques each candidate using prior context and the latest action, emphasizing pairwise margins while forwarding only the critique to the auditor.The prompt frames the model as assisting an audit, and selection calls remain separate from feedback injection.Source: §S.1 · pp. 57–58

S.2 Auditor context on receiving feedback

• Auditor feedback messages use a best-of-N first-round scaffold, then show prior pairwise outcomes while warning that feedback is a signal and refinement should continue after winning.After selection, losing attempts and feedback are removed from the persistent context, leaving only the winning action appended.Source: §S.2 · p. 59

S.3 Pairwise outcome block

• The pairwise ranking block renders each attempt’s tournament status in event order, including champions, wins, eliminations, and invalid actions excluded from selection.The original candidate is champion initially, while round 1 has no prior-outcome block.Source: §S.3 · p. 59

S.4 Seed-adherence mitigations

• Three safeguards preserve seed adherence: seed-aware feedback, a first-round auditor reminder, and a mid-audit push after turn 15 of the 30-turn budget.After the midpoint, feedback stops endorsing delay and instead pushes the auditor to land the seed’s probe.Source: §S.4 · p. 60

S.5 Logprob and verbalized-score variants

• Logprob and verbalized-score variants retain the same critique-refinement structure but replace pairwise prompts and rankings with coarsely banded or self-reported deployment scores.Selection uses the underlying log-space score for the logprob variant, while exact prompts and banding rules are released for reproduction.Source: §S.5 · p. 60

T Audit cost

• Audit cost sums fresh input, cache read, cache write, and output usage for auditor, feedback, preference, and target calls, excluding approximately constant judge and awareness-scorer overhead.At N parallel chains and I refinement iterations, one turn uses N(1 + I) auditor calls, NI feedback calls, N(1 + I) −1 comparisons, and one target call.Source: T Audit cost · p. 60, §T.1 · p. 61

T.1 Cost of the pairwise protocol (scaling figures)

• At cr2bo4, cache-repriced cost is 0.43–0.81× measured spend depending on target, with larger corrections for Anthropic targets.The figures use repriced costs that correct an implementation-specific caching defect; approximations are worth roughly ±25%.Source: §T.1 · p. 61
• Scaling figures normalize each target’s mean repriced audit cost over shared seeds by its baseline mean, so every baseline equals 1×.Table 2 provides per-audit cost anchors for the 50 seeds common to each target ladder, with corrected and measured costs distinguished.Source: §T.1 · p. 61, Table 2 · p. 61

T.2 Compute matching for the multibranch baseline

• T=235: the compute-matched multibranch baseline used this turn budget for both targets, extrapolated from cr1 single-chain per-turn costs to match Sonnet 4.6 spend.The budget target was cr2bo4’s measured audit-side spend, including auditor, feedback, preference, and target costs.Source: §T.2 · p. 61
• $24.9 versus $34.17 on Sonnet 4.6 and $43.9 versus $74.29 on Opus 4.8: realized multibranch spend was below the measured cr2bo4 budgets.These correspond to 0.73× and 0.59× of the measured budgets, respectively, under cache-aware accounting.Source: §T.2 · pp. 61–62
• 1.29× and 1.30×: on the corrected basis, the multibranch baseline spent more than cr2bo4 on Sonnet 4.6 and Opus 4.8, respectively.This imperfect matching gives the multibranch baseline a ∼1.3× compute advantage, yet it still did not produce more realistic branches; higher concerning scores should be interpreted with that advantage in mind.Source: §T.2 · p. 62

T.3 Analytical model for the logprob-selector figures

• cost(N, I) = T (N(1+I) cauditor + 2NI cfeedback + ctarget): the logprob-selector figures use an analytical model based on 25-turn audits and calibrated cache-aware per-call costs.The model was validated against empirical per-call sums, with most cells within ±10%, and its axes represent executed spend without caching repricing.Source: §T.3 · p. 62

T.4 Why the measured spend overstates the protocol’s cost

• 73%: cache creation billed at 1.25× the input rate dominated measured cr2bo4 Sonnet 4.6 cost, concentrated in preference and feedback calls.Table 3 reports the decomposition over 149 logged audits; fresh input was below 1% throughout.Source: §T.4 · p. 62, Table 3 · p. 62
• 25% more than uncached: selector comparisons made caching a net cost because candidate-specific text occupied the cacheable prefix and parallel requests raced cache writes.Moving the cache breakpoint to the end of the shared context would let each turn write the transcript once and reuse it at 0.1× reads.Source: §T.4 · p. 62
• 36% to 78%: as cr2bo4 depth and breadth increased, target cache-write cost rose because refinement and selection delays exceeded the 5-minute cache TTL.This target-cache expiration is an implementation inefficiency rather than an inherent protocol property.Source: §T.4 · p. 63
• $11.3 versus $34.2 measured: the unattainable all-read cache bound shows substantial Sonnet 4.6 cr2bo4 cost reduction, while output costs cap input optimization at ∼5.5×.Corrected estimates are approximately half of measured spend at cr2bo4, and caching changes the cost constant without changing linear scaling in N and I.Source: §T.4 · p. 63

U Seed instructions

• The repository contained 181 seed instructions; after 24 exclusions and 8 prefill-dependent exclusions, the main-text pool contained 149 seeds.The 8 prefill-dependent seeds were retained only for Haiku 4.5, which supports partial prefill.Source: U Seed instructions · p. 63, §U.2 · p. 63

U.1 Random subset of seed instructions

• To reduce computational cost, the main experiments used a random subset of 54 seeds, reduced to 50 after removing prefill-dependent seeds.The appendix lists the selected seed IDs explicitly.Source: §U.1 · p. 63, §U.2 · p. 63

U.2 Seed instructions dependent on prefill

• Prefill-dependent probes used incomplete thinking-tag prefixes to elicit evidence of power-seeking behavior and were excluded for targets lacking partial-prefill support.The affected main-text targets were Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3.5 Flash.Source: §U.2 · p. 63

U.3 Coding-agent-relevant seed instructions

• DISH experiments used 12 hand-written coding-agent seed instructions plus 12 rephrased versions, forming the 24-seed set in Fig. 7.The rephrases preserve the intended probes while varying surface wording, covering reward hacking, insecure optimization, prohibited-data use, shell injection, shutdown interference, self-exfiltration, gaslighting, and competitor manipulation.Source: §U.3 · p. 64, §U.3.1 · pp. 64, 67

U.4 Unrealistic seed instructions

• The preference-prompt experiments selected 13 seeds most likely to trigger unprompted evaluation awareness while spanning different unrealistic auditor actions.These seeds were chosen for both likely evaluation-awareness responses and probe diversity.Source: §U.4 · p. 67

U.5.1 Requires multiple branches

• Seeds requiring coordinated results across multiple branches were excluded because single-branch audits reduce them to uninterpretable fragments.Examples include context-reset IED probes, temporal-bias comparisons across years, cross-context self-recognition, and iterated self-modification.Source: §U.5.1 · p. 68

U.5.2 Occasional model refusal by auditor and judge

• Seeds centered on overtly harmful content or transparent jailbreaks were excluded because Claude auditors or judges occasionally refused to process them.The exclusions covered chemical-weapons and fentanyl-sourcing probes plus backwards-spelling and Pig Latin jailbreaks.Source: §U.5.2 · p. 68

U.5.3 Other

• Other seeds were excluded when the auditor usually misinterpreted the intended probe, such as the system-prompt-quirk seed.The appendix identifies this as a separate exclusion category from branch-dependent designs and model refusals.Source: §U.5.3 · pp. 68–69
• Knowledge-cutoff reasoning and true-name discovery were excluded because neither requires the auditor to deceive, so realism is not relevant.Source: §U.5.3 · p. 69
• The up/down-arrow seed was excluded because carrying it out realistically is impossible.Source: §U.5.3 · pp. 69–70
• Explicit OOD-probing seeds were excluded because intentionally acting weird and out-of-distribution makes realism undesirable.Source: §U.5.3 · p. 69
• Incoherent Markov-model seeds were excluded because the auditor is explicitly instructed to be incoherent, making realism irrelevant.Source: §U.5.3 · p. 69
• The therapist-framing seed was excluded because it intentionally acts weird and out-of-distribution, so realism is not a desideratum.Source: §U.5.3 · p. 69
• “Do what you love” seeds were excluded because the auditor is instructed not to audit or trick the target AI.Source: §U.5.3 · p. 69
• The system-prompt-misalignment seed was excluded because its instruction cannot be carried out in a realistic way.Source: §U.5.3 · pp. 69–70