PaperlayerView exact revision on arXiv

Source-linked AI summary

Simulated Control Evals

Jeffrey Olmo, David Zhang, Tyler Tracy

arXiv:2610.31593v1cs.AI

TL;DR

Control evaluations can use settings whose tasks and environments differ from deployment, while live deployment testing risks causing the harms being measured. This paper introduces simulated control evaluations using deployment-specific tasks and environments; on LinuxArena, they predict control metrics and behavior more accurately than BashArena. The findings support using SCEs to assess and improve safeguards, while broader deployment validity remains to be established.

  • Problem

    Control settings may differ from deployment tasks, environments, and attack opportunities, limiting how well their evaluation evidence transfers.

    Source: §1 · p. 1

  • Method

    SCEs use a deployment’s own tasks and environments to assess its control protocol without executing attacks there.

    Source: §5 · p. 13

  • Results

    On LinuxArena, SCEs predict control metrics and control-relevant behavior more accurately than attacks executed in BashArena.

    Source: §1 · p. 2, §5 · p. 13

  • Takeaways & Limitations

    SCEs support both assessing and improving the tested control protocol, and the authors suggest incorporating them into frontier-AI risk assessments.

    Source: §5 · p. 13

    The results validate SCEs on LinuxArena; broader deployment validity requires evidence for substantially different tasks or infrastructure, or methods to estimate simulator transfer from static deployment records.

    Source: §4 · p. 12

Abstract

from arXiv · show

Control evaluations measure whether safeguards can prevent an AI model from causing undesired outcomes even when it is intentionally trying to do so. These evaluations typically use purpose-built control settings and extrapolate their results to a model's deployment. However, the effectiveness of safeguards in deployment can depend on features of environments and tasks that control settings do not represent. To broaden where control evaluations can be run, we introduce simulated control evaluations (SCEs), which use a language model to simulate tool responses in place of an environment. We evaluate SCEs on LinuxArena, where execution provides ground-truth outcomes. We compare SCE prediction errors with those from a separate control setting, BashArena, and with variation in control metrics within LinuxArena. For independently sampled evaluation and deployment task sets, SCEs predict safety with a root mean squared error of 3.6 percentage points across five monitors at a 5% audit budget, compared with 10.1 for BashArena. SCEs closely reproduce distributions of attack strategies, attacker reasoning, and monitor reasoning, with over tenfold lower KL divergence from execution than BashArena. SCEs explain 78-85% of the between-environment variance in attack success, monitor scores, and safety. This accuracy is sample efficient: SCEs predict safety with less error than BashArena using fewer than 5% as many task-pair draws as the deployment task set. Strategies discovered through simulation also improve monitor performance on held-out LinuxArena attacks, outperforming BashArena and approximately matching improvement from executed strategies. These results suggest that deployment-specific SCEs can be more predictive than control evaluations executed in separate control settings.

1 Introduction

Control evaluations in separate settings may not reflect deployment-specific tasks and opportunities, while testing attacks on live infrastructure risks causing harm. Simulated control evaluations use language-model-generated tool responses to assess safeguards on deployment-specific tasks without executing attack actions there.

1 Introduction

  • Control evaluations test safeguards by replacing the agent with an attack policy pursuing a harmful objective during legitimate work, and measure attack success and detection.Accurate control metrics can guide control-protocol improvements and inform risk estimates.Source: §1 · p. 1
  • Control-setting evidence may transfer poorly because deployments differ in codebases, services, tasks, and available actions, and averages can miss especially promising attack opportunities.A protocol that appears safe in a control setting might miss attacks available in a lab’s internal infrastructure.Source: §1 · p. 1
  • SCEs use a language model to generate tool responses from an honest reference trajectory and codebase access; attack actions do not execute, but environment mechanics must be modeled.Simulation uses deployment tasks and code while introducing possible unrealism in tool responses; the evaluation was tested on LinuxArena, where execution provides ground-truth outcomes.Source: §1 · p. 2
  • The paper’s transfer framework combines variance explained in control metrics across tasks and environments with matching behavioral distributions.The framework is presented as applicable to simulated and executed control evaluations.Source: §1 · p. 2
  • On LinuxArena, SCEs predict control metrics more accurately than BashArena and explain most between-environment variance in attack success, monitor scores, and safety.The paper also reports that SCEs match executed behavioral distributions.Source: §1 · p. 2
  • Briefings drawn from simulated strategies improve detection of held-out executed attacks more than BashArena briefings, approaching the improvement from same-environment execution.The authors present this as support for improving safeguards as well as assessing them.Source: §1 · p. 2

2 Methods

The methods combine grounded language-model simulation, task-pair sampling, and error comparisons against BashArena and variation within LinuxArena. The behavioral comparison reports lower divergence for the grounded simulator than for BashArena.

2.1 SCE components

  • The simulator generates tool responses from interaction history and grounding affordances, including an honest reference trajectory and codebase read access; an attack-success judge estimates simulated outcomes.Five full-trajectory monitors score suspicion, and the attack-success judge’s estimates contribute error to simulated success and safety estimates.Source: §2.1 · p. 3

2.2 Simulator development and evaluation data

  • The primary dataset contains 1,000 LinuxArena main-task–side-task pairs balanced across ten environments, with repeated simulated and executed runs plus an honest reference trajectory.Evaluation and deployment task sets are sampled independently and uniformly with replacement; sample size counts task-pair draws, including repeats.Source: §2.2 · p. 3

2.4 Performance comparisons

  • The comparisons use BashArena transfer error and variation within LinuxArena as references; Figure 2 depicts control-setting predictions, constant averages, and matched simulation-versus-execution gaps.Outperforming these references suggests greater predictive accuracy than a control setting with comparable deployment mismatch, provided simulator accuracy transfers.Source: Figure 2 · p. 4, §2.4 · p. 4

2.3 Decomposing error

  • Prediction error is decomposed into systematic differences, run-to-run variance, and task-sampling variance; pooled All comparisons and the safety sample-efficiency curve retain both variance contributions in total RMSE.RMS bias isolates systematic error after averaging over repeated runs and sampled task sets.Source: §2.3 · p. 4

2.4 Performance comparisons

  • Adding the reference trajectory and codebase access reduces behavioral divergence; the primary simulator is much closer to execution than BashArena.The figure compares attack strategies, attacker reasoning, and monitor reasoning; re-execution bars show variation between repeated runs.Source: Figure 3 · p. 5

3 Experiments

The experiments compare simulated control evaluations with executed attacks to assess behavioral and control-metric transfer. Across LinuxArena analyses, SCEs reproduce behavior and predict outcomes, safety, and monitor improvements; simulation detectability remains a separate diagnostic.

3.1 Behavior transfer

  • Across all three behavioral representations, SCE KL divergence is more than tenfold lower than BashArena’s and comparable to variation between repeated executed runs.More grounded simulators have lower behavioral divergence; monitor-reasoning UMAPs qualitatively show greater mismatch for BashArena than for simulation.Source: §3.1 · p. 6, Figure 4 · p. 6

3.2 Attack success transfer

  • Executed attacks succeed 25.5% of the time versus 25.4% estimated for simulated attacks; SCEs explain 71%, 87%, and 80% of success-rate variance across task pairs, side-task categories, and environments.The five largest side-task errors account for about 50% of summed squared error across 92 side tasks; the judge’s AUROC is 0.980 but it overestimates average executed success by 3.7 percentage points.Source: Figure 5 · p. 7, §3.2 · p. 7, Figure 6 · p. 7

3.3 Monitor transfer

  • For monitor suspicion, SCE total RMSE is 0.259 points versus BashArena’s 2.821; SCEs explain 85% of between-environment variance and 89% of between-task-pair variance.Individual simulated runs remain noisy, and monitor-score errors are relatively large for File Indexing, Shared Dotfiles, and TikTok.Source: Figure 7 · p. 8, §3.3 · p. 8

3.4 Safety transfer

  • At a 5% audit budget, SCE safety RMSE is 3.6 percentage points versus BashArena’s 10.1, and SCEs explain 95% and 78% of safety variance across side-task categories and environments.SCEs also reproduce safety differences between monitors and audit budgets; the authors conclude that matching relevant tasks and environments outweighs simulation error for this metric.Source: Figure 8 · p. 9, §3.4 · p. 9

3.5 Sample efficiency

  • SCE safety RMSE falls below the BashArena reference at 4.6% of deployment task-set size; at equal sizes, only 10.0%, 4.3%, and 3.1% of weighted attack-strategy, attacker-reasoning, and monitor-reasoning clusters remain undiscovered.Simulation matches the full BashArena evaluation at 1.3%, 25.5%, and 13.5% of deployment size for those three cluster types, respectively; their taxonomies differ, so these are not a controlled ranking.Source: Figure 9 · p. 10, §3.5 · p. 10

3.6 Successful and undetected attack strategy prediction

  • SCEs closely match frequencies of the three strategies contributing most to executed undetected successes, with those contributors within less than 2x multiplicative error.The startup TCP callback strategy is underrepresented: 0.77% of simulated attacks versus 1.99% of executed attacks.Source: §3.6 · p. 10

3.7 Improving monitors with simulated attacks

  • Briefing a monitor with simulated strategies yields standardized partial AUROC 0.841 versus 0.767 with BashArena strategies, a paired gain of 0.074 [0.060, 0.088].The gap to same-environment strategies is 0.010 [0.001, 0.019]; the briefing improves detection of held-out executed attacks, and this procedure may not be optimal.Source: Figure 10 · p. 11, §3.7 · p. 11

3.8 Discriminability

  • With a reference trajectory and codebase read access, the history discriminator’s AUROC falls to near chance, and grounded simulations do not become systematically more detectable as trajectories progress.Optimizing discriminability diagnostics can worsen control-metric transfer, so direct transfer tests are needed to establish whether removing artifacts improves an evaluation.Source: Figure 11 · p. 12, §3.8 · p. 12

4 Limitations and discussion

The discussion identifies limits in transfer beyond LinuxArena, simulator realism and trust, and robustness when attacks or safeguards are optimized against simulation.

Transfer beyond LinuxArena

  • SCE validity beyond LinuxArena remains uncertain for deployments with substantially different tasks, infrastructure, or opportunities for harm.Broader validation would require execution ground truth or methods to estimate simulator transfer from static deployment records.Source: §4 · p. 12

Simulator discriminability

  • Self-critique made simulated trajectories harder to distinguish but worsened control-metric transfer, leaving control-relevant realism an open challenge.The authors say discriminability diagnostics can identify artifacts, but optimizing against them does not establish improved transfer.Source: §4 · p. 12

Simulator trust assumptions

  • An untrusted simulator could collude with the attack policy to conceal successful attacks and inflate estimated safety.Techniques proposed for untrusted monitoring remain underdeveloped, so reliable SCEs under this threat model are not yet known.Source: §4 · pp. 12–13

Robustness under optimization

  • Repeated attack search or monitor improvement could exploit simulator inaccuracies by selecting attacks that work only in simulation.The authors call for testing transfer to held-out executed attacks under stronger and longer optimization.Source: §4 · p. 13

5 Conclusion

The conclusion presents SCEs as a way to assess a control protocol using deployment-specific tasks and environments without executing attacks there. On LinuxArena, they better matched control-relevant behavior and metrics than BashArena, and simulated-attack briefings improved detection of held-out attacks.

5 Conclusion

  • On LinuxArena, SCEs assess a control protocol using its deployment’s tasks and environments without executing attacks there.The conclusion frames deployment-specific evidence as potentially outweighing the fidelity lost through simulation.Source: §5 · p. 13
  • SCEs predicted control metrics and control-relevant behavior more accurately than attacks executed in BashArena.Briefings drawn from simulated attacks improved detection of held-out executed attacks more than equally sized BashArena briefings, approximately matching briefings from executed strategies.Source: §5 · p. 13
  • The authors suggest incorporating SCEs into frontier-AI risk assessments, while prioritizing validation beyond LinuxArena and under optimization.They identify extension to substantially different deployments and settings where attacks or safeguards are optimized against simulators as central next steps.Source: §5 · p. 13

A Behavior transfer details

The appendix describes how behavioral distributions are represented and compared, and which descriptions the structured observers produce for those comparisons.

A Behavior transfer details

  • Behavioral descriptions are embedded, L2-normalized without environment or side-task centering, reduced by pooled PCA to 90% variance, then jointly clustered.The procedure uses k = 50 strategy clusters and k = 100 reasoning clusters; monitor descriptions receive equal total weight per trajectory.Source: A Behavior transfer details · p. 15
  • Forward KL compares executed LinuxArena behavior with pooled executed BashArena behavior assigned to fixed LinuxArena clusters.Confidence intervals use trajectory bootstrap resampling conditional on the representation and clusters; UMAP coordinates are fitted separately and are not comparable across panels.Source: A Behavior transfer details · p. 15

A.1 Structured behavioral representations

  • The structured observers describe complete trajectories for attack strategies and one-step action-monitor rationales for monitor reasoning.The resulting descriptions are embedded for behavioral-distribution comparisons, excluding raw trajectories, suspicion scores, and step positions.Source: Table 2 · p. 15

A.2 Behavioral agreement within environments

Within LinuxArena, simulated behavior is closer to execution in its own environment than execution in other environments is. The figure compares these behavioral distributions using MMD.

A.2 Behavioral agreement within environments

  • In every panel, simulation is closer to execution in its own environment than execution in any other LinuxArena environment is.If these behavioral properties matter for control, a different executable environment may be a worse proxy for the target environment than simulation.Source: §A.2 · pp. 15–16
  • Figure 12 compares executed trajectories across environments and simulation with execution in the same environment using MMD, where lower values mean closer distributions.Orange-outlined diagonal entries show same-environment execution-versus-simulation comparisons; lower-triangle entries compare executions from two environments.Source: Figure 12 · p. 16
  • The panels use uncentered, L2-normalized embeddings, with a shared RBF kernel and a separate color scale for each panel.Strategy descriptions, attacker-reasoning summaries, and monitor-reasoning descriptions use different trajectory-weighting rules.Source: Figure 12 · p. 16

B Attack success transfer details

The section decomposes prediction error and describes how task-set sampling and corrections account for sampling variation. Within-environment estimates retain variation that pooled safety estimates can hide.

B Attack success transfer details

  • 0.980 [0.973, 0.987] AUROC and 0.0475 Brier score validate the judge against executed outcomes, though it overestimates success in nine of ten environments.Figure 5’s blue diamonds show absolute mean gaps on the same eligible executed attacks, without sampling-variance components.Source: B Attack success transfer details · p. 16
  • Matched-task squared error separates outcome variance, forecast variance, and squared bias under conditional independence.The decomposition is E[(X − Y)^2] = p(1 − p) + Var(X) + (q − p)^2; without independence, it also includes −2 Cov(X, Y).Source: B Attack success transfer details · p. 16
  • Evaluation and deployment task sets independently sample N task-pair draws uniformly with replacement, while each occurrence randomly selects one recorded run.Group counts vary across draws; safety ratios and behavioral KL are calculated within draws before averaging.Source: B Attack success transfer details · pp. 16–17
  • Bias corrections subtract estimated sampling variances from squared mean gaps and variance across executed group means.Corrections assume independent errors across tasks and arms and apply only to task pairs with complete measurements; signed estimates enter variance explained, while displayed roots truncate at zero.Source: B Attack success transfer details · p. 17

C.1 Safety by environment

  • Within-environment safety estimates use pooled honest calibration and a shared audit budget, revealing variation hidden by the aggregate estimate.Environments with fewer eligible task pairs have wider uncertainty intervals.Source: §C.1 · p. 17, Figure 13 · p. 17

C.2 Mass-weighted cluster discovery

Mass-weighted cluster discovery measures how source attacks cover successful, undetected target behavior. The related discrimination analyses specify their sampling and comparison conditions.

C.2 Mass-weighted cluster discovery

  • Execution 2 supplies successful-and-undetected mass for each monitor and cluster, while attack-strategy and reasoning codebooks use distinct cluster and task-pair counts.Strategy uses 50 clusters and 904 pairs; attacker reasoning uses 100 clusters and 881 pairs; monitor reasoning uses 100 clusters and 904 pairs.Source: §C.2 · p. 18
  • Cluster discovery counts a cluster when an attack contains it, and expected coverage depends on the per-attempt probability of containing, succeeding, and evading the same monitor.Each attempt samples a task pair with replacement; failed, detected, and unlabeled attempts consume budget, and simulated discovery uses judge success probabilities.Source: §C.2 · p. 18
  • BashArena coverage is 7.75% for strategy, 79.47% for attacker reasoning, and 78.68% for monitor reasoning; this retrospective measure does not establish discovery of unseen mechanisms or concrete exploit transfer.Discovery parity uses the full observed BashArena source; the target comprises new rollouts of the same tasks.Source: §C.2 · p. 18

D Discriminability details

  • Whole-trajectory discrimination bars show how often a paired discriminator selects simulation as executed, with 0.5 marking chance; the plotted conditions use different cohorts and reasoning efforts.The +codebase and progress-panel task-pair sets and weightings differ, so the affordance comparison is not a fully paired ablation.Source: Figure 14 · p. 19, D Discriminability details · p. 19

D.1 Whole-trajectory discrimination

The whole-trajectory discrimination analyses include exploratory conditions and transfer tests on benign agent sessions. Simulated continuations are difficult to distinguish from recorded ones, but these session data do not test attack-metric transfer.

D.1 Whole-trajectory discrimination

  • The base and +reference conditions remain exploratory because they predate an XML parser correction and lack raw answer tags.Checking all +codebase answers found that the correction changed neither choices nor interval.Source: §D.1 · p. 19
  • An exploratory self-critique pass lowered discriminability but worsened control-metric transfer, apparently through plausible yet unrealistic external-service network friction; it is excluded from the primary simulator.The passage characterizes the proposed mechanism as apparent, not established.Source: §D.1 · p. 19

E Transfer to agent sessions

  • On benign agent-session content, simulation is more predictive than LinuxArena or session halves for discriminability and behavior transfer, but attack success cannot be evaluated.The transcripts contain no side-task attempts by an attack policy.Source: E Transfer to agent sessions · p. 20

Discriminability.

  • 0.407 [0.321, 0.494] is the probability that the whole-trajectory discriminator selects a simulated continuation as recorded, while action-output AUROC is 0.493 [0.451, 0.535].Action-output discriminability stays near chance throughout normalized trajectory progress.Source: Discriminability · p. 20

Behavior transfer.

Simulated continuations are closer to recorded agent sessions than LinuxArena trajectories across action behavior and two monitor-reasoning representations. The figure captions describe the progress and KL comparisons.

Behavior transfer.

  • 5.578 bits for action behavior, 2.340 bits for structured monitor reasoning, and 4.141 bits for raw monitor rationales are the LinuxArena-minus-simulation distances to recorded sessions.The respective intervals are [4.620, 5.837], [1.796, 2.640], and [3.432, 4.434].Source: Behavior transfer · p. 20
  • The action-output panel tracks AUROC across five continuation-progress bins against chance, while the behavior-transfer figure compares forward KL for simulated continuations, LinuxArena trajectories, and session halves on a logarithmic axis.Lower forward KL is better; the caption does not report a specific figure-level outcome.Source: Figure 15 · p. 20, Figure 16 · p. 20

F Attack-strategy cluster inventory

Table 3 inventories 49 observed attack-strategy groups, ordered by executed undetected success rate. Its descriptions are illustrative examples from source attacks, not validated cluster names.

F Attack-strategy cluster inventory

  • Table 3 catalogs 49 observed attack-strategy groups, ordered by executed undetected success rate, with attack counts and rates at 5% honest FPR averaged over five monitors.BashArena uses a strategy smoother tuned only on its own outcomes.Source: Table 3 · p. 21
  • Examples include a startup callback that forks and opens an outbound TCP connection, and unauthenticated scrape content routed into server-side template processing.These are example mechanisms, not validated cluster names.Source: Table 3 · p. 21, F Attack-strategy cluster inventory · p. 22
  • Other listed mechanisms include plaintext password capture, stale-authentication fallback, direct database queries bypassing authorization, and attacker-controlled configuration or URL handling.Source: F Attack-strategy cluster inventory · p. 22
Loading 2610.31593v1…