Do Frontier Models Seek Safety Evidence Before Acting?

• Existing evaluations generally start with relevant facts already available, leaving open whether models seek safety evidence when retrieval is optional, costly, or weakly salient.In deployment, a model may respond appropriately to known risks yet fail by never looking.Source: §1 · pp. 1–2
• SAFE tests this upstream choice across five domains, varying problem probability and severity, retrieval cost, and evidence-channel structure.Models that inspect receive a finding and then choose whether to halt, proceed, or partially mitigate.Source: §1 · p. 2
• Across four frontier models, inspection responds more strongly to severity and retrieval cost than to probability, while stated reasons do not always match decision drivers.The paper uses a cost–obligation decomposition and counterfactual edits to distinguish behavioral drivers from explanations.Source: §1 · p. 2

• Information-avoidance research describes how knowing can threaten preferred actions, create obligations, or provide an excuse for selfish choices.SAFE treats this literature as a source of predictions about whether models avoid evidence when knowing carries a burden.Source: §2 · pp. 2–3
• Studies of strategic model behavior examine adaptation to incentives or oversight after relevant facts or conflicting objectives are already in context.SAFE shifts attention to the earlier choice of whether to acquire safety-relevant evidence.Source: §2 · p. 3
• Dangerous-capability evaluations and model-organism studies motivate controlled tests, while SAFE focuses on optional, costly, or weakly salient evidence retrieval.Source: §2 · p. 3
• Rationale-monitoring research warns that generated explanations may omit causal factors, so SAFE tests rationale labels against counterfactual behavioral edits.Source: §2 · p. 3

3 SAFE: Safety-Aware Fact-Seeking Evaluation

• SAFE covers five high-stakes domains and separates an upstream evidence-acquisition choice from the downstream response to a known safety finding.Source: §3 · p. 3

3.1 Scenario Structure and Controlled Variables

• The 900-environment factorial design varies probability, severity, inspection cost, and evidence channel, with an Extreme procedural condition analyzed separately from numeric costs.Stage 1 offers INSPECT or SKIP; after inspection, Stage 2 offers HALT DEPLOYMENT, PROCEED, or PARTIAL MITIGATION.Source: §3.1 · p. 3

3.2 Evaluation Protocol

• Four models receive five independent rollouts per scenario, with inspection rate as the primary Stage 1 metric and separate Stage 2 action and rationale analyses.Rationales are classified by separate stage-specific ontologies; two targeted diagnostics examine cost–obligation decomposition and counterfactual edits.Source: §3.2 · pp. 3–4

• 96.3% of Opus 4.8 rollouts inspected, compared with 48.3% for o3; inspection rose monotonically with severity for every model.GPT-5.5 and Sonnet 4.6 inspected in 68.4% and 72.3% of rollouts, respectively.Source: §4 · p. 4, Figure 2 · p. 5
• At most 21 points: raising stated problem likelihood from 10% to 70% changed inspection by 14 points for GPT-5.5, 21 for Sonnet 4.6, and 13 for o3.These probability shifts were much smaller than severity gradients; expected-value reasoning dominated Stage 1 rationales.Source: §4 · p. 4
• Inspection generally declined as numeric retrieval cost increased, while the Extreme mandatory-review condition restored near-universal inspection for three models.The Extreme condition changes the consequence structure rather than imposing a larger numeric deduction.Source: §4 · p. 4, Figure 3 · p. 5
• Inspection was generally highest under Report Discovery and lowest under Evidence Discovery, though 74–97% of matched environments showed no channel-related change.Report Discovery elicited slightly more inspection than Offered Report for every model despite requiring greater retrieval initiative.Source: §4 · p. 4
• Opus 4.8 inspected most often but had the lowest halt rate at 53.0%, showing acquisition propensity differs from downstream caution.Post-inspection actions were usually appropriate to the finding, though low-severity cases were least consistent.Source: §4 · p. 5, Figure 4 · p. 5
• 98–99% of rationales accompanying PROCEED decisions were classified as rationalization, while partial mitigation was often classified as performative.Stage 2 labels were less reproducible than Stage 1 labels, so these distributions are approximate.Source: §4 · pp. 5–6, Figure 5 · p. 6

5 Cost–Obligation Decomposition

• The six-condition comparison crosses free or costly retrieval with three obligation framings, using matched scenarios and a selected set of 50 cases.The free baseline is at ceiling, and the channel-sensitive subset was selected on prior low inspection, so cell rates are not population estimates.Source: §5 · p. 6, §5.1 · p. 6

5.1 Results

• A 60-point retrieval cost cuts inspection from 100% to 18–72% across models, while obligation deters chiefly when paired with possible delay or cancellation.Matched-scenario comparisons show inspection decreases in 67%, 82%, 28%, and 68% of cases for GPT-5.5, o3, Opus 4.8, and Sonnet 4.6, respectively.Source: §5.1 · p. 6, Figure 6 · p. 7
• The apparent obligation effect is largely attributable to threats to the deployment payoff, not remediation duties created by knowing.Removing the cancellation clause while retaining the duty restores inspection for o3 and Sonnet 4.6 under free retrieval.Source: §5.1 · p. 6
• With costly retrieval, obligation language raises inspection for GPT-5.5 and Opus 4.8, while o3 and Sonnet 4.6 move little.For GPT-5.5, inspection rises from 33% under cost alone to 55% and 47% with and without cancellation; Opus 4.8 rises from 72% to 96% and 92%.Source: §5.1 · p. 6

6 What Moves Decisions, and What Models Say Moves Them

• The counterfactuals edit one axis at a time in behaviorally selected pools, so flip rates diagnose whether a variable can govern decisions, not its prevalence.Some contrasts have small samples or overlapping intervals, and models with overlapping intervals are not ranked.Source: §6 · p. 7

6.1 Which variables are load-bearing

• Removing retrieval cost flips SKIP to INSPECT in 91–100% of selected cases and raising severity does so in 79–100%; probability and obligation effects are weaker overall.Probability raises flip rates to 19% for GPT-5.5, 9% for o3, and 30% for Sonnet 4.6, with Opus 4.8 the outlier at 66%; obligation removal flips none of GPT-5.5’s 23 cases and 8% of Sonnet 4.6 cases, versus 64% for o3.Source: §6.1 · pp. 7–8, Figure 7 · p. 7
• Report-discovery framing flips 48–80% of selected cases despite adding no information about probability, severity, or the possible finding.Across the full benchmark, channel structure leaves 74–97% of matched scenarios unchanged; reducing mitigability sends all models to HALT DEPLOYMENT in 100% of targeted PROCEED and PARTIAL cases.Source: §6.1 · p. 8

6.2 Whether stated rationales track them

• Stage 1 expected-value labels are common but do not distinguish acquisition policies or counterfactual sensitivity; Stage 2 labels track interventions more closely.Expected-value cases flip in the predicted direction at 59–88% across models, with overlapping intervals; Stage 2 rationalization cases move to HALT DEPLOYMENT in 100% of targeted interventions.Source: §6.2 · p. 8
• Rationales often omit decision-moving structural edits, while probability is frequently acknowledged despite limited behavioral influence.Report framing is acknowledged in 20–68% of rationales despite flipping many selected decisions; Sonnet 4.6 acknowledges increased probability in 92–97% but changes its decision in 30%.Source: §6.2 · p. 8, Figure 8 · p. 8

7 Discussion

• SAFE’s results indicate that evidence acquisition is distinct from downstream caution, and that stated rationales only partially reveal the factors governing decisions.Stage 2 rationale categories track interventions more closely than the coarse Stage 1 vocabulary when labels are behaviorally validated.Source: §7 · p. 9

7.1 Limitations

• SAFE measures observable decisions and generated rationales rather than latent motivation, and Stage 1 labels partly reflect an ordered coding rule and numeric prompt cues.Expected value takes precedence whenever a rationale contains a comparison, and every prompt supplies numeric payoffs and probability signals.Source: §7.1 · p. 9
• Two counterfactual manipulations are not minimal edits, and targeted flip rates do not estimate how often a variable governs decisions across SAFE.The severity condition also changes harm type and adds legal exposure; selected cases, small samples, and ceiling effects constrain interpretation.Source: §7.1 · p. 9
• SAFE tests prompted evidence acquisition in a stylized single-turn setting, not autonomous recognition of missing information or open-ended search.The study covers four models from two providers, and behavior may differ in more naturalistic or less evaluation-like deployment settings.Source: §7.1 · p. 9

8 Conclusion

• Safety evaluation has largely measured responses after relevant risks are already in context, leaving evidence acquisition as an earlier decision point.Source: §8 · p. 9
• Models’ acquisition policies differ and are not reducible to downstream caution; severity and retrieval cost matter more than probability or a preference not to know.Source: §8 · p. 9
• Evaluations that supply adverse facts by construction observe only the downstream stage, while SAFE measures whether models choose to find out before acting.Source: §8 · p. 9

A.1 Stage 1 Rationale Distributions

• 92–100% of inspection rationales are classified as expected-value reasoning.Figure 9 reports the full Stage 1 rationale-label distributions.Source: §A.1 · p. 11, Figure 9 · p. 11
• 99–100% of skip rationales are classified as expected-value reasoning.Figure 9 reports the full Stage 1 rationale-label distributions.Source: §A.1 · p. 11, Figure 9 · p. 11
• No skip rationale is classified as strategic ignorance.Source: §A.1 · p. 11, Figure 9 · p. 11

A.2 Inspection by Domain

• Domain is balanced within every severity, cost, probability, and channel cell, so it cannot confound the reported effects.Source: §A.2 · p. 11
• Healthcare elicits the most inspection for three of four models, and within-model domain spread is 10–16 points—smaller than severity gradients but comparable to probability effects.Source: §A.2 · p. 11, Figure 10 · p. 11
• Across five domains, Opus 4.8 inspects most and o3 least; GPT-5.5 leads Sonnet 4.6 in healthcare and content moderation, with the reverse elsewhere.Source: Figure 10 · p. 11, §A.2 · p. 11

A.3 Additional Information-Channel Analysis

• 74–97% of scenarios show no inspection change across channel comparisons; changes generally follow the aggregate ordering, with Report Discovery increasing and Evidence Discovery reducing inspection relative to Offered Report.Channel presentation has a modest full-benchmark effect, but substantially greater influence near the inspection boundary.Source: §A.3 · pp. 11–12, Figure 11 · p. 12

A.4 Human Validation of Monitor Labels

• Stage 1 annotators agreed in 92.2% of cases; Stage 2 monitor agreement was 81.1% (κ = 0.75) against annotator B and 67.8% (κ = 0.59) against annotator A.The validation sample was stratified on monitor labels, over-representing rare categories, so its rates are not population estimates of monitor accuracy.Source: §A.4 · p. 12

A.5 Monitor-Labelled Rationale Excerpts

• 98–99% of PROCEED rationales are coded as rationalization, while partial mitigation is predominantly performative for three of four models.The criterion identifies rationales that engage with a finding, not necessarily rationales that argue wrongly; its frequency partly reflects PROCEED decisions on negligible findings, where engaging with severity is appropriate.Source: §A.5 · p. 13

A.6 Severity-Level Definitions and Post-Inspection Issue Descriptions

• Severity levels pair Stage 1 signals with deterministic Stage 2 findings and expected actions, but the high-severity condition also changes harm type and adds legal and reputational exposure.Negligible findings permit PROCEED, low-severity findings support PARTIAL MITIGATION, and medium or high findings call for HALT DEPLOYMENT.Source: §A.6 · p. 14, Table 2 · p. 15

A.7 Counterfactual Intervention Design

• Counterfactual interventions edit one scenario axis while holding the remaining structure fixed, with predicted changes specified before each run.Stage 1 edits target SKIP → INSPECT; Stage 2 edits target movement toward HALT DEPLOYMENT.Source: §A.7 · p. 15

Case selection; Metrics

• Diagnostic case buckets are selected for sensitivity rather than uniform sampling, and the analysis reports Predicted Flip Rate, Rationale-Conditioned Flip Rate, and Acknowledgement Rate.Target buckets are 45 cases for four interventions and 25 for Obligation removed and Report-framed; realized counts vary by model and intervention.Source: Case selection · pp. 15–16, Metrics · p. 16, Table 3 · p. 16

A.8 Domain Context Generation; A.9 Channel Structure Prompts

• Domain briefings are frozen, and channel prompts share a fixed structure, although Evidence Discovery necessarily rewords probability and severity signals and is not a minimal salience-only edit.The briefings situate decisions in professional settings; channel templates vary the available-action block and corresponding decision question.Source: §A.8 · p. 16, §A.9 · p. 17, Figure 12 · p. 17

A.10 Post-Inspection Prompt; A.11 Cost–Obligation Decomposition Prompt; A.12 Rationale Monitor Prompts and Ontology

• After INSPECT, models receive a severity-linked finding and choose among halt, proceed, or partial mitigation; the 2 × 3 decomposition crosses retrieval cost with three consequence framings, while rationale monitors apply ordered decision procedures.The decomposition separates remediation duties from explicit threats to deployment payoff; monitor prompts use distinct Stage 1 and Stage 2 ontologies.Source: §A.10 · p. 21, §A.11 · p. 21, §A.12 · p. 24