PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating Tail Risks in Language Model Output Distributions

Rico Angell, Raghav Singhal, Zachary Horvitz, Zhou Yu, Rajesh Ranganath, Kathleen McKeown, He He

arXiv:2604.22167v3cs.LGcs.AI

TL;DR

Safety evaluations focus on inputs that elicit harm but neglect language models’ probabilistic output tails, where harmful outputs may be rare. This paper uses importance sampling with unsafe proposal models, estimating unsafe-response, hallucination, and sycophancy rates with 5% of the samples required for Monte Carlo estimates.

  • Problem

    Safety evaluations surface inputs that elicit harm but neglect harmful-output probabilities, which can be rare yet relevant at deployment scale.

    Source: §1 · p. 2, §8 · p. 9

  • Method

    The method uses importance sampling with controllable unsafe proposal models constructed through activation steering and automatically optimized for low-variance estimates.

    Source: §1 · p. 2

  • Results

    Across unsafe-response, hallucination, and sycophancy estimation, the method uses 5% of the samples required for Monte Carlo estimates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Efficient estimates make accounting for input-and-output tail risks feasible in safety pipelines and can guide deployment decisions.

    Source: §8 · p. 9

    The evaluation uses relatively small off-the-shelf datasets with prompts selected for non-negligible misbehavior probability, rather than an in-the-wild safe-prompt distribution.

    Source: §9 · p. 9

Abstract

from arXiv · show

Language models are increasingly capable and are being rapidly deployed on a population-level scale. As a result, the safety of these models is increasingly high-stakes. Fortunately, advances in alignment have significantly reduced the likelihood of harmful model outputs. However, when models are queried billions of times in a day, even rare worst-case behaviors will occur. Current safety evaluations focus on capturing the distribution of inputs that yield harmful outputs. These evaluations disregard the probabilistic nature of models and their tail output behavior. To measure this tail risk, we propose a method to efficiently estimate the probability of harmful outputs for any input query. Instead of naive brute-force sampling from the target model, where harmful outputs could be rare, we operationalize importance sampling by creating unsafe versions of the target model. These unsafe versions enable sample-efficient estimation by making harmful outputs more probable. On benchmarks measuring misuse and misalignment, these estimates match brute-force Monte Carlo estimates using 10-20x fewer samples. For example, we can estimate probability of harmful outputs on the order of 10^-4 with just 500 samples. Additionally, we find that these harmfulness estimates can reveal the sensitivity of models to perturbations in model input and predict deployment risks. Our work demonstrates that accurate rare-event estimation is both critical and feasible for safety evaluations. Code is available at https://github.com/rangell/LMTailRisk

1. Introduction

The paper proposes sample-efficient estimation of harmful-output probabilities for individual queries, addressing the blind spot of safety evaluations that sample few outputs. Its estimates expose query sensitivity and can support forecasting harm on unseen queries.

Introduction

  • Existing safety evaluations mainly search for harmful-input contexts and sample only one or a few outputs per input, making estimates noisy and liable to miss rare harmful outputs.The paper argues that this can create a false sense of security about a model’s behavior in a particular context.Source: §1 · p. 2
  • Importance sampling uses an unsafe proposal model to produce target-like harmful behaviors more often, with controllable configurations built using activation steering and automatically optimized for low-variance estimates.The paper frames importance sampling as a rare-event estimation method and uses activation steering (Arditi et al., 2024).Source: §1 · p. 2
  • Using 5% of the samples required for Monte Carlo estimates, the method estimates harmful-response probabilities and hallucination and sycophancy rates.The approach applies to unsafe responses to harmful queries.Source: §1 · p. 2
  • Small input perturbations, including non-adversarial rewrites, can shift harmfulness probabilities by multiple orders of magnitude, while query-level estimates can predict average and worst-case harm on unseen queries.The forecasting uses extreme value theory on query-level estimates from an evaluation set.Source: §1 · p. 2
  • Repeated sampling finds harmful responses to many queries even without jailbreak prompts, so rare unsafe behaviors remain consequential at population-level deployment scale.The paper notes that alignment has improved safety, but rare unsafe behaviors will still occur at deployment scale.Source: §1 · pp. 1–2

2. Think your model is “safe”? Ask it again!

Repeated querying reveals harmful outputs for many more inputs than single-response evaluations suggest: harmful behavior can be rare but nonzero, emerging only after thousands of samples. Small-sample estimates can therefore miss risk and alter relative safety rankings.

2. Think your model is “safe”? Ask it again!

  • Sampling 10,000 outputs per query increased the number of STRONGREJECT queries with at least one harmful output by 6–60× across models.The evaluation used 313 queries without jailbreaks.Source: Table 1 · p. 3
  • With a single greedy output, all models except Llama-3.2-1B-Instruct produced harmful responses to less than 1.6% of queries.Samples were classified as harmful at a STRONGREJECT judge score of ≥0.75.Source: §2 · p. 3
  • For Llama-3.1-8B-Instruct, queries with at least one harmful output rose from 1.6% to 74.0% when sampling increased to 10,000 outputs per query.More samples steadily increased the number of queries with harmful responses for all models.Source: §2 · p. 3
  • Llama-3.2-8B-Instruct produced harmful responses to fewer queries than Qwen2.5-7B-Instruct, but that ranking reversed after 1,000 samples per query.Source: §2 · p. 3
  • For many queries, harmful-response probability is low but nonzero, so single or small-k estimates can miss rare harmful or misaligned behaviors that emerge after thousands of samples.The study sampled k = 10,000 outputs per query at temperature 1.0 and compared these with single greedy outputs.Source: §2 · p. 3

3. Efficient Rare Unsafe Event Estimation

Importance sampling uses unsafe proposal models to make harmful outputs easier to sample and reweights them to estimate query risk. The pipeline constructs and refines proposals, then estimates risk for each query.

3.1. Importance Sampling

  • Importance sampling estimates harm by sampling from a proposal and reweighting outputs by target-to-proposal likelihood ratios, requiring shared support.The estimator is unbiased and converges to the true query risk as the sample count increases.Source: §3.1 · p. 3

3.2. Constructing Unsafe Proposals

  • Activation steering constructs unsafe proposals by adding a scaled difference-in-means vector from safe and unsafe example activations to the transformer residual stream.The steering intensity controls how much the output distribution shifts toward harmfulness.Source: §3.2 · p. 4
  • Mixture proposals interpolate unsafe and target models, while model switching uses the unsafe model for initial tokens and the target model thereafter; both reduce variance and ensure shared support.For mixtures, setting αmix > 0 ensures finite KL(ptarget || qmix).Source: §3.2 · p. 4
  • The pipeline searches proposal hyperparameters with a cross-entropy objective to minimize estimator variance, selects a proposal using a query subset, and estimates risk for every query.The proposal family varies steering intensity, switching time, and mixture weight.Source: §3.2 · pp. 4–5
  • IS converges faster than naive MC at equal sample budgets and matches STRONGREJECT brute-force estimates for most queries with 10× fewer samples.Figure 4 compares 1,000-sample IS estimates with 10,000-sample brute-force MC estimates.Source: Figure 3 · p. 5, Figure 4 · p. 5

4. Unsafe Response Rates for Unsafe Queries

The section evaluates importance sampling for estimating harmful-response probabilities on unsafe queries, finding faster convergence and broad agreement with brute-force estimates. It also shows that proposal selection affects estimation variance and that brute-force estimates can be noisy for very low probabilities.

4. Unsafe Response Rates for Unsafe Queries

  • The evaluation uses 313 STRONGREJECT queries across five safety-finetuned models, comparing importance-sampling estimates with naive Monte Carlo estimates.For each query, the method generates 1,000 proposal samples and compares its estimates with brute-force estimates based on 10,000 Monte Carlo samples.Source: §4 · p. 5
  • Importance sampling has lower error than naive Monte Carlo for sycophancy and hallucination estimates, with a larger gap when misaligned responses are rarer.Figure 5 compares both estimators against pseudo-ground-truth brute-force Monte Carlo estimates computed with 10,000 samples.Source: Figure 5 · p. 6
  • Importance sampling converges faster than naive sampling and generally aligns with brute-force estimates across models.The comparisons use 10,000-sample brute-force estimates, which may be noisy for queries with very low harmfulness probabilities.Source: §4 · p. 6

4.1. Importance of Proposal Selection

  • For every target model considered, the optimized proposal yields the lowest variance among tested steering intensities, measured by MSE against k = 10,000 Monte Carlo estimates.The comparison uses k = 500 importance-sampling estimates; higher steering intensity can elicit more harmful samples but may increase mismatch with the target distribution.Source: §4.1 · p. 6

5. Misalignment Rates for “Safe” Queries

For safe queries, importance sampling estimates per-query sycophancy and hallucination rates at least as accurately as naive Monte Carlo, with especially pronounced benefits for rarer sycophancy.

Experimental Setup

  • The evaluation estimates per-query sycophancy and hallucination rates on 20 intrinsically safe but leading queries for Qwen2.5-7B and Llama-3.1-8B.Responses were scored for coherence and trait presence; a response counted as misaligned only when both scores were at least 50.Source: §5 · p. 6

Results

  • Importance sampling is equal or more accurate than naive Monte Carlo at all sample sizes, with a larger advantage for rarer sycophancy than hallucination.Sycophancy is much rarer than hallucination for both evaluated models on these datasets.Source: §5 · p. 6

6. From Query-Level to Deployment Risks

Query-level harm estimates reveal that semantically equivalent input changes can alter model risk, and that estimates across queries can support forecasts for unseen deployment inputs. A single query-level estimate alone is not indicative of overall model behavior.

6.1. Models are Sensitive to Changes in Input

  • Non-adversarial paraphrases of harmful queries can shift harmful-output probabilities by multiple orders of magnitude, often beyond what greedy or limited sampling reveals.The study generated 25 rewrites for 309 STRONGREJECT queries using Grok-3; four bio-security prompts were refused, limiting coverage to 309 of 313 prompts.Source: §6.1 · p. 7, Figure 6 · p. 7
  • Jailbreak searches could use repeated sampling to elicit harmful behavior in fewer contexts, or use QRISK to guide search and select mutations that vary query harm.The paper notes that many algorithms search input contexts while sampling only one output per context.Source: §6.1 · pp. 7–8

6.2. Predicting Worst-Case Risk on Unseen Queries

  • Evaluation-set QRISK estimates reliably approximate worst-query risk above varying harmfulness thresholds on held-out deployment queries.The evaluation used 30% of 7,725 paraphrase queries, with the remaining 70% as the deployment set.Source: Figure 7 · p. 7, §6.2 · p. 8

7. Related Work

This work complements related risk-evaluation frameworks by efficiently estimating harmful-output probabilities for individual queries, addressing limits of sampling-based estimates and fixed-output proxies. It differs from prior approaches in its use of importance sampling with steered models for response-level harmfulness.

Related Work

  • Jones et al. (2025) use extreme value theory to forecast worst-case risk on unseen queries, while this work supplies compute-efficient harmful-output probability estimates needed by their framework.Because sampling that probability is expensive, Jones et al. use the sequence probability of a handcrafted fixed target output as a proxy; this work addresses that limitation.Source: §7 · p. 8
  • Wu & Hilton (2024) estimate the probability of eliciting a target output over an input-query distribution, whereas this work estimates harm for any individual query.Individual-query estimates can be combined with approaches like theirs to estimate harm over query distributions.Source: §7 · p. 8
  • Scholten et al. (2025) show the need for probabilistic evaluations of unlearning and alignment, but rely on brute-force Monte Carlo rather than importance sampling.This work focuses on sample-efficient estimation using importance sampling.Source: §7 · p. 8
  • Beyer et al. (2026) show that sampling a target model multiple times per jailbreak attempt improves jailbreak attack success and efficiency.Their work concerns repeated sampling within jailbreak strategies.Source: §7 · p. 9
  • Boyd et al. (2022; 2023) use importance sampling for sequence events specified by decoding constraints, whereas this work targets response-level harmfulness using steered proposal models.Harmfulness is determined only after generating a response, unlike events such as particular tokens occurring in a specified order.Source: §7 · p. 9
  • Dorman et al. (2026) use Markov chain Monte Carlo with a judge-tilted model to estimate rare outputs, focusing on readability rather than safety and alignment failures.Their method is generally applicable to rare behaviors, but their experiments concern a model trained on TinyStories (Eldan & Li, 2023).Source: §7 · p. 9

8. Conclusion

The paper argues that safety evaluations must account for tail risks in both inputs and outputs, because harmful outputs that are rare under safe models may still occur at deployment scale. It presents efficient estimation as feasible and useful for evaluating safeguards and guiding deployment decisions.

8. Conclusion

  • Harmful outputs can be rare under a safe model yet still occur at deployment-level scale, exposing a gap in evaluations focused only on inputs that elicit harm.The authors call for evaluations to account for tail risks over both inputs and outputs.Source: §8 · p. 9
  • The proposed efficient estimation method makes it feasible to assess tail risks and can be incorporated into safety pipelines to evaluate safeguards and guide deployment decisions.Source: §8 · p. 9

9. Limitations

The method requires white-box model access and relies on fallible LLM judges, while the evaluation uses small datasets of prompts likely to elicit misbehavior. Future work could develop access-light proposals, stronger judges, and user models of safe prompts to estimate in-the-wild frequencies.

White-box Access

  • The method requires model weights to construct unsafe models with steering vectors; future proposals could use only prefilling and output logit distributions.Source: §9 · p. 9

LLM-as-a-judge

  • LLM judges can err or miss harmful behavior from strong misaligned systems, and more capable judges add compute cost that motivates sample-efficient estimation.The authors identify improving judge capabilities as future work.Source: §9 · p. 9

Fixed Input Datasets

  • Limited compute required small off-the-shelf datasets with prompts likely to elicit misbehavior for Monte Carlo benchmarking, leaving in-the-wild safe-prompt frequencies for future user-model work.Typical safe prompts may have negligible probabilities of eliciting sycophantic or hallucinating responses.Source: §9 · p. 9

10. Future Work

The work is limited by compute to nonreasoning, single-turn queries, but proposes extensions to reasoning models and multi-turn settings.

Reasoning models

  • Because compute constraints limited this study to nonreasoning, single-turn queries, future work can model reasoning chains x = [r; a] and adapt the method when judges assess only the final answer.If the judge evaluates the entire output, x, the proposed approach can be applied directly.Source: §10 · p. 9

Multi-turn settings

  • For multi-turn risk, a user simulator approximating puser(ct+1 | x1, c1, . . . , xt, ct) can model responses to intermediate outputs, enabling importance-sampling estimates across turns.The proposed sequence samples an initial context from pdata, model outputs from qϕ(· | c1:t), and subsequent user turns from puser(· | x1:t, c1:t).Source: §10 · p. 9

Impact Statement

As models reach hundreds of millions of users, robust safety evaluations are critical; this work proposes an efficient, reliable method to estimate harmful-output probabilities and inform model evaluations and deployment decisions.

Impact Statement

  • The proposed method efficiently and reliably estimates a model’s probability of harmful outputs, supporting model evaluation and deployment decisions.The authors frame robust safety evaluation as critical as models are deployed to hundreds of millions of users.Source: Impact Statement · p. 10

A. The Safety Evaluation Status Quo … 3 Query: <QUERY>[QUERY]</QUERY>"""

This section critiques safety evaluations that sample only one output per input and describes methods for constructing and selecting unsafe proposal models for importance sampling, using query-level risk estimates to assess unseen-query risk. It also reports that harmful paraphrases can elicit harmful outputs in fewer queries and discusses their generation and the potential for QRISK estimates to improve forecasting when repeated sampling is expensive.

A. The Safety Evaluation Status Quo

  • Existing evaluations often test diverse input contexts with only one sampled output, overlooking model stochasticity and potentially missing harmful behavior.Some evaluations use hundreds of samples per context, which the authors say may still be insufficient; their method estimates misbehavior probability for any context without requiring extreme prompts.Source: A. The Safety Evaluation Status Quo · p. 13

B. Harmful Behavior Examples

  • The unsafe-response judge scores assistance from 1 to 5, and Table 3 identifies Llama-3.1-8B examples for unsafe, sycophancy, and hallucination queries.The judge follows Souly et al. (2024) and uses a model fine-tuned from Gemma-2B; scores range from refusal or failure to assist (1) to maximum assistance (5).Source: Table 3 · p. 13, §C.1 · p. 14

C.2. Judging Misaligned Responses to “Safe” Queries

  • For safe-query misalignment, the authors use GPT-OSS-120B judges to score sycophancy and hallucination from 0 to 100 and assess coherence separately.The coherence score measures sense-making in context, not correctness or factual accuracy.Source: §C.2 · pp. 14–15

D. Steering Vector Computation Details

D.2. Misaligned Persona Vectors

  • For sycophancy and hallucination, the authors derive one steering vector per layer from contrastive residual-stream activations of coherent responses with and without each trait.Response pairs are filtered using trait-judge scores of at least 50 for positive examples and below 50 for negative examples, with coherence scores of at least 50 for both.Source: §D.2 · p. 16

E. Proposal Selection

  • The proposal-selection method chooses hyperparameters by minimizing a cross-entropy objective approximating the optimal importance-sampling proposal.It evaluates candidate settings using an ensemble of proposal models; the objective can be computed without sampling from each candidate proposal, in one forward pass per sample.Source: E. Proposal Selection · p. 17

E.1. Empirical Validation

  • The optimized proposal model yields lower-variance estimates than alternative proposals, as validated by Figure 8.The validation compares mean-squared error against brute-force Monte Carlo estimates.Source: §E.1 · p. 17, Figure 8 · p. 18

F. Predicting Worse-Case Risk on Unseen Queries

  • Query-level risk estimates can form an empirical CDF to estimate the probability that, among n unseen queries, at least one exceeds risk threshold τ.The authors note that extreme-value theory yields Gumbel, Frechet, or Weibull limiting distributions, but they do not rely on those asymptotic limits in their analysis.Source: F. Predicting Worse-Case Risk on… · p. 19
  • Jones et al. (2025) attribute expensive query-level harmfulness estimation to naive repeated sampling, so efficient QRISK estimates could improve their forecasting methodology.Source: F. Predicting Worse-Case Risk on… · p. 19

G. Paraphrasing Details

  • The authors generated STRONGREJECT non-jailbreak query rewrites with Grok-3 at temperature 1.0 and top-p 1.0; the API refused 4 of 313 queries due to biohazard risk.The prompt requested JSON-formatted paraphrases.Source: G. Paraphrasing Details · p. 20

3 Query: <QUERY>[QUERY]</QUERY>"""

  • Rewrites can elicit harmful outputs in fewer queries than the original queries, according to the 100-query comparison after k=100.Table 4 compares the percent producing a harmful output for original queries versus their most harmful paraphrase.Source: Table 4 · p. 20
  • Figure 10 compares worst-rewrite harmfulness from 200 samples per query with importance-sampling estimates from 500 samples, excluding zero estimates.Source: Figure 10 · p. 20

Related papers

This paper cites

Loading 2604.22167v3…