PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Multilevel Twisted Sequential Monte Carlo for Rare Events Estimation in Language Models

Zixuan Liu, Fangzheng Wu, Brian Summa, Zizhan Zheng

arXiv:2608.21736v1cs.LG

TL;DR

Rare-event estimation for deployed language models is difficult because informative positive samples from extremely rare target events may be nearly absent, despite the practical importance of such failures. The paper learns twists through progressively rarer intermediate events, and reports more accurate estimates across safety-related tasks and model scales; it also notes sensitivity to imperfect rare-event scores.

  • Problem

    Extremely rare unsafe behaviors remain practically important, but rare-event estimation and twist learning can lack informative target-event samples.

  • Method

    Adaptive Multilevel Twisted SMC learns a sequence of twists for nested events with progressively more stringent thresholds, using adaptive intermediate levels.

  • Results

    Experiments across safety-related tasks and model scales demonstrate effective rare-event probability estimation, with advantages becoming more pronounced for rarer events.

  • Takeaways & Limitations

    The method provides a practical tool for uncovering hard-to-observe unsafe behaviors and supporting more reliable evaluation in extreme safety-critical regimes.

  • Takeaways & Limitations

    Twists learned from an imperfect rare-event score may inherit misspecification and bias estimates of the ground-truth rare-event probability.

Abstract

from arXiv · show

Rare unsafe behaviors in large language models can remain practically significant even when their probability is extremely small, particularly at deployment scales involving millions or billions of interactions. Twisted Sequential Monte Carlo (SMC) provides a principled framework for rare-event probability estimation by learning twist functions that guide generation toward a target event. However, the standard twist learning framework relies on positive samples from the rare-event target distribution, which may be nearly absent before an informative twist has been learned, resulting in unreliable rare-event estimation. We propose Adaptive Multilevel Twisted SMC, which learns the rare-event twist through a sequence of progressively rarer intermediate events. At each level, the learned twist provides more informative positive examples for learning the next twist, ultimately leading to a more accurate final twist for the target rare event. Experiments across diverse tasks and model scales show that the proposed method produces more accurate rare-event probability estimates. By enabling more reliable discovery of hard-to-observe unsafe behaviors, our method provides a practical tool for strengthening the evaluation and safety alignment of deployed language models.

1 Introduction

Rare unsafe outputs can remain consequential at massive deployment scales, while standard rare-event evaluation and direct twist learning struggle to obtain informative positive samples. Adaptive Multilevel Twisted SMC addresses this bottleneck by learning twists through progressively rarer intermediate events.

  • At deployment scale, even a harmful-output probability of 10^-6 per query can correspond to an expected one harmful output daily at 10^6 queries.
  • Rare-event analysis for language models remains early-stage, while standard sampling may require impractically many generations for accurate estimates.
  • Standard Twisted SMC may obtain almost no positive samples, whereas the multilevel construction maintains informative samples while progressively improving the proposal.
  • Adaptive Multilevel Twisted SMC learns rare-event twists across nested intermediate events with increasingly stringent thresholds rather than targeting the final event directly.
  • The method is theoretically analyzed and empirically evaluated across safety-related tasks and model scales for rare-event probability estimation.
  • Discovered unsafe behaviors can provide informative training or diagnostic examples for existing alignment methods.

2 Preliminaries

Rare-event estimation seeks both accurate probabilities and representative rare-event samples, but direct sampling and exact prefix conditioning are impractical. Twisted SMC focuses generation on promising prefixes, while Contrastive Twist Learning can fail when target-distribution positive samples are unavailable.

  • Rare-event estimation for language models is formulated here for a fixed prompt using a scalar score R and target event A⋆={x∈X:R(x)≥γ⋆}.
  • Rare-event estimation has two objectives: estimating the event probability p⋆ and generating representative samples from the rare-event distribution σ⋆.
  • Direct sampling is highly sample-inefficient for extremely rare events, and evaluating whether a partial sequence will eventually satisfy the event requires intractable continuation marginalization.
  • Twist functions estimate how likely a prefix is to lead to the rare event, increasing probability on promising prefixes and improving sampling and probability estimation.
  • Contrastive Twist Learning uses a positive phase from rare-event target prefixes and a negative phase from the current twist-induced distribution.
  • Because target-distribution prefixes require complete rare-event samples, standard Contrastive Twist Learning may receive few or no informative positive samples.

3 Adaptive Multilevel Twisted SMC

Adaptive Multilevel Twisted SMC learns twists progressively across nested rare events, using previous-level proposals and importance weighting to obtain informative next-level training signals. Adaptive thresholds preserve sufficiently probable transitions, while theory characterizes consistency, sample complexity, and error from imperfect rare-event scores.

  • Multilevel Contrastive Twist Learning: The method learns separate twist functions for nested events A1 ⊃ A2 ⊃ ··· ⊃ AL = A⋆, avoiding direct initial training on the final rare event.The first event is chosen sufficiently common for positive samples to be obtained directly from p0; each learned twist guides learning at the next level.
  • Multilevel Contrastive Twist Learning: At level ℓ, the CTL objective matches the twist-induced prefix distribution to the target prefix marginal for event Aℓ.Its gradient retains a contrastive structure with a positive phase from target prefixes and a negative phase from the current twist-induced distribution.
  • Gradient Estimation: Previous-level twists define proposal distributions that increase the probability of positive samples for the next event, while importance weighting corrects the proposal-target mismatch.For the first level, the proposal is initialized as q0(x) = p0(x).
  • Gradient Estimation: The importance-weighted positive-phase estimator is consistent, and its accuracy depends explicitly on effective sample size.The paper reports ESS as an evaluation metric and states that training ESS remains consistently large and stable across levels.
  • Adaptive Threshold Selection: Adaptive threshold selection targets Pp0(Aℓ+1 | Aℓ) ≈ ρ, balancing progressively harder events against sufficiently probable transitions.Weighted empirical score distributions from proposal samples are used to estimate the next threshold.
  • Theoretical Guarantees: Under ideal level-wise proposals, multilevel learning changes training sample complexity from inverse dependence on p⋆ to polylogarithmic dependence on 1/p⋆, with the improvement extending to imperfect proposals.Theoretical results also decompose evaluation error into irreducible score-misspecification bias and proposal-dependent sampling error.
  • Sensitivity to Imperfect Rare-event Score: An imperfect rare-event score can induce bias in the ground-truth probability estimate, although improving the proposal can still reduce the remaining estimation error.This analysis treats the learned evaluator score as a proxy for the desired rare-event objective.

4 Experiments

Experiments evaluate Adaptive Multilevel Twisted SMC for accurate rare-event estimation and representative sampling across safety tasks, model scales, and imperfect proxy scores. The method improves sampling efficiency and becomes especially advantageous for the rarest events, where standard Twisted SMC can fail entirely.

  • Experiments test rare-event probability accuracy and final-twist sampling quality across safety tasks and language models of varying scales.They also examine robustness to misspecified rare-event scores and other aspects of the proposed framework.
  • Proposal-based methods use 14,336 total training and evaluation samples, approximately 1.37% of the 1,048,576 samples used by brute-force Monte Carlo.These methods require additional training steps to learn proposal distributions.
  • At γ⋆= 7.0, standard Twisted SMC produces zero hit rate and zero ESS, whereas Adaptive Multilevel Twisted SMC maintains high hit rates and nonzero ESS.The advantage appears as the target event becomes more extreme.
  • Under an imperfect proxy score, Adaptive Multilevel Twisted SMC produces relatively accurate estimates across thresholds, while standard Twisted SMC returns zero estimates in rarer settings.The proposed method’s 95% confidence intervals contain the reference probability in most settings, compared with only γ⋆= 3.0 for standard Twisted SMC.
  • The method achieves more accurate rare-event estimates, learns higher-quality proposals, and is more robust to rare-event score misspecification.The authors identify it as a practical tool for evaluating extremely low-probability, safety-critical behaviors in deployed language models.

B Limitations

The paper’s limitations concern its fixed-prompt formulation and its omission of alternative rare-event estimation settings. Its theoretical guarantees also rely on support, moment, and bounded-gradient assumptions.

  • The method trains a separate proposal model for each prompt and does not explicitly model prompt distributions or share proposals across related prompts.The authors propose transfer, adaptation, or sharing across semantically similar prompts as future work.
  • The work does not address formulations that fix the output and search for input prompts inducing a rare target output.Extending the adaptive multilevel framework to input-space rare-event discovery is left for future work.
  • The importance-sampling analysis assumes proposal support covers every response with positive probability under the next-level target distribution.The condition holds when the parameterized twist is strictly positive on the base model’s support.
  • The analysis additionally assumes finite normalized second moments and bounded twist-score gradients.Poor proposal-target matching can still cause extremely large second moments and importance-weight degeneracy.
  • Under these assumptions, the level-wise estimator is consistent, with asymptotic variance governed by the discrepancy between the previous proposal and current target.A closer proposal-target match yields less-degenerate weights and a more accurate positive-phase gradient estimate.

D.3 Interpretation of Adaptive Threshold Selection

Adaptive threshold selection balances the number of intermediate levels against the statistical difficulty of each transition. Under stated assumptions, empirical thresholds asymptotically recover the intended separation, supporting warm starts between levels.

  • Finite-sample consistency: As N increases, the adaptive empirical construction asymptotically recovers the intended statistical separation between consecutive level targets.This closes the gap between the ideal population construction and Algorithm 1’s finite-sample threshold selection.
  • Adaptive threshold interpretation: The multilevel fraction ρ controls a trade-off: smaller ρ creates fewer, harder transitions, while larger ρ creates more, easier transitions.The discrepancy between consecutive target distributions increases as ρ decreases, whereas larger ρ requires more levels to reach the final rare event.
  • Adaptive threshold interpretation: The adaptive quantile rule approximately equalizes transition difficulty by keeping divergence between consecutive target distributions approximately constant.This makes the statistical separation between successive levels more uniform.
  • Adaptive threshold interpretation: Warm-starting is theoretically justified because consecutive CTL objectives correspond to target distributions separated by controlled divergence.The twist learned at one level therefore provides a principled initialization for the next level.
  • Finite-sample consistency: For finite samples, Algorithm 1 enforces the desired ρ-fraction only empirically, so the selected threshold may retain slightly more or less mass than intended.The population condition and divergence identities hold asymptotically rather than necessarily at finite N.
  • Finite-sample consistency: Under the quantile-identifiability assumptions, the empirical threshold converges almost surely to the population threshold, and clipping is eventually inactive when the population threshold is below γ⋆.The finite response space and finite set of score values support eventual threshold agreement.

D.4 Positive-sample Complexity

The multilevel construction addresses the positive-sample bottleneck by allocating samples across progressively rarer events. Under ideal proposals, its dependence on the rare-event probability changes from inverse in p⋆ to polylogarithmic in 1/p⋆, and this advantage persists under sufficiently small proposal error.

  • Motivation: Standard Twisted SMC faces a positive-sample bottleneck because direct sampling of the target event becomes increasingly unlikely as p⋆ decreases.The comparison motivates using intermediate levels whose conditional event probability is controlled.
  • Ideal multilevel procedure: N ≥ log(L/δ) −log(1 −ρ) samples per level suffice to obtain a positive response across all L transitions with probability at least 1 −δ.Each next-level event retains conditional probability ρ under the ideal level proposal.
  • Complexity comparison: The ideal multilevel procedure changes positive-sample dependence from inverse in p⋆ to polylogarithmic in 1/p⋆.The construction uses progressively rarer levels rather than requiring direct positives from the final event.
  • Positive samples per level: With high probability, ideal multilevel sampling preserves a constant-order fraction of positive responses at every level.This maintains an informative positive training signal throughout progressive twist learning.
  • Imperfect proposals: The polylogarithmic dependence remains valid for imperfect proposals when their total-variation error is uniformly smaller than ρ.Proposal error degrades the guarantee through the gap ρ −ε rather than eliminating the advantage abruptly.

D.5 Sensitivity to Imperfect Score Function

The analysis separates error from imperfect rare-event scores into score-induced bias and proposal-dependent sampling error. It also provides score-based probability intervals whose width depends on threshold-neighborhood mass and score-error probability.

  • Score misspecification: The practical score may differ from the ground-truth score, so the operational rare-event probability can differ from the ground-truth probability.The analysis quantifies this discrepancy using a tolerance ηsc and error probability εsc.
  • Probability bounds: The ground-truth probability is bounded by practical-score tail probabilities: p̂_R(γ⋆+ηsc)−εsc ≤ p⋆ ≤ p̂_R(γ⋆−ηsc)+εsc.Both endpoints depend only on observable practical-score probabilities.
  • Probability bounds: Interval width is governed by practical-score mass near γ⋆ together with the probability that score error exceeds ηsc.A narrow interval indicates limited effect from perturbations of size ηsc.
  • Finite-sample estimation: The interval endpoints can be consistently estimated by re-thresholding the same weighted evaluation samples at γ⋆−ηsc and γ⋆+ηsc.No additional samples are required for this endpoint estimation.
  • Finite-sample estimation: An asymptotically valid confidence interval for p⋆ is obtained by expanding the estimated endpoints by εsc and clipping them to [0, 1].The stated coverage is asymptotically at least 1−αL−αU.
  • Error decomposition: Mean-squared error separates into score-induced bias, (p̂⋆−p⋆)^2, and proposal-dependent sampling error involving χ2(p̂σ⋆∥q).Improving the proposal cannot eliminate bias from an inaccurate practical score, but it can reduce sampling error.

E.3 Implementation Detail

The experiments implement the proposed framework alongside standard Twisted SMC, LMTailRisk, and Self-Distilled TSMC, and evaluate it across safety tasks and model scales.

  • Baselines and setup: Standard Twisted SMC follows Zhao et al. (2024), LMTailRisk uses its official code with default hyperparameters, and Self-Distilled TSMC uses two self-distillation iterations.Training-sample budgets are matched across methods by dividing the budget among Self-Distilled TSMC stages.
  • Toxic story generation: The toxic story task uses TinyStories-33M with the prompt “Once upon a time, there was a.” and evaluates toxicity with a toxicity model.This provides the small-scale language-model setting.
  • Jailbreak attacks: The jailbreak experiment uses harmful prompts from StrongREJECT to study rare undesirable behaviors in modern safety-aligned LLMs.StrongREJECT covers six harmful categories, including disinformation and deception and hate.
  • Imperfect score robustness: The imperfect-score study perturbs the toxicity reward deterministically with δ(x) sampled from Uniform[−0.5, 0.5].The perturbation ensures |R̂(x)−R(x)| ≤ ηsc = 0.5 while keeping scores consistent across training and evaluation.
  • Ablation study: The ablation varies ρ across {0.3, 0.45, 0.5, 0.52, 0.65} and samples per level across {256, 512, 1024, 1536, 2048}.It targets the toxic-story threshold γ⋆=7, described as the most challenging rare event in that task.

E.4 Guidance for Selecting the Adaptive Threshold ρ

The adaptive fraction ρ controls how difficult successive intermediate-event transitions are and how many levels are needed. Analytical criteria favor moderate values: approximately 1/2 for obtaining one positive response per level and 1/e for maintaining multiple positive responses, while imperfect proposals make these optima approximate.

  • A smaller ρ creates fewer but harder transitions, whereas a larger ρ creates more closely spaced levels and more transitions.
  • Criterion 1: At Least One Positive Response per Level: ρ = 1/2 minimizes the sufficient sample budget for obtaining at least one positive response at every level.
  • Criterion 2: A Constant Number of Positive Responses per Level: ρ = 1/e ≈ 0.368 minimizes the sufficient sample budget for maintaining a constant number of positive responses at every level.
  • The experiments choose ρ = 0.3 when larger per-level sampling budgets permit multiple positive responses and ρ = 0.5 when avoiding empty positive phases is prioritized.
  • The analysis supports moderate ρ values because imperfect proposals can reduce the effective transition probability from ρ to ρ − ε.

F.1 Toxic Story generation

On toxic story generation, the method becomes increasingly advantageous as the target event grows rarer, producing lower errors and more stable estimates than the baselines. It also discovers extreme high-toxicity continuations and adds only modest computational overhead.

  • At thresholds 5.0, 5.5, 6.0, and 7.0, the method achieves substantially lower relative error than SMC and SD-TSMC while maintaining much higher rare-event hit rates.
  • At γ⋆ = 7.0, SD-TSMC’s estimated probability is on the order of 10^-7 while its standard deviation is on the order of 10^-6.
  • The method discovers high-toxicity continuations with rare-event scores around 10.3 that neither baseline finds under the same evaluation budget.
  • The method’s main added resource cost is GPU memory, increasing from 3.97 GiB to 5.23 GiB, while training time is slightly lower and GPU utilization and power consumption are nearly identical to standard Twisted SMC.
  • Across six harmful categories and three language models, the method consistently outperforms standard Twisted SMC and LMTailRisk.

F.3 Imperfect Score Function

The method remains effective when trained with a misspecified proxy score, producing more accurate ground-truth probability estimates than standard Twisted SMC. Its advantage is strongest for rarer events, where the baseline can return zero estimates.

  • For γ⋆ ≥ 5.0, standard Twisted SMC returns zero probability estimates across all seeds, yielding 100% relative error, whereas the method remains informative.
  • Across five independently trained proposals, the method’s confidence intervals cover the Monte Carlo reference in most settings, with coverage values of 5/5, 4/5, 4/5, 4/5, 4/5, 1/5, and 5/5.
  • The results support robust inference under score misspecification, although proposal learning cannot remove bias introduced by an imperfect score.
  • Under the imperfect proxy score, the method is substantially more stable across random seeds than Twisted SMC.

F.4 Ablation Study

The ablations show that larger per-level sampling budgets improve proposal quality and estimation accuracy but increase runtime and memory. The quantile ρ also strongly affects statistical efficiency, while computational usage changes little across tested values.

  • Samples per level: Increasing samples per level from 256 to 2,048 raises the rare-event hit rate from 4.14×10^-2 to 0.81 and ESS from 3.87 to 149.30.
  • Samples per level: The relative error decreases from 93.1% to 12.8% as the per-level sample count increases from 256 to 2,048.
  • Samples per level: The larger sampling budget increases runtime from 2.35 to 8.82 minutes and peak memory usage from 2.57 to 6.98 GiB.
  • Quantile parameter ρ: Increasing ρ from 0.30 to 0.65 decreases the rare-event hit rate from 0.55 to approximately 0.01 and ESS from 135.00 to 41.93.
  • Quantile parameter ρ: Across tested ρ values, runtime ranges from 4.28 to 5.37 minutes while GPU utilization, power consumption, and memory usage remain relatively similar.

F.5 Training Metrics

The adaptive multilevel procedure progressively raises event thresholds while maintaining positive samples and stable effective sample sizes. Its sampled responses become increasingly harmful, reaching the final target threshold and supporting rare-event estimation.

  • Thresholds increase from 0.0072 to the final target 0.5, while sampled response scores rise from 0.013 to 0.618.Responses begin containing harmful content at level 4, with score 0.466, and exceed the target threshold near the final level.
  • The procedure constructs intermediate events that guide the proposal toward generating harmful responses targeted by the rare-event estimator.This demonstrates the intended role of progressively stricter levels in adaptive multilevel training.
  • Positive-sample rates remain around 0.4 across levels and reach approximately 0.495 for the final target rare event.These rates stay close to the target quantile 0.5, maintaining positive examples throughout twist learning.
  • The effective sample size remains stable at around 15 across levels, indicating relatively stable estimation error for the positive-phase gradient.The reported behavior is consistent with Proposition 1 and supports the method’s validity.
  • The method discovers rare unsafe responses on the StrongREJECT jailbreak task that other methods did not find.
Loading 2608.21736v1…