Quantifying Behavioral Tails in Black-Box Language Models

1 INTRODUCTION

• Rare-event evaluation lacks statistical resolution for severe behaviors, while automated red teaming finds failures but does not quantify their probability under an independently specified reference prompt distribution.Direct evaluation preserves prevalence under the reference distribution but loses resolution as events become rare.Source: §1 · pp. 1–2
• Prior rare-event approaches often restrict outcomes to specific output features, vary decoding randomness for a fixed prompt, or rely on model-internal representations, motivating a general-purpose prompt-distribution method.The target is an exceedance probability for a response-level behavior under an explicit, reproducible prompt distribution.Source: §1 · p. 2
• RARETRAP maps a Gaussian reference variable through a geometry-aware surrogate transformation to discrete prompts, then uses sequential Monte Carlo to explore increasingly severe behaviors while preserving probability under that prompt distribution.The target model remains black box: the method requires prompt-response access but no gradients, logits, hidden states, parameters, or labeled failures.Source: §1 · p. 2
• Across 10 open-weight and two frontier models, RARETRAP elicits severe resource-consumption behaviors and estimates their exceedance probabilities, which vary substantially across models and prompt distributions.The evaluated behaviors are over-generation and degenerate repetition.Source: §1 · p. 2
• The paper frames behavioral tails jointly by the target model, response-level performance metric, and input distribution, enabling risk comparisons beyond isolated failures or benchmark averages.The authors envision using tail-risk estimates to compare candidate models, manage compute policies, and target alignment effort at behavior-inducing input populations.Source: §1 · p. 2

2 RELATED WORK

• Population-based evaluations measure behavior frequency or severity, and probabilistic guarantees characterize outcomes over specified prompt distributions, but these approaches lack resolution in the tail.Examples include Bloom, XGUARD, Deployment Simulation, and probabilistic guarantees over prompt or conversation distributions.Source: §2 · pp. 2–3
• Rare-event methods study rare outcomes under input distributions, stochastic generation, or model representations, but commonly restrict target events, hold prompts fixed, or use model-internal representations.These methodological differences motivate estimating response-level behavior under an explicit prompt distribution.Source: §2 · p. 3
• Automated red teaming seeks inputs that elicit severe or diverse failures, while resource-exhaustion studies demonstrate excessive generation or reasoning; RARETRAP instead estimates the probability of behavior levels under a specified prompt distribution.ThinkTrap is particularly related through its use of a low-dimensional prompt representation for black-box search.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• The method defines behavioral risk as an exceedance probability under an explicit prompt distribution, separating the distribution induced by a surrogate from the response-level event and its severity threshold.The target model is queried as a black box, and the framework supports distinct behavior definitions.Source: §3.1 · p. 3

3.2 GEOMETRY-AWARE LATENT-TO-PROMPT MAPPING

• A fixed geometry-aware map transforms a low-dimensional Gaussian draw into continuous slot representations, discretizes them to surrogate tokens, and decodes those tokens into a prompt.The map shapes representations using surrogate token-embedding geometry; a shared latent draw jointly determines all prompt slots. The surrogate and target need not share vocabularies, tokenization, parameters, or embedding spaces.Source: §3.2 · p. 4
• Nearest-token decoding induces an explicit discrete prompt distribution by partitioning latent space into token-sequence regions whose Gaussian masses determine sequence and prompt probabilities.The realized map is fixed for evaluation, and its induced distribution provides the reference measure for exceedance probabilities.Source: §3.2 · p. 5

3.3 SEQUENTIAL MONTE CARLO

• Sequential Monte Carlo decomposes a small target probability into conditional probabilities over nested events, using adaptive severity thresholds and conditional sampling to retain samples in progressively more severe regions.Intermediate thresholds use empirical quantiles; modified component-wise Metropolis–Hastings generates samples conditional on each event, and the final estimate combines completed transition probabilities with the remaining conditional fraction.Source: §3.3 · pp. 5–6
• The sequential procedure yields both a target-event probability estimate and conditional prompt-response samples at increasing severity levels, characterizing the behavioral tail.RARETRAP defines the target probability independently of the rare-event estimator and uses Subset Simulation with modified component-wise Metropolis–Hastings.Source: §3.3 · p. 6

4.1 EXPERIMENTAL SETUP

• Across two behaviors, the study evaluates 42 target–surrogate–behavior evaluations; open-weight models use greedy decoding with L = 40, D = 200, and p0 = 0.1, while frontier models use provider-default non-greedy sampling.Source: §4.1 · p. 6

4.2 RQ1: ESTIMATING BEHAVIORAL TAILS

• All 40 open-weight target–surrogate–behavior evaluations reached the target event, with 16 meeting the direct-estimation criterion in 200 samples and every run terminating within four levels.The event remained measurable at thresholds of 32,768–100,000 tokens; higher-capacity models also exhibited both studied behaviors under two external surrogates.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
• Under the common Qwen3-0.6B surrogate, over-generation estimates ranged from 3.9 × 10−3 for Qwen3-14B to 1.0 for Phi-4-reasoning, while changing surrogates shifted estimates by up to two orders of magnitude.Neither surrogate consistently produced the larger estimate; estimates were closest for Qwen3-14B, which shares its vocabulary with Qwen3-0.6B, and diverged most for Nemotron-9B and GPT-OSS-20B.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
• For frontier models under the Qwen3-0.6B reference distribution, RareTrap reached every target event within 1,900 evaluations for GPT-5.4 and 3,700 for Claude Sonnet 4.6, resolving probabilities to approximately 10−4.Nearly every GPT-5.4 response reaching the cap used its full hidden-reasoning budget without visible text; Claude’s safety-filter-blocked share fell from 73% of reference prompts to about 30% by level four.Source: Table 3 · p. 7, §4.2 · p. 8

4.4 RQ3: THE EFFECT OF SUCCESSIVE CONDITIONING

• At equal target-model evaluation counts, successive conditioning shifts the 25th percentile, median, and 75th percentile toward greater severity after the initial set, unlike continued independent sampling.Table 4 reports P25/median/P75; later-level quantiles use newly accepted proposals only.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 RQ4: ABLATION ON GEOMETRY-AWARE MAPPING

• The geometry-aware projection generally yields higher median and upper quantiles for both rlen and rrep than the embedding-agnostic projection across evaluated models.The ablation holds the surrogate, discretization, decoding, latent samples, and target evaluation fixed.Source: §4.5 · pp. 8–9, Table 5 · p. 9

4.6 DISCUSSION

• Estimates are properties of the target together with the surrogate-defined reference distribution, so fixing a surrogate supplies a common reference for cross-model comparison rather than deployment-prevalence estimates.Different surrogates expose different prompt-space regions and can yield different tail risks.Source: §4.6 · p. 9
• Tail-enriched prompt–response populations can supply targeted regression cases or candidate post-training data, while observed high-severity behaviors include continued generation despite confusion or repetition.Mitigation is outside this work’s scope.Source: §4.6 · p. 9

5 CONCLUSION AND LIMITATIONS

• RareTrap demonstrates behavioral-tail characterization across 10 diverse open-weight models under multiple surrogates and two criteria, and on frontier proprietary models accessed through an API.The framework aims to provide a common basis for quantifying and comparing tail risk under a specified prompt distribution.Source: §5 · p. 9
• Probability-estimation accuracy claims are restricted to the probability range covered by direct Monte Carlo comparisons; substantially deeper estimates are not empirically validated here.Additional conditional levels can extend RareTrap to progressively rarer events, but this study does not validate their deeper probability estimates.Source: §5 · p. 9
• Ties in discrete response metrics can bias per-level probability estimates, and the current implementation does not explicitly account for them.Ties may arise from multiple latent realizations mapping to one prompt or from different prompts producing the same discrete severity value.Source: §5 · pp. 9–10

A REPETITION METRIC

• The distinct-n ratio Dn is the number of distinct contiguous n-grams divided by the number of possible n-gram positions, for n no greater than response length.The paper evaluates bigram, trigram, and four-gram diversity.Source: A REPETITION METRIC · p. 15
• At each order, Dn = 1 means all n-grams are distinct, while smaller values indicate lower n-gram diversity.Across evaluated responses, D2, D3, and D4 are defined and lie in (0, 1].Source: A REPETITION METRIC · p. 15
• The repetition score rrep increases strictly when any one of D2, D3, or D4 decreases while the other orders remain fixed.The resulting score satisfies 0 ≤ rrep < 1, and larger values indicate stronger repetition across the considered local scales.Source: A REPETITION METRIC · p. 15
• The target event is rrep(y(z)) ≥ τrep, using the response-level performance function g(z; τrep) = τrep − rrep(y(z)).The threshold therefore defines which responses count as sufficiently repetitive.Source: A REPETITION METRIC · p. 15

B.1 SHARED-LATENT DISTRIBUTIONAL STRUCTURE

• Each prompt position has the same marginal covariance, while the relative transformation RℓR⊤m controls dependence between positions.The factor D/E quantifies expected covariance contraction; choosing αdim = E/D would compensate in expectation, but is not used in experiments.Source: B THEORETICAL PROPERTIES OF THE… · p. 16, §B.1 · pp. 16–17
• The shared-latent continuous prompt distribution has covariance rank at most D, regardless of prompt length, and its position-specific transformations induce structured cross-position dependence.The expected marginal covariance over random subspaces is (D/E)CS; the dimensional scale correction is analytical, while experiments use the unscaled construction.Source: §B.1 · pp. 16–18, B THEORETICAL PROPERTIES OF THE… · p. 16

B.2 NEAREST-TOKEN DECODING AND THE INDUCED PROMPT DISTRIBUTION

• Nearest-token assignments partition latent space into convex polyhedral regions whose boundaries have zero Gaussian probability, assuming no distinct candidate tokens are equidistant everywhere on the affine support.Rotational invariance gives the same marginal token distribution at every prompt position.Source: §B.2 · pp. 18–19
• A surrogate-token sequence receives the Gaussian mass of its latent region, and a decoded text prompt receives the summed masses of sequences that decode to it.The latent preimage of a text prompt is the union of its sequence cells, up to a Gaussian-null set.Source: §B.2 · pp. 18–19
• Positive scaling can change the masses assigned to token sequences and prompts without changing which have positive probability.Under deterministic target decoding, the probability of exceeding a severity threshold is the Gaussian mass of latent regions whose prompts produce responses at or above that threshold.Source: §B.2 · p. 20

B.3 FACTORIZED PROJECTION COMPLEXITY

• The factorized projection realizes the same map as an explicit operator while matching its leading-order evaluation cost and reducing storage as prompt length grows.For L = 40, D = 200, and E = 4096, factorized storage is approximately 7.4% of explicit-operator storage.Source: §B.3 · p. 20

C SEQUENTIAL CONDITIONAL SAMPLING DETAILS

• RARETRAP repeatedly sets a threshold from the current performance-value quantile, then uses retained samples to construct conditional samples until the target boundary is reached.Modified componentwise Metropolis–Hastings targets the Gaussian distribution restricted to each intermediate event; each completed level contributes a factor p0, while direct Monte Carlo needs O(pF^-1) samples for fixed relative precision as pF approaches zero.Source: C SEQUENTIAL CONDITIONAL SAMPLING… · p. 21

D LANGUAGE MODELS EVALUATED

• All evaluated models are run from their publicly released checkpoints.The repository identifier for each model is provided in Table 6.Source: D LANGUAGE MODELS EVALUATED · p. 22, Table 6 · p. 22
• The main output-length and repetition experiments cover instruction-tuned and reasoning models spanning roughly 7B to 30B parameters across multiple model families.The models are listed in Table 6, which provides their public repository identifiers.Source: D LANGUAGE MODELS EVALUATED · p. 22, Table 6 · p. 22
• The surrogate supplies the embedding geometry for the latent-to-prompt map and need not be the target under evaluation.Source: D LANGUAGE MODELS EVALUATED · p. 22

E SENSITIVITY STUDY

• All DeepSeek-8B configurations reach threshold in 200 samples, with estimates from 0.135 to 0.50; prompt-length and latent-dimension effects vary by target and surrogate.For Qwen3-14B, D has a larger effect than L; at D = 100, prompt and response ties prevent probability estimation, so only threshold progression is reported.Source: E SENSITIVITY STUDY · p. 23

E.1 EXTREME OVER-GENERATION

• Within three levels and at most 2,800 evaluations, every tested configuration reaches the raised over-generation threshold, up to 100,000 tokens or the model’s lower maximum sequence length.Estimates span the same range as the main results, from a common event for DeepSeek-8B under the Qwen3-0.6B surrogate to the 10^-3 scale for OLMo-3-7B under the self surrogate; intermediate thresholds are a few hundred to about a thousand tokens.Source: §E.1 · p. 24, Table 8 · p. 24

F WHAT RARETRAP RESPONSES LOOK LIKE

• Tail examples begin with unreadable mixed-script prompts that models often recognize as unclear but continue answering instead of stopping or seeking clarification.Observed patterns include looping on confusion, overriding a stated fallback, drifting or inventing a task, and repeating a surface form to the generation cap; examples illustrate patterns, not their prevalence.Source: F WHAT RARETRAP RESPONSES LOOK LIKE · p. 25

F.1 THE MODEL SAYS IT IS STUCK, BUT KEEPS GOING

• Models voice reasons to stop yet continue, looping over interpretations or policy checks, repeating a sentence, or explicitly recognizing that they are stuck.One GPT-OSS response sustains an unresolved policy check before refusing; a Phi-4 response repeats one sentence, while another response repeatedly revises its interpretation.Source: §F.1 · pp. 25–26

F.2 THE MODEL KNOWS IT SHOULD ASK, BUT ANSWERS ENDLESSLY

• A model identifies asking for clarification as appropriate, then replaces that action with open-ended analysis that continues to the generation cap.The example recognizes that no specific question was provided but proceeds to analyze possible explanations for the string.Source: §F.2 · p. 26

F.3 ANNOUNCING THE FINAL ANSWER, AGAIN AND AGAIN

• Even after formulating and labeling a clarification as its final answer, Qwen3.5 repeatedly returns to the finalization sequence instead of terminating.The response remains in a cycle of finalization and revision through the generation cap.Source: §F.3 · pp. 26–27

F.4 THE SAME SENTENCE REPEATED WITH DIFFERENT WORDS

• A fixed self-checking frame can sustain generation while its topic changes across more than 370 concepts; another model sustains output by inventing both sides of a conversation.In the fixed-frame case, the changing concepts do not alter the frame or end the response.Source: §F.4 · pp. 27–28

F.5 ONE SYMBOL REPEATED TO THE GENERATION CAP

• OLMo-3-7B first interprets the prompt as encoded or ciphered text, then locks onto one character and switches to repeating lion emoji through the generation cap.The example moves from a plausible interpretation and listing glyphs to output dominated by a single character.Source: §F.5 · p. 28
• A model may introduce a symbol while reconstructing a prompt, after which that symbol becomes the dominant portion of its response.The reported Nemotron-3.5 example introduces a glyph absent from the input and then repeats it as the sole surface form.Source: §F.5 · p. 28
• Qwen3-14B repeatedly generates a tool-call marker after first interpreting the control-looking string as linguistic content and later identifying it as tool-call syntax.The prompt-derived marker becomes the focus of almost the entire response.Source: §F.5 · p. 28

• Breakdown occurs in different positions: one response repeats severely inside its reasoning, while another remains coherent through response selection before breaking down in its answer.Source: §F.6 · p. 29
• In the first response, severe repetition consumes almost the entire reasoning, but the model escapes it and produces a concise answer.Source: §F.6 · p. 29
• In the converse response, the reasoning settles on a concise clarification before breakdown appears in the answer.Source: §F.6 · p. 29
• The eventual answer’s semantic quality does not identify whether substantial resource-consuming behavior occurred elsewhere in generation.Source: §F.6 · p. 29

F.7 LONG IS NOT THE SAME AS REPETITIVE

• Over-generation and degenerate repetition overlap, but are not interchangeable; a response at the length threshold can remain below the repetition threshold.Source: §F.7 · pp. 29–30
• GPT-OSS crossed the repetition threshold after only 636 tokens, illustrating that repetition can qualify without an unusually long response.Source: §F.7 · p. 30

F.8 SUMMARY

• The examples reveal qualitatively distinct mechanisms behind similar structural extremes; Table 9 lists examples, not their frequency within the tail.Source: §F.8 · p. 30, Table 9 · p. 30
• Tail samples show several severe-generation mechanisms, with breakdown in reasoning or the eventual answer; rlen, rrep, and answer semantics are related but non-equivalent.Source: §F.8 · p. 31
• Prompt–response populations complement bpF(τ) by describing the prompt-space region to which that probability is assigned.Source: §F.8 · p. 31