PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictive Likelihood Ratios for Language Model Watermark Detection

Li Ma

arXiv:2609.15657v1stat.MLcs.AIcs.LGstat.AP

TL;DR

Keyed watermark detection must test dependence without knowing the generating model’s next-token probabilities. This paper averages predictive likelihoods over uncertain deficits and residual-tail distributions, then evaluates robustness across specified alternatives and matched-temperature outputs. The union-tail mixture has low maximum regret across the tested synthetic alternatives and improves AUC over the equal-tail baseline in all eight non-saturated cells, though the leading reference score generally has higher AUC.

  • Problem

    Pivotal watermark detection removes the unknown next-token probability vector from the null, but leaves a composite alternative whose distribution varies across prompts and token contexts.

    Source: §1 · p. 2

  • Method

    The paper constructs predictive likelihood ratios by averaging the alternative likelihood over prespecified priors on deficits and residual-tail structure, including a mixture of tail-shape and tail-width families.

    Source: §1 · p. 2, §2.4 · p. 6

  • Results

    Across six full-width tail laws, four narrow-width regimes, and three horizons, the union-tail model has maximum regret .0080 versus .0962 for the equal-tail Bayesian model, and higher AUC than the shared-deficit equal-tail model in all eight non-saturated cells.

    Source: §4 · p. 19

  • Takeaways & Limitations

    The findings support a fixed mixture across the evaluated alternatives, but establish neither minimax performance over a larger class nor uniform dominance.

    Source: §4 · p. 19

    Empirical comparisons are conditional on specified grids and Monte Carlo samples, and the archived reference sampler’s conditional-null assumption for anytime validity is not verified.

    Source: §3.6 · p. 16, §4 · p. 19

Abstract

from arXiv · show

Keyed watermark detection tests dependence between observed tokens and pseudorandom variables reconstructed from a secret key. Building on the pivotal framework of Li et al. (2025), we construct predictive likelihood ratios that average over uncertain probability deficits and residual-tail distributions. The aim is robust detection power across alternative specifications without requiring a single signal-strength tuning. A mixture prior combines tail shape and effective width; hierarchical extensions allow within-document variation in deficit or width. The test maximizes prior-averaged power at a fixed size, but is not generally uniformly most powerful or minimax. Under the exact conditional pivot null, normalized predictive alternatives selected before each observation yield a Bayes factor that is also a test martingale: Type I error control is unaffected by alternative misspecification and remains valid under optional stopping. This guarantee does not cover violations of the conditional null, and the interpolated implementation has no certified anytime guarantee. Gumbel marginal likelihoods are evaluated by fixed quadrature. Across the evaluated tail-shape and tail-width alternatives and three horizons, the union-tail mixture has maximum observed Type II error regret .0080, compared with .0962 for the equal-tail mixture, relative to the best tested rule. On temperature-matched outputs from two open models, it improves AUC over the equal-tail baseline in all eight non-saturated model-temperature cells, although the leading reference score generally has higher AUC. Supplementary experiments show retained power under independent null-like replacement and smaller changes from hierarchical dependence modeling. The evidence supports robustness across the evaluated alternatives, not uniform power guarantees or resistance to arbitrary text edits.

1 Introduction

The paper extends pivotal watermark detection to composite alternatives by averaging likelihoods over uncertain signal strength and residual-tail structure. Predictive likelihood ratios retain exact null validity under the conditional pivot null, while their power robustness is assessed across specified alternatives.

1 Introduction

  • The predictive alternative averages over prespecified deficits and residual-tail distributions, maximizing prior-averaged power at fixed size without guaranteeing robustness or uniform power.The authors distinguish misspecification of the deficit from misspecification of the residual-tail distribution.Source: §1 · p. 2
  • Normalized predictive alternatives selected before each observation produce a Bayes factor that is also a test martingale under the exact conditional pivot null.This gives Type I error control under optional stopping despite alternative misspecification; it does not cover conditional-null violations, and the interpolated implementation lacks a certified anytime guarantee.Source: §1 · p. 2
  • The framework builds on pivotal likelihoods that remove the unknown next-token probability vector under the null, while prior-based likelihood detectors either estimate alternatives from labeled data or require NTP information.The construction uses analytic pivots of Li et al. (2025a) with prespecified priors on Δ and the Gumbel residual tail.Source: §1 · p. 2
  • The approach combines exact predictive likelihood construction, a mixture over tail-shape and tail-width families, and shared or tokenwise latent-parameter specifications.Hierarchical extensions place priors on document-level hyperparameters governing token-specific deficits or widths, inducing partial pooling.Source: §1 · p. 3
  • The assessment compares synthetic alternatives at a common nominal 5% size and examines archived and temperature-matched outputs from OPT-1.3B and Sheared-LLaMA-2.7B.The archived and newly generated sequences do not support an anytime-valid claim.Source: §1 · p. 3

2 Method

The method constructs Gumbel-pivot likelihood ratios by averaging over tail-shape and effective-width alternatives, with latent alternatives organized as shared or tokenwise Bayes factors. Under the exact conditional pivot null, predictably selected normalized alternatives provide test-martingale validity; nondegenerate priors target prior-averaged rather than generally minimax power.

2.1 Keyed pivots and the conditional null

  • Sequential validity requires the conditional pivot null with fresh key randomness; context or hash reuse, masking, collisions, or a nonideal pseudorandom function can violate it.Literal independence and identical distribution are sufficient for the conditional statement, but are not necessary.Source: §2.1 · p. 3

2.2 Gumbel-max likelihood

  • With independent uniform Gumbel-max key coordinates, the selected-token pivot is conditionally Uniform(0,1) under H0, yielding an exact null density.The proposition assumes the Gumbel-max key law; the broader conditional-null assumption alone does not specify the pivot law.Source: §2.2 · p. 4

2.3 Averaging over the NTP family

  • The Dirichlet tail layer draws residual proportions from a symmetric Dirichlet, reducing likelihood averaging to a Beta marginal; its top-token interpretation requires Δ≤1/2.The concentration parameter controls how evenly the residual mass is spread, with large α recovering the equal-tail specification.Source: §2.3 · p. 5

2.4 Averaging over the tail width

  • The union-tail prior mixes Dirichlet tail shape and effective width as separate branches; experiments fix w=.5, a working choice rather than an empirically optimized weight.The two-component mixture avoids the product grid; its components are exact submodels.Source: §2.4 · p. 6

2.5 Hierarchical generative model

  • The shared hierarchy multiplies token likelihoods before integrating document-level parameters; the tokenwise hierarchy integrates position-specific parameters first, while union-tail rules keep S document-level.The evaluated rules do not redraw the tail state by token.Source: §2.5 · p. 7, Figure 1 · p. 8

2.6 Bayes factors

  • On 200-token paths, document-level-state union-tail Bayes factors differ by up to 26 nats from fully tokenwise mixtures as the branch posterior updates across positions.The evaluated factor mixes branch-specific products rather than multiplying per-token mixtures.Source: §2.6 · pp. 9–10

2.7 Martingale Bayes factors and e-values

  • Under the exact conditional pivot null, pre-observation normalized alternatives yield a nonnegative mean-one test martingale with level control under optional stopping.For bounded stopping times the Bayes factor has expectation one; for almost surely finite unbounded stopping times its expectation is at most one.Source: §2.7 · p. 10
  • Under the true conditional null, normalized alternatives selected using past information yield conditional e-values and a Bayes factor that is also an e-process; misspecification can affect power, not null validity.The guarantee requires the denominator to be the true conditional null and the numerator to remain predictably selected and normalized.Source: §2.7 · p. 10

2.8 Relationship to the optimal scores of Li et al. (2025)

  • For a point prior in a fixed component family, the likelihood-ratio statistic matches the optimal score of Li et al. (2025a), but this equivalence is family-specific and nondegenerate priors are not generally minimax.The equal-tail baseline uses a different spike component, so its point-prior limit is a different statistic.Source: §2.8 · p. 11

3 Numerical Experiments

The experiments compare calibrated watermark detectors across deficit, residual-tail, and within-document dependence specifications. The union-tail mixture shows the lowest maximum regret in the evaluated width-and-shape grid, while temperature-matched results favor the tail-width component within the Bayesian family; these findings are limited to the tested settings.

3.1 Design and competitors

  • The synthetic comparisons use separately calibrated fixed-horizon scores and evaluate them against independently sampled null and alternative sequences.The shared-deficit design draws one Δ per document; the tokenwise design redraws deficits at each token, and the compared reference scores include the published Gumbel tunings.Source: §3.1 · pp. 11–12

3.2 Shared-deficit equal-tail experiment

  • At n=700, the shared Bayes rule has Type II error 0.0014 versus 0.0062 for the best tested Gumbel reference score in the shared-deficit experiment.The relative reduction is 77.4%; the tokenwise Bayes error is 0.0042. Under independently redrawn deficits, all three tokenwise mixtures have no observed misses by n=100 among 5,000 alternatives.Source: §3.2 · pp. 12, 14

3.3 Robustness to deficit and tail-shape specifications

  • In the deficit sweep at n=100, the three published least-favorable tunings have maximum regrets .2322, .3162, and .6756, versus .0036 for the shared equal-tail mixture.In the tail-shape sweep, the concentration mixture has maximum regret .0014 versus .0046 for the equal-tail rule; close differences are not statistically resolved, and neither sweep establishes minimax guarantees.Source: §3.3 · p. 14

3.4 Robustness to tail-width misspecification

  • Across four widths, six shape laws, and three horizons, union tail has maximum regret .0080 versus .0962 for the equal-tail layer and .4668 for the shape mixture.The comparison is against the best rule scored in the finite sweep, not a minimax guarantee.Source: §3.4 · p. 14, Table 1 · p. 15

3.5 Hierarchical extensions

  • Hierarchical models change estimated Type II error by at most .0012 for deficits and .0038 for widths; the observed changes are smaller than those from extending residual-tail support.No adjusted hierarchical-versus-shared deficit contrast is significant, which does not establish equivalence.Source: §3.5 · p. 16

3.6 Archived benchmark data of Li et al.

  • The archived benchmark compares watermarked outputs at temperature .1 with unwatermarked outputs at temperature one, confounding watermark sensitivity with low-temperature repetition.The reference keyed sampler may reuse seeds, so the conditional-null assumption for anytime validity is unverified; these comparisons are restricted to fixed-horizon empirical errors.Source: §3.6 · p. 16

3.7 Temperature-matched regeneration

  • For matched temperatures .2–.5, ℎars has the lowest mean Type II error, followed by ℎ★ gum,.1 and the union-tail mixture; this ordering does not hold in every cell.The width-only prior’s AUC advantage over the shared-Δ rule persists on the 2,000-prompt holdout, with significant differences in all eight cells.Source: §3.7 · p. 17, Table 2 · p. 18
  • In temperature-matched data, the width-only prior has higher AUC than the shared-Δ rule in all eight cells, and higher AUC than the shape-only prior in all eight.The width-only versus shared-Δ differences are significant and range from .0091 to .0550; the endpoint comparison favors width-only by .0091 to .0475.Source: §3.7 · p. 17

4 Conclusion

The paper develops robust watermark detection by averaging over uncertain alternatives, with gains across evaluated specifications but no uniform-dominance claim. Its exact-null sequential guarantee is conditional, and empirical conclusions remain bounded by the evaluated designs.

4 Conclusion

  • Under the exact conditional null, normalized predictive densities selected before each observation yield a test martingale, but the implemented interpolation lacks certification for this guarantee.The Bayes factor’s interpretation depends on the specified models; posterior probabilities additionally require prior odds, and optimal decisions depend on the loss function.Source: §4 · p. 19
  • The union-tail model has higher AUC than the shared-deficit equal-tail model in all eight non-saturated matched-temperature cells, with each difference significant after adjustment.The leading reference score has higher AUC than the union-tail model in seven of eight cells, with five differences significant after adjustment.Source: §4 · p. 19
  • Hierarchical extensions change estimated Type II error by at most .0038 in synthetic persistence experiments, while residual-tail specification has a larger observed power effect.None of 18 paired hierarchical-versus-shared deficit comparisons is significant; two of 15 width comparisons are significant.Source: §4 · p. 19
  • Maximum Type II error regret is .0080 for the union-tail model versus .0962 for the equal-tail Bayesian model across the evaluated alternatives and horizons.The comparison is relative to the best tested rule, not a minimax guarantee over a larger class.Source: §4 · p. 19
  • Archived benchmark samples with different temperatures do not identify detector rankings under matched generation, and empirical comparisons are conditional on specified grids and Monte Carlo samples.The contamination check supports retained power under independent null-like replacement, not protection against arbitrary edits; nonsignificant differences do not establish equivalence.Source: §4 · p. 19

Computational materials

The report is accompanied by machine-readable summaries, analysis materials, and stored data for reproducing its evaluations.

Computational materials

  • The companion repository provides machine-readable numerical summaries and prefix-curve tables.Supplementary Section A.11 and the README describe contents, sources, and reproducibility limitations.Source: Computational materials · p. 20
  • Document-level rejection indicators and analysis code accompany the report.Source: Computational materials · p. 20
  • Stored temperature-matched generation arrays and prompt tables are also available in the companion repository.Source: Computational materials · p. 20

A.1 Contamination

The contamination extension models departures from the assumption that every watermarked position is coupled to its key, including independently keyed or unwatermarked positions.

A.1 Contamination

  • The clean working alternative assumes every watermarked position is coupled to its key, while the extension permits independently keyed or unwatermarked positions.Source: §A.1 · p. 23
  • The section specifies a contamination model, gives its graphical representation, and evaluates its effect on detection error.Source: §A.1 · p. 23
  • The clean model is recovered when the contamination probability is zero.Source: §A.1 · p. 23

A.1.1 The contamination layer

The contamination layer mixes exact-null positions into watermark likelihoods, with shared or tokenwise latent parameters and validity tied to the exact pivot null.

A.1.1 The contamination layer

  • The contamination model covers independently keyed or null-like positions, but excludes burst edits, context-dependent paraphrases, and insertion/deletion alignment.Source: §A.1.1 · p. 23
  • The model specializes Huber’s gross-error model by fixing the contaminating law to the exact pivot null, and places a prior on ρ jointly with Δ.Source: §A.1.1 · p. 23
  • Products of contaminated factors and their nonnegative prior mixtures are test martingales under the exact null, not evidence of Huber robustness of the null.Source: §A.1.1 · p. 23
  • The shared hierarchy draws one set of deficit, tail, and contamination parameters per document, whereas the tokenwise hierarchy redraws them by position conditional on shared S.Source: Figure 3 · p. 24
  • C_t selects the exact-null component, with document-level probability ρ in the shared model and position-specific probability ρ_t in the tokenwise model.Source: §A.1.1 · p. 23, Figure 3 · p. 24
  • The tokenwise contamination reduction integrates the position-specific probability within each one-token density before multiplying densities, unlike mixing whole-document branch priors.Source: §A.1.1 · p. 24

A.1.2 Independent null-like contamination

The experiment tests independent null-like pivot replacement using contamination-aware and clean Bayesian rules, alongside reference scores. The mixture rules retain low Type II error over the tested replacement rates, while the results are limited to this replacement mechanism.

A.1.2 Independent null-like contamination

  • The generating contamination rate spans ρ★∈{0, .1, .25, .4, .6, 1}; at ρ★=1, the alternative is exactly the null and power should match the evaluation Type I rate up to Monte Carlo error.Contamination-aware rules integrate over ρ, while the clean equal-tail rule fixes ρ=0.Source: §A.1.2 · p. 24
  • At ρ★=.4, contamination-aware and clean shared Bayes both have Type II error .0028, versus .0130 for h★ gum,.005; at ρ★=.6, their errors are .0056 and .0050, versus .0188.Adding the Dirichlet tail prior leaves the Gumbel Bayes rows identical to four decimals for ρ★≤.4 and produces a spread of at most .0008 at ρ★=.6.Source: Figure 4 · p. 25, §A.1.2 · p. 25, Table 3 · p. 26
  • For the shared clean-ρ mixture, mean terminal log Bayes factor at n=700 falls from 898 at ρ★=0 to 270 at ρ★=.6, while its Type II error at ρ★=.6 is .0050 versus .0060 for 280 clean tokens.At ρ★=.8 the mean terminal log Bayes factor is 85, and it is negative at the null-equivalent ρ★=1.Source: §A.1.2 · pp. 25–26
  • Across contaminated levels ρ★<1, the mixture’s empirical maximum regret over tested ρ priors is .0004, and independent null-like replacement does not test burst edits, paraphrase, key desynchronization, or insertion/deletion alignment.At ρ★=1, observed power and Type I-rate differences are consistent with null equivalence at the simulation resolution.Source: §A.1.2 · p. 26

A.2 Attribution of the Gumbel difference

The section separates likelihood-family effects from prior averaging and examines whether shared latent deficits are recovered. The reported comparisons show larger divergences for the spike component at one parameter setting, but do not detect an incremental prior-averaging effect in the fixed-deficit diagnostic.

A.2 Attribution of the Gumbel difference

  • At Δ=0.2 and M=1000, alternative-to-null divergence is 1.028 nats for spike density (9) and 0.077 for density (7), while reverse divergence is 0.245 and 0.081 nats.Both divergences are larger for the spike member at these parameter values, but neither determines finite-horizon power.Source: §A.2 · p. 27
  • The comparison confounds prior averaging with component family: the Bayes rule averages over Π and evaluates spike density (9), unlike h★ gum,Δ, which uses least-favorable density (7).Table 4 separates fixed-Δ0 generating-spike diagnostics from tail-prior mixtures under an equal-tail generator.Source: §A.2 · p. 27, Table 4 · p. 27
  • No incremental difference attributable to prior averaging is detected in the fixed-Δ equal-tail diagnostic; at n=300 and 700, shared spike and Δ0=.01 rules make identical decisions.At n=100, discordance is 2 versus 4 (exact two-sided McNemar p=.69); this diagnostic fixes the deficit and assumes an equal tail.Source: §A.2 · p. 27
  • Under the persistent-deficit generator at n=700, shared-rule Type II error is 0.0014 versus 0.0042 for tokenwise rules, for both spike and Dirichlet specifications.The tokenwise mixture prior-averages each token’s evidence before multiplying.Source: §A.2 · p. 28
  • At n=700, posterior means are 0.006, 0.051, 0.201, and 0.401 for generating deficits 0.005, 0.05, 0.2, and 0.4; 90% interval coverage ranges from 0.843 to 0.925.The Δ=.05, n=700 cell is 3.8 standard errors below nominal coverage; the posterior summarizes the modeled deficit rather than a separate fit.Source: §A.2 · p. 28

A.3 Sensitivity to the mixing weight

The section studies the union-tail prior’s mixing weight and how document-level branch probabilities update. AUC is nearly constant across the evaluated interior weight grid, but excluding the tail-width branch reduces AUC, and the exact interior optimum is not identified.

A.3 Sensitivity to the mixing weight

  • The prior weight w sets mass on the Dirichlet shape block versus the tail-width block, and the document-level tail branch’s marginal likelihood is equivalent to fixing w at its hyperprior mean.This equivalence relies on drawing the branch once per document; redrawing it at every position generally makes the Bayes factor a degree-n polynomial in w.Source: §A.3 · p. 28, Table 5 · p. 29
  • Under an equal-tail generator at Δ=.1, the Dirichlet block’s mean posterior probability rises from .5 to .92 within 50 tokens and to .99 by 200 tokens.At Δ=.005, it is .44 at 50 tokens and .65 by 200; these are branch-learning summaries at w=.5, not regret comparisons across weights.Source: §A.3 · p. 28
  • The near-constant AUC range supports robustness to mixing weight on the evaluated grid, not to removing the width branch; hierarchical estimation of w across documents is outside the prespecified-prior analysis.The AUC comparison uses temperature-matched arms with 2,500 documents per arm at 100 tokens.Source: §A.3 · p. 28, Table 5 · p. 29
  • Across evaluated weights w=0,.1,…,.9, AUC varies by at most .0015 within any temperature-matched cell; at w=1, it decreases by .0094 to .0489 across cells.The AUC at w=1 is within .0076 of the shared-deficit equal-tail baseline, and no value between .9 and 1 was evaluated.Source: §A.3 · p. 28, Table 5 · p. 29

A.4.1 Empirical deficit regimes

The section characterizes empirical token deficits across sampling temperatures for two open models. Support coverage and deficit size vary nonmonotonically, and the measurements do not establish typical deployment conditions or bounds for other settings.

A.4.1 Empirical deficit regimes

  • At temperature .1, more than four fifths of OPT-1.3B tokens fall below the prior support; at temperature 1, close to three fifths lie above it.The reanalysis of archived benchmark data is concentrated in the small-deficit regime, while full-softmax sampling near temperature 1 produces larger deficits for these two models.Source: §A.4.1 · p. 29
  • For OPT-1.3B, the share of tokens within the prior’s support rises from .163 at temperature .1 to .580 at .5, then falls to .345 at 1.Between temperatures .3 and .4, median deficit rises from .0000 to .0546; by .7, the share above one half is .367 and the median exceeds .35.Source: §A.4.1 · p. 29, Table 6 · p. 30
  • These deficit observations cover two open models under the specified decoder and do not estimate typical deployment conditions or bound deficits for other models, decoders, or temperatures.Nucleus sampling can reduce Δ for a fixed pre-truncation distribution, but the amount was not measured in these experiments.Source: §A.4.1 · pp. 29–30

A.4.2 Deficits above one half

Widening the deficit prior leaves equal-tail results unchanged but changes width-based rules, with gains in most cells and a boundary where Δ no longer denotes a top-probability deficit. Null validity remains unaffected by this change in interpretation.

A.4.2 Deficits above one half

  • For the tail-width component, Δ≤1/2 keeps the designated coordinate largest at J=1; above that threshold it ceases to be a top-probability deficit, but null validity is unchanged.The component densities normalize for every Δ in (0, 1), and the exact null does not involve Δ.Source: §A.4.2 · p. 30
  • Equal-tail AUC and Type II error agree to every recorded digit across all eight cells under narrow and widened priors.The score difference is −log 2 with standard deviation zero to recorded precision; the shift preserves document rankings and calibrated decisions.Source: §A.4.2 · pp. 30–31
  • AUC increases by up to .0022 for width-based rules, and the widened union has higher AUC in seven of eight cells; h_ars has higher AUC in seven of eight cells.The prior-support sensitivity analysis reports no adjusted p-value.Source: Table 7 · p. 31, §A.4.2 · p. 31
  • At J=1, Δ above one half is a relabeling of a value below one half, while the profile estimates are J=1 for Sheared-LLaMA-2.7B and J=2 for OPT-1.3B.The J=1 predictive is unchanged by folding the widened symmetric prior; larger-J components are not symmetric, so the widened prior defines a different model.Source: §A.4.2 · pp. 30–31

A.5.1 Dirichlet tail calculations

The section compares Dirichlet tail models with a simulation-snapshot tail and accounts for token selection through size-biased coordinates. It reports slow numerical convergence to the spike limit and distinguishes unselected from selected tail variability.

A.5.1 Dirichlet tail calculations

  • At M=1000 and Δ=0.2, maximum absolute log-density differences from α=∞ are 0.634, 0.0842, and 0.00875 for α=10, 100, and 1000, respectively.The variance-matched symmetric Dirichlet member is approximately α=3; renormalized i.i.d. uniforms do not follow a Dirichlet distribution.Source: §A.5.1 · p. 32
  • Matching unselected variance does not match the size-biased law: the selected variances are 2/9 for Unif(0, 2) and 4/9 for Gamma(3, 3).The scaled-coordinate support endpoint 2 is a limiting boundary; at K=999, roughly 0.7% exceed it.Source: §A.5.1 · p. 32
  • One-token simulation uses a size-biased tail coordinate with a Beta(α+1, (K−1)α) distribution, independently over tokens conditional on α.The size-biasing identity also gives an independent proof of equation (10).Source: §A.5.1 · p. 32

A.5.2 Transform evaluation

This part evaluates the numerical implementation and explores detection performance across tested deficit and tail laws, temperature-matched analyses, and hierarchical models of within-document variation. Its findings are regime-specific and qualified by implementation, calibration, provenance, and conditional-null limitations.

A.5.2 Transform evaluation

  • The transform can be tabulated once per α, giving O(1) component lookups and O(G) cost per token for a joint grid once tables are available.Discretization preserves normalization when nodes are rescaled so that K times their weighted mean equals one; subsequent interpolation is an additional numerical approximation.Source: §A.5.2 · p. 32

A.5.3 Likelihood-ratio computation

  • The grid mixture update uses O(G) time and memory per token once component likelihood ratios are available, while generic Gumbel evaluation may cost O(MG_Δ).Reported finite-grid checks do not bound interpolation error uniformly; the 200,001-node inner table was used, while an interpolated numerator needs a certified mass bound or conservative renormalization to retain a Ville bound.Source: §A.5.3 · pp. 33–34

A.6 Numerical checks and interpretation

  • Across 69 non-diagnostic cells at the three tabulated horizons, realized evaluation-sample Type I error ranged from 0.0442 to 0.0564.The 96-node equal-tail and Dirichlet Gumbel cells had identical Type II errors across 48, 96, and 192 nodes; a separate 300-document-per-arm check found identical Type II errors across node counts in all 48 Gumbel cells, including union-tail cells, but neither check certifies a one-sided interpolation-mass bound.Source: §A.6 · pp. 34–35

A.6.1 Anytime monitoring

  • For the exact analytic densities, rejecting when sup_t≤700 B_t ≥ 20 has Type I error at most .05, but interpolation-based implementations lack a certified anytime guarantee.Their small observed normalization errors are not certified upper bounds; under shared Δ, the shared hierarchy had lower observed Type II error than the tokenwise hierarchy.Source: §A.6.1 · pp. 35–36, Table 8 · p. 36

A.7.1 Benchmark data of Li et al. and calibration

  • At n = 200, the union-tail rule’s Type II errors were .498 for OPT-1.3B and .528 for Sheared-LLaMA-2.7B, versus best reference-score errors of .510 and .524, respectively.Neither margin was statistically resolved; the archived watermarked and unwatermarked arms used different temperatures, so these results do not estimate matched-temperature power or establish a population ranking.Source: §A.7.1 · pp. 36–37, Table 9 · p. 37

A.7.2 Effective tail width and model fit

  • Profile likelihoods favored effective widths J = 2 for OPT-1.3B and J = 1 for Sheared-LLaMA-2.7B, but residual lack of fit remained.Bootstrap selection frequencies were 66% for J = 2 in OPT-1.3B and 94% for J = 1 in Sheared-LLaMA-2.7B; fitted-width goodness-of-fit p-values were descriptive, not calibrated, because width was selected on the same dependent observations.Source: §A.7.2 · p. 39

A.7.3 Replay checks and regeneration provenance

  • The replayed skipgram implementation seeds from the oldest token in a four-token window, so the fresh-pivot conditional-null assumption is not established for archived sequences.Both archived-benchmark replay and temperature-matched regeneration are therefore restricted to fixed-horizon claims; among six reference scores with archived records, the largest common-calibration versus archived Type II-rate difference was .060.Source: §A.7.3 · p. 40

A.7.3 Replay checks and regeneration provenance / A.9.1 Generation and calibration details

  • The archived temperature-matched arrays predate the run-record format and do not verify historical generation settings; at temperature .1, split-half calibration also makes single-split rankings unstable.The archived 500 prompts were reproduced token for token and 2,000 prompts extended the same population, but revised safeguards were not tested through a complete model-generation rerun and do not establish end-to-end reproducibility. The full unwatermarked arm is used to estimate prefix-specific cutoffs, with Type I error assessed by five-fold cross-fitting.Source: §A.7.3 · pp. 40–41, §A.9.1 · pp. 48–49

A.8.1 Numerical settings and boundary randomization

  • Boundary randomization affects only the discrete h_ind,1/e score in the reported evaluation: 274 documents fall on score atoms across 81 cells.Paired comparisons use one realized auxiliary uniform for each document and evaluation cell, shared across methods.Source: §A.8.1 · p. 42

A.8.2 Shared-deficit endpoint results

  • The equal-tail resimulation has Type II errors .0328, .0124, and .0062 at n = 100, 300, and 700, respectively.At score atoms, boundary rejection probability is chosen to attain 5% calibration size, and aggregate error rates average over that randomization; 274 documents fall on an atom across 81 cells, all for the discrete h_ind,1/e score.Source: §A.8.2 · p. 42, §A.8.1 · p. 42, Table 11 · p. 43

A.8.3 Regret across generating regimes

  • Across n = 100, 300, and 700, the tail-shape mixture’s maximum regrets are .0008, .0016, and .0002; it is smallest among ten displayed rules at n = 100 and 700, while equal-tail is smallest at n = 300.These rankings are exploratory: they select the best observed rules across six regimes, and no simultaneous interval for maximum regret is reported.Source: §A.8.3 · pp. 43, 45

A.8.4 Tail shape: regret across tail laws

  • Among fourteen displayed rules, the α mixture has the lowest observed maximum regret at all three horizons, tying the α0 = 1 and α0 = 0.1 point rules at each.At T4, its error is .0182 versus .0186 for the lowest-error point rule; at T3, the α0 = 1 point rule has error .0178 versus .0180 for the mixture. At T6, the mixture has three more misses than the equal-tail rule; these small differences are not statistically resolved.Source: §A.8.4 · p. 46, Table 13 · p. 47

A.8.5 Tokenwise-deficit equal-tail experiment

  • Under tokenwise deficits, every rule common to both experiments has lower observed Type II error than under shared deficits; several Gumbel rules have no misses by n = 100 among 5,000 alternatives.For zero misses, the exact one-sided 95% upper bound is 0.000599; the three tokenwise mixtures reject the same documents at all three recorded horizons. The comparison depends on whether the deficit is shared or redrawn tokenwise.Source: Figure 7 · p. 48, §A.8.5 · p. 48

A.9.2 Bootstrap reporting and sample-size comparisons

  • At 500 documents, estimated union-tail AUC exceeds shared-Δ in all eight cells, while estimated h_ars exceeds h★ gum,.1 in five of eight cells; these comparisons are descriptive.The stored cluster bootstrap covers only the 2500-document analysis, so no adjusted p-value is reported. The 1/2000 floor applies to empirical bootstrap-tail percentile probabilities, not reported normal-tail p-values; among 35 Holm-significant contrasts, 30 equal the floor, three equal .001, and two equal .002.Source: §A.9.2 · p. 49

A.10.1 Hierarchical model for token-specific deficits

  • The deficit hierarchy mixes conditional Beta distributions through document-level parameters, inducing marginal dependence and partial pooling while including independent-uniform and shared-deficit cases.The fixed-marginal Gaussian-copula experiment varies persistence without changing marginal deficits; an earlier Beta-generator experiment changed both marginal distribution and dependence, so its power differences cannot be attributed to persistence alone.Source: §A.10.1 · pp. 52–53
  • At most .0012, hierarchical and shared-deficit models differ in estimated Type II error across persistence levels and horizons, with no Holm-adjusted significant differences; the hierarchical model beats tokenwise by .0030 at P2, n=60.The nonsignificant comparisons do not establish equivalence, and the tokenwise model has higher estimated error than the shared model by up to .0036 under persistent deficits.Source: §A.10.1 · p. 52, Table 14 · p. 54
  • In all eight temperature-matched cells, the hierarchical deficit model has higher estimated AUC than shared deficit, by .0004–.0031 (mean .0015), descriptively.This contrast was outside the Holm family; its mean improvement is one to two orders of magnitude smaller than the union-tail model’s .0328 mean AUC improvement over the same baseline and documents.Source: §A.10.1 · p. 53, Table 15 · p. 54

A.10.2 Hierarchical model for token-specific tail widths

  • The width hierarchy mixes document-level distributions over token-specific widths while keeping the deficit common; its fixed-marginal Gaussian-copula design varies width persistence on a shared five-rung support.The hierarchical model includes document-shared and independent tokenwise widths; the shared-width model includes only the document-shared endpoint.Source: §A.10.2 · p. 55
  • By .0038 at most, hierarchical and shared-width models differ in Type II error; significant reductions occur only under independent-width Q5: .0038 at n=100 and .0024 at n=300.No difference is significant at n=700, and nonsignificant comparisons do not establish equivalence; the shared-width model is correctly specified at Q1.Source: §A.10.2 · p. 55, Table 16 · p. 56

A.11 Computational-materials index

  • The companion repository indexes numerical records and code for hierarchical deficit and width persistence sweeps, alongside supporting analyses and implementation materials.It distinguishes stored-result checks from regeneration and notes that external model checkpoints, tokenizers, C4 records, and reference implementation are not redistributed.Source: §A.11 · p. 57

Related papers

Loading 2609.15657v1…