PaperlayerView exact revision on arXiv

Source-linked AI summary

Token-Level Likelihood-Array Regression for Membership Inference and AI-Generated Text Detection

Jiajun Sun, Zhanrui Cai

arXiv:2608.22179v1stat.MLcs.LGstat.ME

TL;DR

The paper addresses how likelihood information should be used for membership inference and AI-generated text detection. It proposes likelihood-array regression and reports improved detection performance over existing likelihood-based procedures, with additional membership-inference gains from second-order augmentation.

  • Problem

    The paper examines the broader statistical question of how likelihood information should be used for detection, a question central to privacy and copyright audits.

  • Method

    Likelihood-array regression organizes token-level likelihood-derived features across aligned context lengths and token positions for regression-based detection.

  • Results

    0.914–0.983 for membership inference and 0.985–0.996 for AI-generated text detection, substantially outperforming existing likelihood-based procedures.

  • Takeaways & Limitations

    Second-order augmentation substantially improves membership inference, while shorter-context likelihoods provide information beyond conventional full-context probabilities.

  • Takeaways & Limitations

    Threshold-based decisions require calibration beyond ranking performance.

Abstract

from arXiv · show

Membership inference asks whether a text was used to train a language model, whereas AI-generated text detection asks whether it was generated by a language model rather than written by a human. Existing likelihood-based methods typically compress token-level probabilities into a few prespecified scores, most often using only probabilities conditioned on the full preceding context. We propose likelihood-array regression (LAR), which evaluates each target token under nested left-context windows and organizes the resulting likelihood-derived features into a structured array. After aligning arrays across texts of different lengths, LAR learns how detection information varies with context scale, token position, and likelihood features. LAR-1 aggregates learned contributions from individual aligned cells, while LAR-2 adds second-order features formed from pairs of evaluations of the same target token across context lengths. For within-path quadratic model, we establish matching minimax lower and upper bounds, characterize errors from finite-dimensional approximation and random squared projections, and derive conditions under which an oracle spectral sieve attains the minimax rate. Across multiple scoring language models, LAR substantially improves membership inference and AI-generated text detection over likelihood-based baselines. The analyses further show that shorter-context likelihoods contain information beyond conventional full-context probabilities, while second-order features provide additional gains for membership inference.

1 Introduction

The paper addresses membership inference and AI-generated text detection by modeling token-level likelihood information across context lengths and token positions. It proposes likelihood-array regression, which improves detection over likelihood-based baselines and provides theoretical guarantees for its quadratic form.

  • Membership inference tests whether text appeared in a model’s training data, while AI-generated text detection tests whether text was human-written or AI-generated.
  • Existing likelihood-based methods generally compress token-level probabilities into a few prespecified scores, often from full preceding context.
  • LAR evaluates each target token under nested left-context windows, aligns variable-length arrays, and models detection information by context length, token position, and likelihood-derived features.Each array cell is indexed by preceding-context amount and relative document position, with features derived from the scoring model’s predictive distribution.
  • LAR-1 aggregates learned contributions from aligned cells, whereas LAR-2 adds within-path second-order features pairing evaluations of the same target token across context lengths.The second-order terms include same-context quadratic terms and products across different context lengths.
  • Across five scoring models per application, mean held-out AUCs are 0.914–0.983 for membership inference and 0.985–0.996 for AI-generated text detection.These results substantially outperform existing likelihood-based procedures.
  • Shorter-context likelihoods contain information beyond full-context likelihoods, while within-path second-order features provide additional gains primarily for membership inference.
  • For the within-path quadratic model, the paper establishes matching minimax bounds and conditions under which an oracle spectral sieve attains the minimax rate.The analysis also decomposes finite-dimensional prediction error and connects prediction error to AUC convergence.

2 Related Work

Prior detectors compress token-level likelihood information into a few scores, while related regression methods assume fixed-dimensional arrays or single trajectories. LAR instead preserves aligned, multidimensional likelihood structure across context scales and token positions.

  • Existing membership-inference methods mainly differ in how they calibrate and aggregate token-level likelihood information.
  • Other approaches vary conditioning contexts or fine-tune audited models using known nonmembers before recomputing likelihood scores.
  • Likelihood-based AI-text detectors typically reduce token probabilities, ranks, or related signals to one or a few scalar scores.
  • Learned detectors can achieve strong classification performance, but adapting them to new generators or application domains may require substantial labeled data and further fine-tuning.
  • LAR-1 and LAR-2 distinguish first- and second-order effects, but their covariates consist of collections of target-token paths rather than a single trajectory.
  • Likelihood-array regression addresses structures unlike standard functional or fixed-dimensional array regression: variable-size triangular arrays require cross-text alignment while retaining target-token paths.

3 Likelihood-Array Regression

LAR converts nested-context token likelihood evaluations into aligned multichannel arrays and models their contributions to class log odds. LAR-1 uses cellwise effects, while LAR-2 additionally models within-path interactions between evaluations of the same target token.

  • 3 Likelihood-Array Regression: Each text is an independent sampling unit labeled for membership inference or AI-generated text detection, with a fixed scoring model supplying conditional token probabilities.
  • 3.1 Likelihood arrays and alignment: The construction evaluates every target token under nested left-context windows, aligns variable-size likelihood arrays, and attaches likelihood-derived features to aligned cells.
  • 3.1 Likelihood arrays and alignment: A likelihood array is triangular because target token t has t−1 context evaluations, yielding T(T −1)/2 cells for a document of length T.
  • 3.1 Likelihood arrays and alignment: Normalized coordinates map cells to a common context-scale and target-position domain, with u = 1 representing all preceding tokens and v recording relative token position.
  • 3.2 Likelihood-derived cell features: The five channels describe observed-token predictability, context contribution, standardized versions of both, and local predictability change as context increases.
  • 3.3 First- and second-order likelihood-array models: LAR-1 models how aligned cell features and locations contribute to class log odds, averaging across context scales and then target-token paths.
  • 3.3 First- and second-order likelihood-array models: LAR-2 augments cellwise contributions with second-order features formed from pairs of evaluations within each target-token path.
  • 3.3 First- and second-order likelihood-array models: The shared cellwise function allows likelihood-pattern contributions to vary jointly with context scale, document position, and channel values without requiring linear channel effects.

4 Theoretical Analysis

The theoretical analysis formulates LAR features in Hilbert spaces and studies prediction for first- and within-path second-order logistic models. Under boundedness and spectral-decay assumptions, the quadratic model attains a minimax rate whose logarithmic factor is unavoidable, with finite-sieve and AUC convergence results extending the analysis to practical estimators.

  • Finite-dimensional approximation and AUC: Finite path sieves introduce approximation and projection errors, while under the stated choices expected prediction error is of order {log(en)/n}^ν/(ν+1), and AUC converges under an additional condition.The theoretical radius constraint is not imposed in the numerical implementation, and squared prediction loss is equivalent to excess logistic risk up to fixed constants.
  • Minimax rate: The quadratic model is governed by ordered pairwise products of covariance eigenvalues, producing a logarithmic factor because the number of index pairs grows as m log(em).This within-path quadratic spectrum differs from the single covariance sequence governing functional linear regression.
  • Population formulation: The population logistic model combines first-order path effects with position-varying quadratic effects, using a fixed feature map and a covariance operator for the combined features.The Hilbert-space formulation treats the aligned feature map as fixed and models conditional log odds through inner products with first- and second-order coefficient functions.
  • Population formulation: LAR represents text-level features by position-sensitive averages of token-path features and within-path quadratic products, preserving pairwise interactions from the same target token.The first-order feature averages path features, whereas the second-order feature forms a quadratic feature within each target-token path before position-sensitive averaging.
  • Assumptions: Bounded feature maps control empirical logistic risk, while spectral decay constrains feature complexity and permits dependence among different target-token paths within a text.The spectral assumption yields bounds for first-order and symmetric-tensor directions; independence across paths is not required.
  • Minimax rate: Theorems 1 and 2 establish matching minimax lower and upper bounds for within-path ridge estimation, so the logarithmic factor cannot be removed uniformly over the model class.The lower bound applies to all estimators, while the upper bound is attained by ridge estimation using complete Hilbert-space features.

5 Numerical Studies

Numerical studies evaluate LAR across membership inference and AI-generated text detection, examining overall performance, shorter-context information, second-order augmentation, and labeled-sample sensitivity. LAR outperforms comparison procedures across scoring models, with shorter-context cells contributing in both tasks and second-order features helping mainly membership inference.

  • Study design: The experiments compare LAR with existing procedures across scoring models and applications, while also testing shorter-context information, second-order augmentation, labeled sample size, and numerical resolution.Primary comparisons use held-out-text AUC with repeated 80/20 train–test splits and refitting within each split.
  • WikiMIA membership inference: Membership-inference evaluation uses 1,650 WikiMIA texts, comprising 861 members and 789 nonmembers across four text-length groups.The groups contain segments of 32, 64, 128, and 256 words.
  • WikiMIA membership inference: Both LAR estimators obtain higher mean AUCs than every comparison procedure across all five scoring models, and LAR-2 improves on LAR-1 by 0.023–0.039.Among comparison procedures, ReCaLL has the largest mean AUC for each scoring model, with values between 0.765 and 0.871.
  • AI-generated text detection: For AI-generated text detection, LAR-1 AUCs range from 0.985 to 0.996 and LAR-2 differs from LAR-1 by at most 0.005, exceeding every comparison procedure for each scoring model.The benchmark contains 1,200 texts from four domains, including matched human and GPT-4o passages.
  • Sources of detection information: Shorter-context cells outperform full-context cells: LAR-1 reaches AUCs of 0.907 and 0.988 versus 0.802 and 0.937, while LAR-2 reaches 0.952 and 0.989 versus 0.893 and 0.945.The shorter-context results are close to those from the complete array, providing evidence that shorter-context information contributes in both applications.
  • Sources of detection information: First-order coefficient contrasts vary across context scale and text position, change sign across the array, and differ between the two detection tasks.Positive and negative regional contributions can partly cancel when likelihood information is reduced to a global summary.
  • Effect of sample size: As labeled sample size increases, LAR-2 increasingly improves over LAR-1 for WikiMIA, whereas the two estimators remain nearly indistinguishable for generated-text detection.With 100 labeled texts, WikiMIA median AUCs are 0.638, 0.684, and 0.714 for the full-context-only model, LAR-1, and LAR-2; generated-text values are 0.860, 0.956, and 0.955.

6 Discussion

The discussion highlights LAR’s use of nested context windows and structured likelihood features, its empirical gains, theoretical guarantees, and practical trade-offs.

  • LAR evaluates target tokens under nested context windows, preserving variation across context scale, position, and likelihood-derived channels.
  • Shorter-context likelihood cells provide additional ranking information for both membership inference and AI-generated text detection.
  • Within-path second-order augmentation substantially improves membership inference, while first-order features receive little additional information from it in another setting.
  • Under product-form covariance decay, the within-path quadratic model has matching minimax prediction bounds and characterized finite-sieve and random-projection approximation costs.
  • Practical deployment requires calibration for threshold decisions, and likelihood-array construction is more computationally demanding than full-context scoring.
  • Adaptive context-window selection may reduce computational cost, while combining scoring models could help localize membership or generated-text evidence.

Data availability statement

The study’s supporting data are publicly available through the WikiMIA and AdaDetectGPT repositories.

  • The WikiMIA benchmark is available from its Hugging Face repository.
  • The AI-generated text detection data are available from the AdaDetectGPT repository.

Supplementary Material for Token-Level Likelihood-Array Regression for

The supplementary material accompanies the paper on token-level likelihood-array regression.

  • The supplementary material is titled “Supplementary Material for Token-Level Likelihood-Array Regression for Membership Inference and AI-Generated.”

S1 Proofs

The supplementary proofs formalize the functional feature model, covariance and spectrum conditions, effective-dimension bounds, and minimax lower-bound construction.

  • The proof model represents complete likelihood-array data through Hilbert-space path features and target-position basis functions.
  • Labels follow a bounded conditional logistic model based on centered first- and second-order feature representations.
  • Bounded feature assumptions control path and text-level norms independently of text length, without requiring within-text path independence.
  • The covariance analysis includes cross-covariance between first- and second-order feature blocks under specified operator conditions.
  • The ordered product spectrum and effective dimension are bounded using eigenvalue decay and logarithmic factors.
  • The minimax lower-bound proof constructs a finite hypercube inside the model class and applies Assouad’s argument.

S1.4 Proof of Theorem 2

The proof establishes Theorem 2 by combining logistic-loss curvature, localized empirical-process control, and covariance-based effective-dimension bounds. These steps yield the stated estimation rate after controlling approximation and penalty terms.

  • Curvature and risk equivalence: Bounded predictors place the logistic curvature uniformly between positive constants, making excess risk equivalent to squared prediction error.Taylor expansion and centered features also eliminate the intercept–slope cross term.
  • Empirical-process control: The logistic loss has a one-Lipschitz derivative, enabling Rademacher contraction for localized empirical-process bounds.The contraction applies to scalar increments of the form δ + ⟨h, Z⟩.
  • Rate derivation: The proof starts from the estimator’s basic inequality, bounds the penalty difference, and localizes the empirical term using Lemma S6.The localization uses a covariance norm and ridge regularization before substitution of the covariance bound.
  • Rate derivation: The effective-dimension argument absorbs the additional parametric term 1/n for 0 < λ ≤ 1, leaving the theorem’s stated rate.This follows because the right side of the covariance bound remains at least a positive constant.

S1.5 Proof of Theorem 3

The proof compares complete within-path quadratic features with their finite squared-projection representation. It shows that random projections preserve the relevant approximation and covariance complexity, so the finite estimator attains the established minimax rate.

  • Feature construction: Theorem 3 compares complete d-dimensional within-path quadratic features with the finite squared projections used by LAR-2.The construction begins with first-order sieve features and target-token path vectors, then forms quadratic coefficient features.
  • Projection representation: The squared-projection average is unbiased for the quadratic functional after applying the spherical fourth-moment correction.The operator G_d corrects the fourth moment uniformly in d.
  • Projection error: Dimension-free fourth-moment bounds control the variance of the squared-projection estimator.The argument uses Gaussian quadratic-form identities, trace inequalities, and independence across random directions.
  • Effective dimension: The random projection does not replace covariance-based complexity with the nominal number d_vR of projected coordinates.Effective dimension is controlled through operator comparison, rank-d_v interlacing, and monotonicity.
  • Minimax rate: The regularization term satisfies λ ≍ r_n, and the fixed-position term d_v/n ≲ r_n, so every term is at most order r_n.Thus the finite path sieve and squared-projection estimator attain the minimax rate established by Theorems 1 and 2.

S1.6 Proof of Proposition 1

The proof analyzes AUC through pairwise rankings and bounds the loss from an estimated score by its ranking-disagreement event. A margin condition then converts score error into an AUC rate.

  • Pairwise AUC: AUC admits a pairwise representation involving class probabilities and the estimated ordering of two observations.The optimal ranking orders observations by q, equivalently by the strictly increasing log odds η_0.
  • Ranking disagreement: The AUC gap is bounded by the expected probability difference on pairs whose estimated and optimal rankings disagree.The bound includes the factor 1/[2P(Y = 1)P(Y = 0)].
  • Ranking disagreement: Score deviations control ranking disagreement because an incorrect ordering implies that the probability gap is no larger than the two score errors combined.The proof handles both possible order reversals symmetrically.
  • Margin conversion: Splitting pairs by probability gap and applying the margin assumption yields the exponent (τ + 1)/(τ + 2) for the resulting error bound.The choice t = (ε^2)^(1/(τ+2)) balances the two terms.

S1.7 Additional theoretical remarks

The additional remarks clarify which proof ingredients are problem-specific, state assumptions needed for uniform curvature, and reformulate LAR features using empirical measures. They also distinguish LAR’s shared cellwise structure from conventional full-context scores.

  • Theoretical scope: The problem-specific analysis covers dependent token paths, quadratic-product spectra, a path-level lower submodel, and squared projections.The Rademacher construction is used only for a least-favorable submodel, not assumed for observed likelihood arrays.
  • Approximation and assumptions: Theorem 3 retains finite-basis approximation error explicitly for the spline and random-channel bases used numerically.The spectral-sieve approximation condition is stated as d^−ν ≤ C d^−ν.
  • Approximation and assumptions: Uniform boundedness of projected predictors supplies uniform logistic curvature but does not establish that λ = 0.01 is asymptotically optimal.The condition is stated separately because normalized directions alone do not imply pointwise boundedness.
  • Empirical-measure formulation: Equivalent empirical-measure formulations preserve context scale, target position, and channel values for LAR-1, while retaining target-token grouping for LAR-2.Both formulations are probability measures with equal mass across aligned scales and target tokens.
  • Relation to conventional scores: Conventional Min-K% and Min-K%++ scores use document-specific quantiles of full-context empirical distributions rather than LAR-1’s shared cellwise function.The comparison is made for the likelihood and standardized-likelihood channels.

S2 Additional numerical results

Additional analyses show that shorter-context likelihoods improve detection under matched supervision, LAR remains robust across specifications and generators, and quadratic features capture dependence-only signals. Cross-length performance is asymmetric, indicating that text-length generalization depends on training direction.

  • Common supervision: Supervised summaries improve on many fixed likelihood scores, while full-context LAR improves further through flexible modeling of full-context channel distributions.Under matched labels and fold assignments, complete-array LAR-1 outperforms full-context LAR-1 across all ten scoring models; LAR-2 shows the same pattern in WikiMIA and four of five generated-text settings.
  • Generator robustness: Across both alternative generators, LAR-1 and LAR-2 achieve higher mean test AUCs than comparison procedures for every scoring model.AUC ranges are 0.978–0.996 for Claude-3.5-Haiku and 0.979–0.992 for Gemini-2.5-Flash.
  • Numerical sensitivity: 0.947–0.958 for WikiMIA and 0.988–0.990 for AI-generated text detection across five random-feature seeds, while LAR-2 remains above the strongest comparison procedure across displayed settings.AUC changes little with ridge penalty and channel-basis dimension; position-basis dimension and squared-projection count cause somewhat greater WikiMIA variation.
  • Simulation: LAR-1 recovers marginal distribution changes and remains effective when the number of target positions varies.The simulation uses aligned 16 × 24 grids with five Gaussian channels and tests marginal mean, marginal scale, dependence-only, and variable-length settings.
  • Simulation: LAR-2 recovers dependence-only class differences that LAR-1 misses, with AUC increasing from approximately 0.53 to 0.70 as association strength grows.The dependence-only classes share the same first-order marginal distributions, and the null control has mean AUC 0.49 for both models.
  • Generalization: Cross-length AUC is 0.979 when shorter groups train on longer groups, but 0.836 in the reverse direction; the 256-token within-group AUC of 1.000 is descriptive.Leave-one-domain-out AUC differs from within-domain AUC by 0.002–0.021, and the 256-token group contains only 82 texts.
Loading 2608.22179v1…