PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Conformal Constructions for Adaptive Within-Document AI-Text Screening

Marco Mandap, Jerahmeel Hipolito, Arcel Galvez, Charlie Margaret Balagtas, Michael Joshua Buluran, Jeff Roel Durmiendo, Rizzette E. Lopez

arXiv:2609.31547v1stat.MEcs.CL

TL;DR

Adaptive within-document AI-text screening needs false-alert control when inspection prefixes and detectors are selected from observed evidence and the procedure may stop early. The paper gives two finite-sample conformal constructions under document-level exchangeability: one registers actions and allocates error, while the other calibrates a development-fixed route’s complete-path maximum. Both control marginal false alerts within their specified inspection scopes, but neither constructs an e-process or justifies multiplying conformal ranks.

  • Problem

    Selecting prefixes and detectors from observed evidence requires a false-alert bound for the whole inspection path, since a fixed-score guarantee does not cover choosing the most suspicious result.

    Source: §1 · p. 2

  • Method

    Two conformal rules use either a registered prefix–detector family with allocated error or a development-fixed route calibrated by its complete-path maximum.

    Source: §1.2 · pp. 2–3

  • Results

    Both constructions control the marginal probability of any false alert across their permitted inspection paths under document-level exchangeability, without assuming independent tokens.

    Source: §4 · p. 5, §5 · p. 7, §8 · p. 13

  • Takeaways & Limitations

    Construction A permits score-dependent action selection within its registered family, while Construction B permits early termination along its fixed route without splitting the error budget.

    Source: §1.2 · p. 3, §4 · p. 5, §5 · p. 7

    The rank proofs construct no e-process or conditional supermartingale property for products of transformed ranks, so they do not justify multiplying conformal ranks.

    Source: §6.5 · p. 12

Abstract

from arXiv · show

We study false-alert control when screening for text generated by artificial intelligence (AI). The screening procedure selects document prefixes and detectors from observed evidence and may stop before exhausting its inspection budget. We give two finite-sample constructions under document-level exchangeability between human calibration documents and a new null document, with no restriction on dependence among tokens within a document. Construction A registers a finite family of prefix-detector scores and allocates a false-alert budget across their conformal ranks. A union bound protects any executed subset of that family. Construction B calibrates the complete-path maximum of a development-fixed adaptive policy. Each partial-path maximum is bounded by the complete maximum, so a terminal conformal rank protects early stopping without splitting the error budget. We prove marginal control of any false alert across the permitted inspection path and derive necessary calibration counts for rejection. We also state oracle testing, distribution-shift, and independent-audit bounds with their additional assumptions. Both constructions protect stopping within their specified scope; neither proof constructs an e-process or justifies multiplying conformal ranks. Detection power and computational savings remain questions for empirical evaluation.

1 Introduction

Adaptive AI-text screening must control false alerts across the prefixes and detectors selected along a document’s inspection path. The paper positions its contribution as two restricted conformal constructions for that task, not a new general principle of sequential testing.

1 Introduction

  • Selecting the most suspicious result across several prefixes or detectors requires a path-wide false-alert bound; a guarantee for one fixed score is insufficient.The motivating examples include probability-curvature and contrasting-model scores.Source: §1 · p. 2

1.1 Positioning

  • Sequential testing and stopping-valid decisions from fixed-decision guarantees predate this work.The contribution is to specify two within-document detector-selection constructions and their routing and calibration-reuse restrictions.Source: §1.1 · p. 2

1.2 Contributions

  • The paper gives two finite-sample rules: Construction A fixes a registered action family and error allocations, while Construction B calibrates a development-fixed route’s complete-path maximum.Their guarantees permit selection within the registered family or early termination along the frozen route.Source: §1.2 · pp. 2–3
  • The paper derives calibration counts needed to permit rejection and states additional assumptions for oracle budgets, shift bounds, and independent audits; it reports no detector experiments or computational savings.It distinguishes rejection-grid feasibility from useful detection power and offline scoring cost.Source: §1.2 · p. 3

2 Framework

The framework treats each document as the testing unit and permits adaptive inspection that terminates within a finite action budget. Its guarantees concern marginal false-alert probability under document-level exchangeability, not calibration-conditional or multi-document control.

2 Framework

  • A policy selects an inspection extent and detector at each step, then reports an alert or no alert within a finite action budget; non-rejection does not certify human authorship.The framework distinguishes executed actions, distinct tokens inspected, and computational cost because detectors may rescore the same tokens.Source: §2 · p. 3
  • The target is marginal control under the joint null experiment, not control conditional on a realized calibration set, document length, subgroup, or selected route, nor across a collection of documents.The document is tested against a specified human-reference distribution; rejection does not identify an AI alternative because other human populations may differ from that reference.Source: §2 · p. 3
  • Conditional on development information, human calibration documents and a new null document must be exchangeable; score maps, preprocessing, action families, and weights are fixed using development information.Construction B additionally requires its complete-route generator to be fixed without held-out calibration documents or their summaries.Source: §2 · p. 3
  • The assumptions allow arbitrary dependence among tokens within a document, and randomized scores preserve exchangeability when documents are augmented with symmetrically applied independent seeds.The proofs condition on development information and average over it; they do not provide validity conditional on the realized calibration sample.Source: §2 · pp. 3–4

3 A conformal ranking lemma

The conformal ranking lemma gives a conservative upper-tail rank for an exchangeable test score. Its validity follows by comparing the implemented tie-conservative rank with a uniform tie-broken rank.

3 A conformal ranking lemma

  • An auxiliary uniform tie-break makes the test document’s descending rank uniform over {1, . . . , m + 1} conditional on development information.The calibration and test scores, augmented with independent uniforms, remain exchangeable and have distinct ranks with probability one.Source: §3 · p. 4
  • The implemented rank counts all calibration scores tied with the test score, so it is at least the tie-broken rank and is conservative.The auxiliary uniforms are used in the proof; the implemented rank is deterministic given the scores.Source: §3 · p. 4
  • If all scores tie, the conformal p-value is 1, so a constant score cannot trigger an alert at α < 1 regardless of calibration count.Source: §3 · p. 4

4 Construction A: a registered family with an error budget

Construction A controls false alerts by registering prefix–detector actions and allocating a shared error budget across their conformal ranks. Selection and ordering may depend on observed scores or ranks, but the family and allocations remain fixed.

4 Construction A: a registered family with an error budget

  • Each action scores a fixed token prefix with a fixed detector map, and an action failure assigned score −∞ yields p_a(X) = 1 and prevents an alert at that action.The same truncation and failure rules apply to calibration and test documents.Source: §4 · p. 4
  • Deployment may choose an executed subset and its order from observed scores or ranks, while alerting only when an executed action’s conformal p-value meets its allocated threshold.The scores are defined for all documents, including those where deployment does not execute the action.Source: §4 · p. 5
  • Any measurable, score-dependent selection and ordering of executed actions in the registered family satisfies false-alert control, even when action scores are dependent.The guarantee follows from a union bound over the fixed family, not from independence between actions or a rank bound conditional on execution.Source: §4 · p. 5
  • The action weights must remain fixed after observing scores; reallocating weight to the smallest observed p-value can exceed the target even when weights sum to one.For two independent uniform null p-values, the cited example’s error is 1 − (1 − α)^2 > α.Source: §4 · p. 5
  • An action with zero weight cannot reject, and positive-weight actions require enough calibration documents for the rank grid to reach their allocated threshold.The rank is at least 1/(m + 1).Source: §4 · p. 6
  • With twelve equally weighted actions, rejection requires m ≥ 1,199 at α = 0.01 or m ≥ 11,999 at α = 0.001; these counts do not ensure useful power.They permit rejection on the rank grid; treating dependent prefixes from one document as separate calibration units needs a different exchangeability argument.Source: §4 · p. 6

5 Construction B: calibrating the complete policy path

Construction B calibrates the complete-path maximum of a development-fixed route, so its conformal rank protects early stopping along that route. The guarantee depends on route and calibration restrictions, with calibration counts determining whether rejection is possible.

5 Construction B: calibrating the complete policy path

  • Calibration documents follow the complete route, while deployment may follow only an initial segment and leave unvisited actions uncomputed.The complete test route serves as a mathematical reference for the proof.Source: §5 · p. 6
  • Construction B controls false alerts under arbitrary early termination along a prefix of its fixed finite route.The route generator and transformations are fixed using development information, not held-out calibration documents or their summaries.Source: §5 · pp. 6–7
  • Each partial-path maximum is bounded by the complete maximum, allowing the terminal conformal rank to protect stopping without an optional-stopping theorem.The proof uses containment and exchangeability, not validity conditional on a realized calibration set.Source: §5 · p. 7
  • Calibration-informed policy tuning requires fresh independent calibration or another validity argument; a changed policy needs scores from its own complete routes.Recomputing on the same calibration documents is valid under this argument only if the new policy was chosen without those documents or their summaries.Source: §5 · p. 7
  • A failed action must not add alert evidence: the failure sentinel leaves an all-failure path with rank 1, not 1/100 in the stated example.A failure after a successful action leaves the running maximum unchanged.Source: §5 · pp. 6–7
  • Construction B implements an observable conservative lower bound on an induced sequential test, not its conditional expectation or a claim of equality or admissibility.The paper identifies the complete-path conformal implementation and routing restrictions—not a new induction principle—as its contribution.Source: §5 · p. 8
  • At α = 0.01 and 0.001, rejection requires 99 and 999 calibration documents, respectively; these rank-resolution counts establish neither greater power nor lower offline cost.Construction B requires complete calibration routes, and a broader path can raise its calibrated maximum.Source: §5 · p. 8

6 Supporting bounds

The supporting bounds quantify oracle and independence-based sample requirements, characterize the scope of distribution-shift and audit guarantees, and distinguish pathwise stopping control from e-process guarantees.

6.1 An oracle total-variation identity

  • The oracle testing bound applies to two document laws, while independence matters when those laws are represented as product laws—not for treating tokens in one document as independent replicates.A learned detector may also discard information used by the oracle rule.Source: §6.1 · pp. 8–9

6.2 A Hoeffding budget under independence

  • Each error probability is at most exp(−n∆^2/2) for independent bounded scores with known null and alternative means separated by ∆.Identical score distributions are unnecessary; applying this budget to learned conformal rules requires accounting for mean estimation, score selection, and repeated inspection.Source: §6.2 · p. 9

6.3 Distribution shift

  • The shift bound averages over calibration under an added product-law independence assumption; it is not calibration-conditional, and numerical certification needs a justified upper bound on total-variation distance.Document-level exchangeability alone does not imply independence between the new document and calibration set.Source: §6.3 · p. 10

6.4 A zero-event audit bound

  • A zero-alert Clopper–Pearson audit reports 2,995 as the smallest qualifying size, under a fixed-size audit of independent, identically distributed human documents.Its coverage is over repeated audits of fixed size, not a posterior probability or a guarantee for outcome-dependent stopping; shared authorship or prompt lineage can violate the binomial model.Source: §6.4 · p. 11

6.5 Repeated looks, stopping control, and e-processes

  • Theorems 4.1 and 5.1 provide simultaneous control across permitted inspections, protecting data-dependent stopping within the registered family or frozen route.The scope remains one document and the specified action family or finite route.Source: §6.5 · p. 11
  • The rank proofs do not construct an e-process or justify multiplying transformed ranks; marginal rank validity alone does not license those products.A valid p-value can have an infinite reciprocal expectation, and dependence between successive factors requires a separate sequential argument.Source: §6.5 · p. 12

Distribution shift.

The discussion bounds the guarantees to their stated populations and decision scope, identifying additional requirements for subgroup, adversarial, and deployment-wide claims. It also distinguishes alerts from authorship conclusions and leaves power and computational cost for empirical evaluation.

Distribution shift

  • Subgroup protection requires exchangeability and sufficient calibration documents within each stratum, plus authorized access to stratification attributes.Pooled performance alone may miss detector bias against non-native English writing.Source: Distribution shift · p. 12
  • Selecting a human document after detector queries can invalidate null exchangeability, while mixed authorship requires a specified target population and label definition.Paraphrasing generated text is a separate power issue, and the binary reference test does not estimate the extent of AI assistance.Source: Distribution shift · p. 12
  • Screening many documents or rerunning one under several policies requires additional multiplicity accounting for deployment-wide error control.The within-document theorems do not provide that accounting.Source: Distribution shift · p. 13
  • Document-level calibration does not validate an uncalibrated word-level test; a document-level discrepancy needs its own interpretation and calibration.The cited Kolmogorov–Smirnov example assumes independent continuous samples, while repeated words create ties.Source: Distribution shift · p. 13
  • An alert indicates departure from the human reference, not a posterior probability of AI authorship; valid scores may still be powerless.Empirical comparisons should measure detection power and computational cost at matched error targets, including calibration work and abstentions.Source: Distribution shift · p. 13

8 Conclusion

The paper presents two finite-sample conformal rules for adaptive within-document inspection, with stopping-valid false-alert control under document-level exchangeability and stated development restrictions. Their calibration counts establish rank feasibility, not useful power; empirical power and computational savings remain unevaluated.

8 Conclusion

  • Construction A permits score-dependent selection from a registered prefix–detector family with fixed error allocations, while Construction B permits early termination along a development-fixed route using its calibrated complete-path maximum.Construction A uses a union bound; Construction B does not split the error budget.Source: §8 · p. 13
  • Both rules control the marginal probability of any false alert under document-level exchangeability and their stated development restrictions, without assuming independent tokens.Source: §8 · p. 13
  • The calibration counts establish rank feasibility, not sufficient data for useful power, and no empirical evaluation establishes detection power or computational savings.Source: §8 · p. 13
  • Neither stopping proof constructs an e-process, although both provide stopping-valid decisions within their prescribed scope.Source: §8 · p. 13
  • Oracle, distribution-shift, and audit results require additional assumptions stated with each result.Source: §8 · p. 13

Statements and Declarations

The manuscript reports analytic results without empirical evaluation and provides declarations on funding, authorship, ethics, data and code availability, and AI assistance.

Statements and Declarations

  • The manuscript presents analytic results and no empirical evaluation, applying established ranking, union-bound, oracle, concentration, and audit tools to the specified screening constructions.It does not claim new foundational theory or an exhaustive novelty review.Source: Statements and Declarations · p. 13
  • No funding was received, and the authors must confirm relevant financial and non-financial interests before submission.Source: Statements and Declarations · p. 13
  • The authors assign CRediT roles, name Marco Mandap as primary author, and state that all authors reviewed and approved the final manuscript.Source: Statements and Declarations · p. 14
  • The article reports no human- or animal-participant studies or participant data, while noting that institutional requirements must be confirmed.Source: Statements and Declarations · p. 14
  • The theoretical study generated or analyzed no datasets, and the Project Burnwatch prototype is available in a versioned repository.Source: Statements and Declarations · p. 14
  • AI tools assisted with literature synthesis, drafting, editing, and mathematical review; the authors are responsible for verifying the manuscript's claims, proofs, references, and final text.Source: Statements and Declarations · p. 14
Loading 2609.31547v1…