Policy-Conditioned AI-Use Detection: An Evidentiary Framework for Academic Publishing

1 Introduction

• Venue AI rules differ over roles, permitted tasks, disclosure, confidentiality, and responsibility, so there is no single prohibition to enforce.For example, routine language editing needs no disclosure at NeurIPS but must be declared at ICLR.Source: §1 · p. 1
• Origin classification can invert policy judgments: grammar-corrected reviews may look suspicious, while a human-written review based on a confidential chatbot upload may violate policy.Large-scale evidence of model-modified review text or score effects does not establish whether a review broke a rule.Source: §1 · pp. 1–2
• When violations are rare, a strong detector can still flag more compliant authors than violators, while its score alone gives the accused no testable claim or calibration context.The consequences can include rejection, investigation, retraction, or a record notation; detector performance can also degrade under paraphrase or domain shift and penalize non-native English writers.Source: §1 · p. 2
• The proposed target is whether evidence about a workflow fits permitted or prohibited use under the role- and task-specific policy, not whether AI appears in an artifact.The paper frames this as a three-part framework: policy-conditioned targets, inferential reports of hypotheses and uncertainty, and pipeline benchmarks reporting power at a venue-fixed error rate.Source: §1 · p. 2

2 A Framework for Policy-Conditioned AI-Use Detection

• Policy specifies permitted and prohibited AI use for a domain, role, and task, making compliance a relation among workflow, role, and rule rather than a property of the final artifact.For peer review, grammar correction is permitted in the example, while delegating the review and recommendation is prohibited.Source: §2 · p. 3
• Inference reports evidence about whether a workflow violated policy, stating hypotheses and uncertainty rather than converting a score into an accusation.Frequentist reporting can state the null, statistic, calibration regime, and p-value; Bayesian reporting can state the prior, evidence, and posterior probability.Source: §2 · p. 3
• Benchmarks should generate multiple review workflows from shared manuscripts and measure TPR at a fixed false-positive rate under permitted workflows.They can vary evidence access, such as artifact-only versus artifact plus notes and logs; AUROC alone does not name either the tolerated false-accusation rate or detection rate at that tolerance.Source: §2 · p. 3
• The research agenda calls for role-specific policy languages, detectors combining artifact signals with provenance and workflow evidence, and benchmarks built from reproducible human–AI pipelines.The framework is intended to assess the procedure's errors under stated uncertainty, not to create a universal AI-versus-human classifier.Source: §2 · p. 3

3.1 Venue policy is already role- and task-specific

• Venue rules differ in how they constrain authors, reviewers, disclosure, confidentiality, delegation, and responsibility, making permitted use role- and task-specific.The same AI participation can be allowed, required, irrelevant, or prohibited across contexts.Source: §3.1 · p. 3

3.2 Appropriate use is a relation, not a property

• Appropriate use depends on the artifact, its producing workflow, and the governing policy; the same generation can be allowed in one setting and disallowed in another.For example, agentic systems may generate pull requests through approved software workflows, while similar generation would be disqualifying in a written examination.Source: §3.2 · p. 4

3.3 Formalization

• The detector takes the artifact, domain, task, actor role, applicable policy, and evidence beyond the artifact as inputs, and produces a structured report rather than a binary verdict.The report may include a compliance assessment, violation probability, uncertainty, evidence used or missing, calibration, validity conditions, and an action such as abstention.Source: §3.3 · p. 4

3.4 Policies must become machine-readable

• When provenance, disclosures, edit histories, or approved-tool logs sufficiently establish an action, the task can become verification rather than statistical reconstruction.Machine-readable policy specifications should identify role-specific tasks, allowed and prohibited workflows, disclosure, collected evidence, tolerated error, and graduated consequences.Source: §3.3 · p. 4, §3.4 · p. 5

4.1 From detector scores to statistical claims

• TPR at a fixed FPR names both the tolerated false-accusation rate under permitted workflows and the detection rate at that tolerance.A score or AUROC alone does not tell a chair what should trigger an integrity inquiry.Source: §4.1 · p. 5

4.2 Hypothesis testing

• The policy-conditioned test compares a permitted workflow under H0 with a prohibited workflow under H1 for the applicable policy.Type I error is rejecting a permitted workflow; its level bounds the chance that a compliant author or reviewer is flagged.Source: §4.2 · p. 5

4.3 Bayesian inference

• Bayesian concern depends on likelihoods and priors, so identical evidence can mean different things across policies, tasks, and violation base rates.When evidence is weak, abstention can be preferable to guessing, especially for short artifacts with no workflow trace.Source: §4.3 · pp. 5–6
• Reports should state hypotheses, evidence, calibration scope, and uncertainty rather than presenting a score as proof of misconduct.Frequentist reports specify the null, statistic, calibration regime, and p-value; Bayesian reports give prior assumptions, evidence used or missing, and posterior probability.Source: §4.2 · p. 5, §4.3 · p. 6

5.1 Current benchmarks and their limits

• Existing static and transformation benchmarks test text-origin or editing labels, but rarely specify institutional setting, role, available evidence, or error consequences.Static corpora age as models and editing habits change, and transformation-based labels still do not identify the compliance question a chair faces.Source: §5.1 · p. 6

5.2 Pipeline benchmarks

• A pipeline benchmark starts from a human seed, applies a controlled AI-use recipe, and labels the resulting workflow against a stated policy.Recipes can be regenerated for new models or relabeled when venue policies change; benchmark cases should also preserve policy-undefined workflows rather than forcing them into compliant or prohibited labels.Source: §5.2 · p. 6
• Benchmarks should vary evidence from artifact-only access through disclosure, provenance records, and substantially observed workflows to measure what each channel adds.Uncertainty about the workflow and the inferential burden fall across these evidence regimes.Source: §5.2 · p. 7

5.3 Evaluation criteria

• Evaluation should report TPR at a fixed FPR per workflow and state the assumed base rate, separating detection power from false accusations against permitted use.A detector with valid error control but negligible power is safe but useless.Source: §5.3 · p. 7
• At a 1% violation rate, TPR = 0.80 and FPR = 0.01 flags 160 violators but 198 compliant authors; at FPR = 0.05, 990 compliant submissions are flagged.AUROC is unchanged across these settings, while the share of accused authors who are innocent changes substantially.Source: §5.3 · p. 7
• Deployment evaluation must also cover adversarial evasion, calibration, abstention, generalization, adjudicator usefulness, and the privacy cost of required workflow evidence.A benchmark limited to non-adversarial TPR may overstate deployment performance; collecting invasive workflow traces can itself create a governance failure.Source: §5.3 · p. 7

• A structured declaration naming task, tool, stage, and verifier lets a venue check an account of the workflow against the artifact and other evidence.Declarations are less useful if authors expect disclosure itself to invite scrutiny.Source: §6 · pp. 7–8
• Approved-tool routing can create a log of policy-relevant actions without surveilling reviewers’ own machines, though off-platform tool use remains unobserved.A trusted log can support verification, while unobserved workflow elements still require inference.Source: §6 · p. 8
• Reports should enter a graduated, contestable process, with rising evidentiary standards and no detector report sufficient by itself for the final sanction.The accused should receive the hypotheses and calibration information needed to answer the finding.Source: §6 · pp. 7–8

• Origin detectors produce scores about text origin or model-likeness, not direct evidence that a workflow complied with its governing policy.The literature includes signal-based methods, labeled classifiers, and robustness studies.Source: §7 · p. 8
• Graded editing and provenance methods are closer to real workflows, but evidence of model contribution does not settle whether use was allowed, disclosed, or verified.Source: §7 · p. 8
• Review-side studies measure prevalence, effects, and model-generated feedback [Liang et al., 2024a, Latona et al., 2024, Liang et al., 2024b], but do not supply a compliance estimand.Detector degradation under paraphrase and domain shift, and bias against non-native English writers, motivate calibrated evidence, abstention, and human review.Source: §7 · pp. 8–9

Objections and limitations

• Origin detection remains useful where the governing policy is simply “no AI-generated content,” but covers a diminishing share of institutional decisions as AI assistance becomes routine.The worked example is text-centric and does not address code, figures, data, or multi-agent research pipelines; its arithmetic also assumes uniform detection rather than venue triage.Source: §8 · p. 9
• Policy-conditioned inference faces ambiguous rules, partly observed workflows, concealed tool use, and evidence that is noisy, privacy-sensitive, and manipulable.Improving origin-classification accuracy does not resolve this mismatch when compliance depends on institutional rules and workflow evidence.Source: §8 · p. 9
• Workflow datasets require costly, sensitive traces, and research-acceptable evidence may differ from what venues can collect in deployment.Labels are partly interpretive: venues may classify the same workflow differently, and cases between editing and drafting may need graded labels.Source: §8 · p. 9
• The analysis does not model the feedback loop in which reviews and accepted papers enter training data and detection-shaped text influences later models.The example also does not address code, figures, data, or multi-agent pipelines, and assumes a uniform detector despite venue triage.Source: §8 · p. 9

Conclusion

• Detection should assess whether a workflow complies with the governing rule, using artifact signals, disclosures, provenance, and workflow traces as evidence of varying strength.Latent workflows call for calibrated inference; trusted provenance can reduce the task to verification.Source: Conclusion · p. 9
• Venues must specify policies, decide which traces to retain, set tolerated false-accusation rates in advance, and provide a route to contest findings.The framework's account of what the procedure controls and how often it is wrong is distinct from a modeling improvement.Source: Conclusion · p. 9

A A Protocol for Reviewer-Side AI Use

• The peer-review protocol is an end-to-end illustration, but its permitted, gray-zone, and prohibited workflow partition must be re-derived for the policy actually in force.Source: A A Protocol for Reviewer-Side AI Use · p. 14
• A useful protocol states the policy, enumerates permitted and prohibited workflows, defines the inferential target, and evaluates under evidence conditions resembling deployment.These steps do not remove ambiguity, but make claims more interpretable and comparable.Source: A A Protocol for Reviewer-Side AI Use · p. 14

A.1.1 Policy

• Under the stated reviewer policy, language editing and non-sensitive summary support are permitted, while model-written reviews, model-assigned recommendations, unsupported criticism, and unapproved confidential uploads are prohibited.Model-expanded critique from brief reviewer notes remains gray-zone unless the venue resolves its status explicitly.Source: §A.1.1 · p. 14
• Appropriate use depends on the relationship among workflow, reviewer role, and policy, not on the final review alone.The policy permits a reviewer to write an assessment and use a model for grammar, but prohibits delegating the review and recommendation.Source: §A.1.1 · p. 14

A.1.2 Inference

• The hypothesis test treats a policy-permitted workflow as H0 and a prohibited workflow as H1, with rejection of H0 conditioned on available evidence and its uncertainty.Source: §A.1.2 · p. 14
• A Bayesian formulation estimates the probability of policy violation given the review, manuscript, policy, and available workflow evidence, then reports uncertainty, likely workflow, evidence used or missing, and a recommendation category.Recommendation categories range from no concern and insufficient evidence to human review needed or likely violation.Source: §A.1.2 · p. 14

A.1.3 Evaluation

• Multiple workflows generated from the same manuscript isolate workflow differences rather than manuscript differences.The workflow set ranges from human-only review and constrained editing or summary support to model-written reviews, model-assigned recommendations, and adversarial paraphrase or splicing.Source: §A.1.3 · p. 15
• The remaining benchmark workflows include model-assigned recommendations and adversarial variants that paraphrase, impose a style guide, or splice in human-written text.For critique expansion, the benchmark records whether the model added claims absent from reviewer notes.Source: §A.1.3 · p. 15
• Evaluation reports TPR at a fixed FPR under permitted workflows, with per-workflow results, calibration, abstention, and performance by evidence exposed.The proposed benchmark tests whether workflow-aware inference outperforms artifact-only detection, especially on adversarial workflows.Source: §A.1.3 · p. 15
• Evaluation should report false-positive rates by reviewer language background because aggregate rates can conceal excess errors for non-native English writers.The passage states that detectors carry a documented bias against non-native English writers and that many international reviewers write English as a second language.Source: §A.1.3 · p. 15

B A Sketch of a Machine-Readable Reviewer Policy

• A machine-readable reviewer policy specifies scope, permitted and prohibited workflows, gray zones, and disclosure requirements.The specification also identifies collected evidence, retention and access, pre-set error tolerances, sanction standards, and appeal procedures.Source: B A Sketch of a Machine-Readable… · p. 15
• Writing policy down is useful even without a detector because it makes venues set prohibited uses and acceptable error rates in advance.The policy sketch also requires an appeal path and says a detector report alone is insufficient for the highest sanctions.Source: B A Sketch of a Machine-Readable… · p. 16