PaperlayerView exact revision on arXiv

Source-linked AI summary

Likelihood Is Not Authorship: Can We Reliably Measure Human Contribution in AI-Assisted Writing?

Hasan Amin, Zhuoran Lu, Ming Yin

arXiv:2610.19767v1cs.CL

TL;DR

Likelihood-based contribution scores use prompt predictability to estimate how much of AI-assisted writing reflects human input. The paper tests that measure with copied random IDs and a context-swap audit, finding large score shifts on unchanged texts. It concludes that likelihood and transcript evidence alone cannot reliably establish human contribution.

  • Problem

    Likelihood-based scores estimate human contribution from how predictive a person’s prompt is of the final text, but the paper asks whether that relation identifies contribution.

    Source: §1 · p. 1

  • Method

    The paper introduces SPECIFICITY PADDING, which prompts models to copy random IDs, and audits scores by changing evaluator context while holding text fixed.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Showing matching IDs raises fixed-output scores by 32–51 points across tested models and documents, even though the text does not change.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Likelihood scores measure how much context helps predict text, not authorship, and reliable contribution claims need evidence about how the text was produced.

    Source: §7 · p. 7, §9 · p. 8

    If a recorded prompt could have been machine-written, every transcript-only estimator can miss by at least half the contribution at stake in the worst case.

    Source: §6 · p. 7

Abstract

from arXiv · show

AI-text detection generally asks whether a machine wrote a text. As AI becomes an increasingly integral and accepted part of writing, the more relevant question is no longer whether AI was used, but how much of the text reflects human contribution. A prominent line of work estimates this by measuring how predictive a person's prompt is of the final text. We show that this measure can be easily manipulated. We introduce SPECIFICITY PADDING that simply asks the model to copy a few random IDs from the prompt, and that alone makes a title-only request look more human than one that supplies the article's key ideas. A context-swap audit then holds each document fixed, and only shows or hides the IDs to the evaluator. The score of the unchanged text swings by a third to half of its scale, and it rises for every document we test, on news and scientific abstracts, across varied LLMs from 3.8B to 753B parameters. The same articles can average 85\% human or 15\% human depending only on which prompt is on record, and even with no prompt at all, the output-only variant rewards the random strings themselves. An exact identity, the credit ledger, explains why. The score is a surprisal-weighted average over the text, so rare strings that the context makes copyable collect the credit, whatever they mean. A two-point bound marks a deeper limit. If the prompt could have been machine-written, every transcript-only method must, in the worst case, miss by half the contribution at stake. Likelihood-based scores alone cannot reliably establish human contribution without evidence of what the person added and how the text was produced.

1 Introduction

As AI-assisted writing blurs the division between human and model contributions, likelihood-based scores treat prompt predictability as a proxy for human contribution. SPECIFICITY PADDING and a context-swap audit expose how that proxy can be manipulated.

1 Introduction

  • SPECIFICITY PADDING asks a model to copy random prompt IDs, making them predictable and highly credited despite adding nothing to the article.The attack requires no gradients, weights, or evaluator queries, and the audit applies to context-conditioned likelihood metrics.Source: §1 · p. 2
  • A title-only request with copied IDs can outrank a request supplying key ideas when the evaluator sees those IDs; hiding them reverses the ranking on unchanged text.The swing is wider than the full validation ladder from a human draft to a bare subject.Source: Figure 1 · p. 1, §1 · p. 2
  • Showing matching IDs raises fixed-output scores by 32–51 points across tested models and documents, while ordinary padding inflates scores by 13–27 points.The same articles can average 85% or 15% human depending only on which prompt the evaluator sees; scientific abstracts fail alike.Source: §1 · p. 2
  • The credit ledger explains the attack: rare strings receive disproportionate score credit once the context makes them predictable, regardless of their meaning.The score is a surprisal-weighted average over text parts.Source: §1 · p. 2
  • The paper argues that transcript-only contribution claims have a provenance gap because a recorded prompt does not reveal who wrote it.If the prompt could be machine-written, a transcript-only estimator can miss by at least half the contribution at stake in the worst case.Source: §1 · p. 2

2 Contribution as Predictability

The section defines the likelihood-based score as contextual predictability and examines what its validation can establish. Its prompt-detail ordering cannot distinguish contribution from copyable information when both change together.

2 Contribution as Predictability

  • The score ϕ = 0.6 when an output has 100 nats of surprisal without the prompt and 40 with it, meaning the prompt explains 60% of the model’s surprise at the exact tokens.The score is described as normalized contextual predictability; the output-only variant substitutes a model-specific plausibility threshold for input-conditioned surprisal.Source: §2 · p. 3
  • Agreement with the polishing-to-subject prompt ladder cannot distinguish contribution tracking from copyable-information tracking because those quantities move together along the ladder.Separating them requires varying information while holding contribution fixed.Source: §2 · p. 3
  • For the studied news articles, random IDs are task-null because they change no event, claim, or argument, though identifiers may matter in other tasks.The task-null definition treats a score that moves substantially under such a transformation as invalid for the targeted contribution.Source: §2 · p. 3

3 Specificity Padding and the Context-Swap Audit

The section introduces SPECIFICITY PADDING and a context-swap audit to test whether likelihood scores respond to prompt information rather than changes in the text. It also describes the generation setting used to examine the attack in ordinary use.

3 Specificity Padding and the Context-Swap Audit

  • SPECIFICITY PADDING appends random IDs to a prompt and asks the model to use them verbatim; the IDs carry nothing about the article.The dose K is the number of IDs, drawn from nonce names, codes, measurements, labels, and tags.Source: §3 · p. 3
  • The context-swap audit scores one byte-identical padded output under MATCHED, HIDDEN, and WRONG ID contexts, isolating changes due to evaluator context.MATCHED−HIDDEN measures credit from showing the IDs, while MATCHED−WRONG tests whether exact values matter beyond generic specificity.Source: §3 · pp. 3–4
  • For generation-time padding, the study adds eight IDs to title requests and compares attacked articles with clean generations, then swaps contexts on attacked articles.The setup includes 300 news items across seven models and 200 arXiv abstracts across three served models; Table 1 reports paired mean effects and positive-pair counts.Source: §3 · p. 4, Table 1 · p. 4

4 Copied IDs Earn Human Credit

Copied IDs can change contribution scores without changing the article; the reported effects span generated news, output-only estimates, and scientific abstracts.

4.1 A few copied IDs outrank the article’s key ideas

  • 20 points: eight copied IDs lift the clean title-request score, exceeding the 17-point title-to-summary step; the 300-item run scores 0.75 versus 0.55.The IDs do not change what the person contributed.Source: §4.1 · p. 4

4.2 One text, many authors

  • 32–51 points: MATCHED context raises fixed-output scores for all 2,100 model–document pairs, turning one Llama-3.1-8B score from 0.17 to 0.68 without changing the text.Figure 3 compares contexts on the same output.Source: §4.2 · p. 4, Figure 3 · p. 5

4.3 Whoever holds the prompt holds the score

  • 51–84 points: hiding the IDs lowers scores on the same generated articles; DeepSeek-V4-Flash averages 0.85 with the full prompt and 0.15 with only the visible request.Models reproduce 65–99% of requested IDs verbatim.Source: §4.3 · p. 5

4.4 Without any prompt, rarity itself earns credit

  • 0.36 to 0.68: appending eight IDs raises Llama-3.1-8B’s output-only estimate of minimal human contribution across all 300 articles.This estimate requires no matching prompt.Source: §4.4 · p. 5

4.5 Findings extend to other datasets

  • 43–47 points: matching IDs raise fixed-output scores on 200 arXiv abstracts, while generation-time padding inflates scores by 13–20 points.Fabricated measurements can read as scientific claims, so the generation-time comparison tests the score, not content preservation.Source: §4.5 · p. 5

5 Why It Works: Credit Follows Surprisal

The credit ledger expresses a text’s score as a surprisal-weighted average of local scores. Rare, predictable suffixes can therefore draw credit toward their own score, while output-only estimates reward rarity itself.

5 Why It Works: Credit Follows Surprisal

  • The credit ledger makes the text score a surprisal-weighted average of local scores, so rare spans carry more weight regardless of their importance.The identity follows from the chain rule and assumes no token independence, ID format, or architecture.Source: §5 · pp. 5–6
  • A padded score lies between the body’s score and the suffix’s local score, exceeding the body’s score exactly when the suffix scores higher.As the suffix’s surprisal share grows, the padded score can approach 1 regardless of the article’s content.Source: §5 · p. 6
  • 14 points to zero: MATCHED padding’s gain falls from subject-and-title prompts to polishing, vanishing near a clean score of 0.84.Hiding the IDs lowers every assistance level by 35–40 points.Source: Figure 4 · p. 6, §5 · p. 6
  • On a fixed output, the matched–hidden gap reflects both changes in context prediction of the body and the IDs’ surprisal share multiplied by how much context predicts them.When matching makes IDs nearly certain and hiding leaves them unpredictable, the gap approaches the suffix’s surprisal share.Source: §5 · p. 6
  • Above 0.9: under Llama-3.1-8B’s threshold, text averaging six nats per token receives an output-only estimate of minimal human contribution above 0.9.Rare strings raise this estimate even without a prompt.Source: §5 · p. 7

6 The Provenance Gap: What No Transcript Can Certify

A transcript can be compatible with different production histories, so it cannot establish who authored the recorded prompt. Consequently, no transcript-only estimator can universally certify causal contribution.

6 The Provenance Gap: What No Transcript Can Certify

  • For identical recorded prompts and documents, person-authored and machine-generated histories can imply different causal contributions, making them indistinguishable to transcript-only evaluators.The identification set collects contributions from histories compatible with the same observation.Source: §6 · p. 7
  • At least half the contribution at stake is the worst-case error for any transcript-only estimator when the recorded input could be machine-generated.Randomizing the estimator does not help, and a stronger model cannot determine which compatible history occurred.Source: §6 · p. 7
  • The boundary is not a claim that detection is impossible: distributional detection and average-case estimates under a stable workflow prior remain possible.The result rules out universal, instance-level certification when materially different histories fit the same observation.Source: §6 · p. 7
  • Recorded context does not reveal whether a person typed, copied, retrieved, or received its contents from a hidden model call.Ordinary system prompts, retrieved passages, templates, and pasted model text can enter that context; provenance depends on the process.Source: §6 · p. 7

7 What Contribution Scores Should Report

The authors recommend reporting the score’s target and context, auditing sensitivity to context changes, and inspecting where credit falls. Claims about a specific person’s contribution require evidence about the production process.

7 What Contribution Scores Should Report

  • Reports should specify the text, context, evaluator, and estimand rather than label a likelihood score simply as human contribution.The U.S. Copyright Office concluded that prompts alone do not make a user the author of AI output (U.S. Copyright Office, 2025).Source: §7 · p. 7
  • Before deployment, audit fixed outputs under MATCHED, HIDDEN, and WRONG contexts to expose score sensitivity without generating new text.The audit also applies to context-conditioned likelihood metrics such as pointwise V-information and Shannon-game summary metrics.Source: §7 · p. 7
  • Report the credit ledger so reviewers can see whether identifiers, names, or numbers receive concentrated credit.Whether those spans matter remains a task-specific semantic judgment that likelihood cannot make.Source: §7 · p. 7
  • Masking rare strings may blunt this attack but cannot address the mechanism, and the same mask could erase legitimate measurements.Names, quotations, and technical terms can also be copyable high-surprisal spans; masking does not close the provenance gap.Source: §7 · p. 8
  • Claims about who contributed what to a specific document require process evidence such as authenticated logs, revision histories, or content credentials.Such records shrink the identification set but do not themselves assign intellectual value (Lee et al., 2022; Hoque et al., 2024; Coalition for Content Provenance and Authenticity, 2026).Source: §7 · p. 8

8 Related Work

Related work spans graded contribution measurement, information-theoretic metrics, detection, and construct validity; this paper connects their limits to context sensitivity and missing provenance.

Measuring human contribution

  • Prior work classifies text, locates machine-written segments, and estimates graded involvement; this paper adds an invariance audit and an identification bound.The assistance ladder and pairwise judgments vary information and contribution together.Source: §8 · p. 8

Information-theoretic evaluation

Detection and its limits

  • Paraphrasing can degrade detectors, while distributional separation can support detection; graded contribution inherits these dynamics and adds another attack surface.The passage also notes limits for strong watermarking and cryptographically undetectable constructions (Krishna et al., 2023; Sadasivan et al., 2025; Chakraborty et al., 2024; Zhang et al., 2024a; Christ et al., 2024).Source: §8 · p. 8

Construct validity and provenance

  • Construct validity concerns which interpretations a measure supports, while process records capture variables that transcript-only scores lack; the identification result applies causal-identification logic to authorship.Attribution and factuality assess source support, which is distinct from who created claims.Source: §8 · p. 8

9 Conclusion

The paper concludes that likelihood measures how context helps predict text, not who authored it, and that reliable contribution claims need a declared target, context-swap audit, ledger, and production evidence.

9 Conclusion

  • Likelihood alone cannot reliably establish human contribution: it credits context-predictable strings, while transcripts do not reveal who wrote the prompt.The conclusion calls for a declared target, context-swap audit, reported ledger, and evidence of how the text was made.Source: §9 · p. 8

Limitations

The study’s findings are bounded by its evaluator setup, language and genre coverage, and theoretical assumptions. The authors also distinguish synthetic metadata attacks from meaning-preserving or undetectable ones.

Scope

  • Effect sizes may differ across languages, genres, decoding settings, and generation seeds.The study uses English news and scientific abstracts, with one clean and one attacked generation per item.Source: Limitations · p. 9

Evaluator

  • Each model evaluates its own outputs, and the study does not test surrogate evaluators.Scores are model-relative rather than calibrated across families; served endpoints do not expose weight revisions or numerical precision.Source: Limitations · p. 9

Stealth and meaning

  • The fixed suffix is conspicuous, and generation-time padding can introduce unsupported details; the authors do not claim a meaning-preserving or undetectable attack.In one-model pilots, subtler lexical variants produced small or no gains.Source: Limitations · p. 9

Task relativity and theory

  • Treating synthetic metadata as null fits news substance, but not every task; identifiers may count as contributions in lab protocols or under a definition that includes every prompt-supplied bit.The headroom analysis uses four assistance levels from one model, and its body-predictability assumption could be sharpened with span-level measurements; the identification bound is worst-case, not a workflow-specific error estimate.Source: Limitations · p. 9

Reproduction

  • The study reproduces the assistance ordering on Xie et al.’s 300-item news set, not the original four-domain benchmark’s numbers.The authors report this as a reproduction of ordering rather than of the original benchmark’s numerical results.Source: Limitations · p. 9

Ethical Considerations

Across news models, matching IDs raise fixed-output scores, while the paper also documents the study’s data, scoring, and implementation choices. Its ethical discussion recommends treating transcript-only scores as non-dispositive evidence about individuals.

Ethical Considerations

  • Contribution scores can affect judgments of originality, academic integrity, ownership, and compensation, with false confidence harming authors through both inflation and deflation.The authors present the synthetic-string attack as a measurement stress test, not a way to evade institutional policy, and recommend auditable evidence, consent, appeals, and error analysis.Source: Ethical Considerations · p. 9

A.1 Data

  • The news study uses Xie et al.’s 300-item set; its 30-item Llama-3.1-8B subset reproduces the assistance ordering from polishing through subject prompts.The arXiv sample contains 200 abstracts drawn uniformly from 1,999,486 records with nonempty identifiers, titles, and abstracts; models receive titles but not source abstracts.Source: §A.1 · p. 13

A.2 Prompts and IDs

  • Prompts append project metadata as verbatim-copy material, while WRONG IDs preserve types and count but use values from a disjoint seed.The prompt’s metadata includes dataset, protocol, measurement, condition, and label fields.Source: §A.2 · p. 13

A.3 Models and Computations

  • Seven models from six families generate and evaluate their own outputs; local runs use specified hardware and software, while served models use at most four concurrent requests.The experiments use about 15 NVIDIA A10 GPU-hours locally and 16 million input and output tokens across served models, excluding pilots and reruns.Source: §A.3 · p. 13

A.4 Scoring conventions; A.5 Decoding, seeds, and statistics

  • Conditional scores sum response-plus-EOS negative log-likelihood, while absolute scores remain model-relative and are not calibrated across families.The output-only estimate is reported only for Llama-3.1-8B because thresholds were unavailable for other models; decoding uses temperature 0.7, top-p 0.9, and up to 384 new tokens.Source: §A.4 · pp. 13–14, §A.5 · p. 14

B.1 Per-model effects

  • 0.324–0.505: matching IDs raise fixed-output scores in all 2,100 news pairs, and MATCHED beats WRONG in every pair.Generation-time attack−clean effects are positive for 2,040 of 2,100 articles; fixed-output contrasts remove length as an explanation.Source: §B.1 · p. 14, Table 4 · p. 15

B.2 Dose response; B.3 Llama-3.1-8B diagnostics

  • As IDs accumulate, MATCHED scores move toward the suffix’s local score and HIDDEN scores fall; Figure 5 shows these trends in every run shown.At eight IDs, the Llama-3.1-8B output-only estimate rises from 0.36 for clean outputs to 0.68 for padded outputs.Source: §B.2 · pp. 14–15, §B.3 · p. 15, Figure 5 · p. 16

B.4 Replication on arXiv abstracts; B.5 Subtler lexical variants

  • On arXiv, matching IDs raise fixed-output scores by 0.431–0.471 in all 600 pairs; lexical pilots found only about 3-point gains from required words or phrases.Replacing up to two article words with rarer synonyms lowered the score; the authors note that differences of a point or less are not interpretable on that endpoint.Source: §B.4 · p. 15, §B.5 · pp. 15–16

C.1 Proofs

The proofs express contextual scores through weighted local scores and bound what transcript-only observations can identify. Mechanism checks connect these relationships to suffix surprisal and score headroom.

C.1 Proofs

  • The padded score is a weighted average of the body’s and suffix’s local scores, so padding raises it exactly when the suffix score exceeds the body score.The context-swap difference for a fixed output depends on the likelihood change between contexts, normalized by the output’s unconditional surprisal.Source: §C.1 · pp. 16–17
  • For an observation-compatible contribution range of width D(o), any transcript-only estimator has worst-case error at least D(o)/2, while the midpoint attains that bound.The observation map and admissible history class determine the compatible range; authenticated logs or workflow constraints can shrink it.Source: §C.1 · pp. 17–18
  • Transcript-level point identification requires the contribution to be constant across each observational-equivalence class.The result permits prior-based prediction, population estimation, and distributional detection, but does not establish point identification otherwise.Source: §C.1 · p. 18

C.2 Mechanism checks

  • The mechanism checks infer that eight IDs carry about half the padded document’s surprisal, despite occupying a minority of its tokens.The inference uses output-only estimates and an approximation that ignores item heterogeneity and the terminal token.Source: §C.2 · p. 18
  • Across the four levels, regressing the MATCHED shift in Table 6 on clean score gives slope −0.43, intercept 0.36, zero crossing 0.84, and R2 = 0.99.If the IDs leave the body’s own score unchanged, inflation vanishes when the clean score reaches the suffix’s local score.Source: §C.2 · p. 18
  • The near-constant HIDDEN shift of −35 to −40 points across assistance levels implies that suffix weight shrinks as the clean score rises.The account associates higher clean scores with polished outputs that follow a human-written article.Source: §C.2 · p. 18
Loading 2610.19767v1…