Source-linked AI summary
Value Over Language Model: Detecting Original Contribution in Writing
Vibhhu Sharma, Thorsten Joachims, Sarah Dean
TL;DR
Existing LLM detectors measure surface-text involvement rather than how much information originated with the human author. VOLM extracts content at increasing granularity, reconstructs documents with an LLM, and compares them with task-only reference documents; it separates human-authored from matched LLM-generated documents while reducing stylistic residuals through constrained extraction.
Problem
Existing detectors measure how much surface text an LLM produced, not how much of the document’s information or ideas originated with the human author.
Method
VOLM extracts document content at increasing granularity, reconstructs each version with an LLM, and compares the reconstructions with those produced from the task description alone without scoring surface text.
Results
VOLM separates human-authored documents from matched LLM-generated documents across news, peer reviews, and essays while remaining substantially invariant to content-preserving transformations.
Takeaways & Limitations
Increasingly constrained extractors reduce residual differences between LLM-generated and humanized text, supporting extractor design as an important direction for future work.
Takeaways & Limitations
Residual stylistic information can persist after LLM-assisted rephrasing, causing different-model or humanized text to receive non-zero VOLM scores despite little additional original information.
Abstract
from arXiv · showhide
LLMs have been rapidly adopted across writing tasks, prompting the development of tools for detecting LLM-generated text. Yet, these tools largely measure how much of a document's surface text was written by an LLM and aren't fundamentally designed to measure how much of the information content or ideas originated from the LLM itself rather than being supplied by the user in the prompt. In this work, we design a framework that measures how much value a person adds on top of what a language model could have easily produced by itself. The method requires no training or labeled data and never scores the document's surface text, insulating it from stylistic confounders. Instead, it extracts the document's content at increasing levels of granularity, uses an LLM to reconstruct the document from each partial representation, and compares these reconstructions with those produced from the task description alone. We call this framework Value Over Language Model (VOLM), which measures a document's contribution relative to a replacement-level document that an LLM could produce from the task description alone. We evaluate VOLM with a specific instantiation of this framework across three domains: news articles, ICLR peer reviews, and argumentative essays. VOLM separates human-authored documents from matched LLM-generated documents produced from generic task descriptions, while remaining substantially invariant to content-preserving transformations, including LLM-based reconstruction and round-trip translation. We further find that increasingly constrained content extractors reduce residual differences between LLM-generated and humanized text, demonstrating the importance of disentangling informational content from stylistic variation. We hope these results encourage further work on specialized instantiations of the framework and on assessing human contributions in LLM-assisted writing more generally.
1 Introduction
Existing detectors measure LLM involvement in a document’s surface text, whereas VOLM measures the human-supplied information underlying that text. VOLM compares a document with what a language model could produce from a generic task description alone.
- Motivation: Current approaches estimate whether, or how much, a language model produced a document’s final text rather than whose ideas it expresses.A document can be largely rendered by an LLM while reflecting substantial human thought supplied through detailed prompts.
- Framework: VOLM measures human contribution as the gap between the document and what a language model could produce from a generic prompt alone.Short generic prompts define the replacement-level content already available to the model; detailed guidance supplies information it did not have.
- Framework: VOLM extracts document content at increasing granularity, reconstructs the document from each partial representation, and evaluates model surprisal of those reconstructions.Higher surprisal that persists as content is revealed indicates information supplied by the human author.
- Contributions: The framework requires no model training or labeled data and operates through log-probabilities from an off-the-shelf language model.It evaluates reconstructed documents rather than directly scoring surface text.
- Evaluation: VOLM is evaluated on news articles, ICLR peer reviews, and argumentative essays, with sensitivity to genuine content differences and robustness to several content-preserving transformations.The tested transformations include LLM paraphrase, translation round-trips, and humanizer tools.
2 Related Work
Related work measures machine authorship, human contribution, perceptual credit, or model memorization from different perspectives. VOLM differs by using task-conditioned surprisal and an explicit replacement document while avoiding direct surface-text scoring.
- LLM Detection: LLM-detection methods generally identify machine-generated text or estimate the fraction of final text subject to LLM editing.These approaches focus on textual production rather than the origin of a document’s ideas.
- Information-Theoretic Measures: Xie et al. define human contribution as ϕ = I(x; y)/I(y), using mutual information between human input and AI output relative to unconditional output self-information.Their primary measure requires the human input, which an evaluator with only the finished document does not have.
- Information-Theoretic Measures: VOLM measures surprisal relative to the task description T rather than no context, treating task-only generation as the relevant replacement-level baseline.It compares against an explicit reference document Dref = M(T) instead of a scalar threshold.
- Information-Theoretic Measures: VOLM never scores the document’s surface text, instead scoring reconstructions to reduce confounding from stylistic variation.This distinguishes its evaluation target from perplexity-based estimates tied directly to the document’s wording.
- Other Perspectives: HCI work studies contribution attribution through perceived credit and process tracing, while memorization work studies how readily prompts elicit target strings from model parameters.VOLM shares a compression-based intuition with the Adversarial Compression Ratio.
3 Framework
VOLM maps a document to model-score behavior as its extracted content is progressively revealed. An extractor produces increasingly detailed representations, and a reconstructor generates model-written documents from each representation before scoring them.
- Core Idea: VOLM measures how model surprise changes when a document’s actual content is progressively revealed as context.Documents already producible from a generic prompt should stop surprising the model quickly; human-supplied information should preserve surprise longer.
- Pipeline: The pipeline consists of extraction, reconstruction, and scoring stages that produce a sequence of reconstruction score distributions across granularity levels.The displayed pipeline summarizes extraction of granularity levels, reconstruction, and scoring for a single document.
- Extraction: The extractor E condenses document D and task description T into k representations g_i at increasing granularity.Lower levels retain coarser, less specific information, while g_k approximates the document’s full content.
- Extraction: Granularity may increase by selecting more discrete facts or claims, or by using a length-based extraction scheme.The content-based approach allows k to vary with the document.
- Reconstruction: The reconstructor uses the language model to generate D-hat_i from T, extracted content g_i, and the original document length.The original document D is never scored directly; only model-generated reconstructions are evaluated.
3.2 Score Distributions
VOLM estimates when a document’s reconstruction scores persistently diverge from a task-only reference document. It uses repeated stochastic extraction and reconstruction samples, then converts the earliest persistent difference into a score.
- Score Distributions: Extraction and reconstruction are stochastic, so each granularity level yields m × n reconstruction scores from repeated samples.The pipeline maps a document to one score distribution per granularity level.
- Reference Baseline: The replacement reference Dref is generated by the model from the task description alone and passed through the same scoring pipeline as D.This establishes the document the model could produce without additional supplied content.
- Statistical Comparison: At each level, VOLM tests whether the document and reference score distributions are drawn from the same distribution, rejecting at p < 0.05.The comparison is performed separately across the granularity sweep.
- Statistical Comparison: The index i* is the smallest level whose null rejection persists through all subsequent levels, preventing a single early rejection from inflating the score.The persistence condition guards against spurious early divergence.
- Interpretation: VOLM is high when the document departs from the reference early and low when it remains indistinguishable until nearly all content is revealed.Early departure corresponds to i* small; late or absent departure corresponds to i* near k or undefined.
4 Experiments
The experiments evaluate VOLM across journalism, peer review, and argumentative essays using matched human and LLM-generated documents. Results show that human documents diverge from replacement-level LLM documents at low granularity, while content-preserving transformations retain the human-pattern curve.
- Datasets and setup: The evaluation spans journalism, academic peer review, and argumentative essays, pairing each human document with an LLM-generated counterpart using the same task description.Generated counterparts were constrained to within 10% of the human document’s length.
- Procedure: VOLM extracts ordered facts at eleven granularity levels, reconstructs documents from task descriptions plus fact subsets, and scores average per-token log-probabilities conditioned on the task description alone.The All the News instantiation uses WordNet-canonicalized facts and levels gi = i/10.
- Population-level validation: The human population curve separates from the reference Llama curve at granularity 0.1 and decays faster as additional content is provided.The reference curve decays slowly because its extracted information was already implicit in the model and task description.
- Invariance to LLM-style writing: Each content-preserving variant closely tracks the human curve and departs early from the reference Llama curve, indicating that the evaluation is stable under LLM transformations.The reported variants were evaluated with group-averaged score curves on All the News, with analogous curves for ICLR and Persuade deferred to the appendix.
- Robustness and humanization: Reference documents form the slowly decreasing tier, other-model and humanized documents an intermediate tier, and human documents with content-preserving transformations the sharply decreasing bottom tier.The hierarchy distinguishes surface-origin changes from the presence of genuinely new information.
- Per-document procedure: For one illustrative document, the human curve diverges from its reference counterpart at granularity 0.1, producing a VOLM score of 0.9.The same figure was later overlaid with curves for additional transformed variants.
5 Discussion
The discussion shows that reconstruction can preserve stylistic differences, causing non-zero VOLM for content-equivalent text, while increasingly constrained extractors narrow this residual gap.
- Residual stylistic variation: Figure 4 shows both Llama-generated and DIPPER-humanized word choices persisting unchanged through extraction and reconstruction despite different starting wording.
- Residual stylistic variation: Stylistic nuances can persist through condensation and reconstruction, causing the method to measure stylistic gaps alongside information gaps.The reconstructor often reuses source word choices, even after two LLM transformations.
- Residual stylistic variation: The reconstruction procedure reduces the initial log-probability gap between Llama and humanized articles, but not sufficiently to eliminate stylistic confounding.
- Extractor design: 0.367 to 0.101: the Llama-Humanized AUC gap fell across increasingly constrained extractors, a 72.4% reduction from least to most constrained.The sequence was 0.367, 0.166, 0.121, and 0.101 for AUC of average log-probability across the compression sweep.
- Extractor design: 16.2%: WordNet canonicalization alone further reduced the Llama-Humanized gap from 0.121 to 0.101, while the Humanized-Human gap changed by under 4%.Only comparisons with the reference LLM were affected by extractor optimization.
- Future directions: The framework remains a first step, with more principled extractors and reconstructors needed to better isolate information gaps from stylistic differences.The authors also identify multiple evaluator models as a promising direction.
6 Conclusion
The conclusion presents VOLM as a training-free framework for distinguishing human ideas from LLM-produced wording and reports feasibility across three writing domains.
- VOLM measures a document’s value relative to what a language model could produce from the task description alone, rather than the fraction of text attributable to an LLM.
- The framework extracts content at increasing granularities, reconstructs it with a language model, and compares reconstruction surprise with a replacement-level document.
- Across journalism, peer review, and student essays, VOLM separates human-authored documents from matched documents generated from generic task descriptions.
- VOLM remains substantially invariant to content-preserving transformations, including LLM reconstruction and round-trip translation.
- Residual stylistic information can produce non-zero VOLM for text with little additional original information, but constrained extractors can reduce this effect.
- The authors describe the work as a first step toward granular contribution measurement rather than merely detecting AI use.
A.1.1 Condensation-based Extractor
The condensation-based extractor creates representations at specified fractions of a document’s original length.
- At granularity level i, the condensation-based extractor prompts the evaluator model to condense a document to fraction i of its original length.
- A document at granularity 0.1 is generated by asking the LLM to condense the original document to 10% of its original length.
A.1.2 Fact Extractor
The fact extractor represents a document by progressively larger subsets of facts extracted by the evaluator model.
- The fact extractor first obtains facts without a stated limit, then constructs granularity i using the first ith fraction of those facts.
- If 10 facts are extracted, granularity 0.1 contains one fact; if 15 are extracted, it contains two facts using the ceiling operator.
A.1.3 Equalized Fact Extractor
The equalized fact extractor controls the number of facts extracted from each evaluated document to match the reference LLM document.
- The extractor sets a target fact count so every evaluated document yields as many facts as its reference LLM document.
A.1.4 Equalized Schema-constrained Fact Extractor
The equalized schema-constrained fact extractor represents each fact with fixed fields and renders records deterministically, constraining both the information captured and its serialization.
- Each extracted fact conforms to a fixed schema with fields for its structured representation.
- A Ford investment sentence is converted into a structured fact containing subject, action, object, quantity, date, and quote fields.
- The schema constrains represented information and serialization, after which records are rendered with a fixed template instead of free-form model-written sentences.
A.1.5 Equalized Schema-Constrained Fact Extractor with WordNet Canonicalization
The extractor represents documents as increasingly structured facts, reducing stylistic differences and producing clearer curve orderings across evaluation corpora. For Persuade essays, the improved extractor places Qwen between the Llama reference and human-related curves.
- The ICLR and Persuade evaluations plot corpus-level log-probability curves using condensation-based extraction.
- The Equalized Schema-Constrained Fact Extractor with WordNet Canonicalization removes stylistic differences more effectively than the simpler extractor.
- The improved Persuade extractor orders Qwen between the Llama reference curve and the human corpus with its information-preserving variants.
- The sample human review criticizes missing character development, while the Qwen-transformed review introduces a different treatment of the source material.
- Across transformed reviews and essays, recurring content includes criticism of disjointed adaptation, altered source material, and insufficient faithfulness to the original work.