PaperlayerView exact revision on arXiv

Source-linked AI summary

IdeaLens: Detecting AI Ideas in Long-form Writing

Rishanth Rajendhran, Minjoon Choi, Jenna Russell, Ramya Namuduri, Deniz Bölöni-Turgut, Marzena Karpinska, John Wieting, Mohit Iyyer

arXiv:2610.06778v1cs.CLcs.AIcs.LG

TL;DR

AI-use policies increasingly ask who originated a document’s ideas, while prose detectors answer who wrote its words. IdeaLens learns from prose-detector labels using paraphrased outlines, and it tracks idea provenance in mixed-provenance writing while performing strongly across benchmarks and languages.

  • Problem

    Prose detectors identify who wrote the words, but emerging AI-use policies also hinge on who came up with the ideas, and simulated collaboration data are costly to scale and may not transfer.

    Source: §1 · p. 1

  • Method

    IDEALENS predicts prose-detector labels from paraphrased outlines that pair discourse roles with brief content descriptions, using 1M FineWeb documents labeled by Pangram.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    IDEALENS tracks idea provenance when it differs from prose provenance and, across 19 evaluations, flags 91.1% of AI documents at a 0.6% false positive rate.

    Source: §1 · p. 3, §7 · p. 9

  • Takeaways & Limitations

    The results show that human and AI writing can be meaningfully distinguished through their ideas alone.

    Source: §7 · p. 9

    Training uses noisy prose-authorship labels as a proxy for idea authorship, and IDEALENS remains less accurate on mixed-authorship than conventional documents.

    Source: A LIMITATIONS · p. 19

Abstract

from arXiv · show

While modern AI detectors identify who wrote the words, emerging policies on AI use increasingly hinge on a different question: who came up with the ideas? We introduce IdeaLens, a detector that identifies whether a document's ideas came from a human or AI (idea provenance), regardless of who wrote its words. To focus IdeaLens on ideas rather than prose, we represent documents as outlines: lists of items that each pair a discourse role with a brief, paraphrased description of the content, minimizing word-level overlap with the raw text. We train IdeaLens on 1M FineWeb documents with silver labels from Pangram, a prose provenance detector. Since the outlines are largely stripped of surface-level information, the labels must be fit mainly through the ideas. In a controlled study, IdeaLens's AI flag rate drops from 95% to 7% as models write from increasingly detailed human plans, while Pangram 4 still flags 92%; from AI-derived plans, IdeaLens stays above 96%. Conversely, on a new dataset of 50 stories that human authors wrote from AI-generated plans, IdeaLens flags 68% of the stories as AI, compared to 8% for Pangram 4. On a comprehensive suite of 19 existing detection benchmarks, we show that IdeaLens maintains strong detection rates at low false positive rates, suggesting that ideas themselves provide a powerful discriminative signal, and its performance holds across domains, formats, and languages. Finally, we examine 90K predictions from IdeaLens to characterize systematic differences between human and AI ideation. We release our models and labeled datasets to facilitate future research on idea provenance detection.

1 INTRODUCTION

IdeaLens targets idea provenance—the source of a document’s claims, events, and organization—where prose detectors identify who wrote its words. It represents documents as paraphrased outlines and tests whether that input tracks ideas across mixed-provenance writing and established benchmarks.

1 INTRODUCTION

  • Prior collaboration detectors simulate human–AI editing, but such datasets are costly to scale and their detectors may not transfer to real collaborations.Prose detectors use cheaper documents whose ideas and words share a source, leaving idea provenance distinct from the question of who wrote the words.Source: §1 · p. 1
  • On 50 human-written stories following AI-generated outlines, IDEALENS flags 68.0% as AI, versus 8.0% for Pangram and 0.0% for EditLens.It flags 96.0% of the AI originals; Pangram and PROSELENS flag all AI originals but only 8.0% and 0% of the human-written stories, respectively.Source: Figure 1 · p. 2, §1 · pp. 2–3
  • IDEALENS represents each document as a paraphrased outline pairing discourse roles with brief content descriptions, then learns Pangram labels from 1M FineWeb documents.A same-backbone control, PROSELENS, trains on the raw documents instead of outlines.Source: §1 · p. 2, Figure 2 · p. 3
  • As AI-generated documents use increasingly detailed human plans, IDEALENS’s flag rate falls from 94.9% to 6.8%, while Pangram 4 still flags 92.4% with full human outlines.From AI-derived plans, IDEALENS’s flag rate remains above 96%.Source: §1 · p. 2
  • Across 19 evaluations, IDEALENS flags 91.1% of AI documents at a 0.6% false positive rate, versus 81.8% and 0.2% for EditLens-3B.Trained only on English, IDEALENS also detects 95.3% of AI documents at a 0.1% false positive rate across 24 languages.Source: §1 · p. 3
  • The authors release IDEALENS, PROSELENS, training data, evaluation datasets, and their idea-level analysis.The paper also reports systematic differences in its predictions: ideas with specific sources, downsides, or open problems score as more human.Source: §1 · p. 3

2 LEARNING IDEA PROVENANCE FROM OUTLINES

The method addresses the lack of large-scale idea-authorship labels by training on prose-authorship labels while replacing raw documents with outlines intended to preserve ideas but remove wording. It builds format-specific role sets, extracts and paraphrases outlines, and compares the resulting detector with a raw-text control.

2 LEARNING IDEA PROVENANCE FROM OUTLINES

  • The training corpus contains 1M English FineWeb documents of at least 500 words in eight formats, labeled human or AI by Pangram 3.3.2.The authors assume ideas and prose share a source, exclude mixed-labeled documents, and divide the corpus into training, validation, calibration, and test sets.Source: §2 · p. 4
  • Each outline is an ordered list pairing discourse roles with brief descriptions of how those roles appear in the document; role sets are induced separately for each format.The authors describe the representation as separating planning from translating plans into prose.Source: §2 · pp. 3–4
  • Outline construction has three steps: induce format-specific roles, extract ordered role-and-content items, then paraphrase them to remove copied wording.Paraphrasing reduces human-labeled outline text from 8.5% to 0.3%; outlines are paraphrased for training, while inference uses unparaphrased extracted outlines.Source: §2 · p. 4
  • Both models start from a Nemotron-3.5-Lightning-30B-A3B backbone fine-tuned for one epoch with LoRA, and their decision thresholds target a 1% false positive rate.The calibration split contains human documents only; the method section also notes Pangram 3.3.2 supplies training labels while Pangram 4 is used in evaluations.Source: §2 · pp. 4–5
  • IDEALENS predicts Pangram labels from paraphrased outlines, while PROSELENS predicts the same labels from raw documents; the two models share their backbone, training documents, and labels.This controlled comparison isolates the outline representation as the input difference between the models.Source: §2 · pp. 4–5

3 EXPERIMENTAL SETUP

The evaluation spans shared- and mixed-provenance documents, using external benchmarks and three purpose-built datasets to test whether detectors follow idea or prose authorship. It measures idea-detection accuracy and false positive rates across these settings.

Evaluation scope

  • Across 19 external benchmarks and three new datasets, the evaluation tests 17 existing detectors plus IDEALENS and PROSELENS on shared and mixed provenance.The new datasets are IDEASHIFT, IDEASHIFT-X, and TWICETOLD.Source: §3 · p. 5

Evaluation metrics and baselines

  • Idea-detection accuracy measures whether each predicted AI-or-human label matches the source of a document’s ideas.IDEALENS, PROSELENS, and EditLens use a global threshold calibrated to 1% FPR; other baselines use their default thresholds or the specified Pangram label conversion.Source: §3 · p. 5

IDEASHIFT

  • IDEASHIFT varies human-plan detail across six prompt levels, while its AI-generated prose lets the study compare idea detectors’ and prose detectors’ expected responses.For human-seed documents, idea-detector flag rates are expected to fall as prompts convey more of the human plan; for AI seeds, they are expected to remain high.Source: §3 · p. 5

TWICETOLD and IDEASHIFT-X

  • TWICETOLD pairs 50 stories written from AI-generated outlines with their 50 AI originals, while IDEASHIFT-X tests level-0 and level-5 versions in 24 languages.The multilingual set also measures false positive rates on human-authored documents.Source: §3 · pp. 5–6

4 IDEALENS DETECTS AI IDEAS, NOT AI PROSE

Across benchmarks, IDEALENS is accurate on both shared- and mixed-provenance documents, where prose detectors perform poorly on mixed provenance. Controlled studies show that its predictions respond to idea provenance, including across languages and when humans write from AI plans.

Main results

  • 95.3% shared- and 81.3% mixed-provenance accuracy make IDEALENS the only evaluated detector strong in both settings.The weighted mixed-provenance accuracy is 92.6% for IDEALENS, versus 46.8% for PROSELENS and 41.1% for Pangram 4.Source: §4 · p. 6

IDEASHIFT

  • 94.9% to 6.8%: IDEALENS’s flag rate falls as human plans become more detailed, while it stays above 96% for AI-derived plans.PROSELENS and Pangram 4 flag 86–100% of documents across human-plan levels; IDEALENS’s largest drop occurs between levels 3 and 4, when prompts add outline content beyond structural roles.Source: §4 · pp. 6–7

AI rewording of human ideas

  • On 12 benchmarks of AI polishing, paraphrasing, or rewriting, IDEALENS labels 94.7% of documents as human, versus 50.7% for PROSELENS and 43.8% for Pangram 4.Results on GEDE and HART also show low IDEALENS flag rates after edits to human text.Source: §4 · p. 7

TWICETOLD

  • 68% of human-written TWICETOLD stories are flagged as AI by IDEALENS, compared with 0% for PROSELENS and 8% for Pangram 4.IDEALENS also flags 96% of the AI originals; 12 of its 16 errors concern stories with speculative or fantasy elements.Source: §4 · p. 7

Shared-provenance benchmarks and attacks

  • 91.1% of AI documents are detected by IDEALENS at 0.6% FPR on 19 shared-provenance evaluation splits, versus 81.8% at 0.2% FPR for EditLens-3B.IDEALENS’s detection rate falls 6.4 points on average under humanization and adversarial attacks; for DIPPER paraphrasing in GEDE, it detects 77.6%, versus 97.1% for PROSELENS and 95.1% for Pangram 4.Source: §4 · p. 7

IDEASHIFT-X

  • 95.3% of level-0 and 0.7% of level-5 documents are flagged by IDEALENS across 24 languages, despite English-only training.It flags 0.1% of 10K pre-ChatGPT human-authored documents in those languages.Source: §4 · p. 8

Raw-text inference

  • 96.1% of AI documents are detected by IDEALENS on shared-provenance raw text at 0.3% realized FPR, while it labels 89.6% of human-idea, AI-prose documents as human.With outlines, it labels 94.7% of the latter documents as human; on TWICETOLD’s AI-idea, human-writing stories, its raw-text detection rate is 6%.Source: §4 · p. 8

5 WHAT SEPARATES AI IDEAS FROM HUMAN IDEAS?

An ablation indicates that IDEALENS relies primarily on outline content rather than role labels or item order. A feature analysis of 90K item-level predictions describes differences in how human and AI ideas use details and qualify claims.

Content and outline structure

  • Removing role labels or shuffling outlines leaves IDEALENS’s AUROC at 0.993, while role labels alone yield 0.601, indicating reliance primarily on content.The authors extract 79 interpretable features from 90K item-level predictions, most predictive of held-out item scores.Source: §5 · pp. 8–9

Details and specificity

  • AI ideas use details that may fit many subjects or sound specific without actionable substance, while human ideas include details crucial to the author’s point.Examples contrast generic speaker praise and vague “risk reduction strategies” with concrete details relevant to the particular argument.Source: §5 · p. 9

Claims and limitations

  • AI ideas may overstate claims and omit downsides, whereas human ideas tend to bound claims and state their limitations.The examples contrast a prediction of a permanent legacy with a qualified fleet-deployment projection, and an AI review that omits drawbacks.Source: §5 · p. 9

6 RELATED WORK

Prior work studies prose authorship, mixed human–AI writing, and writing-process provenance through several approaches. IDEALENS instead uses prose-level labels with paraphrased outlines to train for idea provenance.

Detecting AI and mixed authorship

  • Prior mixed-authorship research addresses AI-edited text or separates content from expression, while other work logs writing processes or theorizes provenance.IDEALENS grounds its formalization in cognitive theories of writing rather than the text itself or formal semantics.Source: §6 · p. 9

Learning from transformed inputs

  • IDEALENS replaces prose-detector inputs with paraphrased outlines while keeping their labels, so the labels must be fit through ideas.This follows the broader strategy of transforming training inputs to change which features a model learns without changing supervision.Source: §6 · p. 9

7 CONCLUSION

The paper concludes that ideas alone can distinguish human from AI writing, including when idea and prose authorship diverge. It reports transfer across languages and benchmarks, while identifying limits in labels, scope, extraction, and mixed-authorship evaluation.

7 CONCLUSION

  • IdeaLens meaningfully distinguishes AI and human writing through ideas alone, and tracks idea provenance when it differs from prose provenance.Unlike PROSELENS, its control model trained on the same labels but raw text rather than outlines.Source: §7 · p. 9
  • Across 22 datasets, IdeaLens transfers to 24 languages despite English-only training and outperforms the strongest existing open detector on shared-provenance documents.The conclusion also notes noisy training labels, binary handling of mixed ideas, weaker performance on short texts and some attacks, proprietary extraction, and only 50 TWICETOLD stories.Source: §7 · p. 9

A LIMITATIONS

The limitations concern noisy proxy labels, restricted data and extraction settings, and weaker or less certain results in several mixed-authorship and changing-generator conditions. The paper defines authorship by idea source, with prose source treated as a separate axis.

A LIMITATIONS

  • Training uses Pangram prose-authorship labels as proxies for idea authorship, and mixed-authorship documents remain less accurate than conventional ones.The authors do not measure how much of this gap is due to label noise or test ways to reduce it.Source: A LIMITATIONS · p. 19
  • The corpus covers English documents of at least 500 words; shorter documents are weaker, especially for AI-text detection (87.5% vs. 90.4% TPR).Source: A LIMITATIONS · p. 19, Table 2 · p. 19
  • Inference depends on proprietary frontier-model extraction, whose cost and availability may limit adoption; reading raw text instead loses idea-authorship sensitivity.Bulk Gemini 3.7 Flash extraction costs about $0.018 per document.Source: A LIMITATIONS · p. 19
  • Across five extractions, verdicts stay unchanged for 94% of randomly drawn out-of-domain documents but only 52.5% near the decision threshold.Averaging extractions might reduce variance, but the paper does not test it.Source: A LIMITATIONS · p. 19
  • For four newer generators, IdeaLens detects 51–79% at the global cut, versus 97% for older StoryScope generators; keeping it current requires newer-generator training.Pangram 4 does not show the same drop.Source: A LIMITATIONS · p. 19
  • IdeaLens detects 77.6% of DIPPER paraphrases in GEDE, below Pangram 4’s 95.1%; some attacks change content as well as style.The paper cannot fully separate that content change from a genuine weakness.Source: A LIMITATIONS · p. 19
  • Only 50 TWICETOLD stories test model ideas in human prose, so the results cannot establish how well this finding generalizes.Existing benchmarks lack this quadrant or approximate it with human edits retaining most model prose.Source: A LIMITATIONS · p. 20

B LABELS AND QUADRANTS

  • Idea authorship labels documents by who conceived their ideas, separately from who wrote the prose; mixed ideas remain binary-labeled despite unclear attribution granularity.The four quadrants distinguish shared human or model authorship (HH, AA) from mixed authorship (HA, AH).Source: A LIMITATIONS · p. 20, B LABELS AND QUADRANTS · p. 20

C BENCHMARKS

The evaluation spans external and constructed benchmarks, with arms classified by idea authorship and provenance quadrant. The data pipeline describes format-specific roles and how those roles are developed.

C BENCHMARKS

D DATA PIPELINE

  • The pipeline turns web documents into outlines using format-specific discourse roles, an extractor, and paraphrases.Source: D DATA PIPELINE · p. 23

D.1 ROLE DISCOVERY

  • Role discovery proposes format-specific roles, labels items from 200 documents per format with new roles as needed, then consolidates and prunes the vocabulary.For nonfiction, the final vocabulary has 42 roles; Table 6 shows eight and their definitions.Source: §D.1 · p. 23, Table 5 · p. 23, Table 6 · p. 23

D.2 OUTLINE EXTRACTION BENCHMARK

The outline-extraction benchmark favors six-shot Gemini 3.7 Flash as a lower-cost choice, while the pipeline’s paraphrases retain content and remove much surface wording. The section also reports outline statistics, a worked example, and the training-corpus scope.

D.2 OUTLINE EXTRACTION BENCHMARK

  • The benchmark compares 12 extractor configurations on 198 documents against reference outlines; its scores measure agreement with one reference, not correctness.Six-shot Gemini 3.7 Flash has 7.8% of item words copied from the document before paraphrasing, versus 2.6% without few-shot examples.Source: §D.2 · p. 24
  • Six-shot Gemini 3.7 Flash scores 0.774 alignment at about a third of the cost; GPT-5.6 Luna scores 0.796, the best configuration.All configurations reach coverage ≥0.986, and six examples raise alignment by 0.12–0.13 for every Gemini model.Source: §D.2 · p. 24, Table 7 · p. 24

D.3 OUTLINE STATISTICS

  • Across formats, paraphrasing preserves meaning while removing wording: raw and paraphrased outlines have cosine 0.93, and only 4.2% of word 4-grams survive.Source: §D.3 · p. 24, Table 8 · p. 25

D.4 PARAPHRASING REMOVES THE SURFACE SIGNAL

  • Paraphrased outlines retain content but share under 1% of their word 5-grams with the document; Pangram flags almost every human document as AI, while IdeaLens still separates them.The outlines remain close to their documents (cosine 0.76 paraphrased versus 0.79 raw), and 99.2% of raw items are closest to their own paraphrase.Source: §D.4 · pp. 25–26, Table 9 · p. 26

D.5 A WORKED EXAMPLE

  • The worked op-ed outline contains 17 extracted items and their paraphrases; the paraphrases keep each item’s content while changing its wording.Source: §D.5 · p. 26, Table 11 · p. 27

D.6 CORPUS

  • The training corpus contains 1,001,145 FineWeb documents, restricted to English texts of at least 500 words in eight long-form formats with fully human or fully AI Pangram labels.Source: §D.6 · p. 28, Table 12 · p. 28

E TRAINING, CALIBRATION AND THRESHOLDS

IDEALENS and PROSELENS use the same training data and recipe but differ in whether they read paraphrased outlines or raw documents. The section describes how verdict thresholds are calibrated and the costs of training and scoring.

E.1 MODEL CONFIGURATIONS

  • Both models train on the same 842,301 documents and Pangram labels, differing only in whether the input is a paraphrased outline or raw text.The neural models train for one epoch; checkpoints are selected using validation performance at low false-positive rates.Source: §E.1 · p. 28

E.2 CALIBRATION

  • The global threshold targets a 1% false-positive rate, using scores on 80,000 held-out human documents pooled across formats.The realized false-positive rate is reported on each evaluation’s human arm because an in-domain calibration need not hold out of domain.Source: §E.2 · pp. 28–29

E.3 THRESHOLDS AND BASELINES

  • Pangram 4 verdicts count a text as AI when its AI fraction plus half its AI-assisted fraction reaches 0.5.Detectors without a shipped threshold use cuts calibrated on the 80,000 human documents; Pangram 4 keeps its published threshold.Source: §E.3 · p. 29, Table 14 · p. 29

F COST

  • Scoring an average-length document costs about $0.06 through the standard API or $0.018 with batch prompt caching; outline extraction accounts for over 90% of the cost.PROSELENS costs about $0.0005 per document because it skips the format gate and extractor; training costs were $354 for IDEALENS and $710 for PROSELENS.Source: F COST · pp. 29–30, Table 16 · p. 30

G RESULTS BY QUADRANT

Across the provenance quadrants, the detectors differ in whether they respond to ideas or to prose: IDEALENS flags model ideas in human prose while largely leaving human ideas in model-written prose unflagged. Mixed-provenance and successive-edit evaluations further examine that distinction.

G.1 HUMAN IDEAS, HUMAN PROSE

  • 0.6% of human-idea, human-prose documents are flagged by IDEALENS on average across 19 evaluations, alongside 0.7% for PROSELENS and 0.3% for Pangram 4.The quadrant measures the false-positive rate on ordinary human writing.Source: §G.1 · p. 31, Table 17 · p. 31

G.2 MODEL IDEAS, MODEL PROSE

  • 91.1% of model-idea, model-prose documents are detected by IDEALENS, compared with 98.9% by PROSELENS, 97.3% by Pangram 4, and 81.8% by EditLens-3B.These are mean detection rates over 19 evaluations; the detectors that read text do best in this quadrant.Source: §G.2 · p. 32, Table 18 · p. 32

G.3 HUMAN IDEAS, MODEL PROSE

  • 5.3% of human-idea, model-prose documents are flagged by IDEALENS, versus 49.3% by PROSELENS, 56.2% by Pangram 4, and 27.9% by EditLens-3B.The quadrant includes documents in which a model wrote, rewrote, polished, or translated a person’s content.Source: §G.3 · pp. 33–34, Table 19 · p. 34

G.5 ALL DETECTORS AND BASELINES

  • Among open baselines, high true-positive rates on model-written text come with high false-positive rates on human text at their own thresholds.Table 21 organizes detector results by the four combinations of human or model ideas and human or model prose.Source: §G.5 · p. 35, Table 21 · p. 36

G.6 MIXED PROVENANCE

  • On four mixed-provenance benchmarks, IDEALENS places mean P(AI) between clear human and model arms; on AI in Peer Review, person-note reviews score 68.5 versus 77.1 for model reviews and 4.1 for human reviews polished by a model.PROSELENS and Pangram 4 instead score the person-note reviews like fully model-written reviews, at 99.3 and 89.8, respectively.Source: §G.6 · p. 36

G.7 OPAI-BENCH: SUCCESSIVE MODEL EDITS OF A HUMAN DRAFT

  • IDEALENS flags 0.2–0.6% of each OpAI-Bench edited version, while PROSELENS rises to 26.3% and Pangram 4 to 48.5% by v8.The benchmark applies eight model operations to human drafts; ideas remain the person’s even when a model writes almost every word.Source: §G.7 · p. 36

H.1 IDEASHIFT

  • On human sources, IDEALENS’s flag rate falls from 94.9% at IDEASHIFT level 0 to 6.8% at level 5, while model sources remain 96.6–99.2% across levels.Levels progressively supply more of a source’s plan; the share of human-source outline items retained rises from 0.30 to 0.99.Source: §H.1 · p. 37

H.2.1 OUTLINE ADHERENCE

  • In TWICETOLD, 94.1% of the original outline is retained in human rewrites, 20.1% of rewrite-outline content is extraneous, and IDEALENS’s mean P(AI) falls from 79.8 to 69.4 after renaming.The rewrites also preserve chronology: 90.7% of matched items remain in order.Source: §H.2.1 · p. 39, §H.2.2 · p. 39

H.2.3 RESULTS

IDEALENS flags many human-written stories based on AI outlines, and its results extend to multilingual evaluations and several robustness checks. The section also reports extraction variability, weaker detection of newer generators, and an analysis of human- and AI-associated features.

H.2.3 RESULTS

  • IDEALENS flags 68.0% of human rewrites from AI-story outlines and 96.0% of the model originals they were written from.The comparison tests whether a detector identifies model ideas in human prose; the originals serve as a control.Source: §H.2.3 · p. 39, Table 27 · p. 39

H.3 IDEASHIFT-X

  • IDEALENS detects 95.3% of level-0 documents and flags 0.7% of level-5 documents across 24 languages, while flagging 0.1% of the human pages.The documents are scored through English outlines, although IDEALENS was trained only on English.Source: §H.3 · pp. 39–40, Table 29 · p. 40

H.4 TRANSLATION

  • After translation into English, IDEALENS’s level-0 true-positive rate changes by under one point and human false-positive rate does not change; its original and translated scores correlate at 0.946.By contrast, Pangram 4’s level-5 false-positive rate rises from 51.3% to 97.0%.Source: §H.4 · p. 40, Table 30 · p. 40

I.1 PRE-CHATGPT WEB TEXT

  • On 10,000 pre-ChatGPT C4 pages, IDEALENS flags 0.01%, while PROSELENS, Pangram 4, and EditLens-3B flag none.The authors treat every flag as a false positive because the pages predate ChatGPT.Source: §I.1 · p. 41

I.2 ATTACKS AND HUMANIZERS

  • Across seven attacked-benchmark evaluations, IDEALENS’s true-positive rate drops by 6.4 points; attacks that lower it also change content, with missed DetectRL-X paraphrases retaining fewer outline items than caught ones.Character-level attacks are normalized away, while paraphrasing costs IDEALENS the most; the authors caution that changed ideas need not indicate failed idea detection.Source: §I.2 · p. 41, Table 31 · p. 41, §I.3 · p. 42

I.4 CONTENT-PRESERVING TRANSFORMATIONS

  • Verdicts survive outline paraphrasing, source sentence-level paraphrasing, and a change of extractor for 98.6%, 98.6%, and 98.8% of documents, respectively.Scoring 500-word excerpts lowers the true-positive rate from 96.5% to 91.4%; stripping role labels or shuffling items barely changes AUROC, while role sequence alone reaches 0.601.Source: §I.4 · p. 43, Table 36 · p. 43

I.5 FORMAT CLASSIFICATION AND EXTRACTION VARIANCE

  • Repeated extraction leaves verdicts unchanged for all 33 in-domain documents and 94.0% of random out-of-domain documents; near the decision threshold, only 52.5% keep the same verdict across runs.The format gate selects the same format in all three runs for 90.9% of in-domain documents and all five runs for 93.2% of out-of-domain documents.Source: §I.5 · p. 44, Table 41 · p. 45, Table 42 · p. 45

I.6 NEWER GENERATORS

  • IDEALENS detects 78.5%, 62.8%, 51.0%, and 51.2% of stories from four newer generators, below its 97.0% rate on five older StoryScope generators.Feature analysis identified 207 candidate features and retained 79 appearing across several groups; two raters showed similar HUMAN-ASSOCIATION SCORE distributions (r = 0.85), with substantial Gwet’s AC1 but not Krippendorff’s α agreement.Source: §I.6 · p. 45, §J.1 · p. 46, §J.2 · p. 46

J.3 VERIFYING THE LEGITIMACY OF OUR DISCOVERED FEATURES

The feature audit finds that most discovered features lean in their proposed human/AI direction. The section also documents prompts and procedures for outline construction, evaluation construction, and feature analysis.

J.3 VERIFYING THE LEGITIMACY OF OUR DISCOVERED FEATURES

  • 64 of 79 features aligned with their proposed direction, while one conflicted; proposed human and AI features also fired in the expected detector P(AI) quintiles.The HUMAN-ASSOCIATION SCORE compares detector human-likeness for items where a feature fires versus does not fire; the reported agreement is stronger for aggregate directions than interchangeable item-level ratings.Source: §J.3 · pp. 46, 49, Figure 7 · p. 48, Table 46 · p. 49

K PROMPTS

  • The prompt materials include full prompts, output schemas, extraction exemplars, and reference outlines, with exemplars drawn from the training corpus.The prompts and exemplar documents are released with the corpus.Source: K PROMPTS · p. 51

K.1 OUTLINE PIPELINE

  • Outline construction extracts ordered role-labelled content commitments, paraphrases each field without changing propositions, and classifies the resulting role-labelled items.Role discovery proposes functions, assigns roles in context, and consolidates candidates into a cross-topic vocabulary.Source: §K.1 · p. 51

K.2 EVALUATION CONSTRUCTION

  • Evaluation briefs reveal progressively more of a source outline across levels 0–4, alongside full-outline realization and a translation task requiring marked segments to be translated in order.Level 5 asks writers to realize every outline item in order without adding content.Source: §K.2 · p. 52

K.3 FEATURE ANALYSIS

  • Feature discovery contrasts role-matched human- and AI-source items, then rates each item independently using its format, role label, and text.Matching role counts across the two sides prevents role-usage differences from serving as a feature.Source: §K.3 · p. 52

K.4 TWICETOLD CURATION

  • For TwiceTold, writers were asked to produce 500-word stories closely following an outline of 8–12 plot items.The prompt permits minimal additions for coherence while requiring the original event order and outline details to be retained.Source: §K.4 · p. 53

K.4.1 ANNOTATION DETAILS

  • Participating computer science graduate students consented to research use and spent approximately 1–2 hours writing each story.Writing time depended on outline length and complexity.Source: §K.4.1 · p. 53

L WHERE THE REMAINING TABLES ARE

Loading 2610.06778v1…