PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing Narrative Content in Web-scale LLM Pretraining Data

Teagan Johnson, Elliott Ash, Andrew Piper, Maria Antoniak

arXiv:2606.19468v1cs.CL

TL;DR

Narrative composition in LLM pretraining data has received little systematic attention. This paper develops a fine-grained framework and models narrative structure across web-scale data, finding that it is multidimensional and unevenly distributed across sources and topics.

  • Problem

    The narrative composition of heterogeneous LLM pretraining corpora remains largely unmeasured across sub-corpora, genres, and narrative-relevant features.

  • Method

    The authors annotate 400 passages and train NarraBERT classifiers to characterize 11 narrative dimensions across a large-scale pretraining-data corpus.

  • Results

    Narrative dimensions form interpretable axes, vary unevenly across sources and topics, and show substantial within-category variation.

  • Takeaways & Limitations

    Source-level and topic-level curation labels are too coarse to capture narrative diversity in pretraining data.

  • Takeaways & Limitations

    The study does not establish direct causal links between pretraining narrative composition and downstream model behavior.

Abstract

from arXiv · show

The narrative composition of web-scale LLM pretraining corpora remains largely unexplored even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After sampling and annotating a diverse set of 400 passages, we finetune and validate NarraBERT, a RoBERTa-based model for fine-grained narrative prediction. We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma. We find (i) narrative structure is measurable at scale across extremely heterogeneous data, (ii) we uncover a continuous, multidimensional narrative structure underlying web text, and (iii) narrative qualities are unequally distributed across pretraining sources and topics in ways that current curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data and for studying how data composition affects narrative reasoning tasks. We publicly release NarraDolma and NarraBERT.

1 Introduction

The study addresses the largely unexamined narrative composition of web-scale pretraining data by analyzing DOLMA through a theory-grounded framework and scalable prediction pipeline. It shows that narrativity is continuous, multidimensional, and unevenly distributed across sources and topics, with implications for data curation.

  • Motivation: Narrative composition in pretraining corpora has received almost no systematic attention despite its potential influence on LLM narrative capabilities.Existing curation examines quality, toxicity, deduplication, and topic distribution, but not how narrative content is distributed across sub-corpora and genres.
  • Approach: The study analyzes DOLMA, a 3-trillion-token open corpus spanning twelve sub-corpora, defining narrative as structured event sequencing through an agent’s perspective in a grounded setting.The framework builds on narrative-theory accounts while treating narrative as measurable across highly heterogeneous data.
  • Contributions: The annotation framework operationalizes 11 narrative dimensions across events, agency, and setting.These dimensions provide an interpretable basis for characterizing narrative content at passage level.
  • Contributions: NARRABERT predicts all 11 narrative dimensions and is validated against both human and LLM annotations.The classifier enables scalable measurement of narrative qualities in web-scale pretraining data.
  • Contributions: NARRADOLMA contains 3M labeled passages sampled from DOLMA through a principled stratified pipeline.The resulting dataset supports large-scale analysis of narrative qualities in the corpus.
  • Findings: Narrativity forms a continuous, multidimensional structure and is unequally distributed across pretraining sources and topics.Because sources are not narratively homogeneous, source-level inclusion and weighting decisions are too coarse to capture narrative diversity.

2 Related Work

Prior work has developed narrative detection methods and datasets, but largely uses binary or coarse annotations. This work extends that literature with 11 fine-grained narrative dimensions and treats narrative qualities as an additional axis for characterizing pretraining data.

  • Recent NLP studies classify narrativity in online health, health misinformation, and suicide bereavement forum texts.
  • STORYSEEKER provides span-level binary narrative annotations from Reddit, while NARRADETECT covers 18 book and online genres with binary labels and a small five-point corpus.
  • This work annotates 11 fine-grained narrative dimensions on a 5-point Likert scale, providing richer yet computationally tractable narrativity characterization.
  • Prior pretraining-data composition research varies topic, quality, and format, but does not treat narrative structure as a data dimension.

3 Our Annotation Framework

The annotation framework treats narrativity as a scalar, multidimensional construct and translates narrative-theoretic concepts into interpretable ratings. It covers character agency, storyworld realization, and event relations through defined dimensions and annotation procedures.

  • Theoretical basis: Narrativity is modeled as a spectrum of degree or intensity rather than a binary narrative/non-narrative distinction.This scalar perspective motivates rating-based annotation of heterogeneous web text.
  • Agency: Five agency dimensions rate character centrality on a 5-point Likert scale: focalization, internal emotion, internal cognition, change of state, and conflict.The dimensions distinguish experiential perspective, emotional and cognitive interiority, transformations, and multiple forms of opposition.
  • Setting: Storyworld realization is rated on four experiential dimensions: concreteness, temporal grounding, spatial grounding, and sensory detail.Temporal grounding includes historical and cyclical anchoring, spatial grounding combines geographic and proximate anchoring, and sensory detail is distinct from concreteness.
  • Events: An event is a singular, bounded occurrence with identifiable participants and timing; hypothetical, negated, and future events are excluded.Event relations then assess temporal sequencing, causal links, and event density across the passage.
  • Events: The event-relation pipeline detects triggers with a DEBERTA-based model fine-tuned on Sims et al. (2019), achieving an F1 of 0.85 on the web-scale corpus.Annotators order events by start points and classify causal relations using defined categories, including direct cause and enablement.

4 Sampling from Dolma

The study samples narrative passages from twelve heterogeneous Dolma sources through a four-step pipeline combining passage extraction, narrative scoring, topic classification, and stratified sampling. This produces a 400-passage gold dataset and the approximately 3M-passage NarraDolma dataset while preserving source proportions and balancing Common Crawl topics.

  • Source coverage: Dolma sampling covers twelve sources spanning web pages, news, encyclopedic text, books, and social media.Dolma is an open corpus containing over 3 trillion tokens.
  • Initial extraction: Approximately 17M three-sentence passages are extracted from approximately 5M unique documents across raw Dolma v1.73 shards.Passages are allocated proportionally across sources using Table A7 weights and segmented with NLTK sent_tokenize.
  • Scoring and classification: Each passage receives a continuous narrative confidence score p ∈[0, 1] from a DEBERTA-based binary classifier, while Common Crawl passages receive labels across 24 topics.Topic labels balance final Common Crawl samples across subject domains, limiting domination by high-frequency topics such as news.
  • Final samples: The 400-passage gold dataset preserves inter-source proportions, selects 85% with narrative scores p > 0.50 and 15% without score filtering, and balances Common Crawl topics.The passages are split into two non-overlapping gold sets for agency-related annotation.
  • Final samples: Approximately 3M NarraDolma passages spanning approximately 785K unique web documents follow the same stratified procedure, shifting unfiltered sampling from 15% to 25%.The shift increases coverage of non-narrative text while preserving inter-source proportions.

5 Datasets

The authors build narrative datasets through human annotation, validated LLM labeling, and distilled classifiers. NARRABERT then scales annotation to roughly 3M passages across Dolma while producing multidimensional narrative features.

  • Human annotation: The gold dataset contains 400 passages annotated across agency, setting, and event-relation tasks, with verification subsets from additional authors.Verification covered N = 100 for agency, N = 30 for setting, N = 251 for events, and N = 70 for an additional setting annotator.
  • Human annotation: Human agreement is reasonable: agency mean α = 0.76, setting mean α = 0.70, and event relations mean κ = 0.68 and mean F1 = 0.91.Agency and setting also have mean MAE values of 0.62 and 0.55, respectively.
  • LLM labeling: The authors compare CLAUDE SONNET 4.6, QWEN3-235B-A22B, and GEMMA4-31B against human labels, selecting GEMMA for large-scale labeling.Across the three models, agency and setting mean α = 0.71, while event relations mean F1 = 0.78 and mean κ = 0.56; GEMMA is chosen for cost effectiveness and open-source availability.
  • NARRABERT: NARRABERT distills GEMMA labels into two RoBERTa classifiers trained on 5K passages, enabling low-cost labeling of millions of texts.Agency and setting are framed as regression tasks, while event-relation dimensions are binary classification tasks.
  • Scaling up annotation: NARRABERT annotates ∼3M passages across ∼785K unique web documents and all 12 Dolma sub-corpora, producing a 12-dimensional narrative feature vector.It generates nine Likert scores for agency and setting, while event detection supports temporal sequencing, causal density, and event density.

6 The Narrative Landscape of Pretraining Data

The narrative landscape of pretraining data is coherent yet highly heterogeneous: categories occupy distinct regions, while narrative structure varies substantially within categories. PCA reveals three interpretable axes—interiority, grounded eventfulness, and storyworld texture—that show different category concentrations rather than a single narrative scale.

  • Category profiles: Categories form distinct narrative clusters, including high-interiority sources and concreteness- and sensory-detail-heavy rendered-entity topics.Reddit, Literature, Gutenberg, and Adult emphasize focalization, emotion, and cognition while remaining low on setting; Food & Dining, Fashion & Beauty, Travel, Home & Hobbies, and Art & Design emphasize concreteness and sensory detail.
  • Within-category variation: 0.87 is the average within-category standard deviation across 12 narrative features, indicating that category labels remove only a small fraction of narrative variance.Within-category variability is lower for Wikipedia (0.68) and Software Dev. (0.71) but higher for Reddit (1.07) and Literature (1.00).
  • Latent narrative axes: ~72% of total variance is captured by the first three principal components, representing narrative interiority, grounded eventfulness, and storyworld texture.The components respectively emphasize focalization, emotion, and cognition; change of state, conflict, event density, and temporal and spatial grounding; and concreteness, sensory detail, and spatial grounding.
  • Category concentrations: Interiority, grounded eventfulness, and storyworld texture concentrate in different categories, with Reddit and Gutenberg high on interiority, Crime & Law and Wikipedia high on eventfulness, and Food & Dining and Travel high on texture.Software-related categories trail on eventfulness or texture, while Wikipedia, History, Politics, and Crime & Law contribute little interiority.
  • Cross-axis structure: No category dominates all three axes, showing that narrative structure is a multidimensional configuration rather than a single quantity.Gutenberg is high on interiority and texture but moderate on eventfulness; Reddit concentrates on interiority; Wikipedia and Crime & Law lead eventfulness while remaining nearly invisible on the other axes.

7 Conclusion

The paper introduces NarraDolma and NarraBERT to measure 11 narrative dimensions across millions of pretraining passages. It finds that narrative structure is multidimensional, unequally distributed across categories, and varies substantially within them.

  • Contributions: NarraDolma characterizes narrative structure in pretraining data, while NarraBERT efficiently classifies 11 narrative dimensions.The framework is grounded in narrative theory and validated through human annotation and LLM-assisted labeling.
  • Contributions: ∼3M passages across ∼785K documents receive narrative feature vectors through the paper’s annotation and classification pipeline.The annotation framework was applied at large scale after validation through human annotation and LLM-assisted labeling.
  • Empirical findings: Narrative dimensions covary into three interpretable axes: interiority, grounded eventfulness, and storyworld texture.These axes summarize the paper’s first core empirical finding about the structure of narrative dimensions.
  • Empirical findings: Narrative axes are unequally distributed across categories, while substantial within-category variation makes source-level and topic-level labels too coarse to capture narrative diversity.The paper argues that current curation practices neither measure nor account for these distributional differences.
  • Implications and future work: The results establish narrative structure as a measurable, multidimensional property of pretraining data rather than a single “narrativity” quantity.The paper proposes controlled data-mixing experiments and intermediate-checkpoint analyses as directions for studying links to downstream narrative capability.

8 Limitations

The study’s conclusions are limited by the construction and annotation of NarraDolma, uncertainty in event-structure predictions, restricted language and theory coverage, and the absence of causal evidence about downstream model behavior.

  • Sampling and annotation: NarraDolma is a stratified, narrative-overrepresenting subsample of roughly 3M passages from a corpus exceeding 3 trillion tokens, so absolute prevalence estimates do not transfer to raw Dolma.Relative structural patterns remain informative despite this sampling design.
  • Sampling and annotation: Human annotation covered only 400 passages across 11 dimensions, relied on one author for primary validation annotation, and showed modest agreement for temporal ordering (κ = 0.60).The fine-grained framework and months-long calibration process constrained sample size.
  • Model and evaluation: NarraBERT underperforms its LLM teacher on event relations, with F1 of 0.58 temporal and 0.68 causal, making event findings noisier than agency and setting findings.Sparse event annotations also prevented fully independent evaluation across the LLM and classifier validation stages.
  • Scope: The analysis covers only English text and one theoretical operationalization of narrative, so other languages, traditions, and frameworks may reveal different structures.This limitation narrows the generality of the reported narrative patterns.
  • Causal inference: The study does not establish direct causal links between pretraining narrative composition and downstream model behavior.Controlled data remixing and checkpoint analysis remain proposed directions for future work.

Ethical Considerations

The study uses the openly licensed DOLMA dataset, which contains potentially toxic, explicit, and personal web data that was encountered during annotation. Releasing NarraDolma therefore poses risks of amplifying harmful or sensitive narrative styles, addressed through provenance, research-oriented access, and use-case guidance.

  • DOLMA is an open English-language pretraining dataset released under the Open Data Commons Attribution License (ODC-By).
  • DOLMA contains toxic, explicit, and personal web data, including passages not highlighted in the paper but encountered during annotation.The authors note that these narrative qualities may have implications for safe story generation.
  • Institutional review board review was not required, and authors reviewed all AI-assisted data, annotation, analysis, citation, statistical, interpretation, and writing decisions.
  • Potential Risks: NarraDolma could enable identifying, filtering, or prioritizing emotionally intense, personal, violent, or explicit narratives for harmful or manipulative training mixtures.The dataset includes passage text and narrative feature annotations, creating risks of retrieving sensitive personal disclosures.
  • Potential Risks: The authors mitigate these risks by releasing the dataset for research and auditing, documenting provenance and intended use, preserving links to DOLMA sources, and discouraging sensitive-narrative amplification.

A Datasets & Validation · A.1 Agreement Scores · A.2 Automatic Lexical Features

The appendix documents NarraBERT’s validation evidence, including label distributions, agreement scores, and construct-validity checks against automatically extracted lexical features. It also specifies the tools and processing used to compute those lexical features.

  • A Datasets & Validation: The appendix reports gold-label distributions, inter-annotator and model agreement, and correlations between NarraBERT predictions and surface lexical features.
  • A Datasets & Validation: Figure A1 shows distributions of gold annotated labels across the 11 manually annotated features.
  • A Datasets & Validation: Figure A2 shows distributions of the full NarraDolma corpus labels.
  • A.1 Agreement Scores: Tables A1–A3 report per-dimension agreement scores for each validation stage.Table A1 covers human agreement, while Table A2 covers agreement between each of three LLMs and gold split A.
  • A.1 Agreement Scores: Agency and setting agreement uses MAE and Krippendorff’s α, whereas event-relation agreement uses Cohen’s κ and macro F1.Human agreement is measured on a 100-passage overlap set for agency and setting and on the full event annotation set; event annotation begins with pre-highlighted spans.
  • A.2 Automatic Lexical Features: Construct validity was assessed by Pearson-correlating each NarraBERT label with 18 automatically extracted lexical features.The features were produced using spaCy, the Brysbaert concreteness lexicon, and the CardiffNLP RoBERTa sentiment classifier.
  • A.2 Automatic Lexical Features: spaCy token- and entity-level features were extracted in one pass with en_core_web_lg, using sentence boundaries while disabling the parser.Concreteness used mean matched-token scores, with coverage recorded but excluded from correlations; sentiment was truncated to 512 tokens and retained positive, neutral, and negative probabilities separately.

B Analysis Details · B.1 PCA Extreme Examples

The appendix details the corpus analysis through feature correlations, PCA and UMAP analyses, score distributions and dispersion, representative passages, and classification tests. It also presents passages with high scores on the first three principal components.

  • B Analysis Details: The appendix expands corpus analysis beyond the main results by examining feature correlations, PCA structure, UMAP overlays, score dispersion, representative passages, and classification tests.These analyses probe how narrative features organize and separate corpus sources and categories.
  • B Analysis Details: Figure A4 reports the Pearson correlation between every feature in NarraDolma.
  • B Analysis Details: Figure A5 overlays the first three principal components on a UMAP reduction of 20,000 randomly sampled passages.
  • B Analysis Details: Figure A7 shows the distribution of principal-component scores across the full NarraDolma corpus.
  • B Analysis Details: Figures A8 and A9 report the standard deviation of each principal component by source and category, respectively.
  • B.1 PCA Extreme Examples: Table A6 presents passages with high scores on each of the first three principal components.

B.2 Classification Tests … TEMPORAL ORDERING

The appendix validates narrative-feature separability, documents the sampling, training, annotation, and event-detection pipeline, and specifies the annotation framework for agency, setting, and temporal ordering. Together, these procedures support large-scale narrative analysis with interpretable labels and validated automated components.

  • C Sampling & Pipeline: The sampling appendix records Dolma source and topic distributions, exclusions, configuration parameters, and the allocation of gold annotations and NARRADOLMA.Figures A16–A17 show source and topic distributions, Table A7 reports target, gold, and NARRADOLMA allocations, and Table A8 lists excluded sources.
  • C.3 Hyperparameters: RoBERTa training uses nine task-specific regression heads, 5-fold crossvalidation, AdamW with learning rate 2×10−5 and weight decay 0.01, batch size 16, maximum length 200, and early stopping.Inference uses batch size 64 and clips regression outputs to the valid Likert range [1, 5]; LLMs receive identical prompts and elicit integer scores on a 1–5 scale.
  • C.4 Event Span Detection: The event pipeline combines LitBank-based DeBERTa event spans with spaCy verb spans, removes overlapping verb spans, and selects exactly one adjacent pair per passage for manual annotation.Passages contain 2.4 event spans and 6.9 verb spans on average; the LitBank detector achieves F1 of 85% and precision of 90%.
  • D.1 LLM-Generated Document Summaries for Human Annotation: Prompts for agency and setting are held constant across Claude Sonnet 4.6, Qwen3-235B-A22B, and Gemma4-31B, and across validation and large-scale annotation.The agency prompt rates nine dimensions across Agency and Setting on a 1–5 scale, while the document-summary prompt instructs Llama-3.1-8B-Instruct to describe only explicitly present webpage content.
  • AGENCY DIMENSIONS: Agency dimensions measure the presence and centrality of character agency as written, while setting dimensions capture how texts construct place, time, and physical presence.The framework distinguishes textual evidence from inference and operationalizes agency with five dimensions and setting with four dimensions.
  • TEMPORAL ORDERING: Concreteness rates whether language is explainable through physical demonstration versus categories, principles, relationships, institutions, or ideas, while temporal ordering identifies which event started first.Temporal labels include span1_first, span2_first, simultaneous, same_event, and too_hard_to_tell; event-relation prompts are held constant across Claude Sonnet 4.6 and Gemma4-31B.
Loading 2606.19468v1…