Source-linked AI summary
Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data
Marharyta Shvets
TL;DR
Existing annotation approaches compare different people, leaving self-reproduction over time less measured despite its relevance to training-data quality. The paper compares repeated readings by an expert human and three LLM families across three Sumerian myths, finding a human boundary-label split and no consistent model counterpart. It argues that this pattern can inform data-quality assessment and contamination checks.
Problem
Annotation methods commonly measure agreement or disagreement between different people, while how well one reader reproduces their interpretation over time is measured far less often.
Source: §1 · p. 1Method
One expert human and three LLM families annotate three Sumerian myths by segmenting them and assigning each segment one of seven function states.
Source: §3.3 · p. 2, §3.4 · p. 3Results
Takeaways & Limitations
The authors argue that differences between boundary and label reproducibility can serve as a data-quality signal and a contamination check.
Source: §1 · p. 1The dissociation rests on one expert reader across three myths, so a panel is needed to establish whether it is typical of experts or specific to this reader.
Source: Limitations · p. 6
Abstract
from arXiv · showhide
Data quality now matters as much as compute for training language models. Much training data comes from human annotation of text, and interpretive annotation has no ground truth that could settle what is "accurate". Two lines of work respond to this. One combines annotators into a "ground truth" and measures how well they agree with each other; the other treats their disagreement as a signal. Both compare different people at one point in time. We measure something else: how well one reader reproduces their own reading of the same text over time. One expert human reader and three LLM families segmented three Sumerian myths and labelled the causal function of each segment with one of seven states. Across runs months apart, the human cut the text in much the same places but named the segments differently, in every myth. The models show no such consistent pattern: their gap between the two layers is positive in some myths and negative in others, and its size varies. The human's label changes are not random: the runs go through much the same functions but start them one step apart, while model runs start them at the same places. We argue that this pattern is a usable measure of data quality and a contamination check: a "human" annotation whose labels are as stable as its boundaries, and whose functions start in sync, looks like a model's.
1 Data quality, and where it is measured
Training-data quality matters, but interpretive annotations lack a ground truth for deciding what is accurate. This section argues that checking whether one reader reproduces their own reading reveals structure that agreement or disagreement between people misses.
1 Data quality, and where it is measured
- Built structure records where events end and their causal functions, rather than describing the text through paraphrase or summary.Unlike descriptions, this structure can be checked formally for reproducibility.Source: §1 · p. 1
- Existing approaches either combine annotators into a ground truth and measure agreement or treat disagreement as informative, but both compare different people.Self-agreement over time is measured far less often and can distinguish inconsistency from real label variation.Source: §1 · p. 1
- Across repeated readings, the human’s boundaries mostly persist while labels change; the three LLM families repeat both layers, and the human’s label changes follow a one-step shift.The paper calls the split between boundary and label behavior a dissociation and argues both patterns can signal data quality beyond existing approaches.Source: §1 · p. 1
2 What is being measured: attributed causality, no ground truth
The paper measures attributed causal structure: a reader’s interpretation and naming of a text’s segments, not facts read directly from the text. Because there is no ground truth, a reader’s structural signature is meaningful only if it reproduces within that reader.
2 What is being measured: attributed causality, no ground truth
- Causal relations link events that a reader attributes, rather than interventional effects or facts contained in the text.The structure is imposed, not read off, so there is no ground truth.Source: §2 · p. 2
- The human is treated as a subject bringing their own logic to the text, while the model is treated only as a reader.The metrics test for traces of the human’s logic rather than assuming those traces exist.Source: §2 · p. 2
- A causal signature is a stable individual way of attributing structure, and it counts as such only if it reproduces within one reader.Self-consistency measures this for humans and models in the same way.Source: §2 · p. 2
3 Data and method
The study compares repeated human and model readings of three Sumerian myths using a shared seven-state annotation scheme. It projects annotations onto common units and uses chance-corrected and cross-text metrics to examine consistency and structure.
3.1 Corpora
- Three Sumerian myths serve as corpora; although their English translations may be in model training data, there is no tradition of dividing them into episodes for a model to copy.The texts have few retellings or popular commentaries, and the paper therefore treats them as showing logic readers build rather than extract from established episode divisions.Source: §3.1 · p. 2
3.2 Readers and runs
- Three model families each annotate every myth in ten fresh sessions, while one expert human annotates each myth three times, five weeks to ten months apart.Model runs are nearly contemporaneous, whereas human runs are months apart, which the paper treats as different kinds of reliability.Source: §3.2 · p. 2
3.3 The annotation instrument (fixed schema)
- Readers assign each segment one of seven function states and transitions; there is no answer key, so the study measures how one expert’s reading reproduces over time.The same state names are used across runs, and because the human authored the scheme, label changes cannot be explained by unfamiliarity with its categories.Source: §3.3 · pp. 2–3
3.4 Projection onto shared units
- To compare different segmentations, each reading is projected onto shared text-line or aligned-block units, with boundaries retained as a separate layer.Lacunae, manuscript variants, and ellipsis-only lines are excluded, and comparisons use units annotated in both runs.Source: §3.4 · p. 3
3.5 Metrics (chance-corrected)
- Five metrics assess within-reader agreement, disagreement concentration, neighbour-licensed relabeling, phase alignment, and whether function-transition patterns transfer across myths.The metrics use Cohen’s κ, random baselines for label patterns, and a held-out myth prediction test; phase shift is exploratory, while transferable grammar is the only cross-myth metric.Source: §3.5 · p. 3
4 Results
Across all three myths, the human’s boundaries reproduced better than labels, while model boundary–label gaps varied by myth. Human label disagreement followed a one-step shift pattern, and models transferred function order across myths better than the human.
4.1 The dissociation is a human signature
- The human’s boundary–label κ gap was +0.440 for Inanna and Enki, +0.452 for Inanna’s Descent, and +0.298 for Gudea.Boundaries reproduced better than labels in every myth; Table 1 reports within-human self-consistency and Table 2 reports the boundary–label gap by reader and corpus.Source: §4.1 · p. 3, Table 1 · p. 4, Table 2 · p. 4
- Model boundary–label gaps varied in sign across myths; Opus and Gemini were negative on Inanna and Enki, while Gemini’s Gudea gap was +0.291.Unlike the human’s consistently positive gap, models showed no consistent gap pattern.Source: §4.1 · p. 4, Table 2 · p. 4
4.2 Why the time between runs does not explain it
- Because human runs were months apart and model runs almost simultaneous, human–model score levels are compared only qualitatively; timing cannot explain the within-human layer gap.The dissociation compares boundaries and labels within the same human runs, at the same spacing.Source: §4.2 · p. 4
4.3 How the human’s labels disagree: a shift in phase, not a map of hard spots
- With disagreement fixed, units where all three human runs disagreed were rarer than random scatter, but the shuffle baseline also increased total disagreement.The reported ratios were 0.85, 0.88, and 0.72; the passage cautions that these ratios only repeat that labels agree more than chance.Source: §4.3 · p. 4
- In Gudea, the human chose a neighbouring passage’s label in 60% of relabeling events versus 25% chance; all three model families did likewise, often more strongly.The other two myths had too few human events to decide; the paper treats this pattern as a property of the scheme, not a human signature.Source: §4.3 · pp. 4–5
- Human function starts were often one piece apart rather than at the same place, while model triples had same-place starts well above chance.For the human, one-piece lags occurred 2.5 and 1.8 times more often than chance in Inanna’s Descent and Gudea; lags of two or three pieces were at or below chance. On Inanna and Enki, human starts instead aligned at the same places.Source: Figure 1 · p. 4, §4.3 · p. 5
4.4 A transferable event-grammar: models carry it across myths, the human does not
- Mean transferable-grammar gain was +0.04 bits per transition for the human, versus +0.44 for Opus, +0.17 for Qwen, and +0.13 for Gemini.Opus transferred to all three held-out myths; the human transferred only to Gudea, and the overall model-over-human ranking held with three model runs under either tested selection.Source: §4.4 · p. 5
5 The dissociation as a data-quality axis
The two annotation layers answer different data-quality questions: boundaries can provide reproducible structure while labels capture variable interpretation. Their dissociation and the human’s one-step phase shift offer checks that inter-annotator agreement cannot provide.
5 The dissociation as a data-quality axis
- Segmentation supplies reproducible structure, while labeling is where interpretation and human variation appear.The section frames the two layers as answering different data-quality questions.Source: §5 · p. 5
- Repeated labels on stable segmentation retain one expert’s causal-function logic, including places where more than one reading is legitimate, rather than a consensus answer.Combining annotators averages readers toward a “ground truth” and erases this interpretive variation.Source: §5 · p. 5
- The human signature is a gap between stable segmentation and variable labels, while models whose labels are as stable as their boundaries resemble the model pattern.The phase shift offers a finer check: human runs tend to start the same functions one piece apart, whereas model runs start them at the same places.Source: §5 · p. 5
- Table 4 compares where runs start the same function with a baseline that shifts one run along the text.For models, the table reports the most extreme value across sets of three runs per myth.Source: Table 4 · p. 6
- Table 5 measures held-out-myth event-grammar transfer as mean gain in bits per transition over label frequencies alone.Higher gains indicate more transferable event-grammar.Source: Table 5 · p. 6
6 Discussion
The discussion distinguishes reproducible segmentation from variable labeling, locating the human–model difference in interpretation rather than structure. It cautions that stable structure does not establish understanding and that stronger claims about model reading require broader evidence.
6 Discussion
- Segmentation reproduces across readers, while labeling is where interpretation and the human–model difference appear.The discussion relates segmentation to bottom-up and top-down contributions to event segmentation.Source: §6 · p. 6
- The transfer result fits models carrying a fixed, templatelike event-grammar and a human grammar that adapts to each text.A stronger claim that models copy reading organization without intentions would require a panel of readers and matched sampling.Source: §6 · p. 6
- Stable structure is not evidence of understanding: tests rewarding stable, templated answers can miss variable, individual human interpretation.Such tests score models on the layer every reader reproduces.Source: §6 · p. 6
7 Next steps
The next steps are to test whether the observed patterns extend beyond one reader, model reading with separate layer variances and reader-specific phase shifts, and compare human and model reliability over shorter intervals.
7 Next steps
- A panel of expert readers with at least two time-spaced runs each could test whether the boundary–label gap and one-step phase shift recur across people.The panel would distinguish features shared across readers from features specific to the current reader.Source: §7 · p. 6
- With that data, a generative model could assign separate variances to boundaries and labels and a phase shift per reader, then be tested on held-out readers.The authors do not fit it here because one reader and three runs would only reproduce descriptive statistics.Source: §7 · p. 6
- A short-interval retest could separate reinterpretation over months from ambiguity in the moment and make human–model reliability directly comparable.Source: §7 · p. 6
Limitations
The findings are bounded by a single human reader, an uneven human–model timing comparison, and limits in the metrics and data. The phase-shift result is descriptive rather than a calibrated effect.
Limitations
- The human dissociation is a case study based on one expert reader, replicated across three myths rather than independent pairs of runs.Pairs within a myth share runs, making their p-values too optimistic; a panel is needed to test whether the pattern generalizes.Source: Limitations · p. 6
- Human runs were months apart while model runs were nearly simultaneous, so the human–model comparison is qualitative.The main within-human dissociation compares the two layers at the same spacing.Source: Limitations · p. 6
- Equal-run comparisons remain partly unfair because even the most different model runs agree more than the human’s three runs.B1 uses only three runs, and B2 excludes units without a majority label, which are about 30–35% of units.Source: Limitations · p. 6
- The phase-shift counts are small, pairs are counted in both directions, and pieces range from one to thirty lines, so the effect is descriptive, not calibrated.Permutation p-values use 1,000 draws, or 200 per model triple, making values near 0.01 bounds rather than exact values.Source: Limitations · p. 7
Use of AI assistants
Claude assisted with code, statistical computations, and editing, while the author retained responsibility for the research design, annotations, metrics, and interpretation.
Use of AI assistants
- Claude helped write analysis code, run statistical computations, and edit the text; the author verified the reported results.The code is released for inspection.Source: Use of AI assistants · p. 7
- The author determined the research question, annotation scheme, human annotations, metric choices, and interpretation of results.Source: Use of AI assistants · p. 7
- Claude Opus was also a studied reader, but its annotations came from separate fresh anonymous sessions and were not produced or altered by the assistant.Source: Use of AI assistants · p. 7