PaperlayerView exact revision on arXiv

Source-linked AI summary

On the origin of long-range correlations in texts

Eduardo G. Altmann, Giampaolo Cristadoro, Mirko Degli Esposti

arXiv:1207.0658v1physics.data-ancs.CLphysics.soc-ph

TL;DR

Long-range correlations are robustly observed in literary texts, but their sources remain poorly understood. Combining calculations with text analysis, the paper traces how correlations flow from semantic topics to lower linguistic levels, finding bursty event sequences near relevant topics and preserved long-range correlations as burstiness disappears at lower levels.

  • Problem

    How long-range correlations arise across linguistic levels remains unclear, and the choice of observable affects which source memory is measured.

    Source: A. The importance of the observable · p. 2

  • Method

    The paper combines theoretical calculations and text-data analysis to distinguish burstiness from correlations between inter-event intervals across linguistic levels.

    Source: §III · p. 6, §VI · p. 12

  • Results

    Some highly topical words show long-range correlations mostly due to burstiness, while lower-level projections preserve correlations as burstiness disappears.

    Source: §III · pp. 6–7

  • Takeaways & Limitations

    The mechanism offers a general account of how correlations can cascade across hierarchical settings.

    Source: §III · p. 6

    Further studies are needed to connect concept-level correlations to lower-level observations and distinguish their respective sources.

    Source: §III · p. 6

Abstract

from arXiv · show

The complexity of human interactions with social and natural phenomena is mirrored in the way we describe our experiences through natural language. In order to retain and convey such a high dimensional information, the statistical properties of our linguistic output has to be highly correlated in time. An example are the robust observations, still largely not understood, of correlations on arbitrary long scales in literary texts. In this paper we explain how long-range correlations flow from highly structured linguistic levels down to the building blocks of a text (words, letters, etc..). By combining calculations and data analysis we show that correlations take form of a bursty sequence of events once we approach the semantically relevant topics of the text. The mechanisms we identify are fairly general and can be equally applied to other hierarchical settings.

I. THEORETICAL FRAMEWORK · A. The importance of the observable

The framework treats a literary text as the output of a stationary, ergodic source over a finite alphabet and analyzes correlations through chosen observables. Because the observable determines what source memory is measured, the authors use local binary mappings based on matching conditions to avoid introducing spurious correlations.

I. THEORETICAL FRAMEWORK

A. The importance of the observable

  • An observable maps the symbolic text sequence to a numeric sequence, such as a sequence of zeros and ones.Source: A. The importance of the observable · p. 2
  • The framework restricts observables to local mappings, with each output determined by a finite neighborhood of symbols.Source: A. The importance of the observable · p. 2
  • Autocorrelation is defined for the chosen observable, with lag measured in symbols and averages taken over sliding windows.Source: A. The importance of the observable · p. 2
  • The choice of observable determines which source memory is quantified, so long-range correlations characterize the text as a whole only if a class of observables shares the same asymptotic behavior.Source: A. The importance of the observable · p. 2
  • Earlier analyses used varied encodings, but their shared properties and the meaning of encoding-dependent changes in correlations remain unclear.Examples include Huffman codes, arbitrary binary symbol encodings, frequency-rank, and word-level parts of speech.Source: A. The importance of the observable · p. 2
  • To avoid spurious correlations, the authors use binary mappings that mark a symbol as one exactly when a local matching condition is satisfied.For example, the condition can be that the symbol is a vowel.Source: A. The importance of the observable · p. 2

B. Correlations and burstiness

Long-range correlations in a binary event sequence can arise from bursty inter-event times or from correlations between inter-event times. The section distinguishes these mechanisms and sets out how the sequence’s correlation regime is assessed.

B. Correlations and burstiness

  • When Cx(t) decays as t^-β for 0 < β < 1, its sum diverges and the associated random walker spreads superdiffusively.Source: B. Correlations and burstiness · p. 2
  • Integrated indicators are used for more robust estimates: short-range correlations have β > 1 and γ = 1, while long-range correlations have 0 < β < 1 and 1 < γ < 2.The paper uses normal diffusion interchangeably with short-range correlations, and anomalous diffusion with long-range correlations.Source: B. Correlations and burstiness · p. 2
  • For short-range correlations, the power spectrum at zero frequency, S(0), is finite, linking correlation behavior to inter-event-time statistics.Inter-event times are defined as one plus the lengths of the clusters of zeros between consecutive ones.Source: B. Correlations and burstiness · p. 2
  • Long-range correlations have two possible origins: a broad-tailed inter-event-time distribution with divergent στ, or nonsummable correlations among the inter-event times.The two mechanisms contribute differently across linguistic levels of the hierarchy.Source: B. Correlations and burstiness · p. 2

C. Hierarchy of levels

The hierarchy represents linguistic conditions as binary sequences at different levels, linked according to their occurrence patterns. These links establish higher- and lower-level relationships and can be generalized probabilistically.

C. Hierarchy of levels

  • Levels group semantically or syntactically similar conditions, which map text to binary sequences using a corresponding fα [41].Examples include vowels or consonants, letters, words, and topics.Source: C. Hierarchy of levels · p. 2
  • A sequence x is above z when every occurrence of x aligns with an occurrence of z at a fixed offset; same-level sequences have no direct links.A sequence is also above lower-level sequences reachable through a direct path, as prince is above e, which is above vowel.Source: C. Hierarchy of levels · p. 2
  • Links can be interpreted probabilistically for generalizations to higher levels, such as a topic-related word being more likely in writing about war.This probabilistic extension builds on the link definition between sequences.Source: C. Hierarchy of levels · pp. 2–3

D. Moving in the hierarchy

The section derives how long-range correlations and burstiness change as sequences are combined across linguistic levels. Correlations typically flow downward, while burstiness is preserved upward and destroyed downward.

D. Moving in the hierarchy

  • At least two of the three linked sequences must share the largest correlation exponent, constraining how correlations can flow through the hierarchy.The sequences satisfy σX(t) ≤ σZ(t) + σY(t) and corresponding inequalities for the other pairs.Source: D. Moving in the hierarchy · p. 3

Up

  • Upward, a lower-level sequence with γZ > 1 implies at least one sequence above it has γX ≥ γZ, so a letter’s anomaly can originate in a word containing it [42].Source: D. Moving in the hierarchy · p. 3

Down

  • Downward, long-range correlations typically persist with the largest contributing exponent, while broad inter-event-time tails are cut off and information transfers to Cτ(k).With multiple contributing sequences of different exponents, γZ = maxj{γX(j)}; asymptotically, the lower-level correlations can be due solely to nonzero Cτ(k) at large k.Source: D. Moving in the hierarchy · pp. 3–4

Finite-time effects

  • Finite-time estimates can understate downward correlations before the asymptotic regime; in the random-addition example, estimated γZ shifts from normal 1 to anomalous γX.The asymptotic regime starts after a time depending on the parameters; the passage does not provide the completed threshold expression.Source: D. Moving in the hierarchy · p. 3

Burstiness

  • Burstiness is expected to survive upward but be destroyed downward: adding sequences cuts long zero clusters, whereas random deletion cannot make a long-tailed p(τ) short-ranged.Source: D. Moving in the hierarchy · p. 3

II. DATA ANALYSIS OF LITERARY TEXTS

The analysis examines binary sequences from ten English versions of international novels at letter and word levels. It estimates long-range correlations over large time lags, illustrating results with War and Peace.

II. DATA ANALYSIS OF LITERARY TEXTS

  • The study analyzes 41 binary sequences per book across letter- and word-level categories in ten English versions of international novels.The sequences include vowels/consonants, blank space and frequent letters, plus frequent words and nouns and frequency-matched words.Source: §II · p. 4
  • Figure 3 plots burstiness against the finite-time long-range-correlation estimate, with the Poisson process at (1, 1).The diagram generalizes diagrams for long-range correlated sequences in Ref. [33].Source: Figure 3 · p. 4
  • The finite-time estimator of long-range correlations is fitted over large time lags up to 1% of each book’s size.A conservative procedure is used to distinguish short- from long-range correlations; results are illustrated for War and Peace.Source: §II · p. 4

A. Data analysis of correlations and burstiness

The source of long-range correlations varies across linguistic levels: letters and many function words show correlations without burstiness, while many nouns show both. The data also reveal a lower bound linking burstiness to the estimated correlation strength.

A. Data analysis of correlations and burstiness

  • For the letter “e,” long-range correlations persisted without burstiness, whereas for “prince” they were mainly due to burstiness rather than correlations among inter-event times.The original and A2-shuffled “prince” sequences had similar long-range correlations, while A2 shuffling removed them for “e.”Source: A. Data analysis of correlations and… · p. 5
  • Nouns had larger ˆγ in 56 of 70 and larger στ/⟨τ⟩ in 55 of 70 frequency-matched comparisons across 10 books (P-value < 10−6).In every book, at least 4 of 7 noun comparisons showed larger values of both measures.Source: A. Data analysis of correlations and… · p. 5
  • All letters had στ/⟨τ⟩≈1 despite ˆγ > 1.1, while most frequent words and selected frequency-matched words mostly also had στ/⟨τ⟩≈1.Vowel-sequence ˆγ ranged from 1.18 to 1.61 across nine books and was 1.55 ± 0.05 in wrnpc; shuffled null sequences were compatible with γ = 1.0.Source: A. Data analysis of correlations and… · p. 5
  • No sequences combined low ˆγ with high στ/⟨τ⟩, motivating a burstiness-dependent lower bound on ˆγ that restricts the observed range of these measures.For an A2-shuffled sequence with power-law p(τ) and an exponential cutoff τc, ˆγA2 is 4 − µ when the fitting interval is below τc and 1 when it is beyond; interpolating with µ = 2.4 yields the plotted lower bound.Source: A. Data analysis of correlations and… · p. 5

B. Data analysis of finite-time effects

Finite-time effects explain why observed word- and letter-level correlations can differ from their asymptotic behavior. The analysis supports a hierarchy in which keywords connect to specific topics that carry correlations.

B. Data analysis of finite-time effects

  • Long-range correlations at low levels reflect widespread higher-level correlations; keywords, unlike function words, connect strongly to specific topics that carry correlations.This interpretation is supported by the contrast between the sharp transition for “prince” and the slower transition for “so.”Source: B. Data analysis of finite-time effects · pp. 5–6
  • The observed ˆγe < ˆγprince, contrary to asymptotic γe ≥γprince, is explained by tT ≥ 6 × 10^5 for the “e” surrogate and ≥ 7 × 10^8 for the “prince” surrogate.The estimates exceed the maximum time used to obtain ˆγ.Source: B. Data analysis of finite-time effects · p. 5
  • ˆγ(t) rises slowly for letters because sequences with increasingly larger γ dominate for increasingly longer times; ˆγe correlates positively with book length (Pearson Correlation r = 0.44).The passage reports similar results for other letters.Source: B. Data analysis of finite-time effects · pp. 5–6
  • Fig. 4 clearly shows the pre-asymptotic normal diffusion anticipated by the finite-time analysis.Source: B. Data analysis of finite-time effects · p. 5
  • Keyword transitions occur roughly at the paragraph scale (10^2–10^3 symbols); “prince” has a sharp transition alongside the onset of burstiness στ/⟨τ⟩>1.The sharp transition is consistent with fewer independent contributions on higher levels.Source: B. Data analysis of finite-time effects · p. 6

C. Data analysis of shuffled texts

The shuffling results show that long-range correlations mostly disappear when word-level structure is disrupted, but are largely preserved when words are recoded as letter strings. This supports the paper’s account of correlations originating in higher-level, semantically relevant structure.

C. Data analysis of shuffled texts

  • M1 fixes blank-space positions and randomly places word tokens in gaps, whereas M2 consistently replaces each word type with an equal-length random letter sequence.M1 preserves words and letter frequencies while disrupting links above the word level; M2 shuffles links from words to letters.Source: C. Data analysis of shuffled texts · p. 6
  • M1 mostly destroys long-range correlations, while M2 largely preserves them.Average ˆγ in the original texts was 1.40±0.09 in wrnpc and 1.26 ± 0.11 in all books; after M1 it was 1.10 ± 0.08 and 1.07 ± 0.08, respectively, and after M2 it was 1.33 ± 0.08 and 1.20 ± 0.09.Source: C. Data analysis of shuffled texts · p. 6
  • The results support the view that correlations originate in higher-level text structures and remain invariant under the arbitrariness of the sign.The paper links M1’s disruption of above-word-level structure and M2’s preservation of long-range correlations to their semantic basis.Source: C. Data analysis of shuffled texts · p. 6

III. DISCUSSION

Long-range correlations in texts arise from both burstiness and correlations between inter-event distances, whose contributions vary across linguistic levels. The discussion traces how projections shift information between these sources and how the framework may extend to concepts and other hierarchical levels.

III. DISCUSSION

  • At semantic levels, texts are high-dimensional and bursty, while lower-level projections destroy burstiness but preserve long-range correlations by transferring information from burstiness to correlations.The framework explains observations that these two sources contribute differently across linguistic levels.Source: §III · p. 6
  • The cascade may extend above words to concept levels, where topic-linked words need not appear every time a concept does, provided their use probability is high.Further studies are needed to connect concept-level observations with lower levels and distinguish the two correlation sources.Source: §III · p. 6
  • As topics activate linked words with higher probability, correlations flow from topic to word level; highly topical words can show long-range correlations mostly due to burstiness.The renewal case of Eq. (8), where correlations originate only from burstiness, is a simplest toy model compatible with the results and suggests recursively pushing correlations into a renewal sequence to detect hierarchy.Source: §III · p. 7
  • Earlier methods below the letter level or combining letters imply one asymptotic exponent γ, but γ is usually beyond observational range and finite-time ˆγ varies by observable or encoding.The discussion says the present results make this single asymptotic exponent understandable across those methods.Source: §III · p. 7
  • Letters show long-range correlations without burstiness, with exponentially distributed inter-event times, a pattern that can be misread as evidence that letters and most parts of speech follow Poisson processes.The discussion notes this as a possible misinterpretation of the observed lack of burstiness.Source: §III · p. 7

9 Supporting Information … VI. LOWER BOUND FOR ˆγ DUE TO BURSTINESS

The supporting information specifies how text features become binary sequences, how hierarchical transformations affect correlations and burstiness, and how finite-time correlation estimates are tested and bounded.

I. AVERAGE PROCEDURE IN BINARY SEQUENCES

  • Ergodicity lets the authors estimate correlations from one finite text-derived binary sequence by averaging symbol pairs at each time lag.Stationarity makes the correlation depend only on the time lag.Source: §I · p. 9

II. MAPPING EXAMPLES

  • Binary projections encode vowels, letters, n-grams, words, or topic-associated symbols; topic conditions can use latent semantic analysis or concept space.The examples include sliding-window n-grams and word strings with spaces included.Source: §II · p. 9

III. SIMPLE OPERATIONS ON BINARY SEQUENCE AND THEIR EFFECTS ON LONG-RANGE CORRELATIONS AND BURSTINESS

  • Random addition constructs a lower-level sequence from an upper one, while random subtraction moves upward; these operations model correlation flow and burstiness across hierarchy levels.Addition uses y = ξ(1 − x) and z = x + y; subtraction uses ζz for an independent binary sequence ζ.Source: §III · p. 9

A. Transition time from normal to anomalous diffusion

  • Random addition preserves the source sequence’s asymptotic diffusion exponent, but masks it with pre-asymptotic normal diffusion until the transition time tT.Finite-time ˆγ estimates approach γX only for t ≫ tT; when t ≪ tT, ˆγ = 1.Source: A. Transition time from normal to… · pp. 9, 11

B. Random subtraction preserves burstiness

  • Random subtraction preserves the long tail of the inter-event-time distribution, so the asymptotic tail remains controlled by p(τ).For large τ, the finite-k terms are controlled by p(τ), while the remaining sum is exponentially suppressed.Source: B. Random subtraction preserves… · p. 11

IV. DATA

  • The analysis uses ten popular English novels from Project Gutenberg, retaining lowercase letters, digits, apostrophes, and spaces while removing punctuation and capitalization.Words are strings between spaces, with no lemmatization, so singular and plural forms remain distinct.Source: §IV · p. 11

V. CONFIDENCE INTERVAL FOR DETERMINING LONG-RANGE CORRELATION

  • The authors set the reliable upper fitting limit to ts = N/100 and estimate ˆγ by least-squares fits over ts′ < t < ts.They select ts using a randomized surrogate sequence and local-exponent fluctuations; fits use ten time values around ts/100.Source: §V · pp. 11–12

VI. LOWER BOUND FOR ˆγ DUE TO BURSTINESS

  • The finite-time estimate ˆγ exceeds the burstiness-only estimate ˆγA2 for every analyzed sequence, and the cutoff model gives ˆγA2 = γA2 when στ/⟨τ⟩ > s2.The inequality is expected when interval-time correlations contribute positively; the authors report direct confirmation across their analyzed sequences. The cutoff-based bound interpolates between anomalous diffusion below the cutoff and normal diffusion above it.Source: §VI · p. 12

VII. ADDITIONAL SHUFFLING METHODS

Two additional shuffling methods examine how correlations change when word order is randomized or letters are rearranged between fixed blank spaces.

VII. ADDITIONAL SHUFFLING METHODS

  • Word-order shuffling removes correlations beyond the maximum word length and makes the blank-space sequence B uncorrelated.The original blank-space correlations arise from correlations in τi, not from tails in p(τ).Source: §VII · p. 12
  • With blank spaces fixed, both random-letter schemes asymptotically yield γ = γB; for random a/b letters, p >95.8% is needed for tT < ts = 1% book size in wrnpc.The schemes fill the gaps with a and b at probabilities p and 1 − p, or with the book’s letters in random positions; blank-space correlation is preserved by construction.Source: §VII · pp. 12–13

Related papers

Loading 1207.0658v1…