PaperlayerView exact revision on arXiv

Source-linked AI summary

Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure

Shuyang Xiang

arXiv:2609.23551v2cs.CL

TL;DR

Reading order alone cannot determine hierarchical textual structure, so the paper uses separate paragraph, sentence, and token coordinates and intervenes on paragraph position with tokens fixed. Compression occurs in every corpus, but matched random labels compress too; real structure is distinguished by compression depth, whose cross-corpus pattern is not fully explained by the tested corpus-only quantities.

  • Problem

    Reading-order position cannot distinguish tokens at the same separation that occupy different hierarchical relationships, leaving unclear whether paragraph boundaries alter cross-boundary attention beyond distance.

    Source: §1 · p. 1

  • Method

    The paper represents paragraph, sentence, and token indices as separate coordinates and intervenes on paragraph position while holding tokens fixed to measure attention beyond token-distance effects.

    Source: §8 · p. 12

  • Results

    Real paragraph structure compresses more deeply than density-matched random labels where the gap is resolvable, in two of three corpora, while compression depth varies across corpora.

    Source: §8 · p. 13

  • Takeaways & Limitations

    Compression depth, rather than compression location, is the reproducible signature of genuine paragraph structure in the tested setting.

    Source: §8 · p. 13

    The cross-corpus claims are descriptive across only three corpora, and the paragraph-level probe is correlational at Δp1=1 and supplies no corpus-level scale.

    Source: §7 · p. 12

Abstract

from arXiv · show

Standard positional encodings treat position as a one-dimensional reading-order coordinate, but reading order alone does not determine hierarchical textual structure. We use a hierarchical rotary positional encoding (hRoPE) that represents paragraph, sentence, and token indices as separate channels, hold the token sequence fixed, intervene on the paragraph coordinate $p_1$, and measure cross-paragraph attention with a token-distance-exact estimator. Attention is compressed in every corpus, but compression alone is not diagnostic of true structure: an architecturally identical channel with density-matched random labels is compressed too. What distinguishes real structure is the depth of compression: the real-versus-random depth gap is resolvable in two of three corpora and not in the third, and real-structure depth varies more strongly across corpora than the control's. Comparing eight corpus-only quantities across three constructs (lexical persistence, paragraph length, embedding-based coherence), none fully reproduces the cross-corpus ordering of depth, though embedding-based coherence comes closest. At the paragraph level that relation transfers as a common slope, but with the opposite sign to the corpus-level ranking: controlling for length, diversity, and position, paragraphs with more similar neighbors compress less deeply, with no detectable slope difference between any pair of corpora, while length, lexical-diversity, and position effects remain corpus-specific. Compression depth, not its location, is the reproducible signature of genuine paragraph structure in our setting.

1 Introduction

Reading order captures the linear token sequence but cannot distinguish its many possible paragraph hierarchies. The paper tests whether changing paragraph position alone alters attention, then asks whether real paragraph structure produces deeper compression than matched random labels.

1 Introduction

  • The paper tests whether paragraph boundaries alter how a model relates tokens across them beyond reading-order separation, and whether any compression is caused by hierarchical position rather than information density.Source: §1 · p. 1
  • Changing only paragraph position while holding tokens fixed causally changes attention in all three corpora.The paper frames this as a test of whether reading-order position suffices to represent hierarchical textual position.Source: §1 · p. 2
  • Attention compresses near paragraph boundaries in every corpus, but compression alone is not diagnostic because density-matched random labels also compress.The central distinction is compression depth: real structure compresses more deeply than random structure in most corpora, and its depth varies more across corpora.Source: §1 · p. 2
  • No tested corpus-only quantity reproduces the cross-corpus depth ordering, although embedding-based coherence comes closest.At paragraph level, the coherence slope is common across corpora and opposite in sign to the corpus-level ranking.Source: §1 · p. 2

2 Related Work

The related work contrasts scalar reading-order encodings with approaches that represent other structure. The paper uses separate hierarchical coordinate channels to intervene on position and test attention responses rather than improve generation or efficiency.

2.1 Positional encodings beyond reading order

  • Unlike scalar RoPE [Su et al., 2024] and ALiBi, hRoPE assigns hierarchical coordinates independent channel blocks that can be intervened on separately.The paper asks whether a Transformer responds causally to an explicit higher-level coordinate and whether that response has a coherent shape.Source: §2.1 · pp. 2–3

2.2 Hierarchical architectures and long-document modeling

  • Hierarchical encoders use separate sentence-document stacks, while PermGen and HDT encode hierarchy for generation or sparse attention; this paper instead tests coordinate effects with tokens fixed.The construction is close to PermGen and HDT, but is not used to improve generation or efficiency.Source: §2.2 · p. 3

2.3 Geometric framings and causal intervention methodology

  • Related geometric and causal-intervention work studies representation geometry or internal-state changes, whereas this paper swaps only an explicit positional coordinate.Its comparison is with corpus statistics, rather than perplexity or downstream performance.Source: §2.3 · p. 3

3 Experimental Framework

The experimental framework separates paragraph, sentence, and token position, then intervenes on paragraph coordinates while measuring attention beyond token-distance effects. Matched model variants and corpus-specific paragraph units support comparisons across three corpora.

3.1 Hierarchical coordinate system and hRoPE

  • hRoPE gives paragraph, within-paragraph sentence, and within-sentence token indices separate rotary channel groups, allowing p1 to vary independently.The four primary variants are flat, sent axial, hrope axial, and rand axial; the random control resamples p1 labels matched to boundary density.Source: §3.1 · p. 4

3.2 Counterfactual interventions and distance-residualized attention

  • Fake-merge and fake-split manipulate only p1 while preserving tokens and reading order; the attention residual removes the dominant monotone distance-driven component.The linear fit is only a first-order removal, so nonlinear distance dependence may remain in the residual.Source: §3.2 · p. 4

3.3 Normalized boundary effect and bootstrap protocol

  • The normalized boundary effect uses β = 0 for the within-paragraph baseline and β = 1 for the average real-boundary effect.Fake-merge starts at the real-boundary level, while fake-split starts at within-paragraph sentence-boundary pairs.Source: §3.3 · p. 5

3.4 Data and training

  • The study compares WikiText-2, OpenWebText, and Python source files using models with 8 layers, 8 heads, dmodel = 512, context length 1024, and 5000 training steps.Three seeds are used per corpus.Source: §3.4 · p. 5
  • Paragraphs are delimited prose blocks in WikiText-2 and OpenWebText, but blank-line-delimited source blocks in Code.These corpus-specific units differ in length and internal structure.Source: §3.4 · p. 5

4 Layer I: Causal Effect of Hierarchical Position

With the token sequence fixed, intervening on paragraph position changes cross-boundary attention. The response depends on the coordinate’s content, not merely on having a paragraph-position channel.

4.1 Single-example demonstration

  • In the illustrative n = 1 example, fake-splitting leaves tokens unchanged but lowers cross-boundary Deff by 3.105; p1-blind controls are unchanged.This is a single-example demonstration, not a statistical result.Source: §4.1 · p. 5

4.2 Intervention effects in all three corpora

  • Merging paragraphs compresses attention (ΔB < 0), while splitting them dilates it (ΔC > 0).The interventions use the same token sequence; the reported fake-split response falls near the middle of the real-boundary distribution (58–62%).Source: §4.2 · pp. 5–6
  • Across all three corpora, fake-split response βC ranges from 1.06 to 1.52; its 95% interval excludes 1 for Code, is marginal for OpenWebText, and includes 1 for WikiText-2.The merged response overshoots the within-paragraph baseline mildly for WikiText-2 (−0.300) and OpenWebText (−0.302), and strongly for Code (−2.769).Source: §4.2 · p. 6
  • With no paragraph-position channel, falsifying paragraph position changes no input, so flat and sent axial controls have ΔB = ΔC = 0 by construction.Source: §4.2 · p. 6

4.3 Does the channel’s content matter, or only its presence?

  • For hrope axial, validation loss orders real < period < rand < const0 in every seed and corpus, with constant p1 costing 0.30–0.35 nats more than real.The consistent ordering is directional evidence from three seeds, not a formal significance test; real-versus-random depth gaps are largest in Code (0.26 nats) and smallest in OpenWebText (0.10 nats).Source: §4.3 · pp. 6–7
  • Substituting true paragraph coordinates into a period-axial checkpoint raises loss only +0.015 to +0.040 nats, less than hrope axial’s +0.061 to +0.097 when moved to period.The mechanical-coordinate model does not become strongly sensitive to true paragraph structure it never saw.Source: §4.3 · pp. 6–7

5 Layer II: Compression Depth as the Corpus-Sensitive Signature

Attention is compressed in all three corpora, but that alone does not identify genuine paragraph structure: the density-matched random control is compressed too. Real-versus-random depth gaps are resolvable in two corpora, while the third remains unresolved at the available seed count.

5.1 Exact-distance estimation of the compression response

  • The exact-distance estimator regresses distance-residualized attention on paragraph displacement while controlling for token distance, then defines compression depth U* at the response curve’s interior minimum.It retains (d, Δp1) cells with at least 2,000 token pairs and avoids a binned-estimator distance-mismatch bias that had flipped WikiText-2’s sign.Source: §5.1 · p. 7
  • Compression depth is bootstrap-stable in every corpus, but displacement location is not reliably identifiable for WikiText-2 and is fit-sensitive for Code.WikiText-2 depth shifts by about 0.1 across sampling gates; the Code degree-2 and degree-3 fits place the minimum at approximately 5.5 and 3.4, respectively.Source: §5.1 · p. 8

5.2 The depth of compression is not exclusive to true paragraph structure

  • All six model-corpus cells show compression, but the real-versus-random depth gap excludes zero in only two of three corpora; OpenWebText is unresolved at this seed count.Real-structure depth varies more across corpora (−0.31 to −0.77) than random-control depth (−0.22 to −0.33); the cross-corpus pattern is descriptive.Source: §5.2 · p. 8

6 Layer III: Comparing Corpus-Only Candidates with the Observed Depth

Section 6 asks whether corpus-only properties reproduce the observed ordering of compression depth, then examines candidate constructs at corpus and paragraph levels.

6.1 Paragraph-level lexical distributions

  • λstruct estimates excess sequential paragraph similarity over a within-document shuffled-order control and interprets its decay scale as paragraph-level structural persistence.It is a Jensen–Shannon distributional measure of coherence, not a mutual-information decay length.Source: §6.1 · p. 9

6.2 Main comparison

  • λstruct ranks WikiText-2 below OpenWebText and Code, while compression depth ranks Code, WikiText-2, then OpenWebText; thus λstruct does not recover the depth ordering.Its prose-corpus estimates overlap, so it cannot separate WikiText-2 from OpenWebText.Source: §6.2 · p. 10

6.3 Further candidates: at best partial

  • The lexical-persistence candidates and paragraph length fail to reproduce the depth ordering, while embedding coherence separates Code from both prose corpora but not the two prose corpora from each other.The vocabulary-robustness check leaves λstruct’s ordering unchanged at every tested cutoff.Source: §6.3 · p. 10
  • None of the three tested constructs fully reproduces the cross-corpus ordering of compression depth; embedding coherence comes closest.The comparison covers eight quantities grouped as lexical persistence, paragraph length, and embedding-based coherence.Source: §6.3 · p. 10, Table 4 · p. 11

6.4 Moving the analysis down to the paragraph level

  • The exploratory masking probe finds compression depth tracks the attention mass removed rather than which tokens carry it, with a residual effect of similar size across corpora.That residual is 14–18 percentage points of the well and changes no main-text estimate.Source: §6.4 · pp. 11–12
  • More-similar embedding neighbors predict less-deep paragraph compression at a common slope across all three corpora, opposite to the corpus-level ranking.The positive raw slopes show no detectable difference between any pair of corpora; length, lexical diversity, and position effects remain corpus-specific.Source: §6.4 · p. 11, Table 5 · p. 12

7 Limitations

The paper limits its conclusions to the tested corpora, model setting, and measurements, and treats cross-corpus depth ordering as descriptive.

7 Limitations

  • The three tested corpus-level constructs do not explain corpus-dependent depth, and the paragraph-level analysis is correlational at Δp1=1, identifying neither mechanism nor corpus-level scale.Other corpus-level properties may still determine depth.Source: §7 · p. 12
  • The experiments use 8-layer, dmodel = 512 models trained for 5000 steps and report no downstream-task metric.Paragraphs are the relevant unit only under a corpus-specific definition.Source: §7 · p. 12
  • Period axial cannot analyze a coordinate decoupled from token distance, and three seeds restrict seed-level comparisons to directional evidence.The paragraph coordinate is structurally non-identifiable for WikiText-2 and fit-sensitive for Code, so depth, not location, is primary.Source: §7 · p. 12
  • With only three fixed-by-choice corpora, the depth ordering establishes that the corpora differ, not that this particular ranking generalizes.The document-level bootstrap quantifies within-corpus uncertainty.Source: §7 · p. 12

8 Conclusion

Reading order alone cannot determine hierarchical structure, but changing paragraph position while holding tokens fixed changes attention. Real structure compresses more deeply than matched random labels where the comparison resolves, while the tested corpus-only measures do not fully explain cross-corpus depth ordering.

8 Conclusion

  • Real structure compresses more deeply than density-matched random labels in two of three corpora; depth varies across corpora, while compression location is not identifiable everywhere.The conclusion identifies depth, rather than location, as the primary result.Source: §8 · pp. 12–13

A.1 Anchor and β estimation

  • Anchor-normalized intervention magnitudes cannot be compared directly with raw compression depth: Code’s fake-merge effect is −3.77 versus WikiText-2’s −1.30, while their depths are −0.772 and −0.481.The intervention effects use scales normalized to within-paragraph and real-boundary anchors; depth is a raw attention residual.Source: §A.1 · p. 15

A.2 Joint cluster bootstrap

  • Each of 4000 bootstrap resamples draws documents with replacement and recomputes the effective-distance measure and both anchors from the same resampled documents.This keeps anchors and manipulated points aligned within each resample.Source: §A.2 · p. 15

A.3 Equilibrium point estimation

  • The exact-distance response curve contrasts each paragraph displacement with a same-distance, same-paragraph baseline, then fits weighted degree-2 and degree-3 curves to estimate interior stationary points.The degree-2 fit is the headline; the degree-3 fit checks location robustness.Source: §A.3 · pp. 15–16

A.4 Corpus-only candidate quantities and their per-variant verdicts

  • Across eight corpus-only quantities, lexical persistence and paragraph length fail to reproduce depth ordering, while embedding coherence comes closest but does not separate every corpus pair.The lexical variants are one construct, and the two coherence summaries are not independent successes.Source: §A.4 · p. 16

A.5 Corpus-only structural scale estimation

  • λstruct measures the decay scale of sequential paragraph similarity in excess of a shuffled-order control, using lexical distributions and a document-level cluster bootstrap.The shuffle preserves marginal paragraph content while destroying sequential order.Source: §A.5 · pp. 16–17

A.6 A distance-bin pooling artifact, and why we moved to exact-distance estimation

  • Distance-bin pooling can reverse the estimated effect: WikiText-2 showed compression at every distance for ∆p1 = 1, yet pooled estimation indicated dilation, prompting exact-distance comparisons.The discrepancy arose because the compared token pairs had different mean distances within the bin.Source: §A.6 · pp. 17–18
  • WikiText-2’s fitted compression location drifts as the sampling threshold rises, while depth changes far less; its location is therefore non-identifiable even though depth remains stable.The threshold alters which baseline distances contribute to the fitted curve.Source: §A.6 · p. 18

A.7 Robustness of compression depth and location to the sampling threshold

Compression depth is stable across sampling thresholds more broadly than compression location, especially for WikiText-2. Probing and paragraph-level regression add evidence about corpus-dependent effects, while the observed turnover rules out one simple additive mechanism without identifying the alternative.

A.7 Robustness of compression depth and location to the sampling threshold

  • From n ≥1,000 to n ≥2,000, Code’s depth changes by 0.001 (−0.773 to −0.772) and OpenWebText’s by 0.01 (−0.295 to −0.305), while WikiText-2’s depth shifts about 0.1 across thresholds.WikiText-2’s fitted location drifts monotonically and does not plateau; its location is treated as non-identifiable.Source: §A.7 · pp. 18–19, Table 7 · p. 19

A.8 Seed-level inference at n = 3

  • With three seeds, probe and intervention comparisons provide directional evidence only: even a consistent sign cannot reach conventional significance thresholds.The minimum two-sided sign-test p-value is 0.25.Source: §A.8 · p. 20

A.9 Layer-wise probing of paragraph-boundary decodability

  • Paragraph-boundary information is more linearly accessible with either axial channel than with flat; hRoPE’s advantage over random labels is strongest in Code and unreliable in OpenWebText.In Code, hRoPE exceeds rand axial at every seed from layer 2 onward; in OpenWebText, signs fluctuate across seeds.Source: §A.9 · p. 20, Table 8 · p. 20

A.10 Paragraph-level compression regression

  • Controlling for length, diversity, and position, more coherent paragraphs compress less deeply, with no detected coherence-slope difference between corpora; the other predictor effects remain corpus-specific.The positive coherence slope has the opposite sign to the corpus-level coherence–depth ranking.Source: §A.10 · pp. 22–23

A.11 A single-example demonstration; A.12 The compression response curves; A.13 The two counterfactual interventions

  • The single-example maps and intervention illustration hold token order fixed while changing paragraph coordinates; the response-curve figure plots fitted curves whose vertices define compression depth.The single-example display is illustrative, not a statistical result.Source: §A.11 · p. 23, §A.13 · p. 23, Figure 3 · p. 23

A.14 Extended interpretation and scope

  • “Geometry” is descriptive here: the intervention supports a coordinate-dependent attention response, not a complete metric theory or a simulation of human reading.The authors explicitly limit the interpretation to a structured response field.Source: §A.14 · p. 24

A.15 A superposition of monotone processes cannot produce the observed turnover

  • The observed interior minima in Code and OpenWebText cannot arise from adding a monotonically decaying attractive term to a monotonically non-decreasing suppressive term.The result rules out this additive family only; the interaction producing the turnover remains unidentified.Source: §A.15 · p. 25

B Extended Limitations

The paper limits its claims to three selected corpora and to the tested units, models, and methods. Its corpus-level ordering is descriptive, while the paragraph-level coherence relation is correlational and does not explain the corpus-level scale.

B Extended Limitations

  • None of eight tested quantities across lexical persistence, paragraph length, and embedding coherence reproduces the depth ordering or explains why the real-versus-random gap resolves in only two corpora.Other corpus properties, architecture, and training dynamics remain possible explanations.Source: B Extended Limitations · p. 26
  • The paragraph-level regression is correlational, uses ∆p1 = 1 only, and establishes a conditional within-corpus coherence relation rather than a mechanism or corpus-level scale.Its coherence slope runs opposite to the corpus-level coherence–depth ranking.Source: B Extended Limitations · p. 26
  • Cross-corpus ordering is descriptive for three fixed corpora, and the reported models use 8 layers and 5000 training steps without production-scale or downstream evaluation.Whether the findings persist at larger scale or affect downstream performance remains open.Source: B Extended Limitations · p. 26

C.1 Corpus-specific boundary extraction

  • Corpus boundaries are extracted differently by dataset: WikiText-2 uses blank-line paragraphs, OpenWebText follows its web-page filtering pipeline, and Code treats blank-line-separated blocks as paragraphs.The Code hierarchy is file → block → line → token.Source: C Training and Implementation Details · p. 26, §C.1 · p. 27

C.2 Training configuration

The model variants share fixed training hyperparameters and matched axial-channel calibration. The controls differ in the content of the paragraph coordinate, with one mechanical-coordinate control excluded from depth comparisons.

C.2 Training configuration

  • Training hyperparameters are held fixed across flat, sent axial, hRoPE axial, rand axial, and period axial variants and across corpora.The supplied training-volume details include approximately 49,000 Code training files and 8,009,762 OpenWebText training documents.Source: Table 11 · p. 27, §C.2 · p. 28
  • Axial variants use corpus-calibrated, non-wrapping rotary channels with no added learned parameters, matched across variants and seeds.The variants differ only in what occupies the paragraph-position channel.Source: §C.2 · p. 28
  • The mechanical period control uses a density-matched grid coordinate but is excluded from depth comparisons because its paragraph coordinate is nearly determined by token distance.Its use is limited to the loss-based substitution check.Source: §C.2 · p. 28
  • Rand axial matches hRoPE axial in architecture and training setup but replaces true paragraph positions with random labels matched to boundary density and resampled each training step.The labels are not fixed to documents, preventing memorization of boundaries.Source: §C.2 · p. 28

D Language-Modeling Cost of the p1 Channel

The seed-matched comparison finds hrope axial less costly than rand axial, while its cost relative to flat is small in two corpora and undetectable in Code.

D Language-Modeling Cost of the p1 Channel

  • The comparison supports sensitivity to the content of an existing p1 channel, not implicit paragraph-structure representation by an ordinary Transformer without that channel.The authors state that the latter question is outside their scope.Source: D Language-Modeling Cost of the p1… · p. 28
  • The validation-loss comparison uses four primary positional variants and three shared seeds, without a formal bootstrap confidence interval.The reverse substitution result uses an inference-only protocol and is not part of Table 12.Source: D Language-Modeling Cost of the p1… · p. 29
  • Rand axial loss exceeded hrope axial loss in all three matched seeds of every corpus, with non-overlapping seed ranges; evidence is directional (minimum one-sided sign-test p = 0.125).The comparison uses three seeds and is not a precise or formally significant cost estimate.Source: D Language-Modeling Cost of the p1… · p. 29
  • Hrope axial loss was about 0.5% higher than flat in WikiText-2 and OpenWebText, while Code showed no detectable cost at n = 3.In Code, the sign varied across seeds and hrope axial’s range fell within flat’s.Source: D Language-Modeling Cost of the p1… · p. 29

E Supplementary Analysis: Is Compression Uniform or Hub-Driven?

The exploratory masking probe finds that removing cross-paragraph attention mass accounts for most of the change, while hub-specific effects add a similar increment across corpora and have different outcomes relative to zero.

E Supplementary Analysis: Is Compression Uniform or Hub-Driven?

  • The probe masks top-5%-contribution hub tokens at inference on trained hrope axial checkpoints, comparing equal-token and equal-pair random controls without retraining.The analysis is exploratory and does not resolve the origin of corpus-dependent compression depth.Source: E Supplementary Analysis: Is… · pp. 30, 32
  • 81% of WikiText-2, 72% of Code, and 111% of OpenWebText’s original well depth was eliminated by Pair-matched random; Hub-masked raised these shares to 99%, 86%, and 126%, respectively.The Hub-masked increment beyond Pair-matched random was 14–18 percentage points in every corpus.Source: Table 13 · p. 30, E Supplementary Analysis: Is… · pp. 30–31
  • Pair-matched random masking moves U* substantially, unlike masking the same number of arbitrary tokens; the mass effect grows nonlinearly and pushes OpenWebText from −0.305 to +0.034.In WikiText-2 and Code, U* moves monotonically toward zero as more cross-paragraph pairs are removed.Source: E Supplementary Analysis: Is… · pp. 30–31
  • Hub tokens are well approximated by raw-frequency tokens, but their syntactic roles differ: frequent code tokens mark boundaries, unlike frequent natural-language function words.The authors connect this distinction to a compression residual in Code that survives the combined interventions.Source: E Supplementary Analysis: Is… · pp. 31–32
  • After mass and hub effects combine, Code retains compression, WikiText-2’s confidence intervals include zero, and OpenWebText shows dilation from masking rather than in its unmasked state.The respective Pair-matched random depth fractions are 72%, 81%, and 111%.Source: E Supplementary Analysis: Is… · p. 31

F Vocabulary Robustness of the Corpus Structural Scale

The cross-corpus ordering of λstruct remains stable across tested vocabulary cutoffs, but it does not match compression-depth ordering.

F Vocabulary Robustness of the Corpus Structural Scale

  • The main-text λstruct estimates use each corpus’s full frequency-filtered vocabulary: V = 143 for Code, 324 for WikiText-2, and 163 for OpenWebText.Those values coincide with no row of the cutoff table.Source: F Vocabulary Robustness of the Corpus… · p. 32
  • Across every tested vocabulary size, λstruct preserves the cross-corpus ordering Code > OpenWebText > WikiText-2; although WikiText-2’s λstruct decreases moderately as vocabulary expands, its ordering matches compression depth at no size.Source: F Vocabulary Robustness of the Corpus… · p. 32
  • WikiText-2’s λstruct decreases moderately as vocabulary expands, so the ordering is stable but absolute values are not fully vocabulary-independent.The sweep checks ordering across cutoffs, not comparability with the main-text bootstrap medians.Source: F Vocabulary Robustness of the Corpus… · p. 32, Table 14 · p. 32

Related papers

Loading 2609.23551v2…