A Glyph Is Not a Letter, a Token Is Not a Word, a Space Is Not a Space: What the Units of Voynichese Are Not

• A glyph is not yet a demonstrated letter, a blank-delimited token is not yet a demonstrated word, and a separator is not automatically a word boundary.The paper tests these identifications as hypotheses rather than treating transcription units as linguistic units by default.Source: §1 · pp. 1–2
• First-order conditional entropy tests glyph invariance, while BPE learns recurrent multi-symbol units and a calibrated class-merging attack tests for concealed prose under verbose or homophonic ciphers.The learned scale is also tested on held-out quires, and the attack is calibrated against synthetic ciphers.Source: §1 · pp. 2–3
• Vocabulary size and frequency concentration assess token plausibility, while shuffled adjacent-token mutual information tests whether tokens constrain neighbours as words do in matched prose and botanical records.Merging every uncertain separator tests whether false splits account for the observed pattern.Source: §1 · p. 2
• The space tests compare glyphs across separator classes, validate the classes against label-independent image coordinates, and erase spaces before learning to test which former positions are recovered as boundaries.This distinguishes arbitrary spaces from the stronger claim that every space marks a lexical word break.Source: §1 · pp. 2–3
• The paper unifies unit tests with matched controls and cluster-aware uncertainty, then tests which parts of the profile published cipher and self-citation-generator imitations reproduce.It separates lexical appearance from sequence, validates separator classes against layout and ink, relearns units after space deletion, and tests their scale across held-out quires.Source: §1 · p. 3

2.1 Transcription and analysis corpus

• The corpus uses ZL version 3b, distinguishing certain from uncertain separators, and includes 3,880 qualifying lines across 206 pages and 16 effective quire clusters.Frequent EVA composites are collapsed for one representation and retained as multi-character sequences in another; the paper reserves symbol, token, and separator for transcriptional units rather than assuming letters, words, or word boundaries.Source: §2.1 · pp. 3–4

2.2 Token inventory and adjacent order; 2.10 Edge-glyph order, two further controls, and published inventories

• Token-order tests measure adjacent-token information against within-line shuffles and matched controls, while edge-glyph tests assess dependencies that token identities can miss.The analyses also compare whole-profile controls, including the Naibbe cipher and a self-citation text generator, using the same measurement battery.Source: §2.2 · pp. 4–5, §2.10 · p. 8

2.3 Normalised internality index

• The normalised internality index compares glyph-pair association at boundary classes with within-token and independent-pair anchors.Its smoothed association scores use position-independent within-token unigram marginals; continuation line breaks pair the final glyph of one line with the initial glyph of the next on the same page.Source: §2.3 · p. 5

2.4 Cluster-aware uncertainty; 2.5 Currier strata and line-start decomposition

• Uncertainty intervals resample quires, with additional Currier A/B, leave-one-quire-out, and line-start analyses testing stability and stratum differences.The intervals describe stability over observed top-level units, not a population of independent manuscripts; the A–B uncertain-index difference is tested against a prespecified ±0.05 equivalence margin.Source: §2.4 · p. 5, §2.5 · p. 5

2.6 Image-coordinate validation

• Image-coordinate validation aligns ZL and visual token sequences, transfers certainty labels only for matching flanking tokens, and compares certain and uncertain gaps with folio-aware tests.The analysis treats the coordinates as coarse corroboration rather than ground truth and adds a second-reader agreement check that cannot establish lexical status.Source: §2.6 · p. 6

2.7 Conditional-entropy comparison

• Conditional-entropy comparisons use collapsed and decomposed glyph streams, matched language controls, sample-size windows, and within-token-only transitions to test invariance under bijective letter substitution.Spaces and line breaks are removed, and the conclusion is limited to source texts with comparable first-order structure rather than every language or writing system.Source: §2.7 · pp. 6–7

2.8 Learned-unit scale comparison

• BPE curves track learned-unit scale against language and verbose-substitution controls, with a calibrated held-out attack and quire-based cross-fits testing robustness.The study also varies class counts and refits by Currier stratum and quire, compares disjoint quire halves, and repeats the curve after collapsing standard EVA composites.Source: §2.8 · pp. 7–8

2.9 Separator-erasure sensitivity

• Separator-erasure tests fit BPE to continuous line strings, then compare original separator positions with random placements, permuted token lengths, and held-out-quire glyph-pair predictions.The analysis tests whether imposed tokenisation creates the learned-unit trough and applies the comparison to Latin and verbose-cipher controls with plaintext spaces hidden.Source: §2.9 · p. 8

3.1 A token is not yet a word: lexical appearance without prose-like order

• Voynichese’s 7,022 forms and 69.7% singleton types look broadly word-like, but adjacent-token order contributes only 0.79% of token entropy.That share is below the lowest prose control (2.02%) and falls to 0.54% when all uncertain separators are merged; the deficit persists across quires, Currier languages, and vocabulary caps.Source: §3.1 · p. 9, Figure 1 · p. 10

3.2 Order lives at the token edges, not in token identity

• Adjacent edge glyphs carry 0.197 bits of information in Voynichese, exceeding every prose control, while token identity carries little adjacent order.The mixed last-glyph-to-next-token statistic is 2.0% of token entropy, within the control range, and the size of the token-identity deficit varies with vocabulary cap; the robust contrast is where order is concentrated.Source: §3.2 · pp. 10–11, Table 4 · p. 11

3.3 A separator is not one uniform word boundary

• Uncertain separators have higher internality than certain separators (0.494 vs 0.029), exceeding the certain mean in all 16 quires, and bounding-box gap alone distinguishes labels with AUC 0.9053.In leave-one-folio-out validation, bounding-box gap achieved 86.7% sensitivity, 80.1% specificity, and 83.4% balanced accuracy.Source: §3.3 · p. 11, §3.4 · p. 13

3.5 The two separator regimes are stable across Currier A and B

• Uncertain-separator internality is similar in Currier A and B, whereas ordinary-boundary scores differ descriptively between the dialects.The uncertain-separator difference narrowly misses the prespecified equivalence margin; the ordinary-boundary pattern has no identified mechanism, and dialect is confounded with scribal hand.Source: §3.5 · pp. 13–14, Figure 3 · p. 14

3.6 The line-start effect is a mixture of two line populations

• Paragraph-first lines and other lines form distinct opening populations, with line-initial glyph distributions differing by a factor of about 2.9.Paragraph-first lines are strongly enriched for gallows, while ordinary lines reverse the sign for k, f, y, and d; page-opening lines are more extreme still.Source: §3.6 · p. 14, Figure 3 · p. 14

3.7 A glyph is not licensed as a plaintext letter

• Composite-collapsed Voynich glyphs have conditional entropy 0.76–0.83 bits below controls, so EVA symbols are not licensed as one-to-one plaintext letters.A fixed monoalphabetic substitution cannot account for the difference, though other encoding schemes remain possible. The section also introduces Stolfi’s token-as-syllable hypothesis as a testable alternative, but this part supplies no test result.Source: §3.7 · p. 15, Table 6 · p. 15, §3.10 · p. 18

3.8 Recurrent multi-symbol units provide the supported scale

• Voynichese shows a reproducible early multi-symbol regime around 32–64 merges, not a uniquely established inventory of exactly 88 units.The in-sample minimum occurs at 64 merges, while held-out scoring moves it to 32; learned-unit curves alone do not distinguish encoding from non-cryptographic chunking, and the calibrated attack excludes only a limited cipher family.Source: §3.8 · pp. 15–17, Figure 4 · p. 16

3.9 Two accounts that earn part of the profile

• Both tested accounts reproduce several profile features, but neither matches Voynichese’s edge-glyph coupling or hapax-rich vocabulary.Naibbe and self-citation streams match low glyph entropy, learned-unit scale, weak token order, and null attack differentials; neither reproduces the graded separator regime, which their designs do not test.Source: §3.9 · p. 18, Table 7 · p. 18

3.10 A token is not a syllable

• Pinyin streams show 5.2–6.4% shuffle-corrected adjacent order versus −0.5%/+0.4% for Currier A/B and 0.79% pooled, while Voynich tokens remain open.At about 10,700 tokens, pinyin streams use 335–749 syllable types with 10–20% hapax, versus 3,343 types and 72% hapax for Currier A. The test covers Mandarin; larger syllabaries would narrow the inventory gap but not the order gap.Source: §3.10 · p. 19

3.11 Erasing spaces preserves the learned scale and boundary hierarchy

• Space erasure preserves the BPE scale: D32/D64/D128/D256 are 1.102/1.049/1.139/1.500 bits for tokens and 1.115/1.072/1.181/1.581 for line strings.Both curves retain their minimum at 64 merges, so the unit-scale result does not depend on forbidding merges across transcribed spaces.Source: §3.11 · p. 19
• Arbitrary separator constraints fit worse: random positions yield median D64 of 1.241 bits, while reassigned token lengths yield 1.271 bits and select 32 merges in all 50 fits.The random-position fits select 64 merges in 41 of 50 cases and 32 in nine; neither separator count nor token-length distribution explains the observed 64-merge curve.Source: §3.11 · p. 19
• At 64 merges, learned units cross 2.5% of certain separators, 20.3% of uncertain separators, and 60.0% of possible adjacent-glyph positions.Uncertain crossing exceeds certain crossing in all 16 quires; the control rates show the diagnostic can recover genuine word boundaries without assuming lexical status.Source: §3.11 · p. 19, Table 8 · p. 19
• The raw flanking pair predicts space presence out of quire with held-out AUC 0.980 and a 71.9% reduction from marginal cross-entropy over 162,330 positions.The result is positive in every quire fold, rejects random placement, but remains compatible with linguistic morphotactics or a rule-governed surface process.Source: §3.11 · pp. 19–20

4.1 What weak token order does and does not show

• Adjacent-token comparisons test local form order, not paragraph syntax or semantic coherence, so their results constrain only the tested units and text types.The controls include domain-matched botanical, medical, and herbal prose and five narrative-language conditions, but do not exhaust natural language or test long-range dependencies, layout, or illustration links.Source: §4.1 · p. 20
• Voynichese remains below every control at every vocabulary cap, although its deficit shrinks at coarser caps and its corrected estimate is noisy at the 2,000-type cap.At that cap the observed plug-in value is 98% shuffle bias; a slightly negative estimate at the largest cap is the noise floor, not negative mutual information.Source: §4.1 · p. 20
• Erasing all spaces still recovers the 64-merge trough, while 97.5% of certain positions remain uncrossed; preserved strong spaces are not thereby proven lexical boundaries.Merging all uncertain separators lowers the order estimate, but any hidden segmentation that routinely crosses strong spaces must explain why sequence learning and image geometry recover that class.Source: §4.1 · p. 20

4.2 What the coordinate analysis does and does not establish

• The coordinate AUC establishes recoverability of the uncertain-separator distinction, not lexical status; box provenance is undocumented, and the direct-pixel audit shows weaker discrimination.Boxes may encode a human reading of junctures; coordinates are quantised to a 3-pixel grid, and excluded joined cases may be especially compressed, but selection effects are unidentified.Source: §4.2 · pp. 20–21
• The separator-association index may embed familiar joined strings, but box gaps, ink gaps, and space-erased learning converge without using the ZL comma as a measurement.The v101 gap gradient reduces dependence on one reader, yet both transcriptions derive from MS 408 and the check supports only a reproducible weak-space distinction, not lexical analysis.Source: §4.2 · p. 21

4.3 Scope of the learned-unit inference

• BPE units are not a linguistic oracle, and the held-out-quire minimum at 32 rather than 64 merges means the exact scale and 88-unit inventory remain analysis-dependent.The evidential feature is the stable early trough and its replication, not the eventual rise in dependence as rare longer units accumulate.Source: §4.3 · p. 21
• After collapsing nine EVA composites, the pooled gap remains in a broad 32–64-merge trough (1.049 bits at k = 32; 1.046 at k = 64), including after separator erasure.Crossing rates support a graded model in which certain spaces are strong distributional boundaries and uncertain spaces are more permeable; neither class is established as lexical.Source: §4.3 · p. 21
• With only 16 quire clusters and eight continuous-text controls, the analysis emphasizes effect sizes, cluster resampling, leave-one-cluster stability, and matched-length controls over universal claims.The controls do not exhaust linguistic typology, so conclusions are limited to the tested comparisons and substitution invariance.Source: §4.3 · p. 21

5.1 Observed units are not inferred linguistic units

• Glyphs are observed coding units, not established letters: their conditional entropy is too low for tested one-to-one alphabetic relabelings, while recurrent multi-symbol units provide a stable scale.That scale survives Currier splits, quire resampling, composite collapse, space erasure, and out-of-quire application; whether the units are linguistic, cryptographic, palaeographic, or motor is undecided.Source: §5.1 · pp. 21–22

5.3 Implications for language and pseudo-text hypotheses

• Tokens have a plausible vocabulary but their identities are less ordered than in every matched control, so resemblance to words does not establish word-like sequencing.Weak adjacent identity order does not itself imply absence of meaning: lists, labels, paradigms, litanies, tables, and formulaic notations can carry content without continuous-prose behavior.Source: §5.1 · p. 22, §5.3 · p. 22

5.1 Observed units are not inferred linguistic units

• Separators are structural events, but certain and uncertain separators differ in transcription association, physical gap, and permeability to learned units after spaces are erased.The uncertain separators behave more like word-internal junctures; neither separator class is thereby proven lexical.Source: §5.1 · p. 22

5.2 Implications for cipher hypotheses

• Voynichese averages −0.01 language-over-surrogate differential and never exceeds +0.08, unlike calibrated concatenative homophonic ciphers retaining +0.13 to +0.52, or +0.05 to +0.13 against the wrong language.A fixed one-to-one letter substitution cannot explain the symbol-level result, and a fixed one-to-one whole-word codebook cannot erase token-order information without additional operations or a different segmentation.Source: §5.2 · p. 22
• The Naibbe cipher reproduces several manuscript properties but not its 0.2-bit edge-glyph coupling or its 70% singleton types, compared with Naibbe’s 41%.Its verbose homophonic encoding respaces plaintext and spreads letters over several glyphs and many homophones, so the attack rejects calibrated concatenative ciphers, not this broader family.Source: §5.2 · p. 22

5.3 Implications for language and pseudo-text hypotheses

• The self-citation generator shows that meaningless text can produce low entropy, a trough, weak order, excess near-identical neighbours, and a null differential, but those features do not prove meaninglessness.Its authors’ rules generate text with no content, yet the same features are also consistent with the Naibbe cipher of a real text.Source: §5.3 · p. 23
• The manuscript remains unexplained by either published mechanism in its vocabulary tail, edge-glyph coupling, trough depth, and stability across sections.These properties are proposed as targets for testing the hoax hypothesis and for evaluating future cipher designs or proposed plaintext readings.Source: §5.2 · p. 22, §5.3 · p. 23

6 Conclusion

• Call an EVA glyph a glyph, a ZL form a token, and a transcribed blank a separator until evidence establishes letter, word, or boundary function.Treating these observed categories as letters, words, and uniform word spaces would build an untested decipherment into the data.Source: §6 · p. 23
• Fine-grained symbol regularity resolves to quire-stable recurrent multi-symbol units, with an out-of-quire minimum at 32 merges and a full-corpus minimum at 64.Source: §6 · p. 23
• Blank-delimited forms are a plausible static inventory, yet adjacent identity information is below every continuous-text control at any vocabulary cap; edge glyphs exceed every prose control.Source: §6 · p. 23
• Separators form strong and weak regimes recoverable from sequence, image geometry, and learning after every space is erased.Source: §6 · p. 23
• Both published accounts reproduce 2.7-bit conditional entropy, learned-unit scale, below-1% token-order share, and null cipher differential, but neither matches 0.2-bit edge-glyph coupling or 70% singleton vocabulary.Separator regimes were untestable on those accounts; uncertain-separator internality was 0.494 against 0.029, with layout coordinates corroborating the distinction at AUC 0.905.Source: §6 · p. 23

7 Data and code availability

• The ZL transliteration, control corpora, per-token coordinates, and analysis archive are available at the project repository’s recorded snapshot.The analysed snapshot is commit 66f8adaadc120f93e8a4c906685ba66a9d3ab847.Source: §7 · p. 23
• The publication package includes analysis-specific drivers, data, controls, and archived fixed-seed outputs, with explicit input-root arguments for every driver.The drivers cover the headline analyses, learned units, controls, and calibrated attack.Source: §7 · p. 24
• Beinecke page images used in the direct-pixel audit are not redistributed, while the package also accompanies the arXiv version as ancillary files.Source: §7 · p. 24

A.1 Boundary-index specification

• The estimator uses 34 observed symbols and applies add-0.5 smoothing to every cell of the 34 × 34 within-token bigram table before normalisation.The same position-independent unigram vector supplies both PMI marginals and the independent-product expectation.Source: §A.1 · p. 24
• Both PMI marginals and the independent-product expectation use the same position-independent unigram vector.This keeps the reference distribution consistent across these calculations.Source: §A.1 · p. 24
• Bootstrap duplicates of a quire duplicate its lines and counts before all tables and anchors are recomputed.Source: §A.1 · p. 24
• The five boundary pools are defined by label, separator position, or continuation line breaks: uncertain and certain use the ZL comma/dot label, first and mid use position within the line, and continuation joins consecutive nonparagraph-first lines on the same page.Source: §A.1 · p. 24
• Last-of-line intra-line separators count toward certain and uncertain totals but are not shown as a separate main-table row.Source: §A.1 · p. 24

A.2 Coordinate fixed effect

• The pair-fixed-effect coefficient demeans box gap and uncertain-label status within glyph-pair groups, while bootstrap resampling duplicates folios and recomputes pair means.The two-way sensitivity estimate alternately demeans gap and label within glyph-pair and folio groups until convergence.Source: §A.2 · p. 24

A.3 Blind direct-pixel audit

• The audit measured a blind sample of six folios, taking all 23 uncertain boundaries and 277 certain boundaries from the frozen candidate pool.Labels were kept separate from the blind manifest and QC file; blind QC excluded 14 boundaries.Source: §A.3 · pp. 24–25
• Certain ink gaps averaged 4.98 px (n = 265) versus 3.14 px (n = 21) for uncertain gaps, a +1.84 px contrast; all five folios had the predicted sign (p = 0.031).The distributions overlap heavily, so the ink discrimination is much weaker than the bounding-box AUC of 0.905.Source: Figure 6 · p. 25, §A.3 · p. 25
• The positive contrast persisted across the central threshold range and four alternative estimators, but vertical-overlap-only measurement produced little separation.At a 10-pixel edge tolerance, the sign held in all five folios (207 certain, 20 uncertain; permutation p = 0.052).Source: §A.3 · p. 25
• Paired checks also favored wider certain gaps, while direct and locator-box gaps correlated only modestly (r ≈0.23).The direct-gap result corroborates that the coordinate contrast appears in the ink, but it is not an independent study because the same boxes locate tokens.Source: §A.3 · p. 25

Token order

• At caps of 250–4,000 forms, observed-space Voynich token-order shares fall from 2.35% to −0.08%, below continuous-text minima of 3.03% to 1.65%.Across 16 leave-one-quire-out refits, observed-space estimates range from 0.48% to 0.83% and merged-space estimates from 0.23% to 0.55%; at the 2,000 cap, Currier A and B give −0.49% and +0.37% (30 shuffles per stratum).Source: §A.4 · pp. 25–26

Boundary index in raw units

• The raw boundary-index mean is +1.72 bits within tokens, compared with −2.50 across intra-line separators, −2.80 for the independent-pair anchor, and −2.97 at line breaks.Normalisation makes the large negative raw scale readable without changing the ordering.Source: §A.4 · p. 26

Coordinates

• Strict line-by-line alignment retains 17,737 labelled separators, with mean scores of 0.1272 for certain and 0.0023 for uncertain separators.Across 11,306 comparable ZL–v101 decisions on 1,617 lines, the normalised gap is 0.1273 where neither marks uncertainty, 0.0300 where one does, and −0.0044 where both do; alternative pixel estimators keep the contrast positive except the vertical-overlap stress test.Source: §A.4 · p. 26, Figure 7 · p. 26

Nesting

• Quire resampling makes the standard error of mean boundary internality about 9.4 times larger than line resampling, so line-level independence is inadequate for corpus-level uncertainty.Effects have the same sign across all observed quires and persist under leave-one-quire-out checks, supporting inference for this manuscript, not universal generalisation from 16 quires as independent manuscripts.Source: §A.4 · p. 26

Line length

• Restricting analysis to middle-length lines removes apparent page-position gradients in boundary softness, token length, and glyph entropy.For 33–54-glyph lines across 156 pages and 1,768 lines, internality is +0.031 [−0.027, +0.082], token length −0.003, and entropy −0.013; first-separator and line-initial contrasts are constructed within the same line and are not generated by this confound.Source: §A.4 · p. 27

Differential calibration

• Differential calibration used five seeds per synthetic configuration, fitting an order-3 surrogate to each ciphertext and applying the same pipeline to both.Table 9 reports real-minus-surrogate language-match differentials against Latin and German; seed 0 of the three-homophone cipher reproduces the archived run at +0.393.Source: §A.5 · p. 27, Table 9 · p. 27

Naibbe cipher

• Naibbe’s attack differential was +0.002 against held-out Latin and +0.013 against German, with 1.25% order share and 40.5% singleton types.At k = 128, its differential moves only to +0.01 to +0.06, equally for the wrong language; erased-space units cross 2.2% of hidden token boundaries and 7.1% of hidden Latin word boundaries.Source: §A.5 · p. 27

Self-citation generator

• The Timm and Schinner (2020) self-citation generator produced 57–60% singleton types, under 1% order share, and only 0.006–0.009 bits of edge-glyph information.Across three seeds, its Latin attack differentials were +0.011, −0.021, and +0.018; German differentials were −0.003, +0.020, and +0.005. A crude glyph-noise surrogate reproduced none of its patterns.Source: §A.5 · p. 28

A.6 Learned units against published inventories

• Every 64-merge compound is slot-legal, but only five compound types are slot characters; two thirds of occurrences are single Stolfi constituents, while a third cross layer boundaries.The compound units also form 98% contiguous paths through Zattera’s pruned grammar; exceptions are olk, cthy, and ckhy.Source: §A.6 · p. 28, Table 10 · p. 29
• At 64 merges, compound units span three to six slots or two non-adjacent slots, while a third of Stolfi-related occurrences cross layer boundaries.The hand-built models were fitted to this corpus and the slot template admits millions of strings, so the passage identifies the level of coverage as informative.Source: §A.6 · p. 28

A.7 Currier strata and line-start detail

• The Currier A–B uncertain-separator difference is −0.009 bits [−0.052, +0.037] under independent-stratum quire bootstrap, and [−0.048, +0.035] under joint resampling.The passage defines the uncertain share and raw gap and reports class counts for both strata.Source: §A.7 · p. 29
• JSD is 0.528 for paragraph-first lines versus 0.179 for other lines, with enrichments for p, t, k, and f of 24.3, 11.9, 3.0, and 5.0 in paragraph-first lines.Using page-opening and new-left-margin lines instead also gives higher JSD (0.487 versus 0.200); the paragraph-first result has a quire-bootstrap 95% interval of [+0.31, +0.40] for the first-minus-other difference.Source: §A.7 · p. 29

A.8 Edge-glyph and token-identity order: full table

• Observed-space Voynich edge-glyph information is 0.218 bits before correction and remains 0.226 bits for interior token pairs.The table compares shuffle-corrected information across five adjacent-token pairings; y→q contributes 0.097 bits to the observed-space edge information.Source: §A.8 · p. 29, Table 11 · p. 30

A.9 Syllabic-transcription hypothesis: four matched tests

• The toneless-pinyin dependence gap rises from 1.40 bits at zero merges to 1.58 at 64 merges, while Voynich’s falls from 1.60 to 1.05 bits.A trigram pinyin attack gives language-match differentials of approximately zero for Currier A and B, compared with +0.39 for a genuine cipher against its surrogate; this tests homophonic substitution of pinyin letters, not token-as-syllable reading.Source: §A.9 · p. 30, Table 12 · p. 30

A.10 Reproduction

• Reproduction drivers take input roots explicitly, fix random seeds, accept --json-output, and have exact commands and smoke-test flags listed in the README.Defaults include 1,500 bootstrap draws, 300 matched-length windows per entropy control, 100 shuffles, 100 random quire partitions, 50 randomisations, and five cipher seeds; NumPy and Matplotlib are cited as research software (Harris et al., 2020).Source: §A.10 · p. 31