PaperlayerView exact revision on arXiv

Source-linked AI summary

The Price of Token Boundaries: Compression Certificates and Prediction

Yuhao Du, Shunian Chen

arXiv:2609.35869v1cs.AIcs.CL

TL;DR

Pre-tokenisation restricts which text fragments can become prediction units, leaving its compression cost unclear when comparisons use the same boundaries. The paper certifies minimum-token-count optima with and without boundaries and compares prediction from the resulting dictionaries. Boundaries carry a substantial compression cost, while unrestricted fitting has higher mean test bits per byte in the paired multilingual study.

  • Problem

    Tokenisation comparisons often use the same boundaries, while compression and prediction effects of boundary restrictions are usually assessed separately.

    Source: §1 · p. 1

  • Method

    The paper bounds minimum token counts with and without regex boundaries using nonnegative occurrence prices, feasible vocabularies, and independently checked certificates, then compares prediction under explicit protocols.

    Source: §1 · pp. 1–2

  • Results

    Unrestricted fitted dictionaries compress better but have higher mean test bits per byte in every language in a paired 12-language study.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sequence length alone is insufficient to choose prediction units because the compression comparison and prediction comparison answer complementary questions.

    Source: §7 · p. 9, §8 · p. 9

    The language-model findings concern measured architectures, training budgets, and decoder protocols; three seeds per language limit within-language precision.

    Source: LIMITATIONS · p. 10

Abstract

from arXiv · show

Pre-tokenisation restricts which text fragments can become prediction units, but its compression cost is obscured when tokenisers are compared only under the same boundaries. We measure this cost by bounding the minimum token count from both sides, with and without a regular-expression boundary rule. Nonnegative prices on token occurrences yield a lower bound through shortest paths and vocabulary-budget selection; maximising over all prices recovers the linear programming relaxation, and an independent integer checker certifies the reported values. On English Wikipedia, boundaries increase the optimal token count by 28.3--36.8\%. Byte pair encoding lies 2.1\% above the constrained lower bound, but 10.9\% above the unrestricted bound. Compression and prediction favour different dictionaries: at 85M non-embedding parameters and matched training-token budgets, unrestricted fitting yields higher mean held-out bits per byte under a common unrestricted decoder in all 12 languages in the paired study and 11 of 12 under independent tuning and evaluation. To study intermediate boundary policies, we introduce boundary licences, which limit the vocabulary entries permitted to cross cuts and admit the same form of certificate. On separate English and Chinese fitting corpora, licensing 10\% of the vocabulary budget recovers 85.2\% and 100.0\% of the achieved token-count reduction from removing all cuts. These results quantify the compression cost of boundaries while separating it from the prediction quality of the resulting token units.

1 INTRODUCTION

The paper measures the compression cost of regex boundaries independently of whether a vocabulary learner reaches the optimum, then compares prediction from dictionaries fitted with and without those boundaries. Its paired multilingual results show that compression and prediction can favor different dictionaries.

1 INTRODUCTION

  • A regex boundary can prevent tokens from spanning word boundaries, increasing prediction steps while potentially preserving useful prediction units.Compression methods seek shorter sequences, whereas language-model experiments compare the quality of the resulting token streams.Source: §1 · p. 1
  • A certificate within the regex-constrained feasible set cannot measure the cost of the boundary constraint itself.The cited compression observations compare different optima, and unrestricted documents have more candidate occurrences as corpus bytes times maximum token length.Source: §1 · p. 1
  • Nonnegative prices on candidate occurrences yield a lower bound using shortest-path cost minus the largest K aggregate token bids.The certificate specializes Lagrangian relaxation and does not require a general-purpose LP solver; a feasible vocabulary supplies the other end of the interval.Source: §1 · pp. 1–2
  • Unrestricted fitted dictionaries compress better but have higher mean test bits per byte in every language in a paired 12-language study.With independent tuning and evaluation seeds, the direction remains in 11 of 12 languages.Source: §1 · p. 2
  • Boundary licences limit which vocabulary entries may cross regex cuts, with endpoints reproducing constrained and unrestricted optimization and intermediate budgets measuring available compression.The paper identifies these policies as a separate design choice because vocabulary content, token frequencies, and effective byte context change together in the boundary comparison.Source: §1 · p. 2
  • The paper combines two-sided optimal-tax intervals and exact numerical checking with held-out prediction comparisons under an explicit vocabulary and decoder protocol.It also formulates boundary licences and measures their compression frontiers.Source: §1 · p. 2

2 RELATED WORK

The paper places its contribution between vocabulary-optimization methods and work comparing compression with prediction quality. It evaluates the relaxation on unrestricted document graphs and adds a budget on crossing permissions.

2 RELATED WORK

  • The price certificate evaluates the shared ConvexTok vocabulary-selection relaxation on unrestricted document graphs without a general-purpose solver.Its optimization and numerical verification are separate computations, and the decomposition separates shortest-path segmentation from vocabulary-budget selection.Source: §2 · p. 2
  • SuperBPE and BoundlessBPE construct vocabularies spanning conventional boundaries, while this tax interval measures boundary-rule compression cost independently of a vocabulary-construction heuristic.PathPiece and TokEval motivate assessing intrinsic compression alongside language-model quality; this paper connects these questions with certificates, decoder-protocol comparisons, and a crossing-permission budget.Source: §2 · pp. 2–3

3 THE OCCURRENCE-PRICE CERTIFICATE

The certificate compares minimum-token-count optima under a boundary rule and without one, using occurrence prices to produce a valid lower bound. Its price family matches the LP relaxation, while separate integer checking makes reported bounds deterministic.

3.1 VOCABULARY SELECTION AND BOUNDARY CONSTRAINTS

  • The optimal token count is defined by selecting at most K additional vocabulary entries and minimizing token count over corpus documents.The vocabulary includes all 256 byte symbols, and candidate substrings have lengths 2 through L.Source: §3.1 · p. 3
  • Minimum-count encoding is a shortest path through the usable candidate-occurrence graph for the selected vocabulary.A pre-tokeniser deletes occurrences crossing cuts while preserving byte edges, so the constrained optimum cannot be smaller than the unrestricted optimum; their comparison is the optimal pre-tokenisation tax, not a ratio of trained vocabularies.Source: §3.1 · p. 3

3.2 PRICES AND THE CERTIFIED BOUND

  • Occurrence prices separate path costs from vocabulary selection: each path pays for used occurrences, and subtracting the largest admissible aggregate token bids makes the bound conservative.The integer programme uses edge-use and vocabulary-inclusion variables, linking multibyte edges to selected strings; byte edges remain available.Source: §3.2 · p. 3
  • Nonnegative occurrence prices add price to multibyte edges in the complete candidate graph, while each token’s bid sums prices over its occurrences.For any prices, the priced shortest-path cost minus the top-K bid sum is a lower bound, and optimizing over all prices equals the LP relaxation optimum.Source: §3.2 · pp. 3–4
  • The full price family loses nothing relative to the LP, although a finite optimizer may not reach its maximum and the LP may remain below the integer optimum.The implementation uses an inexpensive price initialization and vocabulary proposals; complete vocabularies are rescored before acceptance, and only accepted improvements change the upper bound.Source: §3.2 · p. 4

3.3 FROM A FORMULA TO CONSERVATIVE REPORTED NUMBERS

  • An independent checker converts saved prices to nonnegative multiples of 2^-32 and verifies bids and shortest paths using checked integer arithmetic.Its rational dual value is valid independently of floating-point error estimates or optimizer convergence; reported endpoints are rounded outwards and are deterministic optimization intervals, not sampling confidence intervals.Source: §3.3 · p. 4
  • A positive integer lower bound and a feasible token count in each regime provide the two endpoints used for the reported intervals.The intervals compare the boundary-constrained and unrestricted regimes.Source: §3.3 · p. 4

4 THE CERTIFIED COST OF PRE-TOKENISATION

The section measures the compression cost of pre-tokenisation using certified intervals rather than comparing tokenisers only within the constrained regime. English BPE appears close to the constrained lower bound, but the unrestricted comparison and cross-language results reveal a larger and variable boundary tax.

The certified cost of pre-tokenisation

  • The Wikipedia study uses twelve languages, about 42 MB per language, K = 32768 non-byte entries, and L = 16, scoring minimum-count decoding under specified cuts.Each baseline receives the same corpus, length cap, and vocabulary budget.Source: §4 · p. 4
  • 2.1% above the regex-constrained lower bound, English BPE is 10.9% above the unrestricted bound.SLOTPRICE improves on unrestricted BPE by 4.39% in achieved token count.Source: §4 · pp. 4–5, Table 1 · p. 5
  • 28.3–36.8%: English’s optimal pre-tokenisation tax; the corresponding Chinese interval is 9.8–11.0%.The intervals combine a lower bound from one regime with a feasible vocabulary from the other.Source: §4 · p. 5
  • 10.1–56.2%: the achieved pre-tokenisation tax varies across languages, while unrestricted SLOTPRICE uses fewer tokens than BPE and Two-stage BPE in all twelve.Figure 2 shows the language-specific optimisation intervals.Source: §4 · p. 5
  • Across tested budgets, achieved taxes and both certified interval endpoints rise in English and Chinese; English’s constrained BPE gap narrows from 9.8% to 1.3%.Alternative deployed pre-tokenisers yield overlapping tax intervals within 25.3–37.0% on this corpus.Source: §4 · p. 5
  • On 14 small instances, price optimisation reaches 99.59–99.99% of the LP objective, but interval width can reflect price, vocabulary-search, or LP–integer gaps.The interval remains valid; its width measures remaining compression uncertainty from these sources.Source: §4 · p. 5

5 PREDICTION AT MATCHED TRAINING BUDGETS

The matched-budget study compares regex-trained and unrestricted dictionaries under a common unrestricted decoder, then tests whether the prediction contrast persists with independent tuning and evaluation. Unrestricted fitting has higher mean test bits per byte in all twelve languages in the paired study and in eleven of twelve under independent tuning.

Matched-budget prediction comparison

  • The paired comparison changes dictionary construction while both dictionaries encode complete documents with unrestricted minimum-count Viterbi decoding.Each language uses the same 85M non-embedding-parameter Transformer, 33,152-entry padded vocabulary, and 0.629 G training tokens per arm.Source: §5 · pp. 5–6
  • The compression tax and prediction penalty rank languages differently: Vietnamese has the highest certified tax but a below-median prediction penalty, while Arabic shows the reverse pattern.Figure 2 aligns language-specific measurements from separate corpora and protocols.Source: Figure 2 · p. 6
  • 9.25%: unrestricted dictionaries’ equal-language mean increase in test bits per byte, with higher mean test bits per byte in all 12 languages.The reported relative increases range from 0.93% to 17.31%.Source: §5 · p. 6

Independent tuning and evaluation

  • 11 of 12 languages: unrestricted fitting increases mean test bits per byte with independent tuning and evaluation; the equal-language mean increase is 6.99%.With history capped at 1024 raw bytes, the direction remains in 11 languages and the mean is 6.70%; Thai has small negative means in both modes.Source: §5 · p. 7, Table 2 · p. 7

Scale and training allocation

  • At 24M to 387M parameters, the SLOTPRICE penalty against BPE falls from +18.32% to +2.65% on the measured English D = 20N ladder.At 996M parameters, locally trained Two-stage BPE and released SuperBPE have penalties of 2.87% and 0.72% against native BPE with regex boundaries; these comparisons use different decoder protocols.Source: §5 · p. 7

Byte exposure and linguistic evaluation

  • The scaling and linguistic mechanism remains open: BLiMP’s matched-byte and matched-token deficits are −4.5 and −3.6 points, with the budget-response difference unresolved.The 387M control has two seeds per cell; the 1.88B English comparisons use two checkpoints per arm.Source: §5 · p. 7

6 BOUNDARY LICENCES

Boundary licences vary how many vocabulary types may cross regex cuts while keeping the vocabulary budget and candidate universe fixed. The certificates extend to intermediate licence budgets, where small budgets recover substantial achieved compression gains on the English and Chinese fitting corpora.

Boundary-licence formulation

  • A licence belongs to a vocabulary string, but legality is checked at each occurrence; q counts licensed types, not crossing occurrences.Within-pretoken occurrences can use any string in V, while crossing occurrences require a string in W.Source: §6 · pp. 7–8
  • The licence endpoints reproduce the constrained and unrestricted optimisation problems, and feasible sets expand as q grows.The occurrence rule is also applied when scoring fitting and held-out compression; Section 5 uses a separate unrestricted-decoder comparison.Source: §6 · p. 8

Certificate for intermediate budgets

  • The intermediate-budget certificate separates selected types by crossing bids and evaluates splits between q licensed and K − q unlicensed selections.Prefix and suffix heap passes evaluate all splits in O(m log(K + 1)) time after sorting.Source: §6 · p. 8

Measured compression frontiers

  • 85.2% and 100.0%: a 10% licence budget recovers this share of achieved token-count reduction in English and Chinese, respectively.Certified shares of the optimal endpoint gain are [50.5, 100.0]% for English and [38.7, 100.0]% for Chinese.Source: §6 · p. 8, Figure 3 · p. 9
  • At a 2% licence budget, achieved counts fall below the zero-licence certified lower bound in both languages, proving improvement over every zero-licence vocabulary in this model on these fitting corpora.Chinese takes only 3,276 of 8,192 offered licences at q = 8,192, with held-out compression unchanged from the 10% rung.Source: §6 · p. 8

Cut-placement control

  • With zero licences, permuting cut placement while preserving cut counts and Unicode-character piece-length multisets changes optimal token count by [47.1, 53.5]% in English and [3.9, 6.5]% in Chinese.At full licences, both cut sets define the same true optimum.Source: §6 · p. 8

Measured compression frontiers

  • Figure 3 distinguishes achieved endpoint recovery from certified lower bounds on optimal recovery across licence budgets; its upper bound is 100% at every interior budget.The curves concern optimisation bounds, not sampling uncertainty; connecting lines do not interpolate certificates.Source: §6 · p. 8, Figure 3 · p. 9

7 DISCUSSION

The certificate and language-model comparisons answer complementary questions: one measures compression excluded by boundaries, while the other compares prediction from the resulting dictionaries. Boundary licences make intermediate compression policies measurable, but prediction measurements are still needed to judge their usefulness.

7 DISCUSSION

  • At fixed corpus, length cap, and vocabulary budget, the certificate measures compression excluded by a boundary rule, while language-model comparisons measure prediction from the resulting dictionaries.A small optimisation gap within the constrained problem cannot settle either the constraint's compression cost or its value for prediction.Source: §7 · p. 9
  • Boundary licences allocate permissions to selected string types while legality remains occurrence-specific, exposing compression frontiers for intermediate policies.The measured frontier places much of the achieved compression gain at small licence budgets, making those policies candidates for future language-model evaluation.Source: §7 · p. 9
  • Compression certificates quantify available sequence-length reduction; prediction measurements determine which reductions are useful.Source: §7 · p. 9

8 CONCLUSION

Occurrence pricing certifies the compression cost of pre-tokenisation, while language-model comparisons show that sequence length alone is insufficient to choose prediction units. Boundary licences extend the certificate to intermediate policies.

8 CONCLUSION

  • The multilingual comparisons favour regex-constrained fitting in most languages, including under independent tuning and evaluation, showing why sequence length alone is insufficient to choose prediction units.Source: §8 · p. 9
  • Occurrence pricing certifies the compression cost of pre-tokenisation without requiring an exact vocabulary search.The resulting intervals expose a substantial boundary cost that constrained optimality comparisons conceal.Source: §8 · p. 9
  • Boundary licences extend the certificate to intermediate policies and quantify how much compression they recover.Source: §8 · p. 9

LIMITATIONS

The certificates and language-model findings have defined scope boundaries: certificates are tied to specified corpora and tokenisation settings, while prediction results are tied to measured model and decoder protocols.

LIMITATIONS

  • Certificates apply to specified finite corpora, vocabulary budgets, and maximum token lengths; other text domains require their corresponding corpora.The cut-placement control preserves character-length multisets while allowing byte-length variation.Source: LIMITATIONS · p. 9
  • Language-model results concern the measured architectures, training budgets, and decoder protocols, with three seeds per language limiting within-language precision.Source: LIMITATIONS · p. 10
  • Token matching fixes training-token allocation, but byte exposure and context span vary jointly, so the measured contrast also combines those changes with vocabulary content.Vocabulary-search effort also differs between regimes; matching it would further separate the fitting constraint from optimisation quality.Source: LIMITATIONS · p. 10
  • Extending prediction analysis to larger models and intermediate licence budgets remains future work.The largest reported model has 1.88B non-embedding parameters.Source: LIMITATIONS · p. 10

A PROOFS AND NUMERICAL CERTIFICATION

This section establishes and validates price-based lower-bound certificates for vocabulary selection and boundary licences, alongside corpus constraints, baseline-scoring procedures, and the scope and interpretation of the reported compression comparisons.

A.1 VOCABULARY SELECTION AND PRICE DUALITY

  • The cardinality-constrained integer program equals OPT(K), and the price dual yields a valid lower bound; at a self-certifying solution, the certificate and segmentation count prove optimality.The setup assumes a nonempty corpus, K ≥ 1, and omits candidate types with no surviving occurrences.Source: §A.1 · p. 13, A PROOFS AND NUMERICAL CERTIFICATION · p. 13, §0 · p. 14

0. For the remaining term

  • Maximising the occurrence-price certificate over all nonnegative prices recovers the LP optimum, while finite optimisation, restricted prices, and feasible vocabularies can leave distinct gaps.The reported UB/LB −1 bounds a vocabulary’s relative suboptimality from above without assuming these gaps vanish.Source: §0 · p. 14

A.2 RESTRICTED PRICING AND EXACT VERIFICATION

  • A certificate can be checked in O(BL + |C|) time using shortest paths and top-K selection; the independent integer checker verifies prices, bids, paths, and corpus-derived occurrence groups.Every evaluated price vector remains a valid certificate, but the stated O(T^-1/2) rate does not apply to the production schedule using momentum and background prices.Source: §A.2 · pp. 15–16

A.3 CERTIFIED COST INTERVALS AND GENERAL BUDGETS

  • At fixed vocabulary budget K, the certified boundary tax is nonnegative and nondecreasing as cuts are refined, but this result implies no monotonicity in K.The associated selection-oracle argument extends certificates to admissible vocabulary families through an outer relaxation; excluding an admissible vocabulary need not preserve the bound.Source: §A.3 · pp. 16–17

A.4 OCCURRENCE-AWARE CROSSING LICENCES

  • Boundary licences use vocabulary and licence budgets to admit crossing occurrences, with q = 0 matching cut-constrained paths and q = K matching unrestricted vocabularies.The selection support is exactly characterised by a continuous selection polytope, but this does not guarantee integrality of the joint flow/linking/selection LP.Source: §A.4 · pp. 17–18
  • Occurrence-aware licensing can have an integrality gap: for the example at q = 1, the integer optimum is 3 while the LP optimum is 2.At q = 0 and q = 2, both optima are 3 and 2, respectively; the example does not attribute production intervals to integrality rather than finite price optimisation or primal search.Source: §A.4 · p. 18

B.1 DATA AND CONSTRAINTS

  • The main compression grid uses approximately 42 MB prefixes from 20231101 Wikipedia dumps in twelve languages, with K = 32768 non-byte entries and maximum token length L = 16 bytes.The constrained regime uses an o200k-style regex; the unrestricted regime has no internal cuts, and both preserve document boundaries.Source: §B.1 · pp. 18–19

B.2 BASELINE CONSTRUCTION AND SCORING

  • Compression scores are recomputed from hexadecimal vocabularies with an independent byte trie, using minimum-count shortest-path decoding separately from greedy longest match.BPE and unigram baselines use distinct construction procedures, and native BPE ranked merges are used for BPE language-model streams.Source: §B.2 · p. 19

B.3 PRICE AND VOCABULARY OPTIMISATION

  • Price-based shortest-path bounds guide vocabulary construction, and an independent integer checker verifies the reported conservative lower bound.The search proposes additions, swaps, and ruin-and-recreate moves, retaining only exactly scored strict improvements. The checker quantises saved prices and checks the full graph and rational dual value.Source: §B.3 · pp. 19–20

B.4 VALIDATION AND RESOURCE ACCOUNTING

  • The certificate is validated against small LP and integer programmes, exhaustive checker tests, and verified candidate identities, occurrence counts, and input hashes.The certificate applies specifically to the validated inputs; the LM inventory contains 1186 runs, while the paired main test has 36 seed pairs across twelve languages.Source: §B.4 · p. 20

B.5 COMPLETE LANGUAGE SCORES

  • Table 3 reports achieved bytes per token for twelve languages and outward-rounded certified optimal regex-tax intervals under a common 32768-entry budget.The upper bound is corpus bytes divided by the conservative token-count lower bound; achieved values use the independently scored token count, and each baseline uses its actual length-eligible vocabulary size.Source: §B.5 · p. 20, Table 3 · p. 21

B.6 SMALL-INSTANCE VALIDATION

  • The Table 4 ratio factors multiply, so their percentage gaps do not form an additive decomposition.The comparisons distinguish relaxation gap from price-optimisation and vocabulary-search error.Source: §B.6 · p. 20

B.7 BUDGETS, REGEXES AND DEPLOYED TOKENISERS

  • Tables 5–7 extend measurements across vocabulary budgets and deployed pre-tokenisers, and compare native with minimum-count decoding without claiming optimality for released vocabularies.The English and Chinese measurements are repeated across budgets; bounds and interval endpoints are rounded outwards from exact rational values.Source: §B.7 · pp. 20–21

B.8 ALGORITHMIC ABLATIONS

The ablations distinguish optimization diagnostics from certified bounds, while the language-model comparison specifies how dictionaries are fitted, encoded, and evaluated. The reported table and protocol captions define the scope of those measurements.

B.8 ALGORITHMIC ABLATIONS

  • Preconditioning and momentum improve the dual objective, larger active sets help with diminishing returns, and local search improves feasible vocabularies; iterate averaging makes no measurable difference.These are optimizer diagnostics, not certified optimal-tax intervals.Source: §B.8 · p. 21
  • Table 4 compares checked dual bounds with LP and integer-program reference solves, separating relaxation, primal, and dual gaps.The table’s reference solves use HiGHS with numerical solver tolerances.Source: Table 4 · p. 22
  • The budget sweep reports certified tax intervals and BPE gaps across vocabulary budgets, while separate tables compare deployed pre-tokenisers and released encoders.Released-encoder results report achieved compression without an optimization bound.Source: Table 5 · p. 22, Table 6 · p. 23, Table 7 · p. 23

C.2 TEST-ONLY MULTILINGUAL INFERENCE

  • The primary LM panel pairs three training seeds across twelve languages, uses equal language weighting, and evaluates test loss after within-seed validation-based rate selection.Hindi and Thai lack matching selected BPE test pairs, so that contrast covers ten languages; exact scored-byte normalization does not guarantee identical scored text.Source: §C.2 · pp. 24–25

C.1 VOCABULARY CONSTRUCTION AND STREAM DECODING

  • The LM protocol distinguishes vocabulary fitting, admissible encoding occurrences, and segmentation objectives; regex-trained byte vocabularies do not themselves impose boundary masks at encoding.The compared SLOTPRICE pair holds the inference rule fixed while changing the vocabulary-fitting constraint, although vocabulary-search effort differs.Source: §C.1 · p. 24, Table 9 · p. 25

C.2 TEST-ONLY MULTILINGUAL INFERENCE

  • Table 10 reports mean test bits per byte and paired relative changes for regex-fitted versus unrestricted dictionaries under document-level Viterbi encoding.The table reports three-seed paired-t intervals and Holm-corrected tests across the twelve language contrasts.Source: Table 10 · p. 25

C.3 LEARNING-RATE AND TRAINING-SCALE CONTROLS

The controls examine prediction comparisons across learning-rate, scale, and training-allocation settings, alongside boundary-licence experiments and related work. The accompanying material specifies the scope and reproducibility of these measurements.

C.3 LEARNING-RATE AND TRAINING-SCALE CONTROLS

  • At 996M parameters and K = 200k, SLOTPRICEπ reaches 0.78117 bits per byte against BPE’s 0.78366 in the selected-rate test comparison, but the ranking depends on rate tolerance.The selected comparison uses one seed per arm; at twice the centre rate, the reported means are 0.78162 ± 0.00040 for SLOTPRICEπ and 1.59178 ± 0.00596 for BPE.Source: §C.3 · p. 26
  • Under the D = 20N English ladder, the SLOTPRICE penalty falls from +18.32% at 24M to +2.65% at 387M.Model size and training tokens increase together, fixing training tokens per non-embedding parameter under replacement sampling.Source: §C.3 · p. 26

C.4 BLIMP AND BYTE EXPOSURE

  • SLOTPRICE trails SLOTPRICEπ on BLiMP by −4.0 points at 85M and −2.9 at 996M; the morphology gap narrows from −6.9 to −3.1 points.Because model size, vocabulary size, training budget, and byte exposure all change, the pattern does not isolate a morphological mechanism; at 1.88B, differences are −3.3 points against both BPE and SLOTPRICEπ.Source: §C.4 · p. 28
  • At 387M, the matched-token BLiMP difference is −3.6 points, while the matched-byte difference widens from −3.6 to −4.5 points; the budget response remains unresolved with two seeds per cell.The longer-minus-shorter within-arm changes are +0.9 points for SLOTPRICE and −0.1 for SLOTPRICEπ.Source: §C.4 · p. 28

D BOUNDARY-LICENCE PROTOCOL AND CONTROLS

  • The boundary-licence protocol uses complete UTF-8 fitting records and disjoint held-out slices; its smaller corpora and K = 8,192 cut-placement control keep candidate graphs and exact checks tractable.The candidate class includes distinct substrings of lengths 2 through 16 within fitting records, including strings whose bytes are not independently valid UTF-8.Source: D BOUNDARY-LICENCE PROTOCOL AND CONTROLS · p. 29
  • With K = 8,192, fitting-selected English and Chinese regex/random cut pairs differ by 0.12% and 0.14% in symmetric fitting compression, respectively.Their common-regex crossing-rate differences are 11.46 and 6.33 percentage points, respectively; these pairs were selected using fitting diagnostics alone.Source: D BOUNDARY-LICENCE PROTOCOL AND CONTROLS · p. 30

E RELATED WORK AND SCOPE

  • The paper’s certificate specializes vocabulary-selection pricing to shortest-path segmentation and a top-K support sum, while boundary licences replace that support function with nested selection.The full price family attains the same LP optimum, and the specialized computation avoids a general-purpose LP solve.Source: E RELATED WORK AND SCOPE · pp. 31–32

F REPRODUCTION AND INDEPENDENT REPLICATION

  • The supplementary archive verifies recorded measurement arithmetic and reported intervals; full-corpus certification additionally evaluates candidate graphs and saved occurrence prices.All data and scripts needed for the listed checks are included in the archive.Source: F REPRODUCTION AND INDEPENDENT… · p. 32

F.1 INDEPENDENT TUNING AND EVALUATION

The independent replication selects learning rates using separate tuning seeds, then compares test results on disjoint evaluation seeds under three context policies. The results favor regex-fitted dictionaries in 11 of 12 languages in sliding evaluation, with a scope limited to fitted dictionaries.

F.1 INDEPENDENT TUNING AND EVALUATION

  • The replication selects rates on two tuning seeds and evaluates paired test contrasts on three disjoint seeds across token-block, byte-history, and sliding contexts.Table 15 reports candidate grids and selected rates; each selected rate lies strictly inside its grid.Source: §F.1 · p. 32, Table 15 · p. 33
  • In sliding evaluation, 11 of 12 language contrasts favor the regex-fitted dictionary; Thai is −0.39% [−1.26, 0.47].The listed positive sliding contrasts range from 3.16% for Japanese to 14.04% for Arabic.Source: Table 16 · p. 33, §F.1 · p. 34
  • The reported intervals describe variation across paired evaluation seeds after rate selection, and the measurements cover fitted dictionaries rather than intermediate boundary-licence budgets.Holm correction covers the twelve languages separately within each context mode.Source: §F.1 · p. 34

Related papers

Loading 2609.35869v1…