PaperlayerView exact revision on arXiv

Source-linked AI summary

Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

Hunter Heidenreich

arXiv:2607.05691v1cs.CLcs.LGq-bio.BM

TL;DR

Whether BPE and Unigram-LM produce interchangeable tokenizers for chemistry SMILES remains unclear because prior comparisons did not isolate the algorithm under matched conditions. This study controls the chemistry base, corpus, vocabulary size, and boundary policy, finding that the algorithms produce different vocabularies and segmentation granularity across the tested grid.

  • Problem

    Whether BPE and Unigram-LM still shape different chemistry-SMILES tokenizers under matched conditions has not been established on a fixed chemistry-grammatical base.

  • Method

    The study compares BPE and Unigram-LM using a shared 165-token base across three corpus typologies, vocabulary sizes, and two pre-tokenization boundary policies.

  • Results

    Across 22 matched conditions, BPE and Unigram-LM produce different vocabularies and segmentation granularity, with the contrast stable across corpus typology, boundary policy, and vocabulary scale.

  • Takeaways & Limitations

    The subword algorithm is a genuine modeling decision rather than a neutral default for chemical language-model tokenization.

  • Takeaways & Limitations

    The study does not measure downstream language-model quality or exercise Unigram-LM’s subword-regularization capability.

Abstract

from arXiv · show

Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.

1 Introduction

On chemistry SMILES, BPE and Unigram-LM produce structurally distinct tokenizers rather than interchangeable vocabularies, even under controlled matched conditions. This tokenizer-level study compares them without training language models or claiming downstream superiority.

  • Experimental framing: 165-token chemistry coverage isolates the subword-algorithm axis by eliminating unknown-token failures on conformant SMILES.The study begins from Smirk’s complete OpenSMILES base, which emits no [UNK] on conformant input.
  • Scope: The separation persists across corpus typology, boundary policy, and vocabulary size, including at eight times the small-vocabulary scale.The comparison spans diverse, drug-like, and natural-products corpora and both pre-tokenization boundary policies.
  • Study scope: The study treats tokenizer construction as the object of analysis, training no language models and making no claim that either algorithm is better downstream.Its purpose is to establish whether the resulting vocabularies are interchangeable before downstream comparison.

2 Background and related work

BPE and Unigram-LM make contrasting vocabulary choices through opposite construction procedures, while pre-tokenization boundaries provide a separate chemistry-specific design axis. Prior chemistry evidence shows divergence, but this study isolates the algorithms using fixed chemistry bases, controlled boundaries, diverse corpora, and small learnable vocabularies.

  • Scope: SMILES is the study’s representation; SELFIES and DeepSMILES are out of scope because they change notation rather than the subword algorithm.Representation choice remains consequential but is orthogonal to the BPE-versus-Unigram contrast.
  • Tokenization algorithms: BPE greedily merges frequent adjacent symbol pairs bottom-up, whereas Unigram-LM fits and prunes a large seed vocabulary top-down using corpus likelihood.BPE is deterministic for a given corpus and target size; Unigram-LM provides segmentation probabilities.
  • Boundary policy: Pretokenization confines subword merges within initial chunks, making whether merges cross bracketed-atom boundaries a first-class chemistry design axis.In natural language, boundary policy can have greater impact than vocabulary size or corpus choice.
  • Related chemistry work: The prior chemistry head-to-head learned BPE, WordPiece, and Unigram vocabularies on approximately 2.3M ChEMBL SMILES at V ∈{8K, 16K, 32K}.It found approximately 80% BPE–WordPiece overlap, at most approximately 47% Unigram overlap with the others, and longer, less often RDKit-valid Unigram chemical words.
  • Corpus design: The study spans PubChem, ZINC-22, and COCONUT as diverse, drug-like, and natural-products corpora, respectively, while REAL-Space serves only as an anchor.This corpus design represents alphabet diversity, topology, and functional-group distribution.
  • Vocabulary size: Above roughly 1,000 learned tokens, pieces can fire too rarely to train embeddings, making the small-vocabulary regime especially consequential when high-frequency patterns must be selected.This token-usage dead-zone motivates focusing on vocabulary sizes where token embeddings remain learnable.

3 Methods

The study isolates BPE versus Unigram-LM by training matched tokenizers from a shared 165-token base across corpus, vocabulary-size, and boundary conditions. It evaluates vocabulary membership, granularity, distribution, positional segmentation, learnability, and robustness under controlled preprocessing and sensitivity sweeps.

  • Experimental design: 18 matched headline conditions vary algorithm, three corpus typologies, three vocabulary sizes, and two boundary policies, yielding 36 headline tokenizers.Sensitivity and anchor blocks expand the study to 44 trained tokenizers across 22 matched conditions.
  • Tokenizer construction: Both arms consume identical glyph-id streams from Smirk’s 165-token base, with BPE adding merges and Unigram-LM pruning a seeded pool toward the same target size.The trainers use reference defaults apart from three documented design exceptions, rather than hand-tuning either algorithm.
  • Boundary policy: The sole varied pre-tokenization knob is bracket permeability: NMB makes bracketed atoms opaque, whereas MB permits merges across their boundaries.MB is Smirk-GPE’s default, while the study systematically tests NMB across the small-vocabulary regime.
  • Evaluation measures: Seven structural quantities per condition measure vocabulary overlap, fertility, frequency imbalance, positional boundary agreement, and mechanism diagnostics on realized vocabularies, training data, and held-out splits.Overlap is evaluated with unweighted and frequency-weighted membership variants, including structural-only recomputations.
  • Learnability certification: F95%,100 requires at least 95% of each arm’s learned pieces to fire at least 100 times in training, certifying the learnable regime before mechanism diagnostics are interpreted.The clearance cn excludes the shared base and measures the fraction of learned pieces meeting the count threshold.
  • Sensitivity analysis: One-factor-at-a-time hyperparameter sweeps, four pairwise interactions, and four structural probes test robustness on a representative subsample while the headline grid remains full-corpus.Because BPE has only merge frequency as a free training knob, the sweep is necessarily weighted toward Unigram-LM parameters.

4 Results

Across 22 matched conditions, BPE and Unigram-LM remain strongly separated: their learned vocabularies are near-disjoint, Unigram-LM segments molecules more finely, and the divergence persists across corpus, boundary policy, and scale. Despite this, the algorithms agree on most cut locations, differing mainly in segmentation depth.

  • Membership: Jw is at most 0.05, while J never exceeds 0.161, showing near-disjoint vocabularies with especially little overlap among high-frequency pieces.Self-overlap is J=1.0, whereas cross-arm overlap ranges from 0.010 to 0.161; weighting by frequency deepens rather than softens the disagreement.
  • Granularity: 29.2–41.0% is the relative fertility gap: Unigram-LM emits more tokens per held-out molecule than BPE in every condition.BPE packs ∼1.4–1.8 glyphs per token, while Unigram-LM remains near-atomic at ∼1.0–1.2 glyphs per token.
  • Compatibility: The arms agree on most cut locations, so their near-disjoint vocabularies and fertility gap mainly reflect different segmentation depths rather than fundamentally incompatible boundaries.Granularity is measured on held-out segmentation and is unaffected by unequal realized vocabulary sizes.
  • Distribution: 0.038 is the smallest token-imbalance gap, remaining above the ∼0.002 corpus-draw noise floor in every condition, although its magnitude is corpus-sensitive.On natural-products data at V=1024, |∆D|=0.038 versus ∼0.07 on diverse and drug-like corpora.
  • Stability: Across corpus typology, boundary policy, and vocabulary scale, membership and granularity remain separated, while the fertility gap rises only slightly with vocabulary size.At V=8192, PubChem overlap remains J=0.083 and Jw=0.022, while Unigram-LM still segments 34.2% more finely.

5 Discussion

BPE and Unigram-LM remain structurally distinct on chemistry SMILES: they learn near-disjoint vocabularies and systematically different sequence lengths across corpora, boundary policies, and learnable vocabulary sizes. Thus, the subword algorithm is a genuine modeling decision, although the study limits its mechanistic claims and does not establish downstream language-model effects.

  • Structural divergence: BPE and Unigram-LM build near-disjoint vocabularies and segment molecules to systematically different lengths across corpus types and both boundary policies.The contrast persists across diverse, drug-like, and natural-products corpora, with cuts nesting more often than crossing.
  • Interpretation relative to prior work: The controlled comparison complements, rather than contradicts, field-wide tokenizer surveys because those surveys mix heterogeneous bases and mostly larger native vocabularies.The discussion attributes the different conclusions to coverage dominating under heterogeneous, large-vocabulary conditions.
  • Vocabulary-size limit: V =2048 already places Unigram-LM in the undertrained tail on PubChem, while both arms are unsafe at V =8192.The discussion frames chemistry’s learnable vocabulary ceiling as low and ties it to the fixed 165-token base.
  • Implications for chemical-LM builders: The subword algorithm changes both the embedding units and effective sequence length, with BPE producing roughly one-third fewer tokens than Unigram-LM.The practical implication is that swapping algorithms changes which substructures become atomic units, especially among high-frequency pieces.
  • Mechanistic limits: The supported mechanism is that the algorithms partition the same constrained glyph stream differently, not that SMILES has natural-language-style gold morphological alignment.The claimed evidence includes near-disjoint vocabularies, a consistently signed fertility gap, higher whole-pretoken absorption for BPE, and a one-sided segmentation-entropy signal.

6 Limitations

The study is limited to tokenizer-level evidence: it does not assess downstream language-model quality or Unigram-LM subword regularization. Its headline contrasts also lack per-condition error bars, while corpus size, transferability of the learnability threshold, and the fixed chemistry base introduce methodological qualifications.

  • Scope: The study does not measure downstream language-model quality or exercise Unigram-LM’s subword-regularization capability.These questions are left to follow-on work.
  • Uncertainty: J and Jstruct have no per-condition error bars, although retraining jitter is zero in every cell and J varies ≲0.005 across corpus redraws.The determinism check bounds retraining jitter below ∼0.001 on the J scale; redraws use smaller subsamples than the full comparison.
  • Corpus design: The three headline corpora differ in both typology and size, creating a size confound addressed by size sweeps, REAL-Space, and size-matched probes.The size-matched probe subsamples PubChem and ZINC-22 to COCONUT’s training size and attributes the one corpus-dependent contrast to typology rather than size.
  • Learnability: The NMT-derived learnability bar F95%,100 transfers to chemistry only under an assumption.The threshold is derived from NMT rather than chemistry-specific evidence.

7 Conclusion

BPE and Unigram-LM remain structurally distinct tokenizers for chemistry SMILES across matched conditions and at larger scale, so they are not interchangeable defaults. The conclusion identifies the algorithm—not the corpus—as the key tokenizer choice and motivates downstream-model testing.

  • 7 Conclusion: 22 matched conditions yielded near-disjoint BPE and Unigram-LM vocabularies across corpus typologies, boundary policies, and learnable vocabulary sizes.The algorithms did not converge despite the constrained chemistry alphabet.
  • 7 Conclusion: V =8192, eight times the headline vocabulary, left the algorithms’ contrasts essentially unmoved.The separation therefore persists with scale rather than reflecting only small-vocabulary behavior.
  • 7 Conclusion: The algorithm, rather than the training corpus, is the main tokenizer lever because vocabularies transfer across domains at near-native cost.For BPE, transfer remains within ±1%.
  • 7 Conclusion: The result rejects treating BPE and Unigram-LM as interchangeable tokenizer defaults and motivates testing whether either benefits downstream models.The study establishes tokenizer-level separation, not downstream-model superiority.

Funding · A Methods and reproducibility detail · A.1 Preprocessing and corpus construction

The study received no external funding and documents reproducible methods spanning preprocessing, corpus construction, tokenizer training, sensitivity analyses, structural probes, and learnable-regime confirmation. Corpus preparation uses canonicalization, exact-string deduplication, base-conformance filtering, deterministic subsampling, and held-out splits, with specialized handling for out-of-distribution corpora.

  • Funding: No external funding supported this work.
  • A Methods and reproducibility detail: The methods appendix covers corpus construction, tokenizer training and byte-level determinism, sensitivity analyses, structural probes, and learnable-regime confirmation.
  • A.1 Preprocessing and corpus construction: Each corpus undergoes RDKit canonicalization, exact-string deduplication, conformance filtering, recorded parse-drop rates, and pinned-version processing.Canonicalization uses MolFromSmiles → MolToSmiles in non-Kekulé aromatic form; tautomers and protonation states remain distinct under exact-string deduplication.
  • A.1 Preprocessing and corpus construction: ∼344M molecules were scanned with the 165-token bare-base oracle, which drops molecules emitting [UNK] before subsampling, splitting, or tokenizer training.COCONUT, ZINC-22, and REAL-Space were 100% conformant, with 0 dropped across 738,801, 83.6M, and 136.4M molecules, respectively.
  • A.1 Preprocessing and corpus construction: Deterministic hash-partition subsampling reduces oversized corpora to Table 1 totals, while each held-out test split is 5%, capped at 106 molecules, and training-disjoint.The reported conformance-scan counts are pre-subsample.
  • A.1 Preprocessing and corpus construction: The out-of-distribution corpora use trained PubChem tokenizers; CycPeptMPDB needs no handling, while tmQM requires derived conformant encoding from dative-bond conversion and valence recomputation.The tmQM representation changes implicit-hydrogen bookkeeping on metal-bound atoms.

A.2 Tokenizer training and determinism … B.1 The base glyph alphabet

The study fixes a shared 165-token SMILES base and applies chemistry-aware, deterministic training procedures while testing tokenizer sensitivity and learnability. BPE grows vocabularies bottom-up, whereas Unigram-LM seeds and prunes top-down, preserving structural separation across tested conditions and scales.

  • A.2 Tokenizer training and determinism: Chemistry-aware boundary handling is shared across arms, replacing SentencePiece’s text-oriented splitting with Smirk’s Layer-A/B pre-tokenization.The boundary policy is applied inside shared alphabet-construction and pre-tokenization routines rather than separately by algorithm.
  • A.2 Tokenizer training and determinism: BPE grows its vocabulary bottom-up, whereas Unigram-LM seeds a large substring pool and prunes it top-down through EM re-estimation.Unigram-LM seeds pieces by frequency × length, assigns probabilities, iterates EM, prunes low-use pieces, and finalizes at the target size.
  • A.2 Tokenizer training and determinism: Every tokenizer is trained twice and its artifacts must be byte-identical; training is single-threaded to ensure byte-reproducibility.Three design choices depart from SentencePiece defaults: character coverage 1.0, single-threaded training, and chemistry-aware pre-tokenization.
  • A.3 Sensitivity analysis and structural probes: Sensitivity sweeps vary each training hyperparameter one factor at a time, with geometric ladders for unbounded knobs and within-range grids for bounded knobs.Four pairwise interactions additionally probe covariation between training parameters and corpus or vocabulary conditions.
  • A.4 Learnability confirmation: Both vocabulary-overlap and relative-fertility contrasts remain separated across every sensitivity rung, while the seed pool and BPE merge frequency are inert.Pairwise interaction cells remain more than 70% disjoint, with overlap exceeding the headline range only at maximum piece length 4.
  • A.4 Learnability confirmation: At V =2048, Unigram-LM crosses into the undertrained tail on PubChem with clearance 0.57, below the 0.95 bar.The learnable-regime audit measures the fraction of learned pieces firing at least n times and reports F95%,100 when c100 ≥0.95.
  • B Vocabulary inventory: The vocabulary inventory separates the fixed shared base from each arm’s learned multi-glyph pieces, including a complete PubChem V =256 snapshot and arm-exclusive sets.The shared base’s growth with V and mirror-image arm-exclusive vocabularies provide complementary views of the inventory.
  • B.1 The base glyph alphabet: Both arms share the same 165-token OpenSMILES base, installed as length-1 pieces, so only selected multi-glyph pieces differ.The base contains 158 chemistry-grammatical glyphs and 7 special tokens, and conformant strings decompose without [UNK].

B.2 Learned multi-glyph pieces · V BPE UL BPE UL BPE UL · C Structural character of the learned pieces

BPE and Unigram-LM select largely different learned multi-glyph pieces, with only a small, growing shared core that remains outweighed by arm-exclusive vocabularies across vocabulary sizes and boundary policies. The analyses characterize this separation as a signature of greedy versus probabilistic tokenization and examine the pieces’ structural and chemical properties.

  • B.2 Learned multi-glyph pieces: At V=256, only 10 of 186 NMB pieces are shared (J=0.054), compared with 13 of 183 under MB (J=0.071).The shared pieces are dominated by alkyl chains, with a few short ether and amine linkers.
  • B.2 Learned multi-glyph pieces: 124 of 186 NMB pieces recur under MB, showing that learned-piece selection is largely boundary-robust.Only two recurring pieces change cross-algorithm membership between policies.
  • B.2 Learned multi-glyph pieces: The shared core is strictly nested across V=256, 512, 1024, and 2048, growing from 10 to 311 pieces under NMB and 13 to 345 under MB.Overlap rises from J=0.054 to 0.090 under NMB and from J=0.071 to 0.100 under MB.
  • B.2 Learned multi-glyph pieces: At V=2048 under NMB, BPE and Unigram-LM select 1,581 and 1,578 exclusive pieces, respectively, dwarfing the shared core by roughly 5×.The near-equal exclusive counts and their scale relative to the shared core show that near-disjointness persists beyond small vocabularies.
  • B.2 Learned multi-glyph pieces: Unigram-LM exclusive pieces never exceed 16 glyphs, whereas BPE imposes no length cap and therefore produces longer exclusive pieces.This contrast is reported in the arm-exclusive glyph-length statistics.
  • C Structural character of the learned pieces: The structural analyses treat the learned pieces as one signature of a greedy-versus-probabilistic mechanism.They examine atomic substructures, piece length, glyph co-occurrence, decomposition into in-vocabulary parts, and whether functional bonds remain within one token.

C.1 Substructure classes of the learned pieces … C.4 Compositional closure of the learned vocabularies

Across the learned vocabularies, BPE and Unigram-LM favor different chemical substructures, piece-length profiles, and internal compositional organization. BPE is compositionally closed by construction, whereas Unigram-LM retains longer, more heterogeneous pieces that are less decomposable within its vocabulary.

  • C.1 Substructure classes of the learned pieces: BPE specializes in aromatic-ring fragments, while Unigram-LM favors aliphatic heteroatom chains and, on natural products, long unsaturated motifs.The shared core is heteroatom-led but also contains saturated alkyl and COCONUT aromatic motifs absorbed early by both algorithms.
  • C.2 Piece-length distributions: Unigram-LM’s 16-glyph piece-length wall is a max_piece_length hyperparameter, while BPE is uncapped and has corpus-dependent maxima.Sweeping the cap over {4, 8, 16, 32, 64, 128} shifts Unigram-LM’s tail but leaves its mode short.
  • C.2 Piece-length distributions: Unigram-LM’s longer vocabulary pieces coexist with higher fertility because it samples likely segmentations rather than greedily applying its longest pieces.Thus, longer learned pieces do not coarsen its segmentations; more, shorter tokens remain in use.
  • C.3 Glyph co-occurrence within learned pieces: BPE and Unigram-LM are compared through adjacent base-glyph pair counts within learned multi-glyph pieces under NMB and MB boundary policies.The comparison covers PubChem at V =2048 and groups core backbone glyphs by OpenSMILES role.
  • C.4 Compositional closure of the learned vocabularies: BPE’s realized vocabulary is compositionally closed, whereas Unigram-LM is not constrained to retain in-vocabulary building blocks when pruning its seed pool.The analysis measures both arms from the realized vocabulary rather than merely restating BPE’s training rule.
  • C.4 Compositional closure of the learned vocabularies: 0.36–0.64 is Unigram-LM’s binary-split closure on diverse and natural-products corpora, falling to 0.12 on ZINC-22 at V =256.Roughly half of Unigram-LM’s pieces therefore lack an in-vocabulary binary decomposition, reaching nearly nine in ten on the narrowest alphabet.
  • C.4 Compositional closure of the learned vocabularies: 0.55–0.77 versus ≤0.05 is the full-substring closure contrast between BPE and Unigram-LM across the matched conditions.BPE’s own full-substring closure remains below 1, showing that the measurement captures more than its guaranteed single split.

C.5 Chemical functional-bond locality · D Corpus-specific contrasts

BPE preserves chemically salient multiply bonded heteroatom units, whereas Unigram-LM usually splits them, producing a near-total locality gap across all matched conditions. The corpus-specific analysis extends this membership comparison from diverse PubChem chemistry to drug-like ZINC-22 and natural-products COCONUT.

  • C.5 Chemical functional-bond locality: Functional-bond locality measures whether realized tokens preserve chemically salient multiply bonded heteroatom units, the chemical analogue of morpheme-respecting subword boundaries.The analysis focuses on multiply bonded heteroatoms, including carbonyls, nitriles, imines, and related groups.
  • C.5 Chemical functional-bond locality: 0.95–1.00 of functional bonds remain inside single BPE tokens, versus at most 0.03 for Unigram-LM, yielding gaps of 0.93–0.99 across all 22 conditions.The comparison spans every corpus, vocabulary size, and boundary policy.
  • C.5 Chemical functional-bond locality: C=O locality for Unigram-LM is 0.000 in every condition, while BPE binds the carbonyl =O in essentially every occurrence.The C=O result remains 0.000 to three places across all 22 conditions and held-out splits of up to 106 molecules.
  • C.5 Chemical functional-bond locality: Up to 0.26 nitrile locality is the only functional-bond class Unigram-LM retains with any regularity.This concerns terminal C#N, which is also one of Unigram-LM’s own short pieces.
  • C.5 Chemical functional-bond locality: BPE’s frequency-greedy merging crystallizes short motifs such as =O and C#N, whereas Unigram-LM’s likelihood pruning favors long homo-atomic and unsaturated runs.The resulting segmentation often leaves heteroatoms split from their multiply bonded partners.
  • D Corpus-specific contrasts: The corpus-specific contrast examines learned-piece membership in PubChem’s diverse alphabet, ZINC-22’s narrow drug-like alphabet, and the COCONUT natural-products corpus.Appendix D.3 additionally tests whether cross-algorithm agreement itself depends on corpus typology.

D.1 Narrow-alphabet contrast: ZINC-22 · D.2 Natural-products contrast: COCONUT · D.3 Universality of the agreement across corpora

Across corpora, BPE and Unigram-LM remain structurally distinct, but corpus alphabet and motif distribution shape how strongly their vocabularies overlap. ZINC-22 accentuates divergence through vocabulary saturation, COCONUT reverses the overlap trend, and the pieces shared across corpora are few and chemically narrow.

  • D.1 Narrow-alphabet contrast: ZINC-22: J=0.010 at V =256 (NMB), with only two shared multi-glyph pieces, makes ZINC-22’s small-vocabulary divergence sharper than PubChem’s 0.054.ZINC-22’s narrower alphabet limits the high-likelihood pieces available to Unigram-LM, causing its pruning to saturate before reaching the target vocabulary.
  • D.1 Narrow-alphabet contrast: ZINC-22: ≈0.36 is the BPE | ceiling on ZINC-22’s attainable overlap, while the observed value is about a quarter of that ceiling.The result therefore reflects genuine algorithmic disagreement, not only the combinatorial limit imposed by the narrow alphabet.
  • D.1 Narrow-alphabet contrast: ZINC-22: ≈0.31 caps REAL-Space’s attainable overlap, yet its observed 0.105 sits at about a third of that ceiling.REAL-Space behaves like ZINC-22: Unigram-LM diverges both by selecting different pieces and by running out of pieces to select.
  • D.2 Natural-products contrast: COCONUT: J=0.161 at V =256 (MB) gives COCONUT the highest small-V overlap, about twice the highest small-V overlap on any other corpus.This natural-products corpus completes the typology and reverses the cross-V pattern seen elsewhere.
  • D.2 Natural-products contrast: COCONUT: 0.119 at V =1024 is COCONUT’s overlap, which falls as V grows, opposite to the rising overlap on PubChem and ZINC-22.At V =256, the arms share 24–27 pieces because frequent long alkyl chains and aromatic rings are absorbed immediately; the shared core grows 24 →65 →139 (NMB).
  • D.3 Universality of the agreement across corpora: 23, 31, and 15 pieces are shared between the PubChem–ZINC-22, PubChem–COCONUT, and ZINC-22–COCONUT cores at V =1024 (NMB), respectively.The three shared cores have sizes 128, 99, and 139, so agreement on pieces remains weak across corpora.
  • D.3 Universality of the agreement across corpora: Just 9 pieces belong to all three V =1024 (NMB) cores.The universal core consists of chemically unremarkable long alkyl chains such as CCCCCCCCC, CCCCCCCCCC, and CCCCCCCCCCC.

E Robustness to non-canonical SMILES · F Per-condition measurement tables · F.1 Overlap, vocabulary, and summary scalars

Non-canonical rewrites preserve the central granularity gap while exposing a robustness advantage for Unigram-LM: it remains finer and more write-stable than BPE across all matched conditions. The accompanying per-condition tables document these robustness measurements and define structural vocabulary-overlap variants with frequency-weighted comparisons.

  • E Robustness to non-canonical SMILES: 10,000 molecules: the non-canonical robustness analysis uses a seeded subsample from each held-out split and evaluates segmentation across each molecule’s rewrite orbit.The analysis is model-free and compares the trained tokenizer families directly.
  • E Robustness to non-canonical SMILES: 23–35%: Unigram-LM remains finer than BPE on randomized SMILES in all 22 matched conditions.The relative gap is marginally smaller than on canonical strings in 21 of 22 cells, so canonical measurements slightly overestimate the deployed gap.
  • E Robustness to non-canonical SMILES: 0.11–0.22 versus 0.24–0.38: Unigram-LM has lower randomization bag-instability than BPE in all 22 matched conditions.BPE’s fixed merge sequence is sensitive to reordered glyphs, whereas Unigram-LM independently scores pieces and Viterbi-decodes them.
  • E Robustness to non-canonical SMILES: 0.54–0.83: explicit hydrogens produce catastrophic bag-instability for both arms and inflate token counts several-fold.Ring-digit relabeling leaves token counts exactly invariant, while Kekulé changes pieces substantially despite barely changing token counts.
  • E Robustness to non-canonical SMILES: 0.09–0.15 versus 0.19–0.24: OpenBabel canonicalization moves Unigram-LM’s token bags less than BPE’s in all 22 matched conditions.This cross-toolkit result shows the stability gap is not specific to RDKit randomization or rewrite rules.
  • F Per-condition measurement tables: The exhaustive per-condition measurements are collected in summary tables followed by four families of measurement-specific detail.Analyses tied to compositional closure, functional-bond locality, and non-canonical rewrites remain in their respective appendices.
  • F.1 Overlap, vocabulary, and summary scalars: Table A12 reports four vocabulary-overlap weightings, Table A13 realized per-arm multi-glyph vocabularies, Table A14 dead-zone surplus with per-arm F95%,100 clearance, and Table A15 robustness probes.These tables expand the direct-contrast columns of the body’s Table 2 and use SentencePiece default L=16.
  • F.1 Overlap, vocabulary, and summary scalars: Jstruct and Jw,struct recompute unweighted and frequency-weighted Jaccard overlap after removing bracket-internal pieces.Each arm’s weights are renormalized over the structural held-out mass and carry its own bootstrap confidence interval.

F.2 Fertility and compression detail … F.5 Mechanism-diagnostic detail

Appendix detail shows that BPE and Unigram-LM differ in fertility, boundary nesting, token-distribution uniformity, and mechanism diagnostics across matched conditions. BPE is coarser, more uniform, and absorbs pretokens more often, while Unigram-LM splits more and retains fewer whole pretokens.

  • F.2 Fertility and compression detail: ∼12–19 more tokens per molecule are emitted by Unigram-LM than BPE, despite BPE molecules averaging only ∼30–55 tokens.Per-arm confidence intervals span at most ∼0.2 token on diverse, drug-like, and combinatorial corpora, and ∼1 token on smaller corpora.
  • F.2 Fertility and compression detail: Unigram-LM segments to more tokens at a lower compression ratio than BPE in every matched condition.The compression ratio is glyphs per token, with higher values indicating coarser segmentation.
  • F.3 Boundary nestedness detail: Below 0.7% of positions are genuine crossing conflicts; the arms agree on most cuts and nest rather than cross.The positional comparison uses the same glyph stream, and the fraction of molecules whose BPE parse strictly coarsens Unigram-LM’s is reported per matched condition.
  • F.4 Token-distribution detail: 22 conditions show BPE as the more uniform arm: lower D, higher η, and higher R than Unigram-LM.Both arms remain far from uniform, with absolute D ranging from 0.76–0.97; confidence intervals are ≤0.001 on diverse, drug-like, and combinatorial corpora.
  • F.5 Mechanism-diagnostic detail: 0.89–1.00 of pretokens are absorbed whole by BPE, versus ∼0.72 (0.71–0.74) by Unigram-LM.Permeable (MB) boundaries lift BPE’s absorption further toward 1.0 because merges may cross the bracket.
  • F.5 Mechanism-diagnostic detail: n≤100 is sufficient for BPE to clear the rare-token learnability bar in every condition except COCONUT at V =1024.The clearance sweep evaluates cn at n ∈{50, 100, 200}; the stricter n=200 threshold removes a few narrow-alphabet cases.
Loading 2607.05691v1…