The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages

• A fertility figure shows isiXhosa gradually plateauing, whereas English and Setswana converge early.Source: Figure 1 · p. 1
• The study compares Setswana, English, and isiXhosa across contrasting morpho-orthographic structures.Setswana is disjunctive, isiXhosa is conjunctive, and English is positioned between them.Source: §1 · pp. 1–2, Table 1 · p. 2
• T-SSLM jointly learns subword segmentation and language modelling for pretraining, finetuning, and text generation.It extends SSLM with a Transformer-based architecture.Source: §1 · p. 2
• Across morphology, productivity, and fertility, isiXhosa shows the greatest instability while Setswana converges earliest.The results identify distinct phases of subword learning and show that training stages and languages require different subword units.Source: §1 · p. 2
• IsiXhosa data-to-text BLEU improves by 6.25 over tokenisation-based language models, alongside reduced text degeneration.The result supports learnable subwords as a promising approach for low-resource text generation.Source: §1 · p. 2

2.1 Learning dynamics of language models

• Most studies of syntactic, semantic, and representation change use fixed tokenisation, leaving subword learning trajectories unexamined.BPE token properties may change across merges, but BPE itself remains fixed after preprocessing.Source: §2.1 · p. 2

2.2 Unifying training and tokenisation

• SSLM marginalises over candidate tokenisations during training, extending joint training and segmentation learning to low-resource agglutinative languages.Earlier SSLM work used an LSTM and demonstrated potential over tokenisation-based language models through perplexity evaluation.Source: §2.2 · p. 2
• Prior SSLM research did not conduct finetuning or downstream evaluation, motivating the present study’s broader training setup.Source: §2.2 · p. 2

3 Transformer SSLM

• The architecture is presented as a tool for empirical analysis of subword dynamics rather than as the paper’s primary architectural contribution.The implementation is publicly available.Source: §3 · p. 3

3.1 SSLM background

• SSLM assigns document probability by marginalising over all possible subword segmentations rather than selecting one fixed tokenisation.Dynamic programming computes this marginal efficiently despite the exponential number of possible segmentations.Source: §3.1 · p. 3

3.2 T-SSLM: adapting SSLM for pretraining and finetuning

• T-SSLM replaces the original LSTM parameterisation with a character-level Transformer to support modern pretraining, finetuning, and generation.Its subword probabilities are conditioned on Transformer representations of preceding untokenised characters.Source: §3.2 · p. 3
• T-SSLM mixes character-sequence and lexicon distributions, allowing both its segmentation algorithm and subword vocabulary to adapt during training.Source: §3.2 · p. 3
• Conditional-likelihood dynamic programming enables prompt-based finetuning, while custom decoding enables open-ended generation.Standard beam search is incompatible with the model’s mixture distribution.Source: §3.2 · pp. 3–4

3.3 Extracting learned subwords for analysis

• The model’s preferred segmentation is extracted at checkpoints using Viterbi decoding to identify the probability-maximising subwords under current parameters.After finetuning, segmentation reflects task-specific output-completion likelihood rather than full-document likelihood.Source: §3.3 · p. 4

4.1 Pretraining

• Pretraining is computationally constrained by T-SSLM’s time complexity, so the study prioritises high-quality corpora over scaling data to state-of-the-art sizes.The selected corpora include 13 million isiXhosa words, 4 million Setswana words, and 10 million English words.Source: §4.1 · p. 4
• The experiments span Setswana, English, and isiXhosa, with checkpoints stored throughout pretraining to track how learned subwords change over time.Pretraining uses clean corpora, while finetuning focuses on isiXhosa generation tasks because its morphology creates a challenging adaptation setting.Source: §4.1 · p. 4, §4 · p. 4, §4.2 · p. 4

4.2 Finetuning

• The study finetunes all models on isiXhosa data-to-text and headline generation, with headline generation described as harder because its dataset is smaller.Cross-lingual finetuning tests how Setswana and English models adapt to isiXhosa’s morphological structure.Source: §4.2 · p. 4

4.3 Baselines

• Fixed-tokenisation baselines comprise BPE, ULM, BPE-dropout, character-level segmentation, and byte-based modelling, using the same datasets for fair downstream comparison.These baselines are excluded from learning-dynamics analyses because their tokenisation does not evolve during training.Source: §4.3 · p. 5

4.4 Subword analysis

• At each T-SSLM checkpoint, Viterbi decoding extracts the probability-maximising segmentation for analysis from morphological, productivity, and fertility perspectives.The models jointly learn segmentation and language modelling, while baseline tokenisers remain fixed.Source: §4.4 · p. 5, §4.3 · p. 5

4.4.1 Morphological alignment

• Morphological alignment compares learned subword boundaries with gold morphological segmentations using precision, recall, and F1 on annotated Setswana, isiXhosa, and English words.The models receive no morphological annotations during training.Source: §4.4.1 · p. 5

4.4.2 Productivity and Idiosyncrasy

• Productivity measures how many word types contain a subword, whereas idiosyncrasy measures the average token frequency of those word types.Both metrics are computed from validation-corpus segmentations and word frequencies.Source: §4.4.2 · pp. 5–6

4.4.3 Fertility

• Fertility is the average number of subwords per word, serving as a lexical-coverage measure in which higher fertility indicates finer-grained segmentation.A fertility of 1 means every word is represented directly in the vocabulary.Source: §4.4.3 · p. 6

5 Subword Learning Dynamics

• T-SSLM identifies four stages of subword learning, with isiXhosa and English remaining more variable than Setswana throughout pretraining.Setswana stabilises earlier, whereas isiXhosa exhibits the greatest instability across morphological alignment, productivity, and fertility.Source: §5 · p. 6, Stage 3: Stabilised learning dynamics · p. 7

Stage 1: Rapid evolution of subwords

• During the first 20% of pretraining, subword boundaries change dramatically as models move from random initialisations toward morpheme-aligned units.Productivity also falls as highly productive, frequent subwords are identified first and progressively less common units are incorporated.Source: Stage 1: Rapid evolution of subwords · p. 6

Stage 2: Inflection point for English & isiXhosa

• Between 20% and 40% of pretraining, Setswana subwords stabilise, whereas English and isiXhosa reverse initially decreasing productivity and idiosyncrasy, coinciding with increased fertility indicating finer-grained segmentation.The initially expanded vocabulary shifts toward subwords that improve LM generalisation by occurring in many word types and/or high-frequency words.Source: Stage 2: Inflection point for English… · pp. 6–7
• At the isiXhosa inflection point, morphological-boundary recall rises at the cost of precision, producing finer but sometimes over-segmented units.The segmentations of “umntu” and “zoncedo” illustrate increased boundary coverage alongside morphologically unsound splits.Source: Stage 2: Inflection point for English… · p. 7

Stage 3: Stabilised learning dynamics

• During mid-to-late pretraining, fertility settles at 1.66 for Setswana, 1.8 for English, and 3.05 for isiXhosa, while T-SSLM learns finer-grained units than BPE.Setswana’s disjunctive orthography limits aggressive segmentation, whereas isiXhosa continues changing as better units emerge.Source: Stage 3: Stabilised learning dynamics · p. 7

Stage 4: Task-oriented realignment

• Finetuning increases fertility across languages and shifts isiXhosa units toward frequent task-specific patterns, including character-level segmentation of named entities.Cross-lingual finetuning also improves isiXhosa morphological alignment while reducing alignment in the pretraining language.Source: Stage 4: Task-oriented realignment · pp. 7–8, Table 4 · p. 7

6 Downstream Text Generation

• Finetuning produces task-specific subwords, motivating evaluation of adaptive segmentation against fixed BPE and ULM tokenisation.Fixed tokenisers lack the flexibility to change boundaries during downstream training.Source: §6 · p. 8
• T-SSLM outperforms tokenisation-based models across BLEU and chrF on two isiXhosa generation tasks, with substantial BLEU gains and less text degeneration.The comparison uses same-size models trained on the same datasets to isolate task-adaptable subwords from model scale.Source: §6 · p. 8
• Cross-lingual T-SSLMs pretrained on Setswana or English consistently outperform baselines across BLEU and chrF after isiXhosa finetuning.They achieve competitive text-degeneration scores while adapting subword boundaries to isiXhosa morphology.Source: §6 · p. 8

7 Conclusion

• Dynamic subword learning reveals contrasting trajectories across Setswana, English, and isiXhosa during language-model training.The analysis studies segmentation jointly with other learned aspects of language rather than holding tokenisation fixed.Source: §7 · p. 8
• Task-adaptable segmentation provides insight into the role of subwords across training stages and shows distinct needs in typologically diverse, low-resource languages.The conclusion highlights low-resource and morphologically complex settings as especially relevant.Source: §7 · p. 8
• Dynamic subword learning may be computationally prohibitive for larger-scale pretraining, despite its analytical and modeling potential.The conclusion presents scalability cost as a qualification on the approach.Source: §7 · p. 8

8.1 Computational complexity

• T-SSLM training is about ×10 slower than tokenisation-based language models because it marginalises over multiple candidate subword sequences.This cost prevented pretraining on larger datasets, leaving scaling to larger corpora and higher-resourced languages unresolved.Source: §8.1 · p. 9

8.2 Linguistic scope

• Findings are limited to Setswana, English, and isiXhosa, so generalisation to other language families is not guaranteed.These languages were selected to contrast morpho-orthographic structures for an initial exploration of subword dynamics.Source: §8.2 · p. 9
• Downstream experiments cover only isiXhosa data-to-text and headline generation, so similar gains on other generation tasks are not guaranteed.The available isiXhosa datasets had sufficient training instances for these evaluations, and the paper emphasizes learning dynamics over universal task performance.Source: §8.2 · p. 9

A.1 Mixture model

• The mixture model combines a subword lexicon for common n-grams with a character decoder for arbitrary sequences, including unseen words and names.A learned mixture coefficient determines when to rely on lexicon versus character-level predictions.Source: §A.1 · p. 12

A.2 Finetuning algorithm

• For prompt-based generation, finetuning maximises p(O|C) rather than the likelihood of the complete prompt, adapting boundaries to the completion task.Here C is the input context and O is the expected output.Source: §A.2 · p. 12

A.3 Decoding

• All languages show increasing fertility across training, with especially large distributional shifts for morphologically complex isiXhosa during finetuning.Fertility measures subwords per segmented word in the four training stages.Source: Figure 4 · p. 13
• During isiXhosa finetuning, subword boundaries learned from Setswana or English shift toward isiXhosa morphological boundaries.The figure compares morpheme–subword boundary overlap for models pretrained on the two source languages.Source: Figure 5 · p. 13
• Dynamic decoding generates text character by character while predicting subword boundaries from mixture-model probabilities conditioned on the prompt context.The adapted decoder uses beam size 5 for decoder-only generation.Source: §A.3 · p. 13

B.1 Pretraining

• SSLMs use a 6-layer decoder-only architecture with 8 attention heads and language-specific subword lexicons.The isiXhosa lexicon contains the 10k most frequent character n-grams, with a maximum subword segment length of 5 characters. English and Setswana use a 5k-subword vocabulary.Source: §B.1 · p. 13
• Pretraining continues until validation loss stops improving, reaching 40 epochs for English and isiXhosa and 100 epochs for Setswana.Source: §B.1 · p. 13

B.2 Finetuning

• The T2X prompt template places input text before “# # =” and output text after it, with testing using the special tokens to prompt generation.Source: Table 6 · p. 14
• During completion-only finetuning, models maximise likelihood after “=” and use 20 epochs with validation-based checkpoint selection.Source: §B.2 · p. 14

Baseline pretraining

• Baseline LMs match SSLM size and vocabulary scales, using 6 layers, 8 attention heads, 512-dimensional embeddings, and 5k or 10k subwords by language.Source: C Baseline Hyperparameters · p. 14
• Baseline pretraining uses batch size 64, learning rate 5e-5, no warmup, a linear scheduler, and dropout 0.1.Source: C Baseline Hyperparameters · p. 14

Baseline finetuning

• Baseline finetuning uses batch size 4, learning rate 5e-5, no warmup, a linear scheduler, and validation-selected checkpoints over 20 epochs.Source: C Baseline Hyperparameters · p. 14

Generation comparison

• Tokenisation-based baselines often generate incoherent, repetitive text, whereas SSLM avoids text degeneration in all shown instances.Source: Table 7 · p. 15