PaperlayerView exact revision on arXiv

Source-linked AI summary

Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text

Lifan Deng, Yongwei Zhang, Sen Sun, Bojun Sun, Jingsong Yu

arXiv:2608.18437v1cs.CL

TL;DR

Tangut word segmentation lacks explicit boundary markers and is constrained by costly expert annotation. This study combines expert labels, traditional lexicons, corpus statistics, and unlabeled-text pretraining in a unified framework, with TangutEncoder achieving the highest mean F1 of 0.911 and OOV recall under within-source evaluation.

  • Problem

    Tangut’s unmarked word boundaries and high annotation barrier leave automatic segmentation underexplored despite its importance for downstream language analysis.

  • Method

    The framework combines expert BIES supervision, reliability-calibrated lexicon lattices, explicit distributional statistics, and lightweight character-level MLM pretraining within a BIES–CRF framework.

  • Results

    TangutEncoder obtains the highest mean F1 (0.911) and OOV recall under within-source line-level evaluation, while CRF remains stronger on religious texts.

  • Takeaways & Limitations

    The study establishes initial within-source baselines for automatic Tangut word segmentation across Buddhist and secular texts.

  • Takeaways & Limitations

    The evaluation does not establish transfer to unseen documents because the corpus is small, genre-imbalanced, and limited to two works.

Abstract

from arXiv · show

Tangut is an extinct language whose script does not explicitly mark word boundaries. We present the first systematic study of Tangut word segmentation using 2,750 expert-annotated segments(31,893 tokens), traditional lexicons, and unlabeled text. Our framework combines a reliability-calibrated lexicon-lattice representation, explicit distributional statistics, and a lightweight character encoder pretrained with MLM. Segment-level five-fold cross-validation shows that lexical and statistical features raise CRF F1 to approximately 0.91. The full TangutEncoder reaches the highest mean F1 (0.911) and improves recall beyond the labeled training vocabulary. These results demonstrate generalization beyond the limited supervised vocabulary across thematically diverse held-out passages, while document-level transfer remains to be evaluated.

1 Introduction

This paper presents the first systematic study of Tangut word segmentation, addressing unmarked word boundaries and scarce expert annotation with lexicon integration and unlabeled-text learning. Under segment-level five-fold cross-validation, both complete systems exceed 0.90 F1, with TangutEncoder achieving the highest mean score and OOV recall.

  • Motivation: The study addresses Tangut’s unmarked word boundaries, which make segmentation necessary for lexical retrieval, frequency analysis, POS tagging, alignment, and translation.OCR and transcription therefore produce unsegmented character sequences.
  • Data and task: The expert-annotated corpus contains 2,750 textual segments and 31,893 word tokens from Buddhist scripture and the secular encyclopaedic work Leilin, despite the high annotation barrier.Boundary annotation requires specialists to jointly consider context, phonological reconstruction, translations, and traditional dictionaries.
  • Approach: The framework integrates BIES supervision, a reliability-calibrated lexicon lattice preserving overlapping dictionary candidates, corpus statistics, static character embeddings, and MLM pretraining.The study systematically compares these unlabeled-text signals and develops the lightweight TangutEncoder.
  • Results: Both complete systems exceed 0.90 F1 under segment-level five-fold cross-validation, with TangutEncoder attaining the highest mean score and OOV recall.TangutEncoder learns contextual character representations without assuming word boundaries in advance.
  • Contributions: The paper establishes within-source evaluation baselines for Buddhist and secular texts and provides a word-level foundation for retrieval, POS tagging, alignment, machine translation, and related digital research.It presents the first systematic study of automatic Tangut word segmentation to the authors’ knowledge.

2 Related Work

Prior segmentation research uses sequence prediction and lexicon-enhanced methods across scripts without explicit delimiters, while low-resource and historical-language studies address severe data constraints. Tangut information processing has largely emphasized digital infrastructure, leaving automatic word-boundary identification and contextual representation underexplored.

  • Word Segmentation without Explicit Delimiters: Sequence prediction is a common formulation for scripts without consistent word delimiters, including Chinese, Tibetan, and Myanmar.Prior work includes BIES-style tagging and CRFs for Chinese, syllable-level CRFs for Tibetan, and longest matching for Myanmar.
  • Lexicon-Enhanced Segmentation: Lexicons have been integrated through CRF features, partial annotations, manual segmentation, neural lexical representations, candidate attention, and auxiliary BIES prediction.These approaches incorporate lexical information across supervised, domain-adaptation, documentation, and unlabeled-text settings (Peng et al., 2004; Liu et al., 2014; Okabe et al., 2022; Zhang et al., 2018; Higashiyama et al., 2019, 2020).
  • Low-Resource and Historical-Language Segmentation: Low-resource and historical-language segmentation studies confront limited annotation, text, tools, pretrained models, and support, with dictionary-based weak supervision offering partial compensation.These constraints are particularly pronounced for Sino-Tibetan languages (Nigatu et al., 2024), while historical-language work compares rule-based, lexical, statistical, and learned methods for Akkadian (Homburg and ...).
  • Tangut information processing: Tangut research has focused mainly on digital infrastructure, recognition, databases, manuscript analysis, and translation, while automatic word-boundary identification and contextual representation remain largely unexplored.This study addresses that gap using expert annotation, traditional lexicons, and unlabeled text.

3 Task Definition and Data

Tangut word segmentation is formulated as character-level BIES labeling with exact span-based evaluation. The dataset combines expert-reviewed Buddhist and secular texts with a structured lexicon and separate unlabeled material, whose coverage is broad but uncertain.

  • Task Definition: Tangut segmentation assigns each character a BIES label and evaluates predictions by exact matching of annotated word boundaries.B, I, and E mark multi-character words’ beginning, interior, and end, while S marks single-character words.
  • Data Construction: Expert annotation was independently produced and jointly reviewed by Tangut specialists using context, reconstructed readings, translations, and traditional lexicons across Buddhist and secular texts.The corpus draws Buddhist scripture from Mahāratnakūṭa-sūtra and secular writing from the encyclopedic Leilin.
  • Lexicon: The processed lexicon yields 19,290 multi-character candidates, including 17,082 two-character words that comprise 88.6% of candidates.Single-character entries are removed, duplicate forms are collapsed, and entries retain word forms, pronunciations, definitions, and metadata.
  • Lexicon: 95.9% of corpus word tokens and 83.1% of word types are recorded in the lexicon, but only 14.2% of lexicon entries exactly match annotated words.Another 23.3% of entries occur as corpus substrings, while 76.7% never occur, motivating uncertainty-aware lexical treatment.
  • Unlabeled Data: Unlabeled data contain approximately 318,000 Tangut characters across 663 records and 46 title-level units, with no duplicated annotated segment of three or more characters.The annotated material uses fascicle 68 of Mahāratnakūṭa-sūtra, which is absent from the unlabeled collection.

4 Layered Framework and Setup

The framework combines supervised BIES–CRF segmentation with overlapping lexicon-lattice features and three forms of unlabeled-text knowledge: distributional statistics, static character embeddings, and contextual pretraining. It calibrates dictionary evidence for reliability, prevents leakage through out-of-fold estimation, and evaluates generalization beyond the labeled training vocabulary using fold-specific OOV recall.

  • Layered framework: Tangut segmentation combines supervised BIES tagging with CRF decoding, lexical candidate features, and distributional knowledge from unlabeled text.The distributional layer uses corpus statistics, static character embeddings, or contextual pretraining, while dictionary matching is the main exception to character-level BIES–CRF modeling.
  • Lexicon representation: The 20-dimensional lexicon representation preserves overlapping dictionary spans and combines 11 position-and-length indicators with reliability, unseen-entry priors, and metadata features.Reliability is aggregated by beginning, internal, and ending positions; metadata records semantic glosses, phonetic entries, and book-title labels without treating them as deterministic boundaries.
  • Leakage control: Reliability features are generated out of fold so each training instance’s dictionary statistics exclude its own gold boundary information.Within each outer training portion, reliability is estimated using an additional five-fold split; validation and test features use statistics from the complete outer training portion.
  • Unlabeled-text knowledge: Unlabeled parallel text supplies eight gap-level distributional features based on frequency, association strength, and left- and right-neighbor entropy, alongside Char2Vec and TangutEncoder.TangutEncoder is a compact character-level BERT-style encoder in which each Tangut character is treated as a single token.
  • Evaluation setup: 53.0% of labeled-training OOV tokens are covered by the external lexicon, 34.8% occur in unlabeled text, and 24.8% occur in neither resource.OOV means that a gold test word is absent from the labeled training partition of the current fold, even if it appears in the external lexicon or unlabeled corpus.

5 Results

Across five-fold cross-validation, TangutEncoder provides the strongest overall and secular-text performance, while lexical, distributional, and contextual signals contribute complementary gains. Explicit features remain especially valuable for vocabulary coverage and the smallest genre, whereas corpus and neural methods improve OOV generalization.

  • RQ1: The linear CRF is the strongest supervised baseline overall; dictionary matching recognizes IV words but generalizes poorly, while neural models improve OOV recall at a cost to overall F1.The random Transformer has the highest OOV recall but the lowest overall F1, and religious texts are harder than secular texts for all learned baselines.
  • RQ2: BIE lattice features provide the largest lexical gain, while reliability estimates improve precision and IV recognition but slightly reduce OOV recall.The lattice supplies coverage, whereas reliability and metadata control mismatches between dictionary entries and corpus boundaries.
  • RQ3: Unlabeled-text statistics improve the complete lexicon CRF, with bigram frequency contributing the largest initial gain and character association plus neighbor entropy adding complementary boundary evidence.Improvements extend to IV and OOV recognition and are particularly visible for religious texts.
  • RQ4: MLM pretraining substantially improves F1, OOV recall, and religious-text performance, unlike Char2Vec, whose slight decrease remains within cross-fold variation.The advantage comes from training the entire contextual encoder rather than only supplying static character embeddings.
  • RQ5: Adding the complete lexicon representation to TangutEncoder markedly improves IV and OOV recognition, while corpus statistics provide a smaller additional gain.MLM captures much, but not all, of the available local distributional information.
  • RQ5: TangutEncoder has the highest overall and secular-text F1 and OOV recall, while CRF retains slightly higher overall IV recall and performs better on religious texts.The systems remain complementary: contextual pretraining chiefly improves generalization beyond the labeled vocabulary, while explicit features remain stable in the smallest genre.

6 Preliminary Extension to POS Tagging

A preliminary joint segmentation-and-POS experiment replaces BIES labels with boundary–POS labels and evaluates tagging only on exactly matched predicted spans. The joint CRF reaches approximately 0.88 conditional POS accuracy, while lexical features help segmentation more consistently than POS tagging; results remain preliminary because the POS inventory is under revision.

  • Method: The experiment jointly predicts boundary–POS labels, converts the predicted sequence into word spans and POS labels, and applies the formulation to CRF and BiLSTM–CRF models with or without lexical features.Joint labels include forms such as B-NOUN and E-NOUN.
  • Results: Approximately 0.88 conditional POS accuracy is achieved by the joint CRF on words whose predicted spans exactly match gold segmentation.This evaluation isolates POS-tagging errors from boundary errors.
  • Results: Lexical features improve segmentation but contribute less stably to POS tagging, suggesting dictionary evidence is more informative for boundaries than grammatical categories.The findings are preliminary because the POS inventory remains under revision in Appendix A.

7 Conclusion and Future Work

This study presents the first systematic investigation of automatic Tangut word segmentation, combining expert annotation, traditional dictionaries, and unlabeled text in a unified BIES–CRF framework. The authors identify TangutEncoder’s strongest results and outline broader data, evaluation, resource, and downstream plans.

  • Conclusion: 0.911 mean F1: the full TangutEncoder achieves the highest mean F1 and OOV recall under within-source line-level evaluation.The framework combines lexicon-lattice features, corpus statistics, and contextual MLM pretraining for data-efficient boundary evidence.
  • Future Work: Future work will expand expert annotation across documents, genres, and historical periods while improving genre balance, document-level evaluation, POS inventories, annotation guidelines, and expert consistency checks.The plans address both broader coverage and clearer, more reliable annotation and evaluation.
  • Future Work: The authors plan to seek controlled access to more annotated data and release code, models, and preprocessing resources, then explore larger corpora for pretraining and weak supervision.Planned follow-up also includes multi-seed significance analyses and downstream evaluation in retrieval, lexical analysis, alignment, and translation.

Limitations

The study is limited by a small, genre-imbalanced corpus drawn from two works and does not establish transfer to unseen documents. Additional constraints include shared formulaic expressions, limited statistical testing, restricted corpus release, and a provisional POS inventory.

  • Limitations: The corpus is small, genre-imbalanced, and limited to two works, constraining the breadth of evaluation.Although Leilin is thematically diverse, the evaluation does not establish transfer to unseen documents.
  • Limitations: The evaluation does not establish transfer to unseen documents, despite Leilin’s thematic diversity.Fascicle 68 is absent from the unlabeled collection, and long-string overlap is minimal.
  • Limitations: Shorter formulaic expressions remain shared across Buddhist texts, limiting the strength of separation between labeled and unlabeled material.The passage notes that long-string overlap is minimal, but shorter formulaic expressions remain shared.
  • Limitations: Small model differences lack multi-seed or significance testing, while data-use restrictions prevent full corpus release and the POS inventory remains under revision.OOV refers only to absence from labeled training.

Ethical Considerations · A More Details About Our Corpus · A.1 Textual Sources and Composition

The study combines expert-annotated Tangut texts with digitized historical and unlabeled materials, while institutional restrictions prevent releasing the full annotated corpus. The annotated sources comprise religious and secular works, use layout-based segments, and motivate genre-specific evaluation because of substantial imbalance.

  • Ethical Considerations: The full annotated corpus cannot be publicly released because of institutional and data-use restrictions, but permitted releases will provide reproducibility materials including code, splits, scripts, configurations, manifests, guidelines, metadata, and trained models.The planned release also includes examples and checksums where permitted.
  • A.1 Textual Sources and Composition: The annotated corpus draws on two Tangut works: fascicle 68 of the Mahāratnakūṭa Sūtra for religious text and Leilin for secular stories translated from Chinese.Source-location identifiers preserve traceability to the editions and catalogue records used by annotators.
  • A.1 Textual Sources and Composition: Segments correspond primarily to single vertical columns with location identifiers, and boundaries were adjusted when necessary to preserve lexical continuity across columns.These are layout-based textual segments rather than sentences defined by modern punctuation.
  • A.1 Textual Sources and Composition: Approximately 8.5% of segments and 11.6% of tokens come from the religious text, with Leilin contributing the remainder and motivating genre-specific evaluation.The two annotated sources are therefore substantially imbalanced.
  • A.1 Textual Sources and Composition: The unlabeled corpus contains 663 page-image records spanning 46 title-level units from seven works and approximately 318,000 Tangut characters.It was extracted from Tangut lines in digitized four-line translation materials based on manuscripts held in Russian collections.
  • A.1 Textual Sources and Composition: Mahāratnakūṭa-sūtra is the largest unlabeled source, contributing 269,149 characters, or 84.6% of the corpus, across 38 fascicle-level titles.The supplied passage notes that these records are not all from the same source category, but its final clause is truncated.
  • A.1 Textual Sources and Composition: Only the original Tangut text from the unlabeled corpus is used to compute distributional features and train Char2Vec and TangutEncoder.This establishes the unlabeled-data role in the modeling pipeline.

A.2 Overlap and OOV-Coverage Audit … B.1 Linear CRF

The audit characterizes overlap and out-of-vocabulary coverage between annotated and unlabeled Tangut text, while the appendices document annotation labels and standardized model-training settings. The linear CRF combines local character features with lexicon-lattice and corpus-statistical features under shared optimization conditions.

  • A.2 Overlap and OOV-Coverage Audit: Five exact annotated–unlabeled matches remained after normalization, all one- or two-character fragments, with no duplicated annotated segment of three or more characters.Token-level n-gram overlap is defined as the percentage of annotated n-gram occurrences whose character string appears anywhere in the unlabeled collection.
  • A.2 Overlap and OOV-Coverage Audit: 53.0% of labeled-training OOV tokens are covered by the external lexicon, 34.8% occur as unlabeled-corpus substrings, 12.6% occur in both, and 24.8% in neither.These categories show that OOV-R measures generalization beyond supervised vocabulary rather than forms unseen by every system component.
  • A.3 Annotation Examples: The corpus format removes spaces from raw lines, marks expert word boundaries with vertical bars, and appends linguistic labels after slashes.Religious and secular examples illustrate this representation.
  • A.4 POS and Morphosyntactic Labels: The normalized expert-annotation inventory contains 36 labels spanning conventional parts of speech and finer morphosyntactic distinctions.Frequent categories include nouns, verbs, adjectives, adverbs, adpositions, conjunctions, pronouns, numerals, quantifiers, and particles or auxiliaries, alongside labels for grammatical functions, aspect, direction, person, and number.
  • A.4 POS and Morphosyntactic Labels: POS experiments are preliminary because the fine-grained inventory remains under expert revision and combines lexical classes with morphosyntactic functions.The experiments preserve the original distinctions rather than collapsing them into a newly designed tagset.
  • B Model and Training Parameters: All models use the same data partitions and BIES tag set unless otherwise stated, with base random seed 42 and deterministic CuDNN execution enabled.These settings are applied within each training environment.
  • B.1 Linear CRF: The linear CRF uses L-BFGS, while its variants share optimization parameters and differ only in included external feature groups.The local template uses the current, neighboring, and adjacent-bigram characters, character-type indicators, and a single-character word indicator; lexicon-lattice and corpus-statistical features are real-valued additions.

B.2 BiLSTM–CRF … C.1 Comparison of Character Association Measures

Across the experimental setup, the BiLSTM–CRF is treated as a supporting comparison, while Transformer variants share a controlled architecture and differ in initialization. Character-association experiments show that dPMI has a slight advantage without entropy, but entropy makes the measures nearly equivalent and provides complementary boundary evidence.

  • B.2 BiLSTM–CRF: The preliminary search found that a two-layer BiLSTM with 100-dimensional embeddings and total hidden size 64 achieved the highest observed segmentation F1, while batch size 512 outperformed 32 slightly.The search used a fixed 80/10/10 split outside outer cross-validation, so the BiLSTM remains a supporting architectural comparison rather than evidence for the main model-selection claim.
  • B.2 BiLSTM–CRF: The selected BiLSTM–CRF uses trainable 100-dimensional character embeddings, a two-layer bidirectional LSTM with 32 units per direction, and CRF decoding over four BIES scores.External features are concatenated before the BiLSTM; the principal lexicon-enhanced model uses a 20-dimensional reliability-calibrated dictionary representation with dictionary dropout of 0.2.
  • B.3 Transformer–CRF Variants: Transformer-Random, Transformer-Char2Vec, and TangutEncoder share the same Transformer–CRF architecture and differ only in parameter initialization.TangutEncoder loads all encoder parameters from masked-language-model pretraining, whereas Transformer-Char2Vec initializes its character embeddings with static Skip-gram vectors.
  • B.3 Transformer–CRF Variants: The Transformer encoder has three pre-normalized layers, hidden size 192, four attention heads, a 768-dimensional feed-forward layer, and learned positions for sequences up to 128 characters.Its output is mapped to four BIES scores and decoded with a CRF.
  • B.3 Transformer–CRF Variants: TangutEncoder uses character-level masked-language-model pretraining with approximately 15% masked positions, balanced single-character and two-to-four-character span masking, and a lexicon-and-corpus vocabulary.All Transformer variants freeze the encoder for three epochs before joint fine-tuning; random and Char2Vec models are initialization controls rather than independently optimized scratch upper bounds.
  • C.1 Comparison of Character Association Measures: The study compares discounted PMI, Dice, and t-score for adjacent character pairs, capturing respectively frequency-discounted positive association, normalized overlap, and standardized observed-versus-expected cooccurrence.Dice is bounded between zero and one, while t-score can preserve negative evidence and favors associations supported by higher absolute frequencies.
  • C.1 Comparison of Character Association Measures: After adding neighbor entropy, dPMI, Dice, and t-score converge to nearly identical performance, so the experiments establish no statistically meaningful winner.Without entropy, dPMI performs slightly better, particularly on OOV words; dPMI is retained for the main experiments because its frequency discount suits many low-count observations.
  • C.1 Comparison of Character Association Measures: Neighbor entropy consistently improves association-based segmentation by complementing cohesion estimates with contextual diversity linked to potential word boundaries.Association measures assess whether adjacent characters form cohesive units, whereas entropy reflects whether characters occur freely with many different neighbors.

C.2 Additional Model Variants · C.3 Lexicon-aware Continued Pretraining

Additional variants show that lexical coverage requires reliability-aware integration, while lexicon-aware continued pretraining adds no downstream benefit over canonical MLM. Internal-lattice and genre features provide smaller, condition-dependent gains, whereas MLM pretraining remains complementary to lexicon representations.

  • C.2 Additional Model Variants: Training-vocabulary matching has strong IV recall but weak beyond-vocabulary recall, whereas the external dictionary raises OOV recall at the cost of precision and IV recall.The disagreement with corpus boundaries motivates aggregated lattice representations and reliability calibration rather than relying on broader lexical coverage alone.
  • C.2 Additional Model Variants: Adding candidate counts to the earlier BIE lattice yields F1 0.898 versus 0.897 for BIE alone, while maximum-candidate-length features yield F1 0.897.These experiments compare alternative CRF lattice features under the earlier lattice design.
  • C.2 Additional Model Variants: The internal lattice encodes corpus-derived multi-character words with an 11-dimensional BIE-by-length scheme and uses inner out-of-fold estimation to avoid exposing gold boundaries.Its entries lack reliability scores because they are derived directly from the annotated corpus; external and internal candidate vectors are averaged at each character position.
  • C.2 Additional Model Variants: The BIE lattice provides the largest BiLSTM–CRF lexical gain, followed by reliability estimation and unseen-entry priors, while internal-lattice and domain features add smaller complementary improvements.The internal lattice recovers corpus-specific words missing from the traditional dictionary, and domain features model genre preferences.
  • C.2 Additional Model Variants: Internal-lattice and domain features also improve religious-text performance, including OOV recall, although gains are more stable on secular texts that dominate training data.Domain features require document genre at inference time and therefore remain an auxiliary variant rather than part of the genre-independent model.
  • C.2 Additional Model Variants: Lexicon injection improves randomly initialized Transformer performance, but remains below MLM-pretrained TangutEncoder; combining MLM with lexicon features is stronger than either source alone.This shows dictionary knowledge complements, rather than substitutes for, contextual pretraining.
  • C.3 Lexicon-aware Continued Pretraining: Lexicon-aware continued pretraining uses dictionary spans from structured lexicons and revised Tongyin as positives, contrasts them with five sampled non-dictionary negatives, and combines ranking loss with MLM.Training uses λword = 0.3 for at most 3,000 steps, with a 500-step warmup; the span head is discarded before segmentation transfer.
  • C.3 Lexicon-aware Continued Pretraining: The auxiliary lexicon-ranking objective provides no improvement over canonical MLM in overall F1 or OOV recall, so it is reported only as a preliminary appendix experiment.Both encoders use the same dictionary and distributional features in the downstream evaluation.
Loading 2608.18437v1…