PaperlayerView exact revision on arXiv

Source-linked AI summary

A Character-Level Neural Approach to Sinhala Sandhi Splitting

Yasas Ekanayaka, Deshan Sumanathilaka

arXiv:2609.36131v1cs.CL

TL;DR

Sinhala Sandhi obscures lexical boundaries, and no neural benchmark for splitting it had been established. This study evaluates character-level recurrent sequence transduction on SandhiLex. Its strongest model reaches 68.40% EM on the hard subset, compared with 94.00% EM on affixational Sandhi.

  • Problem

    Sandhi obscures constituent boundaries and can affect downstream Sinhala tools, while no neural benchmark for Sinhala Sandhi splitting had been established.

    Source: §1 · p. 1

  • Method

    The study trains character-level recurrent encoder-decoder models on SandhiLex and compares native Sinhala Unicode with a romanized representation.

    Source: §1 · p. 2, §3 · p. 3, §4.1 · p. 4

  • Results

    68.40% EM on the hard subset and 94.00% EM on the affixational subset; the bidirectional LSTM encoder with unidirectional LSTM decoder is strongest.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    The results establish an empirical Sinhala Sandhi-splitting baseline and suggest that etymological forms may require lexical retrieval rather than sequence transduction.

    Source: §10 · p. 9

    The hard subset has only about 4,500 pairs and combines three linguistically distinct Sandhi types, preventing separate analysis.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Sinhala Sandhi splitting recovers the constituent words or morphemes hidden inside a phonologically merged surface form. The task is important for Sinhala NLP because Sandhi obscures lexical boundaries, but no prior published work has established a neural benchmark for Sinhala Sandhi splitting. We present a character-level sequence-to-sequence study based on SandhiLex, using native Sinhala Unicode input and evaluating recurrent encoder-decoder models for affixational and more complex lexicalized, derivational, and etymological Sandhi. The central challenge is the hard subset lexicalized, derivational, and etymological Sandhi, where our best model, a bidirectional LSTM encoder with a unidirectional LSTM decoder, reaches only 68.40\% exact-match accuracy (82.08\% character-level accuracy), well below the 94.00\% achieved on the more regular affixational subset. Ablations show that bidirectional encoding is the largest contributor to performance, while native Sinhala script improves exact match accuracy over romanized input. Qualitative analysis indicates that many errors are near misses involving boundary adjacent characters or plausible but incorrect phonological substitutions. These results establish an empirical baseline for Sinhala Sandhi splitting and identify data scale, Sandhi type conditioning, and attention-based decoding as the main directions for future work.

1 Introduction

Sinhala Sandhi obscures constituent boundaries through phonological changes, complicating downstream language technologies and practical Sinhala tools. The study addresses the absence of a neural benchmark by evaluating character-level sequence transduction on SandhiLex.

1 Introduction

  • Sandhi merges adjacent words through vowel coalescence, consonant assimilation, or sound deletion, obscuring lexical units that whitespace-based NLP pipelines assume are recoverable.The resulting errors can propagate through morphological analysis, information retrieval, and machine translation.Source: §1 · p. 1
  • Reliable morpheme-boundary detection is needed for Sinhala spell-checkers, dictionary lookup, and predictive typing, and its absence obstructs these tools.Growing Sinhala digital content also increases the importance of accurate segmentation for downstream language technologies serving the community.Source: §1 · p. 1
  • Existing Sinhala computational approaches use rule-based joining, whose reversal cannot principledly choose among plausible word pairs and requires expert maintenance.Despite SandhiLex and evidence for character-level neural models in Sinhala morphology, no neural benchmark for Sinhala Sandhi splitting existed.Source: §1 · pp. 1–2
  • The study evaluates recurrent character-level encoder-decoder models on SandhiLex to compare architectures, encoding factors, and quantitative versus linguistic accuracy.Sinhala has four Sandhi categories: affixational, lexicalized, derivational, and etymological.Source: §1 · pp. 1–2

2 Related Work

Prior work ranges from rule-based Sandhi processing to neural splitting in other South Asian languages. Differences in task direction, linguistic setting, and data scale motivate a data-efficient neural evaluation for Sinhala.

Sinhala Sandhi and Sinhala NLP

  • Sinhala Sandhi splitting recovers constituent boundaries obscured by phonologically conditioned surface changes, which can include vowel coalescence, assimilation, insertion, and deletion.SandhiLex distinguishes affixational, lexicalized, derivational, and etymological forms.Source: §2 · p. 2

Rule-based Sandhi processing

  • Sinhala's rule-based word joiner handles predictable transformations without training data, but it generates merged forms rather than splitting them.Reversing its rules is difficult when multiple underlying pairs could yield the same surface form, and splitting would need an explicit disambiguation mechanism.Source: §2 · p. 2
  • Rule-driven approaches are interpretable and need no training data, but require expert maintenance and struggle to distinguish competing underlying forms.These limitations are reported across related South Asian rule-based studies (V et al., 2014; Kallur and Anami, 2025).Source: §2 · p. 3

Neural Sandhi splitting

  • Neural Sandhi models have been studied for Sanskrit and other Indian languages, but Sanskrit resources exceed 560,000 examples versus approximately 4,500 in Sinhala's hard subset.The study avoids direct numerical comparisons with Sanskrit results because the data-scale difference could confound architecture comparisons.Source: §2 · p. 3
  • The study adopts character-level sequence-to-sequence modeling, prioritizes bidirectional encoding, and evaluates affixational and hard subsets separately to examine complexity and data scale.Prior Sinhala morphology evidence motivates bidirectional encoding, while the paper later reports that its benefit depends on model capacity.Source: §2 · p. 3

3 Task and Data

The task maps a merged Sinhala Sandhi form to its segmented constituents using SandhiLex. The dataset supports an affixational subset and a harder subset combining three less transparent Sandhi types.

3 Task and Data

  • The task maps an input Sandhi sequence to split constituents, and the output can be longer when splitting restores characters changed or deleted during merger.For example, વિહාරාධિકාරviharādhikāra maps to විහාරvihāra + අධිකාරadhikāra.Source: §3 · p. 3
  • SandhiLex pairs each merged Sinhala Unicode form with its segmented constituents, and this study treats each row as a supervised input-target sequence pair.The lexicon was developed by Liyanage and Pushpananda (2024); this paper contributes the first neural modeling study built on it.Source: §3 · p. 3
  • The easy subset contains affixational forms, while the hard subset pools lexicalized, derivational, and etymological forms.More than 300,000 easy pairs were available, with 150,000 sampled for the full-data experiment; the hard subset contains approximately 4,500 forms.Source: Table 2 · p. 3, §3 · p. 3
  • The dataset's category table presents Sinhala Sandhi categories with transliteration and English glosses.SandhiLex distinguishes four categories: affixational, lexicalized, derivational, and etymological.Source: §3 · p. 3, Table 1 · p. 4
  • The hard subset is the main benchmark for opaque forms because original boundaries are less transparent and surface forms may omit constituent characters.Examples include dakunata, split as dakunu + ata, and kammala, split as kam + hala.Source: §3 · p. 4

4 Methodology

The methodology uses character-level sequence transduction on cleaned SandhiLex pairs, compares native and romanized representations, and evaluates recurrent and additional model variants with task-specific metrics.

4.1 Preprocessing

  • Preprocessing removes empty and duplicate entries, then compares native Unicode with a character-mapped romanized version while preserving the same pairings.The controlled representation comparison tests script choice; romanization may merge phonologically distinct Sinhala characters and lose orthographic contrasts relevant to Sandhi boundaries.Source: §4.1 · p. 4

4.2 Encoder-Decoder Model

  • A bidirectional LSTM encoder processes input characters in both directions, concatenates its states into a context vector, and initializes a unidirectional decoder.The decoder generates one output character per step through a softmax projection until the end token or maximum length.Source: §4.2 · p. 5

4.3 Training Protocol

  • Separate easy and hard datasets are trained for 10 and 25 epochs respectively, using a 10% validation split and an 80:20 train-test setup.Model and training settings are summarized in Table 3.Source: §4.3 · p. 5, Table 3 · p. 5

4.4 Compared Systems and Metrics

  • The comparison includes recurrent models with unidirectional and bidirectional encoders, manually configured and double-decoder LSTMs, and a SinBERT Large experiment.Evaluation reports exact match and character-level accuracy, plus word-level precision, recall, and F1 on a randomly selected 500-sample hard-set test subset.Source: §4.4 · p. 5

5 Results

The bidirectional LSTM encoder-decoder is the strongest recurrent model on the hard subset, but its results remain below those on affixational Sandhi. Additional comparisons show effects of data volume and script representation, while alternative model families perform poorly in this setup.

5 Results

  • 68.40% EM and 82.08% CLA: the bidirectional LSTM encoder-decoder leads the hard-subset architecture comparison, with bidirectionality adding 26.0 EM points over the unidirectional LSTM.Bidirectional encoders outperform unidirectional encoders across the recurrent cell types compared.Source: Table 4 · p. 5, §5 · p. 5
  • The reported result is significant at 3.076 × 10−7 (p < 0.05), supporting that BiLSTM’s advantage over BiGRU reflects a genuine difference in predictive behaviour rather than chance.On the randomly selected 500-sample hard-subset test set, word-level exact-match precision, recall, and F1 are each 63.60%.Source: §5 · pp. 6–7
  • Many incorrect predictions are near misses: examples preserve the constituent boundary but substitute or omit a boundary-adjacent vowel, receiving zero exact-match credit.The examples include a final-vowel mismatch in the predicted split of දකුණත and a vowel substitution in the split of සිහසුන.Source: §5 · p. 6
  • Native Sinhala Unicode yields the best exact-match score for the bidirectional LSTM, whereas romanized input yields higher CLA but lower EM.The comparison preserves the same Sandhi and split pairings across script representations.Source: Table 6 · p. 6, §5 · p. 6
  • 94.00% EM on the full affixational subset falls to 68.40% when that subset is size-matched to the hard training regime, indicating a data-volume constraint alongside linguistic opacity.The bidirectional LSTM reaches 98.24% CLA on the full affixational subset.Source: §5 · p. 6, Table 7 · p. 7
  • 2.00% EM for SinBERT Large and 1.00% for Double Decoder LSTM: neither alternative outperforms the recurrent encoder-decoder baseline.The authors caution that these findings apply only within the current dataset and implementation setup.Source: Figure 2 · p. 7, §5 · p. 7

6 Analysis and Discussion

The bidirectional LSTM encoder with a unidirectional LSTM decoder is the strongest tested architecture, with performance varying across Sandhi types and architecture settings. Native Sinhala Unicode favors exact recovery, while the additional architecture results are specific to the available data and implementation.

6 Analysis and Discussion

  • 68.40% EM, 82.08% CLA, and 63.60% F1 on the hard subset; the bidirectional LSTM encoder-decoder outperforms the unidirectional LSTM and bidirectional GRU baselines.Native Sinhala Unicode also improves exact recovery compared with romanized input.Source: §6 · p. 7
  • Bidirectionality adds 26.0 EM points for LSTM, 17.1 for GRU, and 4.0 for RNN, so its benefit varies with recurrent cell type.The paper relates this pattern to the gating capacity needed to preserve and combine two-way context over a Sandhi form.Source: §6 · p. 7
  • 94.00% EM on the full 150,000-example affixational subset contrasts with 68.40% on the hard subset, trained on about 4,500 examples and containing less regular forms.The size-matched easy-subset comparison also shows a substantial EM drop, indicating that data volume contributes to the gap.Source: §6 · p. 7
  • The tested SinBERT Large and Double Decoder LSTM settings do not surpass the bidirectional LSTM encoder-decoder under the available data conditions.The authors limit this finding to the current dataset and implementation setup.Source: §6 · pp. 7–8

7 Ablation Studies

The ablations compare native script, latent dimension, and hyperparameter tuning against the full model; encoder directionality is treated separately as the largest contributor. Native script remains preferable for exact segmentation even though romanization can improve character-level accuracy.

7 Ablation Studies

  • Encoder directionality is not re-ablated because the unidirectional-versus-bidirectional LSTM comparison already isolates its status as the largest contributor, adding 26.0 EM points.The three separate ablations are native script, latent dimension, and hyperparameter tuning.Source: §7 · p. 8
  • Reduced latent dimension has the largest effect among the three separately ablated factors, followed by removing hyperparameter tuning and then romanizing the input.Exact deltas are reported in Appendix A.Source: §7 · p. 8
  • Romanization improves CLA, but native Sinhala Unicode remains preferable for exact segmentation because it preserves orthographic distinctions needed for complete recovery.Source: §7 · p. 8

8 Error Analysis

The analysis distinguishes near-miss errors from failures on etymological compounds, whose historical constituents may leave no trace in the surface form. A qualitative translation evaluation illustrates the practical relevance of splitting, but is not a controlled extrinsic benchmark.

8 Error Analysis

  • Boundary shifts alter a character near an approximate split, while phonological substitutions yield plausible but incorrect decompositions; complete failures occur mostly on low-frequency etymological compounds.Examples include a throne split with a nearby character change, an incorrect equation decomposition, and a semantically wrong smithy split.Source: §8 · p. 8
  • Boundary shifts and phonological substitutions may be reduced by more context or model capacity, but etymological forms can lack any surface trace of the historical constituent.The analysis therefore suggests treating etymological Sandhi as lexical retrieval; capacity ablation narrowed the boundary-shift gap while leaving etymological failures largely unaffected.Source: §8 · p. 8

8.1 Qualitative Translation Impact

  • After splitting, English meanings become recoverable for some merged forms that a general-purpose translator mistranslates or leaves untranslated; this is qualitative evidence, not a controlled extrinsic benchmark.Table 9 summarizes the reported examples.Source: §8.1 · p. 8, Table 9 · p. 8

9 Prototype Implementation

The prototype provides a locally validated demonstration of Sinhala Sandhi splitting, with separate easy and hard endpoints and a frontend displaying predicted splits, romanization, and translation. It is not a user study or an evaluation under real-world deployment conditions.

9 Prototype Implementation

  • The backend exposes /easy and /hard endpoints for the trained model variants, with CORS configured for local frontend communication.It uses FastAPI with asynchronous request handling and Pydantic-based request validation.Source: §9 · pp. 8–9
  • The React frontend accepts a Sinhala Sandhi word and displays its predicted split, romanized output, and English translation.The deployed inference pipeline serializes model weights and related configuration to match training.Source: §9 · p. 9
  • The prototype was functionally validated locally, but usability, latency, and accuracy under real-world deployment conditions were not evaluated.It is an implementation demonstration rather than a user study, and does not replace the quantitative benchmark.Source: §9 · p. 9

10 Conclusion

The strongest model performs substantially better on affixational Sandhi than on the hard subset, where the conclusion points toward lexical retrieval rather than sequence transduction for etymological cases. The paper also identifies data, analysis, and evaluation constraints that shape its future directions.

10 Conclusion

  • 68.40% exact match and 82.08% character-level accuracy on the hard subset contrast with 94.00% on the full affixational subset.The conclusion argues that added context or capacity cannot resolve etymological cases when the needed information is absent from the surface form, making lexical retrieval a more promising direction there.Source: §10 · p. 9
  • Future work should expand the hard subset, evaluate each Sandhi type separately, explore Sandhi-specific models trained on SandhiLex, and measure downstream gains in controlled NLP tasks.The proposed architectures include CNN-based and Sandhi-specific Transformer models rather than general-purpose pretrained models.Source: §10 · p. 9
  • Code and preprocessing scripts are public, but SandhiLex must be requested separately from LTRL-UCSC under its academic-use terms.The dataset is not redistributed with the release.Source: §10 · p. 9

Limitations

  • About 4,500 hard-subset pairs limit learning rare patterns, while grouping three Sandhi types prevents separate analysis of them.The grouped types are lexicalized, derivational, and etymological Sandhi.Source: Limitations · p. 9
  • Automatic evaluation uses a single reference split despite possible multiple valid analyses, so expert review is needed to assess prediction quality and practical translation impact.The model also lacks attention and uses greedy decoding; beam search and Sandhi-type-conditioned decoding remain future work.Source: Limitations · p. 9

A Ablation Details

The hard-subset ablation table varies one component at a time against the bidirectional LSTM baseline. Encoder directionality is addressed separately.

A Ablation Details

  • The table excludes encoder directionality because its unidirectional condition appears in Table 4, where removing bidirectionality reduces performance by 26.0 points.Source: A Ablation Details · p. 10
  • Table 10 changes one component per row relative to the bidirectional LSTM baseline on the hard subset.The baseline is reported as 68.40% EM and 82.08% CLA; the supplied passage does not provide the individual ablation values.Source: Table 10 · p. 10, A Ablation Details · p. 10
Loading 2609.36131v1…