PaperlayerView exact revision on arXiv

Source-linked AI summary

TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages

Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana Pierre

arXiv:2608.26923v1cs.CL

TL;DR

Kinyarwanda lacks a model that handles both its morphological complexity and the relational structure of administrative tables. TabuLM extends KinyaBERT with tabular representations and objectives, then evaluates them on newly collected government tables and TabQA-kin. It achieves 62.0% EM and shows that structural alignment is especially important for tabular reasoning, with morphology providing complementary gains.

  • Problem

    No existing Kinyarwanda model simultaneously handles morphological variation and the relational structure of administrative tables.

  • Method

    TabuLM extends KinyaBERT with row, column, and cell-type embeddings, a learned table-structure bias, MCR and CTP objectives, and pre-training on Rwandan government tables.

  • Results

    62.0% EM on TabQA-kin outperforms all fine-tuned baselines by 5.7–12.7 EM points, while fine-tuned models exceed zero-shot LLMs on aggregation.

  • Takeaways & Limitations

    Structural alignment is decisive for aggregation and important for lookup and comparison, while morphological processing contributes complementary gains.

  • Takeaways & Limitations

    The study is limited by a closed-source Kinyarwanda morphological analyzer and a small evaluation scope with 50 evaluable development items.

Abstract

from arXiv · show

We present TabuLM, the first language model pre-trained on Kinyarwanda tabular data. Kinyarwanda is a morphologically rich Bantu language spoken by over 12 million people in Rwanda, yet lacks any dedicated tabular representation learning resource. TabuLM extends KinyaBERT-large, a two-tier morphological transformer, with additive row, column, and cell-type embeddings and a learned table-structure attention bias that sharpens same-row and same-column attention. Pre-training uses two new objectives: Masked Cell Recovery (MCR), which masks entire cells and forces reconstruction from row and column context, and Column Type Prediction (CTP), which predicts column semantic types from observed cell values. We pre-train on 172 Rwandan government tables (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals, and introduce TabQA-kin, the first native Kinyarwanda table question-answering benchmark comprising 526 QA pairs across 31 tables and four question types. TabuLM achieves 62.0% exact match on TabQA-kin, outperforming KinyaBERT-large by 5.7 EM points and all multilingual baselines (mBERT 49.3%, XLM-R 50.0%) by 11.7-12.7 points. Analysis shows that structural table embeddings are most decisive for comparison and lookup questions, while morphological awareness provides complementary gains. Our code, data, and pre-trained checkpoint are publicly available.

1 Introduction

TabuLM addresses the lack of Kinyarwanda tabular language modeling by combining morphological processing with table-structure representations and objectives. It introduces a government-table pre-training corpus and TabQA-kin benchmark, achieving 62.0% exact match while revealing distinct roles for structural and morphological signals.

  • 1 Introduction: Kinyarwanda administrative data is predominantly tabular, but existing Kinyarwanda models consume free text and miss its row–column relations.The gap combines morphological surface variation with absent tabular pre-training signals.
  • 1 Introduction: TabuLM extends KinyaBERT with row, column, and cell-type embeddings plus a learned table-structure attention bias, adding fewer than 0.1% new parameters.The bias encodes same-row, same-column, and header signals at every transformer layer.
  • 1 Introduction: Masked Cell Recovery and Column Type Prediction provide pre-training signals for reconstructing cells from relational context and predicting semantic column types.MCR masks entire cells, while CTP predicts numeric, textual, categorical, or temporal types.
  • 1 Introduction: TabQA-kin contributes the first native Kinyarwanda table-QA benchmark, containing 526 question-answer pairs over 31 tables across five administrative domains.The benchmark covers census, agriculture, education, health, and infrastructure.
  • 1 Introduction: The tabular components are language-agnostic, while only the Tier 1 morphological encoder requires language-specific resources.This design is described as applicable to other Bantu and morphologically rich low-resource languages.

3 Background: KinyaBERT

KinyaBERT is a two-tier Kinyarwanda model that represents words through morphology before encoding them in a word-level sequence transformer. Its representation combines morphological summaries with stem embeddings and uses morphology-aware positional biases.

  • 3 Background: KinyaBERT: KinyaBERT is the state-of-the-art pre-trained language model for Kinyarwanda, built from two transformers operating at different granularities.Tier 1 processes individual words; Tier 2 processes the resulting word sequence.
  • 3 Background: KinyaBERT: Tier 1 uses POS, stem, affix-set, and individual morpheme embeddings to produce a 128-dimensional morphological summary for each word.The summary is taken from the transformer output at the POS-slot position.
  • 3 Background: KinyaBERT: Tier 2 encodes each word from its projected morphological summary concatenated with its stem embedding into a 512-dimensional token representation.The sequence transformer has 12 layers and 8 heads.
  • 3 Background: KinyaBERT: KinyaBERT combines TUPE relative-position bucketing with a POS-aware relative-position bias and uses word-level masked stem, affix-set, and affix-distribution objectives.These components preserve both sequence-position and morphological information during pre-training.

4 Related Work

Prior tabular language models model relational structure but are English-only, while KinyaBERT models Kinyarwanda morphology without table-aware representations. TabuLM combines morphological processing with structural serialization, tabular embeddings, attention bias, and cell-level pre-training objectives.

  • Related tabular models: English tabular models such as TAPAS, TaBERT, TURL, TABBIE, and TARTE support table tasks but assume English tokenization.These approaches cover table QA, table-to-text generation, entity linking, relation extraction, and masked cell prediction.
  • Low-resource and morphological NLP: KinyaBERT and other morphologically aware approaches target Kinyarwanda or morphology but do not incorporate tabular structure.KinyaBERT’s free-text pipeline is extended here for numeric expressions, administrative names, and agricultural compounds, with BPE fallback for out-of-vocabulary tokens.
  • TabuLM representation: TabuLM serializes tables as flat sequences using [TAB], [ROW], and [CEL] tokens, while attaching grid coordinates and cell types to every token.The coordinates feed row, column, and cell-type embeddings and the two new tabular objectives.
  • TabuLM representation: TabuLM adds row, column, and cell-type embeddings to KinyaBERT representations without changing the model dimension.The learned matrices use dimensionality d = 512, with row and column limits of 64 and 24 respectively.
  • TabuLM representation: A learned attention bias promotes same-row, same-column, and header-aware interactions at every transformer layer.The three terms support co-referent cells, cross-row value comparison, and attention to column labels.
  • TabuLM pre-training: Masked Cell Recovery reconstructs fully masked cells from row and column context, while Column Type Prediction infers semantic column types from observed values.CTP predicts NUMERIC, TEXT, CATEGORICAL, or DATE; both objectives are added alongside KinyaBERT’s three inherited objectives.

6 Tabular Data Collection

TabuLM’s data pipeline extracts Kinyarwanda tables from Rwandan government open-data portals, serializes them structurally, and pre-trains on MCR and CTP before fine-tuning for TabQA-kin. The resulting resources include a 172-table pre-training corpus and a native Kinyarwanda QA benchmark with four question types.

  • Data sources: The corpus combines NISR, Rwanda Agriculture Board, Rwanda Education Board, and Ministry of Health tables.The sources cover census and economic data, crop yields, school records, examinations, health facilities, and disease surveillance.
  • Corpus construction: 172 tables containing approximately 35,000 cells form the final pre-training corpus after rule-based PDF and HTML extraction and filtering.Tables with fewer than 3 rows, fewer than 2 columns, or more than 50% empty cells are discarded.
  • Benchmark construction: TabQA-kin uses native Kinyarwanda templates for LOOKUP, COMPARISON, COUNT, and AGGREGATION questions.The benchmark is constructed from tables and annotated question-answer pairs, with templates authored by a native Kinyarwanda speaker.
  • Training pipeline: TabuLM pre-training serializes extracted tables with structural tokens and applies Masked Cell Recovery and Column Type Prediction.The resulting checkpoint is then fine-tuned on TabQA-kin for final evaluation.

7 Experiments

The experiments evaluate TabuLM’s training setup, baselines, and TabQA-kin performance. TabuLM reaches 62.0% overall EM and outperforms fine-tuned baselines, while aggregation remains difficult for zero-shot LLMs.

  • Baselines: TabuLM is evaluated against multilingual, morphologically aware, ablated, and zero-shot LLM baselines using controlled fine-tuning protocols.The comparison includes mBERT, XLM-R, KinyaBERT, four TabuLM ablations, GPT-4o, and GPT-4o-mini.
  • Results: 62.0% overall EM: TabuLM outperforms all fine-tuned baselines by 5.7–12.7 EM points.Overall EM is computed over each model’s own evaluable set, which differs because tokenizers pack different amounts of table content.
  • Results: 66.7% COMPARISON EM: TabuLM improves from 41.7% after top-2 question-relevant row restriction, while KinyaBERT reaches 59.1%.The inference restriction targets the rows most relevant to the question.
  • Results: 16.7–28.6% LOOKUP EM improvements occur across fine-tuned models with joint row–column filtering.The filtering restricts candidate cells to the most relevant row and column, resolving ambiguity when values repeat.
  • Results: 64.0% overall EM is reached by both GPT-4o and GPT-4o-mini, with aggregation EM of 25.9% and 29.6%, respectively.All fine-tuned models exceed both LLMs on aggregation, including TabuLM at 79.2% and KinyaBERT at 88.9%.
  • Statistical analysis: The strongest statistical separation occurs on AGGREGATION, where fine-tuned-model confidence intervals do not overlap the LLM intervals.For overall EM, TabuLM and KinyaBERT intervals overlap, so that overall gap is described as a strong trend rather than definitive.

8 Analysis

Analysis identifies additive tabular embeddings as TabuLM’s dominant structural component, while attention bias and the new objectives show limited detectable effects at this scale. The model also adapts rapidly and offers local deployment advantages, but the study remains constrained by a small corpus and benchmark.

  • Architectural analysis: 5 × 10^-6 attention-bias scalars are effectively zero, whereas tabular embedding magnitudes reach 0.02–0.23.This indicates that structural information is routed primarily through row, column, and cell-type embeddings in the final checkpoint.
  • Ablations: −4.0 EM: removing tabular embeddings lowers performance from 62.0% to 58.0%, the largest single-component ablation drop.The noTabEmb variant must rely on the near-zero attention bias for structural awareness.
  • Ablations: +2.0 EM for removing either MCR or CTP remains within the 2% noise floor of the 50-item dev set.Neither objective provides a detectable benefit at this evaluation scale.
  • Adaptation: 52% EM at epoch 1: TabuLM starts 18 percentage points above noTabEmb before task-specific updates.TabuLM peaks at epoch 9, while noTabEmb requires all 20 epochs to reach 52%.
  • Training dynamics: MCR loss falls from approximately 3.8 to 0.8 NLL in the first 2,000 iterations, while CTP falls below 0.05 CE after approximately 1,500 iterations.The inherited KinyaBERT objectives converge more slowly and stabilize around iteration 5,000.
  • Limitations and efficiency: The 172-table corpus is both a computational advantage and a limitation compared with English benchmarks using much larger tabular resources.The paper estimates that doubling the corpus to 344 tables and training for 20,000 iterations would cost under 44 GPU-hours.
  • Deployment: 65M parameters: TabuLM runs on one consumer GPU without API dependency or per-query cost.Its aggregation result is 79.2% versus GPT-4o’s 25.9% in the deployment comparison.

9 Discussion

TabuLM’s analysis separates the roles of structure and morphology: structural alignment dominates aggregation, while morphological representations help match entities for lookup and comparison. Zero-shot LLMs instead show a persistent aggregation failure involving row-entity selection.

  • The LLM aggregation ceiling: 7.4% aggregation performance with 3-shot examples shows that prompting did not correct LLMs’ tendency to return numeric values instead of the maximum-value row entity.Fine-tuned models are trained through cell-selection supervision to output an entity cell rather than a numeric derivation.

10 Conclusion

TabuLM jointly models Kinyarwanda morphology and tabular relational structure through architectural extensions and new pre-training objectives. It reaches 62.0% TabQA-kin exact match and surpasses fine-tuned baselines, while domain-specific fine-tuning addresses aggregation failures seen in zero-shot LLMs.

  • 10 Conclusion: TabuLM is the first pre-trained model designed to jointly capture morphological richness and tabular relational structure for a low-resource language.It extends KinyaBERT with additive tabular embeddings, a learned structure bias, and MCR and CTP objectives.
  • 10 Conclusion: 62.0% EM on TabQA-kin surpassed all fine-tuned baselines by 5.7–12.7 EM points.The benchmark evaluates native Kinyarwanda table question answering.
  • 10 Conclusion: 64.0% was the identical overall result for GPT-4o and GPT-4o-mini, with aggregation failure producing a 25–30% ceiling.Fine-tuned TabuLM and KinyaBERT reached 79.2% and 88.9% aggregation EM, respectively, exceeding this LLM ceiling.
  • 10 Conclusion: The paper releases code, the 751 MB TabuLM checkpoint, TabQA-kin, and the 172-table pre-training corpus.The release supports reproduction and further work on low-resource tabular language modeling.

Limitations

TabuLM’s limitations concern reproducibility, corpus scale, evaluation power, and language coverage. The released BPE fallback and public data improve practical accessibility, but the evidence remains limited to Kinyarwanda and a small corpus and benchmark.

  • Reproducibility: The closed-source libkinlp.so analyzer limits reproducibility outside the original KinyaBERT team, although the BPE fallback enables reproduction without it.Fallback-tokenized domain terms lose affix-level granularity, while the tabular architecture and evaluation protocol remain unchanged.
  • Data scale: The pre-training corpus contains 172 tables and approximately 35,000 cells, substantially smaller than English tabular datasets using millions of Wikidata triples.Near-zero attention-bias scalars of approximately 5 × 10^-6 suggest that larger corpora and longer pre-training could make this pathway contribute more meaningfully.
  • Evaluation scope: TabQA-kin covers four Rwandan administrative domains, and only 50 dev items were evaluable for TabuLM, limiting statistical power.Overall confidence intervals overlap between TabuLM and the best baseline, although the aggregation finding is statistically robust at this scale.
  • Language coverage: All experiments are conducted on Kinyarwanda, so broader Bantu-language validation remains outside the reported evaluation.The tabular components are language-agnostic, but extension is bottlenecked by domain-relevant tabular corpora rather than modeling.
  • Data scope: The study uses official Rwandan government open-data under public-domain or government open-data licenses and excludes personally identifiable information.The benchmark contains factual questions about administrative statistics, with anonymized and aggregated census data handled using standard practices.

A TabQA-kin Question Templates

This appendix specifies TabQA-kin’s template construction, illustrates the pre-training data and objectives, and records the training and sensitivity settings. The documented protocol emphasizes reproducibility while showing that 15% cell masking provided the best MCR loss trajectory.

  • A TabQA-kin Question Templates: TabQA-kin templates substitute column headers, row entities, entity types, and numeric thresholds into four Kinyarwanda question types.Templates were authored by a native Kinyarwanda speaker; count questions are excluded from cell-selection evaluation.
  • B Pre-training Data Example: The representative agriculture table combines morphologically complex Kinyarwanda headers with numeric cells, requiring row and column context for masked-cell reconstruction.The example masks Bugesera’s 38,145 t yield value, while CTP predicts NUMERIC for the Umusaruro column.
  • C Hyperparameter Details: The appendix includes complete hyperparameter settings for TabuLM and its ablation variants.The hyperparameter table is presented as Table 10.
  • C Hyperparameter Details: Fine-tuning unfreezes the top four transformer layers and cell-selection head while freezing lower layers and using dev EM for early stopping.The cell-selection head scores cells by averaging token scores before softmax normalization.
  • C Hyperparameter Details: Baseline models use identical fine-tuning hyperparameters, with standard multilingual tokenizers and KinyaBERT’s native tokenizer plus BPE fallback.The fallback handles numerals, administrative compound names, and agricultural terminology.
  • D Hyperparameter Sensitivity: 4 × 10−4 achieved stable, rapid convergence, while 1 × 10−4 converged slowly and 1 × 10−3 caused NaN gradients at iteration 200.The learning-rate sweep compared 1 × 10−4, 4 × 10−4, and 1 × 10−3.
  • D Hyperparameter Sensitivity: 15% cell masking yielded the best MCR loss trajectory, whereas 5% provided insufficient signal and 30% impoverished contextual evidence.The comparison was performed across mask rates of 5%, 15%, and 30%.
  • D Hyperparameter Sensitivity: The half-dimension tabular embeddings matched fine-tuning EM but required 18% longer pre-training convergence.The full-dimension variant was retained to match KinyaBERT’s input dimension.

E TabQA-kin Construction Details

TabQA-kin was constructed from 31 selected Rwandan government tables using entity- and column-aware question templates, with native-speaker verification and explicit exclusions for count questions.

  • Table selection: 31 tables were selected using entity identifiers, numeric columns, and a fewer-than-40-rows constraint.The selected tables supported comparison and aggregation questions while keeping annotation manageable.
  • Template instantiation: 526 questions were generated by pairing entity names and column headers with applicable templates, checking answer-cell uniqueness, and deduplicating surface variants.The questions were shuffled and split 80/20 using seed 42.
  • Quality verification: 53 questions, representing a 10% random sample, received independent native-speaker approval for grammar, answer accuracy, and answer-cell uniqueness.The authors estimate a full-benchmark error rate below 2%.
  • Count question exclusion: 93 count questions, or 17.7% of the benchmark, were retained for release but excluded from reported exact-match evaluations.These questions require derived integer answers absent from individual cells and are reserved for future aggregation-with-counting models.
  • Error analysis: Additional error analysis identifies wrong column alignment from ambiguous column cues and duplicate numeric values that defeat column-wise disambiguation.These failure modes are illustrated in Table 11.
Loading 2608.26923v1…