PaperlayerView exact revision on arXiv

Source-linked AI summary

GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation

Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh, Thanh Danh Le, Khang Nhat Hoang Vo, Huynh Dai Nhan Tran

arXiv:2610.30067v2cs.CL

TL;DR

Vietnamese applications need encoders that are compact and work on raw text, but existing strong models carry deployment costs or require external segmentation. GoiBERT distills complementary representation and task-level signals from two teachers; across eight benchmarks, its large variant leads non-teacher models with fewer than half of BamiBERT’s parameters.

  • Problem

    Strong Vietnamese encoders can be costly to deploy or require external word segmentation, leaving a need for compact, segmentation-free models that perform across diverse tasks.

    Source: §1 · pp. 1–2

  • Method

    GoiBERT uses a tokenizer-compatible Vietnamese teacher for general representation transfer and a stronger cross-tokenizer teacher for out-of-fold task-level soft labels.

    Source: §1 · p. 2

  • Results

    81.13 average score for GoiBERT-large leads non-teacher models across eight benchmarks; compression is nearly free for sentiment and classification but costs accuracy on NLI and NER.

    Source: §1 · p. 2, §5.1 · p. 6

  • Takeaways & Limitations

    GoiBERT-large is the better default, while GoiBERT-base suits tight CPU, edge-device, or batch-labeling budgets; NLI-heavy applications still favor a larger encoder.

    Source: §5.3 · p. 8

    General distillation was run once, reported seed ranges reflect fine-tuning variability rather than significance tests, and three-task ablations do not fully separate component interactions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Vietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers. A same-tokenizer Vietnamese encoder first teaches general-purpose representations. A much larger teacher then provides task-level guidance through soft labels that are generated without ever labeling its own training data. Across eight benchmarks covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition, GoiBERT-large achieves the highest average score among non-teacher models. It does so with fewer than half the parameters of the strongest compact baseline and twice its throughput. GoiBERT-base retains 99% of that baseline's performance at a quarter of its size. Our analysis shows that compression is nearly lossless for sentiment and classification tasks but remains costly for inference and entity recognition, which points to clear directions for future work. Our code and models are available at: https://github.com/Nhantran65/GoiBERT.git

1 Introduction

Vietnamese applications need compact, segmentation-free encoders that hold up across tasks, but existing strong models impose deployment costs or require segmented input. GoiBERT combines two teachers’ complementary signals in compact raw-text models, achieving strong benchmark performance with lower parameter counts and higher throughput than BamiBERT.

1 Introduction

  • Existing strong Vietnamese models bring deployment costs: CafeBERT has 559M parameters, PhoBERT requires external word segmentation, and BamiBERT has 102M parameters.The paper identifies a need for a compact, segmentation-free encoder suited to CPU, small-GPU, or tight-latency settings.Source: §1 · p. 1
  • GoiBERT-large has the highest non-teacher average with less than half of BamiBERT’s parameters and twice its GPU throughput; GoiBERT-base retains 99% of BamiBERT’s average with a quarter of its parameters.GoiBERT-base also has 3.5× BamiBERT’s throughput.Source: Figure 1 · p. 1, §1 · p. 2
  • GoiBERT-base and GoiBERT-large are 26.4M- and 49.4M-parameter encoders that process raw text without word segmentation.The models provide two accuracy–efficiency operating points.Source: §1 · p. 2
  • The two-stage method uses BamiBERT for tokenizer-compatible representation transfer and CafeBERT for out-of-fold task-level soft labels.The CafeBERT labels are generated so that no teacher labels an example it has seen during training.Source: §1 · p. 2
  • Across eight benchmarks, compression is nearly free for sentiment and classification but remains costly for NLI and NER.The analysis evaluates eight benchmarks and 15 metrics across three seeds.Source: §1 · p. 2

2 Related Work

Prior Vietnamese and other language-specific encoders have improved accuracy, while compact-model research has developed distillation methods for transferring teacher knowledge. GoiBERT combines these lines by assigning representation transfer and task supervision to teachers suited to each role.

2 Related Work

  • Vietnamese encoders progressed from multilingual models to monolingual and domain-specific systems, but many remain around or above 100M parameters and prior benchmarks emphasize accuracy.PhoBERT requires external word segmentation, whereas BamiBERT is a general-purpose raw-text encoder.Source: §2 · pp. 2–3
  • Compact-encoder distillation transfers softened predictions and intermediate signals; TinyBERT’s two-stage recipe combines general hidden-state and attention matching with task-specific distillation.Prior findings also indicate that teacher–student size gaps and distillation objectives can affect outcomes.Source: §2 · p. 3
  • Language-specific compact models typically inherit vocabulary and supervision from one teacher, while GoiBERT combines monolingual initialization with task supervision from a multilingual-origin teacher.Pruning, quantization, and parameter sharing are presented as orthogonal techniques that could be applied on top.Source: §2 · p. 3

3 GoiBERT

GoiBERT assigns representation transfer to a tokenizer-compatible teacher and task-level supervision to a stronger cross-tokenizer teacher. Its compact raw-text encoders use these complementary signals in general distillation followed by task fine-tuning.

3 GoiBERT

  • BamiBERT’s matching tokenizer enables token-level hidden-state and attention transfer, while CafeBERT’s different vocabulary limits clean transfer to sequence- or word-level predictions.The method assigns BamiBERT task-agnostic representation transfer and CafeBERT task-specific output supervision.Source: §3 · p. 3

3.1 Compact Architecture

  • GoiBERT-base has 10 layers and 26.4M parameters, while GoiBERT-large has 12 layers and 49.4M parameters; both reuse BamiBERT’s tokenizer to process raw text.Their vocabulary has 20,481 tokens, and the models do not require word segmentation.Source: §3.1 · pp. 3–4

3.2 Stage 1: General Distillation

  • General distillation matches student hidden states and head-averaged attention distributions to frozen BamiBERT on unlabeled text.For GoiBERT-base, an order-preserving mapping selects 10 of BamiBERT’s 12 layers as targets.Source: §3.2 · p. 4

3.3 Stage 2: Out-of-Fold Task Distillation

  • Five CafeBERT teachers, each fine-tuned on four of five folds, label only the held-out fold so each example receives a soft target from a teacher that has not seen it.Task fine-tuning uses gold labels and CafeBERT soft labels; output-level supervision avoids vocabulary alignment, with NER predictions aligned to the first subword of each syllable.Source: §3.3 · pp. 4–5
  • Layer-wise learning-rate decay updates lower student layers more conservatively than upper, task-specific layers.The stated decay factor is γ = 0.8.Source: §3.3 · p. 5

4 Experimental Setup

The evaluation spans eight Vietnamese tasks across four task families, with a consistent benchmark protocol and measurements of both predictive performance and deployment efficiency. Comparisons use established monolingual and multilingual encoders, including BamiBERT as the principal compact baseline.

4 Experimental Setup

  • The eight benchmarks cover text classification, ABSA, NLI, and NER across student feedback, e-commerce reviews, smartphone reviews, restaurants, hotels, and COVID-19 entities.Official train, validation, and test splits are retained.Source: §4 · p. 5
  • Classification and NLI use accuracy and macro-F1, ABSA uses aspect-detection F1 and aspect–sentiment pair micro-F1, and NER uses strict entity-level F1.The evaluation totals 15 metrics; main-text reporting uses one primary metric per task and averages locally trained models over three fine-tuning seeds.Source: §4 · p. 5
  • Baselines include PhoBERT-base-v2, ViSoBERT, BamiBERT, XLM-R-base, and CafeBERT, with BamiBERT serving as the principal comparison under the same downstream protocol.BamiBERT’s learning rate is tuned on validation data and it is fine-tuned with standard cross-entropy.Source: §4 · p. 5
  • General distillation uses 328.6M tokens of unlabeled Vietnamese text, excluding passages matching validation or test content by exact match or 13-gram overlap.Efficiency measurements include GPU throughput and peak memory on an L40S and CPU latency on a 128-token input with eight threads.Source: §4 · pp. 5–6

5 Results and Analysis

GoiBERT-large is the strongest non-teacher model overall, while GoiBERT-base offers a smaller, efficient alternative. Compression costs vary by task, and deployment gains and ablations clarify the trade-offs and contributions of each training stage.

5.1 Main Results

  • 81.13 average: GoiBERT-large leads non-teacher models, ahead of PhoBERT-v2 (80.71) and BamiBERT (80.65), and is best on five of eight tasks.Its full 15-metric average is 84.02, compared with 83.61 for BamiBERT and 82.98 for GoiBERT-base.Source: §5.1 · p. 6, Table 1 · p. 7
  • 80.02 average: GoiBERT-base retains 99.2% of BamiBERT’s average with 26.4M parameters, a quarter of BamiBERT’s size.It exceeds BamiBERT on both UIT-ABSA domains and VSFC-Topic, and averages above ViSoBERT and XLM-R-base despite being 3.7× and 10.5× smaller, respectively.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Where Does Compression Hurt?

  • 80.19 vs. 79.98: GoiBERT-large slightly surpasses its 559M-parameter task teacher on classification, but NLI and NER remain the main compression costs.Without ViNLI, GoiBERT-large averages 81.50 over seven tasks, 0.44 points below CafeBERT; NLI’s base–large gap is 2.66 points versus 0.89 on average elsewhere, and NER supervision omits multi-syllable word boundaries.Source: §5.2 · pp. 6–7, Figure 3 · p. 6

5.3 Efficiency–Accuracy Trade-off

  • 2× GPU throughput: GoiBERT-large also halves peak memory and cuts CPU latency by 37% versus BamiBERT; GoiBERT-base processes 3.5× as many sentences per second and uses less than a third of its memory.GoiBERT-base’s 4.7× FLOP reduction yields only 3.5× GPU and 2.2× CPU speed-ups, while GoiBERT-large’s 2.2× FLOP reduction yields a 2.0× GPU speed-up.Source: §5.3 · p. 7, Table 3 · p. 7
  • 33% and 23%: embeddings account for these shares of GoiBERT-base’s and GoiBERT-large’s parameters, while encoder parameters account for most savings over BamiBERT.GoiBERT-large and GoiBERT-base have 37.8M and 17.7M encoder parameters, respectively, compared with BamiBERT’s 85.1M.Source: §5.3 · p. 7

5.4 Ablation Study

  • +31.2 and +13.5 points: Stage 1 recovers these amounts on ViNLI and ABSA-Restaurant for GoiBERT-large versus training the compact architecture from scratch.Cross-tokenizer CafeBERT task distillation improves five of six settings by up to +3.0 points, while layer-wise learning-rate decay improves every task by 0.9–1.8 points.Source: §5.4 · p. 8, Table 4 · p. 8
  • 0.61 vs. 0.85 points: GoiBERT-large’s average seed range is narrower than BamiBERT’s across fine-tuning metrics.Its seed range is narrower on 11 of 15 metrics.Source: §5.4 · p. 8

6 Conclusion

GoiBERT is a family of compact Vietnamese encoders trained by two-stage distillation from complementary teachers. GoiBERT-large leads non-teacher models across eight benchmarks, while GoiBERT-base retains 99% of BamiBERT’s average with a quarter of its parameters.

6 Conclusion

  • GoiBERT-large achieves the best average score among non-teacher models across eight benchmarks, while doubling BamiBERT’s GPU throughput and coming within half a point of CafeBERT outside NLI.GoiBERT-large has 49.4M parameters; CafeBERT has 559M.Source: §6 · p. 8
  • GoiBERT-base retains 99% of BamiBERT’s average with a quarter of its parameters, using 26.4M parameters.The paper reports compression as nearly free for classification and ABSA, but still costly for inference and entity recognition.Source: §6 · p. 8
  • The two-stage recipe pairs a tokenizer-compatible teacher’s general representations with out-of-fold task supervision from a larger cross-tokenizer teacher.The authors propose extending the recipe to other languages whose strongest models are multilingual.Source: §6 · p. 8
  • Future work targets NLI- and NER-specific distillation, longer contexts, unseen domains, and quantization.The paper identifies inference and entity recognition as areas where compression remains costly.Source: §6 · p. 8

Limitations

The authors identify comparison, statistical, evaluation-scope, and efficiency-accounting limitations that constrain how GoiBERT’s results should be interpreted.

Limitations

  • GoiBERT’s comparison with BamiBERT is not a pure architecture comparison because distillation, layer-wise learning-rate decay, and benchmark training text also differ.The BamiBERT rerun uses plain cross-entropy fine-tuning.Source: Limitations · pp. 8–9
  • The larger variant was developed after earlier test results were examined, so test feedback partly shaped model development.This is an acknowledged source of potential evaluation leakage.Source: Limitations · p. 9
  • General distillation was run once, seed ranges reflect fine-tuning variability only, and the limited ablations do not fully separate component interactions.The 8- and 15-metric averages also mix metrics with different meanings.Source: Limitations · p. 9
  • Distillation used sequences up to 256 tokens, and the benchmarks do not test unseen domains or dialectal variation.Efficiency measurements exclude tokenization, task heads, and the substantial one-time cost of training five out-of-fold CafeBERT teachers per task.Source: Limitations · p. 9

Ethics Statement

The authors flag possible social biases from web data and teachers, and recommend checks before sensitive deployment.

Ethics Statement

  • GoiBERT may inherit social biases from publicly available web text and its teachers, and the models have not been audited for bias.The authors encourage checks before deploying them in sensitive settings.Source: Ethics Statement · p. 9
  • The authors identify lower running costs and access on more modest hardware as benefits of smaller Vietnamese NLP models.Source: Ethics Statement · p. 9
  • The bias warning applies to models trained on public text and teacher outputs, including GoiBERT.Source: Ethics Statement · p. 9

A Architecture and Initialization

The appendix details GoiBERT’s initialization, distillation procedures, and benchmark evaluation setup.

A Architecture and Initialization

  • GoiBERT-large initializes its word embeddings by projecting BamiBERT’s 768-dimensional matrix to 512 PCA components; Transformer blocks are random, and both variants support 2,050 positions.General distillation uses sequences of at most 256 tokens.Source: A Architecture and Initialization · p. 13

B Distillation Details

  • Hidden-state distillation aligns student layers to BamiBERT layers, using learned projections; GoiBERT-base skips teacher layers 3 and 9.The projection layers train jointly with the student while BamiBERT remains frozen; matching uses non-padding positions.Source: B Distillation Details · p. 14
  • Attention matching averages post-softmax probabilities across heads before comparing models, and begins at the first Transformer layer because embeddings have no self-attention matrix.Source: B Distillation Details · p. 14
  • CafeBERT task targets are generated out of fold: each teacher predicts held-out examples after fine-tuning on the other four folds.ABSA targets are produced per aspect slot, while NER targets align to the first subword of each original syllable.Source: B Distillation Details · p. 14
  • Task fine-tuning applies progressively smaller learning rates to lower encoder layers, while the task head receives the full base rate.The top Transformer layer receives the base rate; embedding parameters use ηγ^Lv.Source: B Distillation Details · pp. 14–15

C Benchmark Details

D Training and Measurement Details

The appendix reports training schedules, model selection, and the hardware and input conditions used for efficiency measurements.

D Training and Measurement Details

  • General distillation ran for 120,000 steps with batch size 32 and sequence length 256, using AdamW and a 10^-4 learning rate.It used seed 42 and bfloat16 on one NVIDIA L40S GPU.Source: D Training and Measurement Details · p. 15
  • Downstream fine-tuning ran for 30 epochs with batch size 32; learning rates varied by task, and BamiBERT was trained with cross-entropy only.Model selection used validation data, and reported local test results average fine-tuning seeds 42, 43, and 44.Source: D Training and Measurement Details · p. 15
  • Efficiency measurements cover encoder inference only, using an L40S GPU for throughput and VRAM and an eight-thread CPU for single-input latency.GPU runs used bfloat16 at batch size 32 and length 256; CPU latency used float32 and a 128-token input.Source: D Training and Measurement Details · p. 15

E Complete Benchmark Results

The complete benchmark results are reported across 15 test metrics, alongside parameter counts and estimated computational cost. The section also cautions how to interpret the average and pipeline comparison, and reports seed-wise variation for BamiBERT and GoiBERT-large.

E Complete Benchmark Results

  • The GoiBERT–BamiBERT comparison reflects complete downstream training pipelines, not architecture alone.GoiBERT uses CafeBERT distillation and layer-wise learning-rate decay, while the rerun BamiBERT baseline uses standard cross-entropy fine-tuning.Source: E Complete Benchmark Results · p. 15

F Seed-wise Variation

  • Across seeds 42, 43, and 44, ranges favor GoiBERT-large on nine metrics, BamiBERT on both ViNLI metrics and PhoNER entity F1, and overlap on three metrics.These ranges describe fine-tuning variability, not confidence intervals or significance tests, and exclude repeated general distillation.Source: F Seed-wise Variation · p. 15, Table 6 · p. 16

E Complete Benchmark Results

  • Table 5 reports test results across 15 evaluation metrics, with parameter counts and estimated computational cost.The 15-metric average is descriptive because metrics differ in interpretation and tasks with two metrics receive greater weight than PhoNER.Source: E Complete Benchmark Results · p. 15, Table 5 · p. 16

Related papers

Loading 2610.30067v2…