PaperlayerView exact revision on arXiv

Source-linked AI summary

Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malay

Joanito Agili Lopo, Yunita Sari, Guntur Budi Herwanto

arXiv:2606.11786v1cs.CL

TL;DR

Low-resource Kupang Malay lacks substantial parallel data and digital NLP support, limiting translation resources. The paper introduces linguistically informed instructions with Continual Instruction Tuning, and Lius outperforms standard-instruction, NMT, and multilingual LLM models by 10–13 points across various metrics.

  • Problem

    Kupang Malay lacks substantial parallel corpora and remains underrepresented in digital NLP resources, creating a low-resource translation gap.

  • Method

    The paper combines linguistically informed prompts based on lexical and semantic features with Continual Instruction Tuning for iterative LLM training.

  • Results

    10–13 points: Lius outperformed standard-instruction models, NMT models, and multilingual LLMs across various metrics in zero-shot and few-shot scenarios.

  • Takeaways & Limitations

    The approach shows potential for improving low-resource translation while retaining multilingual capabilities and competence on sentiment analysis and question answering.

Abstract

from arXiv · show

Large Language Models (LLMs) offer new potential for translation tasks but often experience performance degradation when handling low-resource languages. To address this limitation, we propose an approach for fine-tuning LLMs on a low-resource language, Kupang Malay. Our approach involves designing a set of instructions by leveraging explicit lexical and semantic features from a bilingual dictionary, and introducing Continual Instruction Tuning (CIT), a training paradigm that enables iterative instruction-based training. Experimental results demonstrate that our model, named Lius, yields notable improvements over standard instruction-tuned models by outperforming 4-6 points, and surpassing both Neural Machine Translation (NMT) and Multilingual LLM models by 10-13 points on several evaluation metrics. These findings highlight the potential of our approach to mitigate the reliance on large-scale parallel data in low-resource language translation.

1 Introduction

Low-resource languages remain poorly supported because parallel data and commercial translation systems are limited, motivating LLM-based translation with linguistically informed Continual Instruction Tuning. The proposed approach uses bilingual-dictionary features to improve modeling of lexical-semantic relations in low-resource languages.

  • Many languages lack adequate parallel data and are unsupported by commercial translation systems such as Google Translate.
  • Indonesia exemplifies this disparity, with over 718 local languages but only a small subset represented in NLP resources.Javanese has over 84 million speakers but only 12 million parallel sentence pairs, while Dutch has over 400 million in OPUS.
  • LLMs shift machine translation toward models trained on massive, diverse datasets that can generalize across NLP tasks without strict dependence on parallel corpora.Studies have reported competitive translation performance even in zero-shot settings.
  • The research proposes Instructional Linguistics with Continual Instruction Tuning, continuously training LLMs using linguistically informed instructions.Instructions incorporate bilingual-dictionary features including part-of-speech categories, synonyms, antonyms, and grammatical rules.
  • The approach aims to enhance LLM understanding of lexical-semantic relations, particularly for low-resource languages.

2 Related Work

Related work improves low-resource translation through instruction tuning and prompt engineering. These approaches use adapted LLMs, cross-lingual supervision, bilingual resources, and linguistic structure to improve multilingual generalization, interpretability, and lexical alignment.

  • Overview: Recent studies target low-resource multilingual translation with instruction tuning and prompt engineering.These strategies are presented as approaches for addressing challenges in low-resource language translation.
  • Instruction Tuning: LoRA adaptation of Llama-2, MaLA-500, and Mistral leverages monolingual and parallel corpora to improve low-resource language performance.Improvements are especially associated with pretraining aligned to the target language’s linguistic characteristics.
  • Instruction Tuning: Cross-lingual supervision aligns bilingual or multilingual data to enhance generalization across tasks and languages.Instruction-tuning frameworks can extract structured instructions from aligned data.
  • Prompt Engineering: Instruction-specific approaches combine monolingual corpora, bilingual dictionaries, and syntactic patterns to reduce ambiguity and improve lexical alignment.This prompt-engineering direction aims to improve interpretability and output quality without relying on heavy training.
  • Prompt Engineering: BLOOMZ and ChatGPT benefit from dictionary- and syntax-informed prompt engineering for low-resource language translation.The cited approach uses instruction-specific design rather than relying solely on extensive training.

3 Kupang Malay Language

Kupang Malay is a stable Malay-based creole and lingua franca in western Timor, with limited morphological complexity and several dialects. In NLP, it is low-resource because major multilingual datasets lack substantial parallel corpora and existing resources provide limited coverage.

  • Linguistic background: Kupang Malay is a Malay-based creole spoken in western Timor and widely used as a lingua franca across ethnic communities.Although formed through creolization rather than as a native ethnic language, it has developed into a stable, distinct linguistic system.
  • Linguistic features: Its morphology is minimal, with four prefixes and no suffixes or infixes, while its vowel system resembles Indonesian without schwa.Dialects including Air Mata, Alor Malay, and Basa Kupang reflect diverse vernacular influences.
  • Resource limitations: Kupang Malay is an NLP low-resource language because major multilingual datasets lack substantial parallel corpora and available datasets provide limited coverage.Sparse resources include Taxi1500, PanLex, and the Bhinneka Korpus.

4 Instructional Linguistic

Instructional Linguistics designs training instructions from linguistic theories and principles, using four prompt types: context-based, semantic-based, phonetic-based, and list-group-label-based. The prompts derive lexical, semantic, phonetic, and contextual relationships from bilingual dictionaries and monolingual Kupang Malay resources.

  • 4 Instructional Linguistic: Instructional Linguistics applies linguistic theories and principles to design training instructions that guide the model.The approach defines four prompt types: context-based, semantic-based, phonetic-based, and list-group-label-based.
  • 4.1 Background: The instructional approach is motivated by Direct Instruction and Explicit Instruction, emphasizing systematic delivery, clear objectives, and reduced cognitive load.It is also related to Krashen’s Second Language Acquisition theory from a linguistic perspective.
  • 4.2 Sentence Representation: Sentence representations identify salient lexical features for instruction generation using KeyBERT with Indonesian BERT embeddings.The most semantically relevant word is selected through cosine similarity between sentence and word embeddings.
  • 4.3 Context-Based Prompt: Context-based prompts retrieve Kupang Malay equivalents and dictionary examples containing the mapped semantic representation.If no equivalent exists, the original representation is retained; retrieved examples reinforce source-target contextual relationships.
  • 4.4 Semantic Mapping-Based Prompt: Semantic-based prompts map a salient Indonesian word to Kupang Malay and retrieve n nearest neighbors with FastText cosine similarity.The neighboring Kupang Malay words are matched to Indonesian counterparts through the bilingual dictionary.
  • 4.5 Phonetic-Based Prompt: Phonetic-based prompts compare source and target word representations using dictionary-derived phonetic rules and sequence matching.Words are considered phonetically similar when sim(W ∗mkn, Wmkn) ≥τ with threshold τ = 0.7.
  • 4.6 List-Group-Label-Based Prompt: List-group-label prompts identify related words, group them, and assign labels to broaden vocabulary and contextual understanding.For each sentence representation, FastText retrieves n closest Kupang Malay words to support cross-lingual semantic learning.

5 Continual Instruction Tuning (CIT)

Continual Instruction Tuning (CIT) is a training paradigm proposed to improve LLMs’ processing and use of the Instructional Linguistic approach. It uses iterative training, processing one data sample for every four instruction prompts.

  • Continual Instruction Tuning (CIT): CIT is proposed to optimize LLMs’ ability to process and leverage the Instructional Linguistic approach.The paradigm draws inspiration from curriculum learning, rubrics, and self-correction mechanisms.
  • Continual Instruction Tuning (CIT): One data sample is processed for every four instruction prompts in CIT’s continual training paradigm.The model learns iteratively rather than through a single training step.

6 Experimental Settings

The experiments use Indonesian–Kupang Malay parallel data, Kupang Malay monolingual data, multiple Cendol mT5 variants, and comparisons against instruction, multilingual, and existing Kupang Malay models. Translation quality is evaluated through automatic model comparisons and human ratings of adequacy and fluency.

  • Parallel Data: 66,521 Indonesian–Kupang Malay sentence pairs were compiled, with 53,217 for training and 13,304 for testing.English sentences from BibleNLP and The Language Archive were first translated into Indonesian for the study’s Indonesian-to-Kupang Malay direction.
  • Monolingual Data: Kupang Malay monolingual data from news, field-station, taxi, poetry, and pantun collections supported FastText-based instruction construction.The complete monolingual dataset statistics are reported in Table 2.
  • Model and Training Settings: Three Cendol mT5 variants containing 350 million, 580 million, and 1.2 billion parameters were trained with Experience Replay and memory-saving techniques.Training used 1,000 additional samples, BF16 computation, gradient checkpointing, and gradient accumulation over two steps.
  • Model Baseline: Baseline models compare standard instructions with linguistic instructional prompts while keeping all model-training parameters consistent.The standard-instruction baseline follows Cahyawijaya et al.’s translation instruction template.
  • Comparison with Other Models: Multilingual LLM comparisons use zero-shot and few-shot prompting, while Madlad400-3B-MT and Madlad400-7B-MT provide existing Kupang Malay translation baselines.Madlad400 models were trained across 419 languages and require the <2trg> input prefix.
  • Human Evaluation: Human evaluators rate translations for adequacy and fluency on four-level scales, with inter-annotator agreement measured using the Kappa coefficient.Several native speakers assess and rate the translation outputs.

7 Result & Analysis

Lius consistently outperformed standard instruction models and other multilingual or regional systems across translation evaluations, while retaining multilingual, multitasking, robustness, and human-rated translation capabilities. Larger models reduced copying behavior and achieved stronger translation performance, although adequacy and fluency agreement among evaluators remained limited.

  • Overall Performance: Lius variants outperformed standard instruction models across evaluation metrics, with the 1.2-billion-parameter model achieving the highest average performance by 2–6 points on each metric.The 300-million-parameter Lius variant also exceeded the standard instruction model of the same size by two points.
  • Semantic and Contextual Analysis: Lius improved performance by approximately 4–6 points across all evaluation metrics, indicating stronger semantic-feature capture and sentence-context comprehension.A lower TER score indicates that fewer edits were needed to match the reference translation.
  • Copying Behaviour: Copy Accuracy and Copy Rate declined as model size increased, while the Small Lius model had the highest copy accuracy at 23.62%.The Base and Large models’ lower copying tendency was associated with translating rather than copying, such as converting “tahun” to “taon.”
  • Comparison with Other Models: Lius outperformed multilingual NMT and regional LLM baselines across evaluation metrics, including few-shot SacreBLEU scores of 2.15 for mT0-XXL-MT and 1.13 for BLOOMZ-7B1-MT.The corresponding TER values were 173.29 and 372.48, respectively; Lius achieved up to a sixfold improvement over Madlad400 models.
  • Human Evaluation: Human evaluators rated most translations between 3 and 4 for fluency and adequacy, but weighted Kappa was 0.2099 for adequacy and 0.1339 for fluency.Adequacy scores were slightly more dispersed, indicating greater difficulty conveying meaning accurately in some translations.

8 Conclusion

The study proposes a linguistic instructional approach that trains LLMs through Continual Instruction Tuning with iterative, linguistically informed instructions. Using Cendol-mT5, the resulting model outperforms standard-instruction, NMT, and multilingual LLM baselines in zero-shot and few-shot prompting.

  • Method: The proposed linguistic instructional approach trains LLMs using the Continual Instruction Tuning paradigm and iterative instruction-based training.The method incorporates specific instructions during the training process.
  • Model: The model uses the Cendol-mT5 architecture for linguistic instructional training.
  • Results: The resulting model outperforms models trained with standard instructions, NMT models, and multilingual LLMs in zero-shot and few-shot prompting scenarios.

A Robustness Testing Examples

Robustness examples show that Lius can recover intended meaning from typographical errors and reconstruct contextually appropriate sentences after word shuffling, but struggles with word deletion. These results indicate resilience to some input distortions alongside limitations when information is omitted.

  • Typographical errors: Typographical errors do not prevent the model from capturing intended meaning and mapping distorted words to semantically relevant Kupang Malay expressions.For example, it transforms “emudian” into “abis itu” despite the altered input.
  • Word deletion: Word deletion substantially alters meaning, causing inaccurate outputs that may insert unrelated elements such as “bini” and “matua”.The model struggles to grasp holistic meaning when incomplete input removes words from the original phrase.
  • Word Shuffling: Word shuffling generally preserves the model’s ability to reconstruct structured and contextually appropriate sentences despite structural perturbations.Although the translation of “bagitu” is inaccurate, the model still produces diverse outputs reflecting resilience to shuffled input.

B Human Evaluation Process

Human evaluation used native speakers to assess Lius translations on fluency and adequacy. Evaluators came from varied backgrounds and mostly Kupang Malay-speaking areas, with 18 of 40 registered respondents completing the evaluation.

  • Evaluation criteria: Native speakers assessed Lius translations using fluency and adequacy as the two main evaluation questions.Fluency concerned grammar, spelling, and sentence structure; adequacy concerned whether all source information was accurately translated.
  • Evaluation procedure: The evaluation took place over approximately two months, beginning January 21.
  • Evaluators: 18 of 40 registered respondents completed the full evaluation, while four evaluators were excluded for giving identical scores across all translation samples.Participants included university students, lecturers, high school teachers, civil servants, and government language development officials, mostly from Kupang Malay-speaking areas.

C Instruction Template

The instruction template is a core prompt component for translation tasks, using nine variation types that collectively contain 50 distinct instructions. Variations include formal-direct commands, direct questions, narrative prompts, and structured examples such as List-Group-Label instructions.

  • Instruction template variations: Nine instruction template variations were developed, including formal-direct commands, direct questions, and narrative-style prompts.Each variation contains 3 to 5 instructions.
  • Instruction template variations: The variations comprise 50 distinct instructions used for translation tasks.The instruction template is marked with the label INSTRUCTION and forms a core component of the prompt template.
  • Instruction examples: List-Group-Label instructions combine an input sentence, category labels, a prompt template, and an Indonesian-to-Kupang-Malay translation instruction.An example uses “Aku makan kue tiga potong.” as input and labels such as “kala” and “siri.”
Loading 2606.11786v1…