LLMs for Low-Resource Dialect Translation Using Context-Aware Prompting: A Case Study on Sylheti

Research gap

• Sylheti–Bangla translation is underexplored despite Sylheti’s substantial speaker population and limited dialect-specific resources.Bangla itself remains relatively low-resource, while its dialects rarely have large-scale curated datasets.Source: §1 · pp. 1–2

Research questions

• The study asks how multilingual LLMs perform between Bangla and Sylheti and how their translation quality can be improved.The evaluation uses zero-shot prompting before introducing an intervention for the second research question.Source: §1 · p. 2

Initial evaluation

• Zero-shot evaluation shows a directional imbalance, with Sylheti→Bangla achieving 66.8% higher BLEU-1 on average than Bangla→Sylheti.LLaMA 4 and Grok perform best among the tested models in the reported evaluation.Source: §1 · p. 2

Proposed approach

• Sylheti-CAP integrates Sylheti-specific lexical, grammatical, and idiomatic knowledge into prompts, followed by fluency and correctness refinement.The framework is presented as a context-aware strategy for improving low-resource dialect translation.Source: §1 · p. 2, Figure 1 · p. 2

Main contribution

• Results reported across BLEU, METEOR, and ChrF show consistent improvements over zero-shot, few-shot, and chain-of-thought prompting, with fewer mistranslations, omissions, and awkward phrases.The introduction also reports more natural and faithful translations in Lommel (2013) evaluations.Source: §1 · p. 2

Motivation

• Sylheti-CAP addresses the failure of standard prompting assumptions by injecting structured linguistic rules and bilingual lexicons into the translation prompt.The approach targets Sylheti’s distinct phonology, grammar, and vocabulary under low-resource and domain-mismatch conditions.Source: §2 · pp. 2–3

Framework overview

• The framework uses three stages: a linguistic rulebook, a bilingual vocabulary-and-idiom dictionary, and an authenticity and fluency check.The stages are designed to keep outputs from becoming slightly modified Bangla rather than authentic Sylheti.Source: §2 · p. 3

Linguistic rulebook

• The rulebook specifies Sylheti grammatical, phonological, syntactic, and morphological transformations, including pronoun, verb, negation, imperative, and SOV-order rules.Examples include Bangla-to-Sylheti substitutions and phonological simplification.Source: §2 · p. 3

Vocabulary and idioms

• The dictionary supplies contextually appropriate substitutions for frequent words and idioms where direct translation is insufficient.The framework description reports a dictionary of 3,106 word pairs for this purpose.Source: §2 · p. 3

Authenticity check

• The final stage presents the Bangla source sentence with instructions favoring fluent, natural spoken Sylheti over literal translation.This check is intended to distinguish authentic Sylheti speech from formalized Bangla.Source: §2 · p. 3

Dataset

• The evaluation uses the Vashantor corpus, with 2,500 parallel sentences and a 375-sentence test set for each model.The corpus combines Sylheti and Bangla sentences collected from websites, social media, and discussion boards.Source: §3 · p. 3

Dictionaries

• The study derives 2,260 distinct Sylheti–Bangla word mappings from three parallel datasets, while noting that many words are identical across the dialect and standard language.The source datasets are Vashantor, ONUBAD, and a Sylheti dataset.Source: §3 · pp. 3–4

Models

• Five state-of-the-art LLMs from major developers are evaluated, including LLaMA 4, Gemini 2.5 Flash, GPT-4.1, Grok 3, and DeepSeek-V3.The models differ in architecture and technical specifications, enabling a broad comparative assessment.Source: §3 · p. 4

Evaluation setup

• Performance is assessed with BLEU, ChrF, and METEOR to provide complementary views of token-level, character-level, and semantic translation quality.The prompting comparison includes zero-shot, five-example few-shot, and chain-of-thought strategies.Source: §3 · p. 4

4.1 RQ1: Benchmarking LLMs for Sylheti ⇔Bangla

• Baseline LLMs translated into Bangla more effectively than into Sylheti, while zero-shot outputs often normalized Sylheti vocabulary, morphology, particles, and orthography into standard Bangla.Grok 3 led Sylheti→Bangla at 0.4855, whereas LLaMA 4 led Bangla→Sylheti at 0.3565; Grok’s directional ratio was 1.38 and LLaMA’s 1.31.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 RQ2: Enhance LLM’s Translation Performance by Sylheti-CAP

• Sylheti-CAP achieves the highest BLEU-1, METEOR, and ChrF scores across all models and both translation directions, outperforming Zero-Shot, Few-Shot, and CoT prompting.Tables 2 and 3 report consistent improvements, including Grok’s 0.47 BLEU-1 and 46.01 ChrF for Bangla→Sylheti and Deepseek’s 0.52 BLEU-1, 0.45 METEOR, and 53.72 ChrF for Sylheti→Bangla.Source: §4.2 · p. 5, Table 2 · p. 7, Table 3 · p. 7

4.3 Human Evaluation.

• Human evaluation found Sylheti-CAP produced the highest proportion of Good translations in both directions, with Deepseek V3.2 and Grok 3 exceeding 50% Good ratings and Poor outputs below 20%.The study used 200 samples and three native Sylheti speakers to compare four prompting strategies.Source: §4.3 · p. 6

4.4 LLM-as-a-judge.

• Sylheti-CAP also achieved the highest adequacy, fluency, and overall LLM-as-a-judge scores, outperforming other prompting strategies by 3–10 points.GPT-5.0 judged the same 200-sample set on a 0–100 scale against reference Sylheti sentences.Source: §4.4 · p. 6

4.5 MQM Evaluation.

• MQM scores were lowest for Sylheti-CAP in both directions—1.62 for Ben→Syl and 1.93 for Syl→Ben—because mistranslation, awkward phrasing, and omission penalties were reduced.Compared with Zero-Shot, mistranslation penalties fell from 670 to 580 and omission penalties from 220 to 200.Source: §4.5 · pp. 6–7

4.6 LLMs’ Hallucinations.

• Sylheti-CAP consistently achieved the lowest hallucination rates, 12.6–13.8% versus typically 15–17% for CoT, Few-Shot, and Zero-Shot prompting.The reported explanation is that dialect-specific cues and semantic constraints ground token generation and reduce spurious or unaligned outputs.Source: §4.6 · p. 7

• The supplied evaluation materials compare prompting strategies through human preferences, adequacy, fluency, overall quality, MQM penalties, and hallucination ratios.Source: Figure 3 · p. 8, Table 5 · p. 8, Figure 4 · p. 8
• In-context learning performs translation without parameter updates, while fine-tuning improves faithfulness and low-resource adaptability.Source: §5 · pp. 7–8
• LLM translation research evaluates prompt templates, demonstrations, reasoning structure, multilingual coverage, document-level context, low-resource settings, and hallucination resistance.Source: §5 · pp. 8–9
• Bangla MT research includes recurrent, GRU, transformer, morphological, and dialect-specific systems, but Bangla–Sylheti remains constrained by scarce corpora, orthographic variation, and limited resources.Source: §5 · p. 9

• Sylheti-CAP consistently outperforms zero-shot, few-shot, and chain-of-thought baselines across five LLMs, reducing hallucinations, mistranslations, and awkward phrasing.Source: §6 · p. 9
• Grok 3 and LLaMA 4 achieve the highest BLEU and METEOR scores in both translation directions.Source: §6 · p. 9
• The framework integrates linguistic rules, bilingual dictionaries, and contextual fluency constraints directly into prompts.Source: §6 · p. 9

• The framework relies solely on prompting without fine-tuning, while fine-tuned word embeddings are proposed as a possible route to more stable improvement.Source: §7 · p. 10
• The bilingual dictionary contains only 2,260 word pairs, limiting dialect-specific and context-rich vocabulary coverage.Source: §7 · p. 10
• Models trained mainly on high-resource languages may have limited Sylheti exposure, constraining dialectal understanding.Source: §7 · p. 10
• Proprietary models lack publicly available training data, limiting reproducibility and transparency.Source: §7 · p. 10
• Human evaluation used a small number of native Sylheti speakers, so subjective variation and regional coverage may limit generalization.Source: §7 · p. 10

• The dictionary merges Vashantor, ONUBAD, and the Sylheti Dataset to compile Sylheti-specific vocabulary and expressions.The source corpora contain 2,125, 980, and 5,002 sentence pairs, respectively.Source: §A.1 · p. 13
• The resulting word-to-word dictionary contains 2,260 aligned pairs focused on phonological, morphological, and semantic dialect differences.Source: §A.1 · p. 13

Prompt Strategies

• Shared lexical forms between Bangla and Sylheti can support translation fluency because of common linguistic roots and phonetic overlap.Source: Table 8 · p. 14
• The zero-shot strategy directly instructs the model to translate Bangla into Sylheti without examples or prior context.Source: Table 9 · p. 14, §A.2 · p. 14
• The few-shot strategy guides translation with six Bangla–Sylheti example pairs.Examples cover greetings, questions, descriptions, and dialect-specific phrasing.Source: Table 10 · p. 14, §A.2 · p. 14
• The Chain-of-Thought strategy uses a structured, reasoning-based prompt for multi-step contextual translation.Source: Table 11 · p. 15
• Sylheti-CAP integrates explicit linguistic rules and word mappings for authentic Bangla-to-Sylheti translation.Its directives preserve SOV order, adapt pronouns, verbs, negation, and vocabulary, and target spoken rather than formal Bangla.Source: Table 12 · p. 16, §A.2 · p. 16, Table 13 · p. 17
• The context-aware prompt supplies vocabulary, imperative, passive-voice, and classifier mappings to encode Sylheti-specific grammar and usage.Examples include mappings such as পড়াশোনা→পড়ালেখা and passive constructions following an object–subject–dia–participle pattern.Source: §A.2 · p. 16