A Benchmark for Learning to Translate a New Language from One Grammar Book

Real, unseen domain

• MTOB tests adaptation on Kalamang, a language with virtually no web presence, making the translation task unseen while retaining real-world complexity.The benchmark targets a setting where web-scale pretraining offers little or no direct Kalamang data.Source: §1 · pp. 1–2

Learning from explanations

• The benchmark teaches translation through several hundred pages of curated grammar explanations rather than large mined in-domain corpora, resembling L2 learning.The materials contain fewer than 250K English tokens and fewer than 25K Kalamang tokens, and the setup permits in-context learning, retrieval, or finetuning.Source: §1 · p. 2

Human scale

• MTOB enables a data-matched human comparison because its reference materials are small enough for a person to read, while acknowledging that one book cannot support general language mastery.This establishes a lower bound on performance achievable from the same materials, rather than native-speaker application readiness.Source: §1 · p. 2

Data curation

• Documentation-based resources could be curated by small groups of language experts, unlike datasets requiring the diversity and scale expected by current machine-learning methods.This framing is presented as a qualitatively different data source for underserved communities.Source: §1 · p. 2

Multipurpose datasets

• The same learning materials may serve humans and language revitalization, while proprietary community-controlled data could support adaptation without entering pretraining mixtures.These motivations connect the benchmark to multipurpose datasets and data sovereignty.Source: §1 · p. 2

Scope and limitations

• The authors emphasize that MTOB and its methods are speculative, not an immediate complete solution, and may not suit all languages or communities.Community consultation included renewed permission, but the authors stress that useful applications could require years and additional data collection.Source: §1 · pp. 2–3

Linguistic setting

• Kalamang belongs to the Greater West Bomberai grouping but is not closely related to other family members and has features associated with both Austronesian and Papuan languages.Papuan Malay functions as the regional vehicular language.Source: §2 · p. 3

Writing and use

• Kalamang has no formal written tradition or administrative use, though literate speakers write it with Indonesian orthography and limited social-media use is emerging.Recent written use is mainly by non-fluent speakers posting short phrases within Papuan Malay.Source: §2 · p. 3

Grammar

• The language was only studied in detail recently and is analyzed as SOV with nominative-accusative alignment and mixed particle-affix morphology.These grammatical properties are reported from Visser (2022).Source: §2 · p. 4

Suitability for MTOB

• Kalamang suits MTOB because it has little web content, a small speaker population, recent documentation, and Latin-based orthography that avoids script-tokenization complications.The authors also report that community attitudes do not make wider knowledge of the language especially problematic.Source: §2 · p. 4

3 THE MTOB BENCHMARK

• MTOB combines the grammar book, bilingual word list, and parallel Kalamang-English corpus, with the test set sampled from the parallel sentences.All resources derive from Visser (2022) documentation based on fieldwork.Source: §3 · p. 4

3.1 GRAMMAR BOOK

• The 573-page grammar book describes Kalamang phonology, morphology, and syntax using more than 1,000 elicited and naturalistic examples.It is distributed in original LaTeX and preprocessed plaintext formats.Source: §3.1 · p. 4

3.2 BILINGUAL WORD LIST

• The bilingual word list contains 2,531 Kalamang words paired with part-of-speech tags and English descriptions.It is published alongside Visser (2022) through Dictionaria.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 PARALLEL KALAMANG-ENGLISH CORPUS

• After removing grammar-book examples, the parallel corpus provides 500 sentences split into 400 training and 100 test examples.Even including grammar-book sentences, the corpus has at most approximately 2,000 parallel pairs, far below typical machine-translation scales.Source: §3.3 · p. 5

3.4 EVALUATION TASKS

• The benchmark evaluates sentence-level Kalamang-to-English and English-to-Kalamang translation with chrF as the canonical metric.Models receive complete access to the training materials for both directions.Source: §3.4 · p. 5

4.1.1 EXPERIMENTAL SETUP: MODEL

• The study evaluates 12 models, including public pretrained and grammar-book-finetuned models plus API-based systems such as GPT-4 and Claude 2.Chatbot flavor text and refusals are manually handled before automatic scoring.Source: §4.1.1 · p. 5

4.1.2 EXPERIMENTAL SETUP: CONTEXT

• Context conditions range from no reference material to word-list entries, paired sentences, retrieved grammar excerpts, and manually curated medium or long grammar-book contexts.The compared combinations include -, W, S, Ge, Gs, W + S, W + Gs, W + S + Gs, and additional long-context settings for Claude 2.Source: §4.1.2 · p. 6

4.2 HUMAN BASELINE

• The human baseline was one linguistically experienced author who read the grammar twice over approximately 10 hours before translating over several weeks.The author could consult the training materials during translation, producing a data-matched rather than native-speaker baseline.Source: §4.2 · pp. 6–7
• Across kgv→eng and eng→kgv translation, Claude 2 achieves the best results among the tested settings when provided W + S + Gl, although human performance considerably exceeds all model baselines.Source: Figure 3 · p. 7

Overall results

• 44.7 chrF on kgv→eng and 45.8 chrF on eng→kgv remain below the human baseline of 51.6 and 57.0 chrF, respectively.Claude 2 with W + S + Gl is the top baseline, evaluated on disjoint halves of each test direction.Source: §5 · p. 7

Effect of context

• Without retrieved context, models are unable to translate reliably, often hallucinating for kgv→eng or producing Indonesian for eng→kgv.Non-zero chrF scores can arise from copying proper nouns or metric properties rather than successful translation.Source: §5 · p. 7
• Retrieved sentences help most individually, followed by word-list entries and grammar-book excerpts, while combined context usually improves results further.Lexically retrieved grammar excerpts often behave like additional glossed examples because they do not necessarily contain useful grammatical explanations.Source: §5 · pp. 7–8
• Long grammar-book contexts produce a qualitative leap in English-to-Kalamang translation, yielding moderately complex grammatical outputs in the best setting.These outputs remain inconsistent, but their errors are more interpretable and human-like than the ungrammatical gibberish in shorter-context settings.Source: §5 · p. 8

Effect of finetuning

• Finetuning directly on grammar-book text harmed results, although different parameters, data preparation, or larger models could change this outcome.The experiments only finetuned smaller models up to 13B parameters, so the effect of finetuning at larger scales remains uncertain.Source: §5 · pp. 8–9

Comparison to human baseline

• The human baseline outperforms all model baselines by 6.9 chrF on kgv→eng and 11.2 chrF on eng→kgv.Human translations are consistently grammatical in English-to-Kalamang, whereas model outputs are erratic and sometimes hallucinate irrelevant content.Source: §5 · p. 9

• MTOB is instrumentally useful for research, not an immediately useful tool or a general solution to language endangerment and marginalization.The authors caution that strong benchmark performance does not establish immediate social utility or broad generality.Source: §6 · pp. 9–10
• MTOB’s test domain may overestimate generalization because test sentences and training materials derive from the same causally structured corpus.Indirect overlap remains through shared vocabulary, grammatical analysis, and example selection, while independently sampled sentences could contain undocumented structures.Source: §6 · p. 9
• The benchmark measures only sentence-level text translation for one language, leaving discourse phenomena, speech tasks, and cross-linguistic diversity underrepresented.Kalamang is primarily oral, and the authors note that methods must eventually be evaluated across typologically diverse languages.Source: §6 · pp. 9–10
• English–Kalamang translation is not necessarily the most socially useful task because Kalamang speakers also communicate in Papuan Malay.The authors suggest vehicular-language translation may serve broader communication needs.Source: §6 · p. 9
• The task framing is incomplete: learning a language generally from a book is not possible, and more complete systems would combine synthetic material with modest real in-domain data.Small datasets also amplify individual errors, making community participation and thorough application vetting especially important.Source: §6 · p. 10
• Language technology may benefit well-resourced entities disproportionately, impose culturally mismatched L2 goals, or encourage standardization rather than preserve linguistic diversity.The authors therefore stress that each community should decide whether and how technology supports its own revitalization goals.Source: §6 · p. 10

7.1 LARGE LANGUAGE MODELS

• MTOB targets a tension in LLM research: apparent adaptation may reflect genuine general capabilities or exploitation of extensive pretraining data.The benchmark is intended to probe this tension using a language with virtually no web presence.Source: §7.1 · p. 10

7.2 LLM BENCHMARKS

• Unlike prior unseen-task benchmarks centered on synthetic or counterfactual fluid intelligence, MTOB evaluates acquisition of language knowledge and procedures from explanations.It is characterized as closer to crystallized intelligence than abstract reasoning.Source: §7.2 · p. 11, §7.4 · p. 11
• MTOB differs from BabyLM by asking a pretrained model to learn a web-held-out L2 from reference materials, enabling a human baseline under the same data conditions.The task uses fewer than 250K tokens in English and 25K in Kalamang, rather than child-scale L1 pretraining.Source: §7.2 · p. 11, §7.4 · p. 11

7.3 MULTILINGUALITY

• Existing multilingual translation methods typically mine parallel or monolingual data for pretraining, whereas MTOB uses lengthy field-linguistics grammars as dynamic instructions for translation.The authors describe this as a soft form of rule-based translation in which an LLM interprets grammar rules dynamically.Source: §7.3 · p. 11

7.4 NLP FOR HISTORICALLY DISADVANTAGED COMMUNITIES

• MTOB applies community-centered NLP principles by adapting pretrained models to languages with little web data and potentially allowing individual communities to use proprietary documentation.The framing reflects concerns about participatory design, data sovereignty, and the possibility that fieldwork documentation can support supposedly zero-resource languages.Source: §7.4 · pp. 11–12

8 CONCLUSION

• Increasing LLM quality and context-window size improves translation, but current baselines remain below human performance on the one-book Kalamang task.The authors present MTOB as evidence of unrealized value in pretrained data and of promise for in-context learning, without claiming definitive general intelligence.Source: §8 · p. 12

A GRAMMAR BOOK PREPROCESSING

• The grammar book is converted from LaTeX to plaintext through automated cleanup and manual correction, reducing token length from over 500K to roughly 217K–241K tokens.The processed text is released with the code and models for retrieval and finetuning.Source: A GRAMMAR BOOK PREPROCESSING · p. 20
• Future experiments could use the full LaTeX source or rendered PDF once models support 500K-plus-token or long multimodal inputs.This proposal extends access to the book beyond the current plaintext preprocessing choice.Source: A GRAMMAR BOOK PREPROCESSING · p. 20

B BASELINE MODELS

• The benchmark evaluates 12 GPT-style autoregressive Transformers using near-greedy sampling with temperature 0.05.The small stochastic component preserves mostly deterministic model behavior.Source: B BASELINE MODELS · p. 20
• The tested models comprise public-weights pretrained models, public-weights finetuned models, and API-only models.This grouping spans locally available and hosted LLM baselines.Source: B BASELINE MODELS · p. 20

B.1 PUBLIC-WEIGHTS PRETRAINED MODELS

• The public-weight baseline uses instruction-finetuned Llama 2 variants because the evaluation supplies translation instructions.Llama 2-7B-Nous-Hermes, 13B-Nous-Hermes, and 70B-Beluga were chosen as the highest-scoring available variants on the Open LLM Leaderboard at experiment time.Source: §B.1 · p. 20

B.2 PUBLIC-WEIGHTS FINETUNED MODELS

• Llama 2-7B-ft and Llama 2-13B-ft are fine-tuned directly on A grammar of Kalamang using LoRA for four epochs.The largest fine-tuned model is 13B because of resource constraints.Source: §B.2 · p. 20

C PROMPTS

• The prompts place the input before retrieved context so decoder-only models can attend to the sentence while reading the supporting material.The prompt then requests a direct Kalamang or English translation, with language labels reversed for the opposite direction.Source: C PROMPTS · p. 21
• A prompt switch telling models to provide their best guess solved almost all API-model refusals to translate.The revised prompt explicitly permits wrong translations and instructs models not to refuse because they do not speak Kalamang.Source: C PROMPTS · p. 21

B.3 API MODELS

• API experiments vary retrieved context, including similar word-list entries, reference sentences, and grammar-book passages.Grammar-book retrieval uses either embedding similarity or longest-common-substring distance, while longer curated book subsets can be placed directly in context.Source: C PROMPTS · pp. 21–22

D EMBEDDING-BASED RETRIEVAL

• Longest-common-substring retrieval slightly outperforms embedding retrieval because the English-trained embedder may treat Kalamang text as out-of-distribution.All main-text results use longest-common-substring retrieval, but neither retrieval method is designed to find passages describing grammatical features.Source: D EMBEDDING-BASED RETRIEVAL · p. 22

E DATASET SPLITS FOR BASELINES

• The 100-sentence test set is split into two disjoint 50-sentence halves so one human cannot accumulate knowledge across both translation directions.This avoids conflating translation-direction difficulty with differences in human proficiency.Source: E DATASET SPLITS FOR BASELINES · p. 22
• An additional 50 training sentences are held out for human examples, leaving 350 training sentences for scoring two 50-sentence test sets.The human also translates up to 25 extra unscored examples in each direction.Source: E DATASET SPLITS FOR BASELINES · pp. 22–23
• Models retrieve from 450 sentences, including the other direction’s test data, to offset the human’s access to extra monolingual test sentences.The authors acknowledge that this is not perfectly comparable and intentionally favors the model.Source: E DATASET SPLITS FOR BASELINES · p. 23
• Future evaluations are expected to use the traditional shared 400-sentence training and 100-sentence test split for both directions.That split would enable one bidirectional fine-tuned model and produce slightly more stable test metrics.Source: E DATASET SPLITS FOR BASELINES · p. 23

F QUANTITATIVE RESULTS

• 44.7 chrF on kgv→eng and 45.8 chrF on eng→kgv trail human scores of 51.6 and 57.0 chrF, respectively.The strongest baseline is Claude 2 with W + S + Gl context.Source: F QUANTITATIVE RESULTS · p. 23
• 16.3 BLEU versus 24.6 BLEU is the kgv→eng comparison, while eng→kgv reaches 6.9 BLEU versus the human’s 24.6 BLEU.Claude 2 uses W + S + Gl for kgv→eng and W + S + Gs for eng→kgv.Source: F QUANTITATIVE RESULTS · p. 23
• BLEU correlates with chrF for kgv→eng but collapses bad conditions and short sentences to zero, while missing qualitatively impressive eng→kgv outputs.Gloss-level morphemes can break word-level n-gram matches in the English-to-Kalamang direction.Source: F QUANTITATIVE RESULTS · p. 23

G QUALITATIVE RESULTS

• Qualitative comparisons cover highest-context outputs across models and Claude 2 under different retrieved-context types, with chrF and BLEU tables summarizing both directions.The analyses assess whether models execute grammar from A grammar of Kalamang, not necessarily real Kalamang usage by native speakers.Source: G QUALITATIVE RESULTS · p. 23, Table 1 · p. 24, Table 2 · p. 24
• Human translations sometimes outperform references because the references are free translations by a non-native English-speaking linguist, but humans also miss listed spelling variants or lexical senses.One human error could have been avoided by rereading the complete grammar book and word list for the particular sentence.Source: Figure 6 · p. 25, Figure 7 · p. 26, G QUALITATIVE RESULTS · pp. 26–27
• Models often hallucinate or misuse vocabulary, including Indonesian associations, irrelevant retrieved words, and incorrect senses for ambiguous entries.Examples include interpreting Tali as Indonesian “rope,” using irrelevant fish vocabulary, and selecting the wrong meaning of sair.Source: G QUALITATIVE RESULTS · pp. 25–26
• Kalamang ambiguity requires contextual or sociocultural reasoning, such as interpreting tara through community teknonyms and gendered household roles.The analysis treats these interpretations as guesses and notes that strategies explaining ambiguity may be preferable to avoid bias.Source: G QUALITATIVE RESULTS · p. 25
• Grammar-sensitive errors include incorrect demonstrative case, possession, causative structure, locative marking, and progressive-aspect interpretation.The examples show that relevant words alone do not ensure faithful grammatical composition.Source: G QUALITATIVE RESULTS · pp. 25–27
• Claude 2 can produce basically correct content at the gloss level while missing Kalamang phonological alternations and grammatical marking.In one example, its output is nearly correct but omits the phonological change in paruo=kin →paruotkin.Source: G QUALITATIVE RESULTS · p. 27
• Additional grammar-book context can make Claude 2’s outputs more faithful but also more awkward when retrieved words alone already support a translation.Other examples show irrelevant retrieved words causing distraction and English glosses replacing unavailable Kalamang words.Source: Figure 9 · p. 28, Figure 10 · p. 29