PaperlayerView exact revision on arXiv

Source-linked AI summary

From Zero to Hero: An Open LLM Ecosystem for Armenian

Erik Arakelyan, Khatun Avetisyan, Meri Davtyan, Heghine Grigoryan, Nane Khachatryan, Hayk Shahsuvaryan, Henrik Sergoyan, Vahan Martirosyan

arXiv:2609.03350v1cs.LGcs.CL

TL;DR

Armenian lacks open, reproducible training resources, including training-scale math and science solutions. The paper releases validated datasets and adapts Gemma-4-E4B with a documented recipe; the resulting model outperforms open Armenian models and its base, while verified STEM data reverses the knowledge loss from news-only adaptation.

  • Problem

    Armenian has no open training ecosystem with reproducible data and recipes, and no training-scale Armenian math or science text with worked solutions.

    Source: §1 · p. 1

  • Method

    The paper releases validated Armenian datasets and continues pretraining Gemma-4-E4B using a documented recipe that includes verified translated STEM data.

    Source: §1 · pp. 1–2

  • Results

    arm-gemma-e4b outperforms every open Armenian model and its unadapted base, while verified translated STEM data reverses the knowledge loss from news-only continued pretraining.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Per-item provenance and licenses let the released ecosystem grow in place, and the pipeline uses no Armenian-specific machinery.

    Source: §7 · p. 10

    ArmWeb is concentrated in the news domain, and evaluation is MCQA-heavy with no generation-quality or safety assessment.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Pretraining data for Armenian, a morphologically rich and low-resource language, is scarce, and no open Armenian LLM has been released with the data and recipe needed to reproduce it. To address this gap, we curate and release two datasets. ArmWeb is an extensively validated corpus of 4.37M Armenian news documents. ArmSTEM is a parallel English-Armenian collection of 373K math and science problems with step-by-step solutions, translated into Armenian and verified through both answer-preserving LLM judgment and human evaluation. Continued pretraining of Gemma-4-E4B on these datasets yields arm-gemma-e4b, which outperforms every existing open Armenian model as well as its unadapted base, and is the first open Armenian LLM with complete training data and recipe. Our ablations show that news-only continued pretraining improves fluency while eroding knowledge, a pattern we also observe in existing Armenian models, and that a small share of verified translated STEM data reverses the loss. We further find that the largest public Armenian corpora overlap web-derived evaluation panels heavily, including a train/test self-overlap inside FineWeb-2. We openly release all data, models, and code.

1 Introduction

Armenian lacks the curated training data, reproducible open models, and Armenian-scale math and science solutions available in more established language ecosystems. The paper releases an auditable data-and-training pipeline and reports that verified translated STEM data can reverse knowledge loss from news-only continued pretraining.

1 Introduction

  • The paper addresses an Armenian ecosystem gap by releasing curated datasets, an adapted model, evaluations, and a documented, auditable training pipeline.Existing open Armenian models lack training data and recipes, while Armenian training-scale math and science text with worked solutions is absent.Source: §1 · p. 1
  • ArmWeb provides 4.37M Armenian news documents, built with a documented curation pipeline and per-document provenance.Its quality checks include a 410M ablation grid, a repetition study, and a scaling ladder that predicts held-out 1.3B-model loss within 0.47%.Source: §1 · pp. 1–2
  • ArmSTEM releases 373K translated math and science problems with step-by-step solutions, verified through answer checks and human evaluation.Two native speakers independently rated 299 of 300 sampled problems as valid, with perfect inter-annotator agreement.Source: §1 · p. 2
  • arm-gemma-e4b outperforms every open Armenian model and its unadapted base on the six-task Armenian likelihood suite.Figure 1 summarizes mean accuracy across the suite.Source: §1 · p. 2, Figure 1 · p. 2
  • News-only continued pretraining trades knowledge for fluency, while verified translated STEM data reverses the loss and ends 2.2pp above the base while retaining fluency gains.The news-only loss reaches up to −21.2pp on Belebele.Source: §1 · p. 2
  • The three largest public Armenian crawl slices overlap evaluation sets by 7.9–17.4%, including FineWeb-2 Armenian test data in its training split.Without decontamination, perplexity-style evaluation systematically overstates crawl-trained Armenian models' performance.Source: §1 · p. 2

2 Methodology

The methodology combines a curated Armenian news corpus with verified translated STEM data, then evaluates corpus quality, contamination, and scaling. ArmSTEM is translated with protected notation, checked for answer preservation, and decontaminated on both language sides.

2.1 The ArmWeb news corpus

  • ArmWeb uses a 15-year single-operator Armenian news crawl, minimal text cleaning, provenance metadata, and globally applied exact and MinHash deduplication.The signature view applies NFKC normalization, Armenian ligature folding, punctuation stripping, and digit zeroing; the released text remains otherwise verbatim.Source: §2.1 · pp. 2–3
  • Public Armenian crawl slices overlap evaluation sets at 7.9–17.4% of documents, inflating crawl-trained perplexity evaluation and including FineWeb-2 train/test leakage.ArmWeb’s own bpb comparisons are fair because its evaluation panels were decontaminated.Source: §2.1 · pp. 3–4
  • The ArmWeb–CulturaX union leads the mean 410M comparison at 0.532 bpb versus 0.545 for the best single corpus, and its ranking transfers to 1.3B.On held-out news, ArmWeb beats crawl slices by roughly 10%; curated and crawled Armenian data are complements, not substitutes.Source: §2.1 · p. 4
  • Repeated ArmWeb training remains beneficial through eight epochs at both tested scales, with diminishing marginal gains and no saturation cliff.The 410M four-epoch run independently replicates the ablation-grid measurement: 0.559 mean bpb versus 0.561.Source: §2.1 · p. 4, Table 4 · p. 5
  • A three-parameter power law fits four union-recipe rungs within 0.0015 bpb and predicts the independently trained 1.3B model at 0.4675 versus 0.4697 measured.That is a 0.47% extrapolation error; the larger confirmation preserves the 410M corpus ranking.Source: §2.1 · pp. 4–5

2.2 The ArmSTEM translated STEM corpus

  • ArmSTEM translates verified English math and science sources into Armenian, protecting numbers and LaTeX with placeholders before translation and restoring them afterward.The sources were selected for machine-checkable answers, permissive licenses, and freedom from benchmark provenance.Source: §2.2 · p. 5
  • ArmSTEM verification checks placeholder integrity, Armenian language, and blind re-solving against the gold answer, with human reviewers rating 299 of 300 sampled problems valid.After repairs, overall acceptance is 96.6%; two benchmark-side scans reject real English-origin and Armenian-target collisions.Source: §2.2 · p. 6
  • ArmSTEM releases 372,907 English–Armenian pairs, totaling about 311M Armenian and 124M parallel English Gemma tokens.The harder competition-math tranche is in preparation.Source: §2.2 · p. 6, Table 5 · p. 6

3 Adapting Gemma-4 to Armenian

The adaptation method continues pretraining Gemma-4-E4B with a mixed Armenian, English-replay, and code stream, while testing tokenizer extension and contrasting matched news-only and STEM runs.

3 Adapting Gemma-4 to Armenian

  • At the 2B-token ablation budget, mean-initialized Armenian tokenizer extension lands 40–45% above the unadapted base in bits-per-byte, so the recipe retains the stock tokenizer.The ablation uses the smaller E2B; stronger embedding initializers and larger-scale extension remain untested.Source: §3 · p. 6, Table 6 · p. 7
  • The 10B-token recipe uses 69% ArmWeb, 4% Armenian ArmSTEM, 2% English ArmSTEM, 20% English web replay, and 5% code.The released STEM-CPT uses the gentler 3×10−5 learning rate; the 10−4 run is the more aggressive alternative.Source: §3 · pp. 6–7
  • Matched STEM-CPT and News-CPT runs share trainer, seed, budget, replay, and code streams, differing only by swapping six percentage points of ArmWeb for ArmSTEM.The ArmSTEM subset in these runs contains 109,885 items, balanced across math and science.Source: §3 · p. 7

4 Experimental setup

The experiments assess Armenian language knowledge with likelihood and generative suites, compare adapted runs with open-model and base-model baselines, and report single-run task results.

4 Experimental setup

  • Evaluation uses a six-task Armenian likelihood suite scored by zero-shot log-likelihood accuracy and a 24-task generative ArmBench-LLM suite.The main-text generative results focus on tasks measuring knowledge and language competence for base models.Source: §4 · p. 7
  • Baselines include unadapted Gemma-4-E4B, three existing open Armenian models, and Gemma-2-9B as HyGPT’s base model.The setup uses the latter comparison to measure that model’s adaptation delta.Source: §4 · pp. 7–8
  • Table 7 reports likelihood-suite accuracy for five adaptation runs, with one training run per column and per-task 95% binomial halfwidths.The bold learning rate marks released arm-gemma-e4b; “full” denotes STEM-CPT-full.Source: Table 7 · p. 8

5 Results

arm-gemma-e4b leads open Armenian models on the likelihood suite and improves over its unadapted base. News-only continued pretraining erodes knowledge, while verified translated STEM data reverses that loss and gains transfer to generative evaluation.

Results

  • The suite mean is 0.50 for arm-gemma-e4b, versus 0.48 for unadapted Gemma-4-E4B and 0.35–0.47 for existing open Armenian models.It also leads on each of the three knowledge tasks.Source: §5 · p. 8, Table 8 · p. 9
  • News-CPT at 10−4 loses 21.2 Belebele points and 7.1 m-MMLU-hy points, with degradation already complete by mid-training.Checkpoint selection could not recover the loss.Source: §5 · p. 8
  • At 3×10−5, the model retains the full fluency gain, recovers two-thirds of the Belebele loss, and surpasses the base on INCLUDE.The passage attributes INCLUDE's gain to Armenian-context questions benefiting from news knowledge.Source: §5 · p. 8
  • Swapping 6% of news for verified translated STEM data raises the mean 2.2 points above base and Belebele to 0.716, 9.7 points above base.The gains also extend to natively authored INCLUDE and ArmBench evaluations, arguing against translationese as their explanation.Source: §5 · pp. 8–9
  • At the aggressive 10−4 rate, STEM-CPT recovers 18.3 Belebele points over News-CPT, but ending above base also required the gentler schedule.At this dose, STEM-CPT-full matches arm-gemma-e4b within confidence intervals on the likelihood suite, with means of 0.494 and 0.500.Source: §5 · p. 8
  • On generative accuracy tasks, arm-gemma-e4b averages 0.62 versus 0.57 for ArmenianGPT-1.0-3B and improves over its base nearly everywhere.The comparison aggregates small per-task samples; HyGPT leads exam literature, while the base leads POS tagging.Source: §5 · p. 9, Table 9 · p. 9
  • HyGPT-10b gains 2.5pp on MultiBLiMP but loses 17.1 Belebele points and 4.6 suite-mean points; its undisclosed recipe prevents isolating the cause.ArmenianGPT-1.0-3B, the strongest existing open model, still trails unadapted Gemma-4-E4B on the suite mean.Source: §5 · p. 9
  • Reported negative results include a regression on POS tagging and flat exam mathematics.Source: §5 · p. 9

6 Related work

Related work situates the paper among language-specific open ecosystems and corpus-hygiene research. It frames verified translated knowledge data as a distinct lever against forgetting during continued pretraining.

Language-specific ecosystems

  • Prior language-specific ecosystems release corpora, adapted models, evaluations, and recipes, while Armenian resources lacked comparable curation and training-data transparency.Earlier Armenian models were released without training data, and available text was largely multilingual crawl slices.Source: §6 · pp. 9–10

Corpus curation and hygiene

Continued pretraining and forgetting

  • Prior continued-pretraining work uses replay and learning-rate choices to mitigate catastrophic forgetting; this paper proposes verified translated knowledge data as a lever that reverses it.The related-work discussion distinguishes reversal from replay's role in slowing forgetting.Source: §6 · p. 10

Translated training data

  • Prior work studies translated reasoning, multilingual instruction data, and parallel-data pretraining; this paper applies translated data in continued pretraining and uses blind re-solving to check translations.The cited translation-verification approaches include LLM judging (Kocmi and Federmann, 2023) and formula-integrity checks (Chen et al., 2024).Source: §6 · p. 10

7 Conclusion

The paper releases validated Armenian datasets and a strong open model with its complete corpus and recipe. It concludes that verified translated STEM data turns the fluency–knowledge trade-off into a net gain within this adaptation setting.

Conclusion

  • The authors report that the released datasets produce the strongest open Armenian model, published with its complete corpus and recipe.Source: §7 · p. 10
  • They conclude that verified translated STEM data turns low-resource adaptation's fluency-for-knowledge trade into a net gain.Source: §7 · p. 10
  • The pipeline uses no Armenian-specific machinery and should transfer to other low-resource languages; per-item provenance and licenses support ecosystem growth.A competition-math tranche of ArmSTEM and an instruction-tuned arm-gemma-e4b are identified as next releases.Source: §7 · p. 10

Limitations

The authors note limits in corpus diversity, attribution of STEM-swap effects, and evaluation scope. Some benchmark results also remain bounded by small test sets and incomplete contamination audits.

Limitations

  • ArmWeb is concentrated in news, limiting register diversity.Source: Limitations · p. 11
  • At 3.3B Gemma tokens, ArmWeb is about a third of HyGPT's undisclosed corpus only if that count uses comparable Gemma-2 token units.Source: Limitations · p. 11
  • The 6-point mixture swap cannot separate STEM content, QA format, and verification; a format-matched comparison remains future work.Source: Limitations · p. 11
  • Evaluation is MCQA-heavy and lacks generation-quality and safety assessment.Source: Limitations · p. 11
  • The perfect scientific-MCQA score comes from a 50-item set, and a paraphrase-level audit remains future work.Source: Limitations · p. 11

Ethics Statement

The statement distinguishes rights in ArmWeb’s compilation from copyright in the underlying articles and documents personal-data findings. It also limits the corpora and model to research use, with further alignment needed before end-user deployment.

Ethics Statement

  • ODC-BY 1.0 covers ArmWeb’s compilation, while copyright in the articles remains with their publishers and rights holders can request takedowns.Per-document URL and source fields preserve attribution.Source: Ethics Statement · p. 11
  • A scan found 942 email addresses in 782 documents and phone-number-like strings in 8,971 documents, mostly newsroom and institutional contact lines.Author bylines are removed, and the scan report ships with the corpus.Source: Ethics Statement · p. 11
  • The corpora and model are research artifacts; the model has no safety tuning and should not reach end users without further alignment.Source: Ethics Statement · p. 11

A Negative results

The results include a 1.3B scale confirmation, regressions on some tasks after continued pretraining, and low scores on instruction-dependent ArmBench tasks for base-style models.

A Negative results

  • POS tagging falls from 0.18 to 0.01, while exam mathematics stays at 1.75 points for the released model versus 2.75 for STEM-CPT-full.The authors suggest data diversity, rather than difficulty, may be the binding factor.Source: A Negative results · p. 16
  • Instruction-dependent judged generation and BLEU-scored QA remain low for all base-style models, including arm-gemma-e4b.The authors attribute this to the tasks’ formatting demands and defer them to an instruction-tuned variant.Source: A Negative results · p. 16, B Full ArmBench results · p. 16

B Full ArmBench results

  • Table 10 reports scores for every ArmBench task across the base model, News-CPT, arm-gemma-e4b, and STEM-CPT-full.The table includes instruction-dependent tasks on which base-style models score low by construction.Source: B Full ArmBench results · p. 16

C Scale confirmation at 1.3B

  • 0.035 mean bpb is the union’s lead over CulturaX at 1.3B, up from 0.013; its ranking transfers and it overtakes pure ArmWeb on held-out news.Each recipe used 27.6B SP tokens; lower bpb is better.Source: C Scale confirmation at 1.3B · p. 16

D Contamination audit of the perfect MCQA score

The audit checks whether ArmSTEM training data overlaps with the perfect scientific-MCQA benchmark score and finds no shared 8-grams among its 50 items.

D Contamination audit of the perfect MCQA score

E Cross-implementation deduplication check

The deduplication audit compares two implementations and a Russian sister collection, then describes ArmWeb’s composition and reproducibility details.

E Cross-implementation deduplication check

  • On a 100K-document sample, document-level decisions agree 99.2% of the time, and NeMo-Curator reproduces 71.9% of duplicate clusters exactly.Source: E Cross-implementation deduplication… · p. 17
  • The tools’ disagreements cluster near the 0.72 LSH threshold; the Russian collection removes 0.9% of documents versus 22.4% for Armenian.The authors conclude the implementations are equivalent away from the threshold boundary.Source: E Cross-implementation deduplication… · p. 17

F ArmWeb datasheet summary

  • 19 outlets supply ArmWeb; its five largest contribute 2.71M of 4.37M documents, with publication dates spanning 1998–2026.The corpus contains 3.3B Gemma tokens and includes outlet, topic, URL, publication-date, and crawl-date metadata.Source: F ArmWeb datasheet summary · p. 18

G Reproducibility details

  • CPT runs use 4,770 steps and about 10B tokens, taking about 10 hours on 128 H100s; pipeline code, reports, and job scripts are released.The runs use packed sequences of length 4096, and the 410M/1.3B grid, repetition, and ladder models use Megatron-LM.Source: G Reproducibility details · pp. 18–19

Related papers

Loading 2609.03350v1…