Source-linked AI summary
Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer
Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun
TL;DR
Small bilingual models need both language-appropriate tokenization and evaluation: Arkios addresses these through a Devanagari-aware tokenizer, from-scratch training, and paired scoring protocols. It finds real but English-favoring Nepali comprehension, while tool use is conditioned on declared manifests and the model’s scope remains limited by Nepali data availability.
Problem
Existing small open language models are predominantly English-centric, and standard letter-choice evaluation can confound low-resource-language comprehension with response-format following.
Method
Arkios trains a 1.04B-parameter bilingual model from scratch on 150B tokens using a Devanagari-aware tokenizer, custom training stack, manifest-conditioned instruction tuning, and paired evaluation.
Results
0.306 Nepali and 0.387 English answer-text accuracy reveal English-favoring comprehension, whereas letter-format scores are 0.240 and 0.236 against a 0.250 chance baseline.
Takeaways & Limitations
Answer-text scoring is necessary to distinguish genuine Nepali comprehension from failure to follow the multiple-choice-letter convention.
Takeaways & Limitations
Approximately 4.18B unique Nepali tokens constrain Nepali capability, and the report does not expect English or model-size scaling alone to close the gap.
Abstract
from arXiv · showhide
We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-file C/CUDA training stack and a Devanagari-aware byte-level BPE tokenizer built for this project. On ARC-Easy and ARC-Challenge, Arkios exceeds three comparably sized open models (Pythia-1.4B, TinyLlama-1.1B, OLMo-1B) despite an order of magnitude fewer training tokens, likely aided by a match between our educational-web-text pretraining data and ARC's grade-school-science format rather than a general capability advantage. We report full evaluation results under standard protocols, including a correction to an earlier partial-sample estimate, and findings specific to evaluating small models in a low-resource language: the standard multiple-choice-letter prompt format used by common evaluation harnesses places this model at chance on Nepali reading comprehension, and simultaneously at chance on English in the same format, which would lead a naive benchmark run to conclude the model has no Nepali ability when in fact it does. Concretely, both languages score at chance in the letter-choice format (0.240 Nepali, 0.236 English, against a chance baseline of 0.250), while scoring the answer text directly reveals genuine, English-favoring comprehension (0.306 Nepali, 0.387 English). We describe a manifest-conditioned tool-use contract introduced during instruction tuning, where tool calls are permitted only when a tool manifest is declared in context and suppressed otherwise, and report where that contract holds and where it does not. We release both the base and instruction-tuned model weights under Apache-2.0. The training code and a small privately-sourced portion of the Nepali pretraining corpus are not released; everything needed to reproduce the reported numbers from the released weights is included here.
1 Introduction
Arkios is a released 1.04B-parameter bilingual English–Nepali model and evaluation study built around Devanagari-aware tokenization, from-scratch training, and low-resource-language assessment. Its results are deliberately scoped: strong ARC performance may reflect data–benchmark alignment, while Nepali capability remains behind English.
- Contributions: The report combines a from-scratch pretraining run, custom C/CUDA training stack, manifest-conditioned tool-use tuning, and evaluation of small models in low-resource languages.
- Contributions: 1.04B-parameter Arkios is a bilingual English–Nepali base model trained from scratch on 150B tokens and released with full training and evaluation configuration.
- Contributions: Arkios introduces an instruction-tuned chat model whose tool calls depend on a declared manifest, with the manifest masked from training loss.
- Evaluation: Full-test-split evaluation corrects an earlier partial-sample ARC-Challenge estimate and uses paired Nepali/English Belebele items to expose a prompt-format artifact.
- Scope: The model is released under Apache-2.0, but the report does not claim state-of-the-art general capability, a novel architecture, or a new scaling law.
- Scope: Nepali capability is measurably real but trails English, consistent with an approximately 80:1 ratio of unique English to unique Nepali pretraining tokens.
2 Tokenizer
The project uses a 65,536-token byte-level BPE tokenizer modified to treat Devanagari combining marks as part of word units. This prevents the fragmentation caused by the default Latin-oriented pretokenization rule.
- Tokenizer design: 65,536-token byte-level BPE uses a modified pretokenizer regex with [\p{L}\p{M}]+ instead of the default \p{L}+.
- Tokenizer design: The modified word class keeps Nepali vowel signs and virama attached to candidate units, closing the Devanagari fragmentation failure while leaving English pretokenization unchanged.
- Tokenizer evaluation: 1.69 tokens/word is achieved on Nepali, which the companion paper reports as competitive with or better than substantially larger frontier tokenizers on the same measure.
- Tokenizer scope: The tokenizer is frozen for the duration of the project, and the full fertility analysis is deferred to the companion paper.
3 Architecture
Arkios is a dense, decoder-only transformer using grouped-query attention, RMSNorm, SwiGLU, rotary position embeddings, and QK-normalization. Its configuration is compatible with the Qwen3 architecture family.
- Architecture: Arkios is a dense, decoder-only transformer with grouped-query attention, RMSNorm, SwiGLU, rotary position embeddings, and QK-normalization.
- Architecture: The architecture is compatible with the Qwen3 architecture family.
4 Pretraining Data
Arkios was pretrained on 150B tokens, substantially over the Chinchilla-optimal token-to-parameter ratio, while Nepali data availability remained the binding constraint. The report treats new Nepali data acquisition as the appropriate next lever rather than further model or English-token scaling.
- Pretraining mixture: 150B tokens were used for pretraining, equal to 150 tokens per parameter and well above the roughly 20-token-per-parameter Chinchilla-optimal ratio.
- Pretraining mixture: Over-training trades relatively cheap pretraining compute for a smaller model that is cheaper to serve at inference time.
- Nepali data ceiling: 4.18B unique Nepali tokens were available across practical public sources, making Nepali data availability the binding constraint on this model’s Nepali capability.
- Nepali data ceiling: The report identifies new Nepali data acquisition, rather than parameter or English-token scaling, as the correct next lever for improving Nepali capability.
- Pretraining mixture: The training mixture necessarily repeats the much smaller Nepali pools, while English web text is seen well under once.
- Validation: 0.746 English bpb and 0.314 Nepali bpb are reported for held-out diagnostics, but the authors caution that script-level byte differences prevent treating the lower Nepali value as better modeling.
5 Training Infrastructure
Arkios was pretrained with a project-built single-file C/CUDA stack, reaching a reported production run on one 8×H100 node. The training code remains proprietary, although the methodology is described for reproducibility.
- The project used a single-file C/CUDA trainer with custom kernels, checkpointing, and fused scaled-dot-product attention.Initial kernel work reached 3.1% measured MFU before profiling identified correctness and performance defects.
- The final pretraining run achieved 68,052 tokens/second on one 8×H100 SXM5 node and took approximately 79 wall-clock hours.The realized cost was approximately $2,500, including recovery from one uncorrectable HBM ECC fault.
- The C/CUDA trainer, build scripts, and cluster orchestration are proprietary and are not released with the model weights.The report states that its methodological description is sufficient to reproduce the approach, but does not include source code.
6 Post-Training
Instruction tuning progressed through successive checkpoints toward a manifest-conditioned tool-use contract. Evaluation shows reliable suppression of unauthorized calls, but calling when appropriate remains dependent on tool type, language, and request framing.
- 6.1 Iteration history: v1 called tools reliably for trained patterns but unconditionally emitted calls without available tools and sometimes invented nonexistent tool names.
- 6.1 Iteration history: v2 added Nepali safety data, fixing a Nepali refusal gap while measurably reducing tool-calling reliability across the learning-rate sweep.The report interprets this as competition between the two objectives within the same SFT mixture.
- 6.2 Manifest-conditioned tool use: The target contract permits calls only for declared, suitable tools and requires direct answers or refusal when no declared tool applies.The four conditions distinguish declared applicable tools, declared unsuitable tools, declared unnecessary tools, and no manifest.
- 6.2 Manifest-conditioned tool use: Manifests are masked from training loss, so the model conditions on them without learning to generate them.They are synthesized for several conditions and include 2–7 distractor tools to require selection rather than defaulting to the only option.
- 6.2 Manifest-conditioned tool use: The acceptance suite is a small hand-constructed test, so its per-condition pass/fail counts are not precision estimates with meaningful confidence intervals.
- 6.2 Manifest-conditioned tool use: The no-unauthorized-call half is close to fully reliable, whereas appropriate calling is reliable for arithmetic but unreliable for unit conversion.The failures are concentrated in identifiable categories, including a further English–Nepali gap on one condition.
- 6.2 Manifest-conditioned tool use: Without a manifest, the model declines bare arithmetic but may attempt incorrect prose arithmetic when the same computation is phrased as a word problem.One observed rectangle-area answer was 65.25 m2 instead of the true 91 m2.
7 Evaluation
Arkios is evaluated on full-split English benchmarks, paired-language Nepali comprehension, translation target-script selection, and efficiency-oriented scoring procedures. Results show strong but domain-favorable ARC performance, a multiple-choice format artifact, and improved yet incomplete target-language generation in the chat model.
- 7.1 English: 0.416 acc_norm at 10-shot is Arkios’s corrected full-split ARC-Challenge result, replacing an earlier 0.434 estimate from 500 items.The earlier figure used a 500-item subsample and an unrecorded normalization convention; the discrepancy is consistent with sampling variance.
- 7.1 English: Arkios’s ARC-Easy and ARC-Challenge scores exceed Pythia-1.4B, TinyLlama-1.1B, and OLMo-1B despite using an order of magnitude fewer training tokens.The reported ARC gap is attributed partly to a compositional match between educational-web pretraining data and ARC’s grade-school science format, not a general capability advantage.
- 7.2 Nepali: 0.240 Nepali and 0.236 English in letter-choice scoring are both near the 0.250 chance baseline, while direct answer-text scoring reaches 0.306 and 0.387 respectively.The paired-language control indicates that the failure concerns following the A/B/C/D response convention rather than Nepali reading comprehension alone.
- 7.3 Translation: 8% target-script rate for base-model English-to-Nepali generation rises to 58% in the instruction-tuned checkpoint, while the paired direction reaches 81%.Each checkpoint is evaluated in its native prompting format; the remaining 58% English-to-Nepali rate remains a limitation.
- 7 Evaluation: Approximately 40× less dominant scoring memory is allocated by applying the language-model head only at continuation-scoring positions instead of every batch position.This procedure gathers transformer-body hidden states for the batch and avoids materializing full-vocabulary logits at all positions.
8 Limitations •
The report limits its claims around data availability, domain-favorable benchmarking, evaluation artifacts, generation reliability, training scope, context length, and advice behavior. These boundaries constrain how Arkios’s results should be interpreted and deployed.
- Data availability: Approximately 4.18B unique Nepali tokens versus a much larger English pool makes data availability the binding constraint on Nepali capability.The report states that increasing model scale without increasing Nepali data would improve English substantially more than Nepali.
- Benchmark scope: ARC performance is likely domain-favorable and should not be generalized into a broad capability claim.Other English tasks in the same evaluation pass are described as ordinary for the model’s scale and training-token budget.
- Evaluation: The standard multiple-choice-letter format understates Nepali ability, so third-party benchmarks should be interpreted alongside this caveat.The artifact arises because the model is also at chance in the identical English format.
- Translation: English-to-Nepali generation reaches only a 58% target-script rate even in the instruction-tuned checkpoint.This indicates incomplete reliability in requested target-language selection.
- Training scope: The released chat model is SFT-only, with no RLHF, DPO, or other preference-optimization stage.This defines the instruction-tuning scope of the released checkpoint.
- Context: The model uses a fixed 4096-token context window established during pretraining.The limitation is architectural and fixed for the released model.
- Deployment: The chat model can regress relative to the base model on some non-tool interpersonal or practical-advice questions.The report recommends reviewing advice-shaped outputs before unsupervised deployment.
9 Release and Availability
Arkios releases both pretrained and instruction-tuned checkpoints, plus the tokenizer, under Apache-2.0. The training code and a small privately sourced Nepali corpus portion remain unavailable, but reported numbers are reproducible from the released weights.
- Availability: Both the pretrained base model and instruction-tuned chat model are released as HuggingFace repositories under Apache-2.0.The standalone tokenizer is also released.
- Unavailable materials: The C/CUDA training code, cluster orchestration scripts, and privately sourced Nepali pretraining data are not released.The report provides methodology but not source code for the training system.
- Reproducibility: Every reported number is reproducible from the released weights using the evaluation methodology described in the report.Reproduction does not require release of the proprietary training implementation.