PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

Ayoub Kirouane, Christos Petrocheilos

arXiv:2608.17744v1cs.CLcs.LGcs.ROstat.ML

TL;DR

Accuracy benchmarks provide limited evidence about what low-resource-language reasoning fine-tuning changes, because they omit language, efficiency, and auditability dimensions. This paper evaluates those behavioural dimensions across sparse mixture-of-experts models and finds that accuracy remains nearly unchanged while supervised fine-tuning substantially changes reasoning language and related behaviours, with verifiable-reward reinforcement learning correcting key format defects.

  • Problem

    Accuracy-only evaluation does not reveal the language, token efficiency, difficulty sensitivity, or trade-offs of low-resource-language reasoning fine-tunes.

  • Method

    The study fixes active parameter budgets across three sparse mixture-of-experts families, defines six behaviourally gated evaluation dimensions, measures noise and contamination, and pre-registers verifiable-reward reinforcement-learning tests.

  • Results

    Across evaluations, accuracy was nearly unchanged while behavioural properties changed substantially; the best fine-tuned arm scored 76.5 versus 77.2 for the base, and seed variation reached 7.7 points.

  • Takeaways & Limitations

    For low-resource-language reasoning fine-tunes, behavioural dimensions are more informative than accuracy for measuring the properties the fine-tune was intended to change.

  • Takeaways & Limitations

    The transferability of this recipe to truly low-resource languages is untested, and most fidelity measures are automatic rather than human assessments of fluency, terminology, or followability.

Abstract

from arXiv · show

Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource language. On accuracy benchmarks almost nothing happens, and the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the language of the question on ~98% of items, one family at 3x fewer tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them.

1. Introduction: one number, six dimensions · 2. Setup

The paper argues that accuracy is an inadequate, noise-dominated summary of low-resource reasoning fine-tuning, so it measures six behavioural dimensions instead. It evaluates reproducible LoRA adaptations of similarly served sparse MoE models on Greek corpora and multiple distinct instruments.

  • 2. Setup: 3.6–4.0B active parameters per token were held fixed across three sparse MoE families from Alibaba, OpenAI and NVIDIA to separate recipe effects from architectural ones.The models occupy similar serving budgets, making active parameters the meaningful deployment axis.
  • 1. Introduction: one number, six dimensions: 7.7 points separated random-seed variation from every measured data or recipe effect, while the best fine-tuned arm scored 76.5 against the base’s 77.2.Both scores come from the 1,000-item three-axis Greek probe in think mode.
  • 2. Setup: MoE results do not establish dense-model behaviour, while LoRA uses r=32, α=64, one epoch and effective batch 32 on a single DGX B200 node.The setup constrains generalization to sparse MoE architectures and aims for one-node reproducibility.
  • 2. Setup: 118,092 Greek rows were split into 59,107 reasoning examples with explicit traces and 58,985 direct instruction examples, including 9% English replay.The direct half preserves non-reasoning ability, but the paper reports that omitting it collapses the reasoning switch and that it costs most accuracy and format compliance.
  • 2. Setup: Regenerated Greek traces scored S = 0.68 with 97% above 0.5, versus translated traces at S = 0.27 with 0% above 0.5.The structure score prioritizes backtracking, verification, flowing prose and length, and serves as a corpus diagnostic rather than a model metric.
  • 2. Setup: 5,156 Greek benchmark items cover mathematics, commonsense and logic, with greedy decoding and a requested final line scored alongside fallback%.The evaluation separates reasoning and direct modes and includes English controls, sentence-level switching, and a Greek NLU benchmark.
  • 1. Introduction: one number, six dimensions: Six dimensions—correctness, language fidelity, reasoning budget, termination, reasoning steps and budget overrun—replace accuracy as the sole summary.The paper argues these behavioural dimensions measure what the fine-tune was actually for and are stable rather than seed-sensitive.

3. Metrics

The paper evaluates reasoning through six dimensions rather than accuracy alone, using objective measures for correctness, language fidelity, budget, termination, steps, and budget overrun. Diagnostics separately track parsing failures and switch integrity, while controls address class-prior and tokenization confounds.

  • Metric framework: Six dimensions separate correctness from language fidelity, reasoning budget, termination, reasoning steps, and budget overrun.Accuracy is explicitly treated as one dimension, not a summary of all six.
  • M1: Correctness: Logic uses macro-recall over three classes, because a majority-class answerer scores 67% raw but 33% macro.Per-class recalls are required to interpret apparent changes: one seed moved Sosto recall from 84.7 to 17.3 (−67 pp) without changing other classes.
  • M2: Language fidelity: Language fidelity reports median Greek character share g and the fraction of traces with g ≥0.9 after code and LATEX are stripped.This prevents technical Latin characters from distorting the language measure.
  • M3: Reasoning budget: Greek costs 2.3–2.5× English tokens per word, so budget metrics use words rather than tokens to keep cross-language comparisons fair.Measured fertility is 2.32, 2.39, and 2.51× across the three families; a uniform –max-new budget therefore gives Greek a roughly 2.4× tighter ceiling.
  • M4–M5: Process measures: Termination measures rows reaching the generation cap without a final answer, while reasoning steps count intermediate conclusions rather than sentences.Sentence count is rejected because it is w/12 in disguise.
  • M6 and diagnostics: Fallback% measures M1 parse failures and switch integrity measures empty traces when think mode was requested, serving as diagnostics rather than quality scores.Budget overrun is model-independent because its ingredients are computed across all arms.

4. The noise floor

Seed-to-seed training variance exceeds the effects attributed to data and recipe changes, making single-run accuracy deltas unreliable. Yet language and trace-budget behaviours remain stable across seeds, enabling behavioural analysis despite the accuracy noise floor.

  • Seed control: Seed variation is the null control: identical corpus, hyperparameters, recipe, steps, benchmark, and greedy decoding isolate training variance.Only LoRA initialization and data-shuffling order vary; the resulting distribution is the baseline against which every intervention must be read.
  • Observed variance: One run collapsed to 68.7 while two landed near 76, indicating an occasional failure mode rather than symmetric jitter.The low run reduced both the logic axis and instruction-following together; three runs estimate σ = 4.4 pp only loosely, with a 95% interval of [2.3, 27.7] pp.
  • Noise floor: 7.7 pp range and 4.4 pp standard deviation show that training variance, not sampling variance, is the operative error bar.The benchmark’s ±3.4 pp sampling-error figure reflects item selection, while training variance was roughly twice as large.
  • Behavioural stability: 1.00/1.00/1.00 trace-Greek and 132–152-word trace lengths remain stable across seeds despite a 7.7 pp accuracy swing.Trace-Greek is at its ceiling, so unsaturated trace length provides stronger stability evidence; fallback% instead swings 3–41%.
  • Control findings: −0.5 pp selection effect shows that choosing the top 15,607 structurally scored rows did not beat uniform random sampling from the same pool.The pre-registered prediction required selection to win by more than 2 pp, but the apparent earlier advantage was confounded by corpus version.

5. What actually changed

At equal or indistinguishable accuracy, fine-tuning changes the model’s reasoning language, efficiency, and difficulty adaptation rather than benchmark means. These gains coexist with format failures, family-dependent token costs, and no evidence that Greek reasoning itself reduces reasoning quality.

  • Reasoning language: 0 of 1,000 base traces reach a Greek character ratio of 0.9, versus 97.4–98.7% of measured fine-tuned traces reasoning in Greek.The base median Greek-character ratio is 0.33; the four SFT releases reach 97.4–98.1%, while the one-directional Qwen arm reaches 98.7%.
  • Format versus capability: 70.2% of pre-repair Gpt-OSS rows never emit the requested answer line, making its 56.2 Greek-lane anchored accuracy largely a format floor.The reasoning is present and on-language, but the scorer reads the missing closing answer line; the repaired release’s fidelity is measured independently on traces.
  • Accuracy controls: +1.4 pp (0.65σ) is the controlled language effect, replacing the misleading −17.5 pp gap between Greek and English item mixes.The controlled comparison held items and answer language constant while varying only trace language; the earlier gap arose from selection.
  • Reasoning efficiency: 3× fewer tokens on Qwen, parity on NemotronH, and 1.6× more on Gpt-OSS show that serving-cost savings are family-dependent.The reported token comparisons are Qwen 586 vs 1,788, NemotronH 638 vs 681, and Gpt-OSS 640 vs 396.
  • Accuracy and effort: 72.9 vs 72.9 on the same 1,000-item probe accompanies sharply shorter reasoning: 1,010 versus 150 median words per trace for the base and reasoning-only fine-tune.The corresponding median words per correct answer are 1,454 and 239; conditioning on trace length, the base is at least as accurate as the fine-tune.
  • Difficulty adaptation: 98.0% of easy items trigger base-model overruns, versus 0.2–12% for every fine-tune.The base rate is unchanged across easy and hard items, 98.0 vs 98.4, whereas fine-tuned arms spend more on harder items; generation-cap truncation affects hard-item think-mode ratios.

6. Losses that looked like forgetting

Almost nothing attributed to fine-tuning was truly forgotten: language lock, general ability, commonsense, register control, and grammaticality largely remain intact or improve. The genuine loss is answer-format compliance, which is recipe-localized rather than an inherent consequence of learning Greek reasoning.

  • Overall finding: The language lock is conditional on training direction rather than deletion, while register control is retained or gained and grammaticality improves across all four families.The section concludes that benchmark-visible forgetting is usually an artifact of capability expression, measurement, or recipe choice.
  • General ability: −3.2-point Greek residual remained only for the format-repaired Gpt-OSS release, while general ability was flat in both languages on two families.The paper frames this as a limited residual rather than broad catastrophic forgetting.
  • Format compliance: Answer-format failures rose from 12% with single-phase training to 33–40% with two-phase training, identifying the genuine loss as recipe-localized.Reasoning-only training stayed at 2–12%, widening to 2–41% with seed noise; language-matched checkpoints also paid the cost at 24% and 26%.
  • Commonsense expression: −3.0 pp commonsense deficit appeared in 12 of 12 fine-tuned arms, but constrained answer formatting reversed it to +1.7 pp.The capability was not lost; its expression changed under long-form generation.
  • Fluency and register: Grammatical correctness improved on every family: NemotronH 13/58 →27/57, Nemotron-3.5 12/58 →29/58, Gpt-OSS 32/57 →40/58, and Qwen 41/58 →42/58.Register control was never lost, and the Gpt-OSS first-pass scores were corrected after a response-splitting bug.

7. A side experiment: three recipes, and what the second phase costs

This controlled detour compares single-phase, two-phase, and reasoning-only SFT recipes. Reasoning-only training produced the only replicated accuracy effect surviving the noise floor, while two-phase training repaired reasoning-mode switching at an instruction-following cost.

  • Recipe effects: 23.6% of think-mode items had empty traces after single-phase SFT, which collapsed the reasoning switch despite the question’s flag.The rate counts empty traces rather than answer coverage and is described as coverage-independent.
  • Recipe effects: 0–1.3% empty traces followed two-phase training, but fallback rose to 33–40% and accuracy did not improve.Reasoning-only pretraining repaired the mode switch, while the hybrid second phase degraded answer-format compliance.
  • Accuracy effect: 73.6 versus 66.7 accuracy favored reasoning-only training across 15 arms, with 52 of 54 pairwise comparisons favoring it and exact permutation p = 0.0008.On the decontaminated benchmark, the effect was +5.3 pp (p = 0.0164), while reasoning-only won every axis on the 5,156-item benchmark (+5.4 pp mean).
  • Caveats: The accuracy result remains uncertain across designs: corpus version was unbalanced, stratification gave p = 0.125, and Gpt-OSS reversed the ordering at 62.5 versus 68.1.The aggregate result is significant for Qwen, but the stratified test has little resolution and the Gpt-OSS comparison has one run per condition.
  • Accuracy effect: +6.9 pp accuracy across 15 runs (p = 0.0008) was the only replicated effect large enough to survive the noise floor.The effect came from removing the non-reasoning corpus half; the proposed implication applies only to deployments that never need direct mode.

8. Six ways our instruments lied

The section shows that six plausible findings were withdrawn or corrected only after controls exposed biased scoring, contamination, translation artifacts, language-specific parsing errors, and hidden condition-selection factors. These failures demonstrate that low-resource evaluation requires format-anchored scoring, matched-language checks, human-translated controls, locale-aware instruments, and same-day baselines.

  • Answer-format scoring: +29.8 pp (18σ) artifact favouring the fine-tune vanished when extraction was anchored to the requested answer line.The un-anchored scorer let the last option mention win, penalising verbose responses while sparing terse ones.
  • Prompt-format controls: −3.0 pp commonsense deficit across 12 of 12 fine-tuned arms reversed to +1.7 pp under a letter-only prompt.The apparent deficit existed only when models reasoned at length; letter-only output was valid for commonsense but invalid for logic, where the base reached 29.6% against a 33% chance baseline.
  • Contamination controls: 38.9% of the logic probe was contaminated: 175 of 450 items shared a 13-gram with the training pool, while math and commonsense were clean at 0%.The matched English corpus and evaluation-set check found zero collisions, showing that contamination checks must run in both languages.
  • Translation validity: 0.6 pp fine-tune advantage on machine-translated items became a 1.1 pp base lead on the 250-item human-translated subset.The shared commercial translation model adapted fine-tunes to its Greek register, and the human-translated control caught a 3.1σ artifact.
  • Locale-aware scoring: 5 of 1,100 anchored math rows, or 0.45 pp, were mis-scored because Greek thousands and decimal separators were parsed using English conventions.The released Qwen checkpoint had five wrong scores among six Greek-thousands-form answers; decimal commas could also turn a wrong answer into a correct one.
  • Same-day baselines: 1.1% fresh-baseline fallback versus 24% frozen fallback exposed a hidden condition-selection factor, while rescoring reproduced 24.1%.A same-day baseline caught the sixth failure during RLVR evaluation; downstream numbers remained plausible until the known quantity was regenerated.

9. A metric needs a control, not a normalisation

Length normalization cannot rescue behavioural metrics whose values are driven by trace length; valid measurement requires controls that hold length constant or reject length-correlated candidates. A repetition penalty further shows that identical surface degeneration can serve different functions in base and fine-tuned models.

  • Length controls: |corr(words, metric)| of 0.44–0.85 for per-token rates and 0.86–0.95 for fixed-window rates shows that length normalization does not remove length-driven measurement.The proposed fix is a comparison conditioning on the item, which holds length constant by design.
  • Metric gate: |r| ≥0.6 with trace length triggers metric redefinition or removal, eliminating three of seven original dimensions.This included the unsupported claim that fine-tuning reduced looping by 29×.
  • Functional degeneration: −0.4 pp for the base versus −9.7 pp for the fine-tune under repetition penalty indicates that the same constraint has sharply asymmetric costs.The passage interprets base loops as occurring when it has nothing to say, while fine-tune loops are part of its reasoning.

10. Fixing the lock: train on matched language pairs

Language-matched retraining prevents the one-directional fine-tune from ignoring the question-following default and partially reopens explicit English-language overrides. The recovery is family-dependent and carries a measurable Greek-lane cost, so the paper treats language-matching—not full restoration of steerability—as the supported claim.

  • Instruction-channel steerability: 0/1,000 instructed English overrides are obeyed by the one-directional Qwen arm, whereas matched retraining reopens English compliance on Qwen and Gpt-OSS.Reported compliance is 44.8–62.5% on Qwen and Gpt-OSS, but 0.0% on both Nemotron arms; the severity is family-dependent under the same recipe.
  • Matched language-pair retraining: 100% of 1,900 English questions receive English reasoning after language-matched retraining, with median Greek-character ratio 0.000 across four domains.The checkpoint pairs each Greek problem and trace with its English original and trace, excluding traces that drift languages during training.
  • Matched language-pair retraining: −6.9 pp is the raw-lane Greek accuracy cost of the matched checkpoint, scoring 73.7 versus the base’s 80.6 on 5,156 items.The paper places this difference within the family’s 7.7-point seed floor and contrasts it with a −0.7 best-of-15 probe reading under answer-format compliance.
  • Instruction-channel steerability: 83.7–93.3% of releases obey an instruction to reason in Greek, but none fully obeys an instruction to reason in English, leaving Nemotron at zero English compliance.The matched recipe restores the question-following default on every trained family, while the override returns only on two of four checkpoints.
  • Instruction-channel steerability: +9.1 pp is the override-obedience gain from verifiable-reward training against a flat random-reward control, without cost on held directions.The gain remains below the pre-registered trainability bar, while the reason for the family split after identical SFT remains unresolved.
  • Deployment implication: The matched checkpoints are released because they restore the default on every family and are the only checkpoints measured to regain any instruction-channel override.The paper distinguishes default behaviour from instructability: one-directional arms can match or exceed printed default metrics yet read 0/1,000 on the English override probe.

11. Two pre-registered questions, and how they closed

The two remaining pre-registered questions found no general-ability damage except a repairable format-related Gpt-OSS deficit, and no measurable loss in non-reasoning mode. Fixed evaluations showed that the deficits reflected answer-format behaviour rather than destroyed capability, while direct-mode performance remained flat or improved.

  • E2: General ability: E2 found no general-ability damage on the earlier recipe, with one repairable exception on the language-matched recipe.The evaluation used separate Greek and English macros over frozen log-likelihood benchmarks, avoiding generation effects.
  • E2: General ability: −7.3 points Greek was the initial Sophea-OSS-v1 loss; format repair recovered 4.1 points, leaving −3.2 Greek on the released checkpoint.The same repair reduced fallback from 70% to 26%, identifying format-closing behaviour as the fixable cause.
  • E2: General ability: Within 1.1 points of its own base, Lightning retained English, while Qwen and Nano were flat or gained and register control survived fine-tuning.Grammaticality evidence was directional because the judge misread a channel marker as ungrammatical Greek; stripping it reversed the verdict.
  • E3: Non-reasoning mode: E3 found that dropping the direct-training half cost nothing measurable in non-reasoning mode.On the Titan-1 suite, Qwen was −0.1 and Nemotron-3.5 was +1.7 against their own bases; pre-repair Gpt-OSS remained −7.3, matching think mode.
  • E3: Non-reasoning mode: 0 unrequested traces appeared in 9,751 direct-mode generations for the <think>-carrier families, but Gpt-OSS opened its analysis channel on 100% of rows.Neither Gpt-OSS fine-tune installed an off switch that its base never had.

12. Per domain: indistinguishable on accuracy, not on cost

Per-domain accuracy differences are indistinguishable from a 7.7 pp seed-noise floor, so the evidence does not support choosing a model by domain. Cost differs sharply: reasoning-only fine-tunes require 268–294 words per correct answer versus 1,352 for the base.

  • Accuracy: 7.7 pp seed range makes per-domain accuracy winners unreadable as anything more than noisy draws.Figure 11 is explicitly not a leaderboard because reruns could name different winners.
  • Accuracy: Math spans 87.6–95.6, commonsense 70.7–82.3, and logic 40.5–46.3, against the 7.7 pp seed range.The corresponding spreads are 8.0, 11.6, and 5.8 pp; choosing a model per domain is unsupported.
  • Cost: 1,352 words per correct answer for the base versus 268–294 for reasoning-only fine-tunes yields a 4.6–5.0× cost difference.In tokenizer tokens, the gap narrows or reverses by family.
  • Deployment: Deployment choices should therefore rely on cost and trace language, not per-axis accuracy.The passage frames this as the defensible basis when selecting one Greek reasoning model.

13. What each family taught us

Each model family exposed a different failure mode or implementation trap, often producing plausible but invalid measurements rather than crashes. Despite these differences, the language-transfer result replicated on the third, non-transformer architecture.

  • Qwen: At 1,536 tokens, Qwen leaves 670 of 1,000 think-mode rows unfinished and scores 20.8, versus 77.2 at 4,096 tokens.Its long base traces and seed sensitivity make standard-budget comparisons measure the budget; its fine-tune is also completely language-locked.
  • Gpt-OSS: Gpt-OSS emits an analysis channel on 1000/1000 reasoning-off rows, so gating channel splitting on the prompt caused concatenated scoring and 95 unparseable logic rows.The correct rule is to split whenever channel markers appear, regardless of whether reasoning was requested.
  • Gpt-OSS: Dropping the non-reasoning corpus half helps Qwen by +6.9 pp but hurts Gpt-OSS by 5.6, while their English-instruction compliance is 0% versus 95%.The same recipe and corpus therefore produce opposite behavioral responses across families, without an identified mechanism.
  • Nemotron: Nemotron’s fine-tune reasons in Greek on 98% of benchmark items versus 0% for its base, while accuracy changes by −0.6 pp.Generation-cap truncation falls from 10.8 to 2.6%, and measure loops fall from 15.0 to 0.0 per thousand rows.
  • Cross-family pattern: Every family needed a different fix, and three fixes produced plausible numbers rather than crashes, so completed pipelines do not establish correct evaluation.The result also replicates beyond transformer MoEs: the evidence links movable reasoning language to a strong base plus a language-directed corpus, not transformer architecture alone.

14. What we can and cannot recommend · 15. A third pre-registration: verifiable-reward RL, designed before its numbers

The paper recommends only configurations supported by evidence, then pre-registers verifiable-reward RL to test whether it can repair SFT’s formatting, leakage, and instruction-following defects without sacrificing Greek reasoning fidelity.

  • 14. What we can and cannot recommend: Configuration recommendations are tied to explicit evidential bases, because unsupported configuration tables would overclaim what the measurements establish.The paper distinguishes recommendations the evidence supports from those it cannot support.
  • 15. A third pre-registration: verifiable-reward RL, designed before its numbers: 24% fallback and 3.5% answer-channel leakage are structural SFT defects that imitation cannot eliminate through counterfactual learning.The released Qwen checkpoint omits the requested answer line on 24% of Greek items and leaks answers into the reasoning trace on 3.5%.
  • 15. A third pre-registration: verifiable-reward RL, designed before its numbers: The reward design pairs Greek and English versions of each problem and scores correctness, language consistency, format, termination, and override obedience with verifiable terms and gates.The paired pool prevents a Greek-only language-shaped objective from targeting forgetting, while explicit instructions redirect the language-consistency term.
  • 15. A third pre-registration: verifiable-reward RL, designed before its numbers: The four-arm experiment varies only active reward terms and includes a random-reward control whose matching outcomes would be withdrawn as evidence of elicitation rather than learning.All arms share data, steps, seed, hardware, and data order; the control can veto conclusions on any axis it matches.
  • 15.1. The outcomes, against the frozen rules: 24.1% → 2.5% fallback and 3.53% → 0.00% answer-channel leak passed the frozen RLVR thresholds, while the random-reward control moved neither metric.Greek-trace fidelity was held, establishing that RLVR closed both defects rather than merely trading them against language consistency.
  • 15.1. The outcomes, against the frozen rules: +9.1 pp override compliance was reward-responsive but missed the frozen trainability bar, while 98.22% Greek-trace fidelity showed no collapse under an unprotected accuracy gradient.Reverse-direction compliance reached 85.7%, the agreeing direction remained 98.5%, and the study reports bounded conclusions for one Qwen family, seed per arm, final checkpoint, and unscreened pool.

16. What to take away · 17. Related work

The paper’s takeaway is that unchanged accuracy does not imply a failed fine-tune: SFT changes reasoning language, budget, and compliance properties that accuracy misses, while RL fixes specific SFT defects. Related work situates these findings against English-default reasoning, seed-variance reporting, self-taught trace selection, tokenizer fertility, and translated-benchmark artifacts.

  • 16. What to take away: A seed control should precede ablations, and fallback% should accompany accuracy to distinguish inability from refusal to answer in the requested form.The authors also recommend retaining a human-translated subset of machine-translated benchmarks.
  • 16. What to take away: A fine-tune that leaves accuracy unchanged is not necessarily failed: SFT makes models reason in Greek, sometimes using 3× fewer tokens, while other families incur parity or a 1.6× premium.These deployment-relevant trace-language and budget properties are invisible to accuracy tables.
  • 16. What to take away: No fine-tuned arm beats its base on the pooled Greek reasoning benchmark, so the paper reports a measured null with a stated noise floor rather than an unresolved comparison.The NLU retention suite is a different quantity, where two arms gain.
  • 17. Related work: English is the established default for multilingual reasoning, supported by findings that English chain-of-thought outperforms native-language reasoning and by an English-like intermediate representation.Prompting and distillation methods generally translate non-English tasks into English before reasoning.
  • 17. Related work: For low-resource deployment, this paper treats the reasoning trace as the product: an opaque English trace is a defect, and SFT installs Greek reasoning while measuring asymmetric instruction steering.The locked fine-tune obeys “reason in English” on 0% of items, while the base ignores “reason in Greek” on 72%.
  • 17. Related work: 7.7 pp seed variance with sd 4.4 pp occurs on identical MoE + LoRA configurations, while behavioural metrics remain flat across the same three seeds.The authors identify this accuracy-behaviour asymmetry as an unmeasured contribution to seed-variance literature.
  • 17. Related work: The corpus pipeline descends from STaR by retaining traces whose final answers match gold, while tokenizer fertility makes word-denominated budgets fairer than token-denominated comparisons.Greek fertility is measured at 2.3–2.5× that of English across all three families.
  • 17. Related work: Greek evaluation inherits established translated suites, while this paper adds a benchmark artifact mechanism that can favour the fine-tuned model rather than distort all models alike.Prior work documents independent premise-hypothesis translation effects and ranking changes across full versus culturally sensitive subsets.

Limitations

The evidence is limited to LoRA adaptation of sparse MoE models, with sparse routing, seed sensitivity, and several unvaried design choices left unresolved. Greek transfer and auditability remain untested hypotheses because the study lacks dense controls and broad human evaluation, although the released controls are designed for reuse.

  • Model and adaptation scope: LoRA adaptation of sparse mixture-of-experts models limits the claims, because adapters reach only selected MoE layers and the shared expert.A dense baseline was not tested, so the study cannot distinguish LoRA seed sensitivity from amplification by sparse routing.
  • Experimental coverage: Single-seed coverage, small noise-floor estimates, four unvaried conditions, and limited benchmark axes constrain the robustness of the comparisons.The unvaried conditions are LoRA rank and targets, one epoch, the fixed non-reasoning arm, and greedy decoding; the Greek commonsense probe saturated at 96–97%.
  • Generalisability and evaluation: Greek is mid-resource, so transfer to truly low-resource languages remains untested; auditability is also a hypothesis because nearly all fidelity measures are automatic.The only human-labelled switching probe used ∼150 traces, one annotator, no guidelines, and no agreement measurement.
  • Releases: Five fine-tuned checkpoints are released across the three model families, including four SFT releases and one RLVR override arm.The RLVR release passed every hold-gate with fidelity 98.27, while missing the 60% override target.
  • Reusable methodology: The reusable controls require only a few hours of compute, need neither the authors’ models nor corpus, and each changed a previously written conclusion.The paper recommends running them in the stated order because they cost less than one ablation.
Loading 2608.17744v1…