PaperlayerView exact revision on arXiv

Source-linked AI summary

CantoneseLLM v2: Reasoning in a Low-Resource Language

Tsz Chung Cheng, Chung Shing Cheng, Chaak Ming Lau, Cheuk Hei Chong

arXiv:2609.06970v1cs.CL

TL;DR

Cantonese is widely spoken but lacks suitable written data and large-scale native reasoning traces for model training. CantoneseLLM v2 adapts Qwen3 models through staged training, culminating in language- and script-aware RLVR. The final 30B-A3B model reaches 73.16, within 1.20 points of its merged checkpoint, while reasoning and answering consistently in Cantonese.

  • Problem

    Cantonese has limited suitable written data, and available models commonly reason in a language other than Cantonese.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper adapts Qwen3 models through CPT, chat-vector merging, SFT, DPO, and RLVR with a hard format gate and language-and-script reward multiplier.

    Source: §1 · p. 2, §8.1 · p. 11, §8.2 · p. 12

  • Results

    73.16 was the 30B-A3B model’s benchmark score after RLVR, within 1.20 points of its merged checkpoint while enabling consistent Cantonese reasoning and answers.

    Source: §1 · p. 2, §11 · p. 16

  • Takeaways & Limitations

    The final model can reason and answer consistently in Cantonese, unlike the earlier checkpoints and official Qwen3 model.

    Source: §11 · p. 16

    No long original Cantonese reasoning traces were available at the scale needed for SFT; only 131 rows of expert-authored question-and-answer pairs were written or verified by humans.

    Source: §10 · p. 16

Abstract

from arXiv · show

Cantonese is widely spoken but remains low-resource in written data, with no large corpus of native Cantonese reasoning traces available for model training. We develop and release CantoneseLLM v2, comprising models based on Qwen3 8B and 30B-A3B. The models are trained through CPT on 784 million Cantonese and Hong Kong-related tokens, chat-vector merging, SFT, DPO, and RLVR. Evaluation across the training stages shows that chat-vector merging transfers instruction following but preserves the donor model's reasoning language, while SFT with limited Cantonese reasoning data substantially shortens or removes reasoning traces and reduces benchmark performance. DPO restores the reasoning-block format, particularly for the 8B model, but recovers only part of the lost performance. The RLVR training with Cantonese language and Traditional Chinese scripts as multiplicative constraints introduced Cantonese language alignment and restored the lost performance. The 30B-A3B model reaches 73.16 on HKCanto-Eval, within 1.20 points of its merged checkpoint, while retaining the Cantonese reasoning behaviour absent from that checkpoint. We release the model checkpoints, the training environments, and a thirteen-year Traditional Chinese Common Crawl dataset. The models can be accessed at https://huggingface.co/collections/hon9kon9ize/cantonesellm-v20

1 Introduction

Cantonese is widely spoken but remains under-resourced in suitable written data, limiting the development of locally deployable models that reflect its spoken grammar and vocabulary. CantoneseLLM v2 builds Qwen3-based reasoning models through staged training; its final RLVR stage restores benchmark performance while aligning reasoning and answers with Cantonese.

1 Introduction

  • Cantonese’s low-resource status reflects a shortage of suitable written data, despite its approximately 85 million speakers, and available material is concentrated in informal domains.The shortage limits modelling idiomatic, context-sensitive, socially acceptable language across settings relevant to local deployment.Source: §1 · pp. 1–2
  • The five-stage development combines CPT, chat-vector merging, SFT, DPO, and language-aware RLVR to build Qwen3 8B and 30B-A3B reasoning models.CPT installs local knowledge, chat-vector merging transfers instruction following, and later stages address reasoning format and language alignment.Source: §1 · p. 2
  • Chat-vector merging transfers instruction following without training, but leaves the donor checkpoint’s reasoning language unchanged.CPT was used to install local knowledge and Cantonese lexis on a corpus of under one billion tokens.Source: §1 · p. 2
  • DPO restored the 8B model’s reasoning-block format and recovered part of the benchmark regression introduced by SFT.SFT with limited Cantonese reasoning data shortened reasoning traces to one-tenth of their previous length in the 30B-A3B model.Source: §1 · p. 2
  • 73.16 was the 30B-A3B model’s benchmark score after RLVR, within 1.20 points of the merged checkpoint while enabling consistent Cantonese reasoning and answers.The language and script constraints were applied as a reward multiplier.Source: §1 · p. 2
  • The work releases four 8B and 30B-A3B checkpoints, a thirteen-year Traditional Chinese Common Crawl dataset, and training environments and data.The released checkpoints include chat-vector-merged and final reinforcement-learning-trained models.Source: §1 · p. 2

2 Why the Standard Recipe Does Not Transfer

Established adaptation recipes assume resources Cantonese lacks, including large curated corpora and a fluent target-language teacher. Cantonese’s distinct vocabulary, grammar, pragmatics, and social norms also make mechanical Mandarin adaptation an inadequate substitute.

2 Why the Standard Recipe Does Not Transfer

  • Large-scale continual pre-training typically assumes curated corpora of tens of billions of tokens, while distillation assumes a teacher already fluent in the target language; Cantonese has neither.The supplied passage characterizes these conditions as limiting the transferability of established adaptation recipes.Source: §2 · pp. 2–3
  • Cantonese has 784M tokens available in the compared corpus-scale data, while established examples list 35.1B tokens for Taiwan-LLM and 200B each for Swallow and SEA-LION.The table frames corpus scale as the first contrast and notes shortages in other parts of the training pipeline as well.Source: §2 · pp. 2–3, Table 1 · p. 3
  • Mandarin resources cannot simply be translated or converted into Cantonese data because the languages differ in vocabulary, grammar, pragmatics, cultural references, and register norms.Mechanically adapted data may be intelligible yet unidiomatic, culturally incongruent, or socially inappropriate; Cantonese-prompted models also commonly reason in another language.Source: §2 · p. 3

3 Method

The method adapts Qwen3 8B and 30B-A3B through five stages, diagnosing problems after each rather than treating the stages as a fixed recipe. It evaluates benchmark performance and controlled generation behaviours, then uses task rewards, a format gate, and language-and-script constraints in reinforcement learning.

3 Method

  • The five-stage pipeline applies CPT, chat-vector merging, SFT, DPO, and RLVR to Qwen3 8B and 30B-A3B checkpoints.Each stage responds to a problem diagnosed in the preceding checkpoint.Source: §3 · p. 3
  • Evaluation after each stage uses HKCanto-Eval and controlled probes of reasoning-block presence and length, output language, script use, instruction compliance, and termination.The training sequence and its diagnostics are also summarized in Figure 1.Source: §3 · p. 3
  • RLVR combines task-specific verifiable rewards with a hard output-format gate and a multiplicative language-and-script factor rewarding Cantonese reasoning in Traditional Chinese script.The pipeline is presented as a sequence motivated by failures such as missing instruction following, non-Cantonese reasoning, shortened reasoning blocks, and instruction-following regressions.Source: §3 · p. 3

4 Continuous Pre-Training (CPT)

CPT uses a 784M-token corpus to install local knowledge, but the learning-rate sweep shows that lower training loss does not necessarily mean better downstream benchmark performance. Corpus composition and the sweep results reveal important constraints on CPT for Cantonese.

4 Continuous Pre-Training (CPT)

  • The corpus was curated from Common Crawl, web data, and synthetic augmentation, and CPT was run on Qwen3 8B and 30B-A3B to inject Hong Kong-related knowledge.The work describes the corpus as substantially smaller than those used in previous CPT studies.Source: §4 · p. 3, §4.1 · p. 3

4.1 Corpus

  • The corpus contains 784M tokens and 568K rows; Common Crawl supplies 32.9% of total training data, including 27.5% written Chinese and 5.4% Cantonese.Web fiction contributes 17.1% of the tokens and is the only source described as sustaining Cantonese over tens of thousands of tokens.Source: §4.1 · p. 3
  • The corpus’s concentration in historically informal domains may affect content quality and could degrade model performance.The authors describe this concentration as an unavoidable outcome of Cantonese’s historical association with informal domains.Source: §4.1 · p. 4
  • Filtering reduced 477,298 unique Cantonese Common Crawl documents to 31,505 and 2,500,553 written Chinese documents to 141,230.The written Traditional Chinese branch was limited to eight 2025 snapshots and filtered using keywords and a regional check.Source: §4.1 · p. 4

4.2 Training and the Configuration Sweeps

  • The 8B benchmark averages across three CPT step counts were 65.07, 65.12, and 64.85; the 530-step run was selected and used for 30B-A3B as well.The per-category scores were not flat across the runs.Source: §4.2 · p. 4
  • Across the 30B-A3B sweep, cultural-category scores stayed between 67.86 and 69.05, while its best improvement was 1.16% versus 3.25% for the best 8B improvement.The 30B-A3B baseline at 67.69 already exceeded the fully trained 8B model on four of five categories; Cultural was the exception.Source: §4.2 · p. 5
  • The 30B-A3B learning-rate sweep found benchmark average and final training loss moving in opposite directions, with the best benchmark score at the second-highest loss.The lowest-loss run scored below the selected runs, and the text cautions that training loss can anticorrelate with the quantity of interest in low-resource CPT.Source: Table 4 · p. 5, §4.2 · p. 5

5 Chat Vector Merging

Chat-vector merging adds instruction-following ability to continuously pretrained models, but benchmark effects differ by size and reasoning traces retain the donor model’s language. The 8B merge improves against its donor, whereas the 30B-A3B merge regresses.

5 Chat Vector Merging

  • The 8B merge improved against its donor, while the 30B-A3B merge scored 1.94 percentage points lower, with regression concentrated in English MMLU.The 8B model improved across multiple-choice benchmarks except MMLU.Source: Table 5 · p. 6, §5 · p. 6
  • Chat-vector merging transfers instruction following by isolating the direction between official chat and base weights and adding it to the continuously pretrained checkpoint.The method assumes instruction following and alignment can be separated from language knowledge in weight space.Source: §5 · p. 6

5.1 Language of the Trace in the Merged Models

  • Merged models’ reasoning traces followed the donor’s language: the 8B models used Simplified Chinese, while the 30B-A3B merge shifted toward formal or Mandarin-shared wording.The comparison used a machine-translated GSM8K question; all four models returned the correct answer.Source: §5.1 · p. 6

6 Supervised Fine-Tuning (SFT)

SFT targeted Cantonese translation and reasoning, along with judging and data curation, but reasoning traces shortened or disappeared and both models regressed against their merged checkpoints. The 8B and 30B-A3B models exhibited different reasoning failures, with the training mixture and data constraints helping explain the outcomes.

6 Supervised Fine-Tuning (SFT)

  • SFT targeted Cantonese translation and reasoning, LLM-as-a-judge, and data curation using 74,865 rows and 177.4M tokens, 52.2% of them reasoning tokens.No parameter sweep was performed because post-training ran on limited NVIDIA GPUs.Source: §6 · p. 7

6.1 SFT Data Mixture

  • The mixture combined native Cantonese data, translated reasoning data, and English coding or multilingual replay; 25% of 8B rows had reasoning fields stripped.The 8B model’s hybrid reasoning and non-reasoning modes motivated stripping those fields from a quarter of its rows.Source: §6.1 · p. 7

6.2.1 Reasoning Failure in the 8B model

  • On six of eight probes, the 8B SFT model emitted an empty reasoning block, sometimes shifting reasoning into its answer instead.The authors link the empty block to training on rows without reasoning fields, which made the empty token pair a low-loss attractor.Source: §6.2.1 · p. 7, §6.1 · p. 7
  • The 30B-A3B SFT checkpoint produced 1,266 reasoning tokens across eight probes, versus 8,335 for the official model and 11,535 for the merged model.Shortening also affected more complicated mathematical or programming questions, rather than selectively shortening simpler tasks.Source: §6.2.1 · p. 8
  • Replay sets supplied 53% of reasoning tokens but contained no Han characters, while Cantonese traces were short; improving the data further would be costly and require a capable Cantonese model.The Cantonese translation portion averaged 138 tokens per trace, compared with 3,336 and 2,635 tokens in the two replay sets.Source: §6.2.1 · p. 8
  • A translation probe also exposed an instruction-following leak: the 8B SFT model reproduced the written Chinese source despite being told not to explain.Its reasoning trace said no explanation was required, contradicting the output.Source: §6.2.1 · p. 8

6.3 Benchmark Regression

  • SFT reduced average benchmark scores by 20.52 points for 8B and 12.46 points for 30B-A3B versus their merged checkpoints.The 8B regression spanned all categories; the 30B-A3B model’s largest losses included MMLU, CantoMMLU, and Academic and Professional.Source: §6.3 · p. 8
  • The 30B-A3B model generated repetitive reasoning loops and failed to stop on many responses, while the 8B regression was not a termination failure.The 30B-A3B model hit the generation limit on 24.9% of responses; the corresponding 8B figure was 2.7%.Source: §6.3 · pp. 8–9

7 Direct Preference Optimisation (DPO)

DPO was applied to repair reasoning-block failures after SFT, and its effects differed by model size. It restored the 8B model’s reasoning format and partly recovered benchmark performance, while increasing formatting noncompliance on benchmark answers.

7 Direct Preference Optimisation (DPO)

  • DPO addressed an SFT reasoning defect: 64.5% of 8B generations had no reasoning content, compared with 13.2% for 30B-A3B.The reasoning defect was measured over four generations for each of 60,850 prompts.Source: §7 · p. 9, Table 8 · p. 9

7.1 Preference Pairs

  • Preference pairs were selected from four scored candidates per prompt, with eligibility rules for Cantonese reasoning, answer length, and a minimum composite-score difference.The procedure produced 12,204 pairs for 8B and 14,679 for 30B-A3B.Source: §7.1 · pp. 9–10
  • Plain prompts outscored verbose add-ons on the judge’s overall dimension at both model sizes, so prompt perturbation was not a reliable quality label.For 8B, plain candidates won 54.2% of pairwise comparisons and verbose candidates won 19.9%.Source: §7.1 · p. 10

7.2 Evaluation

  • The 8B model’s empty reasoning blocks disappeared in all eight probes, while its reasoning length increased from 211 to 2,652 tokens.The 30B-A3B model’s reasoning length increased from 1,266 to 2,580 tokens across the probes.Source: §7.2 · p. 10

7.2.1 Benchmark Results

  • 10.19 points: the 8B average rose from 48.99 to 59.18, recovering half of SFT’s regression; the 30B-A3B average rose from 61.90 to 64.17, mainly through MMLU.The 8B model improved in every category; for 30B-A3B, the other four categories moved between −2.30 and +0.50.Source: §7.2.1 · p. 10
  • 99% to 70% to 10%: answers beginning with a bare option letter fell from the merged checkpoint through SFT to DPO, underestimating reported benchmark scores by an unknown amount.The pair-selection procedure favored longer responses without rewarding instruction compliance.Source: §7.2.1 · p. 11

8 Reinforcement Learning with a Verifiable Reward (RLVR)

The paper uses RLVR with format, language, and script constraints across a two-stage curriculum and six task environments. The final 30B-A3B model reached 73.16 points while retaining Cantonese reasoning behaviour absent from its merged checkpoint.

8 Reinforcement Learning with a Verifiable Reward (RLVR)

  • RLVR used a multiplicative language-and-script reward to constrain reasoning language and recover capability lost in upstream training.GRPO ran first on GSM8K to establish format and language, then across diverse environments; both models could reason and answer in Cantonese.Source: §8 · p. 11

8.1 Reward Design

  • A hard format gate assigns zero reward unless the output starts with exactly one non-empty reasoning block; task correctness and auxiliary terms contribute equally when the gate passes.The task term uses exact numeric verification in stage one and per-environment graders in stage two.Source: §8.1 · p. 11

8.2 The Language and Script Multiplier

  • The language-and-script multiplier combines a language factor with a script factor, penalizing wrong dominant scripts and reducing reward for Written Chinese reasoning on Cantonese prompts.The Written Chinese reading retains one tenth of its score after rescaling.Source: §8.2 · pp. 11–12

8.3 A Two-Stage Curriculum

  • The curriculum used GSM8K for initial format and language training, then applied the reward across six environments, including language-specific instruction-following and structured-output streams built with rule-based methods.The Cantonese datasets added constraints for colloquial register, sentence-final particles, idioms, and Jyutping rhyme.Source: §8.3 · p. 12, §8.4 · p. 13

8.5 Difficulty Profiling and the Cross-Language Gap

  • Difficulty profiling covered 13,225 rows and about 97,000 rollouts; dropping never-solved and always-solved rows cut rollout cost by 46.4% without loss of gradient signal.For matched mathematics prompts, the Cantonese pass rate was lower by 0.247; coding rollouts also frequently exhausted the generation limit without emitting a program.Source: §8.5 · p. 14

8.6 Results

  • Validation reward increased for both models, but their values are not directly comparable because their training-data selections differed.The 8B reward peaked at 0.443; the 30B-A3B reward peaked at 0.479, with five of six environments improving from starting values.Source: §8.6 · p. 14

8.6.1 Chain-of-Thought Probes

  • Both stage-two models reasoned and answered in Cantonese on the probes, with lexically correct word usage.The 30B-A3B model’s final benchmark average was 73.16, while the 8B recovery remained 6.71 points below its merged checkpoint.Source: §8.6.1 · p. 15, §8.6.2 · p. 15

8.6.2 Benchmark Results

  • 73.16 points: the final 30B-A3B model recovered from 64.17, gaining 8.99 points—88% of the performance lost between the merged checkpoint and DPO—and finished within 1.20 points of the merged checkpoint.The final model also retained target-language reasoning behaviour absent from the merged checkpoint.Source: §8 · p. 11, §8.6.2 · p. 15

9 Results

Across the training pipeline, supervised fine-tuning reduced benchmark scores, while later stages recovered part of the loss. The final 30B-A3B model approached its merged checkpoint while retaining Cantonese reasoning behavior.

9 Results

  • SFT removed 20.52 points from 8B and 12.46 from 30B-A3B; DPO restored 10.19 and 2.27 points, respectively.Final GRPO regained a further 3.62 points for 8B and 8.99 for 30B-A3B.Source: §9 · p. 15
  • 73.16 points: the final 30B-A3B model finished 1.20 points below its merged checkpoint, while the 8B model remained 6.71 points below its starting checkpoint.The 30B-A3B model was described as roughly on par with the merged checkpoint.Source: §9 · p. 15
  • SFT shortened reasoning traces to 158 tokens at 30B-A3B and 26 at 8B; DPO brought the 8B figure to 332 tokens.After reinforcement learning, Cantonese reasoning averaged 232 tokens at 30B-A3B and 155 at 8B.Source: §9 · p. 15
  • Multiple-choice scores alone miss whether a checkpoint reasons in the intended language and script, and formatting failures can also disrupt answer parsing.The paper identifies Cantonese reasoning and correct answer formatting as additional requirements for judging model quality.Source: §9 · p. 15

10 Limitations

The reported post-training results are constrained by limited experimentation, model-judged preference data, and a shortage of long native Cantonese reasoning traces.

10 Limitations

  • Post-training stages were run only once at each size, with no parameter sweep or optimization, because of severe compute constraints.The preference data were model-judged rather than human-annotated, and scores are bounded by the proprietary judge's ability to assess Cantonese reasoning.Source: §10 · p. 15
  • Formatting problems after instruction fine-tuning meant benchmark scores did not reliably reflect the model's underlying knowledge.Source: §10 · p. 16
  • No long original Cantonese reasoning traces were available; only 131 expert-authored human rows were available, and most traces were translated from English or simplified Chinese.The training data also lacked content grounded in Hong Kong entities or current events.Source: §10 · p. 16

11 Conclusion

CantoneseLLM v2 combines staged adaptation with filtered local-language data and generated commentary. Its final model reasons and answers in Cantonese, while the work documents constraints in the data and training process.

11 Conclusion

  • CPT installed local knowledge, chat-vector merging added instruction following but retained donor-language reasoning, and DPO restored the reasoning-block format.No long Cantonese reasoning traces were available; the released models can generate or translate traces for a future training corpus.Source: §11 · p. 16
  • 73.16 points: after reinforcement learning, the 30B-A3B model finished within 1.20 points of its merged checkpoint and could reason and answer consistently in Cantonese.Source: §11 · p. 16

A.0.1 Common Crawl

  • Educational-quality and information-stability filtering reduced 477,298 Cantonese documents to 31,505 (6.6%); regional filtering yielded 141,230 Written Chinese documents.The Written Chinese pool was predominantly Taiwanese before the regional gate: 49.1% Taiwanese versus 14.8% Hong Kong text.Source: §A.0.1 · p. 22

A Corpus Construction

  • Synthetic question-and-answer pairs added a median 301 characters to median 995-character documents, making about 23% of the branch's 42.4M tokens model-generated.Source: §A.0.1 · p. 22

B News Clustering and Persona Commentary

  • Clustering related news shifted training from transient incidents toward recurring patterns, and persona-conditioned generation expressed the local content in Cantonese.Generated personas lacked diversity: 65.3% were variants of old-item restorers, though the commentary remained largely useful.Source: B News Clustering and Persona Commentary · p. 23

C.1 Training Setup

  • CPT processed 2.19 billion tokens over 530 steps, or 2.79 passes over the 784-million-token corpus; the MoE load-balancing loss contributed zero because of a framework bug.Despite the lack of effective load-balancing pressure, the dense and MoE runs converged without diverging or loss anomalies beyond data-driven gradient spikes.Source: §C.1 · p. 25

C.2 Configuration Sweeps

  • The configuration sweep varied dense-model step counts and tested eight learning rates from 1.0×10−5 to 5.0×10−5 for the MoE model, evaluating with 5-shot HKCanto-Eval.Source: §C.2 · p. 26

C.2.1 Step Count & the Final Stretch

  • 65.07, 65.12, and 64.85: the 8B model's average scores across 520, 530, and 540 steps were nearly flat, with the 530-step run selected.Cultural scores rose from 64.68 to 67.46 across those runs, while linguistic scores fell; the remaining three categories moved by less than 0.3 points.Source: §C.2.1 · p. 26

C.2.2 Training Stability

The selected CPT runs were stable, while the supplied appendix passages describe merged-checkpoint traces, remaining gaps, SFT data, and training configurations. The data combine translated reasoning, replay, and Cantonese-focused instruction examples, with distinct settings for the two models.

C.2.2 Training Stability

  • Both selected CPT runs were stable; gradient-norm excursions occurred at matching steps and were amplified by learning rate without changing their locations.The selected 30B-A3B and 8B models moved by 0.044% and 0.085%, respectively, during CPT.Source: §C.2.2 · p. 26, Figure 6 · p. 26

D Merged-Checkpoint Traces and Per-Category Results

  • The merged-model probes show both 8B models reasoning in Simplified Chinese, while the merged 30B-A3B trace begins in Cantonese and later switches to Written Chinese.The supplied tables report HKCanto-Eval results for CPT runs and official or merged checkpoints, but their captions do not provide cell values.Source: Table 21 · p. 27, Table 19 · p. 27, Table 20 · p. 27

D.1 Router Behaviour under a Small-Corpus Adaptation

  • Router behaviour under small-corpus adaptation remains open; the proposed English-token routing explanation is a possibility, not an established result.The authors say per-expert utilisation comparisons across base, CPT, and merged checkpoints would test whether routing shifted and whether English regression followed it.Source: §D.1 · p. 29

D.2 Remaining Gaps after Merging

  • Because merging does not transfer the Qwen3 models’ written-Chinese-to-Cantonese translation ability, SFT was added using Cantonese instructions, optional chain-of-thought, and English replay data.Translation is described as a key task for converting English or Written Chinese resources into Cantonese for future training data.Source: §D.2 · p. 29

E The Supervised Mixture in Detail

  • The SFT mixture includes Cantonese reasoning and examination data, local knowledge and narrative examples, multi-turn dialogue, translation tasks, and expert-curated language instruction.The multi-turn examples target natural English code-mixing, while the linguistically targeted subset comprises 19 expert-curated examples and 112 expansions.Source: E The Supervised Mixture in Detail · pp. 29–30

E.1 Distillation-like Data

  • With no Cantonese reasoning corpus available, the distillation-like set translates questions, reasoning, and answers into a 25% English, 25% Traditional Written Chinese, and 50% Cantonese blend.This class contains 22,120 rows and 57.8M tokens, with 46.9% reasoning tokens.Source: §E.1 · p. 30, Table 23 · p. 32

E.2 Replay Data

  • Replay includes English-language coding data and multilingual prompts whose reasoning traces remain in English; reasoning in those five replay languages is not the aim.The multilingual replay prompts cover Japanese, German, Italian, Spanish, and French.Source: §E.2 · pp. 30, 32

F Supervised Training Configuration and Telemetry

  • The SFT configurations differ by model and framework, and compute constraints prevented a learning-rate sweep or retraining the 8B model in NeMo-RL.Most Cantonese SFT examples were repeated, and 25% of the 8B training entries lacked CoT traces; the 30B-A3B model used the full dataset. The 30B-A3B run used NeMo-RL, while 8B used LLaMA-Factory.Source: F Supervised Training Configuration… · p. 32, §F.1 · p. 32, §F.2 · p. 32

F.4 Training Stability

Both SFT runs remained stable through three epochs, with limited validation-loss improvement late in training. The gradient norms differed, but framework and batch-size differences prevent direct magnitude comparisons.

F.4 Training Stability

  • Both SFT runs were stable and showed no overfitting over three epochs.Validation losses decreased monotonically, from 0.8359 to 0.7467 for 8B and from 0.7397 to 0.6930 for 30B-A3B.Source: §F.4 · p. 33, Figure 9 · p. 34
  • The third epoch added little validation-loss improvement: 96% of the 8B model’s total improvement occurred in its first two epochs.The 30B-A3B run reached 0.6951 at epoch 1.90 and changed by 0.002 over the remaining 1.1 epochs.Source: §F.4 · p. 33
  • The 30B-A3B gradient norm crossed the clipping threshold at isolated steps, while 8B stayed roughly an order of magnitude below it; their absolute magnitudes are not directly comparable.The authors report that the MoE gradient spike did not affect the loss trajectory or convergence.Source: §F.4 · p. 33

G Preference Data Construction

Preference data were generated under plain and verbose prompting conditions, then individually scored by a judge using five dimensions. The judge did not see the system message and therefore could not identify the generation condition.

G Preference Data Construction

  • Each candidate received five integer scores from 1 to 5: relevancy, complete CoT, complete response, CoT Cantonese, and overall quality.The selection table describes normalised judge dimensions and a separate reasoning-language band for the composite score.Source: G Preference Data Construction · p. 34, Table 27 · p. 34

H.1 Candidate Generation

  • Each prompt generated four candidates: two samples under each of the plain and verbose conditions, with the same user turn across conditions.The plain condition preserved the row’s system message; the verbose condition appended or inserted an addon, and 85.5% of rows originally had no system message.Source: §H.1 · pp. 34, 36

H.2 Judging

  • The judge scored candidates separately without seeing their system messages, so it could not tell which generation condition produced a response.Scoring yielded 243,400 unique judgements per model, and 60,848 of 60,850 prompts retained at least one scored candidate.Source: G Preference Data Construction · p. 34, §H.2 · p. 36
  • The judge rubric assessed reasoning-block completeness and Cantonese alongside response completeness, relevancy, and holistic quality.A response with no reasoning block received the lowest score for both complete CoT and CoT Cantonese.Source: §H.2 · p. 36

I Preference Training Configuration and Telemetry

Preference training reached high validation accuracy, but its telemetry shows different learning dynamics across model sizes. The appendix also details reinforcement training costs and attempts, and how its verifiable environments were constructed.

I Preference Training Configuration and Telemetry

  • The 8B preference run used 14 GPU-hours, while 30B-A3B used 115 GPU-hours—more than its 107-hour SFT despite processing 14,679 pairs.RLVR was later restricted to 16 GPUs because the work did not secure comparable GPU availability again.Source: I Preference Training Configuration… · p. 37

I.1 Optimisation Dynamics

  • 98.5% validation preference accuracy at step 100 for 8B rose to 99.2%, while 30B-A3B settled at 92.6%.The rejected-response reward fell to −75.91 for 8B but remained positive for 30B-A3B, whose gradient norm stayed near 50; the telemetry alone did not indicate instability.Source: §I.1 · pp. 37–38, Figure 11 · p. 38

J Reinforcement Configuration, Failed Attempts and Cost

  • The auxiliary reward term r_a averages three bounded curves scoring reasoning markers, response-content length, and reasoning-block length; it cannot rescue generations that fail the format check.The reasoning-marker count uses a 40-term set selected by each row’s language tag.Source: J Reinforcement Configuration, Failed… · p. 38

J.1 Stage 2 Optimisation and the Failed Attempts

  • A 30B-A3B reinforcement attempt reached 0.492 validation reward while approximate policy entropy collapsed by 82%; raising only the upper clipping bound widened reinforcement of low-probability positive-advantage tokens.The first two 30B-A3B attempts showed entropy collapse, and a gradient spike was observed during LoRA training.Source: §J.1 · pp. 39–40, Figure 13 · p. 43

J.2 Cost of the Reinforcement Stage

  • RLVR consumed 1,697 GPU-hours, or 2,472 including failed attempts, versus 327 GPU-hours for continued pre-training, SFT, and preference optimisation combined.The second-stage runs were generation-bound; some 30B-A3B rollouts reached the 16,384-token limit.Source: §J.2 · p. 40

K Environment Construction

  • The environment construction treats verifiable constraints as language-specific when they cannot be transferred by translation, and describes a procedure reusable across target languages with different orthographies.Source: K Environment Construction · p. 40

K.1 Constraint-Type Triage

  • Thirteen constraint types were added across six families, after triaging 48 existing English constraint types against the target languages.The two variants differ only by four types without a written Chinese equivalent.Source: §K.1 · p. 40, Table 35 · p. 41

K.2 Structured Outputs

  • Structured-output environments convert records among JSON, YAML, and TOML, extract labelled records, and generate schema-valid examples; each shipped row was re-verified against its grader.The schema derives from each source document’s real metadata; on the translation probe, preference checkpoints gave more extensive Cantonese reasoning than supervised checkpoints.Source: §K.2 · pp. 40–41, Figure 12 · p. 42

Related papers

Loading 2609.06970v1…