Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch

• The work targets code-switched reasoning as a possible route to improve reasoning capabilities for lower-resource languages, where existing reasoning models are mostly limited to high-resource languages.Source: §1 · p. 1
• The authors ask how LLMs code-switch during reasoning, which switching types help reasoning performance, and which fine-tuning interventions can teach beneficial forms.Source: §1 · pp. 1–2
• The study creates a dataset of reasoning traces from diverse models, languages, tasks, and domains, then uses supervised fine-tuning and analyzes resulting reasoning traces.Source: §1 · p. 2
• The CoRe corpus contains ~7,000 reasoning traces, while the framework adds 1M tokens of training data per task per language across 6 tasks and 7 languages.Source: §1 · p. 2

• Unlike prior studies, the work covers 17 models from 10 families, 21 languages, and diverse reasoning domains and tasks beyond those previously examined.Source: §2 · p. 2
• The paper introduces a structured taxonomy across three dimensions and quantifies code-switching with 9 metrics validated by Linguistics and NLP literatures.Source: §2 · p. 2
• Its classification combines theory-driven and data-driven analysis with manual curation grounded in real-world observations and theories of code-switching.Source: §2 · pp. 2–3
• Instead of prompt engineering, test-time interventions, probes, or logit masking, the authors use supervised fine-tuning to modify reasoning models' underlying code-switching behavior.Source: §2 · p. 3

3.1 Approach

• The framework fuses theory-driven and data-driven taxonomy construction, using model-generated criteria, topic modeling, and manual curation informed by code-switching and reasoning literature.This combination is intended to capture novel model behaviors while retaining alignment with human code-switching concepts.Source: §3.1 · pp. 3–4

3 Understanding Code-Switched Reasoning Behaviors

• CoRe contains about 7,000 reasoning traces from 15 models spanning diverse languages, datasets, tasks, and domains for studying code-switched reasoning.The corpus covers 18 languages, 10 scripts, 8 language families, and prompts from seven datasets.Source: §3.1 · p. 3, §3 · p. 3

3.1 Approach

• The taxonomy organizes code-switching into function, form, and coherence, covering purpose, structural extent and density, matrix language, fluency, and accuracy.Functions include translation and quoting; form ranges from words or phrases to entire traces; coherence concerns naturalness and semantic appropriateness.Source: §3.1 · p. 4
• Manual validation found Gemini 2.5 Flash generated plausible descriptions of code-switching 85% of the time.The authors randomly sampled and manually annotated 100 brainstormed code-switching criteria.Source: §3.1 · p. 4
• The study quantifies code-switching with established metrics, including CMI, M-Index, Memory, Burstiness, and I-Index, and models correctness with language- and model-level random effects.CMI and M-Index quantify multilingual balance or dominance, Memory captures similarity in consecutive span lengths, Burstiness captures switching predictability, and I-Index captures adjacent-language switching probability.Source: §3.1 · pp. 4–5

3.2 Results

• Code-switching varies by prompt language: high-resource languages show more accurate switching, whereas low-resource prompts more often produce reasoning dominated by another language.Chinese commonly serves as the matrix language for Chinese prompts, while English and Chinese frequently dominate reasoning prompted in other languages.Source: §3.2 · pp. 5–6, Figure 2 · p. 6
• A different matrix language from the prompt can benefit reasoning performance, while matrix-language dominance matters more than syntactic integration.In practice, the matrix language is often a higher-resource language such as Chinese or English, whereas increased code-switching fluency is not helpful for performance.Source: §3.2 · p. 6, Figure 2 · p. 6

4.1 Approach

• The study evaluates six interventions using a shared supervised fine-tuning format and a 1-million-token budget per model, language, and task.The evaluated models represent different sizes, families, and multilingual capabilities; Yoruba lacks released native-language and synthetic code-switched datasets because of limited high-quality data.Source: §4 · p. 6, §4.1 · p. 7
• The experiments use Global MMLU across seven languages and reasoning-focused domains, with NLLB supplying additional multilingual translation data.The selected languages span three scripts, three resource levels, four language families, and more than 1 billion speakers worldwide.Source: §4.1 · pp. 6–7
• The native-language, translation, prompt-translation, English-reasoning, and code-switched tasks operationalize distinct hypotheses about how models can develop beneficial multilingual reasoning.Translation tasks target translation into English, English-reasoning uses English intermediate reasoning with a target-language answer, and native-language reasoning provides a monolingual baseline.Source: §4.1 · pp. 7–8
• Strategically code-switched reasoning uses teacher-generated multilingual traces, while synthetic code-switching randomly interleaves English and the target language to test coherence.The synthetic condition is designed to contrast coherent teacher-produced switches with randomly generated ones.Source: §4.1 · p. 8

4.2 Results

• English as the matrix language, semantic accuracy, and greater code-switching by CMI and M-Index benefit performance, whereas overly dense switching can harm correctness.The Integration Index has a negative effect, indicating that switching should occur without high token-level density.Source: §4.2 · pp. 8–9, Figure 3 · p. 9
• Translation fine-tuning significantly increases beneficial code-switching measures and improves code-switching accuracy when translating reasoning prompts into English.Machine-translation fine-tuning raises the Code-Mixing Index and Multilingual Index, while reasoning-prompt translation improves code-switching accuracy.Source: §4.2 · p. 9, Figure 3 · p. 9

• The framework identifies and fine-tunes code-switched reasoning behaviors using data, code, and models.Source: §5 · p. 9
• Matrix language, semantic accuracy, and degree of code-switching all affect reasoning performance.Source: §5 · p. 9
• The interventions significantly increase helpful code-switching aspects and establish code-switching as a means for improving reasoning in languages with limited data resources.Source: §5 · p. 9

• LLMs are used to generate data visualizations in addition to their other roles in the paper.Source: §A.1 · p. 22

• The taxonomy-development models include DeepSeek-R1 variants, Gemini, Claude, Llama, Gemma, Phi, QwQ, Magistral, and Kimi.Source: §A.2 · p. 22

• The model set spans systems with broad multilingual coverage and smaller models supporting selected European, Asian, and African languages.Gemma 3 is assumed to cover 140+ languages, while other models list substantially narrower language sets.Source: §A.3 · pp. 22–23
• The selected models include proprietary and open models such as Gemini, Claude, Magistral, Kimi, DeepSeek, Llama, Phi, and Gemma variants.The passages list these systems alongside their supported and unsupported languages for study selection.Source: §A.3 · pp. 22–23
• DeepSeek distilled models and QwQ-32B share English–Chinese support while also being evaluated with additional languages.The listed sets include languages such as Arabic, Hindi, German, French, Portuguese, Japanese, Korean, Bengali, Swahili, Indonesian, and others.Source: §A.3 · pp. 22–23

• The dataset suite covers coding, math, puzzles, science, legal reasoning, logic, and humanities.These domains are represented in the s1K-1.1 and s1K-Claude datasets.Source: §A.4 · p. 23
• Global MMLU and MMLU ProX extend coverage across mathematical, scientific, humanities, medical, business, legal, logic, commonsense, linguistic, and spatial tasks.Global MMLU covers six broad domains, while MMLU ProX and BIG-Bench Extra Hard add specialized reasoning categories.Source: §A.4 · p. 23
• Additional sources target cultural and social reasoning plus mathematical reasoning traces.UniMoral, OpenMathReasoning, and Kimi-K1.5-Distill-data provide complementary data sources.Source: §A.4 · p. 23

• Examples are newly generated for BBEH, UniMoral, Global MMLU Lite, and MMLU ProX Lite, while remaining experimental data comes from preexisting reasoning datasets.This combines new generation with reused reasoning examples.Source: §A.5 · p. 23
• BBEH and UniMoral generation uses temperature 0.6, top_p 0.95, and up to 32,768 tokens for the specified models.These settings follow recommendations for the DeepSeek distilled models and Phi-4-mini-reasoning.Source: §A.5 · p. 23
• Global MMLU Lite uses model-dependent sampling, with temperature 0.6 or 1.0 and top_p 0.9 or 1.0, while retaining a 32,768-token maximum.DeepSeek-R1 is run with API defaults and the January 2025 checkpoint.Source: §A.5 · pp. 23–24
• MMLU ProX Lite uses distinct settings across Llama, Gemma, and DeepSeek models, including reasoning mode for Llama-3.1-Nemotron-Nano-8B-v1.Some models use temperature 0 with shorter or longer token limits, while others use temperature 0.6 and top_p 0.95.Source: §A.5 · p. 24

A.6 Prompt for brainstorming code-switched reasoning behavior taxonomy

• The brainstorming prompt supplies a problem and segmented reasoning trace, then asks for named criteria describing code-switching strategies.It requires structured output inside <patterns> tags and an empty response when no switching is observed.Source: §A.6 · p. 24

A.7 BERTopic modeling details for code-switched reasoning behavior taxonomy

• The taxonomy pipeline uses BERTopic with all-mpnet-basev2 embeddings, UMAP, HDBSCAN, CountVectorizer, c-TF-IDF, and Gemini 2.5 Flash topic representations.The appendix gives settings including n_neighbors=15, n_components=5, min_cluster_size=15, and cosine or euclidean metrics.Source: §A.7 · p. 25

A.8 Human evaluation of LLM annotations / A.9 Word-level language identification (LID)

• Human evaluation asks whether automatically generated criteria are plausible, while word-level LID combines Lingua with script- and language-specific tools.Tokenization uses jieba, nagisa, PyThaiNLP, and myanmar_words for languages without whitespace word boundaries.Source: §A.8 · p. 25, §A.9 · p. 25

A.9 Word-level language identification (LID)

• 0.933 average word-level LID accuracy is obtained on 140 multilingual reasoning examples covering seven languages.The validation set uses Qwen3-Next-80B-A3B-Thinking on Global MMLU.Source: §A.9 · p. 25

A.10 Prompt for evaluating accuracy of code-switching

• Code-switching accuracy is rated from 1 to 3, where high accuracy means terms are accurate and appropriately used.The rubric distinguishes low, moderate, and high accuracy.Source: §A.10 · p. 26

A.11 Prompt for evaluating fluency of code-switching

• Code-switching fluency is rated from 1 to 3 based on comprehensibility, grammar, syntax, and smooth integration.High fluency denotes natural, easy-to-understand text with seamless switching.Source: §A.11 · p. 26

A.13 MT pipeline for SFT data synthesis

• The SFT data pipeline uses SeamlessM4T v2 or NLLB-200 for translation, Apertus for refinement, and filtering to remove malformed or excessively expanded outputs.Malay uses NLLB-200 because it supports Standard Malay, while other languages use SeamlessM4T v2.Source: §A.13 · pp. 27–28

A.14 Instance-level language identification / A.15 SFT Configuration

• Instance-level LID reaches accuracies of 0.892 for Qwen3-Next-80B-A3B-Instruct and 0.942 for Qwen3.5-27B, while SFT uses LLaMA Factory with model-, language-, and-task-specific cutoff lengths.The configuration uses LoRA, three epochs, cosine scheduling, and an effective batch size of 16.Source: §A.14 · p. 28, §A.15 · pp. 28–29