PaperlayerView exact revision on arXiv

Source-linked AI summary

Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM

Yang Liu, Xiaolong Zhong, Ling Jiang

arXiv:2511.19496v1cs.LGcs.AI

TL;DR

Large language models’ reasoning and tool-use capabilities are costly to deploy as autonomous agents in resource-constrained settings. Xmodel-2.5 is a 1.3-billion-parameter model trained with a µP-based recipe, and it ranks second among 1–2B models on 13 reasoning benchmarks using 1.4T tokens.

  • Problem

    LLMs show strong reasoning and tool-use, but their deployment as autonomous agents is prohibitive in resource-constrained settings; improving agent capability in the 1–2B range is the stated goal.

    Source: §1 · p. 1

  • Method

    Xmodel-2.5 retains a two-stage pre-training recipe and µP, with a WSD curriculum, FP8 mixed precision, and an AdamW-to-Muon switch during decay.

    Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 2

  • Results

    52.49% vs. 56.96% (Qwen3): Xmodel-2.5 ranks second among 1–2B models on 13 reasoning benchmarks, using 1.4T tokens—25.7× fewer than Qwen3.

    Source: §6 · p. 4

  • Takeaways & Limitations

    A 10k-step long-context adaptation phase delivers reliable 16k-context reasoning without extra data or hyper-parameter tuning.

    Source: §6 · p. 4

Abstract

from arXiv · show

Large language models deliver strong reasoning and tool-use skills, yet their computational demands make them impractical for edge or cost-sensitive deployments. We present \textbf{Xmodel-2.5}, a 1.3-billion-parameter small language model designed as a \emph{drop-in agent core}. Training with maximal-update parameterization ($\mu$P) allows hyper-parameters tuned on a 20M-parameter proxy to transfer directly to the full model, even under the parameter-tied \emph{tie-word-embedding} architecture. A 1.4T-token Warmup--Stable--Decay curriculum is used, and we further show that \textbf{switching from AdamW to Muon during the decay phase} improves the 13-task reasoning average by 4.58\,\% while keeping every other hyper-parameter fixed, verifying that early AdamW stability can be paired with late Muon sharpening for better downstream performance. FP8-mixed-precision training balances accuracy and throughput. All checkpoints, recipes, and evaluation code are released under the Apache-2.0 license.\footnote{https://huggingface.co/XiaoduoAILab/Xmodel-2.5 and https://huggingface.co/XiaoduoAILab/Xmodel-2.5-history (training checkpoints).} Training code and evaluation harness: https://github.com/XiaoduoAILab/Xmodel-2.5.

1 Introduction

Large agent backbones make deployment in resource-constrained settings difficult, while small models still fall short on complex multi-step reasoning. Xmodel-2.5 targets stronger reasoning in the 1–2B range with upgrades to its training and tokenizer.

1 Introduction

  • Small language models can match larger models on single-turn benchmarks but still fall short on complex multi-step reasoning needed for agent skills.The stated target is improving this capability within the 1–2B parameter range.Source: §1 · p. 1
  • Xmodel-2.5 is a 1.3B decoder-only model retaining Xmodel-2’s two-stage pre-training recipe and µP, with four targeted upgrades.The µP implementation modifies parameterization, attention scaling, and residual connections and was validated to preserve µP dynamics for hyperparameter transfer.Source: §1 · p. 1
  • The 129k-token DeepSeek-v3 tokenizer replaces Xmodel-2’s 65k-token Unigram tokenizer, improving compression and decoding speed.This is one of the paper’s targeted upgrades.Source: §1 · p. 1
  • FP8-mixed precision raised training throughput by approximately 30% with no observable degradation in pilot experiments.The paper presents this as a numeric-precision upgrade.Source: §1 · p. 1
  • Switching from AdamW to Muon during the decay phase improved the 13-task reasoning average by 4.58%, while keeping all other hyper-parameters fixed.Source: §1 · p. 1
  • The authors hope Xmodel-2.5 will serve as a minimal but strong baseline for lightweight agents with enhanced complex-reasoning capabilities.The motivating deployment setting is resource-constrained environments, where state-of-the-art agent backbones typically exceed 7–13B parameters.Source: §1 · p. 1

2 Background & Related Work

Prior small-model reasoning work emphasizes single-turn benchmarks, while multi-step agent behavior remains less systematically evaluated. Related work also identifies challenges in transferring hyperparameters and jointly applying efficient training techniques.

2.1 Small Language Models for Reasoning

  • Small language models have closed gaps with larger models on mathematical and commonsense reasoning, but systematic evaluation of multi-step agent behavior remains under-explored.Prior work primarily targets single-turn question answering.Source: §2.1 · pp. 1–2

2.2 Hyper-Parameter Transfer with Maximal-Update Parameterisation

  • µP preserves training dynamics across widths to support hyperparameter transfer, but prior Adam integrations report instability below 2B parameters.The original µP derivations assume SGD.Source: §2.2 · p. 2

2.3 Efficient Training of Lightweight Models

  • Prior work studies FP8 mixed precision and fused attention kernels for efficiency, while WSD schedules decouple late-stage annealing from token count.The paper says these techniques have not been jointly studied with µP optimizers and describes domain-weighted data mixing during decay as an ablation absent from prior literature.Source: §2.3 · p. 2

3 Methodology

Xmodel-2.5 scales a deployment-friendly decoder-only design to 1.3B parameters. Its methodology combines µP-compatible architecture, a multi-phase WSD curriculum with a late optimizer switch, and FP8 mixed-precision acceleration.

3.1 Model Architecture

  • The deep-and-thin decoder-only model applies µP attention-logits scaling of 1/dhead while retaining Pre-RMSNorm and SwiGLU.Table 1 notes long-context support via 131K position embeddings and a RoPE base of 500K.Source: §3.1 · p. 2, Table 1 · p. 2

3.2 Three-Phase WSD Pre-Training

  • The 560k-step Warmup–Stable–Decay (WSD) schedule consumes 1.4T tokens, followed by 10k additional Long-Context Reasoning (LCR) steps at 16k context length.Source: §3.2 · p. 2
  • During the final 20k decay steps, replacing AdamW with Muon raises the 13-task reasoning average by 4.58 pp absolute over an AdamW-only baseline.The learning-rate schedule, batch size, gradient clip, weight decay, and 66.9% SFT blend remain fixed.Source: §3.2 · p. 2

3.3 FP8 Hybrid Format & Kernel Implementation

  • The FP8 hybrid format uses E4M3 for activations, E5M2 for gradients, and bfloat16 master weights; Megatron-LM selects the FP8 kernels.The implementation uses Transformer Engine and automatically selects GEMM, Layer-Norm, and GeLU CUDA kernels without source-code modification.Source: §3.3 · pp. 2–3

3.2 Three-Phase WSD Pre-Training

  • Stable-phase data emphasizes broad pre-training diversity, while decay focuses on high-quality instructional and SFT data.This is the data-composition distinction shown for the two WSD phases.Source: Figure 1 · p. 3

4 Experimental Settings

Experiments compare Xmodel-2.5 against publicly released 1–2B decoder-only models using a unified evaluation pipeline. The evaluation spans reasoning categories and reports task-specific metrics.

4.1 Baselines

  • The baseline set comprises eight publicly released decoder-only models in the 1–2B range, with results reproduced through a unified evaluation pipeline.The pipeline uses the Language Model Evaluation Harness to ensure fair comparison.Source: §4.1 · p. 3

4.2 Tasks and Metrics

  • Evaluation spans 13 datasets in commonsense, symbolic and multilingual reasoning, plus mathematical and code tasks, with accuracy for all sets except HumanEval/MBPP (pass@1) and GSM8k/MATH (exact-match).GSM8k uses flexible-extract (5-shot), while MATH is verified by math_verify (4-shot).Source: §4.2 · pp. 3–4

5 Results

Xmodel-2.5 improves the 1–2 B models’ 13-benchmark reasoning average and remains close to Qwen3 despite using fewer parameters and training tokens. Long-context adaptation raises the reasoning average despite a small WikiText-2 perplexity increase.

5.1 Main Reasoning Results

  • 52.49%: Xmodel-2.5 closes 71% of the gap to Qwen3, raising the 1–2 B average from 50.34% (Xmodel-2) to 52.49%.It is 25% smaller than Qwen3 and trained with 96% fewer tokens.Source: §5.1 · p. 4

5.2 Training Loss

  • The WikiText-2 loss curve shows an initial drop after batch size increases from 2M to 4M tokens, followed by a second drop during learning-rate decay.The text says the batch-size change likely replicates the stabilizing effect of a reduced learning rate.Source: §5.2 · p. 4
  • 52.49: After longer-context exposure, the 13-task average rises from 52.36 to 52.49 despite a small WikiText-2 perplexity bump.The context length expands from 3,712 to 8,192 and then to 16,384.Source: §5.2 · p. 4

6 Conclusion

Xmodel-2.5 is a 1.3B-parameter model that ranks second among 1–2 B models on 13 reasoning benchmarks, behind Qwen3. Its WSD recipe adds a short long-context adaptation phase and reaches 16k-context reasoning without extra data or hyperparameter tuning.

6 Conclusion

  • The authors present Xmodel-2.5 as a 1.3-billion-parameter small language model.They hope the findings encourage further exploration of data-efficient pathways toward capable small models.Source: §6 · p. 4
  • A small language model achieves the second-best average score among 1–2B models on 13 widely used reasoning benchmarks, trailing only Qwen3, scoring 52.49% versus 56.96%, with 1.4T training tokens—25.7× fewer.Source: §6 · p. 4
  • A 10k-step long-context adaptation phase extends the WSD schedule to reliable 16k-context reasoning without extra data or hyperparameter tuning.The authors describe the recipe as simple, compute-friendly, and reproducible.Source: §6 · p. 4
Loading 2511.19496v1…