Source-linked AI summary
Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning
Kai Chen, Jifeng Ding, Ning Ding, Jiaye Ge, Lixin Gu, Yicheng Gu, Qipeng Guo, Ermo Hua, Haian Huang, Haozheng Hou, Jie Hou, Xiangyu Hong, Che Jiang, Minxi Jin, Cheng Liang, Dahua Lin, Dawei Liu, Kuikun Liu, Chengqi Lv, Haijun Lv, Han Lv, Ningsheng Ma, Biqing Qi, Jianmin Qian, Shiya Su, Youbang Sun, Huanze Tang, Zhongbo Tian, Hanjing Wang, Rui Wang, Ting Wang, Yi Wang, Baiting Wu, Jun Xu, Bowen Yang, Hui Wang, Weida Wang, Haochen Ye, Jiashuo Yu, Shan Yu, Xiaoyi Yu, Qirui Zeng, Qi Zhang, Ming Zhang, Wenwei Zhang, Bowen Zhou, Xinyu Zhou
TL;DR
Foundation models face diminishing scaling returns and costly, verbose long-chain reasoning. Mobius decouples knowledge storage from reasoning and matches Transformer-level performance with substantially greater data and inference efficiency.
Problem
Foundation models face diminishing scaling returns and costly, redundant long-chain reasoning, motivating architectures that improve capability and efficiency together.
Method
Mobius decouples shared knowledge-vector storage from self-attention reasoning, enabling recurrent latent retrieval and refinement across reasoning operators.
Results
Mobius matches Transformer reasoning performance with 0.626× the training data and substantially higher aggregated request throughput, primarily through shorter CoT lengths.
Takeaways & Limitations
Knowledge-reasoning separation offers a route to more data-efficient training and more compact inference trajectories without reducing fundamental reasoning capabilities.
Takeaways & Limitations
Whether Mobius improves real-world self-evolution remains unvalidated and requires joint design of foundation models, agentic systems, and environmental feedback.
Abstract
from arXiv · showhide
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning. Using hidden states as cache and carrier, reasoners repeatedly query memory for required knowledge-vectors, while the knowledge is transmitted back to reasoning operators. Through this knowledge-reasoning-separation architecture, Mobius achieves better knowledge compression and reasoning efficiency. Built upon Mobius-v0 architecture: 1) Our 7B model trained-from-scratch achieves similar downstream score as a 7B Transformer baseline with 62.6% of baseline's training data. 2) Our Intern-S2-Mobius, continually-pretrained from Qwen3.5-35B, achieves similar downstream score while delivering nearly 4x end-to-end inference speedup.
1. The development bottleneck of the Foundation Models
The section identifies diminishing returns from scaling and verbosity from long chains of thought as development bottlenecks. It proposes Mobius, which decouples knowledge vectors from reasoning operators through shared storage to improve reasoning efficiency.
- Development bottlenecks: Scaling data and parameters is approaching diminishing marginal returns as the Scaling Law gradually plateaus.The paper frames this as a limit of the conventional scaling path.
- Development bottlenecks: Long chains of thought can produce verbose, tangential outputs even when problem difficulty does not require them.The issue is noted in mathematics, code, and physics.
- Mobius architecture: Mobius increases architectural complexity by decoupling knowledge vectors in FFN from reasoning operators in Self-Attn.This separation constructs a shared knowledge vector database accessible to all reasoning operators.
- Efficiency gains: Mobius improves reasoning efficiency through a more flexible inference activation path and more dynamic latent-space iteration.Together, these mechanisms support a more efficient and concise output pattern.
- Efficiency gains: Mobius completes the same reasoning task with fewer high-quality tokens, achieving higher end-to-end inference efficiency than Transformer.The efficiency advantage is attributed to the architecture’s reasoning process.
2. What inspired the design of Mobius?
Mobius is inspired by the conventional division of labor in Transformers—FFNs store knowledge, hidden states transmit information, and self-attention performs compositional reasoning—and by the limitations of hierarchical, unidirectional information flow. Because token-mediated long chains of thought are computationally inefficient, Mobius redesigns the architecture around shared knowledge access and latent reasoning.
- FFNs conventionally store knowledge, hidden states transmit information, and self-attention performs compositional reasoning in Transformer architectures.
- Hierarchical forward propagation restricts shallow hidden states from accessing deep knowledge and deep self-attention from processing knowledge originating in shallow layers.
- Token-mediated chains of thought activate additional FFN knowledge across inference steps, but lengthy redundant reasoning chains expend substantial computation.
- Long CoT is costly because longer chains generate more tokens and generation cost grows non-linearly with chain length.
- Mobius decouples the FFN from layer-wise binding into a globally shared knowledge-vector database, giving every self-attention operator access to the full knowledge body.
3. How about Mobius’ performance?
Mobius is evaluated through training-from-scratch and continual-pretraining experiments, showing higher data efficiency and preserved or improved capabilities. At scale, its inference advantage is associated with shorter chain-of-thought reasoning, with case evidence indicating fewer tokens without changing the correct answer.
- Evaluation setup: Mobius is evaluated with 7B-A1B training-from-scratch models and Qwen3.5-35B continual pre-training, followed by supervised fine-tuning and reinforcement learning.Both experiment tracks use 1TB tokens; the continual-pretraining track starts from Qwen3.5-35B-A3B.
- Reasoning behavior: Both models select the correct answer in a linear-algebra case, while Mobius completes the same reasoning steps with fewer tokens.The comparison attributes the token reduction mainly to eliminating repeated derivations and checks.
- Training-from-scratch performance: 0.626× data: Mobius reaches Transformer’s 1TB-token MMLU score, corresponding to 1.6× Transformer’s data efficiency.Mobius achieves significantly higher MMLU scores than Transformer across all training stages.
- Continual-pretraining performance: Switching from Qwen3.5 to Mobius for continued pre-training preserves and enhances overall capabilities.The evaluation also reports preserved fundamental reasoning capabilities after the architectural switch.
- Inference efficiency: Mobius achieves substantially higher request throughput than Transformer, primarily because its chain-of-thought lengths are shorter.This result is aggregated across multiple evaluation benchmarks and concerns end-to-end inference efficiency.
4. Mobius’ relationship with mainstream research
Mobius relates to latent-reasoning, recurrent, and multi-token inference methods by repeatedly refining dense latent states against shared knowledge Memory. Architecturally, it prioritizes information density and bidirectional knowledge access over directly reducing attention complexity or expanding residual pathways.
- Latent reasoning: Mobius makes latent reasoning native by refining recurrent states against shared Memory, jointly supporting multiple future tokens with higher-density reasoning.This contrasts with approaches that expose a small set of reusable latent states as shared context.
- Looped computation: Mobius applies recurrent updates over only a few layers while retaining full Memory access, enabling more iterative refinement and higher-information-density latent states.Its higher update frequency concentrates reasoning into compact latent representations.
- Latent computation steps: Mobius performs recurrent latent refinement without placeholders or diffusion steps, repeatedly retrieving knowledge vectors, refining latent states, and decoding multiple tokens in parallel.This extends methods that add latent computation steps before generation.
- Inference efficiency: Mobius performs multiple internal knowledge traversals and multi-token iterations before decoding, unlike speculative decoding, which traverses knowledge once before token prediction.Mobius therefore produces higher-information-density hidden states with native multi-token prediction.
- Attention mechanisms: Mobius decouples knowledge from reasoning and uses a globally shared repository to provide attention with flexible selection and higher-quality inputs.Rather than directly reducing attention complexity, this approach aims to complete reasoning with a more compact computation trajectory.
- Residual connections: Mobius extends residual information flow toward bidirectional knowledge access, allowing deeper Reasoners to retrieve beyond their local layer hierarchy.All reasoning stages access the same knowledge repository while maintaining an efficient computation structure.
5. Mobius’ potential on several highlight topics
The section presents Mobius as a knowledge–reasoning-separated architecture with potential for self-evolution, world modeling, scientific discovery, and further scaling beyond current Transformer limitations. These potentials remain conditional on additional system, algorithmic, and hardware advances and require future validation.
- Self-evolution: Mobius separates knowledge and reasoning to address Transformer’s coupled skill acquisition and catastrophic forgetting, but real-world self-evolution remains unvalidated.The passage identifies joint design of foundation models, agentic systems, and environmental feedback as necessary for evaluation.
- World models: Mobius targets continuous-space understanding and reasoning, which Transformer-based world models may require parameter scales on the order of petabytes to model perfectly.The passage also states that terabyte-scale language-model inference already strains existing hardware.
- Scientific discovery: Mobius addresses scientific discovery by internalizing deliberation and supporting knowledge composition and generalization beyond Transformer-plus-long-chain-of-thought procedural problem-solving.Whether it improves real-world scientific discovery depends on jointly optimizing data, infrastructure, training algorithms, and optimization algorithms.
- Scaling: Mobius separates knowledge and reasoning, potentially allowing only reasoning-dedicated parameters to remain stably retained in GPU memory and enabling further scaling beyond current hardware limits.The passage characterizes long-term hardware–software co-design as promising but requiring substantial additional effort.
A. Author List · A.1. Core Contributors (Sorted by Contribution) · A.2. Full List (Sorted by Character)
The paper credits contributors by role across architecture, training, and inference, alongside project advisors and a full character-sorted author list. Core contributor lists identify overlapping participation across these technical areas.
- A.1. Core Contributors (Sorted by Contribution): Architecture contributors are Ermo Hua, Xiangyu Hong, Che Jiang, Baiting Wu, Cheng Liang, Youbang Sun, Biqing Qi, and Qipeng Guo.
- A.1. Core Contributors (Sorted by Contribution): Training contributors include Baiting Wu, Chengqi Lv, Ermo Hua, Xiangyu Hong, Weida Wang, Ning Ding, and Wenwei Zhang.
- A.1. Core Contributors (Sorted by Contribution): A second training group comprises Hanjing Wang, Xiangyu Hong, Yicheng Gu, Ermo Hua, Shan Yu, Haozheng Hou, Jianmin Qian, Jie Hou, Zhongbo Tian, and Hui Wang.
- A.1. Core Contributors (Sorted by Contribution): Inference contributors are Qian Yao, Baiting Wu, Ermo Hua, Jifeng Ding, Ningsheng Ma, Han Lv, Minxi Jin, Zhongbo Tian, and Hui Wang.
- A.1. Core Contributors (Sorted by Contribution): Project advisors are Qi Zhang, Kai Chen, Dahua Lin, and Bowen Zhou.
- A.2. Full List (Sorted by Character): The full author list, sorted by character, contains 44 named contributors spanning the technical teams and advisory group.The list includes Kai Chen, Jifeng Ding, Ning Ding, Jiaye Ge, Lixin Gu, Yicheng Gu, Qipeng Guo, Ermo Hua, Haian Huang, Haozheng Hou, Jie Hou, Xiangyu Hong, Che Jiang, Minxi Jin, Cheng Liang, Dahua Lin, Dawei Liu, Kuikun Liu, Chengqi Lv, Haijun Lv, Han Lv, Ningsheng Ma, Biqing Qi, Jianmin Qian, Shiya Su, Youbang Sun, Huanze Tang, Zhongbo Tian, Hanjing Wang, Rui Wang, Ting Wang, Yi Wang, Baiting Wu, Jun Xu, Bowen Yang, Hui Wang, Weida Wang, Haochen Ye, Jiashuo Yu, Shan Yu, Xiaoyi Yu, Qirui Zeng, Qi Zhang, and Ming Zhan.
B. Expert Activation Patterns
Expert activation differs by training recipe: Mobius-7B shows comparatively uniform activation across reasoning layers, whereas Intern-S2-Mobius-35B uses a wider range while retaining a pronounced block-diagonal pattern. These observations suggest continual pre-training broadens expert utilization without fully removing the routing prior induced by architectural conversion.
- Mobius-7B trained from scratch exhibits a comparatively uniform activation distribution across reasoning layers.
- Intern-S2-Mobius-35B activates experts over a wider range while retaining a pronounced block-diagonal pattern.The model was continually pre-trained from Qwen3.5-35B.
- Continual pre-training can broaden expert utilization without fully removing the routing prior induced by architectural conversion.Training Mobius from scratch may permit more flexible access to the shared expert pool, but the capability impact requires controlled evaluation at comparable scale.
C. Toy Task of Compositional Generalization
On a compositional generalization task, Mobius was compared with Transformer and achieved substantially better convergence efficiency and final scores. The task tests two-hop knowledge composition for entities held out from training.
- C. Toy Task of Compositional Generalization: Mobius achieved substantially better convergence efficiency and final scores than Transformer on the compositional generalization task.The comparison used the newer Mobius architecture against Transformer, as shown in Fig. 7.
- C. Toy Task of Compositional Generalization: The training set contained single-hop knowledge for 500 entities and two-hop knowledge for 400 entities.These examples formed the task’s training data.
- C. Toy Task of Compositional Generalization: The test set contained two-hop knowledge for the remaining 100 entities, evaluating compositional generalization beyond the training entities.The held-out entities differed from those used for the single-hop and two-hop training examples.
D. Case Study: Chain-of-Thought Reasoning
The case study examines chain-of-thought reasoning in biology and shows that Mobius produces a shorter reasoning trace than Transformer.
- The case study presents a biology example of chain-of-thought reasoning.
- Mobius delivers a shorter reasoning trace in this biology example.
- The biology example directly compares Mobius’s reasoning trace with Transformer’s.
E. Layerwise Analysis of Latent Reasoning
A layerwise prediction lens evaluates how Mobius’s token predictions evolve during latent computation under identical teacher-forced contexts. Mobius shows more target-aligned and coherent intermediate-layer predictions than the baseline across standard and subsequent MTP positions.
- Layerwise prediction lens: The analysis decodes each layer’s hidden state for next-token prediction t+1 and four subsequent MTP positions, t+2 to t+5.The context is held constant through teacher forcing.
- Layerwise prediction lens: Mobius exhibits more interpretable, target-aligned intermediate predictions and a more coherent semantic trajectory than the baseline.These observations come from the layerwise prediction comparison shown in Figure 8.
- Layerwise prediction lens: Figure 8 compares Mobius with Qwen3.5 under identical teacher-forced contexts across standard next-token and subsequent MTP predictions.Color intensity indicates predicted-token probability, and black outlines mark agreement with the target continuation.