PaperlayerView exact revision on arXiv

Source-linked AI summary

Safin-1: Safety from Within through Memory-Native State Evolution

Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Zhekai Chen, Cheng Jin, Jingnan Zheng, Yi Zhang, Zhongtian Ma, Jiawei Zhou, Sirui Chen, Qiaosheng Zhang, Xiang Wang, Ning Ding, Xia Hu, Bowen Zhou, Youbang Sun, Chaochao Lu

arXiv:2609.00092v1cs.LG

TL;DR

Long-horizon models need both persistent contextual memory and capabilities that can be reliably invoked, while safety is often imposed separately from the model’s native computation. Safin-1 uses MARCH to route addressable recurrent states and persistent capability states, and its Safety State reduces jailbreak attack success while lowering over-refusal relative to matched LoRA controls. The paper presents these results as initial architecture-level evidence for Safety from Within, not a definitive solution.

  • Problem

    Long-horizon intelligence requires retaining extended context and invoking appropriate reusable capabilities, while safety is commonly represented or imposed through separate mechanisms.

  • Method

    Safin-1 uses MARCH to preserve evolving recurrent states as addressable memory and routes a detachable persistent Safety State through a frozen backbone.

  • Results

    Safety State reduces unweighted average jailbreak attack success rate across five benchmarks by 42.3% at 4B and 52.3% at 35B-A3B, with lower XSTest ORR than matched rank-8 LoRA controls.

  • Takeaways & Limitations

    The routed-state interface provides initial architecture-level evidence that model state can serve as both addressable context memory and an interface for adaptable capabilities.

  • Takeaways & Limitations

    Large-scale evidence covers two Qwen3.5 configurations, while the safety study uses a small English corpus and limited benchmarks and does not cover adaptive, multilingual, or state-tampering attacks.

Abstract

from arXiv · show

Long-horizon complex tasks require foundation models to accumulate information, maintain internal states, and adapt over extended interactions. Safety should be an intrinsic property of the model itself, rather than a behavioral constraint relying solely on external safeguards or post-hoc alignment such as supervised fine-tuning. This motivates Safety from Within, where safety-relevant capabilities are represented and invoked through the model's native computation. We present Safin-1, a family of foundation models realizing this principle through memory routing and state evolution. Safin-1 is built on Memory-Anchor Routing across Context History (MARCH), a network architecture that maintains structured memory states and selectively retrieves relevant historical information through content-conditioned routing. It supports test-time adaptation of persistent capability states without repeatedly modifying the backbone, enabling controlled specialization over a shared foundation. We investigate this interface on downstream safety tasks through a Safety State, demonstrating effective state-based adaptation with substantial safety improvements. More broadly, the routed-state interface unifies contextual memory and persistent capability adaptation within the model's native computation, reframing memory from a passive record of prior context into an active substrate for maintaining and evolving model behavior. Evaluations across general capabilities, long-context understanding, retrieval, and efficiency further validate Safin-1. These findings provide a path toward safety as a state-native and adaptively maintainable capability. This work is only an initial architectural exploration of Safety from Within, and substantial further work is needed to realize this broader vision.

1. Introduction

Safin-1 unifies long-horizon memory and persistent capability specialization through MARCH, which preserves addressable recurrent states and routes relevant history during native computation. Controlled and scaled evaluations report gains in capabilities, long-context retrieval, efficiency, and safety, including a persistent Safety State that improves safety with less over-refusal than LoRA.

  • Architecture: MARCH preserves earlier recurrent states as addressable anchors, enabling selective historical retrieval without altering the underlying recurrent update.Each anchor has a state-aware routing key, and retrieved states are fused with the current recurrent path.
  • Architecture: Safin-1 extends the routed state bank with learned persistent capability states, including a detachable Safety State for specializing a frozen backbone.The router determines how persistent and context-derived states contribute for each token.
  • Evaluation: Controlled 0.8B pretraining across Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 improves language modeling, LongBench, retrieval, and RULER NIAH, including beyond-training-context extrapolation.These matched studies isolate the contribution of the memory-routing architecture.
  • Evaluation: The 4B and 35B-A3B variants raise macroaverage scores over ten capability benchmarks from 66.79 to 69.20 and from 76.25 to 78.35, respectively.The gains are most pronounced on challenging reasoning and competition-level mathematics.
  • Evaluation: The persistent Safety State reduces average jailbreak attack success rate by 42.3% at 4B and 52.3% at 35B-A3B, with substantially less over-refusal than a training-matched rank-8 LoRA control.The backbone remains frozen during state-native specialization.
  • Efficiency: Sparse state routing selects among compressed recurrent memories rather than token-level keys, making historical retrieval practical at sequence lengths up to 128K tokens.This provides a state-level analogue of sparse attention.

2. Technical Methodology

Safin-1’s MARCH backbone unifies recurrent context memory and persistent capability states through content-conditioned routing. It checkpoints evolving states, retrieves relevant history or learned specialization, and fuses the result with the native recurrent path, with sparse routing improving efficiency.

  • MARCH State-Routing Backbone: MARCH augments recurrent computation with context-derived anchors, token-conditioned routing, and persistent capability states in one routable state bank.Selected readouts are fused residually with the current-state path, preserving the underlying recurrent update.
  • Context-Derived State Anchors: At anchor boundaries, the recurrent state is checkpointed without resetting the recurrence, preserving cumulative prefixes as addressable historical states.Each anchor position reads its aligned checkpoint to construct routing metadata, while text positions continue the base recurrent update.
  • Content-Routed State Retrieval: Each token scores causally visible anchors alongside a zero-payload null option, then retrieves a weighted state readout that is added to the current-state readout.Null probability attenuates retrieval when historical memory is unnecessary, and routing is optimized end-to-end with language modeling.
  • Persistent Capability States: Persistent capability states share the routing interface with dynamic anchors, so their token- and context-dependent contributions are jointly normalized with the null option.They are learned parameters available from the first token without consuming input positions.
  • Safety State: The Safety State is learned while the language-model backbone remains frozen and can be attached or removed without changing shared backbone parameters.This detachable state-routed specialization instantiates Safety from Within.
  • Efficient Producer–Reader Implementation: Top-4 routing at 128K tokens more than doubles end-to-end throughput versus dense routing and reduces core runtime by roughly an order of magnitude.The sparse variant restricts aggregation to the highest-scoring visible anchors with minimal performance degradation.

3. Small-Scale Validation of MARCH

Controlled 0.8B studies test MARCH across recurrent backbones and complementary capability groups. MARCH consistently improves language modeling, long-context understanding, retrieval, length extrapolation, and long-sequence efficiency.

  • General Language Modeling: MARCH achieves the best average on all eight zero-shot benchmarks, reaching 41.5 versus 40.1 for Gated DeltaNet and 40.0 for Log-Linear Attention.
  • Length Extrapolation and Associative Recall: MARCH outperforms the strongest recurrent baseline in 19 of 24 NIAH task–length settings and matches it in the remaining five.
  • Long-Context Understanding and In-Context Retrieval: MARCH is the strongest recurrent model on every LongBench task, raising the overall average to 14.9 versus 11.9 for Gated DeltaNet and 12.5 for Log-Linear Attention.
  • Long-Context Understanding and In-Context Retrieval: MARCH raises the best recurrent-baseline average on six real-world retrieval datasets from 20.5 to 23.3.
  • Architecture Analysis: MARCH improves NIAH in 11 of 12 backbone–length settings across GDN, KDA, and GDN2 while consistently improving SWDE and FDA.
  • Efficiency: At 128K tokens, Top-4 MARCH more than doubles dense-MARCH training throughput and reduces core runtime by roughly an order of magnitude.

4. Safin-1 at Scale

Safin-1 scales MARCH to dense 4B and MoE 35B-A3B backbones under matched training protocols, improving challenging capabilities while maintaining broad performance. A detachable persistent Safety State further targets safety specialization with the backbone frozen.

  • Capability and Safety Evaluation: Safin-1 raises MMLU-Pro from 58.27 to 67.55 at 4B and improves GPQA-Diamond by 6.56 points at 35B-A3B.
  • Capability and Safety Evaluation: Safin-1 raises AIME 2025 Avg@64 by 8.02 points at 4B and 8.22 points at 35B-A3B, while GSM8K and MATH remain broadly stable.
  • Capability and Safety Evaluation: Safin-1 improves IFEval and LongBench v2 at both scales, with gains of 0.19 and 0.20 points at 4B and 1.30 and 0.40 points at 35B-A3B.
  • Capability and Safety Evaluation: Average jailbreak ASR falls by 0.60 points at 4B and 1.00 point at 35B-A3B, while XSTest ORR changes from 8.80 to 8.60 and from 16.80 to 17.60, respectively.
  • Capability and Safety Evaluation: Safin-1 improves seven of ten capability benchmarks at 4B and nine of ten at 35B-A3B, while reducing average jailbreak attack success rates.
  • Safety State: The persistent Safety State is trained by freezing language-model parameters and optimizing only persistent states inserted into the routed memory banks.

5. Related Works

Related work spans efficient attention, recurrent and expandable memory, and modular safety alignment. These lines of research motivate separating long-context memory from persistent, selectively invoked behavioral specialization.

  • Efficient Long-Context Modeling: Efficient attention reduces full-attention cost through restricted receptive fields, kernel approximations, systems optimization, or sparse selection.
  • Sparse Attention: Sparse-attention systems preserve content-based retrieval while reducing token-level interactions through predefined or input-dependent selection.
  • Recurrent State and Memory: Recurrent and state-space models compress causal context into recurrent states, while later designs improve state-transition dynamics and associative memory updates.
  • Expandable Memory: Expandable-memory methods increase accessible memory units through multiple states, hierarchical organization, routed experts, sparse state banks, or product-key memories.
  • Safety Alignment: Safety-alignment research evaluates attack success alongside over-refusal and includes data-efficient, parameter-efficient, and modular adaptation methods.
  • Representation-Based Control: Representation-based approaches control behavior through hidden-state interventions, activation directions, and input-conditioned or transferable safety representations.

6. Discussion and Conclusion

Safin-1 treats model state as both addressable context memory and a substrate for persistent capability specialization. Initial results support this architecture-level direction for Safety from Within while identifying scalability, evaluation, and composition limits.

  • Model memory as an architectural substrate: MARCH preserves selected recurrent-state checkpoints as addressable compressed memories, complementing rather than replicating full attention’s token-level recall path.Historical-state access remains useful across GDN, KDA, and GDN2, including after continual pretraining and supervised fine-tuning.
  • State-native specialization for Safety from Within: Safety State uses the same routed state bank for learned persistent capability states and context-derived anchors, with the backbone frozen and specialization attachable or removable.The state’s safety representation and invocation are native to model computation, although its supervision still comes from external harmful and benign examples.
  • State-native specialization for Safety from Within: Multiple persistent states may extend the interface beyond safety, but reliable coexistence and composition remain open questions.Detachability also creates an integrity requirement because persistent states and routing paths must resist removal, substitution, or adversarial suppression.
  • Limitations and research agenda: The state bank grows with context length, while sparse routing reduces reading cost but not storage; fixed-interval anchoring may retain redundant checkpoints.Large-scale evidence covers only two Qwen3.5 configurations under one CPT–SFT pipeline, and safety evaluation is limited in corpus, language, and attack coverage.
  • Limitations and research agenda: Future work should examine adaptive anchor management, broader model and context regimes, state tampering, and composition without routing conflicts or capability interference.The paper also calls for alternative and complementary architectures that make safety representable, maintainable, and selectively invocable through native computation.
  • State-native specialization for Safety from Within: 42.3% and 52.3% reductions in unweighted average ASR across five jailbreak benchmarks occur at 4B and 35B-A3B, respectively, with lower XSTest ORR than matched rank-8 LoRA controls.These are initial architecture-level results rather than a definitive solution.

Project Contributors

The project was led by the Shanghai Artificial Intelligence Laboratory, with the listed authors contributing in contribution order. The author list identifies equal contributors, a technical lead, and a project lead.

  • The Shanghai Artificial Intelligence Laboratory led the work.
  • The listed contributors are Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Zhekai Chen, and Cheng Jin.
  • The remaining listed contributors are Jingnan Zheng, Yi Zhang, Dongcheng Zhang, Zhongtian Ma, Jiawei Zhou, Sirui Chen, Qiaosheng Zhang, Xiang Wang, Ning Ding, Xia Hu, Bowen Zhou, Youbang Sun, and Chaochao Lu.
  • Ming Zhang and Kaisen Yang are equal contributors; Youbang Sun is the technical lead; Chaochao Lu is the project lead.

A. Training Configurations

The appendix reports architecture, data, and optimization configurations for the controlled study, larger Safin-1 runs, and Safety State specialization.

  • The appendix covers the architecture configurations used in the experiments.
  • The appendix reports data configurations for the controlled study and Safin-1 runs.
  • The appendix reports optimization configurations for Safety State specialization.

A.1. Small-Scale MARCH Validation

The controlled small-scale comparison uses a specified recurrent architecture, while MARCH adds periodic state anchors and content-based routing. The appendix also identifies a hierarchical-state baseline.

  • Table 10 reports the training configuration for the controlled small-scale study.
  • The controlled recurrent models use 21 layers, hidden size 1,536, SwiGLU intermediate size 4,096, untied embeddings, and a 32,000-token Llama 2 vocabulary.
  • GDN, KDA, and GDN2 use six recurrent heads with d_k = 192, d_v = 384, and a bias-free short convolution of kernel size 4.
  • MARCH creates state anchors every C = 512 tokens and routes all causally visible anchors using six query heads, one key head, routing dimension d_r = 64, and a learned null candidate.
  • The Log-Linear baseline retains at most 15 hierarchical states.

A.2. Large-Scale Safin-1 Training

Safin-1 training uses dense and MoE backbones with state routing configured across selected GDN layers. The training and model configurations specify separate continual-pretraining and supervised-fine-tuning regimes at both scales.

  • Model and routing configuration: The 4B dense model and 35B-A3B MoE model use 3:1 GDN-to-full-attention layers with routed state anchoring in 16 GDN layers.Anchors are created every 512 text tokens, and the router selects the Top-4 candidates.
  • Model and routing configuration: The models are initialized from Qwen3.5-4B and Qwen3.5-35B-A3B-Instruct, respectively.The initialization procedure also expands and copies pretrained GDN routing projections while leaving remaining backbone weights at the starting checkpoint.
  • Optimization and numerical settings: Both scales use distributed Adam with BF16 model computation, FP32 optimizer states, weight decay 0.1, and global gradient clipping at 1.0.The 35B-A3B MoE router also remains in FP32, and global token batches exclude inserted anchor tokens.
  • Continual pretraining: Continual pretraining uses Intern-S2 with 50.004B training tokens, 32,768-token sequences, and 5,961 steps.The schedule uses 256 warmup steps, 5,109 stable steps, and 596 decay steps.
  • Supervised fine-tuning: Supervised fine-tuning uses Nemotron-Cascade-2-SFT-Data with 30.199B tokens and 32,768-token sequences.The 4B and 35B-A3B configurations use 3,600 and 7,200 training steps, respectively.

A.3. Safety State Specialization

The Safety State is compared with rank-8 LoRA under a matched adaptation protocol. Safety State updates only persistent states with the backbone frozen, while LoRA updates adapters under the same training conditions.

  • Matched adaptation protocol: Safety State and rank-8 LoRA start from the same Safin-1 SFT checkpoint and use matched examples, schedules, batch sizes, and 37 updates.The comparison retains dynamic state anchoring and content-routed historical readout for the LoRA control.
  • Adaptation mechanisms: Safety State freezes the backbone and optimizes only persistent states, whereas LoRA disables Safety State and updates only rank-8 adapters.Both methods therefore isolate different adaptation mechanisms under the shared protocol.
Loading 2609.00092v1…