PaperlayerView exact revision on arXiv

Source-linked AI summary

HRM-Text: Efficient Pretraining Beyond Scaling

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori

arXiv:2605.20613v1cs.CL

TL;DR

Large language model pretraining remains costly because it depends on massive compute and internet-scale raw text. HRM-Text co-designs a hierarchical recurrent architecture with task-completion pretraining, achieving competitive open-model performance with far fewer tokens and less compute.

  • Problem

    Internet-scale raw-text pretraining and massive compute make foundational language-model research inefficient in data-limited settings and inaccessible to many researchers.

  • Method

    HRM-Text co-designs a dual-timescale hierarchical recurrent architecture with a targeted task-completion objective instead of standard raw-text pretraining.

  • Results

    Using 100-900× fewer training tokens and 96-432× less estimated compute, HRM-Text achieves strong performance on most benchmarks and remains competitive with contemporary open models.

  • Takeaways & Limitations

    The results provide an empirical existence proof that competitive pretraining can be achieved without the massive compute and internet-scale raw text dominating current paradigms.

  • Takeaways & Limitations

    The reported scaling experiments cover only up to 1B parameters for HRM-Text, so comparable efficiency gains at larger model scales remain future work.

Abstract

from arXiv · show

The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contrast, biological systems demonstrate highly sample-efficient learning through multi-timescale processing, such as the functional organization of the frontoparietal loop. Taking this as inspiration, we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recurrent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving execution layers. To stabilize this deep recurrence for language modeling, we introduce MagicNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pretraining, we train exclusively on instruction-response pairs using a task-completion objective and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500 budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x less estimated compute than standard baselines, HRM-Text performs competitively with 2-7B parameter open models. These results demonstrate that co-designing architectures and objectives can radically reduce the compute-to-performance ratio, making pretraining from scratch accessible to the broader research community.

1 Introduction

HRM-Text addresses the inefficiency of raw-text scaling by co-designing a hierarchical recurrent architecture with targeted instruction-based training. Trained from scratch on 40B unique tokens, it provides an empirical existence proof that this approach can achieve strong benchmark performance with far less data and compute.

  • Motivation: Internet-scale raw-text pretraining wastes massive compute on prompt-like or task-irrelevant text and creates a major barrier to foundational research.The passage contrasts this brute-force scaling paradigm with data-limited regimes and identifies its computational inefficiency.
  • Architecture: HRM decouples deliberation into a slow-evolving strategic layer and a fast-evolving execution layer, providing a structural inductive bias for sample-efficient learning.The architecture is inspired by the functional organization of the biological frontoparietal loop.
  • Architecture: MagicNorm and warmup deep credit assignment stabilize HRM’s deep recurrence for language modeling, addressing severe gradient-instability risks.The fast L-module performs local iterative refinement, while the slow H-module maintains stable semantic context across cycles.
  • Training objective: HRM-Text pretrains from scratch on instruction-response pairs with a response-only task-completion loss and PrefixLM masking.The objective is −log P(xa | xq), with full bidirectional attention across the input prefix.
  • Results: 40B unique tokens yield strong performance against contemporary open models, using roughly 100-900× fewer training tokens and 96-432× less estimated training compute.The cited comparison includes Llama, Qwen, Gemma, OLMo, Ouro, and Huginn.

2 Methods

HRM-Text combines dual-timescale recurrent computation with MagicNorm and warmup deep credit assignment for stable language modeling. It further improves sample efficiency by training from scratch on instruction-response pairs with response-only loss and PrefixLM masking.

  • 2 Methods: Two high-level cycles each execute three fast L-module updates followed by one slow H-module update, with logits produced from the final H state.The forward pass initializes high-level and low-level recurrent states separately.
  • 2.1 Stabilization Techniques: Gradients initially traverse the final two recurrent steps, then expand to the final five steps as training progresses.This warmup reduces long-path optimization difficulties and lowers average backward-pass computation during early training.
  • 2.2 Task-Completion Pretraining: HRM-Text trains exclusively on instruction-response pairs from scratch using response-only NLL, −log P(xa|xq), instead of raw-text autoregressive pretraining.The task-completion objective concentrates parameter updates on generating responses rather than predicting instruction tokens.
  • 2.2 Task-Completion Pretraining: PrefixLM removes causal masking over instruction tokens while retaining causal masking over responses, enabling bidirectional instruction attention and more global, diverse attention.This creates an encoder–decoder-like separation within a decoder-style implementation.
  • 2.1 Stabilization Techniques: MagicNorm combines internal PreNorm blocks with an exit normalization layer to stabilize forward activations while preserving shorter-horizon gradient flow under TBPTT.It bounds activation variance across N forward steps, while gradients encounter the module-level normalization only K times, with K ≪ N.

3 Results

Under a 40B-unique-token budget, HRM-Text improves benchmark yield through recurrent architecture, task-completion objectives, and PrefixLM masking. Its 1B checkpoint performs competitively with open models trained using substantially larger data and compute budgets.

  • Experimental setting: 40B unique tokens: all reported HRM-Text performance comes from a single checkpoint trained from scratch on task-formatted data.The experiments evaluate a small-budget pretraining design rather than multiple checkpoints.
  • Architecture comparison: Recurrent and looped architectures generally outperform same-size Transformers under matched training compute, with HRM performing strongly against listed baselines, including a larger Transformer.The comparison includes standard Transformers, larger matched-FLOPs Transformers, Looped Transformers, RINS, and HRM.
  • Architecture comparison: HRM achieves generally comparable or stronger performance than TRM while using substantially fewer FLOPs, and maintains stable training dynamics where TRM becomes severely unstable at 1B parameters.At 0.6B parameters, HRM is competitive across most benchmarks while requiring 2× less compute than TRM.
  • Objective and input ablation: Task-completion objectives, PrefixLM training, and HRM each significantly improve performance in FLOPs-matched ablations, with HRM providing the final consistent increase across benchmarks.The task-completion objective establishes initial gains, while PrefixLM further improves results over standard causal masking.
  • Open-model comparison: HRM-Text 1B achieves strong performance among contemporary open models on most benchmarks and remains competitive on MMLU despite its smaller parameter count and 40B-token budget.The comparison includes Llama, Qwen, Gemma, OLMo, Huginn, and Ouro, whose training budgets are substantially larger.
  • Scaling and limitations: Up to 3B Transformer and 1B HRM-Text experiments remain competitive with industrial-scale efforts using datasets up to 36T tokens, while larger-scale efficiency remains future work.The reported scaling evidence is limited to the explored parameter range.

4 Training details

HRM-Text is trained on a stratified mixture of instruction-oriented open-source data, with response conditioning and explicit removal of long chain-of-thought traces. Contamination tests and implementation details characterize the training procedure and its evaluation reliability.

  • Data and sampling: 40B unique tokens are sampled for 60B total training tokens from an initial 176.5B-token, 593.7M-document open-source corpus.The corpus includes general instructions, rewritten knowledge, mathematical and symbolic tasks, textbook exercises, and web-extracted questions.
  • Response control: Four condition tags—direct, cot, synth, and noisy—enable explicit selection of the desired output format at generation time.The tags correspond to answer-only, chain-of-thought, synthetic-answer, and unevenly formatted web-crawl styles.
  • Response control: Text within <think>...</think> boundaries is stripped before training, concentrating the objective on final task completions and internal hierarchical computation.This removes explicit long-CoT traces mostly produced by reinforcement learning with verifiable rewards.
  • Data and sampling: Stratified sampling treats each dataset or task as an independent stratum, caps massive datasets, and upsamples smaller datasets to balance the training mixture.Specific sampling limits and multipliers are provided in Table 6.
  • Contamination analysis: 81.1 is achieved by HRM-Text 1B on DROP’s strictly clean subset, while HRM-Text 0.6B shows no significant contamination in either tested setting.HRM-Text 1B is statistically significant on DROP for n = 13 but not n = 20; overall performance is unlikely to be artificially driven by test-example exposure.
  • Implementation: 46 hours on two 8×H100 nodes cost around $1,472 for HRM-Text 1B, using a 16-layer, 1536-hidden-size configuration with 4,096-token context.The model uses a 65,536-vocabulary BPE tokenizer, RoPE positional encoding, bfloat16 precision, and LeCun-normal initialization.

5 Discussion

The discussion argues that HRM-Text can partially separate reasoning computation from factual coverage, while identifying adaptive computation and inference engineering as important directions for improving efficiency and deployment. It proposes pairing a compact recurrent reasoning core with external factual storage or memory.

  • Reasoning and factual coverage: 40B unique tokens suffice for strong reasoning performance despite knowledge-oriented sources forming only a fraction of the task-formatted mixture.The model performs strongly on MATH and GSM8K while retaining nontrivial performance on MMLU.
  • Reasoning and factual coverage: HRM-Text could serve as a compact reasoning core specialized for computation, planning, and task execution, with factual breadth supplied by curated corpora, retrieval, or learned memory.This direction is motivated by separating reasoning from factual storage rather than requiring one backbone to provide both.
  • Efficiency directions: Adaptive computation could terminate simpler problems earlier, reducing computation while maintaining near-optimal performance, but HRM-Text omits this component for design simplicity.The recurrent schedule adds effective serial depth but also increases inference-time computation.
  • Inference considerations: Multi-turn PrefixLM chat requires careful KV-cache logic so user segments attend fully while assistant tokens preserve causal generation.The passage characterizes this as an engineering constraint that production inference systems should address explicitly.

6 Conclusion

HRM-Text provides an empirical existence proof that highly efficient pretraining is achievable through co-designing a hierarchical recurrent architecture with a targeted task-completion objective. By reducing the compute-to-performance ratio, it makes foundational pretraining computationally accessible beyond highly resourced institutions.

  • Conclusion: HRM-Text demonstrates that highly efficient pretraining is achievable with a hierarchical recurrent architecture and targeted task-completion objective.The approach is inspired by biological multi-timescale processing and avoids reliance on massive compute and internet-scale raw text.
  • Conclusion: Reduced compute-to-performance ratios make foundational pretraining accessible to small labs, academic groups, and individuals.The authors frame this accessibility as enabling broader exploration, training, and innovation on new architectures from scratch.

7 Related Work · Appendix · A FLOPs estimation

The related work situates HRM-Text against scaling-based pretraining, conditional sequence modeling, latent recurrent computation, and recurrent-optimization challenges. The appendix estimates dense-model and recurrent-model FLOPs using distinct forward and backward accounting.

  • 7.1 Scaling laws and efficient pretraining: Scaling laws jointly increase parameters, data, and compute, concentrating large-corpus Transformer pretraining among compute-rich organizations.HRM-Text instead explores efficient pretraining beyond this dominant scaling recipe.
  • 7.2 Conditional sequence modeling and PrefixLM: Decoder-only causal training inefficiently assigns loss to reconstruct prompts that are already known at inference time.This motivates conditional sequence-modeling objectives.
  • 7.2 Conditional sequence modeling and PrefixLM: PrefixLM masking lets prefix tokens attend bidirectionally while outputs remain causal, combining conditional-generation behavior with decoder-only simplicity.HRM-Text uses response-only loss and makes conditional modeling its primary pretraining objective.
  • 7.3 Latent computation and recurrent language models: Recurrent-depth and block-recurrent Transformers reuse parameters across steps or layers but inherit unstable long-range credit assignment.Universal Transformers introduced recurrent depth to self-attention, while later variants extended parameter reuse.
  • 7.3 Latent computation and recurrent language models: Recent latent-reasoning and recurrent-depth language models refine hidden states internally before emitting answers, establishing latent recurrence as an alternative to purely token-level computation.The cited examples include Huginn, Ouro, and CCDD.
  • 7.3 Latent computation and recurrent language models: HRM-Text extends the Hierarchical Reasoning Model’s two-timescale recurrence to language modeling under a small token budget and from-scratch training.Related recursive models such as TRM suggest that hierarchy, temporal separation, and recurrence can support useful serial computation.
  • 7.4 Stable recurrent optimization: Recurrence intensifies normalization and credit-assignment difficulties because repeated transformations create long products of Jacobian-like operators during backpropagation.HRM-Text addresses these issues with MagicNorm and warm-up for deep credit assignment, techniques described as crucial for stable recurrent depth at language-model scale.
  • A FLOPs estimation: Dense-model training FLOPs use F = 6ND, whereas recurrent models count 2ND for forward computation and 4ND for backward computation, scaled by recurrent steps per pass.The recurrent estimate accounts separately for forward and backward recurrent unrolls.

B Evaluation details · C Stable optimization in recurrent-depth models

Evaluation uses benchmark-specific prompts and standardized deterministic decoding, while baseline results come from comparable published settings or direct open-weight evaluation. Few-shot and chain-of-thought comparisons use consistent inference configurations and designated evaluation engines.

  • B Evaluation details: Evaluation prompts include the original benchmark question and required few-shot examples, without an additional system prompt.Few-shot examples are used only for few-shot evaluations.
  • B Evaluation details: Temperature-zero decoding and a 3072-token maximum context length are used unless otherwise specified.
  • B Evaluation details: Baseline scores come from original papers when comparable evaluation settings are available.
  • B Evaluation details: When published baseline numbers are unavailable, the corresponding open-weight model is evaluated directly.
  • B Evaluation details: All few-shot evaluations use the same configuration as HRM-Text and run with the vLLM inference engine.
  • B Evaluation details: Chain-of-thought evaluations are conducted with lm_eval_harness.

C.1 Gradient Stability Under Deep BPTT in HRM

Deep BPTT in HRM produces intermittent, disproportionately large gradient events whose frequency and magnitude increase with backward depth. The experiments identify multiplicative Jacobian amplification as the likely mechanism and show that the chosen truncation is the closest stable alternative to full BPTT.

  • Observed instability: Full BPTT exhibits rare but substantially larger gradient-magnitude spikes than truncated backpropagation, with deeper H cycling increasing log-magnitude dispersion.Values are normalized within diagnostic checkpoints to isolate backward-depth effects from global training-time drift.
  • Observed instability: Increasing the number of backward H and L cycles while holding forward computation fixed reveals more intermittent high-amplitude gradient events as the horizon approaches full BPTT.The study quantifies instability using mean absolute gradient magnitude across training.
  • Mechanism: Jacobian growth increases with backward depth, indicating that deeper recurrent backpropagation amplifies some directions more strongly.This supports a multiplicative mechanism involving products of loop Jacobians.
  • Mechanism: Paired comparisons at identical diagnostic checkpoints show that full BPTT produces rare, disproportionately large gradient events relative to truncated backpropagation.The comparison directly contrasts full and truncated gradient magnitudes under matched checkpoints.
  • Training choice: The experimental truncation setting is the closest setting to full BPTT that remains stable during training.This stability characterization is illustrated by the gradient comparisons in Figures 6a and 7.

C.2 Gradient stability across recurrent architectures

HRM maintains useful gradient signals during recurrent training while avoiding increasingly heavy-tailed or unstable updates. Compared with RINs and the Universal Transformer, these dynamics place HRM in a favorable training-stability regime.

  • Gradient stability metrics: HRM, RINs, and the Universal Transformer are compared using median absolute gradient magnitude and tail-to-median ratio.These statistics assess gradient signal strength and the prevalence of extreme updates across runs.
  • Gradient signal: Higher median absolute gradient magnitudes show that HRM and the Universal Transformer retain stronger training signals than RINs during optimization.This result is reported across training in Figure 8a.
  • Gradient distribution: Lower tail-to-median ratios show that HRM and the Universal Transformer avoid increasingly unstable, rare extreme updates as training progresses.Their gradient distributions become more even and less heavy-tailed, as shown in Figure 8b.
  • Overall stability: HRM retains useful gradient signal while avoiding heavy-tailed gradient dynamics, preserving training stability and supporting better downstream performance.This favorable regime is identified in combination with the paper’s main results.

D Inference-time analysis

HRM inference uses auto-guidance to combine predictions from recurrent depths without extra computation, slightly improving test-time performance. The guidance direction also suggests adaptive control of recurrent depth: interpolation may support early stopping, while extrapolation may motivate deeper test-time scaling.

  • Inference efficiency: Auto-guidance introduces zero computation overhead because shallow-loop hidden representations are already available during decoding.The reported guidance search uses w ∈ {−0.5, −0.1, 0, 0.1, 0.5} with an HRM containing two high-level loops.
  • Auto-guidance mechanism: Auto-guidance interpolates or extrapolates logits from the final hidden state and an earlier recurrent state at decoding time.The guidance scale w controls how predictions from shallow and deep recurrent states are combined.
  • Auto-guidance mechanism: w = 0 recovers the standard final prediction, w > 0 performs extrapolation, and w < 0 performs interpolation between shallow and deep predictions.Positive guidance treats the shallower prediction as a negative direction, whereas negative guidance balances shallow and deep recurrent states.
  • Inference results: Auto-guidance slightly improves test-time performance, with the best guidance scale varying across benchmarks.This variation suggests that different tasks may favor different combinations of shallow and deep recurrent predictions.
  • Adaptive recurrent depth: Interpolation gains suggest early stopping may improve efficiency, whereas extrapolation gains suggest deeper adaptive test-time scaling may benefit the task.Together, these results indicate that HRM inference can adapt recurrent depth to balance efficiency and performance.
Loading 2605.20613v1…