PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space

Xingwei Qu, Shaowen Wang, Zihao Huang, Kai Hua, Fan Yin, Rui-Jie Zhu, Jundong Zhou, Qiyang Min, Zihao Wang, Yizhi Li, Tianyu Zhang, He Xing, Zheng Zhang, Yuxuan Song, Tianyu Zheng, Zhiyuan Zeng, Chenghua Lin, Ge Zhang, Wenhao Huang

arXiv:2512.24617v2cs.LGcs.AI

TL;DR

Token-uniform LLMs allocate computation evenly despite uneven information density. DLCM learns variable-length concepts and shifts reasoning into compressed concept space; at R = 4, it improves average accuracy by 2.69% across 12 zero-shot benchmarks under matched inference FLOPs. The paper also reports a trade-off in fine-grained precision at some mid-concept positions.

  • Problem

    Token-level models apply uniform computation despite language’s uneven information density and the hierarchical structure of reasoning.

    Source: §1 · p. 1

  • Method

    DLCM learns variable-length semantic boundaries from latent representations, reasons over compressed concepts, and reconstructs token predictions through causal cross-attention.

    Source: §1 · p. 2

  • Results

    +2.69% average accuracy across 12 zero-shot benchmarks at R = 4 under matched inference FLOPs, with up to 34% fewer FLOPs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DLCM shows a favorable accuracy–efficiency trade-off while reducing redundant computation and improving results on reasoning-intensive benchmarks.

    Source: §9 · p. 18

    Compression can trade off fine-grained token precision within concepts; BoolQ and RACE show mild regressions of -1.47% and -0.72%, respectively.

    Source: §7.1 · p. 13

Abstract

from arXiv · show

Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose $\textbf{Dynamic Large Concept Models (DLCM)}$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first $\textbf{compression-aware scaling law}$, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a $\textbf{decoupled $\mu$P parametrization}$ that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting ($R=4$, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a $\textbf{+2.69$\%$ average improvement}$ across 12 zero-shot benchmarks under matched inference FLOPs.

1 Introduction

DLCM addresses the mismatch between uniform token computation and language’s uneven information density by learning variable-length concepts and reasoning in compressed concept space. The introduction also presents tools for scaling and training this architecture, alongside benchmark gains under compute constraints.

1 Introduction

  • Token-uniform models spend full computation on predictable spans and critical transitions alike, while latent reasoning, fixed sentence concepts, and byte-level H-NET leave gaps for modern token-level prediction.The introduction characterizes reasoning as hierarchical and notes that H-NET [11] has not been validated against standard modern NTP baselines.Source: §1 · pp. 1–2
  • By separating concept formation from reasoning, DLCM allocates computation according to semantic structure and targets high-information tokens at new concept boundaries.This contrasts with computation tied uniformly to surface token count.Source: §1 · pp. 2–3
  • DLCM learns variable-length concepts from latent representations through encoding, boundary detection, concept pooling and reasoning, then causal cross-attention decoding.Concepts are latent segments, not predefined linguistic units.Source: §1 · pp. 2–3
  • The compression-aware scaling law models total parameters, data, compression ratio, and concept-backbone allocation to guide architecture choices under equal-FLOPs constraints.The law is written L(N, D, R, P).Source: §1 · p. 3
  • Decoupled µP adjusts learning rates independently for token-level components, the concept backbone, and embeddings, with each optimal effective rate scaling inversely with its component width.The parametrization is presented as a way to prevent instability and support performance at scale.Source: §1 · p. 3
  • +2.69% average accuracy across 12 zero-shot benchmarks, with up to 34% fewer FLOPs at R = 4, where capacity shifts to a larger reasoning backbone.The largest gains occur on reasoning-dominant tasks.Source: §1 · p. 3

2 Related Work

Prior work explores reasoning beyond tokens through continuous latent states, sentence-level concepts, and adaptive computation. The discussion positions DLCM against limitations in granularity, interpretability, and applicability to modern autoregressive token generation.

2.1 From Latent Reasoning to Concept-Level Language Modeling

  • Continuous latent reasoning can avoid intermediate token generation and represent parallel reasoning paths, but may reduce interpretability and struggle with precise symbolic manipulation [6].The discussion uses COCONUT as an example of reasoning through successive hidden states [6].Source: §2.1 · p. 3
  • LCMs use fixed sentence-level concepts and frozen encoders and decoders, offering multilingual transfer [16] but requiring separate pretraining and predetermined segmentation.The passage reports 10× sequence-length reduction and generation in 200+ languages for LCMs [16]; DLCM instead learns boundaries end-to-end.Source: §2.1 · pp. 3–4

2.2 Dynamic Compute Allocation in Language Models

  • Universal Transformers [3] and MoE models [14? ] offer adaptive computation, but the discussion says they do not fundamentally address language’s information-density problem.The cited comparison frames these approaches around recurrence or token routing, rather than concentrating computation at semantic boundaries.Source: §2.2 · p. 4
  • H-NET [11] learns boundaries end-to-end and reports 4–8× compression, but focuses on bit-level modeling rather than token-level generation in modern autoregressive LLMs.DLCM adapts dynamic boundary detection to next-token prediction in large autoregressive models [1] [17].Source: §2.2 · p. 4

3 Methodology

DLCM encodes tokens, learns boundaries to form variable-length concepts, reasons over the compressed sequence, and decodes token predictions from those concepts. Global load balancing targets a chosen compression ratio while allowing local variation.

3.2 Encoding

  • The encoder extracts fine-grained token representations that provide the basis for dynamic segmentation and token-level decoding.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 Dynamic Segmentation; 3.3.1 Boundary Detection

  • DLCM detects semantic breaks through local dissimilarity between adjacent latent token representations, with segmentation decisions decoupled from language-model loss for training stability.Source: §3.3.1 · p. 5, §3.3 · p. 5

3.3.2 Concept Formation via Pooling

  • Boundary indicators partition tokens into contiguous segments, each mean-pooled and projected into one concept representation.The resulting concept sequence is shorter than the token sequence.Source: §3.3.2 · p. 5

3.3.3 Adaptive Compression via Global Load Balancing

  • The Global Parser regularizes boundary statistics across the batch toward a global compression rate of 1/R while allowing local fluctuation.R = 4 corresponds to an average of four tokens per concept.Source: §3.3.3 · pp. 5–6

3.4 Concept-Level Reasoning

  • A causal concept-level Transformer performs deep reasoning on the compressed sequence and produces enriched concept representations.Source: §3.4 · p. 6

3.5 Token-Level Decoding; 3.5.1 Concept Smoothing; 3.5.2 Causal Cross-Attention

3.5.2 Causal Cross-Attention

  • Cross-attention projects queries and concept keys and values into a shared head dimension, then projects its output back to the token dimension.Source: §3.5.2 · p. 6

3.6 Training Objective

  • The training objective combines token-level cross-entropy with the adaptive compression load-balancing loss.Source: §3.6 · p. 7

4 Implementation Details

Concept replication converts irregular token-to-concept cross-attention into a regular causal layout for optimized Flash Attention kernels. The approach outperforms Flex Attention in tested configurations, with a larger average speedup at longer sequence lengths.

4 Implementation Details

  • Packed VarLen training supports global compression statistics, and RMSNorm is applied to queries and keys before attention.Source: §4 · p. 7

4.3 Key Observations and Analysis

  • 1.26×–1.73× speedup: Flash Attention Varlen with concept replication outperforms Flex Attention across all tested configurations.Source: §4.3 · p. 7

4.1 Efficient Cross-Attention via Concept Replication

  • Replicating each concept across its token positions changes the irregular L × M attention layout into a standard L × L causal mask for Flash Attention kernels.Source: §4.1 · p. 7, Figure 2 · p. 8

4.3 Key Observations and Analysis

  • Speedup trends are nearly identical across hidden sizes, while Flash Varlen trades a larger K/V cache footprint for regular memory access.Source: §4.3 · p. 8
  • At 2K sequence length the average speedup is ∼1.44×, rising to ∼1.70× at 16K and peaking at 1.73× for hidden size 2048.Source: §4.3 · p. 8

5 Data

The pretraining corpus combines open-source data from multiple domains, selected to balance broad language coverage with structured reasoning and diverse information densities.

5 Data

  • The open-source pretraining corpus uses the DeepSeek-v3 [13] tokenizer and includes English and Chinese web text, mathematics, and code.Source: §5 · p. 8
  • Web text supplies broad coverage, while mathematics and code add structured reasoning examples intended to support robust dynamic segmentation across varied information densities.Source: §5 · pp. 8–9
  • The authors avoid aggressive filtering so data quality aligns with standard open-source corpora; Table 1 summarizes corpus statistics.Source: §5 · p. 9, Table 1 · p. 9

6 Scaling Laws for DLCM

DLCM scaling work pairs decoupled optimization for its heterogeneous architecture with a compression-aware loss law. The law is validated across model scales and compression ratios, while µP-derived hyperparameters transfer from a proxy model to larger models.

6 Scaling Laws for DLCM

  • DLCM’s scaling-law exploration first addresses its heterogeneous architecture with decoupled optimization, then formulates scaling objectives.Source: §6 · p. 9

6.1.1 Formulation of µP

  • µP separately scales token components (E, D) at width dtoken and concept backbone (M) at dconcept, with component-specific initialization and optimization settings.Source: §6.1.1 · p. 9

6.1.2 Hyperparameter Tuning and Verification

  • Hyperparameters tuned on an 87M proxy transfer zero-shot to 274M, 468M, and 834M models; perturbing predicted rates degrades performance.The procedure uses proxy tuning followed by larger-scale verification.Source: §6.1.2 · p. 10

6.2.1 Experimental Setup

  • The scaling study varies concept-layer parameter ratio P across 30%, 50%, and 70% and compression ratio R across 2, 4, and 8, fitting shared exponents once.Models are trained on 200B tokens at 274M, 468M, and 833M parameters; only scale-independent offsets vary by configuration.Source: §6.2.1 · pp. 10–11

6.2.2 Mathematical Formulation

  • Extending Chinchilla [8], the loss law L(N, D, R, P) disentangles token-processing efficiency, concept-processing efficiency, and data scaling.N is total parameters, D dataset size, R compression ratio, and P concept-layer parameter ratio.Source: §6.2.2 · p. 11

6.2.3 Decay-Phase Power Law

  • A log-linear decay law fitted to fractional loss reduction in the 90%–99% token window achieves R2 = 0.93 and predicts late-stage loss drops across scales.The late-stage regime is modeled for the Weight-Sharing-and-Decay training protocol.Source: §6.2.3 · p. 11

6.3.1 Architectural Efficiency

  • R = 4 is selected as the primary configuration because it aligns better with intuitive semantic segmentation and balances training stability with computational efficiency.Source: §6.3.1 · p. 11

6.4 Scaling Law Validation and Verification

  • The joint fit reaches R2 > 0.98, and the predicted effective compute multiplier is approximately 1.4 versus the standard baseline factor of 1.34.The fit compares predicted and empirical loss across model sizes, compression factors, and training budgets.Source: §6.4 · p. 11, Figure 4 · p. 12

7 Experiments

DLCM improves average zero-shot accuracy over the baseline, with gains concentrated in reasoning tasks and trade-offs on granularity-sensitive tasks. A controlled loss analysis finds better performance near concept boundaries but mixed results within concepts.

7.1 Main Results

  • DLCM improves accuracy on reasoning-dominant tasks, including OpenBookQA (+3.00%), PIQA (+2.42%), and ARC Easy (+2.61%).The authors connect these gains to compressing locally predictable spans and concentrating capacity in the concept backbone.Source: §7.1 · p. 13
  • BoolQ (-1.47%) and RACE (-0.72%) regress, which the authors attribute to reduced token-level granularity obscuring fine distinctions.The passage describes the degradation as localized: boundary tokens are modeled more accurately, while mid-concept positions may lose precision.Source: §7.1 · p. 13

7.2.1 Experimental Setup

  • The loss-distribution comparison controls for model size and training data, using 1.3B-parameter models trained on the same 100B-token subset.The analysis aligns token losses by relative position within concepts.Source: §7.2.1 · p. 14, §7.2.2 · p. 14

7.2.2 Loss Distribution Analysis

  • Loss improvements follow a U-shaped pattern: the concept model outperforms the baseline at concept boundaries, particularly positions 0–2 and 16+.Figure 7 compares loss across the first 20 within-concept positions; green indicates lower concept-model loss and red indicates higher loss.Source: §7.2.2 · pp. 14–15

7.1 Main Results

  • 43.92% average accuracy for DLCM surpasses the baseline’s 41.23% by +2.69%, with gains varying across task types.Table 2 presents zero-shot accuracy (%) by task type, with improvements and regressions distinguished by color.Source: §7.1 · p. 12, Table 2 · p. 15

7.2.2 Loss Distribution Analysis

  • At mid-concept positions around 4–15, performance is mixed, with some degradation reflecting a trade-off between token-level precision and semantic coherence.The authors describe this as reallocating capacity toward boundaries and structurally critical tokens rather than uniform token-level predictability.Source: §7.2.2 · p. 15

8 Ablation Studies

The ablations find that global compression regularization improves most downstream tasks and supports content-adaptive concept lengths. They also report that a learned boundary predictor can drift from its target compression rate, unlike a fixed rule-based predictor.

8.1 Analysis: End-to-End Discrete Boundary Learning vs. Decoupled Segmentation

  • The learned predictor’s compressed length rises from about 2,000 to 4,300 tokens, while the rule-based predictor maintains about 2,000 tokens, or 4× compression.The authors attribute the learned predictor’s drift to cross-entropy gradients overpowering compression regularization; the fixed cosine-similarity threshold avoids that conflict.Source: §8.1 · pp. 15–16

8.3 Content-Adaptive Compression Benefits

  • At the 8× target, Technical English averages 10.58 tokens per concept, versus 6.09 for Technical Chinese and 6.14 for Code.The authors report that dense prose is preserved in longer semantic chunks, while code and structured text tend to form shorter syntactic units.Source: §8.3 · pp. 17–18, Table 5 · p. 18

8.2 Global Regularization via Gradient Accumulation

  • The Global Parser improves performance on 5 of 6 tasks and achieves a realized compression ratio of about 3.9, closer to the 4.0 target.The authors argue that batch-level regularization permits content-adaptive compression rather than enforcing a fixed rate on each sequence.Source: §8.2 · p. 16, Table 4 · p. 17

9 Conclusion

DLCM shifts computation from tokens to learned variable-length concepts, creating a compact space for reasoning. The conclusion also highlights tools for allocating and transferring computation, reports empirical gains, and points to adaptive abstraction and multi-level reasoning as future directions.

9 Conclusion

  • DLCM shifts computation from tokens to variable-length concepts learned from latent representations, enabling reasoning in a compact, semantically aligned space.The framework challenges token-uniform computation and avoids repeatedly reasoning at the token level.Source: §9 · p. 18
  • The paper introduces a compression-aware scaling law for allocating compute under fixed FLOPs and a decoupled µP parametrization for stable training and zero-shot hyperparameter transfer.The parametrization is intended for heterogeneous architectures.Source: §9 · p. 18
  • DLCM reports consistent gains on reasoning-intensive benchmarks with reduced redundant computation, while the conclusion identifies adaptive abstraction, planning, and multi-level reasoning as future directions.The authors frame concept-level latent reasoning as a promising direction for more efficient and reasoning-capable language models.Source: §9 · p. 18

Contributions

The contribution roster assigns work across DLCM development, implementation, data construction, optimization, and related infrastructure.

Contributions

  • Xingwei Qu conducted fundamental ablation studies and most engineering implementation, while Shaowen Wang advanced MuP for hyperparameter tuning and Zihao Huang implemented noise-based boundary prediction tricks.Source: Contributions · p. 19
  • Ge Zhang proposed the original idea, developed the demo prototype, and provided the key Global Parser technique; Rui-Jie Zhu resolved DLCM bugs and proposed replacing the encoder with DLCM.Source: Contributions · p. 19
  • Kai Hua constructed training data from open-source data, and Fan Yin contributed to SGLang implementation and optimization.Source: Contributions · p. 19

A Appendix: Segmentation Examples at Different Compression Ratios

The appendix illustrates predicted segmentation on casual English, Python code, and mathematical text at three compression ratios. The examples report segment counts, while the accompanying figure and table describe a Flash Attention speedup comparison.

A.1 Casual English Text

  • For the coffee-routine passage, the examples report 11, 35, and 56 segments at compression ratios 8×, 4×, and 2×, respectively.Source: §A.1 · p. 22

A.2 Python Code

  • For the Python dataset-code sample, the examples report 12, 29, and 58 segments at compression ratios 8×, 4×, and 2×, respectively.Source: §A.2 · pp. 22–23

A.3 Mathematical Text

  • For the Euler’s formula sample, the examples report 13, 32, and 61 segments at compression ratios 8×, 4×, and 2×, respectively.Source: §A.3 · p. 23
  • Figure 9 plots Flash Attention Varlen speedup (Tflex/Tflash) against sequence length; Table 6 identifies a comparison at Batch=1, Heads=32, and Interval=6.Source: Figure 9 · p. 23, Table 6 · p. 24
Loading 2512.24617v2…