PaperlayerView exact revision on arXiv

Source-linked AI summary

NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

The Intern-NCP Team, :, Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge, Ruijun Ge, Jiayi Huang, Yang Jiao, Dahua Lin, Zhouhan Lin, Yifan Liu, Yuliang Liu, Biqing Qi, Mowen Ruan, Junzhe Shen, Yunchong Song, Hao Sun, Zhongbo Tian, Yixuan Wang, Rubin Wei, Jiaxin Xiong, Kangyu Yang, Qian Yao, Qi Zhang, Bowen Zhou

arXiv:2609.10715v1cs.CL

TL;DR

NCP-ArchPreview tackles the limited semantic supervision of next-token prediction by jointly modeling tokens and discrete multi-token concepts in a learned latent space. Its Concept Module predicts codebook-based future concepts and feeds them back into token generation. After 5.73T-token pretraining, it reaches OLMo-3-7B’s final loss with 51.3% of the tokens and improves downstream macro-average by 2.45 points, while remaining limited in its current long-context coverage.

  • Problem

    Standard next-token prediction supervises individual tokens but lacks an explicit objective for how semantic structure unfolds across multi-token spans.

  • Method

    NCP-ArchPreview combines token prediction with next-concept prediction over a product-quantized vocabulary learned from hidden states, feeding predicted concepts back into the token stream.

  • Results

    NCP-ArchPreview matches OLMo-3-7B’s final pretraining loss using 51.3% of its training tokens and improves downstream macro-average by 2.45 points after pretraining.

  • Takeaways & Limitations

    The results establish latent representations as first-class prediction targets at trillion-token scale while preserving standard autoregressive generation.

  • Takeaways & Limitations

    Long-context training is not included in the current architecture preview, and the relationship between lower language-modeling loss and downstream performance depends on training distribution and evaluation domain.

Abstract

from arXiv · show

We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation. NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module. These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end. We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latent-space language model to date. Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B. Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99-point gain on GSM8K. Controlled experiments isolate a clear progression of performance gains stemming from both the latent architecture and the NCP objective. Furthermore, utilizing only 85% of the standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline. The learned latent space remains highly valuable after the pretraining stage: updating just the 17M-parameter VQ module yields a novel, lightweight interface for domain adaptation, while a simple injection of concept representations into a DFlash2 drafter improves the mean accepted length by 4.17% with negligible overhead.

1. Introduction

NCP-ArchPreview addresses the limited semantic supervision of next-token prediction by adding explicit next-concept prediction in a scalable latent-space architecture. Built on OLMo-3-7B, it demonstrates improved optimization and downstream performance while supporting an open pretrained release.

  • Motivation: NCP-ArchPreview explicitly predicts discrete concepts spanning multiple tokens alongside next tokens, addressing the lack of direct supervision for semantic structure across spans.The model jointly uses NTP and NCP while retaining standard autoregressive generation.
  • Contribution: The 8.9B-parameter model uses a product-quantized concept space, an eight-layer Concept Module, and hierarchical residual routing between token and concept processing.It is built on the OLMo-3-7B backbone and trained across 5.73T tokens.
  • Contribution: NCP-ArchPreview matches OLMo-3-7B’s final pretraining loss using 51.3% of its training tokens and improves downstream macro-average performance by 2.45 points, including 5.99 points on GSM8K.The reported convergence speedup is 1.95×, and the downstream gains follow full pretraining.
  • Release: The report formally releases pretrained weights, inference scripts, training recipes, and intermediate evaluation checkpoints for open-source research on non-vanilla foundation architectures.The report also provides a technical account of the architecture, training objective, and empirical validation.

2. Model Architecture

NCP-ArchPreview preserves a token-level autoregressive pathway while adding a compressed, discrete concept pathway between token encoding and decoding. Product quantization, autoregressive concept prediction, causal injection, and hierarchical residual routing connect the two abstraction levels.

  • Overview: The architecture comprises a Token Encoder, Concept Module, and Token Decoder, with concepts summarizing contiguous token groups and predicted concepts guiding next-token generation.The token-level input and output remain consistent with canonical autoregressive decoding.
  • Concept Compression: Mean pooling compresses each group of k token states into a concept sequence of length M = floor(T/k), reducing sequence length by a factor of k.The Token Encoder’s hidden states also support token decoding and codebook learning.
  • Discrete Concept Vocabulary: Product quantization partitions each concept into S segments, whose codebooks yield N^S possible combinations while keeping individual codebooks small.The complete quantized concept concatenates the segment-wise codewords.
  • Next Concept Prediction: The Concept Module predicts each next-concept segment as a probability-weighted combination of codebook entries, producing a differentiable representation constrained to the learned latent space.The prediction is autoregressive over concept history rather than an argmax or sampled codeword.
  • Concept Injection: Predicted concepts are repeated to token resolution and causally shifted before fusion with token states, preventing future-concept leakage into token prediction.The Token Decoder then predicts the next token from the fused states.
  • Hierarchical Residuals: Hierarchical residual routing combines states across depths within modules and transfers aligned representations across modules through IRC and CRC connections.CRC scalings start small so the model initially remains close to the original backbone while learning to use additional connections.

3. Training

Training jointly optimizes token prediction, next-concept prediction, and vector-quantization objectives end to end. NCP supplies multi-token-span supervision through a structured codebook space, while shifted NTP preserves causal autoregressive behavior.

  • Vector Quantization: The VQ objective fits codebook entries to continuous concept representations, while stop-gradient prevents it from updating the Token Encoder.This lets the codebooks track the latent distribution without directly altering token-level hidden states.
  • Next Concept Prediction: NCP supervises dependencies between successive multi-token groups, providing concept-level training signals beyond granular next-token supervision.The objective targets concept representations rather than only individual token positions.
  • Next Concept Prediction: The Concept Module predicts each next concept through probability distributions over segment codebooks, whose weighted combinations provide differentiable structured targets.This avoids non-differentiable discrete selection and unconstrained continuous regression.
  • Optimization: NCP updates the Concept Module and Token Encoder through preceding concept representations, encouraging token states that retain information useful for future-concept prediction.The target concept is detached, so gradients reach the encoder through the concept history.
  • Next Token Prediction: The shifted fused representation receives the usual causal next-token objective, which provides dense supervision while preserving standard autoregressive language-modeling behavior.The zero prefix and shift prevent conditioning on concepts computed from tokens whose predictions are supervised.
  • Joint Training: The three objectives are jointly optimized, with α and β controlling the weights of the auxiliary NCP and VQ losses.Matrix-valued parameters use Moonlight Muon, while embeddings, biases, and other non-Muon parameters use AdamW.

4. Experiments

Across staged pretraining, NCP-ArchPreview lowers training loss and improves downstream performance relative to OLMo-3-7B, while ablations attribute gains to its modules and NCP objective.

  • 4.2.2. Training Loss Performance: 51.3% of Stage-1 training tokens reaches OLMo-3-7B’s final loss, yielding 1.95× faster convergence; Stage-2 reaches the same loss with 66.2% of tokens.In Stage-2, NCP-ArchPreview also achieves a 0.027 lower final loss and a 1.51× convergence speedup.
  • 4.2.3. Downstream Performance: 0.59 points higher macro-average at Stage-2, with improvements on MATH, MC-Non-STEM, and GenQA despite only approximately 10% code data.Stage-1 also reports a 2.45-point higher average across evaluated datasets.
  • 4.3.1. Matched Model Size and Computation: 85% of the computation approaches the training-loss performance of the parameter-aligned Vanilla size-aligned baseline, while outperforming Vanilla and Vanilla computation-aligned.The comparison uses the 34/40 computation ratio described for the matched baselines.
  • 4.3.2. Module Ablation: Adding the Concept Module, hierarchical residual, and NCP loss progressively reduces training loss, with the complete model approaching Vanilla size-aligned.The progressive configurations are compared under the same training recipe over the first 200B tokens.
  • 4.5. Scaling Laws: 1.74× computational efficiency over OLMo-3 is achieved under compute-optimal training across multiple FLOPs budgets.Each reported point is the best validation loss after searching training hyperparameters and model/data allocations at a fixed FLOPs budget.
  • 4.6. Model Analysis: Per-head Q/K normalization suppresses Q/K norm outliers and attention-logit growth, whereas layer-wise normalization is associated with gradient spikes and unstable attention.The main experiments retain layer-wise normalization to preserve a controlled comparison with OLMo-3-7B.

5. Additional Results

Additional experiments evaluate lightweight adaptation, controlled component contributions, training dynamics, and concept conditioning for speculative drafting.

  • VQ Training: 32.69 code average follows VQ adaptation from 30.04, a +2.65 gain that is highest among adapted variants and improves all four code tasks.VQ updates only the VQ codebooks and concept-prediction heads using 17M trainable parameters.
  • VQ Training: 34.83 math average follows VQ adaptation from 30.56, a +4.27 gain exceeding LoRA while preserving a +0.39 general-average change.Full training achieves a larger target gain, but VQ retains a higher general average.
  • VQ Training: 49.47 TriviaQA exact match follows VQ adaptation from 40.28, a +9.19 gain while the general average remains essentially unchanged at +0.03.The target gain is smaller than full training's +18.00 and LoRA's +17.48.
  • Training Throughput and Memory Usage: 15,632 tokens/s/GPU makes VQ training 1.50× faster than LoRA and 2.02× faster than full training, with the lowest reported memory usage.The comparison uses micro-batch size 1 on 8 GPUs; full training exceeds memory capacity.
  • 3B Controlled Comparison: 0.0075 lower loss is achieved by NCP-ArchPreview with MTP than residual-aligned OLMo-3-3B with MTP over the first 150B tokens, using slightly fewer training FLOPs.Without MTP, NCP-ArchPreview also improves over OLMo-3-3B + Residual by 0.0098.
  • Predicted Concept Improves Block-Parallel Speculative Drafting: 6.180 mean accepted length versus 5.933 yields a 4.17% macro-average improvement when concept conditioning is added to the block-parallel drafter.The gain is consistent across GSM8K, MATH, HumanEval, and MBPP, with negligible parameter overhead.

6. Related Work

Related work spans latent-target prediction, hierarchical language modeling, altered prediction objectives, and depth-wise representation mixing.

  • Latent Representation Prediction: Joint-embedding predictive architectures predict latent targets to retain semantic structure while discarding difficult or unnecessary input details.I-JEPA and V-JEPA extend this paradigm to image and video representation learning.
  • Hierarchical Language Models: Hierarchical language models shorten high-level sequences over groups of fine-grained units using fixed or data-dependent patches and predictive context embeddings.Examples include Hourglass Transformer, MegaByte, BLT, H-Net, and ContextLM.
  • Prediction Objectives and Latent States: Multi-token prediction adds auxiliary heads for future tokens, while other methods interleave or predict continuous hidden states; ConceptLM introduced discrete NCP.These approaches modify prediction targets or incorporate latent states into generation.
  • Depth-Wise Representation Mixing: Depth-mixing methods selectively reuse earlier representations through learned averaging, depth-wise cross-attention, or token-conditioned dense connections.DenseFormer, DeepCrossAttention, and MUDDFormer differ in whether their mixing weights are input-independent or input-dependent.

7. Limitations

The report evaluates NCP-ArchPreview at standard context lengths and notes that lower language-modeling loss does not uniformly translate into downstream gains.

  • Long-Context Scope: Long-context training is outside the current architecture preview, which covers 5.73T pretraining and 100B mid-training tokens at standard context lengths.Scaling concept prediction and hierarchical residual routing to longer dependencies remains future work.
  • Loss and Downstream Performance: 2.45 points of downstream macro-average gain after pretraining contrasts with a 0.59-point mid-training gain that varies across benchmarks.NCP-ArchPreview maintains a token-level loss advantage at both checkpoints, but the downstream relationship is stage- and task-dependent.

8. Conclusion

NCP-ArchPreview validates joint token and concept prediction at trillion-token scale, achieving strong efficiency and downstream gains while enabling lightweight adaptation and inference improvements.

  • 8. Conclusion: 2.45 points higher downstream macro-average than OLMo-3-7B, including a 5.99-point GSM8K gain, after pretraining on 5.73T tokens.It reaches OLMo-3-7B's final pretraining loss using 51.3% of its training tokens.
  • 8. Conclusion: NCP-ArchPreview jointly models tokens and learned discrete concepts in an 8.9B-parameter latent-space architecture.The work presents this as the largest-scale validation of latent-space language modeling to date.
  • 8. Conclusion: The complete model approaches a parameter-aligned 40-layer Transformer using 85% of its computation, while scaling-law experiments report 1.74× higher compute efficiency.
  • 8. Conclusion: Updating only 17M VQ parameters supports domain adaptation, and injecting concept representations into a DFlash2 drafter improves mean accepted length by 4.17%.The reported inference improvement has negligible overhead.

B. Additional Architecture Details

The architecture uses hierarchical residual routing to connect token-level and concept-level modules throughout the end-to-end training pass.

  • B. Additional Architecture Details: Table 10 summarizes the complete NCP-ArchPreview architecture configuration.
  • B. Additional Architecture Details: Algorithm 1 routes token and concept information through intra-module and cross-module residual connections during training.The alignment operations are specified by Eqs. 14–20 and Eq. 11.
  • B. Additional Architecture Details: ChunkPool_k converts token-level residual states to concept resolution, while CausalShiftAndRepeat returns concept states to token resolution without exposing future concepts.

C. Analysis of OLMo-3 Mid-Training Data with Proxy Metrics

The proxy-metric analysis evaluates matched-budget Stage-2 recipes with fixed held-out trajectories and task-specific scoring, using an end-to-end training procedure for the model inputs.

  • C. Analysis of OLMo-3 Mid-Training Data with Proxy Metrics: Three matched-budget Stage-2 recipes are compared using a fixed, capability-specific held-out set, with Stage 1 retained only as a cross-stage reference.
  • C. Analysis of OLMo-3 Mid-Training Data with Proxy Metrics: The held-out set contains 177,202 expert trajectories from 63 sources and six domains, totaling approximately 100M teacher-suffix tokens under one generation protocol.
  • C. Analysis of OLMo-3 Mid-Training Data with Proxy Metrics: The training forward pass accepts tokens, PQ codebooks, and loss weights, then aligns token and concept states through chunk pooling and causal shift-and-repeat.
  • C. Analysis of OLMo-3 Mid-Training Data with Proxy Metrics: Scores are aggregated by averaging within sources and then within capability leaves, preventing large benchmarks from dominating by sample count.Free-form tasks use target-token negative log-likelihood, while HellaSwag uses a task-native correct-versus-distractor statistic.

C.2. Recipe-level Results

Recipe-level proxy metrics generally identify V1 as strongest across free-form capabilities, while HellaSwag favors a statistically tied V2/V3 tier; the screening method remains limited to the evaluated setting.

  • C.2. Recipe-level Results: V1 has the lowest NLL and best downstream results across all four free-form capability leaves, consistently outperforming V2 and V3.Compared with V3, V1 improves HumanEval by 5.64 points, MBPP by 4.31, MATH-500 by 6.52, Minerva by 5.50, MMLU-STEM by 4.11, and BBH by 2.79.
  • C.2. Recipe-level Results: HellaSwag favors V2 and V3, whose downstream scores are 67.30 and 67.25 versus 66.40 for V1, so V2 and V3 are treated as effectively tied.The task-native choice margin distinguishes the leading V2/V3 tier from V1.
  • C.3. Interpretation and Scope: A new mid-training recipe can be screened with one frozen forward pass using capability-level NLLs and task-native margins.The evidence does not establish calibrated prediction for unseen model families or causal attribution to individual data components.

D. Scaling Experiments

The scaling-ladder experiments allocate model size and training tokens across fixed FLOPs budgets, using an architecture-aware computation estimate because the Concept Module activates only once per chunk.

  • Scaling-ladder design: Architecture-aware FLOPs account for the Concept Module's once-per-chunk activation, rather than relying solely on C≈6N_paramD.F_tok denotes analytical training FLOPs per token, and D denotes the training-token count; embedding parameters are excluded.
  • Scaling-ladder design: Each FLOPs budget is evaluated with multiple allocations between model size and training tokens.The experiments select several FLOPs budgets and design multiple model-size/token-count allocations for each.
  • Hyperparameter search: Hyperparameter searches identify representative learning rates and batch sizes before tuning other models near the selected learning rate.A representative model is searched at each FLOPs budget, then other models at that budget are searched in a neighborhood around the chosen learning rate.

E. Evaluation Details

The evaluation separates higher-is-better task scores from lower-is-better BPB likelihood scores, defines grouped and overall aggregates, and fixes evaluation procedures across models.

  • Benchmark coverage: The higher-is-better results cover MMLU, mathematical reasoning, code generation, multiple-choice reasoning, and general question answering groups.The reported groups include MMLU AVG, MATH AVG, Code AVG, MC-STEM AVG, MC-Non-STEM AVG, and GenQA AVG.
  • Evaluation controls: Evaluation uses fixed prompts, demonstrations, splits, scoring implementations, and a global random seed of 42 across models.Few-shot selection and generation use the shared seed, while prompting and scoring configurations remain fixed.
  • Likelihood evaluation: Lower BPB indicates better likelihood performance, and its independently computed AVG is excluded from domain and higher-is-better averages.The likelihood block reports bits per byte across ten benchmarks, with MT-MBPP Gold macro-averaged across 17 language subsets before benchmark-level aggregation.
  • Generative evaluation: Generative tasks use task-specific prompting and sampling configurations, with the number of completions reported independently from inference batch size.The configuration table covers benchmarks requiring free-form generation.
  • Aggregation: Higher-is-better Overall AVG averages 26 constituent benchmark results, excluding aggregate and intermediate AVG rows.Section averages are unweighted means of their constituent results, while the overall average is computed directly across the 26 benchmark results.
  • Domain adaptation: Domain adaptation evaluates code, mathematics, and knowledge adaptations on HumanEval+/MBPP+, GSM8K/MATH-500, and TriviaQA, respectively.Stage-1 checkpoint trajectories are reported separately in Table 14 using the same benchmark grouping and separate BPB aggregation.
Loading 2609.10715v1…