PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Autoregressive Language Models

Chenze Shao, Darren Li, Fandong Meng, Jie Zhou

arXiv:2510.27688v1cs.CLcs.AIcs.LG

TL;DR

Token-by-token autoregressive generation makes long sequences computationally costly. CALM encodes K-token chunks as continuous vectors and develops likelihood-free tools for modeling them; experiments report a stronger performance-compute trade-off, including comparable BrierLM performance with fewer FLOPs.

  • Problem

    Autoregressive token-by-token generation makes long sequences computationally costly because computational cost scales with sequence length.

    Source: §1 · p. 1

  • Method

    CALM encodes each K-token chunk as one continuous vector and uses a likelihood-free framework for continuous-domain language modeling.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    44% fewer training FLOPs and 34% fewer inference FLOPs let 371M CALM-M achieve a BrierLM score comparable to the 281M Transformer-S baseline.

    Source: §7.2 · p. 15, §8 · p. 20

  • Takeaways & Limitations

    CALM results highlight scaling the semantic bandwidth of each generative step as a design axis for improving the performance-compute frontier.

    Source: §8 · p. 20

    The exact temperature sampler can become costly near T → 1 and highly rejection-prone at low temperatures.

    Source: §5.2 · p. 11

Abstract

from arXiv · show

The efficiency of large language models (LLMs) is fundamentally limited by their sequential, token-by-token generation process. We argue that overcoming this bottleneck requires a new design axis for LLM scaling: increasing the semantic bandwidth of each generative step. To this end, we introduce Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9\% accuracy. This allows us to model language as a sequence of continuous vectors instead of discrete tokens, which reduces the number of generative steps by a factor of K. The paradigm shift necessitates a new modeling toolkit; therefore, we develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling in the continuous domain. Experiments show that CALM significantly improves the performance-compute trade-off, achieving the performance of strong discrete baselines at a significantly lower computational cost. More importantly, these findings establish next-vector prediction as a powerful and scalable pathway towards ultra-efficient language models. Code: https://github.com/shaochenze/calm. Project: https://shaochenze.github.io/blog/2025/CALM.

1 INTRODUCTION

Token-by-token autoregressive generation makes long sequences computationally costly, and discrete tokens offer limited information density. CALM compresses K-token chunks into continuous vectors and introduces likelihood-free methods for training, evaluation, and sampling; experiments report a better performance-compute trade-off.

1 INTRODUCTION

  • Discrete token-by-token generation makes long sequences costly, while increasing token information density would require untenable vocabulary-softmax growth.Modern LLM tokens carry approximately 15 to 18 bits, and sequence length drives computational cost.Source: §1 · pp. 1–2
  • CALM encodes K tokens in one continuous vector, reducing autoregressive steps K-fold and shifting language modeling to next-vector prediction.The paper describes the vector capacity as expandable by increasing its dimensionality.Source: §1 · p. 2, Figure 1 · p. 2
  • A lightweight autoencoder supplies robust vector representations that reconstruct the original tokens with high fidelity.The autoencoder is presented as a prerequisite for downstream language modeling.Source: §1 · p. 2
  • The framework uses an Energy Transformer for single-step continuous-vector generation rather than iterative diffusion or flow-matching sampling.The paper reports superior generation quality for the adopted architecture.Source: §1 · p. 2
  • BrierLM provides strictly proper sample-based evaluation without explicit likelihoods, while a likelihood-free algorithm supports temperature-controlled sampling.The temperature method includes an efficient batch approximation.Source: §1 · pp. 2–3

2 AUTOENCODER

The autoencoder compresses token chunks into continuous vectors while balancing reconstruction fidelity with latent-space robustness. Variational regularization, KL clipping, and training-time dropout address brittle representations and posterior collapse.

2.1 HIGH-FIDELITY RECONSTRUCTION

  • The context-free autoencoder maps each K-token chunk to a continuous vector and reconstructs tokens; at K=4, just 10 dimensions achieve over 99.9% token accuracy.The model is designed to process chunks independently of surrounding sequence context.Source: §2.1 · p. 3

2.2 ROBUST VECTOR REPRESENTATION

  • Reconstruction-only training produces brittle latent representations, where small vector perturbations can yield unrelated token sequences.The autoencoder therefore needs representations robust to downstream generative-model errors.Source: §2.2 · p. 3
  • Variational encoding samples z from a diagonal Gaussian and penalizes deviation from a standard normal prior to smooth and regularize the latent manifold.The regularization discourages reliance on arbitrarily precise or large-magnitude latent values.Source: §2.2 · p. 4
  • Posterior collapse makes latent dimensions uninformative and can destabilize downstream training; KL clipping encourages every dimension to contribute to reconstruction.The method clips each dimension’s KL loss at a floor.Source: §2.2 · p. 4
  • Latent-vector dropout and token masking, each at p = 0.15, build redundancy and chunk context during autoencoder training only.Both forms of dropout are disabled during later continuous language-model training and inference.Source: §2.2 · p. 4
  • With K=4, a 128-dimensional latent vector retains over 99.9% token-level accuracy despite Gaussian perturbation with σ ≈ 0.3.This robust representation supports subsequent CALM training.Source: §2.2 · p. 4

3 LIKELIHOOD-FREE LANGUAGE MODELING

CALM groups tokens into continuous vectors, requiring likelihood-free alternatives to discrete softmax prediction and likelihood-based training and evaluation. The section develops a single-step generative head, energy-score training, and an autoregressive architecture that feeds reconstructed token chunks forward.

3.1 NEXT-VECTOR PREDICTION

  • Grouping T tokens into L = T/K chunks yields a continuous vector sequence for next-vector autoregressive prediction.Each vector encodes one non-overlapping K-token chunk.Source: §3.1 · p. 5
  • Because the next vector lies in R^l, its explicit likelihood is intractable, precluding maximum-likelihood training and perplexity evaluation.The paper addresses training and evaluation with likelihood-free methods.Source: §3.1 · p. 5

3.2 GENERATIVE HEAD

  • CALM conditions a stochastic generative head on a Transformer hidden state; diffusion and flow-matching alternatives require iterative sampling that counteracts the step-count savings.The design target is high-quality single-step vector generation.Source: §3.2 · p. 5

3.3.1 STRICTLY PROPER SCORING RULES

  • Strictly proper scoring rules train the generative head by rewarding predictive distributions that match the true distribution, extending beyond likelihood-based objectives.The logarithmic score underlying negative log-likelihood is a special case.Source: §3.3.1 · p. 6

3.3.2 ENERGY LOSS

  • The energy score measures predictive alignment through sample distances rather than probability densities and is strictly proper for α ∈ (0, 2).Its first term encourages diversity, and its second encourages fidelity to the observation.Source: §3.3.2 · p. 6
  • An unbiased Monte Carlo energy-loss estimator draws N model samples and M target samples from the autoencoder’s conditional Gaussian posterior.In practice, N = 8 and M = 100; the larger target sample set reduces loss variance at almost negligible overhead.Source: §3.3.2 · p. 6

3.3.3 MODEL ARCHITECTURE

  • Residual MLP blocks fuse noise with the Transformer hidden state and refine it into the predicted continuous vector.Figure 2 depicts the generative head in the model architecture.Source: §3.3.3 · p. 7, Figure 2 · p. 7
  • The model compresses the previous K tokens for Transformer input, predicts a continuous vector, then decodes it into the next K-token chunk.Using latent vectors directly as Transformer input degraded performance, motivating the discrete-token feedback loop.Source: §3.3.3 · pp. 7–8

4 LIKELIHOOD-FREE LM EVALUATION

Because CALM lacks tractable likelihoods, conventional perplexity and energy loss cannot evaluate it directly. The section develops BrierLM, a sample-based metric for assessing language-model predictions across both implicit and conventional models.

4.1 PRINCIPLES OF LM EVALUATION

  • A principled score is uniquely optimized when the model recovers the data distribution; raw observed-outcome likelihood instead favors deterministic predictions and ignores uncertainty.The section uses this distinction to motivate a score that rewards accuracy while representing predictive uncertainty.Source: §4.1 · p. 8

4.2 BRIERLM: BRIER FOR LANGUAGE MODELING

  • BrierLM applies the strictly proper Brier score to language modeling, and its estimator uses model samples rather than requiring tractable likelihoods.The direct Brier score requires the full predictive distribution, which is intractable for CALM.Source: §4.2 · pp. 8–9
  • BrierLM aggregates Brier-n scores for n-grams from n=1 to 4, treating each n-gram as an atomic outcome and scaling the geometric mean by 100.This extends evaluation beyond the first token of each generated chunk, which a next-token teacher-forcing evaluation would omit.Source: §4.2 · p. 9
  • BrierLM can also evaluate conventional autoregressive models from samples drawn from their final softmax, enabling comparisons with likelihood-free models.The section also presents it as a direct, unbiased evaluation option for implicit generative models that otherwise rely on variational-bound estimates.Source: §4.2 · p. 9
  • Pearson correlation was -0.966 and Spearman rank correlation was -0.991, showing BrierLM aligned nearly linearly with cross-entropy across models and checkpoints.Figure 3 plots their joint distribution across models and training checkpoints.Source: Figure 3 · p. 9, §4.2 · p. 9

5 LIKELIHOOD-FREE TEMPERATURE SAMPLING

The section develops rejection-based temperature sampling for implicit models that expose only a sampler, then addresses its temperature-dependent cost with a batch approximation for low temperatures.

5.1 EXACT TEMPERATURE SAMPLING VIA REJECTION

  • For any temperature T in (0, 1), the exact sampler uses rejection sampling in two stages to produce samples proportional to P(x)^(1/T).It handles the integer part of 1/T by accepting identical repeated draws, then handles the fractional part with a Bernoulli Factory procedure.Source: §5.1 · p. 10

5.2 EXPECTED SAMPLING COST

  • The exact algorithm's expected sampler-call cost is temperature-sensitive: as T approaches 1 it can reach the sample-space size, while low T causes extremely high rejection.At low temperatures, the requirement for n identical draws becomes vanishingly unlikely as n grows.Source: §5.2 · p. 11

5.3 BATCH APPROXIMATION

  • At low temperatures T=1/n, the batch approximation searches N samples for repeated candidates, improving sample utilization over repeated all-or-nothing trials.If no candidate meets the n-match requirement, it lowers the matching threshold until it finds a nonempty candidate set.Source: §5.3 · pp. 11–12
  • The batch algorithm is biased for finite N but asymptotically unbiased as N grows, making batch size a lever for trading efficiency against accuracy.Its target distribution is P(x)^n normalized, for T=1/n.Source: §5.3 · p. 12
  • Because it needs only a black-box sampler, the batch approximation can also be used for implicit language models beyond CALM.The paper presents it as a general toolkit for controlled decoding in discrete spaces.Source: §5.3 · p. 12

6 RELATED WORK

The related-work discussion positions CALM against latent compression, continuous autoregressive generation, likelihood-free evaluation, and black-box temperature sampling, emphasizing its focus on robust language representations and efficient generation.

6.1 AUTOENCODER

  • Unlike prompt-compression work that emphasizes reconstruction fidelity, CALM prioritizes a robust, smooth latent manifold for downstream generative modeling.Its discrete-to-continuous mapping is intended to reduce the number of autoregressive steps needed for language generation.Source: §6.1 · pp. 12–13

6.2 LIKELIHOOD-FREE LANGUAGE MODELING

  • For continuous autoregressive generation, CALM adopts the Energy Transformer framework and modifies its head, energy loss, and input structure for language modeling.The discussion contrasts this with Gaussian-mixture expressiveness limits and diffusion's iterative inference, and with prior chunk or sentence approaches that retain sequential or diffusion decoding challenges.Source: §6.2 · p. 13

6.3 LIKELIHOOD-FREE LM EVALUATION

  • BrierLM bridges likelihood-based and sample-based evaluation by using model samples while retaining the strictly proper scoring-rule guarantee.The paper's stated innovations are unbiased likelihood-free estimation of the Brier score and extending it from classification to language modeling.Source: §6.3 · p. 14

6.4 LIKELIHOOD-FREE TEMPERATURE SAMPLING

  • The temperature sampler turns the implicit-distribution problem into a Bernoulli Factory step after isolating a candidate, while offering black-box exact sampling rather than heuristic controls.The discussion contrasts its approach with control methods that require white-box access to model internals.Source: §6.4 · p. 14

7 EXPERIMENTS

The experiments compare CALM with Transformer baselines and test how chunk size, autoencoder design, generative-head choices, and temperature-sampling settings affect performance and compute. CALM with K=4 reaches a comparable BrierLM score to a smaller Transformer while using fewer training and inference FLOPs.

7.2 MAIN RESULTS

  • A 371M-parameter CALM-M matches the 281M Transformer-S BrierLM score with 44% fewer training FLOPs and 34% fewer inference FLOPs.Table 1 includes autoencoder parameter and FLOP overhead; the results also report that CALM improves consistently with model size.Source: §7.2 · p. 15, Table 1 · p. 15
  • At K=4, CALM-L surpasses the Transformer performance-compute frontier; K=2 nearly halves cost with a marginal performance drop, while K=8 drops performance.At K=1, CALM-L requires more FLOPs for lower performance than its discrete counterpart; the authors attribute the K=8 drop likely to model-capacity limits.Source: §7.2 · p. 15, Figure 4 · p. 16
  • CALM-XL initially trails Transformer-M but progressively closes the gap with Transformer-L as its learning curve steepens.The authors attribute CALM’s slower start to modeling a complex continuous-vector distribution rather than a low-information discrete token.Source: §7.2 · p. 16, Figure 5 · p. 16

7.3 EFFECT OF AUTOENCODER

  • KL clipping remedies posterior collapse, where 71 of 128 latent dimensions collapsed, and dropout on input tokens and latent vectors adds orthogonal performance benefits.The reconstruction-only autoencoder provides a reasonable baseline, while naively adding a variational objective degrades downstream performance.Source: §7.3 · p. 16, Table 2 · p. 17
  • BrierLM performance peaks at β=0.001 and latent dimension l=128; aggressive KL regularization at β=0.1 reduces reconstruction accuracy to about 99%.Reconstruction stays consistently high across tested latent dimensions, but downstream performance varies; scaling the autoencoder’s layers, width, or training data yields no significant BrierLM improvement.Source: §7.3 · pp. 16–17, Figure 6 · p. 17, Figure 7 · p. 17

7.4 EFFECT OF MODEL ARCHITECTURE

  • Flow matching and the energy-based head outperform diffusion, with flow matching converging faster initially and the energy-based head reaching a higher performance ceiling.The energy-based head avoids iterative decoding; flow matching’s midpoint sampler achieves decent quality in 2 steps and near-optimal performance within 4.Source: §7.4 · p. 18, Figure 8 · p. 18, Figure 9 · p. 18
  • Increasing energy-score sample counts improves BrierLM at nearly linear training-cost growth, while discrete model inputs outperform combined or purely continuous inputs.The energy-score exponent performs best at α=1; training fails for α<1 and BrierLM falls to 0 at α=2.Source: §7.4 · pp. 18–19, Table 3 · p. 18, Table 4 · p. 19

7.5 TEMPERATURE SAMPLING

  • Larger sampling batches and lower temperatures increase accuracy at the cost of diversity, with batch size covering more of the trade-off than temperature; CALM’s trajectory nearly matches Transformer temperature sampling.Matching Transformer temperature 0.6 requires about N=100 samples in CALM, while matching 0.5 requires N=200.Source: §7.5 · pp. 19–20, Figure 10 · p. 20, Figure 11 · p. 20

8 CONCLUSION AND FUTURE WORK

CALM models chunks of tokens as continuous vectors with a likelihood-free toolkit, and its experiments show a superior performance-compute trade-off. The conclusion identifies remaining opportunities in autoencoding, generative modeling, sampling, scaling, and adapting the algorithmic toolkit.

8 CONCLUSION AND FUTURE WORK

  • CALM shifts generation from discrete tokens to continuous vectors representing K tokens, supported by an autoencoder, energy loss, BrierLM, and temperature-sampling algorithms.The authors report a superior performance-compute trade-off and identify semantic bandwidth per generative step as a scaling axis.Source: §8 · p. 20
  • The current autoencoder prioritizes reconstruction over semantic structure; semantically grounded latents and context-aware or autoregressive designs are proposed future directions.The latter designs could offer superior robustness and more reliable reconstruction.Source: §8 · pp. 20–21
  • The current model pairs a Transformer backbone with a lightweight generative head; an integrated generative Transformer or alternative training objectives may improve generative modeling or sample quality.The authors also suggest exploring other strictly proper scoring rules or generative models.Source: §8 · p. 21
  • Rejection sampling for temperature control can add significant inference overhead, motivating exploration of lighter heuristic methods for the diversity-fidelity trade-off.Suggested directions include scaling input noise or fine-tuning with a modified loss.Source: §8 · p. 21
  • CALM’s scaling properties remain to be investigated, including whether larger models can support higher semantic bandwidth and how to formulate scaling laws that include K.The proposed laws could help select K for a compute budget.Source: §8 · p. 21
  • Sample-based CALM lacks directly computable log-probabilities and full probability mass functions, leaving policy optimization and KL-based distillation as open reformulation challenges.The authors identify these as algorithmic-toolkit questions for future work.Source: §8 · p. 21

A.1 PROOF OF THEOREM 1

The proof establishes that Algorithm 1’s rejection sampling yields the temperature-adjusted distribution by accepting outcomes with probability proportional to P(x)^(1/T). It decomposes the inverse temperature into integer and fractional stages.

A.1 PROOF OF THEOREM 1

  • Algorithm 1 accepts sample x with probability P(x)^(1/T), so rejection sampling yields the desired normalized temperature distribution.The proof identifies the acceptance probability as the basis for the result.Source: §A.1 · p. 30
  • Writing 1/T = n + α splits the acceptance probability into integer and fractional stage success probabilities.Here n is the integer part and α the fractional part of the inverse temperature.Source: §A.1 · p. 30
  • In stage 1, the candidate x must be drawn n consecutive times, giving probability P(x)^n; stage 2 handles the fractional exponent.The fractional-stage acceptance probability is expressed as a cumulative sum over possible iterations.Source: §A.1 · p. 30

A.2 PROOF OF THEOREM 2

The proof analyzes Algorithm 1’s expected sampler-call cost by treating repeated trials as geometric and separating the integer and fractional temperature components. It gives cost bounds for low- and high-temperature regimes.

A.2 PROOF OF THEOREM 2

  • Repeated independent trials continue until acceptance, so expected total sampler calls equal expected calls per trial divided by the trial success probability.The proof identifies success as acceptance of any sample and sums acceptance probabilities over outcomes.Source: §A.2 · p. 31
  • Each trial uses n sampler calls for the integer stage; the fractional stage is skipped when α = 0 and is conditionally run when α > 0.The fractional stage is entered only if stage 1 produces a candidate.Source: §A.2 · p. 31
  • The proof establishes expected-call bounds separately for low-temperature and high-temperature regimes, using an exponent bound in the former and concavity with a uniform distribution in the latter.For the low-temperature regime, the numerator of the cost formula is bounded by n + 1.Source: §A.2 · p. 32

A.3 PROOF OF THEOREM 3

The proof shows that Algorithm 2’s batch-based sampling probability converges to the target temperature distribution as batch size grows. Its argument uses convergence of sample counts and bounded convergence of the resulting sampling probabilities.

A.3 PROOF OF THEOREM 3

  • For batch size N, the algorithm’s sampling probability is the expectation of a random variable XN, and the proof targets convergence to PT(x).The target is the temperature distribution defined in the theorem.Source: §A.3 · p. 33
  • The proof uses the weak law of large numbers to show sample proportions converge to P(x), then derives convergence of normalized weights and XN.The weights are expressed as a polynomial in the count Cx and normalized by N^n.Source: §A.3 · p. 33
  • Because XN is bounded and converges in probability to PT(x), bounded convergence establishes that the algorithm is asymptotically unbiased as N grows.The proof concludes that the expectation of XN converges to the target probability.Source: §A.3 · p. 33
Loading 2510.27688v1…