PaperlayerView exact revision on arXiv

Source-linked AI summary

SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens

Nikita Dragunov, Temurbek Rahmatullaev, Elizaveta Goncharova, Nikita Kurdiukov, Aysel Mirzoeva, Anna Borisiuk, Andrey Kuznetsov, Anton Razzhigaev

arXiv:2508.05305v2cs.CL

TL;DR

SONAR-LLM addresses the tension between efficient sentence-level generation and stable likelihood-based training. It predicts sentence embeddings with a decoder-only Transformer but supervises them through token-level cross-entropy via a frozen SONAR decoder. Across generation and summarization evaluations, it matches or outperforms relevant baselines, with scope limited by English-only experiments and model-based quality judging.

  • Problem

    Token-level decoding can bottleneck long-sequence generation, while LCM sentence-embedding models replace token-level likelihoods with MSE or diffusion objectives.

  • Method

    SONAR-LLM autoregressively predicts SONAR sentence embeddings and propagates token-level cross-entropy through the frozen SONAR decoder.

  • Results

    SONAR-LLM outperformed both LCM variants in GPT-4o evaluations, matched the standard token-level LLM on standard NLG metrics, and achieved stronger performance than prior sentence-level baselines on XSum and CNN/DM.

  • Takeaways & Limitations

    The results support SONAR-LLM as a promising sentence-level generation approach combining semantic abstraction with likelihood-based supervision.

  • Takeaways & Limitations

    Experiments are limited to English-language data, and GPT-4o-based quality assessments remain subject to the behavior and biases of the underlying model.

Abstract

from arXiv · show

The recently proposed Large Concept Model (LCM) generates text by predicting a sequence of sentence-level embeddings and training with either mean-squared error or diffusion objectives. We present SONAR-LLM, a decoder-only transformer that "thinks" in the same continuous SONAR embedding space, yet is supervised through token-level cross-entropy propagated via the frozen SONAR decoder. This hybrid objective retains the semantic abstraction of LCM while eliminating its diffusion sampler and restoring a likelihood-based training signal. Across model sizes from 39M to 1.3B parameters, SONAR-LLM attains competitive generation quality. We report scaling trends, ablations, benchmark results, and release the complete training code and all pretrained checkpoints to foster reproducibility and future research.

1 Introduction

SONAR-LLM combines sentence-level embedding prediction with token-level cross-entropy through a frozen SONAR decoder, addressing limitations of both token-by-token decoding and LCM objectives. The paper evaluates scaling, text quality, summarization, inference efficiency, and reproducibility.

  • Token-level language models provide simple likelihood training but can become throughput bottlenecks on long sequences, whereas LCM reduces trajectory length using sentence embeddings but relies on MSE or diffusion objectives.The paper motivates SONAR-LLM as a hybrid alternative to these approaches.
  • SONAR-LLM predicts SONAR sentence embeddings autoregressively while propagating token-level cross-entropy through the frozen decoder.This couples continuous sentence-level reasoning with discrete token supervision.
  • Scaling Laws Analysis: SONAR-LLM is evaluated across model sizes with scaling-law fits for validation losses spanning LLM, LCM, and SONAR-LLM architectures.The scaling analysis quantifies exponents across the compared model families.
  • Text Quality Evaluation: SONAR-LLM is reported to achieve consistently higher generated-text quality than LCM-based sentence-level models using BLEU, ROUGE-L, METEOR, and GPT-4o judging.The GPT-4o evaluation is part of the paper’s text-quality assessment.
  • Summarization Evaluation: On XSum and CNN/DM summarization benchmarks, SONAR-LLM matches or exceeds other sentence-level approaches.
  • Inference Efficiency Analysis: Sentence-level inference has favorable long-sequence FLOP scaling compared with token-level decoding, and the authors release training code, evaluation scripts, and checkpoints.The paper presents both efficiency analysis and an open-source release for follow-up research.

2 Related Work

Related work spans token-level Transformers, latent-variable sentence generation, diffusion and flow-based text models, and LCM. SONAR-LLM extends this progression by predicting sentence embeddings while retaining likelihood-based supervision.

  • Standard language models use next-token cross-entropy over discrete vocabularies, while MAMBA and MAMBA-2 explore alternatives to self-attention for faster long-sequence processing.The cited work frames efficiency as a major direction in autoregressive modeling.
  • Continuous and discrete VAEs generate sentences from latent codes, with VQ-VAE compressing sentences into short sequences of discrete indices before autoregressive decoding.
  • LCM builds an autoregressive prior over SONAR embeddings and explores MSE, quantization, and diffusion losses, while SONAR-LLM restores token-level cross-entropy through the frozen decoder.SONAR provides a language-agnostic, multimodal sentence-embedding space covering 200 languages.
  • Diffusion-based text models denoise continuous or discrete representations, while score-entropy objectives and masked diffusion seek to narrow their gap with autoregressive baselines.
  • Flow matching applies continuous normalizing flows to text, and SONAR-LLM bridges these directions through autoregressive sentence-embedding prediction with likelihood-based supervision.FLOWSEQ is cited as generating sentences in a handful of ODE steps.

3 SONAR-LLM

SONAR-LLM is a decoder-only Transformer that predicts continuous SONAR sentence embeddings from embedding prefixes, then decodes them into sentences through the frozen SONAR decoder. Training uses teacher-forced token prediction rather than MSE or diffusion objectives.

  • 3 SONAR-LLM: SONAR-LLM predicts the next sentence embedding from a prefix of sentence embeddings and decodes that prediction with the frozen SONAR decoder.The overall architecture is illustrated in Figure 1.
  • 3 SONAR-LLM: Text is segmented with Punkt, encoded by the frozen multilingual SONAR encoder into 1024-dimensional vectors, and used to train next-sentence predictions against the true sentence.
  • 3.2 Model Architecture: The model is a decoder-only Transformer with an embedding vocabulary of size one, so each step predicts a continuous vector rather than a discrete token.Variants scale from 39 M to 900 M parameters and use rotary position encodings and RMS-norm.
  • 3.3 Cross-Entropy Through the Frozen Decoder: Predicted sentence embeddings are decoded into token logits with teacher forcing, and cross-entropy is minimized against the ground-truth token sequence while gradients pass through the frozen decoder.Teacher forcing also supplies the ground-truth embedding at the next time step.
  • 3.3 Cross-Entropy Through the Frozen Decoder: Inference appends an encoded “End of sequence.” sentence and stops when cosine similarity exceeds τ_stop=0.98 or when 32 sentences have been generated.

4 Results

Across scaling, generation, summarization, and efficiency analyses, SONAR-LLM scales effectively and outperforms sentence-level LCM baselines, while remaining competitive with token-level LLMs on several tasks.

  • Scaling laws: The fitted validation-loss scaling exponent for SONAR-LLM is α ≈0.596, comparable to embedding-based models, and all architecture fits achieve R2 > 0.97.The fits use epoch-4 losses across all model sizes and three random seeds, yielding 15 data points per architecture.
  • Automatic Evaluation with GPT-4o: The 900 M SONAR-LLM achieves grammar 7.1, creativity 5.2, coherence 6.0, and plot consistency 5.2, outperforming both LCM variants across all four GPT-4o-rated metrics.The 900 M token-level LLM remains strongest overall, reaching grammar 9.1, coherence 8.1, and plot consistency 7.0 with beam sampling.
  • NLG Metrics: At 900 M, SONAR-LLM matches the LLM on short-prefix ROUGE-L (39.4 vs. 39.4) and remains competitive on METEOR (32.9 vs. 34.2) and long-context ROUGE-L (35.2 vs. 35.7).Both MSE-based and diffusion-based LCMs perform substantially worse across the reported NLG metrics.
  • Summarization Evaluation: SONAR-LLM substantially outperforms MSE and diffusion LCMs on XSum and CNN/DM, matches token-level LLMs on XSum, but trails them on CNN/DM.CNN/DM favors more extractive approaches, whereas XSum is more abstractive.
  • Unfreezing the SONAR Decoder: Unfreezing the decoder reduces validation cross-entropy by more than 10% without improving standard summarization quality, but raises RULER NIAH exact-match accuracy from 21.6% to 41%.The reported benefit is therefore concentrated in numerical or symbolic retrieval rather than standard natural-language generation.
  • Inference Efficiency: SONAR-LLM surpasses standard LLM inference efficiency from approximately 4096 input tokens onward because sentence-level decoding requires fewer decoding steps.Standard token-level LLMs retain an advantage on shorter sequences.

5 Conclusion

SONAR-LLM combines sentence-level embedding prediction with token-level cross-entropy, achieving competitive generation, summarization, scaling, and long-context efficiency results. Decoder unfreezing helps precise symbol reproduction but is unnecessary for semantically grounded text.

  • Generation Evaluation: SONAR-LLM outperformed both MSE-based and diffusion-based LCMs across GPT-4o evaluations and consistently matched the standard token-level LLM on NLG metrics.The proof-of-concept study used TINYSTORIES and evaluated grammar, coherence, creativity, plot consistency, and standard metrics.
  • Overall Findings: SONAR-LLM achieved stable optimization and competitive scaling trends across model sizes, combining semantic abstraction with a likelihood-based training signal.The model predicts sentence embeddings while propagating token-level cross-entropy through a frozen SONAR decoder.
  • Summarization Evaluation: SONAR-LLM achieved stronger summarization performance than sentence-level baselines, matched token-level LLMs on XSum, and remained behind them on CNN/DM.These evaluations used pretrained models on diverse instructional and open-domain corpora.
  • Decoder Ablation: Unfreezing the SONAR decoder improves pretraining cross-entropy and nearly doubles exact-match accuracy on symbol-heavy retrieval tasks, without measurable summarization gains.Decoder adaptation is therefore useful for precise numeric or alphanumeric reproduction but unnecessary for semantically grounded text.
  • Inference Efficiency: Beyond 4096 tokens, inference cost grows almost linearly with sequence length up to 1 million tokens, although the underlying complexity remains quadratic.The favorable behavior results from operating on sentence-level segments rather than individual tokens.

Limitations

The evaluation is limited by reliance on GPT-4o-based judgments and by experiments conducted only on English-language data. These constraints leave human-annotated and multilingual validation for future work.

  • Scope: The paper identifies several limitations in the evaluated architectures and calls for more complete validation beyond the reported experiments.The stated limitations concern evaluation methodology and language coverage.
  • Evaluation Scope: GPT-4o assessments of grammar, coherence, creativity, and plot consistency remain limited by the evaluator’s behavior and biases.The paper identifies direct human annotation and broader qualitative analysis as more complete alternatives.
  • Language Scope: The experiments use English-language data, so the reported results may not directly generalize to other languages.The multilingual SONAR encoder–decoder suggests possible broader applicability, but thorough multilingual evaluation remains future work.

Impact Statement

SONAR-LLM offers a sentence-level alternative for efficient long-context language generation while retaining token-level likelihood training. Its abstraction does not create fundamentally new generative capabilities, and higher-level operation may alter text properties in application-dependent ways.

  • Potential Benefits: Sentence-level processing in continuous embedding space is intended to support more efficient handling of long contexts and scalable language generation.The approach predicts and processes sentences rather than individual tokens while retaining token-level likelihood training.
  • Capability Scope: Operating in sentence embeddings does not introduce fundamentally new generative capabilities compared with existing large language models.The types of outputs and potential misuse scenarios remain largely similar to standard LLMs.
  • Application Considerations: Higher-level abstraction may amplify or suppress properties such as verbosity or structural coherence, affecting downstream applications in subtle ways.The paper frames these effects as potential consequences rather than established outcomes.
  • Ethical Considerations: The paper identifies no novel ethical risks specific to SONAR-LLM beyond those already present in current language models.Its expected impact is to support research on efficient long-context modeling, semantic abstraction, and larger reasoning units.
Loading 2508.05305v2…