Large Concept Models: Language Modeling in a Sentence Representation Space

Motivation

• The approach addresses a limitation of current decoder-only LLMs, which predict tokens but lack explicit hierarchical reasoning and planning across levels of abstraction.The paper motivates this contrast with human workflows such as outlining a talk or document before producing its words.Source: §1 · p. 2

Concept-level modeling

• Large Concept Models represent abstract reasoning independently of any particular language or modality, using concepts rather than token-level representations.In this proof of concept, a concept is typically a sentence or equivalent speech utterance represented in an embedding space.Source: §1 · p. 2

Architecture

• An LCM encodes input sentences into SONAR concepts, autoregressively predicts a new concept sequence, and decodes the generated concepts back into language or speech.The architecture uses a fixed sentence-embedding space, with SONAR encoders and decoders; the concept encoder and decoder are frozen in the illustrated architecture.Source: §1 · pp. 2–3, Figure 1 · p. 3

Long-context structure

• The explicit hierarchy is intended to improve long-form readability and enable local interactive edits, while concept-level sequences can be at least an order of magnitude shorter than token sequences.The paper presents this shorter sequence length as relevant to handling long context and long-form output, whose token-level transformer cost grows quadratically with sequence length.Source: §1 · p. 4

Modularity and extensibility

• The design is modular: concept encoders and decoders can be optimized independently, and new languages or modalities can be added to an existing system.This modularity is presented as avoiding modality competition or interference within multimodal systems.Source: §1 · p. 4

2 Main Design Principles

• SONAR was selected as a semantic sentence-embedding space because of its performance on semantic similarity and prior use in large-scale bitext mining.The paper uses the existing, freely available SONAR space rather than introducing a new embedding space.Source: §2.1 · p. 5
• SONAR combines an encoder/decoder bottleneck with translation, denoising, and embedding-level MSE objectives, then extends to speech through teacher-student training.The bottleneck replaces cross-attention in the text embedding architecture.Source: §2.1 · p. 5, Figure 2 · p. 5

2.2 Data preparation

• SaT predicts sentence boundaries at the token level and is designed to resist missing punctuation or capitalization, but its quality depends on the split-probability threshold.SpaCy provides a rule-based multilingual alternative that is thoroughly tested for high-resource languages.Source: §2.2 · p. 6
• The pipeline adds maximum-length caps to SpaCy and SaT, splitting long sentences into smaller coherent fragments using punctuation or SaT split probabilities.The resulting methods are named SpaCy Capped and SaT Capped.Source: §2.2 · p. 6
• Auto-BLEU measures segmentation quality by comparing each segment with text decoded after SONAR encoding, with higher scores indicating less signal loss.The evaluation samples 10k documents, approximately 500k sentences, and stratifies results by original sentence length.Source: §2.2 · pp. 6–7
• SaT Capped shows a slight but consistent advantage over SpaCy Capped, while uncapped segmenters underperform especially beyond 250 characters.The comparison uses average Auto-BLEU across sentence-length strata; capping is therefore central to the reported result.Source: §2.2 · p. 7, Figure 3 · p. 7
• The training data uses SaT Capped after segmentation analysis and applies robust automatic segmentation to handle noisy formatting and very long sentences.Long or malformed segments can negatively affect the quality of their SONAR embeddings.Source: §2.2 · pp. 6–7

2.3.1 Base-LCM

• The Base-LCM uses a decoder-only Transformer with PreNet normalization and projection into the model dimension, then regresses the next SONAR embedding.The PreNet normalizes SONAR vectors and maps them to dmodel; the model is trained on next-concept prediction.Source: §2.3.1 · p. 8

2.3.2 Diffusion-based LCM

• Diffusion LCMs model next-concept generation autoregressively as pθ(xn|x<n), using forward noising and learned reverse denoising processes.The reverse process predicts clean sentence embeddings from noisy states under a chosen noise schedule.Source: §2.3.2 · pp. 9–10
• One-Tower and Two-Tower diffusion variants differ in whether one backbone jointly handles context and denoising or separate contextualizer and denoiser models are used.The supplied passages describe the autoregressive diffusion role and the reverse-process formulation, while the architectural split is represented by the two model variants.Source: §2.3.2 · pp. 9–10
• Diffusion training explores cosine, quadratic, and sigmoid noise schedules that control the log-SNR distribution across diffusion timesteps.The sigmoid schedule introduces γ and δ to control the scale and center of the log-SNR distribution.Source: §2.3.2 · p. 10

2.3.5 Quantized LCM

• Quantized LCMs learn residual quantizers for SONAR and use discrete codebook units, with auto-encoding BLEU improving as more codebooks are added.Using all 64 codebooks reaches around 70% of the auto-encoding BLEU obtained with continuous SONAR embeddings.Source: §2.3.5 · pp. 14–15

2.4.2 Pre-training evaluation

• Diffusion and Quant-LCM variants have similar ℓ2 and ℓ2-r scores, whereas Base-LCM has lower ℓ2 but no improvement in ℓ2-r.Base-LCM directly optimizes ℓ2, while its lower round-trip performance is attributed to averaging plausible continuations in SONAR space.Source: §2.4.1 · pp. 18–19
• Diffusion models achieve higher MI than Quant-LCM variants, while Quant-LCM variants significantly outperform Base-LCM on MI.Diffusion outputs also paraphrase context more, which correlates with increased MI in the reported analysis.Source: §2.4.1 · p. 19

2.4.1 Experimental setup

• Diffusion-based methods give clearly better next-sentence prediction results than the other LCM variants overall.The comparison covers the reported pretraining metrics and datasets across the ablation variants.Source: §2.4.1 · pp. 18–20

2.4.2 Pre-training evaluation

• One-Tower and Two-Tower diffusion models show no consistent metric difference across the evaluated datasets.The architectural variants therefore do not yield a stable winner in the supplied comparison.Source: §2.4.1 · p. 19

2.4.3 Finetuning evaluation

• Finetuning R-L and Coherence scores follow the pretraining ordering: diffusion models outperform Quant-LCM, and both outperform Base-LCM by a large margin.The passage states that both metrics correlate with the MI-based model ordering.Source: §2.4.1 · p. 20

2.4.4 Noise schedule ablations

• The wide sigmoid schedule (δ, γ) = (3.5, 0) achieves the highest CA on C4 and Wikipedia-en while matching cosine closely in MI.The same schedule shows higher contrastive accuracy across guidance-scale settings in the cited analysis.Source: §2.4.3 · p. 21

Inference complexity

• Only sentences of ≤10 tokens favor LLMs in theoretical inference FLOPs, while LCMs scale better as context size increases.LCM cost includes SONAR encoding, sentence-space prediction, and decoding.Source: §2.5.1 · pp. 23–24, Figure 13 · p. 24

Fragility of SONAR space

• Small embedding perturbations can cause drastic semantic loss after decoding, motivating fragility analysis for SONAR codes.Fragility evaluates semantic similarity between original and perturbed decoded text.Source: §2.5.2 · pp. 24–25
• Auto-Encoding BLEU decreases faster than cosine similarity as noise increases, while the finetuned decoder degrades more slowly than the base decoder.The study uses Auto-Encoding BLEU and external cosine similarity as complementary score functions.Source: §2.5.2 · pp. 25–26
• Sentences longer than 250 tokens are extremely challenging for SONAR and LCM training, although incorrectly splitting long sentences can also create fragile short segments.Fragility is more sensitive to sentence length than Auto-Encoding BLEU, which drops only 1–2% for long sentences.Source: §2.5.2 · p. 26
• Fragility varies substantially across SONAR samples, with the 5% most fragile embeddings often representing hyperlinks, references, identifiers, code-switched text, or numerical entries.These samples may reflect limited SONAR exposure or tokenizer failures and could be filtered from training data.Source: §2.5.2 · p. 26

3 Scaling the model to 7B

• The 7B Two-Tower LCM is trained on 2.7T tokens and 142.4B sentence concepts, with context extended to 2048 concepts.The model uses five contextualizer layers, fourteen denoiser layers, and d_model=4096.Source: §3 · pp. 26–27

3.1 Evaluation

• The evaluation covers summarization and long-context summarization using Rouge-L, overlap, repetition, attribution, main-idea coverage, fluency, and coherence metrics.The benchmarks include CNN DailyMail, XSum, and LCFO at 5%, 10%, and 20% summary lengths.Source: §3.1.1 · p. 27, §3.1.2 · p. 28

3.1.1 Summarization

• The LCM produces competitive Rouge-L scores against T5-3B and instruct-finetuned LLMs, while generating more abstractive, less repetitive summaries than the LLM baselines.Its lower OVL-3 indicates less direct source copying, and its repetition rate is closer to ground truth.Source: §3.1.2 · pp. 28–29, Table 10 · p. 29

3.1.2 Long-context summarization

• The LCM outperforms Mistral-7B-v0.3-IT and Gemma-7B-IT on LCFO Rouge-L at 5% and 10% conditions and remains close to Gemma at 20%.It also obtains high SH-5 scores across conditions, indicating summaries can be attributed to the source.Source: §3.1.2 · p. 29, Table 11 · p. 30
• Llama-3.1-8B-IT substantially outperforms the other LLMs on Rouge-L, although the LLMs have similar performance on CNN DailyMail and XSum.The paper suggests contamination or difficulty handling long contexts as possible explanations.Source: §3.1.2 · p. 29

4.1 Summary expansion

• Summary expansion asks models to generate a much longer document from a concise summary while preserving essential elements and their logical structure.The task is more freely generative than summarization and therefore includes an additional requirement beyond shortening or copying content.Source: §4.1 · p. 30, §4 · p. 30
• In summary expansion, LLMs obtain higher Rouge-L than the LCM, while the LCM produces paraphrastic outputs and lower fluency, especially on CNN DailyMail.The comparison reflects reconstruction overlap rather than the task’s broader requirement to generate a longer text preserving essential content and structure.Source: Results · pp. 30–31, Table 12 · p. 31

4.2 Zero-shot multilingual generalization

• The LCM substantially outperforms Llama-3.1-8B-IT on English XLSum Rouge-L, 23.5 versus 20.7, and on the shared-language average, 20.2 versus 19.7.It also achieves Rouge-L above 20 on several low-resource languages and 30.4 on Vietnamese.Source: §4.2 · pp. 31–32, Figure 16 · p. 32

4.3 Exploring explicit planning

• The proposed planning system conditions LCM generation on a high-level plan produced after a break concept, allowing plans to span multiple subsequent concepts.The initial experiment simplifies this into a single multitask LPCM that predicts break and plan concepts itself.Source: §4.3 · p. 33, Figure 17 · p. 33
• The LPCM achieves significantly higher coherence than a parameter-matched One-Tower LCM baseline, with significance reported at the 99.9% level.The result is attributed in the paper to additional structure from predicted plan concepts.Source: Methodology · p. 33, Results · p. 34

Multilingual sentence representations

• Multilingual sentence-representation research aligns texts across languages in shared embedding spaces using dual encoders, contrastive objectives, translation data, and task-specific adaptations.SONAR extends this encoder-decoder paradigm with a fixed-size sentence representation and multilingual support.Source: §5.1 · pp. 34–35

Joint speech/text sentence representations

• Joint speech-text representation work extends multilingual sentence embeddings to speech and multimodal settings through self-supervised learning and distillation.These efforts provide context for SONAR’s text and speech representation capabilities.Source: §5.1 · p. 35

LLM based sentence representations

• LLM-based sentence embeddings derive representations from instruction-tuned models, modified attention or pooling mechanisms, and commercial embedding services.The paper distinguishes these approaches from its use of a dedicated reconstructable sentence space.Source: §5.1 · p. 35

Sentence embeddings for language modeling

• Prior sentence-level language models either select from finite candidates, combine sentence connections with token-level modeling, or require decoding and re-encoding between steps.These constraints differ from LCM’s replacement of token-level connections with sentence representations.Source: §5.3 · p. 36

Novelty relative to prior work

• The paper claims LCM is the first fully generative language model operating entirely in a highly semantic, reconstructable sentence representation space.Related diffusion and planning approaches still rely on token-level inputs or outputs or do not target arbitrary-length text generation.Source: §5.3 · pp. 36–37

Choice of the embedding space

• SONAR’s local geometry and training on short bitext create a mismatch for global next-sentence prediction and fragile links, references, numbers, and code.The paper warns that factuality of fragile generated sentences may be compromised.Source: §6 · p. 37
• A frozen encoder may be suboptimal for LCM modeling, whereas end-to-end multilingual and multimodal training is challenging, less efficient, and risks modality competition.The encoder must balance modeling suitability with shared semantic representations across languages and modalities.Source: §6 · p. 37

Concept granularity

• Sentence-level concepts make next-sentence prediction difficult because the possible output manifold is much wider than a token vocabulary.Small embedding errors can also produce vectors that do not decode into syntactically and semantically correct sentences.Source: §6 · p. 38
• Long sentences may contain multiple concepts, yet finer splitting does not necessarily separate them cleanly, limiting fixed-size one-sentence embeddings.Text splitting or one-to-many mappings are identified as future directions.Source: §6 · p. 38

Data sparsity and abstraction

• Unique sentences create data sparsity, while semantic abstraction trades lossless encoding of names and numbers against generalization.SONAR provides good auto-encoding quality but may sacrifice generalization capacity relative to the desired abstraction.Source: §6 · p. 38

Continuous versus discrete

• Diffusion struggles with text because continuous SONAR vectors still represent discrete combinatorial sentences, while continuous objectives lack cross-entropy’s contrastive behavior.Quantization could address discreteness, but current SONAR quantization uses many codebooks and units because SONAR was not designed for efficient quantization.Source: §6 · p. 38

Conclusion

• LCMs replace token-level modeling with autoregressive prediction in a high-dimensional, language- and modality-independent concept space.For feasibility, the paper defines a concept as a sentence or equivalent speech segment and uses SONAR embeddings.Source: §8 · p. 39

Results

• LCMs trained only on English outperform Llama-3.1-8B-IT on English and the average over foreign languages officially supported by that LLM, without additional multilingual data.The paper presents languages and modalities as equally handled at every LCM stage.Source: §8 · p. 39

Challenges

• Next-sentence prediction is harder than next-token prediction because sentence choices are virtually unlimited, more ambiguous, and lack a fixed softmax vocabulary.Small embedding errors can yield predictions that do not correspond to valid sentences.Source: §8 · pp. 39–40

Future work

• The authors plan alternative concept embeddings and acknowledge that reaching flagship-LLM performance requires architectural improvements, data curation, instruction tuning, ablations, and models above 70B parameters.The paper describes its current results as a step toward scientific diversity rather than a finished replacement for flagship LLMs.Source: §8 · p. 40

Reproducibility

• The project releases training, evaluation, and data-processing code to support reproducible research on alternative LLM architectures.The released materials include the full training code for LCM variants and supporting scripts.Source: §8 · p. 40

Precomputation

• Precomputed SONAR embeddings accelerate repeated training on the same data mix by trading GPU computation for storage capacity.The fixed encoder and segmentation method make on-the-fly encoding unnecessary for each run.Source: A Technical consideration for data… · p. 48

Storage trade-off

• Storing 1024-float embeddings requires roughly 15–20 TB per terabyte of raw text, increasing storage demands while reducing GPU memory and compute load.The paper reports this trade-off as enabling faster iteration.Source: A Technical consideration for data… · p. 48

Data format

• The pipeline stores SONAR embeddings in FP16 Parquet datasets aligned with segmented texts and supports additional quality metadata and filtering.Parquet supports efficient loading and last-mile data transformation for these structured records.Source: A Technical consideration for data… · p. 48

Corpus scale

• The training corpus contains about four billion documents and 310 billion sentences, averaging 27 tokens and 88 characters per sentence.The processed corpus totals slightly more than 889 TB of raw text.Source: A Technical consideration for data… · p. 48

Released resources

• The released code includes training, evaluation, and data-processing components for reproducing the LCM experiments.The repository is provided under an open-source license.Source: B Open Sourced Code · p. 48

Training framework

• Fairseq2 provides composable modules for building and iterating over the paper’s different model architectures while sharing training components.Its modular design supports side-by-side architecture development rather than a monolithic toolchain.Source: B Open Sourced Code · p. 48

Evaluation framework

• The evaluation framework separates predictors, tasks, and data loading, enabling local or distributed evaluation at scale.It can run on a SLURM cluster and supports reproducible comparisons with other models.Source: B Open Sourced Code · p. 48

Data processing

• The updated stopes framework supports sentence segmentation and SONAR encoding across SLURM, local, and Ray-compatible data-processing environments.It schedules and monitors large job collections and processes datasets in blocks.Source: B Open Sourced Code · pp. 48–49

• The generator must produce a topic description of at most one sentence using as few words as possible.The description must remain specific rather than generic and must not contain profanity.Source: C System prompt: Generation of Topic… · p. 49
• The example extract describes destruction in a neighborhood and Superman identifying an old enemy, Voltar, who challenges him.Source: C System prompt: Generation of Topic… · p. 49
• The example’s preferred description states that Superman’s old enemy Voltar appeared and challenged him.Single-word outputs such as “Superman” or “Voltar” are labeled bad descriptions.Source: C System prompt: Generation of Topic… · p. 49

• The evaluator must score the extract on a 0–5 scale according to its logical coherence and clarity.The scale ranges from completely incoherent at 0 to highly coherent and easy to follow at 4; the supplied scale text begins defining 5 but is truncated.Source: D User prompt: LLM As a Judge -… · p. 49
• Intermediate scores represent increasing connections between ideas, from disjointed text at 1 to moderately coherent text at 3.Source: D User prompt: LLM As a Judge -… · p. 49
• The response must contain only the coherence score without an explanation.Source: D User prompt: LLM As a Judge -… · p. 49