From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning

Chen Shani; Liron Soffer; Dan Jurafsky; Yann LeCun; Ravid Shwartz-Ziv

Source-linked AI summary

From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning

Chen Shani, Liron Soffer, Dan Jurafsky, Yann LeCun, Ravid Shwartz-Ziv

arXiv:2505.17117v7May 21, 2025cs.CLcs.AIcs.IT

TL;DR

Whether LLMs balance compression and semantic richness like humans remains unclear. Using an Information Bottleneck framework and categorization benchmarks, the paper finds broad human category alignment but weaker fine-grained semantics, stronger compression, and better human alignment from smaller encoder models than decoder models.

Problem
It remains unclear whether LLMs navigate the compression-meaning trade-off like humans, whose understanding requires balancing statistical efficiency with semantic richness.
Method
The study compares human category ratings with embeddings from 40+ LLMs using mutual information, Spearman correlation, and a rate-distortion objective.
Results
LLMs broadly align with human category boundaries but miss fine-grained semantic distinctions, achieve lower (better) L scores than humans, and show stronger human alignment in encoder than decoder models.
Takeaways & Limitations
The findings suggest that human-like understanding may require preserving semantic richness and apparent representational inefficiencies rather than optimizing statistical compression alone.
Takeaways & Limitations
Polysemy is rare in the data and cannot account for the findings; optimizing solely for statistical efficiency may still risk misinterpretation or oversimplification in socially sensitive applications.

Abstract

from arXiv · show

Humans organize knowledge into compact conceptual categories that balance compression with semantic richness. Large Language Models (LLMs) exhibit impressive linguistic abilities, but whether they navigate this same compression-meaning trade-off remains unclear. We apply an Information Bottleneck framework to compare human conceptual structure with embeddings from 40+ LLMs using classic categorization benchmarks. We find that LLMs broadly align with human category boundaries, yet fall short on fine-grained semantic distinctions. Unlike humans, who maintain ``inefficient'' representations that preserve contextual nuance, LLMs aggressively compress, achieving more optimal information-theoretic compression at the cost of semantic richness. Surprisingly, encoder models outperform much larger decoder models in human alignment, suggesting that understanding and generation rely on distinct representational mechanisms. Training-dynamics analysis reveals a two-phase trajectory: rapid initial concept formation followed by architectural reorganization, during which semantic processing migrates from deep to mid-network layers as the model discovers increasingly efficient, sparser encodings. These divergent strategies, where LLMs optimize for compression and humans for adaptive utility, reveal fundamental differences between artificial and natural intelligence. This highlights the need for models that preserve the conceptual ``inefficiencies'' essential for human-like understanding.

1 THE ENIGMA OF MEANING IN LARGE LANGUAGE MODELS

The paper asks whether LLMs balance compression and semantic richness as humans do, using information-theoretic comparisons of human categories and model embeddings. It finds broad categorical alignment but weaker fine-grained semantic structure, with LLMs favoring efficient compression over human-like nuance.

The study compares human conceptual structures with embeddings from 40+ LLMs using Rate-Distortion Theory and Information Bottleneck principles.The analysis is grounded in classic categorization benchmarks and evaluates compression, semantic structure, and their trade-off.
LLMs achieve mathematically efficient compression, whereas humans maintain apparently inefficient representations that preserve multidimensional semantic nuance for flexible reasoning.The contrast frames human inefficiency as a possible consequence of preserving meaning rather than minimizing representation cost.
LLMs broadly align with human category boundaries but miss fine-grained semantic distinctions central to human understanding.Their representations capture recognizable categories while failing to preserve the richer internal structure reflected in item typicality.
Encoder models outperform much larger decoder models in human alignment, suggesting that understanding and generation may depend on different representational strategies.This architecture-level difference appears despite the scale advantage of decoder models.
Across 57 OLMo-7B checkpoints, conceptual structure develops through rapid initial formation followed by architectural reorganization and migration of semantic processing toward mid-network layers.The later phase produces increasingly efficient encodings as semantic processing shifts from deeper to middle layers.
The findings challenge the assumption that statistical optimality equals understanding and motivate systems that preserve conceptual inefficiencies supporting human-like comprehension.The paper presents these inefficiencies as potentially important for adaptive semantic flexibility.

2 RESEARCH QUESTIONS AND SCOPE

This section connects LLM conceptual analysis with cognitive information theory and defines three questions about category alignment, internal typicality structure, and compression versus semantic fidelity. A unified framework evaluates these questions using human ratings and model embeddings.

The paper bridges two research streams that rarely intersect: LLM conceptual analysis and cognitive information theory grounded in established human benchmarks.Prior work often studied these areas separately or focused on narrower domains.
The first question asks whether LLM-emergent concepts align with human-defined categories.It focuses on how the two systems partition semantic space.
The second question asks whether LLMs reproduce human-like internal structure, especially item typicality.This examines how semantic meaning is preserved within categories.
The third question asks how humans and LLMs balance compression against semantic fidelity.The study treats this as an integrated trade-off rather than an isolated compression measure.
The analysis derives Mutual Information, Spearman correlation, and a rate-distortion objective from human category ratings and analogous model embedding scores.These metrics target category recoverability, human typicality structure, and representation complexity versus meaning preservation.

3 BENCHMARKING AGAINST HUMAN COGNITION

The study benchmarks LLM representations against digitized classic human categorization datasets and embeddings from diverse models. The resulting evaluation spans curated category judgments, multiple architectures and scales, and both static and contextual representations.

The benchmark studies prototype structure and graded category membership rather than treating categories as strictly absolute.Rosch’s datasets emphasize prototypical members, while McCloskey and Glucksberg examine fuzzy boundaries.
The unified benchmark contains 1,049 items across 34 categories aggregated from three classic human categorization studies.The datasets provide category assignments, typicality ratings, and, in one study, graded membership certainty.
The model comparison covers 40+ LLMs ranging from 300M to 72B parameters across encoder and decoder architectures.This design separates potential effects of architecture, scale, and training objective.
Representations are extracted as static input-layer embeddings and contextual hidden-layer embeddings obtained with controlled prompts.This dual approach traces context-free lexical knowledge and context-shaped conceptual structure across network depth.
The findings are robust to prompt templates and pooling strategies, while token count and tokenization patterns do not correlate with the results.The study also reports that polysemy is rare enough not to account for the findings, although 141 lemmas absent from WordNet were omitted.

4 A FRAMEWORK FOR COMPARING COMPRESSION AND MEANING

The framework combines information-theoretic complexity with geometric distortion to evaluate how conceptual clusters compress item identity while preserving semantic coherence. Its metrics separately assess categorical alignment, internal typicality structure, and the overall compression-meaning trade-off.

The objective L(X, C; β) combines mutual-information complexity with within-cluster semantic distortion, weighted by β.Complexity measures information retained about item identity, while distortion measures geometric spread within clusters.
Complexity is low when clusters reveal little about individual item identity, corresponding to stronger compression.Uniform clusters maximize compression, whereas singleton clusters preserve item identity and maximize complexity.
4.2.2 THE DISTORTION TERM: MEASURING SEMANTIC COHERENCE: Distortion is the average squared distance of items from their cluster centroids, measuring semantic spread within categories.The variance term captures within-cluster embedding dispersion around each centroid.
4.2.2 THE DISTORTION TERM: MEASURING SEMANTIC COHERENCE: Low distortion indicates tight, semantically coherent clusters, whereas atypical or semantically distant items increase cluster spread.The bat-versus-bird example illustrates how mismatched membership can raise distortion.
Mutual Information evaluates categorical recoverability, and distortion evaluates whether typical items lie near category centroids in a prototype-like structure.Together, these terms distinguish category boundaries from the internal organization of meaning within categories.
Varying the number of clusters K produces L curves that expose whether systems prioritize aggressive compression or preservation of semantic distinctions.Aggressive compressors reach low L rapidly by sacrificing nuance, while richer representations retain higher L.

5 AN EMPIRICAL INVESTIGATION OF REPRESENTATIONAL STRATEGIES

Across 40+ models, LLMs align broadly with human category boundaries but capture fine-grained semantic structure only weakly. They achieve more efficient compression than human categories, while training reveals rapid concept formation followed by architectural reorganization.

5.1 [RQ1] THE BIG PICTURE: ALIGNMENT OF CONCEPTUAL CATEGORIES: AMI ≈0.55 confirms broad above-chance alignment with human categories across 40+ models, while peak-AMI layers trade off within-category structure.Static embeddings average approximately AMI = 0.45, rising to peak AMI ≈0.55 with contextual processing.
5.1 [RQ1] THE BIG PICTURE: ALIGNMENT OF CONCEPTUAL CATEGORIES: BERT-large-uncased reaches AMI = 0.60, matching or exceeding models 100× larger and showing architecture can matter more than scale.Classic Word2Vec and GloVe embeddings also rival contemporary LLM peak performance.
5.2 [RQ2] ZOOMING IN: FIDELITY TO FINE-GRAINED SEMANTICS: Typicality correlations remain modest: BERT reaches ρ = 0.38, whereas most decoder models fall below ρ < 0.15.Representation-focused and encoder models generally show stronger fine-grained semantic alignment than decoder-only models across scales.
5.3 [RQ3] THE EFFICIENCY ANGLE: THE COMPRESSION-MEANING TRADE-OFF: LLM-derived clusters achieve lower L values than human categories across tested K, indicating superior information-theoretic compression-distortion efficiency.Human concepts have higher entropy and greater internal diversity, while encoders achieve lower distortion than decoders at matched complexity.
5.3 [RQ3] THE EFFICIENCY ANGLE: THE COMPRESSION-MEANING TRADE-OFF: Compression efficiency does not predict downstream performance, with L scores showing no correlation with MMLU performance.The reported relationship is r = −0.20, ρ = 0.51 on MMLU.
5.4 EMERGENCE DURING TRAINING: HOW DIVERGENT STRATEGIES DEVELOP: Conceptual organization develops in two phases: rapid early category formation, followed by slower reorganization that shifts semantic processing toward mid-network layers.AMI rises from near zero to approximately 0.45 during 1K–100K steps, while processing migrates from layer 29 to layer 23 during 100K–500K steps.

6 DISCUSSION AND CONCLUSION

The paper argues that LLMs and humans use divergent representational strategies: LLMs favor statistical compression, whereas humans preserve semantic richness. This challenges equating information-theoretic optimality with understanding and motivates architectures that better preserve human-like structure.

DISCUSSION AND CONCLUSION: LLMs achieve broad categorical alignment but weak typicality correspondence, lower L scores than humans, and encoder models outperform larger decoders in human alignment.Training analysis adds rapid category formation followed by a shift of semantic processing from deep to mid-network layers.
DISCUSSION AND CONCLUSION: The findings challenge the assumption that statistical optimality equals understanding because efficient representations can sacrifice semantic nuance.The paper links decoder training objectives and human-model divergence to different representational strategies.
DISCUSSION AND CONCLUSION: Progress toward human-like AI may require preserving conceptual inefficiencies that support flexible reasoning and generalization.The conclusion contrasts LLM statistical compression with human semantic richness and flexible cognition.

7 ETHICS STATEMENT

The study uses public models and digitized classic cognitive datasets without collecting new human-subject data. It reports no anticipated privacy, security, or fairness risks, while cautioning that efficiency-only optimization could worsen misinterpretation or oversimplification in sensitive applications.

ETHICS STATEMENT: The analyses rely on publicly available LLMs and digitized classic cognitive-psychology datasets, with no new human-subject data collected.Released benchmark data are attributed and curated to preserve research integrity.
ETHICS STATEMENT: The authors foresee no privacy, security, or fairness risks from the analyses but caution against statistical-efficiency optimization without semantic richness.They identify possible misinterpretation or oversimplification risks in socially sensitive applications.

8 REPRODUCIBILITY STATEMENT

The paper frames the compression-meaning trade-off as balancing efficient abstraction against semantic richness for flexible reasoning and generalization. It acknowledges that its metrics are only proxies and suggests cognitively motivated biases and hybrid architectures as directions for closer human alignment.

A.1 A COGNITIVE INTUITION OF THE COMPRESSION-MEANING TRADEOFF: The compression-meaning trade-off balances minimal information against semantic richness needed for flexible generalization, inference, and communication.The paper illustrates this as retaining meaningful abstractions rather than every detail of an experience.
A.1 A COGNITIVE INTUITION OF THE COMPRESSION-MEANING TRADEOFF: The proposed metrics quantify compression and geometric efficiency but do not directly measure all aspects of human conceptual abstraction or reasoning.The authors present them as interpretable proxies rather than definitive evidence of human-style processing.
FUTURE DIRECTIONS: Cognitively motivated inductive biases, hierarchical and compositional structure, and feature- or relation-based mechanisms could improve human-like conceptual organization.These mechanisms target nested category relationships and meaningful perceptual or functional structure.
FUTURE DIRECTIONS: Causal priors, hybrid exemplar-rule learning, and modular architectures could support flexible generalization while preserving richer conceptual structure.The proposed combination aims to approximate human category learning through examples, rules, and specialized representational components.

B LIMITATIONS

The study identifies limitations concerning scope, benchmarking, model coverage, modality, and methodological choices. It also reports robustness analyses for pooling strategies and prompt templates.

Scope limitations: The analysis primarily focuses on English, textual input, and selected LLMs, leaving cross-lingual, multimodal, and broader architectural generalization open.The authors propose expanding to other languages and testing different architectures and applications.
Benchmark limitations: Human categorization benchmarks may not fully capture cognitive complexity and could introduce biases.
Methodological limitations: The IB-RDT objective is applied to specific LLMs, so other models or representations might behave differently.
Pooling robustness: Average pooling provides consistent performance and the tightest distribution across tested pooling strategies.The study compares average, first-token, last-token, and sentence-average pooling.
Prompt robustness: The neutral template "This is a {word}. " is selected to minimize semantic bias and balance performance with consistency for baseline comparisons.Eight templates spanning different linguistic framings were evaluated.

F STATIC VS. CONTEXTUAL AMI EXPLORATION

LLMs encode above-chance human-like category structure in static embeddings, while contextual processing generally strengthens alignment. Alignment varies with architecture, language, and training stage rather than computational cost alone.

Static and contextual AMI: Static embeddings already encode human-like categorical structure, and contextual processing consistently improves alignment through peak AMI.Static models can rival modern LLMs’ peak contextual performance, while peak alignment typically occurs in middle-to-late layers.
Architecture and scale: Encoders and static representation models achieve comparable or superior alignment to much larger decoder models, indicating architecture matters beyond scale.This pattern appears across BERT, ViT encoders, Word2Vec, and GloVe compared with decoder families.
Robustness: Peak AMI remains highly stable across clustering seeds, supporting the robustness of layer-wise alignment patterns.The reported peak values and profiles show negligible variation across random initializations.
Compute scaling: Computational cost alone does not predict human-aligned conceptual representations, because peak AMI shows no systematic relationship with FLOPS per token.The comparison directly evaluates peak AMI against each model’s computational cost.
Multilingual scaling: Non-English languages show a positive model-size–AMI relationship absent in English, which the authors interpret as reflecting limited multilingual training data.The analysis covers Spanish, German, Italian, and Russian alongside English.
Training dynamics: OLMo-7B develops concepts in two phases: rapid early alignment followed by slower reorganization toward more efficient internal representations.Attention sparsity, effective rank, and L exhibit the same early-shift then restructuring pattern.

J TOKENIZER ANALYSIS

Tokenizer analyses find substantial lexical overlap across most model families, reducing tokenization as an explanation for performance differences. Supplementary evaluations confirm broad category alignment but weak typicality correspondence, while matched-family tests preserve encoder advantages.

Tokenizer overlap: Most tokenizer families share over 60% lexical overlap, while first-generation BERT WordPiece tokenizers share under 16% with other groups.The overlap analysis normalizes tokenizer-specific markers before computing pairwise Jaccard similarity.
Cluster validation: ARI and NMI reproduce AMI’s above-chance category-alignment trends across tested LLMs, while silhouette scores generally indicate reasonable cluster cohesion.Relative model performances are also broadly similar across these supplementary metrics.
Matched architectural comparisons: Matched-family comparisons show encoders consistently achieve higher AMI and lower L than decoders, despite reducing training-data differences.The matched families are GPT, Pythia, Cerebras, and T5, although residual confounds remain possible.
Typicality structure: Human typicality correlations are modest, weak, and mostly non-significant across static and peak-AMI analyses, indicating limited semantic-nuance alignment.The analyses span the Rosch (1973), Rosch (1975), and McCloskey (1978) datasets.
Compression–meaning trade-off: The L objective favors LLM-derived representations over fixed human categories, with lower L indicating a more optimal compression–distortion balance.Human categories show higher L values, while encoder profiles remain flatter as β varies.

S L OBJECTIVE VS. DOWNSTREAM TASK PERFORMANCE

Compression quality does not directly predict downstream benchmark performance, while representation properties differ across languages, architectures, and model types. The analyses also show that static and contextual embeddings occupy distinct compression–meaning regimes.

Downstream performance: Across 13 instruction-tuned models, L scores do not significantly correlate with MMLU performance (r = −0.202, p = 0.508).The result suggests concept compression and standard NLP benchmark accuracy are not directly interchangeable.
Language effects: Non-English representations are more compressed than English representations, especially in smaller models, consistent with less flexible representations under limited training data.The comparison covers Spanish, German, Italian, and Russian.
Training dynamics: OLMo training exhibits coordinated two-phase changes in effective rank, Gini coefficient, and Hoyer sparsity, indicating network-wide representational reorganization.These measures show rapid early change followed by slower restructuring across training.
Category alignment: Static LLM clusters align above chance with human categories, and BERT is notably strong on both NMI and ARI evaluations.These results are averaged across three psychological datasets.
Compression regimes: Contextual embeddings achieve better-than-human compression–meaning trade-offs, while static embeddings attain still lower L values than both contextual embeddings and human categories.Lower L denotes a more optimal balance between compression and semantic fidelity.
β sensitivity: Encoder models maintain flatter distortion-to-complexity profiles across β, indicating greater stability under compression than decoder models.Decoder models show stronger shifts in representational capacity as β changes.