PaperlayerView exact revision on arXiv

Source-linked AI summary

A Formal Limitation on Learning Human Language From Textual Corpora

Emily Cheng, Ryan Cotterell

arXiv:2608.28560v1cs.CL

TL;DR

The paper asks whether intended meaning can be recovered from utterance form alone, and develops information-theoretic bounds for any text representation. Modeling meanings, contexts, and utterances jointly, it shows that recovery is limited by information carried by form and by context-dependent uncertainty, with experiments supporting the bounds across discrete and continuous tasks. The main scope boundaries are shared-context, single-turn modeling and evaluation of statistical decodability rather than causal use of meaning.

  • Problem

    The paper asks what any system can infer about language and speakers’ intended meanings from utterance form alone, when raw text may omit extralinguistic information.

  • Method

    The authors model communication probabilistically and derive upper bounds on decoding meaning from any utterance featurizer, including LLM representations, for exact and ϵ-accurate recovery.

  • Results

    Experiments confirm the predicted bounds for artificial languages, Mandarin zero-pronoun resolution, and color naming across discrete and continuous meaning spaces.

  • Takeaways & Limitations

    Utterance forms alone bottleneck meaning representation, so additional text or extralinguistic supervision cannot surpass limits intrinsic to the linguistic channel.

  • Takeaways & Limitations

    The model assumes shared context and single-turn communication, and evaluates meaning representation as statistical decodability rather than causal use.

Abstract

from arXiv · show

Can a listener recover what a speaker means from the form of an utterance alone? We answer this question information-theoretically, and for a listener given by any featurizer of text, including the hidden states of contemporary large language models. Modeling language use as a joint distribution over meanings, contexts, and utterances, we derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance. The bounds are governed by the uncertainty that form leaves about meaning, which splits into an irreducible part and a part that only (extralinguistic) context, but never the utterance alone, can resolve. Because these quantities are intrinsic to language, no representation, however much text or supervision produced it, can surpass them; the bounds hold whether the space of meanings is discrete or continuous. Experiments on artificial languages, Mandarin zero-pronoun resolution, and color reference provide empirical evidence in support of the theory.

1 INTRODUCTION

The paper revisits whether natural language can be learned from text alone, focusing on the information available for recovering speakers’ intended meanings from utterance forms. It frames this as a probabilistic question because context and extralinguistic supervision may supply information that raw text lacks.

  • Classical language-learning theory showed that unbounded positive strings do not generally guarantee convergence on the target grammar.Gold’s result concerns exact grammar identification, while Angluin characterized identifiable language classes that natural-language grammars do not satisfy.
  • Probabilistic modeling makes learning from positive data conceivable by replacing exact grammaticality with estimation of statistical structure.This perspective connects to the distributional hypothesis, under which linguistic form and potentially meaning may be recovered from contextual regularities in text.
  • The paper asks what any system can infer about language—and especially intended meaning—from utterance form alone, given the distribution of language use.The question remains relevant because modern systems increasingly receive extralinguistic supervision, such as human preference judgments through RLHF.
  • Meaning may be analyzed through reference, sense, convention, or pragmatic context, so the paper treats meaning as dependent on the chosen conceptualization.The distinction between conventional and pragmatic meaning highlights why extralinguistic context can matter for interpreting an utterance.

2 A MODEL OF COMMUNICATION

The communication model represents language use as a probabilistic, single-turn interaction among context, intended meaning, utterance, and inferred meaning. It also distinguishes the utterance representation available to a decoder from the original utterance.

  • The model defines M as intended meaning, C as shared context, U as produced linguistic form, and inferred meaning as the listener’s reconstruction of M.Communication succeeds to the extent that inferred meaning matches intended meaning.
  • A representation Z is produced from the utterance by a featurizer g: U → Z and decoded into meanings by f: Z → M.LLM activations are an example of such utterance representations.
  • Language use is modeled as a distribution over utterances, contexts, and meanings generated across dyadic speaker–listener interactions.The formulation abstracts away from individual exchanges to study information-theoretic properties of the entire communication system.
  • Context influences intended meaning and utterance production, while the listener infers meaning from the utterance and shared context.The model allows context to include extralinguistic factors such as speaker identities and linguistic information such as prior conversation history.
  • The analysis assumes speaker and listener share context and focuses on single-turn one-way communication, while allowing the main results to extend to noisy channels.These assumptions exclude pragmatic repair across differing contexts from the analyzed setting.

3 BOUNDING MEANING INFERENCE FROM TEXT-BASED REPRESENTATIONS

The paper bounds meaning recovery from any representation of an utterance by separating information carried by form from information supplied uniquely by context. The bounds cover exact classification and approximate recovery over discrete or continuous meaning spaces.

  • Text-based meaning inference is formalized as finding g: U → Z such that a decoder f: Z → M recovers meaning within distance ϵ.The framework includes exact recovery for ϵ = 0 and approximate recovery for ϵ > 0, and applies to any text featurizer, including a perfect learner trained on infinite data.
  • I(M; U; C) = I(M; U) + I(M; C | U) separates information jointly supplied by utterance and context from context-only information unavailable in the utterance.The conditional mutual information I(M; C | U) adjusts the ceiling for recovering meaning from text alone.
  • 3.1 EXACT RECOVERY FOR FINITE MEANING SPACES: Theorem 1 upper-bounds exact classification accuracy for discrete meanings using utterance informativity, context-only information, entropy, and the binary entropy of error.The theorem applies to any representation map g: U → Z when meaning entropy is finite.
  • 3.2 APPROXIMATE RECOVERY FOR INFINITE MEANING SPACES: For continuous or uncountable meanings, Theorem 2 converts ϵ-accurate recovery into a packing-point classification problem and applies Fano’s inequality to the quantized variable Jϵ.A maximal 2ϵ-packing and nearest-point quantizer make accurate recovery imply correct auxiliary classification.
  • 3.2 APPROXIMATE RECOVERY FOR INFINITE MEANING SPACES: Theorem 2 becomes vacuous when the packing number N(ϵ) equals 1, because every decoder then achieves ϵ-accurate recovery with probability 1.This is a degenerate metric-space case rather than an informative recovery bound.
  • 3.2 APPROXIMATE RECOVERY FOR INFINITE MEANING SPACES: The resulting success ceiling depends on purely linguistic informativity I(M; U), which equals overall meaning decodability minus context-only information.The approach assumes finite meaning entropy for classification and finite quantized entropy H(Jϵ) for approximate recovery.

4 EXPERIMENTS

Experiments test the theoretical bounds on categorical and continuous meaning recovery using artificial languages, Mandarin zero-pronoun resolution, and color naming. Across settings, empirical performance remains below the predicted ceilings.

  • Artificial languages: Artificial-language experiments test categorical and continuous meaning recovery while varying the mutual information between meanings and utterances.Six categorical languages vary I(M; U), while eight continuous languages vary I(Jϵ; U) under the valid-language constraint.
  • Natural language: Natural-language experiments use LLM or VLM activations as utterance featurizers and MLP decoders for Mandarin pronoun resolution and color reference.The models include three LLMs and three VLMs ranging from 2B to 14B parameters.
  • Artificial languages: Theoretical bounds exceed empirical test performance for every tested artificial language in both discrete and continuous settings.For continuous meanings, this holds for ϵ = 0.2 and other tested values of ϵ.

5 RELATED WORK

Related work situates the paper within debates over whether ungrounded language models can acquire meaning. The paper’s theoretical limits align with some empirical shortcomings while remaining compatible with positive results in pragmatic inference.

  • LLMs and language theories: The work responds to debates about whether LLM behavior or internal activations can instantiate models or theories of language.The discussion distinguishes meaning inference requiring context from linguistic representation derivable from utterance alone.
  • Formal limits: Merrill et al. (2021) show that some languages prevent semantic equivalence recovery from ungrounded linguistic input, even with unlimited data and semantic assistance.The present work addresses a similar high-level question through information-theoretic bounds on intended meaning recovery.
  • Empirical evidence: The paper’s result that text-based systems learn impoverished meaning representations aligns with reported weaknesses in implicature and presupposition, despite positive pragmatic-inference findings.The authors present these results as consistent with, rather than eliminating, successful pragmatic inference in some settings.

6 CONCLUSION

The paper concludes that meaning inference from utterance-only representations has theoretical limits confirmed in experiments, including for contemporary language and vision-language model activations.

  • Conclusion: Theorems 1 and 2 establish limits on meaning inference from any representation of an utterance alone.The bounds apply regardless of the representation used.
  • Conclusion: Experiments corroborate the theory and support the conclusion that linguistic channels impose a bottleneck on meaning representation.The authors frame this as no free lunch in meaning inference.
  • Conclusion: The bottleneck remains whether models train on trillions of text tokens or receive extralinguistic supervision.In natural communication, humans also use context as a conduit for meaning inference.

LIMITATIONS

The experiments construct artificial languages that separately control utterance informativeness and verify theoretical bounds for categorical and continuous meanings. The continuous-language design varies mutual information across its full range while preserving valid communication through context.

  • Artificial languages: categorical: The categorical construction makes each utterance–context pair determine one meaning while tuning utterance-only information through p.Informative utterances identify meanings independently of context; context-dependent utterances require context as a key.
  • Artificial languages: categorical: The categorical experiments compare decoder accuracy with numerically computed theoretical upper bounds across target information levels.The empirical best-case accuracy is evaluated over hyperparameters and five random seeds, then checked against the bound.
  • Artificial languages: continuous: The continuous construction quantizes meanings into J_ϵ bins and tunes I(J_ϵ; U) from 0 to H(J_ϵ) while maintaining H(M | U, C) = 0.The informative contribution is p × H(J_ϵ), while the shared contribution is zero.
  • Artificial languages: continuous: For ϵ ∈ {0.2, 0.4, 0.6}, the construction produces 25, 9, and 5 quantization bins with entropies of 4.64, 3.17, and 2.32 bits, respectively.Uniform bin frequencies make H(J_ϵ) equal log2 |J_ϵ| in this setup.

B.3 MANDARIN PRO-DROP DATASET (WANG ET AL., 2018)

The Mandarin dataset supports zero-pronoun meaning inference experiments using true pronoun labels and frozen LLM representations. A supervised MLP decoder predicts among 12 pronouns, with theoretical bounds estimated from the empirical distribution.

  • Dataset construction: Filtering 2,150,945 Mandarin TV subtitles to utterances containing exactly one explicit or dropped pronoun yields 904,996 utterances.The resulting pairs combine each utterance with its true pronoun label from Wang et al. (2018).
  • Dataset construction: The dataset contains 12 personal-pronoun classes and is skewed toward first-person singular 我, which represents 37.6% of true pronouns.Possessive forms are normalized to base pronouns, and 咱们 is included among the observed forms.
  • Meaning inference pipeline: The baseline selects an expressed pronoun directly and otherwise predicts the most frequent class 我, while the learned decoder maps frozen LLM embeddings to 12-class probabilities.The MLP is trained with cross-entropy, and the most likely class is used at test time.
  • Bound validation: Theoretical upper bounds are estimated from H(M) and I(M; U), then compared with empirical test accuracies for the Mandarin task.The largest p_e satisfying the bound is found numerically and checked to exceed empirical accuracy.
  • Meaning inference pipeline: The natural-language experiments use 80/20 train-test splits and lightweight supervised MLP decoders trained on utterance–meaning pairs.For each frozen model, last-token hidden states are extracted at every layer, normalized, and probed on held-out data.

D ADDITIONAL ARTIFICIAL LANGUAGE RESULTS (CONTINUOUS)

Additional continuous-meaning experiments test Theorem 2 across several tolerance values and show that its bound continues to hold. The experiments vary ϵ while using quantized continuous meanings and empirical ϵ-accuracy.

  • Additional continuous results: For ϵ ∈ {0.2, 0.4, 0.6}, Figure 4 tests the Theorem 2 bound in artificial languages with continuous meanings.The black line denotes the theoretical bound across these settings.

E.2 TIGHTNESS OF THEOREM 2

Theorem 2 becomes tighter as the tolerance ϵ increases, while finer quantization produces looser bounds. In color naming, the bound holds across all tested tolerances and reflects ambiguity in how utterances partition color meanings.

  • Tightness mechanism: Theorem 2 holds for every tested ϵ, but the bound is looser below ϵ = 0.1 because finer quantization increases I(J_ϵ; U).Smaller tolerances create more bins, so one utterance form corresponds to more quantized meaning bins.
  • Color naming: The color-naming pattern reflects that language coarsely parcels the meaning space, making utterance–bin ambiguity part of the bound.An utterance such as “purple” maps deterministically to one bin while the quantized meaning space remains structured by tolerance.
  • Color naming: In color naming, bounds hold across all ϵ values and typically tighten with larger ϵ.Figure 6 reports the corresponding number of quantization bins for each tolerance.

F PROOF OF THEOREM 1

Theorem 1 bounds classification accuracy for recovering discrete meaning from an utterance representation using Fano’s inequality. The proof relies on a Markov chain and finite meaning entropy, while an entropy-based form improves on classical Fano bounds.

  • Proof setup: The decoder operates on Z=g(U), and its prediction forms the Markov chain M → Z → f(Z).Here, g maps utterances to representations and f predicts the original speaker meaning.
  • Fano reduction: Fano’s inequality bounds prediction success through the representation’s information about meaning and the probability of error.The proof denotes prediction success by pe and error by pi=1−pe before applying the inequality.
  • Bound: Substituting pi=1−pe and using H(M)=I(M;U)+H(M|U) yields the theorem’s final classification bound.Finite entropy H(M)<∞ ensures that the resulting bound is non-vacuous.
  • Bound choice: The entropy-based denominator requires only finite entropy, produces a numerically tighter bound, and adds little computational overhead relative to classical Fano’s inequality.Classical Fano instead uses log(|M|−1), while still requiring estimation of H(M|U).

G PROOF OF THEOREM 2

Theorem 2 extends the classification argument to continuous meanings by discretizing the meaning space with a maximal packing. Accurate regression then implies an auxiliary classification success event, allowing Fano’s inequality to yield the regression bound.

  • Theorem statement: Theorem 2 considers ε-accurate recovery under a metric d and assumes finite entropy H(Jε) for the discretized meaning variable.The variable Jε is used to represent meanings at the target accuracy level.
  • Discretization: A maximal 2ε-packing maps each continuous meaning to its nearest packing point, producing the discrete variable Jε.Packing points are separated by at least 2ε under d.
  • Reduction: The proof constructs an auxiliary classifier that predicts the packing-point label Jε from the decoder’s continuous estimate.This converts ε-accurate regression into a discrete classification problem.
  • Error relation: Because ε-accurate recovery implies correct packing-point classification, the regression success probability is bounded by the classification success probability.The implication follows from the 2ε separation of packing points.
  • Final bound: Since Z is a deterministic function of U, H(J|Z)≥H(J|U), and the resulting entropy relations yield the final regression bound.The proof also uses H(Jε|U)=H(Jε|U,C)+I(Jε;C|U).
Loading 2608.28560v1…