Language Models Hit a Formal Limit: What Text Alone Can Never Teach Them
A formal information-theoretic proof demonstrates that no amount of textual training data can enable language models to fully recover a speaker's intended meaning without extralinguistic context.
Researchers have established a hard upper bound on the probability that any text-based featurizer, including the hidden states of contemporary large language models, can accurately decode a speaker's intended meaning from an utterance alone. The study, submitted to arXiv on 28 Aug 2026, models language use as a joint distribution over meanings, contexts, and utterances to derive these limits. The analysis splits the uncertainty inherent in linguistic form into two distinct components: an irreducible part and a resolvable part that depends exclusively on extralinguistic context. Because these quantities are intrinsic to the structure of language itself, no representation generated from text corpora, regardless of scale or supervision intensity, can surpass these theoretical bounds. This limitation holds whether the space of meanings is defined as discrete or continuous.
The paper moves beyond theoretical derivation to provide empirical evidence across three specific domains: artificial languages, Mandarin zero-pronoun resolution, and color reference tasks. In each case, experiments confirm that the gap between utterance form and intended meaning persists even when models are exposed to vast amounts of textual data. The findings specifically target the assumption that scaling data alone will eventually allow models to infer full semantic intent. The authors show that certain ambiguities are not artifacts of insufficient training but are fundamental properties of communication where form leaves uncertainty about meaning that only context can resolve.
This work challenges the prevailing strategy of relying solely on expanded textual corpora to improve semantic understanding in AI systems. It suggests that architectures designed to ingest only text hit a ceiling defined by information theory, not engineering constraints. For practitioners building systems for high-stakes interpretation or dialogue, the results indicate that achieving human-level comprehension requires integrating non-textual context signals directly into the model input or inference process, rather than hoping for emergent capabilities from larger parameters or datasets.