Source-linked AI summary
From task structures to world models: What do LLMs know?
Ilker Yildirim, L. A. Paul
TL;DR
The paper asks whether LLMs have knowledge and how their task-performing abilities relate to human worldly knowledge. It uses structured world models to examine possible recovery from language, while emphasizing that such recovery remains uncertain and may be limited.
Problem
The paper asks whether instrumental knowledge in LLMs incorporates worldly knowledge, and how a text-prediction objective could yield it.
Source: Introduction · p. 2, The leap: From next word prediction to… · p. 4Method
The paper uses structured world models as a framework for examining how LLMs might recover worldly content.
Source: Measured recovery of world models in… · p. 5Results
Domain-specific studies suggest that next-token prediction can recover aspects of underlying data-generating processes and domain semantics.
Source: Measured recovery of world models in… · pp. 6–7Takeaways & Limitations
In realistic settings, LLMs may primarily acquire instrumental knowledge reflecting only limited or partial worldly knowledge.
Source: Measured recovery of world models in… · p. 8Whether LLMs recover causal world abstractions remains uncertain, and studies rely on densely sampled small domains with enumerable building blocks.
Source: The leap: From next word prediction to… · p. 5, Measured recovery of world models in… · p. 7
Abstract
from arXiv · showhide
In what sense does a large language model have knowledge? The answer to this question extends beyond the capabilities of a particular AI system, and challenges our assumptions about the nature of knowledge and intelligence. We answer by granting LLMs "instrumental knowledge"; knowledge defined by a certain set of abilities. We then ask how such knowledge is related to the more ordinary, "worldly" knowledge exhibited by human agents, and explore this in terms of the degree to which instrumental knowledge can be said to incorporate the structured world models of cognitive science. We discuss ways LLMs could recover degrees of worldly knowledge, and suggest such recovery will be governed by an implicit, resource-rational tradeoff between world models and task demands.
Introduction
The paper asks what it means for LLMs to have knowledge and connects that question to human knowledge grounded in structured models of the world. It distinguishes this worldly knowledge from instrumental knowledge, which it attributes to LLMs’ ability to produce successful responses across prompts.
- The paper frames LLM knowledge as a question with implications for cognitive science, neuroscience, philosophy, and AI.Source: Introduction · p. 1
- World models are causal abstractions that preserve the structure of real-world entities and processes, including objects, scenes, and agents.Source: Introduction · p. 1
- “Worldly” knowledge uses a model structurally matched to the world to generate approximately truth-preserving, relevant answers.The authors contrast this with instrumental knowledge, which they use for sufficiently successful responses across a wide range of prompts.Source: Introduction · p. 2
Instrumental knowledge
Instrumental knowledge is framed as an ability to perform tasks across domains, potentially supported by inferring task structure from context. The paper argues that this can exceed formal linguistic competence, while leaving open how much it depends on worldly knowledge.
- Task cues such as “TL;DR” or “In summary” can signal summarization, while paired sentences across languages can signal translation.Source: Instrumental knowledge · pp. 2–3
- Machine translation could use learned relations among words and phrases without projecting shared physical-world knowledge across languages.The paper links this possibility to relational theories of word meaning, where meaning is defined through relations among words or concepts.Source: Instrumental knowledge · p. 3
- Compelling answers from fine-tuned LLMs are difficult to explain solely by fine-tuning, since much of the pre-trained model remains frozen.Source: Instrumental knowledge · p. 3
- Pre-trained LLMs may infer deeper task structure and condition next-word prediction on it, extending beyond language rules and patterns.Source: Instrumental knowledge · p. 4
The leap: From next word prediction to world models
The paper asks how next-word prediction could yield worldly knowledge and considers whether compression might recover world structure from text. It treats that possibility as speculative, not established.
- The central question is how a text-prediction objective could yield worldly content and, in turn, some degree of worldly knowledge.Source: The leap: From next word prediction to… · p. 4
- Structured world models offer a framework for considering recovery: they are lower-dimensional state spaces that capture causal structure in physical processes.Source: The leap: From next word prediction to… · p. 5
- Compression might recover data-generating processes reflected in text, including worldly content about entities, physical processes, and situations.Source: The leap: From next word prediction to… · p. 5
- Whether LLMs actually recover causal world abstractions remains a leap of faith at present.Source: The leap: From next word prediction to… · p. 5
Measured recovery of world models in LLMs under domain-specific settings
Domain-specific sequence models show that predictive training can expose underlying world structure, while evidence from natural-language-trained models is more limited and study designs face substantial caveats.
Othello
- Othello-GPT generated legal moves and linearly decodable full board states from prior moves, and decoder interventions appropriately changed its move predictions.The training traces came from randomly simulated, non-strategic play.Source: Figure 2 · p. 6, Measured recovery of world models in… · p. 6
Other domain-specific models
- Jin & Rinard (2023) 40 suggest next-token training recovers deeper program-language semantics, while protein-sequence models made coarse 3D contact structure linearly decodable.The program-synthesis result was in a toy domain.Source: Measured recovery of world models in… · pp. 6–7
Future research
- The authors call for systematic tests across training objectives, network architectures, and world-model complexity.Source: Measured recovery of world models in… · p. 7
Limits of recovery
- The domain-specific studies require dense sampling, tokenizable world-model building blocks, and may recover models that are computationally costly.Whether Internet-scale language datasets meet the sampling requirements is unknown.Source: Measured recovery of world models in… · p. 7
Natural-language-trained models
- Abdou et al. 46 reported significant correspondence between color-pair embedding similarities and perceptual distances; Patel & Pavlick (2022) 33 found above-chance generalization from examples of one hue.Source: Measured recovery of world models in… · p. 7
Limits of recovery
- Abdou et al.'s color-space correspondence was roughly 0.2, and the authors suggest realistic language use may yield only partial worldly knowledge.They note that spatial structures and intuitive physics are more complex than color space.Source: Measured recovery of world models in… · pp. 7–8
A resource rational view of instrumental knowledge
Worldly knowledge can be costly, and its useful granularity depends on task demands. The authors propose that LLMs' instrumental knowledge reflects an implicit tradeoff between those demands and world-model detail.
Task demands and representation granularity
- A coarse qualitative simulation may predict whether liquid flows left or right, whereas trajectory details can require finer-grained simulation.The authors treat representation granularity as a measure of the extent of worldly knowledge.Source: A resource rational view of… · p. 8
Resource-rational tradeoff
- The degree of worldly knowledge in LLMs may reflect an implicit tradeoff between costly world models and tasks that occur frequently during training.The authors say coarse simulation may suffice for accurate prediction, while finer-grained demands may make it less accurate.Source: A resource rational view of… · p. 8
Task-driven world models
- Task-driven world models allocate fewer resources to knowledge that does not benefit the agent's task while retaining a causal abstraction.In the maze example, simplified representations retain a subset of walls while maintaining expected plan accuracy and path length.Source: Figure 3 · p. 9
Implications for AI and cognitive science
- The authors propose that future AI systems integrate task demands during world-model formation; human task-driven representations can be structured yet sparse.Ho et al. 58 offer a computational account of simplified representations relative to planning objectives in maze navigation.Source: A resource rational view of… · p. 9
World models as programmable, mid- or high-level interfaces for safety and alignment
The authors connect world models to AI safety and alignment, arguing that interpretable, structure-preserving representations could let engineers or users impose explicit values and safety constraints.
Safety motivation
- LLMs' black-box operation and fabrication of situations, events, and persons raise concerns about truthfulness and alignment with human values.Source: World models as programmable, mid- or… · p. 10
World models as interfaces
- World models and domain-specific high-level programming languages offer interpretable representations that can support explicit value and safety constraints.Source: World models as programmable, mid- or… · p. 10
Implementation directions
- The authors describe hybrid LLM–world-model pipelines, neuro-symbolic architectures, and natively programmable neural networks as ways to exploit these features.Source: World models as programmable, mid- or… · p. 10
Conclusion
The article frames LLM abilities as instrumental knowledge and examines whether this knowledge incorporates world models associated with ordinary, worldly knowledge. It proposes that recovery of worldly knowledge may trade off the cost of structured models against common task demands, while explicit world models could support safer, more aligned AI.
Conclusion
- LLM performance challenges assumptions about knowledge and intelligence, and the article grants LLMs instrumental knowledge defined by a set of abilities.It examines how that knowledge may incorporate world models associated with worldly knowledge.Source: Conclusion · p. 10
- Recovery of worldly knowledge may involve an implicit resource-rational tradeoff between costly structured world models and commonly occurring tasks.Source: Conclusion · p. 10
- Future work should construct goal-conditioned world models that combine structured abstractions with an agent’s planning objectives.Source: Conclusion · p. 10
- More explicit world models could support safer, aligned deployment by exposing an interpretable mid- or high-level interface for control and intervention.Source: Conclusion · pp. 10–11
Glossary
The glossary distinguishes ability-based instrumental knowledge from worldly knowledge grounded in world models, and defines world models as structure-preserving representations. It also characterizes pre-trained and fine-tuned LLMs by their training and response-generation roles.
Glossary
- Instrumental knowledge is knowledge of true propositions held in virtue of a system’s abilities, including an LLM’s performance across relevant domains and its ability to infer task structure from prompts.The account proposes that task inference conditions activations for next-word prediction.Source: Glossary · p. 12
- Worldly content concerns real-world entities and processes, while worldly knowledge requires using a world model to grasp that content.Source: Glossary · p. 12
- Ordinary knowledge is a subject’s standing in the knowledge relation to a true proposition.Source: Glossary · p. 12
- A homomorphic mapping preserves the structure of relations among symbols in two systems.Source: Glossary · p. 12
- World models are homomorphic, structure-preserving representations of real-world processes that can support inference, prediction, and reasoning queries.They represent processes in efficient abstract forms, such as simulating optics with computer graphics or object motion with physics engines.Source: Glossary · p. 12
- Pre-trained LLMs autocomplete text using Internet-scale natural-language data, while fine-tuned LLMs receive further training to follow instructions and generate aligned, humanlike responses.Fine-tuning typically uses much less data than pre-training and often uses supervised and reinforcement learning.Source: Glossary · pp. 12–13
Outstanding questions
The outstanding questions identify directions for formalizing instrumental knowledge, testing worldly knowledge, and integrating world models into AI. They also ask how the knowledge distinction bears on fine-tuning, foundation models, linguistic competence, and understanding.
Outstanding questions
- How can instrumental knowledge be formalized, including through task-conditioned world models, and related to amortized inference or resource-rational solutions to intractable problems?Source: Outstanding questions · p. 18
- How does fine-tuning affect instrumental knowledge and the recovery of worldly knowledge?Source: Outstanding questions · p. 18
- How can structured world models be incorporated into AI systems to support safer and better-aligned systems?Source: Outstanding questions · p. 18
- How should domain-specific benchmarks assess worldly knowledge across dimensions such as world-model granularity and training-data distribution?The proposed benchmarks draw on world models and computational theories in cognitive science, including intuitive physics and cognitive maps.Source: Outstanding questions · p. 18
- Can the distinction between instrumental and worldly knowledge inform foundation models in language, computer vision, and reinforcement learning?Source: Outstanding questions · p. 18
- Do LLMs internalize formal linguistic competence, and what is its format in relation to linguistic theories?The article also asks how its account relates to conceptual role semantics and embodiment-based arguments, and whether worldly knowledge offers a way to analyze understanding.Source: Outstanding questions · p. 18
Highlights
The article asks what it means for LLMs to have knowledge and grants them instrumental knowledge, defined by abilities. It explores how that knowledge may incorporate world models and how LLMs might recover degrees of worldly knowledge under a resource-rational tradeoff.
Highlights
- The article asks in what sense LLMs have knowledge, a question it says challenges assumptions about knowledge and intelligence beyond any particular chatbot.Source: Highlights · p. 19
- LLMs are granted instrumental knowledge, defined by a certain set of abilities.Source: Highlights · p. 19
- The article examines how instrumental knowledge relates to human worldly knowledge and whether it incorporates structured world models.Source: Highlights · p. 19
- LLMs could recover degrees of worldly knowledge through an implicit, resource-rational tradeoff between world models and task demands.Source: Highlights · p. 19