Source-linked AI summary
Talking About Large Language Models
Murray Shanahan
TL;DR
LLMs’ convincing human-like language makes it tempting to attribute human capacities such as knowledge, belief, and thought to them. The paper clarifies their sequence-prediction basis and argues for more careful language that distinguishes models from the larger systems embedding them. These systems can be versatile and useful, but their pattern completion is not guaranteed to be logically faithful and current embodied applications remain limited.
Problem
As LLMs increasingly resemble humans in behavior despite fundamentally different operation, anthropomorphic descriptions risk obscuring how they work.
Method
The paper analyzes LLMs and LLM-based systems by tracing their sequence-prediction function, system components, prompting, and relationships to human language use.
Results
LLM applications are versatile, but simple LLM-based systems lack communicative intent, and their pattern completion has no guarantee of logical faithfulness.
Takeaways & Limitations
Scientific precision about mechanisms can support more philosophically nuanced discourse and help users and developers avoid attributing capacities LLMs lack.
Takeaways & Limitations
Current systems remain unlike humans in their construction, shared-world language learning, and breadth of embodied language use.
Abstract
from arXiv · showhide
Thanks to rapid progress in artificial intelligence, we have entered an era when technology and philosophy intersect in interesting ways. Sitting squarely at the centre of this intersection are large language models (LLMs). The more adept LLMs become at mimicking human language, the more vulnerable we become to anthropomorphism, to seeing the systems in which they are embedded as more human-like than they really are. This trend is amplified by the natural tendency to use philosophically loaded terms, such as "knows", "believes", and "thinks", when describing these systems. To mitigate this trend, this paper advocates the practice of repeatedly stepping back to remind ourselves of how LLMs, and the systems of which they form a part, actually work. The hope is that increased scientific precision will encourage more philosophical nuance in the discourse around artificial intelligence, both within the field and in the public sphere.
1 Introduction
LLMs achieve striking capabilities through next-token prediction, yet their human-like behavior makes anthropomorphism increasingly tempting. The paper argues that users and developers should keep the models’ underlying operation in view when describing and deploying them.
- LLM benchmark performance scales with training-set size, while model size has a smaller effect.
- Many tasks that demand human intelligence can be reduced to next-token prediction by a sufficiently capable model.
- Because AI systems have substantial utility and commercial potential, describing their actual operation supports informed decisions about trustworthiness and safety.
- LLMs are generative mathematical models of token distributions in large text corpora, producing statistically likely continuations of prompts.
- The same sequence-prediction mechanism can produce answers about real people, fictional worlds, and familiar rhymes, although their relationships to truth differ.
- Repeatedly recalling what LLMs do can help users and developers avoid misleading terms such as belief, knowledge, understanding, self, and consciousness.
3 LLMs and the Intentional Stance
Anthropomorphic language can be useful shorthand for complex artefacts, but LLMs’ convincing behavior makes that shorthand harder to treat as harmless. The paper therefore urges careful attention to their fundamentally nonhuman operation.
- The intentional stance treats an entity as if it were a rational agent, making anthropomorphic descriptions convenient shorthand for complex processes.
- For LLMs, human-like behavior such as improved reasoning after being told to “think step by step” can make anthropomorphism especially tempting.
- The paper does not deny that some LLM-based systems might warrant terms such as beliefs or intentions in principle.
- The paper’s central caution is that systems can behave human-like while remaining fundamentally different from humans in construction.
- The comparison between human Bob and LLM-based BOT raises whether “know” has the same sense in both cases.
4 Humans and LLMs Compared
An LLM-based question-answering agent is not merely an LLM: it combines the model with system components and prompt engineering to produce conversation-like behavior. This makes the application versatile while leaving sequence prediction at its core.
- Human conversation relies on understanding interlocutors and assessing statements against agreed criteria external to the speakers.
- A bare-bones LLM is not a conversational agent and must be embedded in a larger system to manage dialogue turn-taking.
- Prompt engineering uses prefixes and formatted dialogue examples to elicit appropriate responses from an LLM.
- LLM applications can perform question-answering, summarization, screenplay generation, logic solving, and translation without further training.
5 Do LLMs Really Know Anything?
The paper distinguishes sequence prediction from human knowledge and belief. Although LLM-based systems can generate propositional answers, they lack the communicative and world-involving capacities that make such concepts apply straightforwardly to humans.
- A simple LLM-based question-answering system has no communicative intent and does not know who asks questions or receives its answers.
- BOT does not properly speaking have beliefs, even though the intentional stance may license casual language suggesting that it knows an answer.
- A bare-bones LLM performs sequence prediction, and propositional outputs do not have a special relationship to truth for the model itself.
- Predicting that “Burundi” follows a prompt is not the same as knowing that Burundi is south of Rwanda.
- A simple dialogue agent cannot fully participate in the human language game of truth because it does not inhabit the shared human world.
6 What About Emergence?
Convincing LLM behavior can invite anthropomorphism, but the bare-bones model remains a predictor of likely word sequences rather than a judge of truth or belief.
- State-of-the-art conversational exchanges make anthropomorphism difficult to resist, especially as capabilities emerge at scale.The paper distinguishes this persuasive behavior from human-like cognition.
- The bare-bones LLM models which words are likely to follow other words rather than making judgements about truth.
- LLMs can encode or contain knowledge in an encyclopedia-like sense, without literally knowing or believing as humans do.
- Because the bare-bones model lacks access to external reality or independent truth criteria, belief ascription requires the larger system.
7 External Information Sources
External information sources may appear to give an LLM the evidence needed for belief updating, but this possibility belongs to the complete system, not the language model alone.
- Beliefs about a shared world require the ability to update appropriately using evidence from that world.
- A system could consult trustworthy websites and use model editing to maintain factual accuracy, raising the question of whether it can update beliefs from evidence.
- External-source systems include LaMDA, Sparrow, and Toolformer, while tool-use also covers calculators, calendars, and programming environments.
- The language model remains a sequence predictor without external-world access; any stronger case for intentional language applies only to the whole system.
- Human belief updating reflects participation in a shared world and community, whose foundations are absent from LLM interaction.
8 Vision-Language Models
Vision-language systems connect words with images, but most rely on frozen text-image correlations rather than verifying propositions against external reality.
- Question-answering systems supplement an LLM with dialogue management that queries the model as needed.
- Vision-language models combine language models with image encoders and multimodal text-image training to predict continuations in an image context.
- These models support visual question-answering and dialogue about user-provided images.
- A VLM could generate image hypotheses, verify them against the image, and fine-tune away false statements, potentially supporting talk of belief.
- Most contemporary VLM systems model joint text-image distributions, making their word-image relationship correlational rather than causal.
- Spurious image features can produce dog-related statements, so a VLM’s words are not guaranteed to concern an actual dog.
- Whether knowledge or belief language is appropriate depends on each VLM’s model and overall architecture.
9 What About Embodiment?
Embodiment makes LLM systems more human-like in some respects, but appropriate talk of knowledge, belief, and intent depends on how language is integrated with perception, action, and the world.
- 9 What About Embodiment?: Inference-time computation has causal structure, but that does not establish causal relations between words and what they refer to.
- 9 What About Embodiment?: A system inhabiting a shared 3D world is more human-like than a disembodied model, yet its status depends on exactly how the LLM is embodied.
- 9 What About Embodiment?: SayCan embeds an LLM in a physical robot that maps high-level instructions to low-level actions for everyday tasks.
- 9 What About Embodiment?: SayCan combines the LLM’s estimate of action usefulness with a perceptual module’s estimate of environmental feasibility.
- 9 What About Embodiment?: Although SayCan’s action descriptions may be grounded through perception-action mediation, its language use remains unlike human language use.
- 9 What About Embodiment?: Systems like SayCan use text-only pretraining and a narrow instruction-and-action language repertoire, unlike humans’ broad collective language activity.
- 9 What About Embodiment?: Even for embodied LLM systems, engineers should remain cautious about describing their behavior as genuine understanding.
10 Can Language Models Reason?
LLMs can produce multi-step arguments and correct answers, but prompting them to reason does not establish a truth-preserving reasoning process. Trustworthy reasoning therefore requires caution about whether apparent reasoning reflects logic or learned language patterns.
- Why Language Models Appear to Reason: Reasoning is harder to attribute to LLMs than belief because formal logic is content-neutral.The validity of modus ponens does not depend on what the premises are about.
- Why Language Models Appear to Reason: Chain-of-thought prompting encourages LLMs to generate explicit sequences of reasoning steps before a final answer.The prompt prefix supplies examples whose continuation follows the same style.
- Limits of Apparent Reasoning: These generated arguments are often valid and answers often correct, but mistakes remain and scaling may not resolve whether the process is trustworthy.The paper distinguishes occasional correctness from a demonstrated reasoning mechanism.
- Trustworthy Reasoning: A theorem prover is trustworthy when its computational process mirrors the truth-preserving structure of deduction.The relevant criterion is faithfulness to logic, not merely the appearance of a valid argument.
- Trustworthy Reasoning: Embedding an LLM in an algorithm faithful to logic offers one route to a trustworthy reasoning system.A pure prompted LLM would require reverse engineering to establish an emergent mechanism conforming to that prescription.
11 How Do LLMs Generalise?
LLMs generalise by completing patterns in token sequences, allowing suitably prompted models to solve previously unseen reasoning problems. This capability can resemble reasoning while remaining distinct from guaranteed logical inference.
- Generalisation Through Pattern Completion: Next-token prediction is sufficient for LLMs to solve previously unseen reasoning problems, albeit unreliably.This is not explained by cutting and pasting fragments from the training set.
- Generalisation Through Pattern Completion: LLMs model a complex distribution of token sequences rather than responses from an average individual human.Their resulting intelligence can differ from human intelligence, being more capable in some ways and less capable in others.
- Generalisation Through Pattern Completion: Reasoning prompts elicit likely continuations because published human text contains reasoning problems paired with correct answers.Correct outputs reflect likely collective human responses rather than likely responses from an individual person.
- Few-Shot Prompting: Few-shot prompting is more accurately described as pattern completion than as teaching an LLM how to reason.The model completes a partial token sequence that follows a pattern established by the prompt examples.
- Few-Shot Prompting: Chain-of-thought prompts transform reasoning problems into next-token prediction problems that can be solved by pattern completion.This relies on a meta-pattern in human language: sequences of tokens conforming to a shared pattern.
- Limits of Generalisation: Pattern completion may generalise broadly with sufficient parameters and suitable training data, but it is not guaranteed to preserve truth.The capability is useful for reasoning applications while lacking a guarantee of faithfulness to logic.
12 What about Fine-Tuning?
Fine-tuning changes the distribution of responses toward outputs preferred by users or raters while preserving the basic account of LLMs as token-distribution models. The thought experiment further indicates that the final model matters more than how it was produced.
- Effects of Fine-Tuning: Fine-tuning and RLHF can shape responses to reflect user norms, filter toxic language, improve factual accuracy, and mitigate fabrication.The paper presents human-feedback fine-tuning as a potent technique when applied at scale.
- Effects of Fine-Tuning: RLHF produces a slightly skewed model of the distribution of tokens in human language rather than a fundamentally different kind of model.The shift is illustrated as changing which continuations are most likely for a prompt.
- Effects of Fine-Tuning: For a prompt about Boris Frump, RLHF can favor a politically neutral continuation over complimentary or incendiary raw-model alternatives.The example depends on raters preferring politically neutral responses.
- Interpreting RLHF: Human-preference fine-tuning can be viewed as training on an augmented dataset containing enough rater or user examples to dominate less-favoured responses.The resulting model makes approved responses more likely.
- Interpreting RLHF: An impractically long process of training from human preferences could eventually produce capabilities comparable to a conventionally trained LLM.This thought experiment separates the production process from the nature of the final product.
13 Conclusion: Why This Matters
Because contemporary LLMs are convincingly intelligent yet unlike humans, anthropomorphic language can carry more weight than intended. The paper therefore urges greater care and resistance to anthropomorphism while acknowledging that better terminology may emerge over time.
- Why Language Matters: Philosophically loaded terms such as “belief” and “knowledge” are safe shorthand only when readers cannot assign them more weight than warranted.The concern applies to both scientific writing and public communication.
- Why Language Matters: Conversational LLM systems can create a compelling illusion of a thinking creature despite lacking the shared form of life underlying human understanding and trust.Their behaviour combines human-like and inhuman capacities in ways that can be inscrutable.
- Practical Stance: It may take extensive interaction with these artefacts before familiar psychological language or new terminology settles into appropriate use.The paper presents this as an open process rather than a settled vocabulary.
- Practical Stance: The paper recommends resisting anthropomorphism when discussing LLMs and the systems embedding them.This recommendation follows the contrast between their convincing behaviour and fundamentally different construction.