Source-linked AI summary
Six misconceptions about large language models: A minimal model and diagnostic taxonomy
Zhicheng Lin
TL;DR
Debates about LLMs are shaped by folk theories that mistake genuine features for complete accounts, creating confusion about capabilities, mechanisms, and impacts. This Perspective introduces a four-distinction model and applies it to six misconceptions and publisher AI policies, showing how conceptual conflations produce predictable reasoning errors.
Problem
Persistent folk theories about LLMs recur across scientific, public, and institutional discourse, but their conceptual structure has not been systematically delineated or connected to downstream reasoning errors.
Method
The paper builds a diagnostic taxonomy around four distinctions and applies its matrix to misconceptions and publisher-policy language.
Results
The framework identifies six misconceptions whose partial truths conflate distinctions and generate predictable errors in capability evaluation, deployment design, and institutional AI policy.
Takeaways & Limitations
Treating LLMs as simulators of discourse and task performance avoids the parrot-mind binary and supports more precise evaluation, system design, and governance.
Abstract
from arXiv · showhide
Large language models (LLMs) are now embedded in scientific, educational, and governance workflows, with debates centering on their capabilities, mechanisms, and impacts. Yet these debates remain structured by persistent folk theories--intuitive, informal explanatory models that guide attitudes and actions. Deflationary slogans ("just autocomplete," "stochastic parrots," and "average of the internet") and anthropomorphic framings ("emergent agents" and "proto-minds") each capture genuine features of current systems but mistake those features for the whole. This Perspective proposes a minimal working model of LLM-based systems centered on four distinctions: between pretraining and deployed systems; between the learned distribution and particular samples; among parametric, contextual, and external memory; and between task competence and agency. The model is used to diagnose six misconceptions about LLMs: next-token prediction, regression to the mean, training-data regurgitation, model memory, alignment, and understanding. For each, the analysis identifies what the misconception gets right, which distinctions it conflates, and what follows for capability evaluation, system design, and governance. Applied to publisher AI policies as governance case studies, the framework shows both how policy language can conflate these distinctions and how such errors can be corrected. The model thereby avoids the parrot-mind binary by treating LLMs as simulators of discourse and task performance, offering a diagnostic toolkit for locating and correcting the errors these folk theories perpetuate.
Folk theories of large language models
The paper treats recurring LLM slogans as folk theories: partial explanations that become misleading when generalized beyond their original scope. A four-distinction diagnostic matrix connects these conflations to errors in evaluation, system design, and publisher policy.
- Folk theories of large language models: Folk theories are intuitive, informal explanatory models that guide how people understand and act toward technological systems.They can coexist with technical accounts and are defined by explanatory mode rather than speaker sophistication.
- Folk theories of large language models: Slogans such as “stochastic parrots” and “a blurry JPEG of the Web” began as scoped critiques or analogies before becoming broad theories of what LLMs are.Their original scopes included data provenance, synthetic-text harms, and compression rather than complete accounts of system behavior.
- Memory and alignment: For memory, the paper distinguishes fixed weights, bounded contextual working memory, and separate product storage rather than assuming systems remember everything or nothing.The corrective question concerns storage location, access, retention, and training reuse.
- Memory and alignment: Treating fine-tuning or RLHF as a removable safety filter is mistaken because these procedures alter the same weights that encode knowledge and skills.The recommended response is to evaluate regressions and value shifts while documenting training objectives separately from deployment-time filters.
- Folk theories of large language models: The framework rejects both anthropomorphizing models as agents and trivializing their capabilities as mere parroting.It instead separates task competence from agency and functional from human-like understanding.
- Diagnostic framework: The taxonomy organizes six misconceptions around four distinctions: training versus deployment, distribution versus sample, memory layers, and competence versus agency.Its matrix moves from folk claim and kernel truth through diagnosis to corrective questions and practical fixes.
- Policy application: Publisher policies emphasize disclosure, citation verification, and confidentiality but can blur architecture-level distinctions when explaining reliability, memory, or cognition.One policy example treats incomplete knowledge as a function of training cutoff rather than deployed information acquisition and validation.
- Understanding and agency: Policy claims that systems cannot replicate human creative or critical thinking can impose an all-or-nothing cognitive thesis instead of specifying tasks and evaluation conditions.The paper recommends describing concrete functional performance without treating fluency as evidence of beliefs, desires, or moral standing.
A minimal working model of LLM-based systems
The minimal working model separates offline training from deployed products and distinguishes learned distributions, decoded samples, and three memory layers. It also treats the model as one component in systems that may use prompts, retrieval, tools, filters, and external action loops.
- System boundary: The model distinguishes an LLM from an LLM-based system: the former is a model class, while the latter is a deployed product built around such a model.The framework uses “AI” and “generative AI” more broadly for policy language or non-LLM cases.
- Training and inference: Pretraining minimizes next-token cross-entropy to learn a high-dimensional conditional distribution rather than a table of canned completions.Training is offline, and the base model’s weights are frozen after training.
- Training and inference: Inference produces a next-token distribution that decoding policies such as greedy, temperature, or top-p sampling turn into particular output sequences.These policies trade diversity against predictability.
- Post-training: Instruction tuning and RLHF directly update model parameters, reshaping knowledge- and competence-encoding weights rather than adding a detachable filter.Fine-tuned variants of one base model can therefore have different behavioral profiles.
- Deployment: Deployed products wrap models with prompts, interfaces, safety filters, tools, retrieval, and sometimes actuators, creating feedback loops when action results return to subsequent prompts.The model is therefore one central component rather than the entire system.
- Memory: System memory comprises parametric information in weights, contextual information in the bounded current input, and external product-level stores such as logs, profiles, and knowledge bases.These layers differ in how information is retained, retrieved, and reused.
- Cognitive status: LLMs can support substantial abstraction, transfer, and in-context learning while lacking human-like understanding, unified beliefs or intentions, and phenomenal consciousness.This separates functional understanding from human-like cognitive status.
Six misconceptions
The paper places six recurring misconceptions in a diagnostic space defined by four distinctions. Each misconception preserves a kernel of truth but conflates distinctions and produces predictable errors in evaluation, deployment, or governance.
- Six misconceptions: The six misconceptions concern statistics and generativity, memory and alignment, and cognitive status.The framework groups them as three statistical or generative misconceptions, two concerning memory or alignment, and one concerning cognition.
- Diagnostic matrix: Table 1 diagnoses each misconception through its kernel truth, conflated distinction, downstream mistake, corrective question, and practical heuristic.The same structure supports analysis of claims about creativity, artificial general intelligence, emotional experience, and attachment.
Misconception 1: “LLMs are just next-token predictors” or “stochastic parrots”
Calling LLMs next-token predictors is mechanistically accurate but incomplete for deployed systems. Instruction tuning, RLHF, tools, and feedback loops can change system behavior and competence beyond the pretraining objective alone.
- Kernel truth and limitation: Current LLMs are conditional next-token predictors trained by cross-entropy minimization, but that mechanism does not fully characterize deployed-system competence.The “stochastic parrots” framing captures concerns about grounding, data provenance, and scale without establishing that pretraining bounds system capability.
- Post-training: Instruction tuning and RLHF alter the learned distribution by updating model parameters rather than merely applying an external safety filter.These changes reshape the same weights that encode knowledge and skills.
- Deployed systems: Tool-using loops can implement action policies by feeding observations back into subsequent next-token predictions.Capability and risk claims should therefore specify the deployed wrapper, tools, and decoding policy.
Misconception 2: “LLMs regress to the mean” or are “the average of the internet”
The “average of the internet” misconception confuses the learned conditional distribution with particular outputs. Decoding, alignment, and search or evaluation loops shape whether systems produce modal or less-explored outputs.
- LLMs learn conditional distributions, not a single average answer, including long-tail and rare structures.
- Output homogenization is not mathematically inevitable because samples depend on decoding, prompting, and alignment regimes.Bland output often reflects low temperature or greedy decoding, generic-phrasing pressures, and uniform assistant prompts.
- Search, reinforcement learning, evaluation, or human curation can move generative systems into low-probability or underexplored regions.
- FunSearch, AlphaGeometry, and AlphaEvolve illustrate evaluator-guided routes beyond typical human outputs in formal domains.AlphaEvolve rediscovered best-known solutions in most of 67 problems and improved on several others.
Misconception 3: “LLMs just regurgitate the training data”
LLMs can reproduce some training-corpus snippets, but treating them as giant lookup tables wrongly assumes every output is copied text. The relevant question is empirical: when are outputs actually copied or near-verbatim?
- LLMs can reproduce boilerplate or famous training-corpus passages, creating privacy and copyright concerns.
- Most generations are recombinations of patterns learned across many sources rather than direct copies.
- The analysis directs mitigation toward data curation, leakage auditing, and product-level memory design.
Misconceptions about memory and alignment
Memory misconceptions arise because deployed systems combine fixed model weights, bounded conversational context, and external product storage. Alignment likewise concerns the same parameters encoding knowledge and skills, rather than a detachable surface layer.
- Memory: Parametric memory is frozen, lossy information encoded in model weights during training.
- Memory: Contextual memory includes conversation history and retrieved material only while it remains within the context window.
- Memory: External memory stores logs, profiles, documents, and retrieval indices outside the model, concentrating important privacy and governance risks.
Misconception 5: “Fine-tuning and RLHF are just superficial filters on a neutral core”
Fine-tuning and RLHF do not simply attach superficial filters to a neutral core: they alter the model’s parameters, distribution, and policy. Deployment choices further shape behavior, so evaluation must specify the complete deployed configuration.
- Supervised fine-tuning and RLHF update knowledge- and skill-encoding parameters rather than adding a detachable alignment mask.
- Alignment changes the model’s conditional distribution and produces a different policy, not a pristine core plus a surface filter.
- DeepSeek-R1’s behavior depends heavily on post-training alignment and deployment choices, including server-side filters in its official app.
- The framework treats evaluation, deployment design, and institutional governance as its three practical targets.
- Capability evaluation should analyze the deployed configuration, specifying prompts, decoding, retrieval, tools, memory, and workflows.Evaluations should also test stability under perturbation and examine failures in realistic use.
- In a 2024 clinical trial, GPT-4 access improved diagnostic reasoning by only 2 percentage points over conventional resources, a nonsignificant difference.GPT-4 alone outperformed the control group by 16 points in the same example.
Deployment and system design
The framework links LLM misconceptions to concrete design and governance choices, emphasizing architecture-sensitive distinctions in memory, alignment, reliability, and deployment. Publisher policies illustrate how conflated concepts can produce broad or poorly targeted rules.
- System design: Distinguishing parametric, contextual, and external memory clarifies what is stored in weights, retained in short-lived context, or kept in auditable storage with separate access controls.This distinction addresses architectures that assume nonexistent continuity or reuse user data through opaque product-level memory.
- Governance: The minimal model directs governance toward training-data curation, alignment objectives, preference-data oversight, tool and memory interfaces, and decoding and prompting defaults.These are specific design levers rather than abstract judgments about whether systems are parrots or proto-persons.
- Governance: Moffatt v. Air Canada assigned accountability for chatbot misinformation to operators of the deployed system rather than to the chatbot itself.The case illustrates how deployment-level distinctions shape legal responsibility.
- Governance: The NIST Generative AI Profile distinguishes pretrained, adapted, and deployed systems and calls for documentation of fine-tuning and retrieval augmentation.This architecture-sensitive approach contrasts with regulation of AI as an undifferentiated category.
- Publisher AI policies as governance case studies: Publisher policies vary in what counts as AI use and how it must be disclosed, while one analysis found roughly 70% of journals had adopted AI policies.Policies were mostly disclosure requirements, yet AI-assisted writing continued to rise and explicit disclosure remained rare.
- Publisher AI policies as governance case studies: Policy language often links AI-generated text or training cutoffs directly to unreliability, overlooking retrieval, tool use, and verification.Warnings about reproducing text also emphasize literal copying more than paraphrastic or structural borrowing.
- Publisher AI policies as governance case studies: Confidentiality policies can treat material entered into generative AI systems as inherently accessible to providers, merging deployments with different storage, logging, and reuse properties.The framework instead recommends targeting data flows and data policies.
- Publisher AI policies as governance case studies: Statements that LLMs cannot replicate human creative and critical thinking build a strong cognitive thesis into policy instead of tying restrictions to task-level competence, documented reliability, and accountable use.These policies also say little about fine-tuning, RLHF, or interactions between weight-level alignment and external filters.
Conclusion
The framework offers a diagnostic toolkit for evaluating claims about what LLMs are, can do, or imply. It preserves open questions about understanding, reasoning, agency, and related issues while directing attention to distinct descriptive levels and practical consequences.
- Conclusion: The framework asks which descriptive level is invoked, which distinction is conflated, and what follows for evaluation, deployment, and governance when that distinction is preserved.It is intended to locate and correct errors in claims about LLMs.
- Conclusion: The paper does not settle whether LLMs genuinely understand, reason, or possess agency.Its contribution is diagnostic rather than a final resolution of these questions.