PaperlayerView exact revision on arXiv

Source-linked AI summary

Six misconceptions about large language models: A minimal model and diagnostic taxonomy

Zhicheng Lin

arXiv:2608.20421v1cs.CYcs.AI

TL;DR

Debates about LLMs rely on folk theories, but their conceptual structure has not been systematically delineated. This Perspective uses four distinctions to diagnose six misconceptions and shows how conflations can produce predictable errors in evaluation, deployment, and policy.

Abstract

from arXiv · show

Large language models (LLMs) are now embedded in scientific, educational, and governance workflows, with debates centering on their capabilities, mechanisms, and impacts. Yet these debates remain structured by persistent folk theories--intuitive, informal explanatory models that guide attitudes and actions. Deflationary slogans ("just autocomplete," "stochastic parrots," and "average of the internet") and anthropomorphic framings ("emergent agents" and "proto-minds") each capture genuine features of current systems but mistake those features for the whole. This Perspective proposes a minimal working model of LLM-based systems centered on four distinctions: between pretraining and deployed systems; between the learned distribution and particular samples; among parametric, contextual, and external memory; and between task competence and agency. The model is used to diagnose six misconceptions about LLMs: next-token prediction, regression to the mean, training-data regurgitation, model memory, alignment, and understanding. For each, the analysis identifies what the misconception gets right, which distinctions it conflates, and what follows for capability evaluation, system design, and governance. Applied to publisher AI policies as governance case studies, the framework shows both how policy language can conflate these distinctions and how such errors can be corrected. The model thereby avoids the parrot-mind binary by treating LLMs as simulators of discourse and task performance, offering a diagnostic toolkit for locating and correcting the errors these folk theories perpetuate.

Folk theories of large language models

The section presents a four-distinction framework for diagnosing six LLM misconceptions and applies it to capability evaluation, system design, and publisher AI policies. It argues that folk theories arise when a real feature of a system is treated as a complete account of what it is or can become.

Folk theories

  • Folk theories emerge when a scoped feature is inflated into a global account, polarizing debate between dismissive deflation and anthropomorphic accounts of nascent minds or agents.Both extremes can mislead capability evaluation, system design, and governance.Source: Folk theories of large language models · p. 2

Framework

  • The framework addresses a gap in prior work by linking conceptual confusions to downstream errors through four distinctions and a diagnostic matrix for six misconceptions.The distinctions separate pretraining from deployed systems, learned distributions from particular samples, parametric from contextual and external memory, and task competence from agency.Source: Folk theories of large language models · pp. 3–4

Memory

  • The claim that LLMs remember everything or nothing overlooks fixed weights, bounded context, and product-level storage, risking overtrust in chat learning or assumptions of zero state.The diagnostic asks where information is stored and reused, with explicit, auditable policies for weights, context, and external stores.Source: Folk theories of large language models · p. 4

Alignment

  • Treating RLHF or fine-tuning as a removable safety filter ignores that instruction tuning changes weights encoding knowledge and skills, potentially shifting values and capabilities.The framework recommends evaluating regressions and value shifts and documenting training objectives separately from deployment-time filters.Source: Folk theories of large language models · p. 4

Understanding

  • The framework rejects both treating LLMs as human-like agents and trivializing them as parrots, distinguishing substantial task competence from embodied, human-like minds.It recommends describing capabilities functionally rather than inferring beliefs, desires, or moral standing from stylistic fluency.Source: Folk theories of large language models · p. 4

Publisher policy examples

  • Publisher policies that attribute an incomplete picture to training cutoffs can obscure whether deployed systems use current retrieval and how they verify claims.The diagnostic asks whether the system relies on parametric knowledge or retrieval and how key claims are checked.Source: Folk theories of large language models · p. 5
  • Publisher policies focused on near-verbatim reproduction understate risks from uncredited paraphrase and structural copying of arguments or methods.The diagnostic distinguishes near-verbatim output from idea-level borrowing that still requires citation and attribution.Source: Folk theories of large language models · p. 5
  • Policy language can conflate data entry with broad organizational access and treat human-like creativity as all-or-nothing; the reviewed policies do not explicitly illustrate the alignment misconception.The framework instead asks where data is stored and who can access it, and which creative or critical tasks are reliably performed.Source: Folk theories of large language models · p. 5

A minimal working model of LLM-based systems

The minimal working model distinguishes training from deployment, learned distributions from sampled outputs, three forms of memory, and task competence from agency. These distinctions help locate where claims about LLM-based systems elide important features.

Distribution and samples

  • A learned distribution is distinct from the particular sequence selected by decoding, and defaults such as low temperature and cautious prompts help create an assistant voice.Greedy, temperature, and top-p decoding trade diversity against predictability.Source: Figure 1 · p. 6, A minimal working model of LLM-based… · p. 7

Training and deployment

  • Pretraining freezes weights after learning a conditional next-token distribution, while instruction tuning and RLHF reshape those same parameters and deployed products add system-level behavior.Fine-tuning steers outputs toward preferences such as helpfulness, harmlessness, or honesty; variants of the same base model can therefore behave differently.Source: A minimal working model of LLM-based… · pp. 6–7

Deployed systems

  • A deployed system wraps a model with prompts, interfaces, filters, tools, and retrieval, and can feed tool outcomes back into context as a closed-loop controller.The same pretrained model can support qualitatively different systems depending on how it is wrapped, aligned, and deployed.Source: Figure 1 · p. 6, A minimal working model of LLM-based… · p. 7

Memory

  • System memory comprises parametric information in weights, bounded contextual working memory, and external product-level stores selectively injected through retrieval or orchestration.Confusing these layers can produce mistaken claims about recall, forgetting, and learning from chats.Source: Figure 1 · p. 6, A minimal working model of LLM-based… · p. 7

Cognitive status

  • LLMs can support abstraction, transfer, and in-context learning, but this functional competence is distinct from human-like understanding, unified intentions, and consciousness.Agency terms such as belief, desire, and intention should be reserved for carefully specified functional roles.Source: A minimal working model of LLM-based… · p. 8

Scope and use

  • The model covers deployed products shaped by wrappers, tools, retrieval, memory, decoding, or closed-loop action, and flags claims that elide these distinctions as misconceptions.It serves as a conceptual tool for locating errors and mistaken inferences about the system at issue.Source: A minimal working model of LLM-based… · p. 8

Six misconceptions

The six misconceptions each preserve a kernel of truth but conflate distinctions in the minimal model, leading to characteristic errors in evaluation, deployment, or governance.

Six misconceptions

  • The taxonomy groups six recurrent misconceptions into three about statistics and generativity, two about memory and alignment, and one about cognitive status.Each preserves a kernel of truth while conflating at least one distinction in the minimal model, producing characteristic downstream errors.Source: Six misconceptions · p. 8
  • Table 1 maps each misconception from its folk claim and kernel truth through its conflated distinction and downstream mistake to a corrective question and practical heuristic.Claims about human-like creativity, artificial general intelligence, emotional experience, or attachment can be treated chiefly as variants of misconception 6’s competence-agency conflation.Source: Six misconceptions · p. 8

Misconceptions about statistics and generativity · Misconception 1: “LLMs are just next-token predictors” or “stochastic parrots”

Current LLMs are conditional next-token predictors, but that mechanism alone does not fully describe the competence of deployed systems. Capability and risk claims should specify the deployed system—including its wrapper, tools, and decoding policy—not just its pretraining objective.

Misconception 1: “LLMs are just next-token predictors” or “stochastic parrots”

  • GeneGPT achieved 0.83 on GeneTuring, compared with 0.00–0.44 for LLMs without tool augmentation (45).GeneGPT augments Codex with NCBI Web APIs for genomics tasks; the predictor is unchanged, but the system is not.Source: Misconception 1: “LLMs are just… · p. 8
  • Conditional next-token prediction accurately describes current LLMs’ pretraining mechanism but becomes misleading as a complete account of LLM-based systems or as grounds for dismissing them as cognitively trivial.The stochastic-parrots concerns about grounding, data provenance, and scale do not establish that the pretraining objective circumscribes deployed-system competence (44).Source: Misconception 1: “LLMs are just… · p. 8
  • Instruction tuning and RLHF alter the learned distribution, while tool-using loops can make next-token prediction implement an action policy that incorporates observations into subsequent predictions.Capability and risk claims should specify the deployed system’s wrapper, tools, and decoding policy.Source: Misconception 1: “LLMs are just… · pp. 8–9

Misconception 2: “LLMs regress to the mean” or are “the average of the internet”

LLMs learn conditional distributions, not single average answers; apparent blandness and departures from typical human outputs depend on decoding, prompts, retrieval, search, selection, and human judgment. The section presents three routes to outputs beyond modal human practice: evaluator-guided search, high-variance generation with human filtering, and synthesis unconstrained by human embodiment.

Core diagnosis

  • The learned distribution is not a single average answer; bland, consensus-style output usually reflects low-temperature or greedy decoding, alignment pressures, and uniform assistant prompts (29, 30).The proposed diagnostic is to vary prompts and decoding settings and report which regimes evaluations and deployments use.Source: Misconception 2: “LLMs regress to the… · p. 9
  • Retrieval improved GPT-4’s FreshQA factual accuracy by 32.6–49.0% relative to the same model without retrieval; PaperQA2 reached 85.2% precision versus 73.8% for PhD-level human experts on LitQA2.Both examples reflect retrieval and orchestration around the same parametric model.Source: Misconception 2: “LLMs regress to the… · p. 9

Evaluator-guided search

  • Evaluator-guided search can find outputs beyond historical human practice, as AlphaEvolve improved on several of 67 problems; explicit evaluators work best when criteria are checkable (46–50).AlphaGo’s Move 37 was highly effective despite being judged vanishingly unlikely for a professional, and later entered professional play.Source: Misconception 2: “LLMs regress to the… · p. 10

Synthesis unconstrained by human embodiment

  • Visual systems can generate impossible camera movements, fantasy transformations, stylized action sequences, and biomechanically difficult or implausible maneuvers without the same reliance on live performers, sets, stunt teams, animation, or conventional visual-effects pipelines.This shifts part of the workflow from filming and performance to generation, selection, and editing.Source: Misconception 2: “LLMs regress to the… · p. 10

Misconception 3: “LLMs just regurgitate the training data”

LLMs can reproduce training-corpus snippets, but most generations recombine patterns rather than copy text. Treating them as lookup tables conflates parametric compression, contextual prompting, and external storage, obscuring when outputs are actually copied.

Misconception 3: “LLMs just regurgitate the training data”

  • LLMs can reproduce training-corpus snippets, especially boilerplate or famous passages, raising privacy and copyright concerns (62).The passage frames the frequency of copied or near-verbatim outputs as an empirical question.Source: Misconception 3: “LLMs just… · p. 11
  • Most LLM generations are recombinations of patterns learned across many sources, not copies.Treating an LLM as a giant lookup table conflates parametric compression, contextual prompting, and external storage.Source: Misconception 3: “LLMs just… · p. 11
  • The framework treats copying as an empirical question and links mitigation to data curation, leakage auditing, and product-level memory design.Table 1 reframes the issue around when outputs are copied or near-verbatim.Source: Misconception 3: “LLMs just… · p. 11

Misconceptions about memory and alignment

Deployed LLM systems have distinct parametric, contextual, and external memory layers, so asking whether a model simply remembers or forgets misses where information is stored and reused.

Misconceptions about memory and alignment

  • Deployed systems distinguish fixed parametric weights, context-window-limited prompt history, and external product memory selectively injected into prompts.Parametric memory is a lossy compression of training data and stays fixed during ordinary inference (28); external memory includes logs, profiles, documents, and retrieval indices (36).Source: Misconceptions about memory and… · p. 11
  • The key governance question is where information is stored, retained, and reused, with safeguards made explicit and auditable across memory layers.The passage identifies external, product-level memory as a concentration of privacy and governance risks.Source: Misconceptions about memory and… · p. 11

Misconception 5: “Fine-tuning and RLHF are just superficial filters on a neutral core” … Implications for evaluation, deployment, and governance

Fine-tuning and RLHF reshape learned behavior rather than merely filtering a neutral core, while cognitive claims should distinguish bounded task competence from agency. The taxonomy directs practical attention to capability evaluation, deployment design, and institutional governance.

Misconception 5: “Fine-tuning and RLHF are just superficial filters on a neutral core”

  • Supervised fine-tuning and RLHF update knowledge- and skill-encoding parameters, reshape the conditional distribution, and inscribe institutional preferences rather than adding a reversible filter to a neutral core.Pretraining already reflects its data’s norms, omissions, and sampling biases; external filters can be added or removed without retraining, but conflating them with RLHF obscures trade-offs in coverage, style, and refusals.Source: Misconception 5: “Fine-tuning and RLHF… · p. 11
  • DeepSeek-R1’s official deployment combines post-training alignment with server-side filters that can interrupt or overwrite responses to politically sensitive questions [63–65].Its underlying weights support strong math and coding performance, while sensitive mixed prompts can prompt refusals, party-line answers, or loss of step-by-step reasoning.Source: Misconception 5: “Fine-tuning and RLHF… · p. 12
  • Open weights make DeepSeek-R1’s learned policy partly editable: reported fine-tuning and weight-editing approaches reduce political refusals while largely preserving reasoning or benchmark performance [63–65].The case shows censorship is neither a detachable mask nor immutable; for closed-weight systems, the boundary between weight-level behavior and external filtering is empirically opaque.Source: Misconception 5: “Fine-tuning and RLHF… · p. 12

Misconception 6: “LLMs think like humans” vs. “LLMs do not understand anything”

  • Anthropomorphic and deflationary accounts alike conflate competence with agency, respectively treating fluent performance as evidence of human-like agency or next-token prediction as proof of meaninglessness [37, 66].Current models show bounded functional understanding through systematic generalization, in-context learning, and multistep reasoning, yet lack embodiment, persistent online learning, and stable internal state supporting human-like experiential understanding [28, 37, 67].Source: Misconception 6: “LLMs think like… · p. 12
  • Assess which tasks models can reliably perform and which forms of agency are being attributed, rather than inferring agency or moral standing from fluent style [39].Describing competence functionally helps avoid both over- and under-attribution.Source: Misconception 6: “LLMs think like… · p. 12

Implications for evaluation, deployment, and governance

Evaluation and capability modeling … Publisher AI policies as governance case studies

The sections argue that capabilities and risks depend on deployed configurations, workflows, and governance choices, not on abstract claims about LLMs alone. They apply this framework to system design, legal accountability, and publisher policies, identifying both practical effects and recurring conceptual conflations.

Evaluation and capability modeling

  • Evaluate the deployed configuration—not the base model or default sample—by specifying prompts, decoding, retrieval, tools, memory, and workflows, then testing stability under perturbation and realistic-use failures.This avoids missing closed-loop behavior, tool use, and in-context learning while also avoiding overgeneralizing from striking demonstrations to stable competence.Source: Evaluation and capability modeling · p. 13
  • A redesigned clinician–AI workflow raised diagnostic accuracy to 82–85%, versus 75% with conventional resources; GPT-4 alone scored 16 points above control, while physician access yielded a nonsignificant 2-point gain (68, 69).The follow-up used the same vignettes and scoring rubric, but independent clinician and AI assessments followed by AI synthesis; the model did not change, only the workflow.Source: Evaluation and capability modeling · p. 13

Deployment and system design

  • Distinguishing parametric, contextual, and external memory supports auditable storage and access controls; fine-tuning and RLHF should be treated as value-laden, measurable, and preferably reversible or auditable interventions.The DeepSeek-R1 case shows that one open-weight starting point can be aligned, filtered, or realigned into divergent normative regimes through post-training and deployment.Source: Deployment and system design · p. 13

Governance and public reasoning

  • Governance should scrutinize training-data curation, alignment objectives and preference-data oversight, tool and memory interfaces, and decoding and prompting defaults, rather than reducing LLMs to parrots or proto-persons.The former framing understates risks from unreliability, scale, and misuse; the latter diverts debate toward premature questions of moral standing or AI rights.Source: Governance and public reasoning · p. 13
  • Moffatt v. Air Canada held the airline liable for chatbot misinformation, while Mata v. Avianca prompted disclosure and certification orders that sometimes regulate AI use abstractly rather than target fabricated sources (70, 71).NIST’s Generative AI Profile distinguishes pretrained, adapted, and deployed systems and calls for documentation of fine-tuning, retrieval augmentation, and postdeployment monitoring (72).Source: Governance and public reasoning · pp. 13–14

Publisher AI policies as governance case studies

  • Roughly 70% of journals had AI policies, mostly disclosure requirements, yet AI-assisted writing rose, disclosure remained rare, and policy and non-policy journals showed no detectable difference in writing rates (75).Publisher and journal policies vary in what counts as AI use and how it must be disclosed (73, 74).Source: Publisher AI policies as governance… · p. 14
  • Publisher policies link statistical framing or training cutoffs directly to unreliability and emphasize large-text reproduction, overlooking retrieval, verification, and paraphrastic or structural borrowing.These rationales echo misconceptions 1–3 by treating next-token prediction or cutoff dates as determinative and memorization as lookup-table reproduction.Source: Publisher AI policies as governance… · p. 14
  • Policies conflate memory types and deployment regimes, embed a strong cognitive claim about creativity and critical thinking, and largely omit fine-tuning, RLHF, and weight-level alignment (misconceptions 4–6).The passages instead treat material entered into generative AI as inherently provider-accessible and often merge deployments with different storage, logging, and reuse properties.Source: Publisher AI policies as governance… · p. 14

Conclusion · Funding · Data Availability

The paper presents its minimal working model and six-misconception taxonomy as a diagnostic toolkit for evaluating claims about LLMs. It does not settle questions about understanding, reasoning, agency, or moral standing, but aims to reduce reliance on slogans in policy and practice.

Conclusion

  • The framework offers a diagnostic toolkit for claims about what LLMs are, can do, or imply.It asks which level of description is invoked, which distinction is conflated, and what preserving that distinction means for evaluation, deployment, and governance.Source: Conclusion · p. 15
  • It does not settle whether LLMs genuinely understand, reason, possess agency, or warrant moral standing.Instead, it aims to reduce the risk that policy and practice are steered by slogans rather than by systems’ architecture, behavior, and deployment conditions.Source: Conclusion · p. 15

Funding

  • The author received support from a national science and technology project, Yonsei University Research Fund, and Yonsei Fellowship.The listed support includes Brain Science and Brain-like Intelligence Technology—National Science and Technology Major Project (2021ZD0204200), Yonsei University Research Fund (5355661), and the Yonsei Fellowship funded by Lee Youn Jae.Source: Funding · p. 15
  • The listed funding sources are the National Science and Technology Major Project, Yonsei University Research Fund, and Yonsei Fellowship.The project number is 2021ZD0204200, the fund number is 5355661, and the fellowship is funded by Lee Youn Jae.Source: Funding · p. 15

Data Availability

Related papers

Loading 2608.20421v1…