Source-linked AI summary
Not Even Wrong: On the Limits of Prediction as Explanation in Cognitive Science
Mark Orr, Drew Cranford, Ken Ford, Kevin Gluck, Will Hancock, Christian Lebiere, Pete Pirolli, Frank Ritter, Andrea Stocco
TL;DR
Centaur has been presented as a step toward unified theories of cognition, but this comment questions whether behavioral prediction establishes a model of cognition. By examining its mechanisms, generality, and measurement, the authors conclude that Centaur is better understood as a model of behavior than cognition, and recommend focusing on suitable predictive uses.
Problem
Centaur is presented as a step toward unified cognition, raising whether behavioral prediction without explicit cognitive mechanisms supports that theoretical claim.
Source: Concerns · pp. 4–5Method
The comment examines Centaur’s mechanisms, generalization, Newell Test criteria, and the measurement implications of translating experiments into natural language.
Source: Concerns · pp. 4, 5, 9Results
Centaur’s apparent generality comes from translating experimental paradigms and data into natural language for next-token prediction, while its brain-activity correlation offers no additional insight into algorithmic or implementational solutions.
Source: Concerns · pp. 5, 14Takeaways & Limitations
The authors recommend refining Centaur’s predictive capabilities for suitable purposes, especially low-criticality or dynamically stable contexts, rather than imposing theoretical obligations about cognition.
Source: Conclusion8 · p. 15The authors report no experimental data to determine whether humans show similar behavior or use the same cognitive processes across original and translated versions of Psych-101 experiments.
Source: Concerns · p. 10
Abstract
from arXiv · showhide
We offer a comment on the Centaur (Binz et al., 2025) transformer-based model of human behavior. In particular, Centaur was cast as a path towards unified theories of cognition. We offer a counter claim with supporting argument: Centaur is a path divergent from unified theories of cognition, one that moves towards a unified model of behavior sans cognition.
Introduction
This comment describes Centaur as a pretrained transformer fine-tuned on experimental human-behavior data and questions its presentation as a unified model of cognition. It acknowledges the model’s scale and novelty while setting out concerns about its claims and limitations.
Introduction
- Centaur was trained on over 10 million single trials from approximately 60 thousand participants across 160 cognitive tasks and outperformed every task-specific state-of-the-art computational model.Its authors initially called it the first true unified model of human cognition, a claim tempered upon publication.Source: Introduction · p. 1
- The commentary presents overlapping perspectives on Centaur’s deficiencies, limitations, omissions, and contentious claims.It also notes responses from Bowers et al. (2025) and O’Grady (2025).Source: Introduction · p. 1
- Simulating an experiment with Centaur requires translating its paradigm and behavioral data into natural language.The passage presents this as a requirement for any experiment simulated by Centaur.Source: Introduction · p. 1
- The authors’ unified-cognition claim is tempered in the published paper, where the term appears only in the conclusion and is not well defined.The commentary notes that the earlier Psych Arxiv version made a stronger claim.Source: Introduction · p. 1
Contributions
The authors recognize contributions in collaborative science, data infrastructure, usability, and automated scientific practice. They then turn to concerns about the approach.
Contributions
- Centaur is a cooperative computational effort that may add to lessons about organizing research in cognitive, neural, and psychological sciences.The authors place it alongside large-scale community efforts such as the Human Connectome and Cognitive Atlas, and cognitive architectures such as Soar and ACT-R.Source: Contributions · p. 2
- Centaur and Psych-101 contribute a harmonized repository combining 160 experiments and 10 million single-trial responses from approximately 60 thousand participants.The commentary identifies harmonization across experiments as an impressive feature.Source: Contributions · p. 2
- Centaur shifts model-design decisions to a workflow requiring a pretrained LLM, natural-language task and data translation, and fine-tuning.The authors say this removes implementation burdens from cognitive modelers and removes the need for a cognitive modeler.Source: Contributions · p. 2
- Centaur and Psych-101 add another approach to automated scientific practice in cognitive science and psychology.The passage situates this approach alongside recent work by Rmus et al. (2025) and Dillion et al. (2023).Source: Contributions · p. 2
- The authors describe these contributions as valuable while noting that novel approaches also bring issues they address next.Source: Contributions · p. 2
Concerns
The authors challenge claims that Centaur’s predictive breadth and task-related neural alignment establish it as a unified theory of cognition. They argue that its predictions do not by themselves explain cognitive processes, map model representations to brain organization, or support cumulative theoretical progress.
A Category Error
- Centaur is non-mechanistic and therefore cannot provide a path to a unified theory of cognition, for which cognitive mechanisms are foundational.The authors call the comparison with integrative benchmarking a category error because Centaur does not propose mechanisms of cognition.Source: Concerns · pp. 3–4
- Centaur’s potential to model behavior beyond human cognition is behavior-only coverage, not mechanistic unification across cognitive processes.The authors distinguish predicting behavior from the environment from integrating cognitive or neural mechanisms, and argue that statistical fit across experiments may suggest process sameness rather than unification.Source: Concerns · pp. 4–5
- Centaur’s apparent generality is an illusion: it performs one next-token prediction task, with breadth supplied by translating experimental paradigms and data into natural language.The authors contrast this with Newellian hopes for more general models of human cognition across tasks.Source: Concerns · p. 5
Newell’s Criteria or Newell’s Test
- The Newell Test conflates transformer training with the resulting frozen Centaur system, which lacks real-time learning and straightforward adaptation to new tasks.The authors also question real-time claims because processing duration depends on hardware, and say fine-tuning on extensive human experimental data is implausible for human participants.Source: Concerns · pp. 5–7
- Centaur’s Newell Test performance remains limited or unclear on knowledge integration, grounded semantics, metacognition, development, evolution, and brain-level correspondence.The authors say its semantic capacities derive from language statistics without external grounding, its professed awareness lacks evidence of metacognition, and its neural alignment claim overreaches the available system-level mapping.Source: Concerns · pp. 6–7
Measurement and its Discontents
- Centaur has not tested its natural-language task versions on humans, leaving no direct version-to-version comparison to establish similar behavior or cognitive processes.The proposed Go/No-Go comparison would give participants 135 K choices for training before testing; the authors report no data to settle whether humans behave similarly across versions. Centaur’s response times are next-token predictions, not simulated cognitive-processing durations.Source: Concerns · pp. 9–10
Centaur is Non-Mechanistic and Atheoretic
- Theories do more than predict: they can explain, guide interventions, generate questions, and advance discovery, while Centaur’s failures would be difficult to connect to a principled account of cognition.The authors argue that if Centaur’s predictions were poor, it would be unclear what had been falsified or how that failure could improve cognitive understanding; hence their conclusion that Centaur “isn’t even wrong.”Source: Concerns · pp. 11–13
Concerns
- Centaur’s better brain-activity prediction than LLAMA does not establish neural alignment, since PCA-reduced hidden states were tested through separate regressions for each participant and cortical parcel.The connection to brain data is indirect and highly mediated by these fitted models.Source: Concerns · p. 13
- The analysis appears to capture task-functional correlation, offering no additional insight into the algorithmic or implementational solutions of the human brain.The fine-tuned model’s representations contain marginally more useful information than the standard model for predicting brain activity.Source: Concerns · p. 14
- The authors provide no other evidence of substantive similarity between Centaur’s internal representations and brain function, noting differences in learning, memory, and architecture.LLMs lack human-like long-term episodic memory, rely on context-window-limited memory, and use a fixed transformer architecture that learns matrix operations over large inputs.Source: Concerns · p. 14
- Regression into neural-activity space provides no explanatory account of how Centaur’s representations map to brain regions or whether the results are valid.The authors contrast this with detailed experimentation matching brain structures and architectural mechanisms (Borst, et al., 2013; Borst, et al., 2015; Anderson, et al., 2008; Stocco, et al., 2010; Stocco, et al., 2024).Source: Concerns · pp. 13–14
Conclusion8
The authors argue that Centaur’s predictive performance does not establish a unified theory of cognition, and that its comparison set leaves important cognitive-modeling approaches out. They recommend refining its predictions for suitable purposes and settings rather than imposing theoretical claims about cognition.
Conclusion8
- Centaur’s prediction is unsurprising because transformers capture stimulus–response regularities in translated experiments, though task and person variability merit analysis.The authors also suggest that models with fewer parameters might suffice, giving 1.5 billion and 500 million as examples.Source: Conclusion8 · pp. 14–15
- The comparison set omits state-of-the-art cognitive architectures, dense process measures, and complex tasks with mechanistic accounts, limiting its coverage of cognitive modeling.The authors say such models and process data have been used to provide detailed mechanistic explanations of cognitive processes.Source: Conclusion8 · p. 15
- They recommend purpose-specific prediction in low-criticality or dynamically stable, extrapolation-insensitive settings, with possible personalization for intervention and prevention in disaster response, medicine, and psychology.Source: Conclusion8 · p. 15
- 90% of approximately 10 million responses from 160 experiments and ~60,000 participants were used for training; the authors caution that Centaur’s human-like appearance may be illusory.The remaining 10% was held out; the authors compare the illusion to LLM claims of consciousness based on training data containing literature and socially directed internet content.Source: Conclusion8 · pp. 15–16