Source-linked AI summary
Why machines will still not rule the world
Jobst Landgrebe, Barry Smith
TL;DR
The essay asks whether physical computability and rising benchmark scores rebut the claim that complex behavior exceeds mathematical modeling. It examines theoretical arguments and benchmark audits, concluding that neither establishes that the requisite models can be obtained, while benchmark failures alone do not prove AGI impossible.
Problem
Method
The essay develops a formal ergodicity-based argument and evaluates theoretical and benchmark objections using recent audit evidence.
Source: §5 · p. 6, §6 · pp. 9–10Results
The authors conclude that theoretical arguments do not show the requisite models can be obtained, while compromised benchmark evidence establishes neither machine mastery of complexity nor AGI’s impossibility.
Source: §6 · pp. 9–10Takeaways & Limitations
High performance on a fixed benchmark distribution cannot establish performance under distribution shift, and continuously refreshed benchmarks offer only partial remedies for contamination.
Source: §5.3 · p. 9, §6 · p. 9Benchmark failures are specific to compromised instruments and do not prove AGI impossible; models still show measurable, more modest performance on contamination-resistant evaluations.
Source: §6 · p. 9
Abstract
from arXiv · showhide
In our book Why machines will never rule the world [13, 14] we argue that arti- ficial general intelligence is mathematically impossible. This is because the human beings and the processes which exhibit intelligence are complex systems whose be- haviour cannot be captured by the kinds of models that we can generate with or without computers. Proponents of contemporary machine intelligence respond with two lines of argument: a theoretical one, grounded in the universal approximation theorems for neural networks and the Church-Turing-Deutsch principle; and an em- pirical one, grounded in rapidly rising scores on standardized benchmarks. In this communication we examine and reject both responses. First, we show serious issues in the physicalist counter-argument based on the Church-Turing-Deutsch principle. Second, we review recent evidence to the effect that prominent benchmarks are compromised by training-data contamination, flawed test construction, and strate- gic optimization. Our central argument remains: That models required to perform cognitive behaviour in open-ended, thermodynamically complex and non-ergodic environments are not and will not become achievable.
1 Introduction
The introduction argues that AGI is impossible because the systems underlying intelligence cannot be adequately modeled mathematically. It evaluates theoretical and benchmark-based objections to that claim.
1 Introduction
- The authors argue that AGI is impossible because the systems that would need to be modeled to produce it are beyond mathematics’ capacity.They frame this as a counter-claim to claims that AGI is merely distant or difficult.Source: §1 · pp. 1–2
- Rising benchmark scores are often interpreted as evidence that general intelligence is approaching.The introduction describes each new LLM generation as arriving alongside benchmark-score tables.Source: §1 · p. 1
- The essay examines theoretical and empirical objections, arguing that the former establishes nothing against the thesis and recent audits undermine the latter.The theoretical objection invokes neural-network universal approximation and the Church-Turing-Deutsch principle; the empirical objection relies on benchmark performance.Source: §1 · p. 2
2 Complex systems limit mathematical modeling
The authors distinguish equation-capturable logic systems from complex systems with thermodynamic properties, and argue that intelligence depends on the latter. Their claim concerns whether adequate models can be constructed, not computational speed.
2 Complex systems limit mathematical modeling
- Logic systems are captured by explicit equations, whereas complex systems exhibit thermodynamic properties such as context dependence and non-equilibrium dynamics.The authors caution that technically complex systems are not simply systems that are complicated.Source: §2 · p. 2
- The authors describe human intelligence as produced by the organism embedded in natural and social environments, with dialogue and spontaneous adaptation expressing that complexity.They characterize these capacities as open-ended responses to novel situations.Source: §2 · p. 2
- Because available mathematics models complex systems only partially, approximately, and for limited purposes, the authors argue that programs cannot reproduce their behavior.They connect this claim to the need for the entire system’s processes, noting that partial dysfunctions can already harm brain function.Source: §2 · p. 2
- The argument concerns model availability rather than speed or hardware: without an adequate mathematical model, additional computation cannot reproduce the system’s behavior.The authors characterize algorithms as implementations of mathematical models.Source: §2 · p. 2
- The authors distinguish technical complexity from mere complication; not every complicated system is complex in their technical sense.The distinction is flagged as a source of confusion in AI literature.Source: §2 · p. 2
3 Ergodicity and non-ergodicity
The section uses ergodicity to distinguish a single trajectory’s time average from an ensemble expectation, then argues that non-ergodic human intelligence defeats the fixed-distribution assumption behind machine-learning generalization.
3 Ergodicity and non-ergodicity
- A process is non-ergodic when a trajectory’s time-average limit fails to exist or differs from the ensemble expectation on a positive-measure set.For an ergodic process, the time average along one realization converges to the ensemble average.Source: §3 · p. 3
- In the repeated 1.5-or-0.6 gamble, ensemble growth is 5% per round while almost every individual trajectory declines about 5%; the example does not generalize to real complex systems.The section uses this contrast to illustrate how ensemble and time averages can diverge in a non-ergodic process.Source: §3 · pp. 3–4
3.1 Machine learning
- Machine-learning generalization relies on training, test, and deployment data sharing a fixed distribution, while supervised learning minimizes empirical risk as an estimate of true risk.The section argues non-ergodicity can make the deployment distribution differ from the training distribution.Source: §3.1 · p. 4
- For human intelligence and its environment, non-ergodicity means even small empirical risk gives no guarantee of future performance, and the authors say adequate sampling is impossible.They identify this as a necessary condition of human intelligence and the environment in which it operates.Source: §3.1 · p. 4
4 The Connectionist Counter-Thesis
The section argues that neither universal approximation nor the Church-Turing-Deutsch principle establishes that human cognitive behavior can be modeled. The former concerns fixed functions on compact domains; the latter presupposes a model of the system and its initial state.
4.1 Universal Function Approximation
- Cybenko and Hornik, Stinchcombe, and White proved that feedforward networks with a single hidden layer are universal approximators, with a non-polynomial activation required to approximate an arbitrary functional.Source: §4.1 · pp. 4–5
- Universal approximation establishes representational capacity for fixed functions on compact domains, but not whether training finds the parameters or whether a stable target function exists for complex systems.The authors conclude that the theorem cannot show the right function can be learned or specified for such systems.Source: §4.1 · p. 5
4.2 The Church-Turing-Deutsch Principle
- The CTD-based argument concludes that a universal quantum computer can simulate a brain if it is a finitely realizable physical system governed by physical laws.The authors answer that CTD is a conjecture, and that simulation presumes the Hamiltonian and initial state are known; specifying a brain model would require roughly 10^26 interacting particles and its environment.Source: §4.2 · pp. 5–6
- Even an exact simulation of one brain would reproduce one trajectory, not provide a predictive model of an open environment; the authors therefore say CTD does not rebut their thesis.They describe their position as ‘nomological monism,’ a version of strict materialism [14, chapter 2], rather than a claim that intelligence is non-physical.Source: §4.2 · p. 6
5 So-called benchmark evidence
Benchmark performance is often treated as evidence of general capability, but audits raise substantial doubts about benchmark validity and the inferences drawn from scores. The section examines contamination, flawed test construction, leaderboard incentives, and optimization against measures.
5.1 Contamination and non-validity of benchmarks
- 59.4% of 138 audited SWE-bench verified problems had flaws in test design or descriptions, and models could reproduce some reference solutions without search.The authors cite this as evidence of possible training-data contamination amid public availability of the benchmark and its source repositories.Source: §5.1 · p. 7
- Roughly 23% on SWE-bench pro versus above 70% on SWE-bench verified shows a striking score gap, though pro tasks were deliberately longer and harder.The authors say verified scores materially reflected training configuration and test-harness idiosyncrasies, while cautioning that task difficulty explains some of the gap.Source: §5.1 · p. 7
- High software-benchmark scores do not establish reliable production performance; generated code can overlook architectural context, raise maintenance costs, or contribute to outages.The section identifies software engineering as a production domain where these practical consequences are evident.Source: §5.1 · p. 7
5.1.1 Knowledge Benchmarks and Web-Scale Contamination
- Web-scale training can expose models to knowledge-benchmark items indirectly, and contamination is difficult to rule out across detection settings and training stages.A review of 55 studies found no detection method consistently reliable across contamination types, model-access settings, and training stages; contamination weakens the inference from score to capability but does not mean every high score is contaminated.Source: §5.1.1 · pp. 7–8
- LLM judges can favor verbosity, authority cues, citations, and formatting independently of substantive correctness, so evaluations may reward expert-sounding fluency rather than reasoning.The reported judge biases include longer responses and superficial presentation features.Source: §5.1.1 · p. 8
5.2 Human-preference leaderboards
- Chatbot Arena leaderboard dynamics include selective release of privately tested variants and unequal sampling and removal policies, producing data-access asymmetries and overfitting to Arena-specific dynamics.The authors conclude that preference leaderboards measure persuasiveness to typical users on typical queries rather than performance in sustained, consequential, multi-turn tasks.Source: §5.2 · p. 8
5.3 Goodhart’s law
- When benchmarks become optimization targets through data exposure, selective reporting, or tuning to raters, scores increasingly measure optimization rather than underlying capability.A static benchmark measures empirical risk on its fixed distribution and cannot by construction establish performance under distribution shift; contamination is an extreme case of this limitation.Source: §5.3 · pp. 8–9
6 Discussion
The discussion argues that complex systems produce non-ergodic behavior that mathematics and computers cannot capture, while theoretical computability arguments do not establish that adequate models can be obtained. It also argues that benchmark audits weaken empirical claims about general intelligence without proving its impossibility.
6 Discussion
- Benchmark audits substantially weaken the empirical case for general intelligence but do not establish an empirical case against its possibility.The authors distinguish their universal impossibility thesis from contingent failures of particular measurement instruments.Source: §6 · p. 9
- The authors argue that complex systems generate non-ergodic behavior that mathematical models, and therefore computers, cannot capture.They maintain that universal approximation theorems and the Church-Turing-Deutsch principle establish representational and in-principle computational adequacy, not that the requisite models can be obtained.Source: §6 · pp. 9–10
- The human brain can respond adequately to non-ergodic processes without preparation, which the authors identify as intelligence.They cite this capacity as distinct from mathematically modeling those processes.Source: §6 · p. 10
- The authors propose held-out, continuously renewed evaluations using non-ergodic data, and predict that models will fail on token sequences absent from training material.They also note that evaluation regimes can themselves become optimization targets.Source: §6 · p. 10
- The authors argue that practical intelligence operates below consciousness, grounds cognitive intelligence, and is inaccessible to machine configuration because pertinent training data cannot be made available.They state that human-created mathematics and optimization algorithms cannot holistically model complex systems.Source: §6 · p. 10