PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?

David Ilić, Gilles E. Gignac

arXiv:2310.11616v3cs.CLcs.AI

TL;DR

The study asks whether LLM performance differences across tasks reflect shared ability dimensions, as human cognitive scores do. It analyzes benchmark-score correlations and factor structure, finding a positive manifold, a general factor, and a combined knowledge/reading-writing factor.

  • Problem

    LLMs differ in capacity across tasks, raising the question of whether their scores are positively intercorrelated and reflect underlying ability dimensions.

    Source: §1 · p. 1

  • Method

    The authors analyzed correlations across LLM benchmark scores and tested general and nested group factors using factor models.

    Source: §3.1 · p. 5, §3.2 · p. 5

  • Results

    The authors found a positive manifold and general ability factor, a combined Grw/Gkn group factor, and positive associations of parameter count with both factors.

    Source: §3.1 · p. 5, §4 · p. 6

  • Takeaways & Limitations

    The findings describe the structure of artificial-system capabilities and point to a balanced approach to optimizing LLM performance.

    Source: §5 · p. 9

    The test battery omitted spatial abilities and may have inflated the general factor's variance, so the findings are tentative and do not resolve intelligence versus achievement.

    Source: §4.1 · p. 7, §4.5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are advanced artificial intelligence (AI) systems that can perform a variety of tasks commonly found in human intelligence tests, such as defining words, performing calculations, and engaging in verbal reasoning. There are also substantial individual differences in LLM capacities. Given the consistent observation of a positive manifold and general intelligence factor in human samples, along with group-level factors (e.g., crystallized intelligence), we hypothesized that LLM test scores may also exhibit positive intercorrelations, which could potentially give rise to an artificial general ability (AGA) factor and one or more group-level factors. Based on a sample of 591 LLMs and scores from 12 tests aligned with fluid reasoning (Gf), domain-specific knowledge (Gkn), reading/writing (Grw), and quantitative knowledge (Gq), we found strong empirical evidence for a positive manifold and a general factor of ability. Additionally, we identified a combined Gkn/Grw group-level factor. Finally, the number of LLM parameters correlated positively with both general factor of ability and Gkn/Grw factor scores, although the effects showed diminishing returns. We interpreted our results to suggest that LLMs, like human cognitive abilities, may share a common underlying efficiency in processing information and solving problems, though whether LLMs manifest primarily achievement/expertise rather than intelligence remains to be determined. Finally, while models with greater numbers of parameters exhibit greater general cognitive-like abilities, akin to the connection between greater neuronal density and human general intelligence, other characteristics must also be involved.

1. Introduction

LLMs vary in task performance, prompting the question of whether their scores reflect shared ability dimensions rather than only specialized strengths. The study examines whether LLM scores form a positive manifold and general factor, and whether model parameters predict ability.

1. Introduction

  • LLMs differ in their task capacities, motivating tests of whether performance covaries across human-like cognitive abilities and reveals underlying dimensions.The study asks whether those dimensions include a general ability factor.Source: §1 · p. 1

1.1. General intelligence: human and artificial

  • The paper uses artificial general ability (AGA) rather than artificial general intelligence (AGI), leaving open whether LLM abilities reflect intelligence or achievement.The proposed AGI perspective treats a general factor as reflecting a positive manifold rather than simply a level of ability.Source: §1.1 · p. 2

1.2. Group-level factors of intelligence

  • The proposed benchmark dimensions include fluid reasoning, knowledge, reading and writing, quantitative knowledge, memory, and processing speed.The paper notes that LLM benchmarks can be classified under CHC dimensions, including commonsense reasoning, reading comprehension, and programming.Source: §1.2 · p. 2
  • The authors consider whether LLM benchmarks form general and group-level factors, which could support comparisons of overall performance and inform study of human intelligence.Human cognitive abilities are described as having a general factor alongside smaller group-level factors.Source: §1.2 · pp. 2–3

1.3. Number of parameters and artificial general ability

  • The authors predict that model parameter count will positively predict LLM performance, while prior evidence came from varying parameters within one model.They argue that larger parameter counts may support learning complex patterns, but prior work found increases were not linear across all tests.Source: §1.3 · p. 3

1.4. Summary and purpose

  • The study tests for positive correlations across LLM benchmarks, a general ability factor, CHC-aligned group factors, and associations with parameter count.The authors expect a positive manifold and a positive parameter-performance association.Source: §1.4 · p. 3

2. Method

The study curated LLM samples from public benchmark data, assembled measures spanning several ability domains, and analyzed their relationships using correlations and latent-variable models.

2.1. Sample

  • 591 usable models formed the reported subsample after deduplication based on model-name similarity and parameter count.The initial repository sample had 3,862 models; results from the other two subsamples were reported as highly similar.Source: §2.1 · pp. 3–4

2.2. Measures

  • Fluid reasoning measures included Hellaswag commonsense continuation, multi-step GSM8K arithmetic, and selected MMLU Elementary Mathematics items.Hellaswag presents four possible continuations; GSM8K items were selected to represent quantitative reasoning.Source: §2.2 · p. 4
  • MMLU items supplied domain-specific knowledge composites and quantitative knowledge measures, including statistics, algebra, and econometrics.The knowledge composites drew on subject-area subtests, while the quantitative composite measured mathematical knowledge.Source: §2.2 · p. 4
  • The benchmark table presents selected tests with theorized CHC categories, item counts, and score reliabilities.Reading and writing measures included history comprehension and Winogrande, which tests sentence completion using contextual cues.Source: Table 1 · p. 4, §2.2 · pp. 4–5

2.3. Data analysis

  • The analyses used Pearson correlations and latent-variable models, with maximum-likelihood estimation and fit indices to assess model fit.The theoretical model specified a second-order general factor and four first-order factors, with the general-factor variance fixed at 1.Source: §2.3 · p. 5

3. Results

Across 12 tests, LLM scores were positively intercorrelated and supported a strong general factor plus a weaker Gkn/Grw factor. Parameter counts were positively associated with both factors, with non-linear patterns in each relationship.

3.1. Descriptive statistics and inter-subtest pearson correlations

  • All 12 test correlations were positive (mean r = 0.73, range 0.35 to 0.99), with Gkn inter-correlations ranging from 0.98 to 0.99.Mean LLM test performance was 48.62% (range 18.09 to 74.27).Source: §3.1 · p. 5, Table 2 · p. 6

3.2. Confirmatory factor analysis

  • The bifactor model showed a strong general factor across all 12 tests (mean loading 0.81; 65.6% of variance) and a weaker, positive Gkn/Grw factor (mean λ = 0.50).The model included three correlated residuals and had acceptable close-fit; the unrestricted one- and two-factor solutions did not fit acceptably.Source: §3.2 · p. 5, Figure 1 · p. 6

3.3. Associations with number of parameters

  • Pearson r = 0.54 (95% CI: [0.48, 0.59]) between parameter count and AGA scores; gains were steepest below 10 billion parameters and more gradual at higher counts.The association was positive, and a cubic spline fit better than a linear model; only 10 models exceeded 80 billion parameters, so the scatter-plot analysis was restricted to below 80 billion.Source: §3.3 · pp. 5–6
  • Pearson r = 0.11 (95% CI: [0.05, 0.18]) between parameter count and Gkn/Grw scores; scores rose through about 10–15 billion parameters, then fluctuated or declined, with essentially no association beyond 45 billion.A cubic spline fit the relationship significantly better than a linear model.Source: §3.3 · p. 6

4. Discussion

The discussion reports a strong common ability pattern alongside a combined knowledge/reading-writing factor and a mathematics composite with the strongest AGA loading. Parameter count tracks both ability factors nonlinearly, while limited benchmark breadth and unresolved intelligence criteria qualify the interpretation.

4. Discussion

  • LLM scores showed a positive manifold and AGA factor, with one combined Grw/Gkn group factor; parameter count associated positively with both.Source: §4 · p. 6

4.4. Association with number of parameters

  • About 25% of AGA variance was accounted for by parameter count; gains were strongest from 100 M to 10B and tapered thereafter.Source: Figure 2 · p. 7, §4.4 · p. 8

4.1. Positive manifold and AGA

  • Mean inter-test correlation was 0.73, and the general factor explained 66% of variance, but the verbal-heavy battery leaves true AGI unresolved.Source: §4.1 · p. 7, §4.5 · p. 9

4.2. Group-level factors

  • Only one group-level factor emerged: combined Grw/Gkn, consistent with their content overlap; no independent Gf factor was observed using limited Gf measures.Source: §4.2 · pp. 7–8

4.3. Strongest indicator of AGA

  • The mathematics composite including algebra and statistics had the largest AGA loading, aligning with human findings that arithmetic can strongly indicate g.Source: §4.3 · p. 8

4.5. Limitations

  • The 12-test battery omitted important dimensions including visual processing, short-term memory, and processing speed, and its Gf measures were limited.Source: §4.5 · pp. 8–9
  • The conservative analysis retained 591 of 3,862 models, while parameter count was the sole statistically examined predictor because broader model data were unavailable.Source: §4.5 · p. 9
  • Whether AGA reflects intelligence or achievement remains open; the study did not assess awareness or self-improvement, and LLM generalization remains inconsistent.Source: §4.5 · p. 9

5. Conclusion

LLM capacities show a strong positive manifold, suggesting underlying general processes. Model parameters positively predict general and Gkn/Grw performance, but the effect may diminish without complementary characteristics.

5. Conclusion

  • LLM capacities show a strong positive manifold, with good performance on one task tending to accompany good performance on others.This pattern suggests underlying general processes.Source: §5 · p. 9
  • Model parameter count positively predicts general and Gkn/Grw performance, but its effect may diminish without complementary characteristics such as token count.The conclusion frames this as a reason to consider a balanced approach to model design.Source: §5 · p. 9
  • Further differential-psychology research on LLM performance may advance research on both artificial and human intelligence.Source: §5 · p. 9

CRediT authorship contribution statement

David Ilić's listed contributions span writing, software, methodology, investigation, formal analysis, data curation, and conceptualization. Gilles E. Gignac's listed contributions span writing, methodology, investigation, formal analysis, and conceptualization.

CRediT authorship contribution statement

Loading 2310.11616v3…