PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-scale factor analysis shows machine intelligence is only partially interpretable

Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad Ardi

arXiv:2609.36515v1cs.CLcs.AIq-bio.NC

TL;DR

Prior studies of language-model intelligence often used narrow, human-informed measures, leaving bottom-up evidence about how abilities are organized limited. The authors analyze benchmark performance with factor analysis and find that general intelligence is only partially interpretable: similar-content benchmarks need not cluster, and high-g benchmarks lack a common theme.

  • Problem

    Prior studies often used 6–23 narrow, human-informed measures, leaving bottom-up LLM-specific structure underexamined.

    Source: §1 · p. 2

  • Method

    The authors apply bottom-up, theory-free factor analysis to 13,251 evaluation scores across 1,618 models and 456 benchmarks, triangulating across treatments of missing data.

    Source: §2.3 · p. 3, §6 · p. 10

  • Results

    Across benchmarks, g explains at most 70.8% of variance in model performance; similar-content benchmarks need not cluster, and high-g benchmarks lack a common theme.

    Source: §4.1 · p. 6, §4.2 · p. 7, §4.3 · p. 7

  • Takeaways & Limitations

    If the mutualist account is correct, g is a summary scalar rather than a real, targetable ability, making a g-oriented research program untenable.

    Source: §5.3 · p. 9

    Factor analysis measures correlations and is compatible with, but does not establish, a causal model; a mutualist account is harder to rule out.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The $g$ factor is not dominated by any common theme, and there is a lack of evidence that it is well-proxied by standard "intelligence" benchmarks. Our findings go against current endeavors of defining, identifying, and targeting general intelligence as a tangible construct in language model development. This leaves the strategy of targeting a single conceptual ability without support, since the first-order abilities it would have to reach are often partially idiosyncratic and not identifiable in practice.

1 INTRODUCTION

Language-model research often assumes a content-free intelligence underlies and transfers across specific tasks, but that assumption has rarely been tested directly at broad scale. This study analyzes diverse benchmark scores and finds that factor patterns are only partially interpretable, with limited evidence that standard reasoning benchmarks proxy general intelligence.

1 INTRODUCTION

  • Language-model development often assumes a content-free capability underlies and transfers across specific tasks, an assumption borrowed from human fluid-intelligence theory.Fluid intelligence is defined as performing well on any task regardless of content.Source: §1 · p. 1
  • Prior studies tested machine intelligence using only 6–23 benchmarks, often narrowly focused on mathematics, academic knowledge, and common-sense reasoning.Such sampling can conflate the extracted factor with model scale, and prior work often imposed human-informed structures.Source: §1 · p. 2
  • Factor analysis of 13,251 evaluation scores across 1,618 models and 456 benchmarks finds partially interpretable, often incoherent factor patterns.The benchmark pool includes diverse tasks, including those outside the mainstream.Source: §1 · p. 2
  • The study finds limited evidence that commonly targeted reasoning benchmarks proxy general intelligence; highest loadings more often belong to miscellaneous tasks without a common theme.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Psychometric research infers latent structure from broad measurements, while machine-intelligence research often assumes a causal general factor explains benchmark correlations and transfer. This section motivates a bottom-up test of that structure and reviews how prior studies differ in scale and method.

2.1 THE PSYCHOMETRIC PARADIGM

  • Psychometrics infers latent structure by measuring observable behaviors broadly and applying dimension reduction; intelligence research identifies a higher-order g factor across school-subject scores.Source: §2.1 · p. 2

2.2 CAUSAL ASSUMPTIONS IN MACHINE INTELLIGENCE

  • The assumed hierarchy places g above task-specific latent factors, which in turn influence performance, with no causal path back to g.The field often expects improving reasoning, mathematics, or coding to improve g and thereby transfer to other tasks.Source: §2.2 · pp. 2–3

2.3 INTERPRETING STRUCTURAL PATTERNS

  • The study interprets factor patterns without prespecifying what a general factor should look like, using bottom-up analysis across the benchmarks the field uses.It argues that human theories may not fit neural networks’ different cognitive processes.Source: §2.3 · p. 3

2.4 RELATED WORK

  • Prior benchmark factor studies were smaller or confirmatory; PCA does not separate systematic from error variance, and earlier EFA studies lacked higher-order analysis or used narrow batteries.Item-response methods require per-item results, while full-information maximum likelihood requires overlapping observed models across benchmark pairs.Source: §2.4 · pp. 3–4

3 METHODOLOGY

The study analyzes a highly sparse corpus of language-model benchmark scores by combining data collection, matrix densification, imputation, hierarchical factor analysis, and label-cohesion tests. It triangulates across methods with differing assumptions to identify recurring structure.

3 METHODOLOGY

  • The raw 1,618 × 456 benchmark matrix is about 1.8% dense and missing not at random, so the analysis triangulates across densifiers and imputers.Source: §3 · p. 4

3.1 DATA COLLECTION

  • The corpus includes generative language models evaluated on text-only benchmarks, excluding encoder-only classifiers, narrow task-specific systems, and undocumented community uploads.Benchmarks enter if they have at least one in-scope result row.Source: §3.1 · p. 4

3.2 DATA PROCESSING

  • Processing averages duplicate model–benchmark scores, collapses near-duplicate model variants, selects the widest comparable metric, and peels matrices to 10% density using three axis-prioritizing strategies.The C, R, and S densifiers retain broad benchmark coverage, broad model coverage, or neither axis preferentially.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.3 FACTOR ANALYSIS

  • Exploratory factor analysis uses minimum residual estimation and oblique promax rotation, followed by a Schmid-Leiman bifactor transformation to obtain a higher-order factor.The ωh coefficient quantifies variance explained by g; parallel analysis selects dimensions, capped at 20 factors.Source: §3.3 · p. 5

3.4 LABEL COHESION ANALYSIS

  • Label cohesion compares within-label distances in factor-loading space with coverage-matched random sets, using permutation tests and false-discovery-rate correction.Labels with fewer than four members or the entire benchmark set are not scored; Benjamini–Hochberg controls FDR at q = 0.05.Source: §3.4 · p. 6

4 RESULTS

Across the factor analyses, the estimated share of performance variance attributed to general intelligence varies widely, and same-domain benchmarks do not consistently cluster. The benchmarks most strongly associated with the general factor are not dominated by standard intelligence tests or a common subject area.

4.1 VARIANCE EXPLAINED BY g

  • At most, a universally causal g factor accounts for 70.8% of variance in model performance; the best-performing imputer, SoftImpute on S Std., yields 25.7%.Table 3 reports point summaries of the factor analyses.Source: §4.1 · p. 6, Table 3 · p. 6
  • Newer models score higher on g, but the variance it explains shows no release-year trend; leave-one-benchmark changes in ωh correlate only weakly with benchmark frequency (mean |r| = 0.18).Source: §4.1 · p. 6

4.2 BENCHMARK CLUSTERS

  • Across nearly all imputations, same-subject benchmarks only occasionally cluster; only code shows weak cohesion, while other labels show no structure distinguishable from chance.Narrow labels such as finance are cohesive but have very few members.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 BENCHMARKS’ g-CENTRALITY

  • The strongest g proxies span creativity, legal use cases, and emotional intelligence, while standard tests such as ARC and GPQA_Diamond rank near the middle.The results provide no evidence that g resembles abstract reasoning or that reasoning, mathematics, and coding benchmarks are the best proxies.Source: §4.3 · p. 7, Table 5 · p. 8

5 DISCUSSION

The discussion argues that LLM abilities lack a coherent higher-order structure: content-similar tasks need not cluster, and a general factor cannot be coherently approximated. It considers possible sources of benchmark correlations and notes that factor analysis alone cannot establish a causal model.

5.1 UNINTERPRETABLE HIERARCHICAL STRUCTURE

  • LLM abilities appear more horizontal than hierarchical: content-similar tasks need not cluster, and a substantially effective g factor cannot be coherently approximated.The discussion contrasts this pattern with human theories that assign fluid intelligence higher-order causal effects.Source: §5.1 · p. 9
  • A proposed test of causal generalization fine-tunes a model on one task, then checks whether performance improves on other tasks.For example, fine-tune on mathematics and measure coding before and after; an improvement would empirically verify a causal mechanism.Source: §5.1 · p. 9
  • Without causal generalization, improving one task family offers no blanket improvement; broad capability would require training on all relevant tasks.The discussion says this could demand an increasingly expansive dataset covering the universe of possible tasks.Source: §5.1 · p. 9

5.2 WHY DO ABILITIES CORRELATE?

  • The proposed explanation for benchmark correlations is that they may arise from unintuitive conditional token probabilities rather than human-like analogies.The discussion notes that input similarity identifies the best transfer source for at most 2 of 8 target tasks, while transfer is better predicted by task-induced gradients than content.Source: §5.2 · p. 9

5.3 ALTERNATIVE TO A LATENT FACTOR VIEW

  • Factor analysis reveals correlations but is only compatible with a causal model, and mutualist processes can produce a positive manifold without a common cause.The discussion therefore treats mutualism as difficult to rule out.Source: §5.3 · p. 9
  • Mutualism models abilities as mutually facilitating, but it leaves inter-task correlations unintuitive and does not explain their uninterpretability.If correct, this view makes g a summary scalar rather than a real, targetable ability.Source: §5.3 · p. 9

A LIMITATIONS

The analysis relies on published scores without controlling their evaluation conditions, and some completion methods were not tested across the full design. The authors also describe how language models assisted with research and writing under author review.

A LIMITATIONS

  • Published scores were not independently evaluated, and undocumented differences in evaluation setup were not corrected.When sources disagreed on an evaluation, their scores were averaged.Source: A LIMITATIONS · p. 30
  • Metric direction was recorded but not applied, affecting benchmark distances and the sign of correlations assigned by mean filling.The authors note that orienting every column before analysis would have been cleaner.Source: A LIMITATIONS · p. 30
  • The reported results cover only the Table 3 imputers; some correlation-level estimators lack full design coverage, and regularized EM-PCA was excluded.Its built-in cross-validation was intractable at this matrix size.Source: A LIMITATIONS · p. 30

B THE USE OF LARGE LANGUAGE MODELS

  • Language models assisted with methodological exploration, drafting and editing, pipeline code, and extraction, while authors reviewed the work and verified numerical output.The authors state that they take responsibility for the data and the paper’s claims, numbers, and citations.Source: B THE USE OF LARGE LANGUAGE MODELS · p. 30

C.1 DATA SOURCES

This section describes how evaluation records were collected and processed into a corpus, including source verification, scope decisions, deduplication, and score handling. It also records the limits of release-date evidence and several remaining score-scale defects.

C.1 DATA SOURCES

  • The corpus draws published evaluation records from four source tiers, ranging from curated suites and aggregators to benchmark-specific leaderboards and primary evaluation papers.Extraction routes included archived leaderboard tables, staged HELM data, papers, repositories, and dataset cards.Source: §C.1 · pp. 30–31, §C.2 · p. 31

C.3 INFERRED FIELDS

  • When records omitted fields, the authors inferred closed-model platforms from vendor APIs and recorded zero temperature for documented EleutherAI-harness evaluations.The inference rules were limited to those two fields.Source: §C.3 · p. 31

C.4 RELEASE DATES

  • Direct sources date 78% of models, but benchmark dates are weaker; repository creation can predate release, and single-lookup dates may still be early.The authors report that weaker-tier corrections moved 119 model dates, 85 to a later month.Source: §C.4 · pp. 31–32

C.5.1 MODELS

  • Included models had to generate text from arbitrary prompts; the corpus excluded narrow or non-deployable systems, while recording setup variants on result rows rather than as separate models.Multimodal models were included when their LLM backbone still handled arbitrary text prompts.Source: §C.5.1 · p. 32

C.5.2 BENCHMARKS

  • Benchmarks were restricted to text-only scope, while translation duplicates were removed or merged and independently sourced multilingual benchmarks were retained.Cross-language aggregates were kept when they did not duplicate an existing column and measured transfer across languages.Source: §C.5.2 · p. 32, §C.5.3 · p. 32

C.6 DUPLICATE DETECTION AND INTEGRITY CHECKS

  • Duplicate rows had to match on model, benchmark, metric, setup, source, model identifier, and language; rows differing on any field were retained and aggregated by model-benchmark pair.Integrity checks also verified row references and flagged benchmarks with fewer than five rows.Source: §C.6 · p. 32

C.7 SCORE NORMALIZATION

  • Scores were generally normalized to 0–100, while 92 multi-metric benchmarks kept the metric covering the most models, subject to overrides and tie-break rules.Accuracy and exact match remained separate because no model was scored both ways on benchmarks carrying both.Source: §C.7 · p. 32, §C.8 · p. 33

C.9 REMAINING COLUMN DEFECTS

  • The authors dropped seven raw GPQA scores because their ranges did not overlap normalized scores, and removed ELEPHANT because its metric field contained configurations.WildBench, SEA-Exam, and MultiPL-E were left unchanged because a score gap alone did not establish a scale conflict.Source: §C.9 · p. 33

D SCORE-REDUNDANCY PRUNING

The authors audited benchmark redundancy using score correlations, then collapsed model identifiers and prepared matrices for analysis. These steps left 456 benchmarks, 1,618 models, and 13,251 result rows before densification.

D SCORE-REDUNDANCY PRUNING

  • Correlation auditing kept the paper-sourced MultiLoKo across-language aggregate, dropped all 31 per-language identifiers, and identified a spliced score series as a scraping artifact; ThaiExam’s ONET/IC/A-Level cluster was r = 0.92–0.95.TGAT and TPAT1 correlated weakly with that cluster (r = 0.70–0.88).Source: D SCORE-REDUNDANCY PRUNING · p. 34
  • Other redundancy decisions kept LiveCodeBench’s aggregate over six versions and TwitterAAE’s parent over dialect splits, while dropping duplicate Kaggle GPQA variants.LiveCodeBench versions had mean pairwise r = 0.995; TwitterAAE splits correlated r = 0.993–0.999 with one another and the parent.Source: D SCORE-REDUNDANCY PRUNING · p. 33
  • The final corpus contained 456 benchmarks, 1,618 models, and 13,251 result rows after redundancy pruning and subsequent data filters.The redundancy pass deliberately left two of seven audited families partly or wholly intact where correlations did not justify removal.Source: D SCORE-REDUNDANCY PRUNING · p. 34

E.1 STANDARD (VARIANT-LEVEL)

  • Variant-level identity collapse preserves source identifiers alongside canonical models, averages repeated evaluations, and standardizes names by stripping prefixes, dates, generic tokens, and context lengths.Named model tiers and parameter counts are retained, with safeguards against confusing version numbers for parameter counts.Source: E MODEL-IDENTITY COLLAPSE · p. 34, §E.1 · pp. 34–35

E.2 AGGRESSIVE (FAMILY-LEVEL)

  • Aggressive family-level collapse takes the first alphabetic family token, or the first alphabetic token from a cleaned identifier when family data is unavailable.Source: §E.2 · p. 35

E.3 POST-COLLAPSE FILTERING

  • After post-collapse filtering, the raw matrices were 1,266 × 404 at 2.2% density and 334 × 380 at 3.5%, derived from 2,183 source-level identifiers.Source: §E.3 · p. 35

F DENSIFICATION ALGORITHM

  • At a target density of 0.10, the three peeling strategies prioritize dropping sparse benchmarks, sparse models, or whichever row or column has the lowest fill rate.C is column-primary, R row-primary, and S symmetric.Source: F DENSIFICATION ALGORITHM · p. 35
  • After peeling, rows or columns below the observation floor are iteratively removed, then columns with fewer than two observations or zero variance are dropped.The analyzed tables used a floor of three observations.Source: F DENSIFICATION ALGORITHM · p. 35

G IMPUTATION METHODS

The imputation methods share an interface that returns a completed matrix and held-out error measures, allowing the factor-analysis stage to remain the same across methods. One-sided matrix completion instead recovers benchmark-space covariance information and uses it to predict held-out cells.

G IMPUTATION METHODS

G.1 ONE-SIDED MATRIX COMPLETION

  • One-sided matrix completion estimates benchmark-space covariance from pairs of observed standardized scores, fitting a low-rank covariance representation to those products.Its premise is that benchmark-space singular vectors may remain recoverable even when observations are too sparse to complete cells.Source: §G.1 · pp. 36–37
  • For held-out cells, the method uses a conditional-Gaussian best linear predictor in the recovered factor space rather than inverting a rank-deficient covariance block.The hold-out masking ensures each row has at least two observations.Source: §G.1 · p. 37

H.1 METRICS

The paper validates sparse-score imputations, extracts general and domain factors, and embeds benchmarks using distances between their factor-loading profiles. It then compares benchmark labels against those distances.

H.1 METRICS

  • Imputation evaluation masks about 20% of observed cells with a column-stratified split, retains at least two training observations per benchmark, and filters fits with held-out R2 < 0.2.Columns are standardized using training-cell moments only; each benchmark with more than two observations contributes a held-out cell.Source: §H.1 · p. 37

H.2 CORRELATION-LEVEL IMPUTERS

  • Correlation-level imputers predict each held-out cell from the row’s remaining observed cells using a conditional-Gaussian best linear predictor based on recovered correlations.The prediction is a weighted combination of the row’s other cells, with weights fixed by the covariance structure.Source: §H.2 · p. 37

I.1 ESTIMATOR

  • Exploratory factor analysis uses minimum-residual estimation, with promax rotation for solutions extracting multiple factors and a unity-diagonal retry after hard errors.The analysis uses the R package psych.Source: §I.1 · p. 37

I.2 FACTOR COUNT

  • Horn’s parallel analysis compares observed eigenvalues with random-matrix eigenvalues, while factor counts are capped at 20 and reduced as needed for feasible, successful fits.The random comparison uses the 95th percentile from 100 standard-normal matrices; rank-deficient or high-dimensional matrices also constrain the count.Source: §I.2 · pp. 37–38

I.3 BIFACTOR DECOMPOSITION

  • The bifactor step applies the Schmid-Leiman transformation and records general and specific benchmark loadings, ωh, variance explained, and inter-factor correlations.It uses EFA without sign flipping.Source: §I.3 · p. 38

J.1 COMPOSITE DISTANCING

  • Composite benchmark distances average pairwise cosine distances between loading vectors across solutions where both benchmarks appear, making distances comparable across factor rotations and signs.The resulting distance matrix is embedded in two dimensions with UMAP; missing pairwise distances are filled with a benchmark-specific mean or, when needed, the global mean.Source: §J.1 · p. 38, J BENCHMARK EMBEDDING · p. 38, §J.2 · p. 38

J.3 LABELS

  • Benchmark labels cover subject, task, and language as independent, non-exclusive axes, and are curated from benchmark documentation independently of the distance and cohesion analyses.UMAP receives the precomputed composite distances; labels may be multiple or nested.Source: §J.3 · p. 38

J.4 UMAP LIMITATIONS

Across release-year groups, factor structure shifts without a consistent direction, while newer models score higher on the pooled general factor. Shared imputation may pull group structures toward the pooled solution.

J.4 UMAP LIMITATIONS

  • UMAP preserves neither density nor global distance, so apparent tightness and separation in the two-dimensional plot are only visual summaries; clustering claims rely on the composite distance matrix.The release-date analyses are exploratory and use main-analysis matrices at 10% target density.Source: §J.4 · p. 38, K RELEASE-DATE ANALYSIS · p. 39

K.1 SETUP

  • The analysis splits models into four release-year groups and compares each group’s factor solution with size-matched random subsets, but shared pre-split imputation can pull groups toward the pooled structure.This can understate differences when benchmark correlations vary by year; up to 42 of 124 benchmarks in the oldest S group had no observed score there.Source: §K.1 · p. 39, Table A4 · p. 39

K.2 MODELS BY RELEASE YEAR

  • 8 of 24 release-year fits fall outside the 90% random-subset band, and median congruence with the pooled general factor is 0.82 versus 0.97 for random subsets; shifts have no consistent direction.The imputers disagree on which year has the strongest general factor, with no trend by release year.Source: §K.2 · p. 39, Figure A1 · p. 40
  • Newer models score higher on the pooled general factor, but it explains no more variance in later generations, providing no sign of convergence on more general intelligence.In four of six solutions, mean scores rise from between −1.5 and −1.0 for models from 2022 or earlier to between 0.0 and 1.2 for models from 2025 or later.Source: §K.2 · p. 39, Table A5 · p. 40

K.3 WITHOUT IMPUTATION

  • With only six benchmarks observed in every year group, first-eigenvalue variance shares are 0.43, 0.71, 0.72, and 0.57 from the oldest through newest group, and the authors read no trend into this check.The oldest models are near benchmark floors, and the newest group has only 20 models with an interval overlapping 2023’s.Source: §K.3 · p. 40

K.4 BENCHMARK RELEASE DATE

Benchmark release year does not reliably predict absolute general-factor loading, although same-era benchmarks tend to sit closer in loading space. Much of that proximity is associated with shared model coverage.

K.4 BENCHMARK RELEASE DATE

  • For observed benchmark pairs, S correlations are 0.36 when release years differ by at least four years versus 0.50 within a year; C correlations are 0.51 in both groups.The authors conclude most of the era effect comes from missingness patterns, while the remaining part matches the year-to-year model shift.Source: §K.4 · p. 41
  • In 9 of 10 solutions, same-era benchmarks have smaller mean within-era loading-vector distances than under permuted era labels (z < −1.96).Release year predicts absolute general-factor loading significantly in only 3 of 10 solutions, twice positively and once negatively.Source: §K.4 · p. 40, Table A6 · p. 41
  • Co-observation has the larger standardized rank-regression coefficient in all 10 solutions (0.20 to 0.54 versus 0.01 to 0.20 for year gap), while year gap remains significant in 6 of 10.Benchmark year correlates with the mean year of models taking it at 0.67 on S and 0.68 on C.Source: §K.4 · pp. 40–41, Table A6 · p. 41

L BENCHMARK g-RANKINGS

Benchmark frequency is associated with g loadings, and within most factor cells more frequently measured benchmarks rank higher. The section adjusts benchmark rankings for that within-cell frequency association before comparing results across cells.

L.1 FREQUENCY AND g-RANKING CORRELATIONS

  • Naively, benchmark frequency correlates with g loadings (unsigned average r = 0.3432), with some dataset-imputation combinations reaching ~0.5.The correlations are dispersed, motivating frequency adjustment when identifying benchmarks that proxy g.Source: §L.1 · p. 41, Table A7 · p. 41

L.2 FREQUENCY ADJUSTMENT

  • Benchmarks are ranked by absolute g loading within each method–dataset cell, then ranks are normalized from 0 at the top to 1 at the bottom.A benchmark’s raw score averages its normalized ranks across cells where it appears; benchmarks appearing in fewer than two cells are excluded.Source: §L.2 · p. 42
  • Within most cells, frequently measured benchmarks have higher |g_i| and better normalized ranks, so their raw ranking may partly reflect measurement frequency.The adjustment removes the linear within-cell rank–frequency association before averaging ranks across cells.Source: §L.2 · pp. 42–43, Table A8 · p. 43

L.3 DIAGNOSTICS

  • Frequency–rank associations vary in direction across cells, so the pooled raw correlation is small despite nontrivial within-cell associations.The primary ordering therefore uses cellwise frequency-adjusted average normalized ranks; raw average ranks are retained for comparison, and the orderings agree closely.Source: §L.3 · pp. 43–44, Table A9 · p. 43

M LABEL COHESION RESULTS

Benchmark clusters are weakly cohesive across the reported labels. The subject-label results are strongest for two specific categories, while language labels also show weak cohesion.

M LABEL COHESION RESULTS

N IMPUTATION RESULTS

Only 20 imputation combinations pass the R2 ≥ 0.2 gate, and USVT yields no valid solutions or crashes mid-estimation. A leave-one-covariate-out analysis finds a modest association between benchmark frequency and changes in ωh.

N IMPUTATION RESULTS

  • Only 20 imputation combinations pass the R2 ≥ 0.2 gate, and USVT is the only method with no valid solutions or crashes mid-estimation.The table reports held-out RMSE, R2, and selected configuration for dataset–imputer combinations.Source: N IMPUTATION RESULTS · pp. 45–46, Table A13 · p. 46

O OMEGA SENSITIVITY

  • The mean absolute correlation between benchmark frequency and LOCO ωh deltas is r = 0.184, while the signed average is r = 0.054.The authors note that the signed average may be small because positive and negative correlations cancel.Source: O OMEGA SENSITIVITY · p. 47, Table A14 · p. 47
  • LOCO factor analyses leave out each benchmark in turn and use the change in ωh to measure sensitivity.The analysis is run for each valid dataset.Source: O OMEGA SENSITIVITY · p. 47

P FULL g-RANKINGS

This section presents benchmark-level g-factor rankings and compares how benchmarks from the same domain are distributed across imputation strategies. It also includes common-subject UMAP plots across datasets and imputation approaches.

P FULL g-RANKINGS

  • Ranks 1–12 are bhasa, mtrag, creativityprism, eqbench, mceval, pwc_svamp, pwc_drop_test, ProphetArena, dialogbench, pwc_piqa, pwc_timequestions, and tablebench_data_analysis.The table defines rank 0 as first and rank 1 as last; these entries are ordered by residualized average normalized rank order.Source: Table A15 · p. 48, P FULL g-RANKINGS · p. 48
  • The excerpted ranking places quac at 101, mbpp at 119, mmlu_pro at 125, math500 at 174, and chatbot_arena at 192, with listed values -.095, -.082, -.073, -.017, and +.001, respectively.Source: P FULL g-RANKINGS · pp. 50–52
  • The excerpt shows benchmark ranks 128–141, including mmlu at 129 and gpqa_diamond at 139; Table A15 lists all 380 benchmarks sorted by rank order.The ranking method and its diagnostics are described in Appendix L.Source: P FULL g-RANKINGS · pp. 48, 51
  • The full g-rankings list continues from benchmark entry 195, ewok_social_properties, through entry 380, pwc_conala.The supplied entries show benchmark-level rankings but do not provide the table's column headings.Source: P FULL g-RANKINGS · pp. 52, 57

Q COMMON-SUBJECT UMAP PLOTS

Related papers

Loading 2609.36515v1…