PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Compression of Vision-Language Model Benchmarks

Hyunjong Ok, Seunggu Kang, Jaeho Lee

arXiv:2609.37515v1cs.LGcs.AIcs.CLcs.CV

TL;DR

VLM benchmark suites are expanding, making evaluation costly, while benchmark compression for VLMs remains underexplored. PRIMEBench hierarchically prunes benchmarks and items using vision-aware selection. Its released suite removes over 97% of items while closely preserving rankings, and VAW performs best at 5% retention on held-out models.

  • Problem

    VLM evaluation spans expanding capabilities and is costly, while prior ranking-preserving benchmark compression has been studied mainly for language models and remains underexplored for VLMs.

    Source: §1 · p. 1

  • Method

    PRIMEBench hierarchically cleans items, selects category representatives, prunes items with VAW, and prunes category counts, combining inter-model variance with an embedding-based vision-dependence prior.

    Source: §1 · pp. 1–2

  • Results

    The released suite removes over 97% of items while closely preserving model rankings, and VAW performs best at 5% retention on held-out models.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The results support hierarchical compression as a way to reduce broad VLM evaluation cost while retaining ranking information for comparing models.

    Source: §8 · p. 10

    With 10 to 20 held-out models per split, split-to-split variation is comparable to gaps between stronger selectors, so results are averaged over 20 splits rather than relying on one split.

    Source: J LIMITATIONS · p. 38

Abstract

from arXiv · show

Thorough evaluation of vision-language models (VLMs) has become prohibitively expensive, as benchmarks span an ever-broader spectrum of capabilities and new models arrive at a relentless pace. Benchmark compression methods that preserve model rankings at a fraction of the cost are well studied for language models, but for VLMs the question remains under-explored. We present PRIMEBench (Pruning Redundant Items for Multimodal Evaluation), a vision-aware hierarchical benchmark compression framework that substantially reduces evaluation cost while preserving model rankings. This hierarchical framework operates in four stages: data cleaning to remove items answerable without the image and all-correct items, category representative selection to pick one benchmark per capability category, item pruning with Vision-Aware Variance (VAW), and category-count pruning. VAW combines inter-model variance with a vision-dependence score computed from multimodal embeddings alone, while encouraging coverage of diverse items within each benchmark. On models held out from item selection, it has the highest mean fidelity at the released 5% retention. The hierarchical design lets practitioners stop at any stage to match their compute budget; the released suite removes over 97% of items while preserving model rankings. Beyond compression, our analyses show how VLM evaluation behaves as model panels grow and evolve, providing guidance for designing future benchmarks that are more efficient, robust to model turnover, and explicit about the limits of evaluation-side pruning.

1 INTRODUCTION

VLM evaluation costs are rising as benchmark suites expand, while VLM benchmark compression remains underexplored. PRIMEBench addresses this with a hierarchical framework that compresses benchmarks and items while aiming to preserve model rankings.

1 INTRODUCTION

  • The growing breadth of VLM capabilities and recurring model releases make full-suite evaluation costly, reaching hundreds of GPU-hours per model.The costs compound as new models must be evaluated against an expanding suite.Source: §1 · p. 1
  • Prior compression work mainly targets within-benchmark redundancy, and large-scale validation of benchmark-level compression for VLMs is absent from the cited studies.The paper frames VLM benchmark compression as an underexplored problem.Source: §1 · p. 1
  • PRIMEBench addresses redundancy across benchmarks and within benchmarks by selecting category representatives and pruning items while preserving full-suite rankings.Benchmarks within a capability category induce correlated model rankings; items solved by all models or failed by all provide little separation signal, though all-wrong items may diagnose future models.Source: §1 · pp. 1–2
  • Its four stages clean data, select category representatives, prune items with Vision-Aware Variance, and prune the number of categories.The pipeline can be stopped at different stages to match a compute budget.Source: §1 · p. 2
  • VAW combines inter-model variance with a label-free vision-dependence prior and coverage cells, with the prior's effective weight shrinking as the model panel grows.The prior is computed from embeddings alone, and items are selected within each benchmark's share of the category budget.Source: §1 · p. 2

2 RELATED WORK

Prior benchmark-compression research primarily selects items for text-only LLMs, while multimodal work has not jointly compressed benchmarks and items in a released VLM suite. PRIMEBench combines both compression levels and validates them on VLM panels.

2 RELATED WORK

  • The comparison table organizes selected methods by modality, compression level, and largest validation panel; PRIMEBench combines benchmark- and item-level compression.The caption reports validation on 30 VLMs for item selection and 231 VLMs for benchmark-level ranking preservation.Source: Table 1 · p. 3
  • Earlier compression methods mostly target text-only LLMs, using approaches such as IRT anchors, disagreement ranking, sparse-anchor optimization, rank-correlation prediction, and embedding coverage.Multimodal prior work includes multimodal IRT and point-biserial item discrimination for VLM benchmarks.Source: §2 · p. 3
  • The cited multimodal methods do not release a suite jointly compressed across benchmark and item axes, and DatBench's correctness-only criterion does not account for visual dependence.PRIMEBench combines the two axes and checks benchmark-level ranking preservation on a large external panel.Source: §2 · p. 3

3 PROBLEM STATEMENT

The paper formulates benchmark compression as selecting benchmarks and items within them so that the resulting model ranking remains close to the full-suite ranking.

3 PROBLEM STATEMENT

  • The evaluation catalogue contains M models and B benchmarks, each benchmark comprising a set of items with model-level item scores.Scores may be binary correctness values or continuous values, such as judge outputs.Source: §3 · p. 3
  • A model's full-suite score is its average score across benchmarks, and the full ranking orders models by that mean score.Benchmark scores are themselves averages over items.Source: §3 · p. 3
  • Compression selects a benchmark subset and item subsets within those benchmarks to make the compressed ranking close to the full ranking.The formulation uses k for the number of retained benchmarks, r for item retention, and Spearman correlation ρ to measure ranking similarity.Source: §3 · pp. 3–4

4 METHOD

PRIMEBench compresses VLM evaluation in four sequential stages: cleaning items, choosing category representatives, pruning items with VAW, and reducing the category count.

4.1 STAGE 0: DATA CLEANING

  • Stage 0 removes all-correct items and items answerable from text alone, producing a pool intended to measure VLM-specific capability.All-wrong items are retained because later model generations may solve them; text-only solvability is filtered at a threshold of 2/3 when image-free scores are available.Source: §4.1 · p. 4

4.2 STAGE 1: CATEGORY REPRESENTATIVE SELECTION

  • Stage 1 groups benchmarks into 18 capability categories and selects the benchmark whose normalized score vector correlates best with the category mean.For categories with weakly correlated representatives, the method supplements the representative with variance-ranked items from other category benchmarks.Source: §4.2 · p. 4

4.3 STAGE 2: ITEM PRUNING WITH VISION-AWARE VARIANCE

  • Stage 2 scores items for both inter-model discrimination and vision dependence, then selects high-scoring items within coverage cells to discourage near-duplicates.The vision-dependence score measures how visual input shifts the text-only embedding and is label-free; the vision-text embeddings also define the coverage cells.Source: §4.3 · p. 5
  • VAW's vision-dependence tilt fades with panel size because its effective weight shrinks at the same O(M^-1/2) rate as the variance-estimation error.For a common M, rescaling does not change item rankings; α is fixed at 0.187, while α = 0 yields variance alone.Source: §4.3 · p. 5

4.4 STAGE 3: CATEGORY-COUNT PRUNING

  • Stage 3 selects a category subset by maximizing Spearman correlation with the ranking from all Stage 2-pruned category pools.The released k=5 category choice matches the top five singular values, which the figure caption says capture 97.6% of spectral energy.Source: §4.4 · p. 6, Figure 2 · p. 6

5 EXPERIMENTAL SETUP

The experiments evaluate benchmark and item compression on a 30-model panel and 43 benchmarks across 18 capability categories, with ranking fidelity as the primary metric. Stage 2 comparisons use held-out models, while end-to-end compression is evaluated against rankings from all Stage 0-cleaned benchmarks.

Evaluation data

  • The study evaluates 30 vision–language and omni models across 43 benchmarks organised into 18 capability categories.Stage 2 uses vision-text embeddings from Qwen3-VL-Embedding-8B, which is not among the evaluated models.Source: §5 · p. 6

Baselines

  • Stage 2 compares VAW with five selectors and Variance at retentions from 1% to 50%, using common category pools.DISCO's selected items are scored by their mean for a consistent comparison.Source: §5 · p. 6

Metrics and protocol

  • Fidelity is Spearman rank correlation between compressed- and full-pool rankings, reported over 20 balanced 15/15 splits and a chronological split.Stage 2 selection uses build models and is evaluated on held-out models; end-to-end rankings are compared with all Stage 0-cleaned benchmarks.Source: §5 · pp. 6–7
  • Figure 3 evaluates mean held-out Spearman ρ over 20 random splits, with selection and evaluation performed on separate halves of the model panel.Stage 2 comparisons use the 18 PRIMEBench category pools.Source: Figure 3 · p. 7, §5 · p. 7

6 RESULTS

The four-stage pipeline substantially reduces the evaluation suite while preserving model rankings. The released suite retains high end-to-end fidelity, including under held-out model evaluation and external leaderboard validation.

End-to-end compression

  • From 386,324 items across 43 benchmarks, the released suite retains 9,377 items and reaches ρ=0.959 against the Stage 0-cleaned reference.Table 2 presents the Stage 0–3 pipeline and defines the released-suite correlation relative to rankings over all Stage 0-cleaned benchmarks.Source: §6 · p. 7

Stages 0–1

  • Stage 0 retains 270,408 items after cleaning, and Stage 1's representative benchmarks reach median per-category ρ=0.951.Stage 0 removes visually solvable and all-correct items while retaining all-wrong items; Stage 1 selects the benchmark most correlated with each category mean.Source: §6 · p. 7

Stage 2

  • On held-out models, VAW has the highest fidelity at retentions of 5–50%, while AnchorPoints leads at retentions of 2% or less.Benchmark shares and coverage cells account for most of VAW's gain over plain Variance; embedding-based cells outperform random partitions.Source: §6 · pp. 7–8

Stage 3 and generalization

  • At k=5, Stage 3 reaches ρ=0.988 against the 18-category pruned reference; the held-out end-to-end pipeline reaches ρ=0.892 against the Stage 0 reference.The latter is within 0.01 of Stage 3 without item pruning (0.901).Source: §6 · p. 8

External validation

  • On 231 OpenVLM models, Stage 3 at k=5 reaches ρ=0.988 against the ranking from 14 overlapping benchmarks.The 14 benchmarks overlap with PRIMEBench; the models were scored on at least half of them.Source: §6 · p. 8

7 DISCUSSION

The discussion examines why VAW works, how compression behaves as model panels grow, and what the selected suite can and cannot support. It reports stable benchmark selection over the dated range studied, while treating future-model separation and training-data pruning cautiously.

Why VAW works

  • At the tested panel size, vision-text coverage cells raise held-out ρ from 0.908 to 0.925, while benchmark shares alone yield 0.908 versus Variance at 0.906.The vision-dependence tilt is annealed with panel size; at r=2%, its cross-seed-selected version improves ρ over variance-only selection at every tested size from M=4 to 192.Source: §7 · p. 8

Shelf life

  • A frozen five-benchmark selection remains at ρ≈0.99 as the dated OpenVLM panel grows to 203 models, ranking at or above the 99.9th percentile among five-benchmark subsets.The comparison covers models with known release dates and stays within 0.0004 of the best subset refitted on each enlarged panel.Source: §7 · p. 8

Capability Frontier

  • The Capability Frontier contains 1,653 current-family hard items that still separate the oldest and newest Qwen generations, but future-model separation is untested.The authors treat it as a diagnostic of the current panel rather than a forecast.Source: §7 · pp. 8–9

Beyond ranking

  • Left-out benchmark prediction error falls as the number of models used to fit the predictor grows, though the analysis uses full benchmark scores rather than the released suite.The paper notes that estimates may improve as sparse cross-benchmark leaderboard coverage grows.Source: §7 · p. 9

Checkpoint ranking

  • At r=5%, VAW raises ranking ρ over Variance by 0.082 on average across eight stable LoRA variant sets, but the comparison is descriptive because variants from one run are not independent draws.VAW leads in six sets, trails on one, and ties on one; the caption identifies the plotted quantity as variant-ranking ρ.Source: §7 · p. 9, Figure 4 · p. 9

Training-data pruning

  • No evaluation-side selector beats random training-subset sampling beyond the seed spread, and Variance trails it by 10.5–20.2 test-accuracy points at r≤30%.The authors distinguish training-data selection from evaluation compression and recommend uniform random sampling as the safer training-data baseline.Source: §7 · p. 9

8 CONCLUSION

PRIMEBench compresses VLM evaluation across benchmarks and items, reducing the original 43-benchmark suite by over 97% while closely preserving model rankings. The evaluation panel comprises 30 vision–language and omni models from ten families.

8 CONCLUSION

  • Over 97% of the original suite’s items are removed while model rankings are closely preserved, and VAW performs best at 5% retention on held-out models.The benchmark-level mechanism is also validated on a substantially larger external OpenVLM panel.Source: §8 · p. 10

A.1 MODEL DETAILS

  • The evaluation panel contains 30 vision–language and omni models from ten families.Models are evaluated with greedy decoding through lmms-eval.Source: Table 3 · p. 17, §A.1 · p. 17

A.2 BENCHMARK DETAILS

  • The benchmark catalogue groups benchmarks by capability and includes binary, continuous, GPT-judged, and aggregate scoring.It covers 386,324 items before Stage 0 cleaning.Source: §A.2 · p. 17, Table 4 · p. 18

B STAGE 0: DATA CLEANING

The section describes cleaning the benchmark pool, selecting category representatives and supplements, and pruning items before evaluating ranking preservation on held-out models. The reported results include both per-category fidelity and end-to-end comparisons.

B STAGE 0: DATA CLEANING

  • 115,916 of 386,324 items (30.0%) are removed by filters for all-correct answers and text-only solvability; all-wrong items are retained.The filters run once before any model split, and 11,247 removed items fail both filters.Source: B STAGE 0: DATA CLEANING · p. 19, Table 5 · p. 20

C.1 CATEGORY REPRESENTATIVE SELECTION

  • The 18 category representatives are selected by correlation with their category means; their median correlation is 0.925, but eight fall below 0.90.Hallucination Detection has the lowest representative correlation (0.699), followed by Visual Reasoning (0.758).Source: §C.1 · p. 21, Table 6 · p. 21

C.2 MULTI-SOURCE ITEM SUPPLEMENTATION FOR LOW-ρ CATEGORIES

  • Twelve of 18 categories reach ρ ≥0.95 after supplementation, while six remain below target even at the best supplement prefix.Supplements are added in descending score variance, and the six below target include Visual Reasoning (0.764) and Web/UI Understanding (0.800).Source: §C.2 · p. 21, Table 8 · p. 23

D.1 SELECTION RULE

  • The item-selection rule uses vision-text embeddings to form coverage cells, while items without an embedding receive no cell and δVL_i = 0.A cell share that cannot be filled is completed with the benchmark’s highest-scoring remaining items; scores are scaled to [0, 1].Source: §D.1 · p. 23

D.2 PER-CATEGORY VARIATION

  • At r=5%, VAW retains ρ ≥0.90 in 13 of 18 categories on unseen models, with the weakest fidelity in Spatial Reasoning (0.801).Academic Knowledge (0.848) and Object Recognition (0.858) also have low fidelity; the passage recommends full benchmarks or higher retention for fine-resolution evaluation of these categories.Source: §D.2 · p. 23, Table 9 · p. 24

D.3 UNSEEN-MODEL PROTOCOLS

  • At r=5%, VAW has the highest mean held-out fidelity, 0.925 versus 0.920 for AnchorPoints, 0.909 for IRT 2PL, and 0.906 for Variance.The advantage is modest across splits: VAW is best in 10 of 20, with split-to-split standard deviations of 0.01–0.03.Source: §D.3 · p. 24, Table 11 · p. 25
  • In end-to-end evaluation at r=5%, VAW reaches 0.892 versus 0.901 for Stage 3 alone, and the selectors overlap substantially across splits.The end-to-end split-to-split standard deviation is 0.05–0.09; without Stage 3, VAW reaches 0.956 and IRT 2PL reaches 0.958 across all 18 pruned categories.Source: Table 12 · p. 25, §D.3 · p. 25

D.4 CELL-SPACE ABLATION

The cell-space ablation finds a ranking-fidelity gain from vision-text coverage cells over benchmark shares alone, while the paper also reports category-count pruning and theoretical conditions for the variance-based selector.

D.4 CELL-SPACE ABLATION

  • At r=5%, vision-text cells add 0.017 over benchmark shares alone in 18 of 20 splits, and outperform same-size random partitions by 0.011 on average.Text-only cells are within about 0.005 of vision-text cells; the DPP variant reaches 0.919 versus 0.925 for cells.Source: §D.4 · p. 26, Table 13 · p. 26
  • At r=5%, swapping the default encoder for any of three alternatives lowers VAW by 0.011–0.012 on the four categories all encoders cover.At r=2% and r=10%, each alternative encoder is above the default.Source: §D.4 · p. 26, Table 14 · p. 27

E STAGE 3: CATEGORY-COUNT PRUNING 

  • The released suite uses k=5 categories, while k=8 can omit Visual Reasoning and retain 1,136 items at end-to-end ρ=0.962.Stage 3 searches for ranking fidelity at a chosen k rather than minimizing item count; selected category subsets are not nested across k.Source: E STAGE 3: CATEGORY-COUNT PRUNING  · p. 27, Table 15 · p. 27
  • Across six family pairs, item-level agreement on the released suite ranges from 0.115 to 0.463, indicating that families differ in which items they find hard.Source: E STAGE 3: CATEGORY-COUNT PRUNING  · p. 28

F THEORETICAL ANALYSIS

  • Theoretical results show plug-in item variance is estimated with noise of order M^-1/2; a fixed tilt can change the population top-K selection as the panel grows.A decaying tilt approaches the population variance selector at an explicit rate under the stated margin condition, and recovers it exactly with high probability under the boundary-gap condition.Source: F THEORETICAL ANALYSIS · p. 28

F.1 NOTATION AND STANDING ASSUMPTIONS

  • The theoretical analysis assumes independent, identically distributed correctness indicators within each item and requires the number of valid records per item to grow for recovery.For binary scores, the variance score is based on the empirical correctness rate; the results also discuss bounded scores in [0, 1].Source: §F.1 · p. 28

F.2 BEHAVIOUR OF VAW AS THE NUMBER OF MODELS GROWS

The analysis examines how VAW’s vision tilt behaves as model panels grow, including its theoretical convergence and empirical ranking effects. It also evaluates benchmark and item selection on OpenVLM.

F.3.2 PROOF OF THEOREM 2 (FIXED NORMALISED TILT IS INCONSISTENT WHEN IT CHANGES THE POPULATION TOP-K)

  • With a fixed tilt that changes the population top-K, empirical selection converges to that perturbed top-K rather than the variance-only top-K.This result applies under the stated assumptions and boundary-gap conditions.Source: §F.2 · p. 29, §F.3.2 · p. 31

F.3.1 PROOF OF THEOREM 1 (VARIANCE-ESTIMATOR CONCENTRATION)

  • For the plug-in variance estimate bvi, the generic rate is Θ(M^-1/2), and at p_i=1/2 its standard deviation is Θ(1/M); moreover, Pr(|bvi−vi|≥ε)≤2exp(−2Mε²).Source: §F.2 · p. 29, §F.3.1 · p. 31

F.3.3 PROOF OF THEOREM 3 (ANNEALED TILT APPROACHES THE POPULATION VARIANCE SELECTOR)

  • For an annealed tilt, the selector’s disagreement with the population variance top-K vanishes when the bound’s terms vanish; at fixed retention, the M-dependent terms scale as O(M^-1/2) up to logarithmic factors.The boundary-atom term must vanish separately; for a fixed pool with a positive top-K gap, Proposition 1 gives exact recovery.Source: §F.2 · p. 30, §F.3.3 · p. 32, §F.3.4 · p. 33

F.2 BEHAVIOUR OF VAW AS THE NUMBER OF MODELS GROWS

  • The convergence bounds provide no finite-sample guarantee at the panel sizes tested, and the analysis does not establish that variance is the strongest large-panel selector or that tilt helps small panels.The bounds concern a fixed pool; the assumptions include independent model draws and an unestimated population margin constant.Source: §F.2 · p. 30

G FINITE-M SCAN OF THE VISION TILT INSIDE CELLS

H.1 OPENVLM DATA

  • The OpenVLM scan uses aligned benchmarks and model coverage of at least 95%; at the largest panels, only benchmarks with sufficient models remain, down to seven.The scan evaluates model panels from M=3 to 192 across 10 OpenVLM benchmarks.Source: G FINITE-M SCAN OF THE VISION TILT… · p. 33, §H.1 · p. 34

H.2 STAGE 2 ON OPENVLM: SELECTOR COMPARISON

  • On the eight OpenVLM benchmarks with closed answer spaces and at least 200 scored models, DISCO scores ρ=0.910 versus Variance’s ρ=0.907 at r=5%.Selectors are compared against the full-benchmark ranking on the same models.Source: §H.2 · p. 34

H.3 STAGE 3 ON OPENVLM: BENCHMARK SELECTION

The OpenVLM benchmark-selection analysis tests whether a small subset preserves the ranking across 14 benchmarks. It also measures how the score matrix’s spectral concentration varies across model-release eras.

H.3 STAGE 3 ON OPENVLM: BENCHMARK SELECTION

  • Exhaustive Stage 3 selection of five OpenVLM benchmarks reaches ρ=0.988 against the 14-benchmark ranking on 231 models; greedy selection stays within 0.002 of exhaustive at every k.The selected benchmarks are AI2D, MMMU, OCRBench, RealWorldQA, and ScienceQA; the subset is re-fitted on OpenVLM.Source: §H.3 · p. 34, Table 18 · p. 35

H.4 STAGE 3 ON OPENVLM: LOW RANK ACROSS RELEASE ERAS

  • Across OpenVLM release eras, the top five components contain 86–89% of spectral energy and effective rank is 7.2–8.1.The last era has 13 benchmarks because one has no scores.Source: §H.4 · p. 35, Table 19 · p. 35

I.1 SHELF LIFE OF A FROZEN BENCHMARK SELECTION

The analyses test how frozen benchmark selections behave as model panels grow, whether compressed evaluation preserves rankings of model variants, and whether evaluation selectors transfer to training-data pruning. Results are strong for benchmark and variant ranking, but do not support using evaluation-side selectors for training-data selection.

I.1 SHELF LIFE OF A FROZEN BENCHMARK SELECTION

  • ρ stays 0.988–0.990 as dated OpenVLM panels grow from 132 to 200 models, with the frozen five-benchmark subset at or above the 99.9th percentile of 2,002 subsets; adding 62 undated models lowers ρ to 0.971.The selection is at most 0.0004 below the best subset on dated panels; release dates are an imperfect proxy for model generation, and this benchmark-level analysis does not test the item-level release.Source: §I.1 · pp. 35–36, Table 20 · p. 36

I.2 CAPABILITY FRONTIER: A DESCRIPTIVE STRESS SET

  • The Capability Frontier contains 1,653 difficult items; the Qwen-VL-Chat→Qwen3-VL-8B generation gap on it is 0.62× its full-pool value, though selection partly reflects the Qwen family’s scores.Items are selected from those with panel mean scores in [0.10, 0.50], excluding items almost every model fails; future-generation discriminativeness has not been tested.Source: §I.2 · p. 36, Table 21 · p. 36

I.3 LEFT-OUT BENCHMARK PREDICTION ACROSS PANEL SIZES

  • Left-out benchmark prediction MAE falls from 0.11–0.21 at M=1 to 0.04–0.10 at the largest panels, with smaller improvements beyond M=64.The analysis predicts full benchmark scores, not scores from the released suite.Source: §I.3 · p. 36, Table 22 · p. 37

I.4 CHECKPOINT AND HYPER-PARAMETER RANKING

  • At 5% retention, VAW’s mean Spearman ρ is 0.939 versus Variance’s 0.857 across eight ceiling-passing variant sets, and VAW wins in six.Variant sets below bootstrap ρ=0.90 on the full-test ranking support no conclusion; the comparison is descriptive because variants from one run are not independent.Source: Table 23 · p. 37, §I.4 · p. 37

I.5 EVALUATION-COMPRESSION SELECTORS DO NOT TRANSFER TO TRAINING-DATA PRUNING

  • No evaluation-compression selector reliably beats random training-data sampling beyond seed variation, so the authors advise against using evaluation-side item scores, including VAW, for training-data pruning.At r≤0.3, VAW is within 3.1 accuracy points of Random and ahead only at r=0.3 by 0.8 points, against seed standard deviations of 2–3 points.Source: Table 24 · p. 38, §I.5 · p. 38

J LIMITATIONS

The authors qualify the evidence by noting limits in panel size, external validation, theory, modality coverage, and prediction accuracy. They also caution that aggressive compression may narrow attention to already represented capabilities, so the suite should not replace comprehensive evaluation in high-stakes settings.

J LIMITATIONS

  • With only 10–20 held-out models per split, variation in ρ is comparable to gaps between stronger selectors, so results are averaged over 20 splits rather than based on a single split.Source: J LIMITATIONS · p. 38
  • End-to-end validation of the released suite is limited to the authors’ panel; OpenVLM validation covers Stage 3 at benchmark level and Stage 2 selectors under same-model scoring, while blind-solvability filtering cannot be rerun there.Source: J LIMITATIONS · p. 38
  • The theory gives no finite-sample guarantee at tested panel sizes and does not establish that the tilt or cells improve ranking fidelity; those questions rely on experiments.Source: J LIMITATIONS · p. 38
  • Embedding cells improve ρ over random partitions by about 0.011, but text-only cells perform as well as vision-text cells, so image inputs are not required to build the cells.Source: J LIMITATIONS · p. 38
  • Left-out benchmark prediction remains at 0.04–0.10 MAE at the largest available panels, and the released suite itself has not been tested as an estimator.Source: J LIMITATIONS · pp. 38–39
  • Judge-scored results inherit judge errors, and the panel excludes proprietary API models and video or audio benchmarks; Stage 2 would need new dependence scores and embeddings for other modalities.Source: J LIMITATIONS · p. 39

K BROADER IMPACT

  • Aggressive compression may narrow attention to represented capabilities, so the compact suite is not a substitute for comprehensive high-stakes evaluation or audits of emerging capabilities; users can stop at less compressed stages.Source: K BROADER IMPACT · p. 39

Related papers

Loading 2609.37515v1…