PaperlayerView exact revision on arXiv

Source-linked AI summary

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Yuanteng Chen, Qiwei Lai, Chen Tianqi, Peisong Wang, Yuantian Shao, Nanxin Zeng, Zhilei Liu, Chuangyi Li, Jing Liu, Jian Cheng

arXiv:2609.25809v1cs.LGcs.AI

TL;DR

Evidence on dynamic pruning in fine-grained MoEs has been limited, leaving unclear how much expert selection is redundant and when adaptive allocation helps. Across twelve checkpoints in nine architecture families, the study finds that retaining about two thirds of selected experts preserves 98.8% of baseline performance, while dynamic allocation matters most under aggressive pruning.

  • Problem

    Existing pruning evidence focuses largely on coarser architectures and likelihood-scored multiple-choice benchmarks, leaving the redundancy of fine-grained expert selection and the value of dynamic allocation unclear.

    Source: §1 · p. 2

  • Method

    The study evaluates dynamic expert pruning across twelve checkpoints spanning nine architecture families, comparing uniform truncation with dynamic allocation at matched budgets.

    Source: §1 · p. 2, §8 · p. 12

  • Results

    Uniformly retaining ⌈2K/3⌉ of the natively selected experts preserves 98.8% of baseline performance on average, while dynamic rules recover up to 3.0 points under aggressive pruning, especially on generative tasks.

    Source: §8 · p. 12

  • Takeaways & Limitations

    Uniform truncation is the essential reference for dynamic pruning; the clearest opportunity for new allocation methods is preserving generative capabilities that uniform truncation begins to lose.

    Source: §1 · p. 3, §8 · p. 13

Abstract

from arXiv · show

Fine-grained mixture-of-experts (MoE) architectures have become a mainstream design for open-weight LLMs, with hundreds of experts and increasingly many selected per token. This shift makes dynamic expert pruning an attractive route to cheaper inference. Yet existing evidence comes largely from coarser architectures and likelihood-scored multiple-choice benchmarks, leaving three central questions open in the fine-grained regime: how redundant per-token expert selection is, how effectively existing pruning methods exploit that redundancy, and what governs a model's sensitivity to pruning. We fill this gap with a systematic empirical study of twelve fine-grained MoE checkpoints spanning nine architecture families, with a core suite of eleven benchmarks covering knowledge QA, mathematics, code generation, and general reasoning. We find that expert selection is far more redundant than the field's operating points assume: uniformly retaining about two thirds of the selected experts preserves 98.8% of unpruned performance on average, requiring only a one-integer change and delivering 1.2-1.7x measured speedup across two serving backends. This simple baseline leaves little room for dynamic allocation at conservative budgets: even the best published rules differ from it by under 1% at matched expert budgets. Their value emerges under aggressive pruning, where the best rules recover up to 3.0% over uniform truncation, with gains concentrated in the generative tasks that suffer the sharpest degradation. Sensitivity to aggressive pruning also depends on the model: larger and thinking models are more resilient, whereas multimodal models are more vulnerable. Together, these findings reveal how much expert computation fine-grained MoEs can dispense with, and establish when dynamic allocation earns its complexity, informing both practical deployment and future pruning methods.

1 INTRODUCTION

Prior pruning evidence leaves open how much fine-grained MoE expert selection is redundant, when dynamic allocation improves on uniform reduction, and which model or task properties shape pruning sensitivity. Across broad models and tasks, the study finds that uniform pruning preserves most performance, while dynamic allocation matters most under aggressive pruning.

Motivation and questions

  • Existing work has not established how much selection can be removed in fine-grained MoEs or when dynamic allocation is needed, in part because studies use coarser models and likelihood-scored multiple-choice benchmarks.The study frames these gaps as questions about redundancy, dynamic allocation, and pruning sensitivity.Source: §1 · p. 2

Study scope

  • The study evaluates twelve checkpoints across nine architecture families, using eleven benchmarks spanning knowledge QA, mathematics, code, and general reasoning.Nine models support uniform budget sweeps, four support dynamic-rule comparisons, and three additional checkpoints probe reasoning mode, scale, and modality.Source: §1 · p. 2

Central findings

  • Uniformly retaining about two thirds of selected experts preserves 98.8% of unpruned performance on average across nine models and three task families.This requires changing only a single integer, with no calibration or token-specific allocation.Source: §1 · p. 2
  • At conservative budgets, the best published rules differ from uniform truncation by −0.53 to +0.67 percentage points; under aggressive pruning, they recover up to 3.0 points.The winning rule changes across models and budgets.Source: §1 · p. 2
  • Generative tasks deteriorate more sharply than likelihood-scored QA, while larger and thinking models withstand aggressive pruning better and multimodal models are more sensitive.Generative tasks account for most gains from dynamic allocation; the reported sensitivity differences are small at conservative budgets.Source: §1 · pp. 2–3

Practical implication

  • The findings establish uniform reduction of K as a practical starting point, with dynamic allocation as a way to push further where uniform truncation begins to fail.Future methods have room to improve the quality–compute trade-off at those more difficult budgets.Source: §1 · p. 3

2 RELATED WORK

Fine-grained MoE expands the range of expert budgets and routing signals available for pruning. Prior dynamic methods use different token, layer, and expert information; this study compares their added value beyond simply lowering K.

2.1 FINE-GRAINED MOE

  • Fine-grained designs offer longer selected-expert lists and more diverse router scores, broadening the budgets and signals available for pruning.The passage contrasts top-2 routing with fine-grained selections and names softmax, sigmoid with learned biases, and square-root softplus scoring.Source: §2.1 · p. 3

2.2 DYNAMIC EXPERT PRUNING

  • Dynamic pruning reduces the experts executed at inference while retaining the expert pool, unlike structural pruning, which permanently removes experts.Source: §2.2 · p. 3
  • Existing rules allocate computation using gate-weight ratios, expert-output similarity, cumulative gate weight, layer sensitivity with token signals, or expert traffic.The rules include NAEE, DIEP, DYNROUTE, BAN, and EAC-MoE; EAC-MoE restricts pruning to prefill.Source: §2.2 · p. 3
  • The study compares representative rules across fine-grained architectures, task families, and matched expert budgets to measure gains beyond simply lowering K.This comparison is intended to establish when dynamic allocation improves on uniform truncation and where future methods have room to advance.Source: §2.2 · p. 3

3 EXPERIMENTAL SETUP

The experiments use routed expert budget as a common axis for studying redundancy, dynamic allocation, and pruning sensitivity across fine-grained MoEs. They combine broad architectural coverage with focused model comparisons and benchmarks spanning likelihood-scored QA and generated reasoning tasks.

3 EXPERIMENTAL SETUP

  • The experimental setup uses routed expert budget to connect analyses of redundancy, dynamic allocation, and pruning sensitivity across architectures and task families.Source: §3 · p. 3

3.1 ROUTING AND EXPERT BUDGETS

  • FIXED-K keeps the top k experts for every token and layer, while dynamic rules vary retained experts using token- and layer-dependent signals.Both reduce routed expert execution and data movement while shared experts continue processing every token.Source: §3.1 · p. 4

3.2 MODELS

  • Nine checkpoints span nine architecture families, with routed pools of 32–512 experts and native selections of 4–10; three further Qwen3 checkpoints support comparisons of reasoning mode, scale, and modality.The model set includes different shared-expert configurations and softmax, sigmoid, and square-root-of-softplus routers.Source: §3.2 · p. 4

3.3 TASKS AND EVALUATION

  • The core evaluation uses eleven benchmarks across knowledge QA, mathematical and code reasoning, and knowledge and general reasoning, with QA scored by likelihood and reasoning evaluated through generated answers.The modality analysis adds a nine-dataset multimodal suite and expands text QA to nine datasets.Source: §3.3 · pp. 4–5

4 HOW MUCH ROUTING DOES A FINE-GRAINED MOE ACTUALLY NEED?

Uniform truncation provides a baseline for measuring how much fine-grained MoE routing redundancy remains as expert budgets tighten across task families.

4 HOW MUCH ROUTING DOES A FINE-GRAINED MOE ACTUALLY NEED?

  • FIXED-K lowers expert counts uniformly across tokens and layers without changing model parameters or router ranking, isolating redundancy already present in the model.The reduced-budget quality establishes the baseline that a more elaborate pruning rule must improve upon.Source: §4 · p. 5

4.1 TWO THIRDS OF THE SELECTED EXPERTS ARE ENOUGH

  • At three of eight experts, Qwen3-30B-A3B retains 73.6% of its mathematical and code reasoning score; at two of six, Ling-lite-1.5 retains 57.8%.At two experts, Qwen3-30B-A3B retains 13.5% on mathematics and code, while GPT-OSS-20B retains 96.3%, showing that the retained fraction of native selection matters.Source: §4.1 · pp. 5–6
  • 98.8% of unpruned performance is preserved by retaining ⌈2K/3⌉ experts, averaged over nine models and three task families.Figure 2 shows task-family scores normalized to each unpruned model; the gate-weighted mixture concentrates weight on the preferred experts after lower-weight choices are removed and remaining weights renormalized.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 KNOWLEDGE SURVIVES WHAT REASONING DOES NOT

  • At two experts, Qwen3-30B-A3B retains 60.4% on knowledge QA but 13.5% on mathematical and code reasoning; long-form generation lets local disruptions propagate across dependent steps.On eight of nine models, mathematical and code reasoning is the least robust family under aggressive pruning.Source: §4.2 · pp. 6–7

4.3 FROM ROUTING REDUNDANCY TO INFERENCE SPEED

  • 1.18–1.65× decode and 1.15–1.25× prefill throughput on vLLM; HF Transformers reaches 1.44–1.73× decode and 1.21–1.31× prefill.These are measured speedups at tested reduced budgets relative to the same unpruned model on each backend; separate unfused GEMMs in HF Transformers allow individual matrix multiplications to be removed, while vLLM's fused kernels amortize part of that cost.Source: §4.3 · p. 7, Table 2 · p. 7

5 WHEN DOES DYNAMIC ALLOCATION BEAT UNIFORM PRUNING?

Uniform truncation is a strong baseline, but dynamic allocation becomes valuable once aggressive pruning causes losses, especially on generative tasks. The matched-budget comparison shows that both the pruning budget and the evaluated tasks shape the benefit of adaptation.

5.1 COMPARING ALLOCATION AT MATCHED BUDGETS

  • The comparison matches four published allocation rules against FIXED-K on four fine-grained MoEs at matched average active-expert budgets.NAEE uses gate weights, DYNROUTE uses cumulative probability mass, DIEP uses similarity-adjusted skipping, and BAN combines token routing with calibrated layer sensitivity.Source: §5 · p. 7, §5.1 · pp. 7–8, Table 3 · p. 8

5.2 DYNAMIC ALLOCATION PAYS OFF UNDER AGGRESSIVE PRUNING

  • At conservative budgets, the best dynamic rule’s margin over FIXED-K ranges from −0.53 to +0.67 points across four models, averaging +0.06.The comparison pairs FIXED-K with the best of four dynamic rules at matched measured budgets; configurations spending more than 10% over a tier are excluded.Source: §5.2 · p. 8, Table 3 · p. 8, §5.1 · p. 8
  • At aggressive budgets, the best dynamic rule gains 2.45 points on Qwen3-30B-A3B, 2.71 on Qwen3-Next-80B, 2.98 on Ling-lite-1.5, and 0.90 on GPT-OSS-20B over uniform truncation.These budgets retain between one third and one half of native selection; the average gain is 2.26 points.Source: §5.2 · p. 9
  • BAN wins the aggressive-budget comparison on three of four models, while the best rule changes with model and budget.DYNROUTE has the highest conservative-budget mean on Qwen3-30B-A3B; BAN wins there under aggressive pruning.Source: §5.2 · p. 9

5.3 GENERATIVE TASKS REVEAL THE VALUE OF REALLOCATION

  • At Qwen3-30B-A3B’s aggressive budget, BAN improves the mean by 3.17 points on seven generative datasets versus 1.20 on four Knowledge QA datasets.On GPT-OSS-20B, DYNROUTE raises LiveCodeBench from 48.0 to 62.5, a 14.5-point gain, while the suite-wide improvement is 0.90 points.Source: §5.3 · p. 9

6 WHAT GOVERNS A FINE-GRAINED MOE’S SENSITIVITY TO EXPERT PRUNING?

Pruning sensitivity differs across model properties, with the clearest separations emerging under aggressive budgets. Thinking and larger models retain more generative capability, while the multimodal model declines more sharply.

6 WHAT GOVERNS A FINE-GRAINED MOE’S SENSITIVITY TO EXPERT PRUNING?

  • The study compares reasoning-mode post-training, parameter scale, and modality across three pairs drawn from four Qwen3 checkpoints, with differences emerging mainly under aggressive pruning.All checkpoints have 128 experts and natively select eight; curves are normalized to each model’s unpruned performance.Source: §6 · p. 9

6.1 THINKING PRESERVES MORE GENERATIVE CAPABILITY

  • At four experts, Qwen3-30B-A3B-Thinking leads its Instruct counterpart by 2.7 percentage points on four challenging generative datasets; at three experts, the lead is 5.1 points.Performance retention is close between eight and five experts; the task breakdown locates the advantage in generation, with little difference on Knowledge QA.Source: §6.1 · pp. 9–10, Figure 3 · p. 10

6.2 SCALE MATTERS MOST UNDER AGGRESSIVE PRUNING

  • At three experts, Qwen3-235B-A22B-Instruct retains 76.7% of unpruned performance versus 62.3% for Qwen3-30B-A3B-Instruct, a 14.4-point gap.The larger model has no advantage over the smaller one from seven to five experts; the gap appears as pruning deepens and is located in generative tasks, not Knowledge QA.Source: §6.2 · p. 10

6.3 MULTIMODAL MODELS ARE MORE SENSITIVE TO AGGRESSIVE PRUNING

  • At three experts, the multimodal model falls 11 points behind the text model, and at two experts the gap widens to 42 points.Their normalized curves remain within one percentage point from eight to four experts; the matched suites contain nine datasets each.Source: §6.3 · p. 10, Figure 4 · p. 11
  • Within the selected eight experts, the leading three hold 49.8% of routed weight for the multimodal model versus 59.8% for the text model.The selected experts hold 15.5% of the full 128-way softmax mass for the multimodal model versus 44.3% for the text model; the direction is consistent across all 48 MoE layers.Source: §6.3 · pp. 10–11, Figure 4 · p. 11

7 IMPLICATIONS FOR DYNAMIC EXPERT PRUNING

The paper recommends first establishing a uniform-pruning budget, then measuring what dynamic allocation adds at that budget. Allocation design should account for task, model, layer sensitivity, and the cost of executing the selected assignments.

7 IMPLICATIONS FOR DYNAMIC EXPERT PRUNING

  • The recommended workflow is to establish an operating budget with uniform truncation, measure dynamic rules’ added value, then design allocation for tight budgets.Uniform pruning reveals removable redundancy; its remaining losses show where better allocation has something to recover.Source: §7 · p. 11

7.1 ESTABLISHING THE BUDGET WITH UNIFORM PRUNING

  • Retaining ⌈2K/3⌉ experts is a useful initial operating point, after which progressively lowering the count reveals where capability begins to decline.The recommendation is based on the broad tolerance observed across the studied architectures.Source: §7.1 · p. 11
  • For derivations, programs, or open-ended answers, track complete generations because they expose sharper pruning losses than other evaluations.Larger and thinking checkpoints retain more generative capability under aggressive pruning, while the multimodal checkpoint loses performance more sharply.Source: §7.1 · pp. 11–12

7.2 EVALUATING THE ADDED VALUE OF DYNAMIC ALLOCATION

  • Measure dynamic allocation’s margin over uniform truncation across budget levels; remeasure realized average expert use and recalibrate when transferring settings.Thresholds can spend different budgets across models and workloads, so comparisons should use the realized average number of activated experts.Source: §7.2 · p. 12

7.3 DESIGNING ALLOCATION FOR TIGHT BUDGETS

  • For tight budgets, combine token routing with layer sensitivity; cumulative retained mass and per-rank statistics keep meaning across router widths better than top-1-relative thresholds.Kernel support and low allocation overhead are needed for reduced expert computation to translate into practical inference savings.Source: §7.3 · p. 12

8 CONCLUSION

Uniformly retaining about two thirds of selected experts preserves 98.8% of baseline performance on average, while dynamic allocation matters most under aggressive pruning. The appendix details the implementation and calibration used to compare pruning rules at matched budgets.

8 CONCLUSION

  • ⌈2K/3⌉ retained experts preserve 98.8% of baseline performance on average, with 1.2–1.7× decoding throughput across tested serving configurations.At aggressive budgets, dynamic rules recover up to 3.0 points; larger and thinking models are more resilient, while the multimodal checkpoint is more sensitive.Source: §8 · pp. 12–13

A IMPLEMENTATION

  • The four published dynamic rules were reimplemented in a shared vLLM router patch, with appendix sections documenting application, rule computations, calibration, and budget matching.The rules are extended to fine-grained routers for comparison at iso-budget settings.Source: A IMPLEMENTATION · p. 16

A.1 WHERE THE RULES ARE APPLIED

  • The pruning hook acts after router scores are computed and before experts reach the kernels, leaving shared experts untouched and outside reported budgets.Retained gate weights are renormalised, and a single implementation covers the four architectures.Source: §A.1 · p. 16

A.2 THE FOUR RULES AS IMPLEMENTED

  • NAEE stops at a relative-weight threshold, DYNROUTE retains a prefix reaching target mass, and DIEP adjusts the threshold using expert similarity.BAN combines token-level router uncertainty with layer sensitivity; the rules do not all share the same minimum-expert floor.Source: §A.2 · p. 17

A.3 OFFLINE CALIBRATION

  • DIEP and BAN calibration statistics are collected per checkpoint on 512 C4 documents before evaluation, without using evaluation data.DIEP stores pairwise expert-output CKA and median w2/w1; BAN stores per-layer sensitivity and token-sensitivity range.Source: §A.3 · p. 18

A.4 SOLVING EACH KNOB TO A TARGET BUDGET

  • Each rule’s knob is solved offline to match a target average expert count, with separate solutions for generation and likelihood-scored QA.Configurations exceeding their target budget by more than 10% are excluded from the main comparison.Source: §A.4 · p. 18

B EVALUATION PROTOCOL: HOW A SWEEP CAN SILENTLY BE WRONG

Comparisons can mislead when rules are matched by hyperparameter rather than actual budget, or when noisy reference rows favor a conclusion. The authors use measured budgets and conservative FIXED-K readings to address these problems.

B EVALUATION PROTOCOL: HOW A SWEEP CAN SILENTLY BE WRONG

  • A shared hyperparameter can spend different amounts across generation and QA, so comparisons require separately solved thresholds and measured average expert counts.The authors retain FIXED-K readings that make reported margins conservative when repeated measurements disagree.Source: B EVALUATION PROTOCOL: HOW A SWEEP CAN… · p. 19
  • Some FIXED-K configurations differed by up to a point across measurements, and the authors kept the older, lower reading so comparisons would not favor their conclusion.The reported margins are therefore conservative in the direction unfavorable to the claim that FIXED-K is hard to beat.Source: B EVALUATION PROTOCOL: HOW A SWEEP CAN… · p. 19
  • Rules that undershoot FIXED-K’s budget can score lower simply because they receive less computation, with deficits reaching 0.95 experts in the worst case.Such rows remain in Appendix E with measured budgets visible, but their deficits are not evidence about rule quality.Source: B EVALUATION PROTOCOL: HOW A SWEEP CAN… · p. 19

C GENERALISING A TOP-2 SKIPPING RULE TO TOP-K ROUTING

Extending a top-2 similarity-based rule to wider routers requires a choice not dictated by its original formulation. The appendix reports that its generalised form does not perform uniformly across models, alongside the broader uniform-truncation pattern.

C GENERALISING A TOP-2 SKIPPING RULE TO TOP-K ROUTING

  • The top-2 DIEP rule is generalised by comparing each rank above one with the leading expert using its own similarity factor, judged independently.A prefix variant instead stops at the first failure; neither choice rescues the rule at the budgets reported in Table 3.Source: C GENERALISING A TOP-2 SKIPPING RULE… · p. 19
  • DIEP’s conservative-budget deficit versus FIXED-K varies across models and does not simply track router width.The reported deficits range from −11.27 on GPT-OSS-20B at K=4 and −11.11 on Qwen3-30B-A3B at K=8 to −2.38 on Ling-lite-1.5 at K=6 and −2.16 on Qwen3-Next-80B at K=10.Source: C GENERALISING A TOP-2 SKIPPING RULE… · p. 20

D FULL RESULTS: UNIFORM TRUNCATION

  • Uniform-truncation tables cover every budget and dataset for the four core models and five further architectures, with scores reported as percentages and row means across available datasets.The reported curves show a near-flat region at higher budgets, while collapse arrives at a fraction of native budget rather than a fixed expert count.Source: D FULL RESULTS: UNIFORM TRUNCATION · p. 20, Table 5 · p. 20

E FULL RESULTS: EVERY RULE AT BOTH BUDGETS

The expanded results report every evaluated dynamic rule at both budget tiers and show that rule ordering can change between tiers. Measured budgets and incomplete configurations qualify how those comparisons should be read.

E FULL RESULTS: EVERY RULE AT BOTH BUDGETS

  • The expanded tables list every evaluated rule by core model, budget tier, and dataset, with FIXED-K at the head of each tier for direct comparison.The ordering of rules changes between tiers on the same model, and the spread among rules at matched budgets can exceed the gap between the best rule and FIXED-K.Source: E FULL RESULTS: EVERY RULE AT BOTH… · pp. 20–21
  • The additional uniform-truncation results cover five further architectures, reporting every budget and dataset with percentage scores and per-row dataset means.The top row in each block is unpruned; rows above and at or below half the native K are marked separately.Source: Table 6 · p. 21
  • Budgets are measured average expert counts solved separately for generative and QA settings; configurations absent from a tier either missed the 10% tolerance or did not complete.The tables do not silently substitute a cheaper configuration.Source: E FULL RESULTS: EVERY RULE AT BOTH… · p. 21

F FULL RESULTS: THE MATCHED PAIRS

The matched-pair results show that pruning sensitivity varies by task and model: exact-string generation breaks especially sharply for multimodal models, while reasoning mode and scale help on long-derivation tasks. Throughput measurements use native FIXED-K selection, excluding allocation-rule overhead.

F FULL RESULTS: THE MATCHED PAIRS

  • Tables 7–10 report per-dataset results for each rule at both budget tiers, with measured generative/QA budgets; configurations overspending a tier by more than 10% are ineligible for Table 3.Uniform truncation is the first row of each tier for comparison.Source: Table 7 · p. 22, Table 8 · p. 22, Table 9 · p. 23
  • Below four experts, multimodal degradation is uneven: ChartQA falls furthest, and at two experts transcription-heavy datasets approach zero while multiple-choice sets retain one-third to one-half of unpruned scores.The pattern points to exact-string output as an early failure point.Source: F FULL RESULTS: THE MATCHED PAIRS · pp. 22–23, Table 12 · p. 24
  • Reasoning-mode post-training and parameter scale help at three experts on long-derivation columns, but are absent or slightly negative on the four likelihood-scored datasets.Figure 3 averages AIME-24, AIME-25, LiveCodeBench, and GPQA-D.Source: F FULL RESULTS: THE MATCHED PAIRS · p. 22, Table 11 · p. 24
  • The Figure 4 model pairings use nine short-answer datasets each, but the suites differ, so compare their means only against each model’s own unpruned row.The multimodal suite includes nine vision-language datasets; the text suite combines four likelihood-scored datasets with five additional short-answer datasets.Source: F FULL RESULTS: THE MATCHED PAIRS · p. 22, Table 12 · p. 24

G THROUGHPUT MEASUREMENTS

  • Throughput tests use native FIXED-K selection, so measured speedups exclude per-token allocation-rule overhead; prefill and decode are timed separately, with rates reported as trial medians.Runs use matched batch size and prompt length, and generation continues to the requested length.Source: G THROUGHPUT MEASUREMENTS · pp. 23–24

H SINGLE-FACTOR STUDIES: FURTHER DETAIL

The multimodal router is flatter than the text model’s at both prompt and decode stages, and the model difference persists across layers. The accompanying short-answer suites contain different datasets, so their means are comparable only relative to each model’s own unpruned baseline.

H SINGLE-FACTOR STUDIES: FURTHER DETAIL

  • Across prompt and decode, the multimodal model’s top three experts hold a smaller share of selected weight than the text model’s, and its routing remains flatter when generating.At decode, the leading three hold 51.7% of the selected eight experts’ weight; at prompts they hold 49.8%.Source: H SINGLE-FACTOR STUDIES: FURTHER DETAIL · pp. 24–25
  • The multimodal and text short-answer suites each contain nine datasets, but they are not the same datasets, so compare means only against each model’s own unpruned row.Seven of the nine multimodal datasets are lite subsets.Source: Table 13 · p. 25
  • The model gap is stable across depth: the top-eight mass gap is −26.0 points (95% interval [−31.1, −21.2]), and multimodal router entropy is 0.70 nats higher.The entropy difference’s 95% interval is [+0.54, +0.85] nats.Source: H SINGLE-FACTOR STUDIES: FURTHER DETAIL · p. 25
Loading 2609.25809v1…