GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

1. INTRODUCTION

• Metadata-based prediction could reduce candidate executions relative to lookup tables, but remote header-only acquisition is outside scope.Raw memory rooflines are complicated by quantized-kernel efficiencies, MoE activation sparsity, and hybrid KV allocation.Source: §1 · p. 1
• The study tests transfer across model families and hardware using pre-inference GGUF features on two Apple systems and an NVIDIA GPU.It isolates activated-parameter and per-layer KV accounting, and reports partial coefficient transfer, system-varying low-bit behavior, and unsuccessful RTX prefill replication.Source: §1 · p. 1
• Related work spans accelerator design, serving simulation, and hardware-aware inference analysis [4] [5], while this estimator targets unmeasured single-sequence GGUF families.It combines metadata with a small reference set and an explicit cross-system falsification test.Source: §1 · p. 1

2. PREDICTOR

• Decode throughput is modeled from bytes per generated token, including GGUF size adjusted by active-parameter share and context-dependent KV traffic.The model also includes fixed recurrent-state traffic.Source: §2 · p. 1
• For each host–quantization pair, the decode scale factor is the median training-row ratio; unseen formats use the host-wide median.The fit uses ratio space so the fastest small models do not dominate least squares; the factor is an empirical scale, not literal utilization.Source: §2 · p. 1
• B0 uses an unfitted total-size roofline, B1 fits a host–quantization scale while charging total weights, and B2 adds active-weight and per-layer KV accounting.Comparing B1 with B2 isolates active-weight accounting under the same fitting rule.Source: §2 · p. 1
• The zero-prefix prefill model back-solves a throughput ratio against calibrated matrix throughput and twice the activated-parameter count.P1 uses a host median coefficient, whereas P2 uses a host–quantization-group median.Source: §2 · p. 1
• Prefill coefficients for unseen quantizations fall back to the host median, and greater prefix depths test scope because neither model includes a prefix term.The B2 decode diagnostic separates predictions by host, with held-out points marked orange and dashed ±25% lines.Source: §2 · p. 2, Figure 1 · p. 2

3. EXPERIMENTAL METHOD

• Measurements use a 64-GB MacBook Pro M4 Max, a 128-GB Mac Studio M4 Max, and an RTX 5080, with host-specific bandwidth and matrix-throughput calibrations.The Macs use 12 performance-core threads; the RTX uses CUDA and 16 threads.Source: §3 · p. 2
• The scored cohort contains 53 configurations across 22 GGUF files, split by base family into host-specific training and held-out sets.MacBook contributes 15 training and four held-out configurations, Studio 15 and five, and RTX 12 and two.Source: §3 · p. 2
• Fresh-process llama-bench runs measure separate 512-token prompt-processing and 128-token generation microbenchmarks at prefix depths 0, 4096, and 16384.Runs use flash attention, F16 K/V, five repetitions, and 45 seconds of settling; physical RTX residency was not instrumented.Source: §3 · p. 2
• Decode rows pass a stability gate, while prefill rows are retained despite 8/66, 0/69, and 29/42 rows exceeding 3% CV on MacBook, Studio, and RTX.Table 1 reports host-separated MAPE; decode includes three depths per configuration, while P1/P2 are scored at depth zero.Source: §3 · p. 2

4.1. Host-adapted decode

• B2 train/test MAPE is 10.4/13.1% on MacBook, 12.4/14.4% on Studio, and 10.1/36.1% on RTX.RTX test median and maximum error are 25.8% and 69.0%; B2 improves on B1 for every held-out host.Source: §4.1 · p. 2
• Active-weight accounting lowers held-out MAPE from 49.4% to 13.1% on MacBook, 55.3% to 14.4% on Studio, and 51.9% to 36.1% on RTX.Per-layer rather than uniform-global KV accounting further lowers held-out MAPE from 21.2% to 13.1%, 21.0% to 14.4%, and 37.9% to 36.1%, respectively.Source: §4.1 · pp. 2–3, Figure 2 · p. 3

4.2. Cross-host transfer is incomplete

• Leave-one-host-out transfer yields test-family MAPE of 11.6% on MacBook, 16.8% on Studio, and 36.0% on RTX, but most target files also occur on a source host.Across matched configurations, decode ratios span 1.76–3.26×, ruling out one universal multiplier.Source: §4.2 · p. 3
• The Qwen3.8-27B low-bit ladder varies by host: Q2 is 2.2% faster than IQ2 on Apple systems, while IQ2 is 13.7% faster on RTX.Across conventional Q4/Q8 pairs, Q4 is faster for decode on all three systems, but its prefill rate is usually slightly lower; file compression alone does not specify kernel efficiency.Source: §4.2 · pp. 3–4

4.3. Prefill is a negative result

• P2 held-out zero-prefix MAPE is 18.7% on MacBook, 22.2% on Studio, and 108.2% on RTX, rising at 16K to 76.2%, 68.1%, and 133.3%.P2 omits prefix-dependent work, and most host–quantization groups have only one training configuration; the authors treat prefill as unsupported under this protocol.Source: Figure 3 · p. 3, §4.3 · p. 4

4.4. Protocol and robustness checks

• Across 159 scored decode rows, within-run CV shows no obvious descriptive association with residual error (pooled Spearman ρ = −0.05).Correlated depth rows and absent Studio/RTX between-run studies limit noise attribution.Source: §4.4 · p. 4, Figure 4 · p. 4

5. LIMITATIONS AND CONCLUSION

• The RTX holdout contains only two Q4_K_M files, and the largest analytical working set exceeds reported VRAM before graph buffers, leaving physical residency unverified.No GPU clock, power, or spill telemetry was recorded; the RTX holdout also lacks an independent between-run repeatability campaign.Source: §5 · p. 4
• The active-byte proxy scales shared tensors with expert tensors and assumes balanced k/E routing, while no learning curve establishes the reference-set size needed.Tensor-level active-byte validation was unavailable.Source: §5 · p. 4
• Within the stated boundary, active-weight and metadata-aware KV corrections reduce held-out decode error on all three systems, but fitted efficiencies are not universal.Low-bit behavior differs across host/runtime stacks, with runtime versions and IQ2’s slightly different architecture remaining confounds.Source: §5 · p. 4

Companion Results Appendix: GGUF Throughput Prediction

• The appendix presents protocol-complete evidence from three hosts for the ICASSP manuscript.It is a locally generated companion appendix.Source: Companion Results Appendix: GGUF… · p. 6
• The appendix does not claim publication, archival acceptance, or independent replication.Its status statement limits what the companion results establish.Source: Companion Results Appendix: GGUF… · p. 6
• The appendix identifies itself as locally generated rather than as a publication or independent replication.This status qualifies how its evidence should be read.Source: Companion Results Appendix: GGUF… · p. 6

1 Scope and cohort

• The scored cohort has 318 rows—159 decode and 159 prefill—across 53 host–file configurations on two M4 Max systems and an RTX 5080.The 22 scored files yield 42 training and 11 held-out configurations; the broader selected protocol cohort has 354 observations including 36 reference-probe rows.Source: §1 · p. 6
• The two failed MacBook attempts for gpt-oss-120B were prompt-batch failures, while calibration probes were host-specific and none were used on RTX.The MacBook and Studio each had three named Qwen calibration probes; probes can be excluded on one host while the same file is scored on another.Source: §1 · p. 6

1.1 Cohort accounting

• A protocol duplicate is a successful protocol row removed by the atomic host–model–phase–depth selector, not an extra scored observation.The cohort accounting separates raw records, legacy rows, protocol attempts, selected observations, and scoring exclusions.Source: §1.1 · p. 6

1.2 Excluded reference-probe observations

• Reference-probe rows complete the 354-observation selected cohort and support descriptive comparisons, but are measurements rather than prediction errors.Probe-exclusion sensitivity refits B2 after adding probes back, evaluating original training rows separately from the held-out test rows.Source: §1.2 · pp. 6–7, Table 1 · p. 7

2 Aggregate prediction results

• On test rows, B2 fitted with active parameters records 13.11, 14.37, and 36.15 for MacBook Pro M4 Max, Mac Studio M4 Max, and RTX 5080, versus B1 total-parameter values 49.36, 55.25, and 51.85.Source: §2 · p. 8

2.1 Leave-one-host-out transfer

• Leave-one-host-out B2 fits use median per-quantization coefficients from the other two hosts and the target host’s calibrated bandwidth; the target formats absent from source training use a source-wide median fallback.The target set includes eligible rows on that host and mostly shares files with source hosts, so it tests host/runtime transfer rather than simultaneous model-and-hardware holdout.Source: §2.1 · p. 8

2.2 Per-layer KV correction

• Per-layer KV metadata replaces the uniform-global denominator in the refitted variants, but the supplied passage gives no resulting error comparison.The aggregate results remain host-separated because fitted efficiency factors are host-specific.Source: §2 · p. 7, §2.2 · p. 9

2.3 Decode context behavior

• All 53 scored configurations slow monotonically from 0 to 16,384 tokens, with normalized throughput measured against each model’s own depth-0 rate.The normalization supports within-model comparisons across context depths.Source: §2.3 · p. 9

2.4 Prefill scope diagnostic

• P2 prefill is fit and scored at depth zero, while Figure 1 contrasts host-separated zero-prefix bandwidth with held-out error when those coefficients are applied at larger prefix depths.The figure notes that the displayed Qwen quantizations are not a strictly identical-network ladder because IQ2 has fewer parameters and one fewer layer.Source: §2.4 · pp. 9–10

• Together, the tables reproduce every frozen metadata field for 23 selected files, while making explicit that scored host coverage is not identical.Source: §3 · p. 11
• Table 2 identifies selected models, their sources, exact file sizes, and host coverage.Source: Table 2 · p. 11, §3 · p. 11
• Table 3 reports selected-model geometry, including exact KV bytes at measured depths and the routed-to-total expert ratio.Source: Table 3 · p. 12

• The exported rows report throughput in tokens/s and absolute percentage error, ordered by host, split, model ID, and depth.Source: §4 · p. 13
• Table 4 pairs protocol-complete decode measurements with B2 predictions.Source: Table 4 · p. 13
• The retry gate uses the worst decode CV within each host–file invocation.Source: Table 4 · p. 13

5 All scored prefill observations

• Depth 0 is the primary prefill evaluation; larger-depth predictions are scope diagnostics, and P1/P2 coefficients are fit only at depth 0.Source: Table 5 · pp. 18–19, §5 · p. 18

6 Repeatability and calibration

• The six-run repeatability series is MacBook-only, with decode CV 1.613% and prefill CV 0.373%; it is a single-cell diagnostic, not a general threshold.Source: §6 · p. 23

6.1 Host settings and calibration

• The requested n_gpu_layers=99 asks for maximal offload but does not verify accelerator residency; no device-memory trace was recorded.Source: §6.1 · p. 23

7 Output-projection sensitivity

• The additive output-projection term is not selected because held-out gains are inconsistent across losses and hosts, and it partly recharges streamed weights.Source: §7 · p. 24

8 Provenance and explicit limits

• Leave-one-host-out results do not establish universal transfer, and held-out sets are small: four, five, and two configurations across the three hosts.Source: appendix generator… · p. 27

appendix generator paper/icassp2027/generate_supplement.py f557811bc7d7084fd7acba1c3d61c236b7e40ec1dc4d12757625c3da9817e934

• The derived parameter counts and per-depth KV bytes come from a frozen metadata snapshot, not a fresh independent extraction.Source: appendix generator… · p. 27
• The Qwen3.8-27B IQ2 file differs in layer count and parameter count from the other shown files, so the quantization plot is a near-family comparison, not a controlled bit-format substitution.Source: appendix generator… · p. 27