Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

Calibration and validation

• R2 = 0.910 across 93 open-weight models from 19 vendors, with 72% within 2× and 86% within 3× of known size.The resulting estimator is coarse and intended for order-of-magnitude sizing and relative ranking rather than precise parameter counts.Source: §1 · pp. 2–3, Figure 1 · p. 3

Incompressibility framework

• IKPs measure incompressible factual capacity from black-box API access, separating stored facts bounded by Shannon entropy from compressible procedural capability.The benchmark targets rare facts that cannot be derived or computed from general knowledge.Source: §1 · pp. 2–3

Densing Law falsification

• Across 100 dated open-weight models, IKP accuracy changed by +0.0013/month (95% CI [−0.0004, +0.0033]), indistinguishable from zero and rejecting the Densing prediction of +0.0129/month at p < 10−15.Factual capacity does not compress.Source: §1 · pp. 3–4

Frontier estimates and the MoE total-params rule

• 97 proprietary frontier models receive coarse effective-capacity estimates with a ∼3× prediction interval, while total MoE parameters predict knowledge better than active parameters, R2 = 0.67 vs. 0.41.The calibration places several closed models plausibly above the largest open-weight release.Source: §1 · pp. 3–4

Knowledge fingerprinting

• Rare-fact Jaccard overlap combined with hallucination similarity distinguishes weight-sharing siblings, post-training lineages, and full retrains without model weights.The fingerprinting method also separates closed-vendor version families.Source: §1 · p. 4

Open-source release

• A released toolkit, probe set, and results enable effective knowledge-capacity estimation through OpenRouter’s standard API endpoint.The release includes code and a companion website for reproducible evaluation.Source: §1 · p. 4

2.1 Knowledge Capacity of Language Models

• Knowledge capacity is theoretically tied to parameter count: models store roughly 2–4 bits of factual knowledge per parameter, while factual capacity scales with model size and training exposure.Mechanistic studies further identify transformer feed-forward layers as key-value memories containing causally editable knowledge representations.Source: §2.1 · p. 4

2.2 Long-Tail Knowledge and Scaling

• Rare-fact accuracy scales log-linearly with model size, but long-tail knowledge remains unreliable because popularity overshadowing and post-training degradation suppress obscure facts.These mechanisms motivate using increasingly obscure probes to discriminate model capacity.Source: §2.2 · pp. 4–5

2.3 Existing Model Size Estimation

• Existing parameter-count estimates rely on inference economics or output fingerprints, while standard capability benchmarks fail as size proxies because capability density changes over time.Fingerprinting can identify served models, but the cited methods do not estimate the size of unknown models; knowledge-based estimation is therefore complementary.Source: §2.3 · p. 5

2.4 Parameter Specialization

• Stronger models allocate increasingly specialized parameter vectors to factual knowledge, while MoE knowledge efficiency depends on total and active model size, expert counts, and sharing.These findings motivate analyzing effective parameters for MoE models.Source: §2.4 · p. 5

2.5 Distillation Detection

• Distillation detectors use weights, synthesized inputs, or MoE routing, whereas IKP’s knowledge fingerprints compare rare-fact overlap and shared errors using black-box API access alone.This makes IKP complementary to existing distillation-detection approaches without requiring model weights or MoE structure.Source: §2.5 · p. 5

3.1 Factual Knowledge as Incompressible Information

• IKPs target factual associations whose values cannot be derived, computed, or inferred from other facts or language regularities.Although prior distributions provide limited compression, probe design minimizes it through specific-year questions, non-predictable researcher entities, and collision filtering.Source: §3.1 · p. 6

3.2 The Capacity Bound

• ∼2N bits is the ideal factual-knowledge storage limit for N parameters, while empirical estimates reach ∼3.6 bits/param [Morris et al., 2025], and accuracy scales as acc ≈a·log(N) [Kandpal et al., 2023; Lu et al., 2024].Thus, accuracy on a probe set with known information content constrains parameter count from below.Source: §3.2 · p. 6

3.3 Why Procedural Improvements Do Not Help

• N = Nfact + Nproc + Nling, and IKPs primarily probe Nfact, so aggregate accuracy remains a lower bound on total N despite procedural or linguistic efficiency gains.The Densing Law can improve Nproc and Nling, freeing capacity for factual storage or reducing total parameters, but cannot remove the information bound on Nfact.Source: §3.3 · p. 6

3.4 The Frequency-Capacity Relationship

• Zipf’s law implies that increasing capacity memorizes progressively rarer facts, producing aggregate accuracy that scales as log(N).Each tier follows a logistic sigmoid in log N, while the mean of seven shifted sigmoids is approximately linear across a wide range.Source: §3.4 · p. 6

4.1 Probe Generation

• The seven-tier probe set combines 401 LLM-generated candidates with corpus-grounded Wikidata and DBLP/arXiv probes targeting harder factual knowledge.The full design contains 1,400 probes, with 200 per tier; corpus-grounded sources dominate T3–T7 because LLM generation is circular for obscure facts.Source: §4.1 · p. 7
• 1,311 cleaned probes remain after collision and ambiguity audits, with calibration R2 improving from 0.900 to 0.910 and frontier estimates shifting by less than 5%.The released pipeline initially contains 1,400 probes; 89 are flagged and removed, and all results use the cleaned subset.Source: §4.1 · p. 8

4.2 Landmark-Based Tier Assignment

• Tiers are assigned by six landmark models from 0.5B to frontier, retaining probes answered by landmark Tk but not Tk−1 and dropping approximately 15% non-monotonic cases.The landmarks span Qwen 2.5 0.5B, Qwen 2.5 7B, Qwen 3 32B, Qwen 3 235B, Kimi K2.5 1T, and Gemini 3.1 Pro Frontier.Source: §4.2 · p. 8

4.3 Scoring

• At λ = 0, IKP accuracy is the fraction of correctly answered facts, with seven-tier scores averaged equally and researcher responses weighted STRONG 1.0, WEAK 0.5, and refusals or wrong answers 0.Across λ ∈[0, −2], calibration R2 varies by at most 0.03, while estimates can swing 2–3× by vendor; negative-penalty flooring can inflate frontier estimates several-fold.Source: §4.3 · p. 9

4.4 Calibration Curve

• The calibration fits A = α · log10(N) + β by ordinary least squares on open-weight models, then estimates target size by inverting the regression.A denotes aggregate no-penalty accuracy and N denotes parameter count in billions; reported R2 is the forward-direction fit predicting accuracy from size.Source: §4.4 · p. 9

• 1,400 probes span seven tiers and 16 domains, including researcher subfields, founding years, geography, history, science, and culture.The probe set contains 200 probes per tier.Source: §5 · p. 9
• 201 models from 27 vendors are evaluated, with 93 open-weight models calibrating parameter-count estimates and 97 proprietary models serving as targets.The open-weight calibration range is 135M to 1,600B parameters; proprietary targets include GPT-4 [OpenAI, 2023].Source: §5 · p. 9
• Gemini 3 Flash Preview judges responses at temperature = 0 and low reasoning effort, with an estimated 0.1–0.2% error rate verified by manual audit.Source: §5 · p. 9
• The 345 researcher probes require a subfield plus verifiable evidence, while a four-way rubric distinguishes strong and weak correctness, wrong answers, and refusals.CS scoping reduces name collisions, and artifact requirements discourage plausible but unattested labels.Source: §5 · pp. 9–10

6.1 Calibration Curve Quality / 6.2 Leave-One-Out Cross-Validation

• IKP calibration achieves R2 = 0.910, with each 10× parameter increase adding ~15.9 percentage points; LOO-CV yields 1.48× median error, 72% within 2×, and 86% within 3×.For MoE models, total parameters predict factual capacity better than active parameters, with R2 = 0.67 versus 0.41.Source: §6.1 · p. 10, Figure 2 · p. 11, §6.2 · p. 11

6.3 Frontier Model Estimates

• GPT-5.5 Pro and GPT-5.5 lead proprietary models at ~5.3T and ~4.7T effective parameters, while a broad ~2T cluster indicates frontier capacities within ~1.5×.Estimates are 90% prediction intervals, not point values; the smallest estimated model is ~9B, giving a ~580× frontier range.Source: Table 2 · p. 13, §6.3 · p. 13

6.4 Per-Tier Discrimination

• T3 is the strongest single-tier proxy for overall ranking, while T6 separates frontier models: GPT-5.5-Pro reaches 72.2% versus GPT-5.5 at 67.4%.T1–T2 largely saturate, T3–T5 provide most discrimination, and T7 remains least saturated but is contaminated by guessing and miscategorized probes.Source: §6.4 · p. 14, Figure 4 · p. 16

6.5 Thinking Mode Analysis / 6.6 Cross-Generation Improvement

• Across 30 base/think pairs, thinking improves accuracy in 18 cases by a mean 2.3 percentage points, with benefits peaking at T3–T4 and vanishing at T7.The range is −11.0 to +17.3 percentage points; DeepSeek V4 Pro has the largest gain (+17.3 pp), while Claude 3.7 Sonnet has the largest regression (−11.0 pp).Source: §6.5 · p. 15, Figure 5 · p. 15

6.7 Densing Law Falsification

• +0.0013 IKP-accuracy points per month is indistinguishable from zero and rejects the Densing prediction of +0.0129/month, showing no per-parameter factual improvement over release time.Adding release date increases R2 by only +0.0033; newer models’ apparent time advantage is explained by their larger sizes.Source: §6.7 · pp. 17–18, Figure 6 · p. 17

6.8 Why not just use MMLU? Standard knowledge benchmarks as size proxies

• IKP outperforms standard benchmarks as a parameter proxy on every matched subset, with GPQA Diamond at R2 = 0.52 versus 0.90 and SimpleQA at 0.90 versus 0.99.GPQA gains approximately 2 pp/month at fixed size, whereas SimpleQA drifts only +0.03 pp/month; IKP also covers models from 135M to 1.6T.Source: Table 3 · p. 18, §6.8 · p. 19, Figure 7 · p. 19

6.9 Knowledge Fingerprinting: Lineage vs. Retraining

• Hallucination-similarity separates shared-base, lineage, and retrained models: HSS ≥0.30 indicates shared weights, 0.10–0.30 lineage, and <0.10 retraining.All GPT-5→5.x transitions fall in the retrained regime, while GPT-5/GPT-5-pro remain shared-base at HSS = 0.56.Source: §6.9 · pp. 20–22, Figure 9 · p. 22

6.10 What Determines Whether an LLM Knows Something? / 6.10.1 Researcher Probes

• Researcher recognition correlates moderately with citations (Spearman ρ = 0.575), but citations and h-index explain only ~35% of individual-probe variance, making high prominence necessary but insufficient.No researcher below 50 citations exceeds 15% recognition, while many researchers with 10K+ citations remain below 25%; named artifacts can produce exceptions.Source: §6.10 · p. 23, §6.10.1 · pp. 23–24, Figure 10 · p. 24

6.10.1 Researcher Probes

• Researchers attached to widely adopted artifacts are recognized at least 86% of the time, versus 0.34 recognition without such artifacts, regardless of citation count.A tool with ≥10K GitHub stars or a dedicated Wikipedia page has stronger recognition effects than an order-of-magnitude increase in citations.Source: §6.10.1 · p. 26
• Recognition depends multiplicatively on citations, name uniqueness, named-artifact amplification, and subfield ecosystem density rather than citation count alone.Common East Asian names are recognized at 22.6% versus 44.6% for uniquely spelled names, while ML researchers avoid the low-recognition floor observed in Systems because of denser derivative content; name-collision errors affect three of four Systems high-citation low-recognition cases.Source: §6.10.1 · pp. 26–27

6.10.2 Factual Probes About Entities

• Pageviews predict entity recognition better than sitelinks, with r = 0.774 versus r = 0.502 and pageviews subsuming sitelinks in joint regression.Pageviews approximate English-language discourse volume, while sitelinks measure multilingual coverage.Source: §6.10.2 · p. 27
• Specific facts require their own mention-frequency threshold: names scale with prominence, but founding years remain sparse and reach 37–47% wrong-answer rates versus <1% for capitals.Journal founding years are +0.22 easier than sitelinks predict, whereas place founding years are −0.31 harder; entities founded in 1900–1950 have mean recognition 0.40 versus 0.25 before 1800 and 0.26 after 2000.Source: §6.10.2 · pp. 27–28

6.10.3 Synthesis: Effective Mention Frequency

• LLM knowledge is governed by effective mention frequency for the specific fact, making downstream derivative text and structured metadata stronger predictors than citations, sitelinks, or h-index.Named artifacts and bibliographically enforced metadata are especially well memorized, while hard IKP probes clear the mention-frequency threshold only for the largest models.Source: §6.10.3 · p. 28

6.11 Key Empirical Findings

• Aggregate accuracy rises ∼15.9 percentage points per 10× parameter increase, while total parameters predict MoE knowledge better than active parameters (R2 = 0.67 versus 0.41).The scaling result spans 135M–1,600B parameters; the MoE pattern is consistent with factual storage being distributed across experts.Source: §6.11 · p. 29
• Thinking mode provides only a flat ∼2 pp bonus, T7 remains unsaturated at median ∼8% and frontier ∼22–28%, and safety tuning can suppress measured knowledge by 21.5 pp.The thinking benefit peaks at T3–T4 and vanishes at T7; Sonnet 4’s T5 refusal rate rises from 15% to 85%, indicating refusal policy rather than capacity.Source: §6.11 · p. 29
• Hallucination rates distinguish vendors, while hallucination-answer similarity separates shared-base models (HSS ≥0.30) from independent retrains (HSS < 0.10).Gemma hallucinates on 89–97% of unknown T5–T7 probes versus 3–28% for Claude; several nominal point releases fall into the retrained regime, while others preserve clear lineages.Source: §6.11 · p. 29

7.1 The Silent Tax of Safety Tuning

• A 21.5-percentage-point Sonnet gap may reflect refusal policy rather than capacity: Sonnet 4 scores 51.1% versus Claude 3.7 Sonnet’s 72.6%.Sonnet 4 refuses 85% of probes on T5, compared with 15% for Sonnet 3.7; under λ = 0, such refusals score as ignorance, making estimates lower bounds.Source: §7.1 · p. 29

7.2 Hallucination Rate as a Vendor Signature

• OpenAI’s hallucination rate shifts from 98–99% for GPT-4.1 variants to 3–4% for GPT-5 nano/mini, making wrong-versus-refuse behavior a vendor-generational signature.Google spans 10–90% across larger Gemma and Gemini models, while Anthropic averages 11%, with thinking variants as low as 3%.Source: §7.2 · pp. 29–30

7.3 Knowledge Regression Between Generations

• GPT-3.5 Turbo outperforms GPT-4o-mini on IKP, 62.2% versus 55.4%, while later Claude and DeepSeek generations partially recover earlier regressions.The discussion attributes regressions to increasing RLHF conservatism and possible size differences, which cannot be cleanly separated.Source: §7.3 · p. 30

7.4 Robustness to Benchmark Saturation

• IKP is structurally resistant to benchmark saturation because it measures long-tail stored facts rather than reusable procedures, while its generation pipeline can produce fresh probes from disjoint entities.Direct contamination of the released probe file remains a separate concern.Source: §7.4 · p. 30

7.5 Limitations

• The 90% prediction interval spans ∼3.20× in either direction, and estimates above ∼1T parameters are less certain because only two calibration anchors exceed 1T.The calibration absorbs training-data and post-training variation as residual; combining IKP with inference-economics constraints could narrow uncertainty.Source: §7.5 · pp. 30–31
• IKP results can be distorted by retrieval augmentation, landmark circularity, nondeterministic API responses, probe contamination, uneven tier allocation, and misclassified T7 probes.LOO-CV gives R2 = 0.907 for landmark robustness, while nine T7 probes exceed 25% correct across 201 models and should be reassigned.Source: §7.5 · p. 31

7.6 Wikidata Long-Tail Data Quality

• Auditing 557 T5–T7 Wikidata probes led to 119 changes, expanded fact-type diversity from a founding-year monoculture to 16 types, and required recalibration after answer edits.The audit found incorrect facts, ambiguous questions, contested attributions, and sharply varying reliability across properties, including 25% for headquarters.Source: §7.6 · p. 32

7.7 Open Questions

• Future work should test when T7 saturates, partition within-family score variation, and recover RLHF-hidden answers to estimate the alignment tax.The best models currently answer only ∼25–28% of T7, while recovery methods could bound capacity obscured by safety tuning.Source: §7.7 · p. 32

• Knowledge fingerprints survive fine-tuning and distillation.Source: §4 · p. 32
• The section asks whether fingerprints also persist through substantial continued pretraining on fresh data.Source: §4 · p. 32
• If they persist, fingerprints could enable training-free provenance tools for open-weight licensing enforcement.Source: §4 · p. 32

8 Conclusion

• R2 = 0.910 across 93 open models supports IKP estimates of effective knowledge capacity, with 1.48× median leave-one-out error and 72% within 2×.Total parameters predict MoE knowledge better than active parameters (R2 = 0.67 vs 0.41), and the paper reports effective-capacity estimates for 97 proprietary models.Source: §8 · p. 33

A Probe Generation Methodology

• The probe pipeline calibrates seven tiers to model-size ranges and combines LLM-generated, Wikidata-grounded, researcher, and manually authored probes with independent verification.Phase A contributes 401 of 1,400 probes, while Phase B supplies externally verifiable corpus-grounded probes; tier assignments are checked against the landmark ladder.Source: §A.1 · p. 39, §A.2 · p. 39, §A.3 · pp. 39–41

A.3 Phase B: Corpus-Grounded Probes

• The researcher-probe artifact requirement forces verifiable evidence beyond guessed subfield labels, reducing failures from name collisions and plausible fabrication.The released format requests a subfield plus an associated paper, system, institution, or co-author, with structured evidence bundles and four-way response classification.Source: §A.3 · pp. 40–41

A.4 Quality Filters and Drop Rates

• The quality filters remove computable facts, non-monotonic probes, ambiguous names, contaminated machine-learning researchers, and poorly grounded templates.Grounded Wikidata templates were independently recalibrated because grounding made approximately 30% of T7 candidates too easy and approximately 5% non-monotonic, leaving approximately 65% in the released T6/T7 pool.Source: §A.4 · pp. 41–42

A.5 Final Composition

• The final composition diversifies frontier-difficulty probes beyond founding years, with 259 of 600 T5–T7 probes covering computer-science researcher subfields.Approximately 50% of T6/T7 Wikidata probes were replaced with diverse fact types while preserving unambiguous answers.Source: §A.5 · p. 42, Table 8 · p. 42

A.6 Verification Procedure

• Gold answers are source-specifically verified and corrected or dropped when conflicts arise, with corrected Wikidata probes recalibrated because answer changes alter tier assignments.Researcher gold subfields come from dominant DBLP venue tags, while Phase A answers receive independent LLM verification and manual spot-checking.Source: §A.6 · pp. 42–43

B Hallucination Penalty Sensitivity

• λ = 0 is adopted because fit quality remains nearly flat across penalties while avoiding both the hallucination-penalty and flooring choices.Across λ values, R2 ranges approximately 0.91–0.94 and the 90% prediction-interval factor approximately 2.6–3.2×, whereas individual estimates can shift 2–3×.Source: B Hallucination Penalty Sensitivity · pp. 43, 45, Figure 11 · p. 45

B.1 Flooring and its interaction with λ

• Unfloored negative-penalty scoring degrades calibration sharply, from R2 = 0.91 to 0.72 and prediction-interval factors from 3.2× to 9.9×; λ = 0 makes flooring irrelevant.An earlier discrepancy under λ = −1.0 reported GPT-5.5 as 4.9T floored versus 1.5T unfloored, whereas no-penalty scores are identical with or without flooring.Source: §B.1 · pp. 45–46, Table 11 · p. 46

C Dense vs. MoE Calibration

• R^2 = 0.67 for MoE-total versus 0.41 for active parameters, showing factual capacity tracks total weights; the combined curve remains headline because MoE-only validation does not improve.The combined curve has median fold error 1.52× versus 1.58× for MoE-only models, while per-model behavioral deviations exceed the architecture gap.Source: Table 12 · p. 46, C Dense vs. MoE Calibration · pp. 46–47
• Switching to the MoE-total curve raises top frontier estimates by ∼18%, but combined-curve values remain headline within the 3.2× prediction interval.Examples include Claude Fable 5 rising from ∼3.5T to ∼4.0T and GPT-5.5 from ∼4.7T to ∼5.5T.Source: Table 13 · p. 47, C Dense vs. MoE Calibration · p. 47

D Wikidata Long-Tail Audit; D.1 Failure-Mode Taxonomy

• The audit organizes every problematic probe into five reusable failure-mode buckets spanning incorrect facts, extraction bugs, ambiguity, contested entities, and multi-actor attribution.The appendix reports that answer-field corrections require recalibrating prior tier assignments.Source: D Wikidata Long-Tail Audit · p. 47, §D.1 · p. 47

A. Wikidata-itself-wrong.

• ∼75% of headquarters probes were stale, while other Wikidata errors involved wrong field semantics, circular claims, or incorrect years and values.Examples include outdated Roku and Vista Outdoor headquarters and mistaken sculpture-creator attributions.Source: A. Wikidata-itself-wrong · pp. 47–48

B. Pipeline extraction bugs (on our side).

• Pipeline extraction bugs can substitute a different date field even when Wikidata’s intended inception value is correct.The George Eastman Museum was extracted as 1905 instead of Wikidata’s P571 value of 1947.Source: B. Pipeline extraction bugs (on our… · p. 48

C. Probe-construction ambiguity (Wikidata correct, question unanswerable).

• Probe-construction ambiguity was the most prevalent class, especially for generic religious-art, geographic-feature, place, and institution names.Examples include “Stone Bridge,” “Putnam,” and “Maritime Museum,” where multiple reasonable referents exist.Source: C. Probe-construction ambiguity… · p. 48

D. Politically/legally contested entities; E. Multi-actor attribution

• Contested entities and multi-actor attributions can make Wikidata’s answer reasonable but non-unique, requiring probe dropping, caveats, or credit for co-contributors.The audit distinguishes disputed sovereignty and statehood from valid alternative founders, co-authors, or screenwriters.Source: D. Politically/legally contested… · p. 48, E. Multi-actor attribution · p. 48

D.2 Per-Round Sourcing Telemetry

• 10 SPARQL rounds produced 100–300 candidates each, with empirical verification rates guiding later sourcing and total effort of ∼10,000 API calls plus ∼120 manual checks.Replacement probes were calibrated through the landmark ladder and targeted fact types that verified poorly.Source: Table 14 · p. 49, §D.2 · p. 49

D.3 Per-Fact-Type Reliability Ranking

• P403 river mouth achieved a 100% pass rate, whereas P159 headquarters achieved only 25% because Wikidata often retains outdated company locations.The ranking aggregates empirical reliability at T6/T7 obscurity across rounds 4–7.Source: §D.3 · p. 49
• P57 director achieved 67–100% attribution reliability, but generic film-title ambiguity required templates adding year, country, and genre.“Night Train” and “Hit Man” are cited examples of ambiguous titles.Source: §D.3 · p. 49
• P17 country scored 75–100%, but politically contested entities consistently failed web cross-checks because Wikidata selects one claimant.The affected examples include Crimea, the Spratlys, and South Georgia.Source: §D.3 · p. 49
• P571 inception reached ∼95% reliability, with most failures caused by pipeline date-field selection bugs rather than Wikidata errors.It is described as the most-curated date field.Source: §D.3 · p. 49
• P170 creator for sculptures scored ∼70%, with foundries frequently recorded instead of sculptors, while P112 founder scored 75% and sometimes returned an organization rather than a person.These semantic mismatches make otherwise direct questions awkward or misleading.Source: §D.3 · p. 49
• P58 screenwriter scored 50%, P170 creator for paintings was unusable at T7 because of title collisions, and P138 named after showed mixed reliability.Screenwriter errors conflate “story by” with “screenplay by”; painting probes had ∼100% title-collision rates at T6/T7.Source: §D.3 · pp. 49–50

E Full Model Results

• T4 has the widest interquartile accuracy range, while T1–T2 are near ceiling and T6–T7 distinguish only the strongest models.Figure 12 summarizes per-tier accuracy across all 201 models, with T1–T2 median accuracy above 98%.Source: Figure 12 · p. 50
• Anthropic models hallucinate at 10% on beyond-knowledge T5–T7 probes, versus 58–66% for Google models.These vendor-specific hallucination differences directly affect IKP scoring.Source: Figure 13 · p. 51

Full regression results

• Time coefficients are statistically indistinguishable from zero once log10(NB) is controlled, while the time-only model’s R2 = 0.22 collapses after parameter count enters.The regression analysis uses OLS with IKP accuracy (λ = 0) as the dependent variable and n = 100 models from 23 vendors.Source: Table 17 · p. 64, F Densing Law Falsification: Full… · p. 64

Tests against the Densing prediction

• +0.01285/month is the Densing target, but every specification rejects it with |t| > 10 while failing to reject zero time trend.The Densing target is derived from the observed log10(N) slope β1 = 0.149; all tests have p > 0.3 for zero trend.Source: Table 18 · p. 64, F Densing Law Falsification: Full… · p. 64

Bootstrap inference

• The 4000-replicate bootstrap estimates a 95% time-coefficient CI of [−0.00037, +0.00326], far below the Densing target.The upper confidence bound remains approximately 11× standard errors below the prediction.Source: F Densing Law Falsification: Full… · pp. 64–65

Robustness

• Robustness checks retain the null time trend after excluding distilled or sub-1B models, while active-parameter regressions produce an artifact from MoE parameter allocation.The preferred specification uses total parameters because MoE models store knowledge across all experts, whereas active parameters correlate with release date through changing architecture mix.Source: F Densing Law Falsification: Full… · p. 65

Date uncertainty

• Small release-date noise would inflate standard errors but not shift estimates in expectation; hiding a Densing-sized effect would require unsupported systematic backdating exceeding three months per post-2025 release.Post-January-2026 dates were collected contemporaneously, while other dates came from vendor-announced releases.Source: F Densing Law Falsification: Full… · p. 65

GPT-5 family structure

• GPT-5 T5 accuracy separates nano (5%), mini (25%), and base (80%), indicating an approximately 16× knowledge gap and a higher pro shelf.GPT-5.4-pro reaches 77.8% overall accuracy, the strongest OpenAI model below the 5.5 generation.Source: G Cross-Generation and Model Family… · p. 65, Figure 15 · p. 67
• GPT-5.4-nano falls to 59.0% on T3 from GPT-5-nano’s 75.4%, while fingerprinting supports a shared GPT-5, 5-pro, and 5-think base followed by retraining.Each .x release from 5.1 through 5.4 sits in the retrained regime relative to its predecessor; only 5.2-pro and 5.4-pro remain lineage-tied to their base.Source: G Cross-Generation and Model Family… · p. 65

Cross-generation evolution

• GLM shows the steadiest improvement across six generations, while other families exhibit occasional knowledge regressions.Figure 14 traces knowledge evolution across seven major families and highlights regressions associated with efficiency optimization or safety tuning.Source: G Cross-Generation and Model Family… · p. 65, Figure 14 · p. 66

H Knowledge Fingerprinting: Detailed Tables

• HSS is preferred because independent models rarely share normalized wrong answers, while weight-sharing siblings converge on 30–55% of joint-wrong probes; HSS = 0.30 is the empirical separation threshold.Raw Jaccard can remain high for independent models, while lift is noisy when correct sets are small.Source: H Knowledge Fingerprinting: Detailed… · p. 67
• Nemotron-70B versus Llama-3.1-70B shows why all three metrics are needed: J = 0.81 and lift = 6.8 indicate a shared base, but HSS = 0.08 reflects post-training changes to wrong-answer wording.Pairs are classified as shared-base only when J, lift, and HSS agree; one-sided signals are reported as possible lineage.Source: H Knowledge Fingerprinting: Detailed… · p. 68, Limitations · p. 69
• Fourteen thousand cross-vendor comparisons flag limited outliers, with Baidu ERNIE 4.5-300B-A47B exceeding the threshold against four independent models and Llama 3.1 70B appearing against many models.The ERNIE pattern is consistent with training on mixtures of distilled Western frontier outputs, whereas Llama’s pattern is consistent with widespread use for synthetic-data generation but does not prove direct distillation.Source: H Knowledge Fingerprinting: Detailed… · p. 68
• The method cannot distinguish direct distillation from shared training data, and HSS is noisy when rare-fact joint-wrong samples are small; expanding probes into T6–T7 would improve resolution.Models scoring 70–80% on T5–T6 jointly miss only 20–50 probes, limiting statistical power.Source: Limitations · p. 69
• Known-provenance controls show elevated shared-training signals, but heavily post-trained derivatives can suppress HSS, reinforcing the recommendation to interpret J, lift, and HSS jointly.Every control pair with at least 10 joint-wrong probes had lift 2.1–7.9 and J 0.15–0.81.Source: Limitations · p. 69, Table 19 · p. 69
• Across tracked families, consecutive releases separate into shared-base variants, incremental lineages, and full retrains, with Table 20 reporting J, lift, HSS, both_w, and classifications.Shared-base variants include reasoning or pro versions; incremental lineages include DeepSeek, GLM, Kimi, Nova, and Command families; frequent retrains include Gemini, Gemma, Qwen, GPT-5.x, and Opus 4.6/4.7.Source: Limitations · p. 69, Table 20 · p. 70

I Case Studies

• Case studies present concrete probe questions and truncated verbatim model responses to illustrate the fingerprinting findings from the main text.The examples are intended to make the reported response patterns directly inspectable.Source: I Case Studies · p. 75

I.1 USTC Hackergame — Watching a Fact Arrive Over Three Years

• Claude 3.7 Sonnet was the first tested model to list real USTC Hackergame challenges, after GPT-4o repeatedly hallucinated a deterministic fake list.The transition occurred in a ∼9-month window, and later Claude, GPT, and Kimi models inherited or extended the knowledge.Source: §I.1 · pp. 75–76, Table 22 · p. 77
• Knowing that USTC Hackergame began in 2014 did not imply knowing its yearly challenges, showing that structural facts and long-tail content can be stored independently.DeepSeek V4 Pro correctly stated the start year but fabricated per-year challenges, while refusal and hallucination policies also varied across vendors.Source: §I.1 · p. 76
• IKP operationalizes these observations across 1,400 facts and 201 models with quantitative scoring rather than human verification.The case study probes 12 representative models, whereas the systematic probe set generalizes the same type of long-tail knowledge measurement.Source: §I.1 · pp. 75–76

I.2 Tier Discrimination: Universal Knowledge (T1); I.3 Tier Discrimination: Mid-Tier Knowledge (T3); I.4 Tier Discrimination: Frontier-Only Knowledge (T6)

• T1 facts are universal, T3 facts separate small from larger models, and T6 facts are typically answered only by the largest frontier models.Even 1B-parameter models answer T1 probes correctly; models above ∼30B typically answer T3 probes, while strong models such as GPT-4o can still refuse T6 probes and top-scoring Gemini models retrieve them.Source: §I.3 · p. 78, §I.2 · p. 78

I.5 Hallucination vs. Refusal; I.6 Name Collision in Researcher Probes; Claude Opus 4.6; GPT-4o-mini; GPT-4o

• Refusal is preferable to confident hallucination in IKP evaluation, but researcher probes remain vulnerable to name collision that produces plausible answers about the wrong person.Claude Opus 4.6 answered or refused on the examples, GPT-4o-mini hallucinated Zhongzhi Luan’s field, and it confused wireless researcher Geoffrey Ye Li with machine-learning researchers.Source: §I.5 · p. 79, Claude Opus 4.6 · pp. 79–80, GPT-4o-mini · pp. 79–80

I.7 Thinking Mode Enabling Correct Recall; Claude Sonnet 4; Claude Sonnet 4 (think)

• Thinking mode enabled correct recall when base models refused: Claude Sonnet 4 identified Fujiwara no Teika as the compiler of Hyakunin Isshu.The thinking trace helped reason through partial knowledge rather than merely retrieving an immediately available answer.Source: §I.7 · p. 80, Claude Sonnet 4 · p. 80, Claude Sonnet 4 (think) · p. 81

I.8 Thinking Mode Correcting Hallucination; DeepSeek-V3.2; DeepSeek-V3.2 (think)

• Thinking mode also corrected a hallucinated historical date, changing DeepSeek-V3.2’s answer from 1529 to the gold answer 1683 for the repelled Ottoman siege of Vienna.The variant disambiguated the targeted 1683 siege from the first siege in 1529.Source: §I.8 · p. 81, DeepSeek-V3.2 (think) · p. 81, DeepSeek-V3.2 · p. 81

J Ethical Considerations

• The estimates measure effective capacity rather than literal architecture, provide only ∼3.2× precision, and should support transparency rather than competitive surveillance.All estimates use public behavioral measurements and an open-weight calibration set, without internal or confidential vendor access.Source: J Ethical Considerations · p. 81