Source-linked AI summary
Time Capsule of Testable Human Knowledge: 41 Years of Jeopardy! in a Single Free Local Model
David Noever, Forrest McKee
TL;DR
The paper asks whether Watson-like question-answering capability can be compressed into a portable local model and tested beyond training-data exposure. It evaluates a 9 GB Qwen2.5-14B model across the complete 529,939-clue Jeopardy! corpus with a post-cutoff held-out test. The local model answers 67.0% of all clues and 65% of post-cutoff clues, while Claude Opus 4.8 answers 95% post-cutoff, showing compact transferable capability without making the file a lossless archive.
Problem
The paper addresses whether Watson-like question-answering ability can fit in a small local model and be evaluated beyond absorbed training material.
Method
The authors run Qwen2.5-14B, quantized to 4 bits and approximately 9 GB, across the complete 529,939-clue, 41-season Jeopardy! corpus with a post-cutoff held-out condition.
Results
67.0% of in-distribution clues and 65% of post-cutoff clues are answered by the local 14B model, while Claude Opus 4.8 answers 95% post-cutoff.
Takeaways & Limitations
Broad question-answering competence can be compressed into a portable file that answers clues beyond its training cutoff rather than serving as a lookup table.
Takeaways & Limitations
The model is a compressed, lossy representation, and the forced-response protocol with conservative lexical scoring understates precision achievable with confidence gating.
Abstract
from arXiv · showhide
In 2011, IBM's Watson was something like a sealed capsule of its era's queryable knowledge. Its DeepQA system defeated the strongest human Jeopardy! champions, but the knowledge that let it do so lived in a curated billion-document corpus running on a cluster of POWER7 servers, frozen at build time and impossible to move or copy. We show that the same kind of artifact, a snapshot of what a culture can answer, is now portable and essentially free. We evaluate a single 9 GB open-weight model (Qwen2.5-14B, 4-bit) against the complete open Jeopardy! clue dataset, 529,939 clues across all 41 broadcast seasons from 1984 to 2025. To our knowledge this is the first time a model has been run over the full corpus. The 41 years mark only how long the questions were collected. What they test is far older and broader: the accumulated body of human general knowledge a culture considers worth knowing, from ancient history and dead languages to science, literature, and geography, with a verified answer for every item. The model answers 67.0% of all clues under a strict forced-response protocol with exact and fuzzy matching, and exceeds 85% on factoid categories. We treat training-data exposure as something both systems share rather than a flaw unique to language models. Watson's case is in fact the more extreme one. Its corpus was assembled to contain Jeopardy answers and it was tuned on past clues, and it could not answer anything outside that curated distribution. The decisive test is whether a model can answer clues that did not exist when it was built. On clues aired after its training cutoff, the local model holds 65% and Claude Opus 4.8 holds 95%, while Watson by construction scores zero. The capability survives the move from a server room to a file you could seal in a time capsule, and unlike Watson it is not frozen to its own moment.
1 Introduction
The paper asks how much of Watson’s question-answering ability can fit in a small local model, while treating training-data exposure as a shared condition. It benchmarks modern open models on the complete 41-season Jeopardy! corpus and tests whether they answer clues beyond their absorbed material.
- Transformer models acquire broad world knowledge through general pre-training and answer in a single forward pass without task-specific retrieval.
- The paper treats prior exposure to source material symmetrically, noting that Watson operated with a curated corpus and past-clue tuning.
- The study benchmarks language models against all 529,939 clues from Jeopardy!’s 41 broadcast seasons.The authors describe this as the first such complete-corpus benchmark, as far as they know.
- Its evaluation analyzes performance by difficulty, question type, and broadcast year across open transformer models ranging from 4.7 to 19 GB.
- A post-cutoff held-out test measures whether models can answer clues that did not exist during training rather than merely policing contamination.
2 Background
Jeopardy! is a demanding question-answering benchmark because its clues use dense, adversarial language and require responses in question form. Watson addressed this difficulty with selective responding, making coverage and precision distinct measures, while benchmarks can lose discriminative power as systems improve.
- Jeopardy! clues are declarative, category-labeled, and linguistically adversarial, with wordplay, misdirection, wagering, and question-form responses.
- Watson’s confidence threshold traded coverage for precision by answering only clues above a learned response threshold.
- Benchmark saturation occurs when a task that once distinguished systems becomes solved within a few model generations.
- Newer evaluations respond by constructing harder expert-level or specialized targets, while this paper measures how far Jeopardy!’s benchmark has fallen.
3 Method
The method exhaustively evaluates an open local model on a structured, full-coverage Jeopardy! dataset using reproducible difficulty and question-type slices. It combines a lexical correctness rule with a single-GPU sweep designed for large-scale, fair comparison.
- Dataset: The dataset contains 529,939 clues across all 41 broadcast seasons, with round, value, category, clue, canonical response, and air date recorded.
- Dataset: Clues are grouped into EASY, MEDIUM, and HARD tiers from normalized dollar values, while DAILY-DOUBLE and FINAL remain separate categories.
- Dataset: A lightweight regular-expression taxonomy assigns question types including person, place, science, definition, date/year, quote/fill-in, wordplay, and general.
- Evaluation setup: Qwen2.5-14B-Instruct is evaluated locally in 4-bit quantization as an approximately 9 GB model using greedy decoding.
- Evaluation setup: The complete sweep took about 4.6 hours on one workstation-class GPU at a sustained median latency of 0.83 s per clue.
- Scoring: Primary lexical accuracy uses normalization followed by exact or fuzzy token-set matching at an 88 threshold, providing reproducibility but conservatively rejecting some semantically correct answers.
4 Results
Across the complete Jeopardy! corpus, compact open models reproduce much of Watson-class question answering, with performance varying more by question type than by model size or clue value. The model remains strong on factoid categories but drops on wordplay and Final Jeopardy clues.
- 4.1 Full-corpus accuracy: 67.0% lexical accuracy: Qwen2.5-14B answers 529,939 clues under forced response, compared with Watson’s 75.4% all-clue figure.The 9 GB model reaches this result using a much smaller and cheaper setup than Watson’s system.
- 4.2 A class of models, not one model: 59.7% to 73.9%: all five open models answer a clear majority of clues, with Qwen2.5-32B within a point of Watson’s 75.4%.The models span roughly 4.7 to 19 GB and were evaluated on a stratified 1,384-clue sample with judge-rescued scoring.
- 4.2 A class of models, not one model: Consumer-runnable open models reproduce most Watson-class question answering across architectures, vendors, and a four-fold size range.The authors present this as a class property rather than a quirk of one checkpoint, not as a scaling-law claim.
- 4.3 Difficulty: 71% to 67% to 64%: accuracy is nearly flat from EASY through HARD tiers, while FINAL clues fall to 48%.Daily Double clues score 67%; dollar value therefore barely predicts model accuracy.
- 4.4 Question type: 88% and 87%: SCIENCE and PLACE are the strongest question types, while WORDPLAY is lowest at 58%.Factoid categories have crisp canonical answers, whereas wordplay requires manipulating surface forms.
- 4.5 The year curve and shared exposure: 65% to 70%: accuracy stays essentially flat across individual broadcast years from the late 1990s through 2025.The flat curve is consistent with absorbed historical material, but it does not by itself establish generalization beyond that distribution.
5 Out-of-Distribution Reach
The held-out post-cutoff test measures whether Jeopardy! performance transfers beyond clues plausibly seen during training. Both models retain nearly all their accuracy on post-cutoff clues, unlike Watson, whose fixed corpus could not support this regime.
- Cross-model comparison: Watson could not enter the post-cutoff regime because its corpus and scoring models were fixed at build time.The comparison distinguishes historical corpus exposure from the ability to answer questions outside that absorbed material.
- Held-out validation: 65% post-cutoff accuracy leaves Qwen2.5-14B only two points below its 67.0% in-distribution score.The gap is presented as evidence of transferable knowledge rather than clue-specific recall.
- Held-out validation: 95% post-cutoff accuracy leaves Claude Opus 4.8 one point below its 96% in-distribution score.The frontier model’s advantage therefore persists on clues aired after its knowledge cutoff.
- Practical reach: The local 14B reaches 65% on genuinely unseen clues and lands close to Watson’s 75.4% all-clue figure on backpack-sized hardware.The passage compares held-out local-model performance with Watson’s full-set reference while emphasizing the difference in deployment scale.
- Cross-model comparison: 67→65% and 96→95% are the held-out transitions reported for the local and frontier models, respectively.Figure 6 frames both small declines as the signature of transferable knowledge rather than memorized clues.
6 Why the Benchmark Aged
Jeopardy! became easier for modern models because instruction tuning largely solved indirect linguistic comprehension, while symbolic word manipulation remains difficult. The benchmark’s challenge therefore split into capabilities that aged differently rather than disappearing uniformly.
- Why the benchmark aged: Instruction-tuned models largely solve parsing of indirect phrasing, the explicit comprehension problem that required substantial Watson engineering.The paper characterizes this capability as a default property of general models and effectively free.
- What instruction-tuned models get for free: 71%, 66%, 64%, and 67% are the reported accuracies for EASY, MEDIUM, HARD, and DAILY-DOUBLE tiers, respectively.The nearly flat pattern indicates that clue dollar value and its phrasing-based difficulty grading barely predict accuracy.
- What remains genuinely hard: WORDPLAY scores 58%, versus 88% for science and 87% for place, showing that surface-form manipulation remains harder than fact retrieval.Wordplay requires operating over letters and sounds rather than only retrieving knowledge about the world.
- What remains genuinely hard: FINAL JEOPARDY scores 48%, making it the lowest-performing category after WORDPLAY among the reported question types.The paper describes these clues as especially lateral by design.
- What remains genuinely hard: Jeopardy!’s original NLP challenge split into linguistic indirection over open-domain recall and a narrower symbolic-manipulation problem.Modern models have effectively closed the comprehension half while only denting genuine wordplay.
7 Discussion
The paper frames the model as a portable, lossy compression of broad question-answering knowledge rather than an archive of clues. Its strongest comparison with Watson comes from answering post-cutoff clues, while the evaluation protocol and training exposure impose important limits.
- Knowledge compression: A 9 GB model encodes comparable breadth of trivia knowledge in its weights, making Watson-class question answering portable and queryable on commodity hardware.The paper frames this as knowledge compression and identifies the minimum model size that can rival Watson as an open question.
- What the capsule holds: The model is a compressed, lossy, generalizing representation, not a reproducible copy or lookup table of its 18 trillion-token training corpus.The 9 GB figure refers to the model itself, whose demonstrated capacity includes clues that postdate its training.
- Portability: A 9 GB model fits on common portable media and costs about a cent per month in deep archival cloud storage, unlike Watson’s server-bound system.The paper gives microSD cards, Blu-ray discs, USB sticks, and LTO tape as concrete storage comparisons.
- Generalization: The artifact matters because it can answer new practical questions beyond the historical clue-and-answer lookup table.The paper illustrates this with science, nature, history, geography, agriculture, and other knowledge sampled by the corpus.
- Corpus scale: The 529,939-clue corpus represents roughly 8,700 complete games and about 2,200 hours of human answering effort.This scale contextualizes why exhaustive human inspection is impractical.
- Evaluation boundary: The post-cutoff condition, rather than the headline 67.0%, is the paper’s preferred test of generalization beyond absorbed training material.The authors treat prior exposure symmetrically because Watson’s corpus and tuning were also deliberately oriented toward Jeopardy! answers and past clues.
8 Conclusion
Across the full Jeopardy! corpus, a single compact model approaches Watson-class accuracy while broad question-answering knowledge compresses into a portable file. The conclusion emphasizes that the model can answer beyond its absorbed historical material, unlike Watson’s architecture.
- Conclusion: 67.0% of 529,939 clues are answered by a single 9 GB model under a strict reproducible protocol, with higher performance on factoid categories.The result approaches Watson’s all-clue accuracy on hardware costing a tiny fraction as much.
- Conclusion: The comparison is framed as a measure of how broadly question-answering knowledge compresses into compact general models, not as a verdict on Watson.The authors explicitly preserve Watson’s status as a pioneering system judged by the standards of its time.
- Conclusion: A model can answer clues beyond the absorbed historical distribution, entering a regime that Watson’s architecture could not.The paper treats this distinction as the decisive contrast between the two eras.
- Conclusion: The 41-year corpus is presented as a verifiable record of knowledge a culture considers worth knowing, now stored in a small portable file.Every clue is paired with a known-correct response, while the model can answer questions postdating its own construction.
A Multi-model breakdowns
Across five open models, difficulty and question-type patterns are shared rather than checkpoint-specific. Accuracy declines toward harder tiers, drops sharply on FINAL clues, does not favor older decades, and is highest for factoid categories but lowest for WORDPLAY.
- Difficulty: Accuracy falls gently from EASY to HARD for every model, then drops sharply on FINAL clues.Model ordering is roughly preserved across tiers, while the difficulty effect is attributed to the task rather than one checkpoint.
- Broadcast decade: Accuracy is flat or mildly declining from the 1990s to the 2020s across all models, rather than rising for older clues.The paper interprets this pattern as evidence favoring durable knowledge over verbatim recall.
- Question type: Factoid categories with crisp answers, especially science and place, are easiest for every model, while WORDPLAY is hardest.The residual difficulty is linked to surface-form symbol manipulation, and the pattern is shared across the model class.