Source-linked AI summary
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language
Avia Efrat, Uri Shaham, Dan Kilman, Omer Levy
TL;DR
Existing ambiguity datasets can be solved relatively easily by native speakers, so Cryptonite tests ambiguity through naturally sourced cryptic clues. T5-Large scores below a rule-based solver, while top-tier human experts approach 100% accuracy.
Problem
Many existing ambiguity datasets can be solved by a native speaker with relative ease, motivating a dataset that challenges competent native speakers.
Source: §1 · p. 1Method
Cryptonite is a large-scale dataset of naturally sourced cryptic clues whose solutions require resolving semantic, syntactic, and phonetic wordplays as well as world knowledge.
Source: §1 · p. 1Results
Takeaways & Limitations
Cryptonite presents a challenging benchmark for current models and a candidate for assessing future models’ disambiguation capabilities.
Source: §1 · p. 1, §6 · p. 5Some clues do not fit the standard wordplay-definition split, and T5’s SentencePiece tokenization might lose information needed for character-level manipulations.
Source: §3 · p. 2, §4.1 · p. 3
Abstract
from arXiv · showhide
Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease. We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a misleading surface reading, whose solving requires disambiguating semantic, syntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues pose a challenge even for experienced solvers, though top-tier experts can solve them with almost 100% accuracy. Cryptonite is a challenging task for current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6% accuracy, on par with the accuracy of a rule-based clue solver (8.6%).
1 Introduction
Cryptonite uses naturally sourced cryptic clues to test ambiguity that challenges even experienced solvers. Its baselines perform poorly, while clue difficulty, answer length, and answer-exclusive splits matter to evaluation.
1 Introduction
- Cryptonite contains 523K professionally authored cryptic clues with misleading surface readings, designed to challenge even humans experienced in cryptic crossword solving.The authors say this source makes the dataset less prone to artifacts and biases than crowdsourced examples (Gururangan et al., 2018; Geva et al., 2019).Source: §1 · p. 1
- Solving requires separating definition from wordplay and disambiguating semantic, syntactic, and phonetic wordplays, along with world knowledge.In the illustrated clue, “shifting” signals an anagram, “earth” supplies the letters, and “hater” fits the definition.Source: §1 · p. 1, Figure 1 · p. 1
- Fine-tuning generic T5-Large on Cryptonite achieves only 7.6% accuracy, below the rule-based cryptic clue solver’s 8.6%, underscoring Cryptonite’s challenge as a test of future models’ disambiguation capabilities.Source: §1 · p. 1
- Baseline performance correlates with human-rated clue difficulty, enumeration is highly informative, and mutually exclusive train/test answers are critical for candidly estimating T5’s general solving ability.The example answer is a five-letter anagram, “hater,” matching the definition “one doesn’t like.”Source: §1 · p. 1
2 Cryptic Crosswords
Cryptic crosswords are human puzzles whose clues mislead on the surface and require wordplay disambiguation. A clue generally combines wordplay and a definition, though some clues depart from that split.
2 Cryptic Crosswords
- A cryptic crossword is a human puzzle whose clues supply answers for a letter grid, but unlike regular crosswords, its clues have misleading surface readings and require deciphering wordplay.A cryptic clue has one possible answer even outside the grid.Source: §2 · p. 2
- A clue generally has wordplay and a definition, whose unsupplied split is usually the solver’s first parsing task; the definition relates directly to the answer while indicators help decipher wordplay.The clue is designed to have only one possible answer.Source: §2 · p. 2
- Examples span different wordplay types, but some clues resist the standard wordplay-definition split and instead point to the answer through the entire clue.The examples offer only a glimpse of cryptic crosswords’ diversity.Source: §2 · p. 2
3 Dataset
Cryptonite is a dataset of 523,114 cryptic clues drawn from professionally authored English-language crosswords. Its train, validation, and test sets share no answers, making the benchmark more challenging than a naive random split.
3 Dataset
- Cryptonite is a dataset of 523,114 cryptic clues from 17,375 English-language crosswords published in The Times2 and The Telegraph3 between October 2000 and October 2020.Source: §3 · p. 2
- Preprocessing removes clue-answer duplicates, mismatched answer enumerations, and repeated clues paired with different answers.The latter cases occur in less than 0.1% of the data and violate the single-solution principle.Source: §3 · p. 2
- Unlike regular crossword clues, cryptic clues are designed to have one answer without relying on letters from a grid.Regular clues may remain ambiguous without existing grid letters.Source: §3 · p. 2
- Examples illustrate concatenation, homophones, and compound wordplays involving reversal and insertion, while one clue lacks the usual wordplay-definition split.The figure marks answers, definitions, wordplays, and indicators with distinct colors.Source: §3 · p. 2, Figure 2 · p. 2
- Train, validation, and test sets share no answers, a split that creates a more challenging benchmark than naive random splitting.The split follows Lewis et al. (2020).Source: §3 · p. 3
4 Experiments
Experiments benchmark T5-Large and a rule-based solver, then analyze clue difficulty, enumeration, and data splitting. The baselines solve only a small portion of clues, and answer-exclusive splits are critical for estimating generalization.
4.1 Baselines
- T5-Large predicts answers from clues, while the rule-based solver ranks wordplay-definition parses using expert resources and omits some less frequent wordplay types.T5 uses SentencePiece tokenization, which might lose information on clues requiring character-level manipulations.Source: §4.1 · p. 3
4.2 Main Benchmark
- Despite training on roughly half a million examples, T5 does not exceed the rule-based solver, and both solve only a small portion of clues.Accuracy is measured using exact string match.Source: §4.2 · p. 3, Table 2 · p. 3
4.3 Analysis
- Both baselines perform better on quick clues, suggesting their performance correlates with human assessment of linguistic difficulty.Quick cryptic clues are designed to be easier for beginners.Source: §4.3 · p. 3, Table 3 · p. 4
- Removing enumeration lowers accuracy for both baselines, and T5 also leverages this answer-length signal.Enumeration gives the number of letters in the answer.Source: §4.3 · p. 4, Table 4 · p. 4
- A naive random split grossly overestimates T5: it solves an additional 50% of the test set, with solving probability correlated with how often answers occur in training.Because answers can be shared by paraphrased clues, answer-exclusive splits provide a more candid estimate of generalization to new clues.Source: §4.3 · p. 4, Table 5 · p. 4, Figure 3 · p. 4
5 Related Work
Related work spans formal and rule-based approaches to cryptic crosswords, disambiguation datasets, and regular-crossword systems. Cryptonite is positioned as a large-scale, varied-wordplay dataset with answers held out across splits.
5 Related Work
- Prior cryptic-crossword work formalized clue languages and syntactic identification, while this paper focuses on a large solving dataset and baselines; generating clues from answers is another possible use.The cited approaches include Williams and Woodhead (1979), Hart and Davis (1992), and Hardcastle (2001, 2007).Source: §5 · p. 4
- Cryptonite is presented as the first dataset combining large scale with varied wordplays, unlike pun-disambiguation and sarcasm datasets.The related-work discussion cites Levesque et al. (2011), Raganato et al. (2017), Sakaguchi et al. (2020), Joshi et al. (2017), and Oprea and Magdy (2020).Source: §5 · pp. 4–5
- Regular-crossword systems may search external clue databases, whereas Cryptonite guarantees cryptic clues and keeps answers mutually exclusive across splits to shift focus from memorization toward reasoning.Some regular-crossword datasets contain a small proportion of tricky clues that require human identification.Source: §5 · p. 5
6 Conclusion
Cryptonite is a large-scale cryptic-crossword dataset requiring solvers to disambiguate varied wordplays. T5-Large does not outperform the rule-based baseline, and the results underscore the task's difficulty.
6 Conclusion
- Cryptonite is a large-scale cryptic-crossword dataset whose clues require disambiguating a variety of wordplays.Source: §6 · p. 5
- 7.6% accuracy for T5-Large versus 8.6% for a rule-based solver, while human experts achieve close to 100%.The comparison highlights the challenge posed by Cryptonite.Source: §6 · p. 5
- The results highlight the challenge of Cryptonite and encourage research on disambiguation tasks that native speakers cannot easily solve.Source: §6 · p. 5
A Example Predictions
Tables 6 and 7 present examples of T5-Large predictions, respectively, correct and incorrect.
B Example Predictions
- Tables 6 and 7 show examples of correct and incorrect T5-Large predictions, respectively.Source: Table 6 · p. 7, Table 7 · p. 7
B Similar Clues
Table 8 presents pairs of similar clues drawn from the train and test sets of the naive split.
B Similar Clues
- Table 8 pairs similar clues from the train and test sets of the naive split.Source: Table 8 · p. 7