Source-linked AI summary
Roget's Thesaurus and Semantic Similarity
Mario Jarmasz, Stan Szpakowicz
TL;DR
The paper tests whether computerized Roget’s Thesaurus can measure semantic similarity as an alternative to WordNet-based measures. It proposes a Roget’s taxonomy-based measure, compares it with human judgments and NLP systems, and finds strong similarity-test correlations and synonym-selection performance across TOEFL, ESL, and Reader’s Digest questions.
Problem
The paper investigates whether Roget’s Thesaurus, sometimes treated as a book of synonyms, can provide an effective alternative to WordNet for measuring semantic similarity.
Method
The authors convert a Roget’s taxonomy-based semantic-distance measure into a similarity measure and compare it with human judgments and six WordNet-based measures across noun-pair and synonym-selection tasks.
Results
Roget’s performs strongly across the evaluated tasks, including 82% accuracy on the ESL questions and higher performance than WordNet measures on the reported synonym tests.
Takeaways & Limitations
The electronic 1987 Penguin Roget’s Thesaurus is reported to be as good as, if not better than, WordNet for semantic similarity and especially effective on standard synonym tests.
Takeaways & Limitations
The Rubenstein and Goodenough list is not considered a suitable semantic-similarity benchmark because its preparation methodology is unexplained and its 0-to-10 rating scale is difficult for subjects to use.
Abstract
from arXiv · showhide
We have implemented a system that measures semantic similarity using a computerized 1987 Roget's Thesaurus, and evaluated it by performing a few typical tests. We compare the results of these tests with those produced by WordNet-based similarity measures. One of the benchmarks is Miller and Charles' list of 30 noun pairs to which human judges had assigned similarity measures. We correlate these measures with those computed by several NLP systems. The 30 pairs can be traced back to Rubenstein and Goodenough's 65 pairs, which we have also studied. Our Roget's-based system gets correlations of .878 for the smaller and .818 for the larger list of noun pairs; this is quite close to the .885 that Resnik obtained when he employed humans to replicate the Miller and Charles experiment. We further evaluate our measure by using Roget's and WordNet to answer 80 TOEFL, 50 ESL and 300 Reader's Digest questions: the correct synonym must be selected amongst a group of four words. Our system gets 78.75%, 82.00% and 74.33% of the questions respectively.
1 Introduction
The paper examines semantic similarity as a graded relation and tests whether Roget’s Thesaurus can measure it effectively against human judgments and WordNet-based systems.
- Semantic similarity describes the degree of synonymy between two words as a continuous variable.
- Native speakers can order word pairs by similarity, including nonsynonymous but closely related pairs such as airplane–wing.
- Rubenstein and Goodenough established human synonymy judgments for 65 noun pairs, from which Miller and Charles selected 30 pairs for further study.
- The paper proposes a Roget’s-based semantic distance measure, converts it into semantic similarity, and compares it with human judgments and NLP systems.
2 Roget’s Thesaurus Relations as a Measure of Semantic Distance
The system represents semantic distance through paths in Roget’s regular taxonomy, then converts shortest path length into a similarity score.
- Roget’s offers a larger lexicon than WordNet and includes links between parts of speech and topical groupings that WordNet lacks.The computerized system uses the 1987 edition of Penguin’s Roget’s Thesaurus.
- The system calculates semantic distance as the shortest path between word references in Roget’s taxonomy.Given two words, it looks up their references and calculates all paths between those references.
- Roget’s taxonomy is organized into eight Classes, 39 Sections, 79 Sub-Sections, and 596 Head Groups before reaching lower-level groupings.
- Distance equals the number of edges in the shortest path, with shorter paths corresponding to closer taxonomic groupings.Examples range from length 0 for the same semicolon group to length 12 for the same section.
- sim (w1, w2) = 16 – [min distance(r1, r2)], where r1 and r2 are the reference sets for the compared words or phrases.
3 Evaluation Based on Human Judgment
The Roget’s-based measure was evaluated against human similarity judgments and WordNet-based measures across established noun-pair benchmarks. It correlated strongly with human judgments on the Miller and Charles and Rubenstein and Goodenough lists, while results on a larger, heterogeneous list were lower and its validity was questioned.
- Human-judgment benchmarks: The 65-pair and 30-pair human-judgment benchmarks were created by Rubenstein and Goodenough and Miller and Charles, respectively.Rubenstein and Goodenough used 51 judges and scores from 4.0 to 0.0; Miller and Charles selected 30 pairs from their list.
- Evaluation method: The evaluation repeated both human-judgment experiments with Roget’s and compared results with six WordNet-based similarity measures.The WordNet comparisons used WordNet 1.7.1 and Pearson product-moment correlation with human judgments.
- Miller and Charles: r=.878 correlation with human judgment was obtained for the Miller and Charles list, nearly matching the human-replicated upper bound of r=.885.The outlier was crane – implement, whose words have nothing in common in Roget’s Thesaurus.
- Rubenstein and Goodenough: r=.818 correlation with human judgments was obtained for the 65-pair Rubenstein and Goodenough list.Five low-similarity pairs received intermediate Roget’s scores because they appeared under the same Head and noun groups.
- Larger benchmark: .539 correlation with human judges on Gabrilovich’s 353-pair list was below the smaller-list results but exceeded WordNet’s best score of .375.The result was comparable to Finkelstein et al.’s combined metric, which obtained r=.550.
- WordNet comparison: WordNet edge counting performed well with WordNet 1.7.1 despite earlier criticism based on the assumption that taxonomy links have uniform distances.The authors suggest this may reflect improvements in the newer WordNet version, including more uniform distances between words.
4 Evaluation Based on Synonymy Problems
The paper evaluates semantic-similarity measures on synonym-selection questions from TOEFL, ESL, and Reader’s Digest tests. Roget’s performs strongly across these benchmarks and remains best when questions contain only nouns.
- TOEFL: 78.75% of 80 TOEFL questions are answered correctly by the Roget’s Thesaurus system.Hirst–St-Onge and PMI-IR score 77.91% and 73.75%, respectively, while the combined approach of Bigham et al. reaches 97.50%.
- ESL: 82% of 50 ESL questions are answered correctly by the Roget’s system, ahead of PMI-IR at 74% and Hirst–St-Onge at 62%.Other WordNet measures answer no more than 36% correctly.
- Reader’s Digest: 74.33% of 300 Reader’s Digest questions are answered correctly by Roget’s, compared with 45.65% for the next-best WordNet measure.All other measures do not surpass 25%.
- Cross-task interpretation: The experiments favor measures that evaluate similarity between words from different parts of speech.When the experiments are repeated with noun-only questions, WordNet measures perform more uniformly and better, but Roget’s remains best.
5 Discussion
The paper concludes that computerized Roget’s performs at least as well as WordNet for semantic similarity and excels on synonym tests. It also identifies applications for similarity measures and notes important limitations in WordNet-based systems.
- Overall comparison: Roget’s was better than WordNet in seven of eight experiments and correlated well with human similarity judgments.The exception was the Rubenstein and Goodenough list of 65 noun pairs.
- Synonym tests: Roget’s answered standard synonym questions strongly despite using a simple distance measure without context or word-sense disambiguation.The authors describe this performance as impressive given those constraints.
- Limitations: WordNet-based systems performed poorly on synonym questions partly because their similarity measures are largely restricted to nouns and struggle with phrases.Their reliance on WordNet’s hierarchical structure limits applicability beyond noun relations.
- Applications: Semantic similarity measures can support lexical-chain construction by linking significant words that represent the same topic.The paper notes an implementation using the electronic Roget’s Thesaurus.
- Applications: Roget’s can also support analogy problems in which systems select the word pair expressing the same relation as a given example.The paper illustrates this with cat:meow and dog:bark.