PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill, Anna Korhonen

arXiv:1608.02117v2cs.CL

TL;DR

Binary inventories and much NLP research do not represent lexical entailment’s gradual category membership. HyperLex collects crowdsourced graded judgments for concept pairs and compares them with automatic systems, finding a substantial gap between human judgments and current models.

  • Problem

    Large-scale semantic inventories and much NLP work treat lexical entailment as binary, despite evidence that category membership and typicality are gradual.

    Source: §9 · p. 42

  • Method

    HyperLex crowdsources graded judgments of how much one concept is a type of another across 2,616 concept pairs.

    Source: §1 · p. 2

  • Results

    Current lexical-entailment models optimized for ungraded LE perform poorly, and evaluated systems remain far below human performance on graded LE.

    Source: §9 · p. 43

  • Takeaways & Limitations

    HyperLex provides a repository of human graded lexical-entailment scores for cognitive-linguistics and other cognitive-science research.

    Source: §8 · p. 41

    The authors leave the underlying causes of the observed patterns speculative and beyond this work’s scope.

    Source: §7.2 · p. 35

Abstract

from arXiv · show

We introduce HyperLex - a dataset and evaluation resource that quantifies the extent of of the semantic category membership, that is, type-of relation also known as hyponymy-hypernymy or lexical entailment (LE) relation between 2,616 concept pairs. Cognitive psychology research has established that typicality and category/class membership are computed in human semantic memory as a gradual rather than binary relation. Nevertheless, most NLP research, and existing large-scale invetories of concept category membership (WordNet, DBPedia, etc.) treat category membership and LE as binary. To address this, we asked hundreds of native English speakers to indicate typicality and strength of category membership between a diverse range of concept pairs on a crowdsourcing platform. Our results confirm that category membership and LE are indeed more gradual than binary. We then compare these human judgements with the predictions of automatic systems, which reveals a huge gap between human performance and state-of-the-art LE, distributional and representation learning models, and substantial differences between the models themselves. We discuss a pathway for improving semantic models to overcome this discrepancy, and indicate future application areas for improved graded LE systems.

1. Introduction

HyperLex treats lexical entailment as a gradual, human-rated relation rather than a binary category link. Its human judgments capture typicality and graded membership, while current automatic approaches do not accurately model graded LE.

1. Introduction

  • Binary inventories such as WordNet miss the graduality of category membership, even though type-of reasoning supports ordinary language inference.A cat in the garden supports the inference that an animal is in the garden.Source: §1 · p. 2
  • HyperLex assigns graded lexical-entailment judgments to 2,616 concept pairs, each rated by at least 10 people on how much X is a type of Y.The crowdsourced ratings capture typicality and semantic category membership as degrees of LE.Source: §1 · p. 2
  • Human judgments capture typicality, vagueness, and graded LE, with inter-annotator Spearman correlations around 0.85 across the dataset.WordNet hypernymy-hyponymy pairs receive the highest average ratings among its relations, and ratings reflect the asymmetry between pairs and their reversals.Source: §1 · p. 3
  • None of the evaluated LE, distributional, or representation-learning approaches accurately models the graded relation reflected in human judgments.The evaluation spans unsupervised and supervised approaches, WordNet-based methods, multimodal approaches, and word embeddings.Source: §1 · p. 3
  • The authors hope HyperLex will encourage language technology that better reflects human semantic memory and models linguistic entailment more effectively.The paper also identifies potential applications for graded LE in later sections.Source: §1 · p. 3

2. Graded Lexical Entailment

The paper uses lexical entailment in its strict taxonomic sense and extends it from a binary TYPE-OF relation to a graded strength between concepts. This graded score brings together typicality and category membership, which the study does not explicitly distinguish.

Definitions

  • Lexical entailment (LE) here means the taxonomic hyponymy-hypernymy, or TYPE-OF, relation, not the broader substitutable-entailment relation.The broader use can include relations such as synonymy, metonymy, and meronymy.Source: §2 · p. 4
  • A graded LE score s specifies the degree to which X is a type of Y, extending the classical binary question of whether every X is a Y.A threshold on s can convert graded relations into discrete decisions.Source: §2 · p. 4

(Proto)typicality, Graded Membership, and Graded LE

  • The intuitive question “to what degree is X a type of Y?” combines typicality, or how representative X is, with graded membership, or how fully X falls within Y.The two phenomena can vary independently, but the crowdsourcing study deliberately does not distinguish them.Source: §2 · pp. 4–5
  • Typicality and category membership are related but distinct, and the study leaves their further differentiation for future work.The distinction is often not evident for verb concepts.Source: §2 · p. 5

Relation to Relational Similarity

  • Graded LE differs from relational similarity: it scores the strength of LE for a pair, while relational similarity scores how typical the pair is of a relation.For example, graded LE asks how much snake is an animal; relational similarity asks how well the pair represents an LE relation.Source: §2 · p. 5

3. Design Motivation

Existing lexical-entailment evaluation protocols and datasets largely reduce the relation to binary detection or directionality, leaving graded TYPE-OF judgments unmeasured. HyperLex supplies crowdsourced graded ratings, broad pair coverage, and a resource that can also support conventional evaluations.

3.1 Lexical Entailment Evaluations in NLP

  • Existing protocols evaluate binary entailment detection, directionality, or both, rather than the strength of the relation.Standard distributional approaches rely on context inclusion or lexical generality, while supervised approaches learn an asymmetric operator from training data.Source: §3.1.1 · p. 7
  • HyperLex instead assigns an order-sensitive pair (X, Y) a graded score for the strength of lexical entailment from X to Y.The score reflects typicality and graded membership of X for category Y.Source: §3.1.1 · pp. 7–8

3.1.2 Evaluation Sets.

  • Existing evaluation sets lack graded LE ratings, so they cannot directly evaluate how well models capture gradience in the TYPE-OF relation.HyperLex ratings can also be thresholded into ungraded annotations for detection and directionality evaluations.Source: §3.1.2 · p. 10
  • HyperLex is presented as an intrinsic evaluation resource for LE-based vector-space models, addressing a focus on similarity and relatedness in representation evaluation.The authors expect it to guide development of LE-specialized representations, noting a large gap between human and system performance.Source: §3.1.2 · p. 10
  • HyperLex covers different LE-related phenomena, including typicality, graded membership, entailment depth, concreteness, word class, and other lexical relations.Its 2,616 crowdsourced pairs can support linguistic and cognitive-science analyses as well as model evaluation.Source: §3.1.2 · p. 10

4. The HyperLex Data Set

HyperLex was designed as a broad, clearly defined, and consistently annotated resource for graded lexical entailment, built from 2,616 noun and verb pairs rated by human annotators. Its construction combines USF norms and WordNet-guided sampling with intuitive type-of instructions and quality controls.

4. The HyperLex Data Set

  • The final set contains 2,616 pairs—2,163 noun and 453 verb pairs—with at least 10 ratings each on a 0–10 type-of scale.Pairs are asymmetric: (X, Y) and (Y, X) are distinct.Source: §4 · p. 11

4.1 Choice of Concepts

  • USF association norms supplied varied candidate pairs, while WordNet assigned lexical relations and guided coverage across taxonomic levels and other relations.Pairs span nouns and verbs, semantic relations, and concreteness levels.Source: §4.1 · pp. 12–13

4.2 Sampling Procedure

  • Authors and two native English speakers manually checked sampled relation pairs, retaining pairs only when most checkers agreed on the relation.Reversed hyponym pairs and unassociated pairs were separately generated and checked.Source: §4.2 · p. 14

4.3 Question Design and Guidelines

  • Guidelines explained lexical entailment through the type-of relation, distinguished it from semantic relatedness, and used examples to introduce graded membership and typicality.Annotators judged a pair to stand in a type-of relation if any senses of its words did, leaving disambiguation to their intuition.Source: §4.3 · pp. 15–16

4.4 Questionnaire Structure and Participants

  • Participants first answered a qualification question, then rated pairs in groups of six or seven using sliders that posed “Is X a type of Y?” as a degree judgment.Groups overlapped by one pair for inter-group calibration; the study recruited native English speakers aged 18–50 living in the US or UK.Source: §4.4 · pp. 16–17, Figure 5 · p. 17

4.5 Post-Processing

  • After excluding failed qualification attempts and inconsistent or suspicious ratings, the authors averaged accepted ratings per pair and linearly rescaled scores from 0–6 to 0–10.85% of total exclusions followed incorrect qualification answers; final acceptance was 85.7% including those workers, or 97.5% excluding them.Source: §4.5 · p. 18

5. Analysis

Human ratings show graded membership and typicality across HyperLex, with strong annotator agreement and sensitivity to lexical relation, taxonomic distance, and direction. The analyses also identify scope considerations, including possible effects of source norms and the absence of explicit balance by concreteness.

Inter-Annotator Agreement

  • HyperLex achieves Spearman’s ρ = 0.854 (IAA-1) and ρ = 0.864 (IAA-2), with agreement reported as comparable to prominent crowdsourced semantic benchmarks.IAA-1 averages pairwise rater correlations; IAA-2 compares each rater with the mean of the others.Source: §5 · pp. 18–19

Typicality in Human Judgements

  • Human judgments assign higher scores to more prototypical category members, while still showing graded membership among WordNet hyponyms.Examples include higher ratings for cat, monkey, and cow than for mongoose or snail as animals.Source: §5 · pp. 19–20

Hypernymy/LE Levels

  • For nouns, graded LE scores rise with WordNet path length, and gradience remains visible even among pairs with longer paths.The authors suggest that prototypical instances contribute to higher ratings for broad categories.Source: §5 · pp. 20–21
  • For verbs, average scores decrease as WordNet LE levels increase; the authors attribute this pattern possibly to greater abstractness and ambiguity higher in the hierarchy.They relate verb concepts to cognitive differences in learning and comparing nouns versus verbs.Source: §5 · p. 21

LE Directionality

  • 94% of hyp-N pairs with a reversed counterpart receive higher ratings in the hyponym-to-hypernym direction, with direction detection improving as WordNet distance increases.The authors report difficulties especially for taxonomically closer hyp-1 pairs.Source: §5 · p. 21, Table 8 · p. 21

Other Lexical Relations

  • Average ratings follow the expected relation order syn > cohyp > mero > ant > no-rel, with antonyms and unrelated pairs receiving especially low scores.Lower cohyp than hyp-N scores suggest annotators distinguish cohyponymy from hyponymy.Source: Table 7 · p. 20, §5 · pp. 21–22

Concreteness

  • HyperLex was not explicitly balanced for concreteness, and the authors note that USF norms may make prototypical instances more prevalent, especially among longer-distance pairs.The dataset nevertheless includes 125 highly abstract pairs.Source: §5 · pp. 20, 22

Data Splits: Random and Lexical

  • The authors provide random and lexical train/dev/test splits; the lexical split has zero concept overlap between train and test but discards cross-set pairs.The random split reserves 70% for training, 5% for development, and 25% for testing.Source: §5 · p. 23

6. Evaluation Setup and Models

The evaluation compares model scores with human HyperLex ratings using rank correlation, alongside preliminary supervised experiments. Its directional entailment measures draw on distributional feature inclusion and are designed to reflect the asymmetry of type-of relations.

6. Evaluation Setup and Models

  • Spearman’s ρ correlations between model-score ranks and HyperLex human-score ranks are used for evaluation, with off-the-shelf unsupervised models tested on the full set and preliminary supervised experiments on provided splits.The setup follows intrinsic word-pair scoring evaluation protocols.Source: §6 · p. 24

6.1 Directional Entailment Measures

  • Directional entailment measures are grounded in the distributional inclusion hypothesis: features of a narrower term X should also occur among the features of broader term Y.The models use distributional features and weights such as positive PMI or local mutual information.Source: §6.1 · p. 24
  • WeedsPrec measures weighted feature inclusion, WeedsSim combines it with symmetric similarity, and InvCL incorporates both inclusion and non-inclusion of context features.WeedsSim uses the geometric average of WeedsPrec or another asymmetric measure and a symmetric similarity measure; InvCL is a variation of ClarkeDE.Source: §6.1 · pp. 24–25
  • The scores represent the asymmetric question “Is X a type of Y,” so reversing a pair should produce a different score.The evaluation uses the same word-pair scoring setup adopted in semantic-similarity research.Source: §6.1 · p. 24, §6 · p. 24

6.2 Generality Measures

Generality measures use the idea that broader terms have higher-entropy distributional representations. The section presents SLQS as an entropy-based measure of the relative specificity or generality of related terms.

6.2 Generality Measures

  • Generality-based entailment measures assume that broader terms have higher-entropy distributions, reflecting their greater generality compared with narrower terms.The section presents SLQS as the best-performing model in prior work among this group.Source: §6.2 · p. 25
  • SLQS identifies highly associated context features and computes each feature’s entropy from the distribution of co-occurring features.The context features may be selected using positive PMI or LMI.Source: §6.2 · p. 25
  • A concept’s median entropy across its most associated context features is denoted E_X, and a higher E_X indicates greater semantic generality.The measure is based on the N most associated context features.Source: §6.2 · p. 25
  • SLQS-BASIC can be used for ungraded entailment directionality, while SLQS-SIM adapts it to entailment detection by setting negative scores to zero.SLQS(X, Y) > 0 indicates that X is a type of Y in the directionality setting.Source: §6.2 · p. 25
  • The proposed generality account also expects a hypernym’s contexts to include features of its hyponym while extending to contexts beyond that hyponym.The animal–crocodile example illustrates the contrast in context coverage.Source: §6.2 · p. 25

6.3 Visual Generality Measures

The visual models estimate concept generality from the variability of images associated with a word. The section adapts this idea to graded lexical-entailment evaluation using image-generalilty measures and semantic-similarity thresholds.

6.3 Visual Generality Measures

  • Image sets for broader WordNet concepts tend to show greater visual variability than sets for narrower concepts, motivating image-based generality measures.The examples contrast images for animal, bird, and owl.Source: §6.3 · p. 26
  • Image dispersion measures generality as the average pairwise cosine distance among a concept’s image representations, while a related measure uses distance to their centroid.The image sets contain multiple representations for each concept.Source: §6.3 · p. 26
  • The visual model is tested for ungraded lexical-entailment directionality and detection as well as in graded evaluations.The section introduces it as a visual model for directionality and detection.Source: §6.3 · p. 26
  • The visual model uses an image-generality function with threshold α for a minimum generality difference and threshold θ to zero scores when semantic similarity is low.VIS-ID and VIS-CENT use different image-generality functions; word representations provide semantic similarity.Source: §6.3 · p. 26

6.4 Concept Frequency Ratio

Concept frequency ratio uses word frequency as a proxy for lexical generality. It is included as a baseline because it has been competitive in standard binary lexical-entailment evaluations.

6.4 Concept Frequency Ratio

  • Concept word frequency ratio is used as a proxy for lexical generality.The model uses word frequency counts from a large corpus as f(X).Source: §6.4 · pp. 26–27
  • Frequency ratio is a competitive baseline in standard binary lexical-entailment evaluation protocols.The cited prior work reports this competitiveness across several evaluation settings.Source: §6.4 · p. 26
  • The frequency-ratio model differs from the referenced formulation by setting f(X) to the corpus word-frequency count.The count is obtained from a large corpus.Source: §6.4 · p. 27

6.5 WordNet-Based Similarity Measures

The section evaluates WordNet-based similarity measures as proxies for graded lexical entailment. Their design can misrank direct co-hyponyms above more distant hypernymy-hyponymy pairs.

6.5 WordNet-Based Similarity Measures

  • The WordNet variant of equation (13) uses standard WordNet similarity measures to quantify the degree of the TYPE-OF relation.The three resulting models are WN-BASIC, WN-LCH, and WN-WUP.Source: §6.5 · p. 27
  • WN-BASIC scores concepts using the shortest path between them in the WordNet taxonomy.Source: §6.5 · p. 27
  • WN-LCH combines shortest-path length with taxonomy depth, while WN-WUP uses concept depths and their most specific ancestor.The LCH score is −log(path/2 · depth).Source: §6.5 · p. 27
  • By design, all three measures can rank direct co-hyponyms as more similar than distant hypernymy-hyponymy pairs.Source: §6.5 · p. 27

6.6 Order Embeddings

Order embeddings map lexical entailment relations into a partial order over vectors, then use coordinate-based measures to assess graded entailment. Their training approximates the order constraint with penalties and positive and negative examples.

6.6 Order Embeddings

  • Vendrov et al. (2016) developed embeddings specialized for lexical entailment and applied them to binary detection and directionality tasks.Source: §6.6 · p. 27
  • Order embeddings map a visual-semantic hierarchy’s partial order into an order-preserving embedding space rather than preserving distances.The section illustrates the hierarchy with a slice of the visual-semantic hierarchy.Source: §6.6 · p. 27, Figure 6 · p. 28
  • The reversed product order compares nonnegative vector coordinates, with smaller coordinates higher in the partial order and the origin at its top.The coordinate-wise order is defined by conjunction of total orders.Source: §6.6 · p. 27
  • Because the order condition is a soft constraint, training penalizes violations and uses a max-margin loss to separate positive from negative pairs.For standard ungraded evaluation, positive pairs come from WordNet and negatives are created by replacing a concept with a randomly selected one.Source: §6.6 · p. 28
  • For graded evaluation, ORDEREMB-COS uses cosine similarity, while ORDEREMB-DISTALL and ORDEREMB-DISTPOS use coordinate distances.DISTALL sums absolute coordinate differences as an indicator of entailment strength; DISTPOS includes only coordinates meeting the reversed-product-order criterion.Source: §6.6 · pp. 28–29

6.7 Standard (“Similarity”) Embeddings

This section tests whether models designed for symmetric semantic similarity can quantify graded lexical entailment, given that some strong entailment pairs are also judged semantically similar.

6.7 Standard (“Similarity”) Embeddings

  • The models target symmetric relatedness, typically measured by symmetric measures such as cosine similarity.Source: §6.7 · p. 29
  • The benchmark tests whether symmetric similarity models can quantify lexical-entailment strength and how the two relations are interlinked.Source: §6.7 · p. 29
  • Pairs with strong HyperLex entailment ratings can also receive high similarity ratings, including assignment–task and author–creator.Their HyperLex scores are 9.33 and 9.30, and their SimLex-999 scores are 8.70 and 8.02, respectively.Source: §6.7 · p. 29
  • The tested distributional models use skip-gram negative sampling with bag-of-words or dependency contexts, while resource-based models include NON-DISTRIBUTIONAL, PARAGRAM, and PARAGRAM+CF.The latter models draw on linguistic resources, paraphrase data, or linguistic constraints.Source: §6.7 · p. 29

6.8 Gaussian Embeddings

Gaussian embeddings represent concepts as probability distributions, offering soft spatial regions for modeling lexical entailment. The section evaluates variants trained with expected likelihood or KL divergence and paired with different graded-entailment measures.

6.8 Gaussian Embeddings

  • Gaussian embeddings represent each concept as a multivariate Gaussian, with a mean vector and a covariance parameterization.The covariance may be full, diagonal, or spherical.Source: §6.8 · p. 29
  • Gaussian regions provide a soft representation in which their intersection can quantify lexical entailment, and asymmetric KL divergence has an inclusion interpretation.Source: §6.8 · pp. 29–30
  • A low KL divergence from X to Y indicates that Y can be encoded easily as X, corresponding to Y entailing X.The passage interprets this as soft inclusion between the Gaussians’ ellipsoid level sets.Source: §6.8 · p. 30
  • The evaluated variants combine expected-likelihood or KL training objectives with cosine similarity or KL divergence for graded lexical-entailment evaluation.The four variants are WORD2GAUSS-EL-COS, WORD2GAUSS-EL-KL, WORD2GAUSS-KL-COS, and WORD2GAUSS-KL-KL.Source: §6.8 · p. 30

7. Results and Discussion

The results compare human judgments with lexical-entailment models, examining how performance varies across methods, tasks, word classes, evaluation splits, and representation choices. They also discuss the relative performance of Ridge and OLS and possible directions for improving graded lexical-entailment models.

7.1 Training Data and Parameters

  • Direct TypeDM scores and negative LMI variants were not reported because they performed significantly worse than the reported setups.Source: §7.1 · p. 30

7.2 Results

  • The FR model was strongest in the first comparison, while visual approaches were competitive with textual distributional models; VIS-ID reached ρ = 0.229 and VIS-CENT ρ = 0.228 at α = 1.Varying α led to even better visual results, and the authors call for more expressive multi-modal LE models.Source: §7.2 · pp. 31–32
  • Frequency is the strongest model in the first comparison, while directional measures lag other approaches and systems remain far below human agreement.These results motivate models designed specifically for graded lexical entailment.Source: §7.2 · pp. 31–32, Table 11 · p. 32
  • Word embeddings remain well below human performance; dependency contexts have a slight edge, while hierarchical embeddings show no clear advantage and symmetric cosine poorly captures entailment asymmetry.ORDEREMB improves slightly with DISTPOS rather than COS, and WORD2GAUSS with KL rather than COS.Source: §7.2 · pp. 32–33, Table 12 · p. 33
  • PARAGRAM and PARAGRAM+CF score 0.685 and 0.742 on SimLex-999 but 0.243 and 0.320 on HyperLex, supporting that graded LE is not semantic similarity.On the 111-pair intersection, all evaluated embeddings perform better at semantic similarity than graded LE.Source: §7.2 · p. 34, Figure 7 · p. 34
  • Performance differs by word class: directional measures score lower on verbs, visual models score better on nouns, and embeddings score higher on verbs even after controlling for test-set size.The authors offer possible explanations involving verb context variability, visual abstraction, and human reliance on similarity or syntax, while leaving causes speculative.Source: §7.2 · pp. 34–35, Table 13 · p. 35
  • On 940 high-scoring hyp-N pairs, frequency indicates directionality, but models’ graded-LE scores are lower than on the full set and ungraded outputs poorly estimate entailment strength.Reliable hypernym direction or relation detection does not translate into a global estimate of graded LE strength.Source: §7.2 · pp. 35–36, Table 14 · p. 36

7.3 Supervised Settings: Regression Models

  • Supervised regression is sensitive to the split and model choices: lexical-split OLS and many RIDGE variants fail to beat unsupervised baselines, while random-split gains partly reflect lexical memorisation.Prototypical hypernyms can cause models to assign high scores even to clear negatives; outcomes also depend on representations, feature transformations, and regression algorithm.Source: Figure 8 · p. 38, §7.3 · pp. 38–39, Table 16 · p. 39
  • RIDGE consistently outperformed OLS across all splits, models, and feature transformations, showing that regression-model choice matters.The preliminary analysis advocates more sophisticated learning algorithms; future work could also draw on training data from resources beyond HyperLex.Source: §7.3 · pp. 39–40

7.4 Further Discussion: Specialising Semantic Spaces

  • Specialising vector spaces for graded lexical entailment remains open, but specifying pairwise entailment degree may help end-to-end natural-language inference systems.By analogy, similarity-specialised spaces have boosted statistical systems in dialogue state tracking (Mrkši´c et al. 2016, 2017; Vuli´c et al. 2017).Source: §7.4 · p. 40
  • Neural-inspired representation architectures (Mrkši´c et al. 2016; Vendrov et al. 2016) are promising for future graded-LE modeling in supervised and unsupervised settings, despite low current performance.Source: §7.4 · p. 40

8. Application Areas: A Quick Overview

HyperLex’s graded lexical-entailment resource has potential applications in NLP and representation learning, knowledge-base extension, cognitive science, relational search, and multimodal modeling.

Natural Language Processing

  • Graded LE introduces a new evaluation challenge because methods effective on binary LE protocols have fundamental limitations on graded entailment.The paper also connects graded LE with phrase- and sentence-level textual entailment and tasks such as question answering and paraphrasing.Source: §8 · p. 40

Representation Learning

  • HyperLex offers a wide-coverage evaluation set for representation-learning models targeting the TYPE-OF relation, which has been largely overlooked beside similarity and relatedness.The passage identifies a lack of a comprehensive intrinsic evaluation resource for lexical-entailment embeddings.Source: §8 · p. 41

Data Mining: Extending Knowledge Bases

  • HyperLex can evaluate statistical models that extend knowledge bases by learning graded TYPE-OF, INSTANCE-OF, or IS-A relations.Source: §8 · p. 41

Cognitive Science

  • HyperLex’s human graded-entailment scores may support cognitive-science research on prototypicality, graded membership, and analogical transfer.The paper illustrates transfer from a familiar source concept to an unfamiliar target concept.Source: §8 · p. 41

Information Search

  • Graded LE could help relational search rank more prototypical category members ahead of less prototypical ones.The paper illustrates this with animal queries, placing dogs, cats, or elephants before huhblubs or wampimunks.Source: §8 · p. 41

Beyond the Horizon: Multi-Modal Modeling

  • Graded lexical entailment may support multimodal systems that organize images and captions at different abstraction levels, though its contribution to image captioning remains uncertain.The resource may also help evaluate models grounding language in the physical world and could be extended to hierarchical image databases.Source: §8 · p. 42

9. Conclusion

HyperLex provides a broad gold-standard resource for graded lexical entailment, with human ratings that consistently capture gradience and differ from similarity. Evaluations reveal a substantial gap between human and current model performance, while further analyses and downstream applications remain open.

9. Conclusion

  • HyperLex is a large, wide-coverage gold-standard resource for intrinsic evaluation of semantic representations targeting graded lexical entailment and the TYPE-OF relation.The authors also anticipate use in cognitive science because category membership is central to semantic representation.Source: §9 · p. 42
  • HyperLex focuses on continuous strength of TYPE-OF relations and contains 2,616 word pairs, with analyses finding prominent effects of prototypicality and graded membership.Source: §9 · p. 42
  • More than 600 native English annotators consistently quantified graded LE and distinguished it from similarity, relatedness, and other lexical relations.Inter-annotator agreement was high across the full dataset and subsets.Source: §9 · p. 43
  • HyperLex covers nouns and verbs, varying concreteness, and different WordNet relations, enabling analyses across word classes and supervised learning with standard data splits.Source: §9 · p. 43
  • Current lexical-entailment models optimized for ungraded LE perform poorly overall, leaving a substantial gap between human and system performance on graded LE.The authors identify room below the inter-rating ceiling and call for more expressive graded-LE models.Source: §9 · p. 43
  • The paper notes that it has only begun exploring HyperLex analyses and its use in broader textual-entailment systems and other applications.Source: §9 · p. 43

Related papers

Loading 1608.02117v2…