Source-linked AI summary
Towards a Seamless Integration of Word Senses into Downstream NLP Applications
Mohammad Taher Pilehvar, Jose Camacho-Collados, Roberto Navigli, Nigel Collier
TL;DR
Word-level NLP can leave intended meanings unresolved, and integrating sense-level information into downstream applications has been relatively understudied. This paper adds graph-based disambiguation to a text classifier and reports gains across multiple datasets, especially with coarser senses and longer documents.
Problem
Word-level NLP leaves semantic ambiguity unresolved, while integration of sense-level information into downstream applications has been insufficiently studied.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Sense representations should be evaluated in downstream applications rather than relying only on artificial benchmarks.
Source: §5.2.2 · p. 6The approach does not explicitly tackle co-reference resolution, although shared-sense mappings partly address it.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Lexical ambiguity can impede NLP systems from accurate understanding of semantics. Despite its potential benefits, the integration of sense-level information into NLP systems has remained understudied. By incorporating a novel disambiguation algorithm into a state-of-the-art classification model, we create a pipeline to integrate sense-level information into downstream NLP applications. We show that a simple disambiguation of the input text can lead to consistent performance improvement on multiple topic categorization and polarity detection datasets, particularly when the fine granularity of the underlying sense inventory is reduced and the document is sufficiently large. Our results also point to the need for sense representation research to focus more on in vivo evaluations which target the performance in downstream NLP applications rather than artificial benchmarks.
1 Introduction
Word-level NLP can miss intended meanings because it leaves ambiguity unresolved, and this limitation has received less study than subword or character-level approaches. The paper introduces a graph-based disambiguation pipeline and evaluates sense-level input in downstream classification tasks.
1 Introduction
- Word-level representations can hamper handling of rare forms and restrict semantic understanding to ambiguous word meanings.The passage identifies multiwords, inflections, and derivations as examples of infrequent forms.Source: §1 · p. 1
- The under-studied limitation is semantic ambiguity; the paper addresses it by mapping input words to intended senses before downstream processing.The proposed pipeline uses a graph-based WSD algorithm and is evaluated on polarity detection and topic categorization.Source: §1 · p. 1
- Sense-level input improves classification on multiple datasets, particularly for longer inputs and reduced sense-inventory granularity.The pipeline is described as modular, allowing different WSD and sense-representation techniques to be integrated.Source: §1 · p. 1
2 Motivation
The motivation is that word-based models can miss contextually intended meanings, while ambiguity also intersects with multiword expressions and co-reference. The pipeline is presented as addressing these issues to differing degrees.
2 Motivation
- Ambiguous words such as Mercedes, race, Hamilton, and Formula can refer to multiple entities or meanings, challenging word-level understanding.The authors identify word-level functionality as a barrier to deeper language understanding.Source: §2 · p. 2
- Most word-based models treat the components of multiword expressions independently, although those expressions are idiosyncratic lexical units.The pipeline is described as facilitating the handling of expressions such as Lewis Hamilton and Formula 1.Source: §2 · p. 2
- Co-reference resolution is not explicitly addressed, but expressions referring to the same contextual meaning can map to one concept and partly address it.The passage gives Formula 1/F1 and German Grand Prix/German GP/Hockenheim as examples.Source: §2 · p. 2
3 Disambiguation Algorithm
The algorithm builds a graph of candidate senses and their semantic links, then selects high-degree candidates iteratively. A confidence threshold controls when it stops disambiguating and backs off to the surface word form.
3 Disambiguation Algorithm
- The pipeline semantifies text before classification by replacing surface-form units with word senses, aiming to provide input with reduced ambiguity.The reduced-ambiguity input is intended to facilitate the classifier's decision making.Source: §3 · p. 2
- The graph-based joint disambiguation and entity-linking algorithm accepts arbitrary semantic networks and requires no sense-annotated training data.The authors describe it as fast and scalable, in contrast to computationally expensive graph methods.Source: §3 · p. 2
- The algorithm represents each input text as a graph of candidate senses and semantic links, then iteratively selects a highest-degree candidate and removes its lexical alternatives.The graph is a small subgraph of the semantic network; after a sense is selected, candidate senses for its corresponding word are removed.Source: §3 · pp. 2–3
- A threshold stops selection when the maximum degree falls below θ|S| / 100, leaving uncertain words or out-of-inventory senses as surface forms.This stopping rule is intended to avoid introducing noise in uncertain cases.Source: §3 · p. 3
4 Classification Model
The classifier combines convolutional features with an LSTM, and compares random with pre-trained embedding initialization. The paper also constructs WordNet and Wikipedia sense embeddings and groups senses into supersenses.
4 Classification Model
- Convolution and max pooling extract salient features, which an LSTM processes before the classifier’s softmax layer.The LSTM is added before softmax to better capture long-distance dependencies.Source: §4 · p. 3
4.1 Pre-trained Word and Sense Embeddings
- DeConf uses WordNet and Personalized PageRank to find sense-biasing words, then derives sense embeddings in the word-embedding space.The sense vector averages a decay-weighted biasing-word representation with the corresponding word embedding.Source: §4.1 · p. 4
- Wikipedia sense embeddings extend coverage to named entities and domain-specific terms, using NASARI salient-word lists and BabelNet mappings to WordNet.For WordNet synsets mapped to Wikipedia pages, the corresponding embeddings are averaged.Source: §4.1 · p. 4
4.2 Pre-trained Supersense Embeddings
- The model reduces sense granularity with WordNet’s 44 supersense clusters, averaging the embeddings of their member senses.The paper notes that WordNet’s fine-grained distinctions may be too fine for many NLP applications.Source: §4.2 · pp. 4–5
5 Evaluation
The evaluation covers topic categorization and polarity detection, using the same general experimental setup for both tasks.
- The paper evaluates its model on topic categorization and polarity detection, then introduces the common experimental setup.Source: §5 · p. 5
5.1 Experimental setup
Both tasks use the paper’s classification model with task-specific hyperparameters and shared evaluation conventions. The experiments compare embedding initializations, sense inventories, and supersenses.
5.1 Experimental setup
- The same classification architecture is used for both tasks, with task-specific hyperparameters fixed across configurations and 300-dimensional embeddings.Evaluation uses 10-fold cross-validation unless standard train-test splits are available; the disambiguation threshold is tuned on training data over seven values from 0 to 3.Source: §5.1 · p. 5
- Disambiguation uses a joint semantic network of Wikipedia hyperlinks and WordNet connected through BabelNet.The paper uses “Wikipedia” as shorthand for this joint inventory, which also includes WordNet senses.Source: §5.1 · p. 5
- Pre-trained embeddings comprise 757,262 sense vectors in the 300-dimensional Word2vec space, derived using WordNet 3.0 and a November 2014 Wikipedia dump.The word vectors are from Google News, truncated to the 250,000 most frequent words.Source: §5.1 · p. 5
- Supersenses are obtained by clustering disambiguated senses in the same WordNet supersense, and results are reported with accuracy and F1.Source: §5.1 · p. 5
5.2 Topic Categorization
With random embedding initialization, sense integration consistently improves topic categorization over the word-based model, especially with supersenses. The results vary with embedding initialization and domain: Ohsumed is challenging, while Wikipedia sense embeddings attain the best overall performance.
5.2.1 Datasets
- The three topic-categorization datasets include BBC and Newsgroups newswire collections and Ohsumed medical abstracts categorized into 23 cardiovascular diseases.Source: §5.2.1 · pp. 5–6
5.2.2 Results
- With random initialization, sense-based models consistently improve over the word-based model, with statistically significant gains on all three datasets when using supersenses.Table 2 identifies settings that improve the word-based model.Source: Table 2 · p. 6, §5.2.2 · p. 6
- Supersenses perform better than fine-grained WordNet senses, suggesting WordNet distinctions are too fine-grained for topic categorization; pre-trained representations show no improvement over the word-based model.Source: §5.2.2 · p. 6
- Ohsumed is the most challenging dataset, and pre-trained word embeddings reduce performance there; pre-trained Wikipedia sense embeddings attain the best overall performance.The paper notes Ohsumed’s medical domain and 23 classes, and attributes the Wikipedia-sense result to its broad inventory of domain-specific entities.Source: §5.2.2 · pp. 6–7
5.3 Polarity Detection
Polarity detection evaluates whether text is positive or negative, using five datasets spanning full reviews, short snippets, critic reviews, and phrases. Sense-enriched input did not yield consistent gains, particularly on relatively short reviews.
5.3 Polarity Detection
- Polarity detection is binary classification of a sentence or document as positive or negative.Source: §5.3 · p. 7
5.3.1 Datasets
- The five datasets include full movie reviews, short review snippets, critic reviews, IMDB reviews, and binary Stanford Sentiment phrases.For Stanford Sentiment, neutral phrases were removed and the remaining labels were assigned using the stated score thresholds; its labels are not evenly distributed.Source: §5.3.1 · p. 7
5.3.2 Results
- No consistent gain from disambiguated input appeared on polarity detection, particularly for relatively short reviews; Table 4 reports accuracy for the model and its variants.Results were generally better than Kim (2014), which the authors attribute to adding a recurrent layer.Source: §5.3.2 · p. 7
5.4 Analysis
The analysis finds that sense-based performance gains tend to increase with document size, while sparsity and disambiguation quality qualify the benefit. Coarser sense distinctions can help across both classification tasks and sense inventories.
Document size
- Performance gain generally increases with average document size across classification tasks for Wikipedia supersenses with random initialization.Figure 4 presents the relation between average document size and performance improvement for this configuration.Source: §5.4 · pp. 7–8
- Splitting training data across fine-grained senses can reduce instances per sense and hurt classification, while supersenses partly address this sparsity.Source: §5.4 · p. 8
- The disambiguator needs sufficiently large text to build a graph with enough coherent connections; longer topic-categorization documents yield more high-confidence disambiguations.Lower performance of graph-based disambiguation on short texts is a known issue that remains under exploration.Source: §5.4 · p. 8
Senses granularity
- Reducing sense granularity benefited both tasks across WordNet and Wikipedia, suggesting text classification does not require fine-grained semantic distinctions.The study coarsened senses using WordNet lexicographer files and leaves alternative techniques for both inventories to future work.Source: §5.4 · p. 8
6 Related Work
Prior work developed both context-clustered and inventory-based sense representations, but integrating sense representations into deep models and evaluating them in downstream tasks remained challenging. This paper positions its modular pipeline as a way to test those components in applications.
6 Related Work
- Sense representations include unsupervised methods that cluster contexts and methods that use external sense inventories and lexical resources.Examples include Schütze (1998) and Reisinger and Mooney (2010) for the former, and Chen et al. (2014) and Johansson and Pina (2015) for the latter.Source: §6 · p. 9
- Li and Jurafsky (2015) integrated unsupervised sense embeddings into downstream applications, but coupled disambiguation and representations made alternative components difficult to evaluate.The proposed pipeline instead separates preprocessing disambiguation from inventory-linked representations, allowing components to be plugged in.Source: §6 · p. 9
- Flekova and Gurevych (2016) added supersense representations as features, whereas this work focuses on reducing lexical ambiguity in the input.Source: §6 · p. 9
7 Conclusion and Future Work
The paper proposes a pipeline integrating sense-level knowledge into a state-of-the-art text classifier. It reports gains particularly for longer documents and coarser sense inventories, and frames the pipeline as an in vivo evaluation framework.
7 Conclusion and Future Work
- The pipeline integrates sense-level knowledge into a state-of-the-art text classifier, with reported gains particularly for longer documents and reduced sense granularity.Source: §7 · p. 9
- The authors present the modular pipeline as an in vivo framework for evaluating word-sense disambiguation and sense-representation techniques in downstream applications.They release code, data, and pretrained sense and supersense embeddings to support reproduction and further comparison.Source: §7 · p. 9
- Future work includes extending the approach to other languages and applications, and exploring task-specific coarsening or SentiWordNet for polarity detection.Source: §7 · p. 9