Source-linked AI summary
Evaluating Neural Word Representations in Tensor-Based Compositional Settings
Dmitrijs Milajevs, Dimitri Kartsaklis, Mehrnoosh Sadrzadeh, Matthew Purver
TL;DR
Evidence comparing neural and co-occurrence-based representations in compositional settings has been limited. This paper evaluates them with tensor-based and simpler composition methods across focused and larger-scale tasks. Co-occurrence vectors are competitive on constrained tasks, while neural embeddings perform better on larger-scale tasks.
Problem
Prior comparisons of neural and co-occurrence-based representations in compositional settings were limited to additive and multiplicative composition against a neural autoencoder.
Source: §1 · p. 1Method
The study compares three semantic spaces using tensor-based and simpler compositional methods across verb disambiguation, sentence similarity, paraphrase detection, and dialogue act tagging.
Source: §1 · p. 1, §5.3 · p. 7, §7 · p. 9Results
Takeaways & Limitations
The best compositional operator depends strongly on the task and dataset, while neural word2vec vectors perform steadily off-the-shelf across varied tasks.
Source: §7 · p. 9The large-scale advantage of neural vectors may reflect their neural nature or the larger amount of data used to train them.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We provide a comparative study between neural word representations and traditional vector spaces based on co-occurrence counts, in a number of compositional tasks. We use three different semantic spaces and implement seven tensor-based compositional models, which we then test (together with simpler additive and multiplicative approaches) in tasks involving verb disambiguation and sentence similarity. To check their scalability, we additionally evaluate the spaces using simple compositional methods on larger-scale tasks with less constrained language: paraphrase detection and dialogue act tagging. In the more constrained tasks, co-occurrence vectors are competitive, although choice of compositional method is important; on the larger-scale tasks, they are outperformed by neural word embeddings, which show robust, stable performance across the tasks.
1 Introduction
The paper addresses the limited comparison of neural and co-occurrence-based representations in compositional settings. It evaluates both approaches across tensor-based and simpler composition methods, finding that neural vectors are generally more effective.
1 Introduction
- Neural and co-occurrence-based representations had been compared at the word level, but prior compositional comparison covered only additive and multiplicative methods against a neural autoencoder.The authors identify this as a limited basis for assessing the approaches in compositional settings.Source: §1 · p. 1
- The study compares neural word embeddings with two co-occurrence-based spaces across tensor-based models and simpler additive and multiplicative approaches.Its aim is a more complete comparison of these representations in compositional tasks.Source: §1 · p. 1
- The evaluation covers verb disambiguation and transitive sentence similarity, plus paraphrase detection and dialogue act tagging.The first two are small-scale experiments on predefined sentence structures; the latter two use larger-scale tasks.Source: §1 · p. 1
- Neural vectors are more effective than traditional approaches in almost all of the evaluated tasks.The authors describe this as the general picture across the experiments.Source: §1 · p. 1
2 Meaning representation
The section contrasts formal compositional meaning with distributional representations that express similarity. It introduces co-occurrence and neural word vectors as representations whose meanings can be composed at phrase and sentence level.
2 Meaning representation
- Compositional approaches are needed because natural languages are creative and sentences are rarely repeated, so larger text segments must be built from their parts.The ideal method would capture similarity both among constituents and among their composed meanings while fitting linguistic structure.Source: §2 · p. 2
- Formal semantics treats sentence meaning as a function of the meanings of its parts, while distributional representations express meaning similarity through distances between vectors or tensors.Formal representations can capture phenomena such as quantifier interaction, whereas distributional methods address similarity in meaning.Source: §2 · p. 2
- Co-occurrence vectors use context words as dimensions and encode how frequently a target word occurs with those contexts.The examples use BNC co-occurrence frequencies; similar contexts yield closer vectors, as illustrated by John being closer to Mary than to idea.Source: §2 · p. 2, Table 1 · p. 2
- Neural word embeddings are trained to predict surrounding words, producing vectors that can capture lexical similarity.The section notes that predicting context may make neural models more robust to data sparsity, a problem for co-occurrence spaces.Source: §2 · p. 3
- Applying distributional representations beyond words requires compositional models that extend them to sentence meaning and more complex semantic phenomena.The section identifies this extension as the focus of the next section.Source: §2 · p. 3
3 Compositional models
The section distinguishes composition by order sensitivity and grammatical structure, then explains how tensor-based models represent predicates and compose them with noun vectors. It presents several implementations for transitive verbs.
3 Compositional models
- Compositional models range from order-agnostic combinations to methods that use linear word order or grammatical structure.The paper focuses on grammatical composition alongside simple additive and multiplicative models, not task-specific deep-learning composition.Source: §3 · p. 3
- Simple models combine word vectors using addition or point-wise multiplication, but cannot distinguish sentences that differ only in word order.Deep-learning approaches can capture word order but are excluded because their vectors and operators are trained for specific tasks.Source: §3 · p. 3
- In tensor-based models, nouns are vectors and predicates are tensors whose multilinear maps apply to vectors to form sentence meanings.Tensor contraction generalizes matrix multiplication for higher-order tensors; simplified tensors can use operations such as multiplication, tensor product, point-wise multiplication, and addition.Source: §3 · p. 4
- Relational models build a transitive verb matrix from subject–object vector pairs, while Kronecker models form a matrix from the outer product of the verb vector with itself.Both reduce a transitive verb’s order-3 tensor to an order-2 matrix.Source: §3 · p. 4
- Five Frobenius models expand relational verb matrices to order-3 tensors by copying the subject or object dimension and applying addition, multiplication, or outer product.The variants are Copy-Sbj, Copy-Obj, Frobenius-Add, Frobenius-Mult, and Frobenius-Outer.Source: §3 · p. 4
4 Semantic word spaces
The section instantiates two co-occurrence spaces with different corpora and weighting schemes, alongside neural word embeddings. It describes the spaces and training choices used in the experiments.
4 Semantic word spaces
- Two co-occurrence spaces use different corpora and weighting schemes, and the neural comparison uses skip-gram embeddings trained with negative sampling.The neural vectors are 300-dimensional and learned from Google News.Source: §4 · pp. 4–6
- GS11 uses lemmatized BNC co-occurrence counts over frequent nouns, verbs, adjectives, and adverbs, weighted with PPMI.Its probabilities use a symmetric five-word window; p(c|t) is the probability of context word c given target t, and p(c) is its overall probability.Source: §4 · p. 5
- KS14 uses lemmatized ukWaC counts, local mutual information, and a five-word context window on either side of each target.The space is normalized and projected to 300 dimensions using SVD.Source: §4 · p. 5
- SVD is used to produce a compact representation; the paper notes that dimensionality reduction can make representations robust to corpus noise.The cited prior work is reported to show SVD performing well on similar tasks.Source: §4 · p. 5
- Negative sampling adds randomly selected incorrect word-context pairs to the neural training objective, which distinguishes target words from noise draws.For these experiments, context words are selected from a five-word window around each target.Source: §4 · p. 6
5 Experiments
The experiments compare three vector spaces and compositional operators across four semantic tasks, finding operator-dependent results in focused tasks and stronger neural performance in broader tasks.
5.1 Disambiguation
- NWE copy-object led verb disambiguation (Spearman ρ = 0.456), significantly exceeding KS14 Frobenius outer, the nearest co-occurrence competitor (p < 0.05).KS14's best result was Frobenius outer (0.350), while GS11's was point-wise multiplication (0.348). For simple composition, only multiplicative GS11 and additive NWE beat their own verb-only baselines; tensor copy-subject fell below co-occurrence baselines.Source: Table 3 · p. 6, §5.1 · pp. 6–7
5.2 Sentence similarity
- Addition led sentence similarity in all three spaces: KS14 scored 0.732, GS11 0.682, and NWE 0.689; no tensor-based model outperformed addition.KS14 addition's lead over NWE addition was not statistically significant; GS11 and NWE generally improved over verb-only baselines, with the stated model exceptions.Source: Table 4 · p. 7, §5.2 · p. 7
5.3 Paraphrasing
- Additive NWE led MSR paraphrase detection, with un-lemmatized accuracy of 0.73; multiplication performed worse than addition across all vector spaces.Tensor models were not tested because they could efficiently handle only fixed-structure sentences. KS14's accuracy dropped only 0.04, with no F-score drop.Source: §5.3 · pp. 7–8, Table 5 · p. 8
5.4 Dialogue act tagging
- Un-lemmatized NWE addition led dialogue-act tagging with 0.63 accuracy and 0.60 F-score; lemmatization sharply reduced accuracy, and addition consistently beat multiplication.The reported accuracy and F-score are averaged over tag classes.Source: Table 5 · p. 8, §5.4 · p. 8
6 Discussion
The discussion finds that small-scale comparisons favor no single vector-space approach, while broader tasks show more consistent advantages for neural word embeddings. Compositional operator choice also varies in importance and effect by task.
6 Discussion
- Small-scale tasks produced fairly competitive results across methods, but the best compositional operator was task-specific.Frobenius copy-object performed best for verb disambiguation, whereas addition performed best for sentence similarity; in KS14 sentence similarity, other compositional methods fell below the verb-only baseline.Source: §6 · p. 9
- On larger-scale tasks, NWE consistently yielded the best results, and it was the only space with adequate dialogue-act tagging performance.GS11 lagged KS14 and NWE on paraphrase detection, despite behaving stably in the small-scale tasks.Source: §6 · p. 9
- The apparent small-scale similarity between approaches is qualified by the spaces' differing comparisons with baselines: GS11 and NWE behaved similarly, while KS14 could be tuned to the best result.GS11 and NWE outperformed their baselines with several compositional methods, though both had lower overall performance than KS14 in those tasks.Source: §6 · p. 9
7 Conclusion
Neural word embeddings are competitive with count-based vectors on constrained small-scale tasks and more successful on large-scale tasks. The best compositional operator depends on the task, while differences in training corpora leave the source of the large-scale advantage unresolved.
7 Conclusion
- On constrained small-scale tasks, neural embeddings match or outperform count-based vectors, while they are more successful on large-scale tasks; Table 6 summarizes performance against baselines.The large-scale comparison does not establish whether the advantage reflects neural training or greater training-data volume.Source: Table 6 · p. 9, §7 · p. 9
- Word2vec neural vectors perform steadily off the shelf across tasks, but differences in corpus size and nature may confound comparisons with co-occurrence vectors.The authors note that task-specific neural language-model training can achieve even better performance.Source: §7 · p. 9
- Tensor-based composition performs best for verb disambiguation, whereas point-wise operators perform best for sentence similarity across vector spaces.The preferred operator depends on task and dataset, rather than on whether vectors are neural or co-occurrence-based.Source: §7 · pp. 9–10