Source-linked AI summary
Experimental Support for a Categorical Compositional Distributional Model of Meaning
Edward Grefenstette, Mehrnoosh Sadrzadeh
TL;DR
Compositional sentence meaning remains difficult for empirical distributional models because they must combine contextual word meanings with grammatical structure. This paper implements Coecke et al.’s categorical model using BNC data and learned matrices for relational words. It matches competing methods on intransitive disambiguation and outperforms them in a more syntactically complex transitive experiment.
Problem
Empirical distributional models have limited methods for composing word meanings into phrase and sentence meanings while respecting grammatical structure.
Method
The paper implements a categorical logico-distributional model using BNC-based noun vectors and learned matrices for relational words applied to sentence arguments.
Results
The model performs on par with leading approaches for intransitive disambiguation and outperforms other models in the more complex transitive experiment.
Takeaways & Limitations
The categorical model can be implemented practically and shows stronger results as syntactic complexity increases.
Takeaways & Limitations
The evaluation excludes a slightly better mixed baseline because its improvement was not statistically significant and it required parametric optimisation unavailable for the full test set.
Abstract
from arXiv · showhide
Modelling compositional meaning for sentences using empirical distributional methods has been a challenge for computational linguists. We implement the abstract categorical model of Coecke et al. (arXiv:1003.4394v1 [cs.CL]) using data from the BNC and evaluate it. The implementation is based on unsupervised learning of matrices for relational words and applying them to the vectors of their arguments. The evaluation is based on the word disambiguation task developed by Mitchell and Lapata (2008) for intransitive sentences, and on a similar new experiment designed for transitive sentences. Our model matches the results of its competitors in the first experiment, and betters them in the second. The general improvement in results with increase in syntactic complexity showcases the compositional power of our model.
1 Introduction
Computational models struggle to compose sentence meaning from empirical word meanings. This paper implements a categorical framework that unifies logical structure with distributional data and evaluates it on intransitive and transitive disambiguation.
- Machines struggle with phrase and sentence semantics, while common search systems often ignore syntax or use superficial lexical relations.
- Formal semantic and distributional models provide partial, complementary solutions based respectively on logical form and contextual use.
- Coecke et al. unified these approaches by defining sentence vectors from word vectors and grammatical structure.
- This paper trains relational-word matrices over the BNC and applies them to argument vectors, providing an empirical implementation of the categorical model.
- The model matches competitors on intransitive disambiguation and performs better in a new transitive experiment, while motivating more compositional evaluation.
2 Two Orthogonal Semantic Models
Formal semantics composes meanings through grammatical derivations, whereas distributional models learn word meaning from contextual patterns but generally lack compositional sentence representations. Their operations also face ordering or dimensionality limitations.
- Formal Semantics: Formal semantic models derive sentence interpretations from grammatical structure, substituting meanings for constituents and applying β-reduction.
- Formal Semantics: Logical analyses support reasoning but reduce sentence meaning to truth or falsity and require an underlying domain and valuation function.
- Distributional Models: Distributional models represent words as vectors learned from the contexts in which they occur, supporting geometric similarity measures.
- Distributional Models: Their principal drawback is non-compositionality: they ignore grammatical structure and logical words when computing phrase and sentence meanings.
- Distributional Models: Kronecker products preserve word order but produce length-dependent vectors whose dimensionality grows exponentially, making computation quickly intractable.
3 A Hybrid Logico-Distributional Model
The hybrid model combines type-logical grammatical structure with distributional word representations through syntax-guided linear maps. Relational words are matrices acting on argument vectors, yielding sentence vectors in a shared sentence space.
- Vector-space composition lacks an obvious counterpart to semantic composition, motivating a categorical connection between grammar and distributional meaning.
- Relational words such as verbs and adjectives are represented as matrices, while nouns retain atomic vectors that can serve as arguments.
- Type-logical grammatical structure determines which words are matrices or vectors and maps word vectors to sentence vectors.
- Pregroup Grammars: Pregroup types encode grammatical roles: a transitive verb has type n^r s n^l, relating two noun arguments to a sentence output.
- Syntax-guided Semantic Composition: The categorical reduction uses inner products to substitute subject and object weights into verb argument positions, producing vectors in a common sentence space.
- Syntax-guided Semantic Composition: Sentence synonymity is measured by cosine similarity, with the sentence space instantiated according to the intended semantic interpretation.
4 Building Matrices for Relational Words
The model builds matrices for relational words from corpus relations and argument vectors, using tensor products to represent multi-argument structure. Transitive-verb matrices encode subject–object co-occurrence information and generalize to other relational words.
- Matrix weights encode the extent to which basis words are related in the argument positions of a relational word.This distributional abstraction avoids requiring exact lexical relations to have appeared in the corpus.
- Relational words are represented as m-dimensional matrices over a fixed basis, with one dimension for each adjoint argument type.The basis vectors come from a restricted set of frequent corpus words, allowing varied contextual information.
- The learning procedure retrieves each argument vector, forms their Kronecker product, and uses corpus relations to accumulate matrix weights.The procedure applies to sequences containing a relational word and its arguments in the order specified by the grammatical type.
- A transitive verb is represented as a 2 dimensional matrix whose weights reflect subject and object co-occurrence evidence.For ‘show’, the vector is constructed from tensor products such as ‘table’ ⊗ ‘result’ and ‘map’ ⊗ ‘location’.
- A sample matrix weight 79.24 is obtained by summing products from two observed ‘show’-relations for the basis pair (‘far’, ‘far’).The products are 6.6×7 and 5.6×5.9, yielding 46.2 + 33.04.
- The same construction yields 3-dimensional matrices for ditransitive verbs and 1-dimensional matrices for adjectives and adverbs.
5 Computing Sentence Vectors
Sentence vectors are computed by combining argument vectors with relational-word matrices through point-wise multiplication in tensor space. The resulting vectors support comparison of sentence meanings by cosine similarity.
- The meaning of a transitive sentence is computed by applying the verb matrix to the Kronecker product of its subject and object vectors.This implements the categorical prescription as point-wise multiplication followed by summation.
- The subject and object vectors supply argument-position weights that are combined with the verb’s matrix coefficients to produce a sentence vector.
- In the transitive case, the sentence space is S = N ⊗N, while more complex relational words use higher-dimensional tensor spaces.The computations reduce to point-wise multiplications and summations, so the full tensor space need not be explicitly built.
- The same procedure computes meanings for sentences containing adjectives and adverbs.
- Sentence vectors can be compared using cosine measure to estimate their degree of synonymy.
6 Evaluation
The evaluation tests the categorical model on intransitive verb disambiguation and a new transitive-sentence task, comparing it with established compositional baselines using human similarity judgements. The model matches competitors in the first experiment and shows stronger alignment in the transitive experiment, while the authors note limitations in the evaluation measures and comparison set.
- First experiment: The first experiment tests whether compositional models disambiguate ambiguous intransitive verbs using noun context, with model similarities compared against human judgements.Each dataset entry contains a noun, target verb, landmark verb, High/Low classification, and evaluator similarity score from 1 to 7.
- First experiment: The categorical model performs significantly better than Kintsch and achieves a ρ quasiidentical to the multiplicative model in the first experiment.Each reported ρ has p < 0.05 according to the Table 3 caption.
- Evaluation methodology: The comparison excludes Mitchell and Lapata’s slightly better mixed model because its ρ difference was not statistically significant and it required parameter optimisation on held-out data.The paper instead compares against their multiplicative model, identified as the best included comparator.
- Second experiment: The new transitive dataset contains 200 sentence pairs, with subject and object nouns providing context for target and landmark transitive verbs.Twenty-five evaluators formed sentences and rated verb-pair similarity from 1 to 7; entries were classified as HIGH or LOW.
- Evaluation methodology: Spearman’s ρ is the primary ranking metric because High/Low classifications are imprecise and model similarity scores can be renormalised in practice.The authors caution that High–Low mean differences are not reliable indicators of performance: the categorical model can align better with annotators despite a smaller difference.
- Second experiment: 0.17 to 0.21: the categorical model’s alignment with human judgements improves significantly in the transitive experiment.The authors report the improvement as statistically significant at p < 0.05; additive performance changes little, while multiplicative alignment becomes statistically indistinguishable from the baseline.
7 Discussion
The paper implements a categorical distributional model on BNC data and evaluates it on intransitive and transitive verb-disambiguation tasks. Results are competitive for intransitive sentences and stronger for transitive sentences as syntactic complexity increases.
- Implementation: The implementation combines empirical distributional data with logical form using matrices for relational words and vectors for atomic words.Verbs and adjectives are represented as matrices acting on noun arguments.
- Evaluation: The model was evaluated on Mitchell and Lapata’s intransitive disambiguation task and a newly developed transitive-verb experiment.The two experiments test composition at different levels of syntactic complexity.
- Results: The categorical method performs on par with leading existing approaches in the intransitive experiment.The paper attributes this partly to the small context, where the method becomes similar to Mitchell and Lapata’s multiplicative model.
- Results: The categorical model outperforms other models in the transitive experiment and aligns more closely with human judgements as sentence complexity increases.This experiment distinguishes models with sensitivity to grammatical structure from commutative composition models.
- Implications: The results support implementing the categorical distributional model and indicate better performance in experiments involving higher syntactic complexity.The authors argue that its mathematics scales uniformly to more complicated sentences and supports comparing differently structured sentences.
8 Future Work
Future work extends the model’s linguistic coverage and examines how its flattened pregroup grammar compares with alternative syntactic approaches.
- Linguistic coverage: Function words and logical words, including quantifiers and conjunctives, are left for future treatment.The paper notes that it is not yet clear how existing set-theoretic approaches apply to this setting.
- Grammar comparison: The pregroup grammar flattens sentence representation by applying a transitive verb to its subject and object simultaneously.In CCG, the verb is instead applied to the object first and then to the subject.
- Grammar comparison: Advantages, disadvantages, and comparisons with CCG and other systems remain ongoing work.