PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain and Function: A Dual-Space Model of Semantic Relations and Compositions

Peter D. Turney

arXiv:1309.4035v1cs.CLcs.AIcs.LG

TL;DR

Existing vector-space models handle individual words well, but semantic relations and compositions require a unified treatment that supports novel phrases and multiple structural properties. The paper introduces a dual-space model combining domain and function similarities, and reports performance matching prior relation and composition models while improving order-sensitive phrasal similarity. Its scope remains limited by unaddressed polysemy and future work on longer phrases and sentences.

  • Problem

    Vector-space models represent individual words effectively, but prior relation and composition models are substantially different and struggle with novel phrases and shared semantic requirements.

  • Method

    The dual-space model combines separate domain and function spaces and composes similarity measures rather than mapping component vectors into one phrase vector.

  • Results

    Across analogy, noun-modifier composition, and phrasal similarity, the dual-space model matches relevant prior performance and significantly outperforms vector addition and element-wise multiplication on order-sensitive phrasal similarity.

  • Takeaways & Limitations

    A unified dual-space treatment can model relations and compositions while addressing linguistic creativity, order sensitivity, adaptive capacity, and information scalability within the paper’s evaluated scope.

  • Takeaways & Limitations

    The model avoids polysemy and leaves longer phrases and sentences for future work.

Abstract

from arXiv · show

Given appropriate representations of the semantic relations between carpenter and wood and between mason and stone (for example, vectors in a vector space model), a suitable algorithm should be able to recognize that these relations are highly similar (carpenter is to wood as mason is to stone; the relations are analogous). Likewise, with representations of dog, house, and kennel, an algorithm should be able to recognize that the semantic composition of dog and house, dog house, is highly similar to kennel (dog house and kennel are synonymous). It seems that these two tasks, recognizing relations and compositions, are closely connected. However, up to now, the best models for relations are significantly different from the best models for compositions. In this paper, we introduce a dual-space model that unifies these two tasks. This model matches the performance of the best previous models for relations and compositions. The dual-space model consists of a space for measuring domain similarity and a space for measuring function similarity. Carpenter and wood share the same domain, the domain of carpentry. Mason and stone share the same domain, the domain of masonry. Carpenter and mason share the same function, the function of artisans. Wood and stone share the same function, the function of materials. In the composition dog house, kennel has some domain overlap with both dog and house (the domains of pets and buildings). The function of kennel is similar to the function of house (the function of shelters). By combining domain and function similarities in various ways, we can model relations, compositions, and other aspects of semantics.

1. Introduction

Vector-space semantics represents individual words effectively but struggles with novel phrases, semantic relations, and the requirements of compositional meaning. The paper introduces a dual-space model that combines domain and function similarities while composing similarity measures rather than vectors.

  • VSMs capture individual-word meaning from word–context distributions, but extending them to phrases such as dog house remains unresolved.
  • Holistic phrase representations do not scale to the vast number of possible and previously unseen phrases, although they suit idioms whose meanings are not compositional.
  • Vector addition and averaging can lose information and fail to distinguish order-sensitive meanings, while fixed-dimensional representations also limit scalability and adaptive capacity.
  • The proposed dual-space model unifies relational and compositional semantics by measuring domain similarity and function similarity.
  • Across analogy, noun-modifier composition, and phrasal similarity tasks, the dual-space approach matches or significantly improves on relevant prior models.
  • The model composes cosine similarity measures, preserving component-word vectors and enabling order-sensitive, compositional linking structures rather than stand-alone phrase vectors.

2. Related Work

Related work identifies linguistic creativity, order sensitivity, adaptive capacity, and information scalability as central requirements for semantic models. Existing vector, holistic, tensor, and neural approaches address some requirements but retain important limitations that motivate the dual-space design.

  • The paper evaluates prior models against four desiderata: linguistic creativity, order sensitivity, adaptive capacity, and information scalability.
  • Simple vector addition, multiplication, and averaging lack order sensitivity, adaptive capacity, or information scalability, despite useful empirical performance.
  • Tensor-product composition preserves information structurally but grows exponentially with phrase length, whereas circular convolution compresses representations at the cost of information loss.
  • Holistic relation and phrase models cannot handle the full range of novel combinations because they depend on observed phrase or pair patterns.
  • Neural language models capture exchangeability or functional similarity but lack an equivalent mechanism for domain similarity.
  • The dual-space model resembles earlier two-part approaches but combines distinct domain and function spaces to address relational direction and semantic composition.

3. Three Vector Spaces

The model builds separate domain, function, and merged mono spaces from word–context matrices. These spaces differ in contextual patterns, enabling experiments on whether complementary representations improve semantic modeling.

  • Domain space uses nouns as contexts, function space uses verb-based patterns, and mono space merges the two context types.
  • All three spaces are built through common extraction and matrix-construction stages, with their differences introduced by the contextual-pattern stage.
  • The pipeline extracts lexicon terms, finds their corpus contexts, tokenizes and tags phrases, and converts tagged phrases into contextual patterns.
  • It constructs sparse term–context frequency matrices by selecting frequent candidate terms and counting term–pattern occurrences.
  • The matrices are weighted with PPMI and smoothed using truncated SVD, whose output comprises low-rank component matrices.
  • WordNet-based term selection produced subjectively higher quality than selection based only on high corpus frequency.

3.2 Domain Space

Domain space represents a term’s domain through nearby nouns, using broad contextual windows and ignoring syntactic relations. Its construction retains frequent noun-context patterns to form a sparse semantic matrix.

  • Domain-space intuition: Domain space characterizes a word’s topic using nearby nouns within a relatively wide window, while ignoring syntactic context.Each tagged phrase contributes the first noun to the left and right of the n-gram when available.
  • Domain-space matrix: 50,000 selected contextual patterns produce 114,297 rows and a PPMI matrix with 149,673,340 nonzero values at 2.62% density.The patterns are ranked by the number of rows generating them, with ties broken alphabetically.
  • Domain-space construction: For the n-gram boat, three tagged phrases generate five domain contextual patterns.The resulting row vector has nonzero entries for contexts such as lake and summer when those patterns survive filtering.
  • Function-space contrast: Function space characterizes roles through syntactic contexts involving nearby verbs, using narrower windows and more complex patterns than domain space.The richer patterns distinguish roles such as subject, direct object, and indirect object.
  • Function-space construction: Function-space construction can generate up to six patterns per tagged phrase, each containing the marker X and at least one verb.Patterns are formed from general and specific variants, with additional split patterns when material occurs on both sides of X.
  • Function-space matrix: 50,000 selected function patterns produce 114,101 rows and 68,876,310 nonzero PPMI values at 1.21% density.The source passage reports these matrix dimensions and density for function space.

3.4 Mono Space

Mono space merges domain and function contexts into one baseline matrix, while similarity is measured from cosine-based latent-space representations. The experiments vary latent dimensionality and singular-value weighting, with separate parameter tuning for the two spaces.

  • Mono-space construction: Mono space merges the 50,000 domain columns and 50,000 function columns, yielding 100,000 columns and 114,297 rows.Its PPMI matrix contains 218,222,254 nonzero values with 1.91% density.
  • Role of mono space: Mono space serves as a baseline for testing whether separating domain and function spaces is useful.The three spaces are built from word-context matrices, differing in their contextual columns.
  • Similarity measure: Similarity between terms is measured by the cosine of their corresponding row vectors.Cosine is the inner product after normalizing both vectors to unit length.
  • Latent-space parameters: The parameter k controls the number of latent factors, while p weights factors by raising singular values to p.Higher p produces broader, fuzzier neighborhoods; lower p produces sharper neighborhoods.
  • Parameter exploration: Experiments test 14 values of k from 100 to 1400 and 21 values of p from −1 to +1.The corresponding common settings include p = 0 and p = 1.
  • Parameter optimization: Tuning four domain/function parameters avoids an exhaustive 86,436-combination search by alternating optimization from kf = kd = 700 and pf = pd = 0.The procedure relies on relatively independent domain and function similarities and stops when performance no longer improves.
  • Term coverage: The matrices’ rows derive from WordNet terms, with exact matches attempted first and alternate forms searched when necessary.The same matrices are reused across experiment sets without dataset-specific customization.

3.7 Composing Similarities

The model combines domain and function similarities according to the task or phrase syntax. It uses a geometric mean to favor jointly high and balanced component similarities, assigning zero when any component is negative.

  • Similarity composition: Semantic relations and compositions combine domain similarity simd(a, b) and function similarity simf(a, b) in task-dependent ways.The intended combined similarity is high when both component similarities are high and balanced.
  • Geometric aggregation: The geometric mean combines component similarities because it encourages both high values and balance between them.Unlike the arithmetic mean, it is unsuitable for negative numbers.
  • Negative similarities: If any component similarity is negative, the model defines the geometric-mean combination as zero.Otherwise, it computes the nth root of the product of the component similarities.

3.8 Element-wise Multiplication

The element-wise multiplication approach is adapted for SVD-based vectors by enforcing nonnegative components and transforming the composed vector back into the comparison space. NMF is identified as an alternative that was not scalable to the matrices used.

  • Motivation: Element-wise multiplication composes vectors component by component, but negative elements can produce unintuitive positive products.The issue occurs when both input elements are highly negative.
  • Baseline constraint: The baseline cannot directly multiply rows of UkΣp_k because those vectors are typically about half positive and half negative.Using the sparse PPMI matrix would avoid negatives but would forgo SVD’s smoothing effect.
  • Adapted composition: The model maps latent vectors into a smoothed, nearly nonnegative space, clips any remaining negative elements to zero, and multiplies them element-wise.The composed vector is then transformed by VkΣp−1_k so it can be compared with rows in UkΣp_k.
  • Nonnegative conversion: The nonnegative conversion sets each vector element to itself when positive and to zero otherwise.This operation is denoted nn(x).
  • Alternative factorization: NMF could provide another solution, but no implementation was found that scaled to the matrices used in the paper.In earlier experiments with smaller matrices, SVD and NMF had similar performance.

4. Experiments with Varieties of Similarities

The paper presents four experiment sets evaluating the dual-space model on relations, composition, phrase similarity, and three classes of word pairs.

  • Four experiment sets evaluate the dual-space model on analogies, noun-modifier composition, phrase similarity, and three classes of word pairs.

4.1 Similarity of Relations

The dual-space model measures semantic relation similarity by combining function similarity with a domain-similarity constraint, and evaluates this approach on SAT analogy questions. It performs as well as the state-of-the-art holistic model, while reformulated ten-choice questions show the value of using two distinct spaces.

  • Evaluation: The SAT evaluation uses 374 multiple-choice analogy questions requiring selection of the choice pair most analogous to the stem pair.One example maps lulling a person into trust to cajoling a person into compliance.
  • Model and task: The model selects the choice pair c:d that maximizes relational similarity simr(a:b, c:d) for a stem pair a:b.Relational similarity is defined using Equation 25.
  • Model and task: simr combines function similarity across pairs with the constraint that domain similarity within pairs, sim2, is at least cross-pair domain similarity, sim3.sim1 measures function similarity, sim2 measures internal domain similarity, and sim3 measures cross-domain similarity.
  • Results: 51.1% accuracy with ten-fold cross-validation was near the 51.6% fine-grid-search accuracy, while coarse-search accuracy ranged from 31.0% to 48.7%.The authors interpret the results as evidence that heuristic search is effective and parameter tuning matters.
  • Results: 47.9% accuracy for simr on the ten-choice reformulated SAT test exceeded sim1 alone at 27.5%, a statistically significant difference.The result supports the claim that function similarity alone is insufficient.
  • Results: The dual-space model matches the current state-of-the-art holistic model and addresses linguistic creativity; the ten-choice results support maintaining separate function and domain spaces.The model was more accurate than the modified models, and the two-space benefit was clearer on the more difficult test.

4.2 Similarity of Compositions

The dual-space model evaluates noun-modifier composition by combining domain and function similarities, using WordNet-derived multiple-choice questions. It outperforms vector addition on the original test and handles order sensitivity and compositionality, while holistic evaluation is artificially favorable.

  • Evaluation: The experiment tests whether dog house can be matched with kennel using WordNet-generated seven-choice noun-modifier questions.The dataset contains 2,180 questions, split into 680 training and 1,500 testing questions.
  • Results: 58.3% accuracy: Equation 31 correctly answers 874 testing questions, skips 22, and incorrectly answers 604.The best testing parameters are kd = 800, pd = 0.3, kf = 100, and pf = 0.6.
  • Limitations of the Holistic Approach: Holistic performance is significantly better than the other models only under ideal conditions requiring rows for the tested bigrams, which does not address linguistic creativity.The experiment design gives the holistic baseline access to representations that would be unavailable for many novel phrases.
  • Constraints: Dropping the constraints a ≠ c and b ≠ c reduces dual-space accuracy from 58.3% to 13.7%, showing that these constraints are important to the tested decision rule.All models benefit from the constraints, which prevent choices corresponding to the modifier or head noun from dominating.
  • Order Sensitivity: On reformulated fourteen-choice questions, dual-space is significantly more accurate than element-wise multiplication and vector addition, unlike those models addressing order sensitivity.The original seven-choice questions do not test word order as strictly.

4.3 Similarity of Phrases

The paper evaluates phrase similarity by combining domain and function similarities for corresponding phrase components. Vector addition performs slightly better without order testing, but dual-space performs significantly better when reversed phrase order is penalized.

  • Evaluation: The evaluation uses 324 phrase pairs across adjective-noun, noun-noun, and verb-object phrases, with human ratings from 18 participants on a 1–7 scale.The dataset is divided into development and evaluation phrase pairs for parameter tuning and testing.
  • Model: Phrase similarity is computed as geo(simd(a, c), simd(b, d), simf(a, c), simf(b, d)) for phrase pairs ab and cd.The equation reflects the requirement that corresponding components have high domain and function similarity.
  • Results: 0.48 average correlation: dual-space is significantly below vector addition using function space at 0.51, while mono-space multiplication reaches 0.47.The reported averages are based on 162 rho values, and the dual-space versus mono-space multiplication difference is not significant.
  • Order Sensitivity: The original dataset does not test order sensitivity, so the authors add reversed phrase pairs and assume humans would rate them 1.Examples include tax charge ∼ rate interest and produce effect ∼ result achieve.
  • Results: With the expanded order-sensitive dataset, dual-space performs significantly better than both vector addition and element-wise multiplication.The authors report that word-order-sensitive instructions are relevant because people may mentally correct incorrectly ordered phrases.
  • Summary: Without order testing, vector addition is slightly better; with order testing, dual-space surpasses both alternatives by a large margin.This contrast is the central comparison across the two phrase-similarity settings.

4.4 Domain versus Function as Associated versus Similar

The paper interprets domain similarity as association and function similarity as semantic similarity, then tests three combined measures against labeled word-pair classes. The results support this division, while suggesting function similarity is harder to measure.

  • Hypothesis: The hypothesis maps domain similarity simd(a, b) to association and function similarity simf(a, b) to semantic similarity.The distinction is tested using similar-only, associated-only, and similar+associated word-pair labels.
  • Similarity Measures: simso is intended to be high when function similarity is high and domain similarity low; simao reverses this pattern; simsa is high when both are high.These three measures operationalize the three labeled classes.
  • Evaluation: Across all three parameter settings, desired labels are concentrated at the tops of the sorted similarity lists, supporting the three-measure classification.The density of desired labels decreases gradually lower in the rankings.
  • Evaluation: Logistic-regression classification with the three similarity measures as features further supports the domain-association and function-similarity hypothesis.The classifier uses ten-fold cross-validation on feature vectors containing simso, simao, and simsa.
  • Limitations: Similar-only appears more sensitive to parameter settings than associated-only and similar+associated, possibly because function similarity is harder to measure.The paper notes that constructing function space is more complex than constructing domain space.

5. Discussion of Experiments

The experiments evaluate the dual-space model across relational, noun-modifier, phrasal, and word-pair similarity tasks. Results support its ability to address order sensitivity and unify semantic relations with composition, although holistic models remain strongest on some tests.

  • Relational similarity: 51.1% accuracy on 374 SAT analogy questions was not significantly different from the 56.1% accuracy of the best past holistic model.Reformulated questions supported the need for both domain and function spaces; function space alone ignored order, while mono space significantly reduced performance.
  • Phrasal similarity: The reformulated 324-pair phrasal-similarity dataset showed a significant dual-space advantage over vector addition and element-wise multiplication on order sensitivity.The evaluation used Mitchell and Lapata’s dataset and a reformulated version designed specifically to test order sensitivity.
  • Word similarity: Results on 144 labeled word pairs supported the hypothesis that domain similarity measures association whereas function similarity measures word similarity.The findings connect the two spaces with distinct aspects of lexical semantic relatedness.
  • Overall findings: The experiments support the dual-space model as a unified approach addressing linguistic creativity, order sensitivity, and adaptive capacity.The paper reports this conclusion across its relational and compositional evaluations.
  • Relational similarity: 50.8% SAT accuracy from function similarity alone significantly exceeded the 43.0% best WordNet result at the 95% confidence level.This comparison used the original five-choice SAT questions and Fisher’s Exact Test.

6. Theoretical Considerations

The dual-space model constructs task-specific similarity from domain and function representations rather than producing one general-purpose phrase vector. Its general scheme combines cosine similarities, supports different text-chunk sizes, and preserves information while leaving automation and complexity as open considerations.

  • Task-specific representation: The model hypothesizes that phrases lack a single stand-alone, task-independent representation suitable for all purposes.A phrase such as dog house is composed differently depending on whether the task is relational or phrasal similarity.
  • Task-specific representation: Similarity composition produces linking structures connecting phrases to other phrases, such as dog house to kennel.These links are presented as an explicit part of meaning and understanding.
  • General similarity scheme: The general scheme encompasses relational, compositional, and phrasal similarity, with task-specific instances expressed through different constraints and cosine combinations.The relational and compositional examples show how the same framework can instantiate distinct similarity measures.
  • General representation: Each text chunk is represented by paired domain and function matrices whose rows encode the words’ domain and function semantics.The representation supports ordered chunks and allows the two spaces to be compared through cosine matrices.
  • General similarity scheme: The composition function f takes cosine similarities among domain and function matrices, not the original row vectors, as its inputs.Its exact specification depends on the task and may be determined by syntactic structure for sentences.
  • Scalability: Representational size grows linearly with phrase length, while the general composition in Equation 37 can grow quadratically rather than exponentially.Some instances may grow less than quadratically, and quadratic growth is described as often acceptable in practical applications.
  • Automation: The paper manually constructed composition functions, but proposes automated similarity composition through constrained search and potentially monotone submodular maximization.Turney’s analogical-mapping algorithm is offered as a possible basis for searching compositions that maximize composite similarity.
  • Scalability: The model avoids fixed-dimensional information loss because representations grow with phrase length, although automating composition may affect the complexity of longer-phrase scaling.The paper identifies automation of composition as an unresolved computational question.

7. Limitations and Future Work

The paper identifies scope and design limitations concerning longer text, morphology, polysemy, language coverage, parameter tuning, and the integration of holistic and dual-space models. It also leaves formal logic and textual entailment, plus automated composition functions, for future work.

  • Future work: Future work should test longer phrases and sentences and investigate how parsing can constrain automatic searches for similarity composition functions.The current discussion identifies longer-text evaluation and syntax-guided search as open directions.
  • Scope limitations: The model currently simplifies morphology by assigning singular and plural forms the same semantic representation.A more sophisticated model would use different representations for different morphological forms.
  • Scope limitations: Polysemy is not addressed, although prior VSM work suggests a possible extension to the dual-space model.The paper treats this as an unresolved research direction rather than presenting an implemented solution.
  • Model integration: Holistic representations remain necessary for idiomatic expressions, while dual-space composition handles linguistic creativity; integrating both approaches remains future work.The paper therefore treats the two approaches as complementary in different cases.
  • Model design: The dual-space model has four tunable parameters: k_d, p_d, k_f, and p_f.The paper characterizes parameter tuning as a possible limitation requiring further research.
  • Model design: Domain and function space construction involved intuition-guided design decisions, leaving systematic exploration and experimental evaluation of that design space for future research.The function-space construction is specific to English and may challenge some other languages.
  • Open semantic problems: Formal logic and textual entailment remain unresolved because the approach focuses on similarity while sentence-level meaning also involves truth and falsity.The paper does not yet know how to join spatial similarity models with symbolic models for truth.

8. Conclusions

The paper concludes that the dual-space model unifies semantic relations and compositions while addressing several central representation challenges. Its novelty lies in combining complementary domain and function spaces and composing similarities rather than vectors.

  • Conclusion: The dual-space model is presented as unifying semantic relations and compositions while addressing linguistic creativity, order sensitivity, adaptive capacity, and information scalability.The authors conclude that the model achieves this goal while acknowledging room for improvement and further research.
  • Novelty: The model’s novelty is its use of two distinct, complementary word-context matrices that work together synergistically.The two matrices support separate domain and function similarity measures.
  • Novelty: Similarity composition offers an alternative to vector composition by combining domain and function similarities in multiple ways.The paper gives geometric-mean combination of the two similarities as one example.
Loading 1309.4035v1…