PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Space of Parts of Speech

Jiří Milička, Ivan Kraus, Arnold Stanovský, Anna Vysloužilová, Barbora Štěpánková, Lenka Fárová, Vojtěch Cink, Šárka Dohnalová

arXiv:2608.15443v1cs.CL

TL;DR

Parts-of-speech categories can be fuzzy because some words do not properly belong anywhere. This study maps words from five languages into a three-dimensional neural embedding space, producing visualizations of semantic relationships and category structure.

  • Problem

    Some words do not properly belong to a single part of speech, motivating analysis of fuzziness in parts-of-speech categorization.

  • Method

    The study trains a classifier on embeddings and maps words from five European languages into a three-dimensional space for parts-of-speech analysis.

  • Results

    The resulting visualizations show relationships among parts of speech, including prominent NOUN, VERB, and ADJ regions.

  • Takeaways & Limitations

    The visualizations help navigate the semantic latent space of parts of speech.

  • Takeaways & Limitations

    Individual words are represented as dots, and the representation does not capture the real shape of the space.

Abstract

from arXiv · show

Parts of speech categorization is understood in the European linguistic tradition as crisp categorization, which is also reflected in corpus linguistics, where each disambiguated token is assigned exactly one POS. However, the assigned categories are largely determined by arbitrary decisions distilled into annotation manuals. Since some words stand between parts of speech in their semantics or typical syntax, and some parts of speech are closer to each other than others, POS categorization seems inherently fuzzy. We analyze this fuzziness using word2vec embeddings, training a neural network to reduce their high dimensionality to three dimensions relevant for determining parts of speech. This creates a three-dimensional space onto which we map several thousand words, revealing which are prototypical and which lie on the boundaries, and visualizing relationships between parts of speech. The study uses Universal Dependencies POS tags for French, Czech, Finnish, Russian, and English.

1 Introduction

The study treats parts-of-speech categories as empirically fuzzy and analyzes their semantic and syntactic relationships using embeddings. A neural network reduces embedding information to three visualizable dimensions for examining POS relationships across five European languages.

  • Motivation: Words can be borderline between parts of speech, while some categories are closer than others, and traditional classification often reflects convention rather than systematicity.Examples include Arabic masdars, which have verbal semantics but noun-like morphology, and Slavic ordinal numerals, which behave like adjectives.
  • Method: The study analyzes this fuzziness empirically using corpus data and embedding representations of semantic and syntactic relationships.Word2vec reduces high-dimensional collocation profiles to manageable vectors whose dimensions encode semantic value.
  • Method: A classifier trained on embedding–part-of-speech pairs maps semantic relationships between words onto POS categorization.Its internal representation is intended to predict parts of speech for unknown words with reasonable accuracy.
  • Contribution: Three dimensions reduce several hundred embedding dimensions to information relevant for determining parts of speech and form the study’s main visual output.The three-neuron bottleneck is chosen because humans can easily visualize and navigate three-dimensional space.
  • Scope and limitation: The analysis examines existing Universal Dependencies categories within the European linguistic tradition for French, Czech, Finnish, Russian, and English.The language restriction provides reasonably comparable data but leads the study to avoid extensive typological comparison.

2 Broader context of the study

European parts-of-speech classification descends from an eight-category Greek tradition, while other linguistic traditions use fewer categories and different criteria. The study adopts Universal Dependencies but frames POS categories as overlapping, with prototypical cores and fuzzy boundaries.

  • Historical background: The European tradition traces eight parts of speech to Greek grammarians, who classified them using morphological, syntactic, and semantic properties.The system was probably introduced by Aristarchus in the 2nd century BCE and described by Dionysius Thrax.
  • Cross-linguistic context: Other linguistic traditions often distinguish fewer parts of speech using different criteria, and the universality of POS categories remains debated.The debate concerns both the grammatical versus semantic nature of categories and their cross-linguistic applicability.
  • Universal Dependencies: The study uses Universal Dependencies because its categories provide cross-linguistic consistencies based on experience across many languages.Although UD follows the traditional eight-part classification, it distinguishes 17 UPOS classes and other textual elements.
  • Universal Dependencies: UD assumes that every word in any language can be assigned to one class, while class definitions vary across languages despite shared names.The shared names indicate at least partially similar syntactic and, to a greater extent, semantic features.
  • Category fuzziness: Modern grammatical analysis recognizes that word classes overlap rather than mutually exclude one another, with prototypical cores and fuzzy peripheries.This motivates treating category boundaries as non-neat and is supported by the study’s data.

3 Data

The study uses comparable 100-dimensional static word2vec embeddings trained on unlemmatized CoNLL17 corpora, while restricting each language to its 10,000 most frequent forms. POS distributions are derived from filtered InterCorp frequency dictionaries as normalized vectors over the UPOS categories retained for analysis.

  • Embeddings: The embeddings use Continuous Skipgram word2vec with a context window of size 10 and are static, enabling analysis of word forms without additional texts.They come from the NLPL word embeddings repository.
  • Embeddings: 100-dimensional embeddings are trained on unlemmatized CoNLL17 corpora compiled from CommonCrawl and Wikipedia, facilitating comparisons between embeddings.The corpora are associated with Ginter et al. (2017).
  • Vocabulary: 10 000 most frequent forms are selected for each language to reduce the overall data size.This frequency-based reduction defines the vocabulary analyzed in each language.
  • POS frequency data: Filtered InterCorp v16ud with automatic UD annotation supplies form-frequency dictionaries for each analyzed language.Undesirable UPOS are excluded during compilation because PUNCT and SYM are non-word tokens, while INTJ, PROPN, and X often involve strange-looking or misclassified tokens.
  • POS frequency data: Each form receives a 10–12-item POS vector whose values are relative frequencies for the retained UPOS and sum to 1.The vector length depends on the language, and each item ranges from 0 to 1 inclusive.

4 Methodology

The study prepares POS-annotated word embeddings by splitting lemmas across two roughly equal sets and training paired neural networks for cross-validation. Each network compresses embeddings through hidden layers to a three-neuron bottleneck, whose activations are mapped directly into 3D space for visualization.

  • Data preparation: Data were split into roughly equal sets A and B, with all forms of each lemma confined to one set to prevent validation leakage.The split supports training on one set and validation on the other.
  • Model training: Two structurally identical networks, NNA and NNB, were trained on opposite sets and validated against each other while classifying embeddings into UD POS.Different closed-class POS shares across the training sets could leave some POS unrepresented in a network.
  • Network architecture: The networks used hidden layers of 128, 64, and 16 neurons, followed by a 3-neuron bottleneck before final POS classification.The bottleneck was introduced to visualize relationships between parts of speech.
  • Network architecture: All three hidden layers used ReLU activation and dropout rate 0.3, and each network was trained for 30.
  • Visualization: For visualization, activation values from the bottleneck layer were mapped for each wordform into 3D space without further transformations.

5 Results · 5.1 French

The French projection clearly separates the main VERB, ADJ, NOUN, and NUM groups while revealing extensive boundary blending caused by polyfunctionality, homonymy, and collocation. These results also highlight differences between French grammatical traditions and French UD conventions, especially for determiners, numerals, conjunctions, and presentatives.

  • 5.1.1 French POS: French grammatical definitions commonly use morphology, semantics, and syntax to distinguish nine parts of speech, although grammars disagree about presentatives and conjunction subdivisions.Some grammars add presentatives/introducers and distinguish coordinating from subordinating conjunctions.
  • 5.1.2 French UD: French UD differs from grammar traditions by splitting articles into DET and PRON, omitting presentatives, defining three auxiliaries, splitting conjunctions, and assigning cardinal numerals to NUM.UD treats faire as an auxiliary in causative constructions, while French grammars generally do not; other numerals receive ADJ or ADV tags.
  • 5.1.2 French UD: French UD tokenization sometimes mishandles hyphenated expressions and contractions, including c’est-à-dire and multi-word tokens such as duquel.These tokenization issues can contribute to misclassification.
  • 5.1.3 Analysis: The French visualization clearly distinguishes VERBs, ADJs, NOUNs, and NUMs, while ADVs form a less distinct group and PRONs occupy several mixed locations.Some PRONs cluster with SCONJs, CCONJs, and DETs, whereas others blend into the ADJ tentacle.
  • 5.1.3 Analysis: Blending among VERB, ADJ, and NOUN tentacles reflects polyfunctional forms, homonymy, and context-dependent interpretations such as oublié, inconnu, and pêcher.ADVs also blend with ADJs through frequent near-exclusive collocation, while possessive and interrogative DETs and possessive PRONs blend into ADJs.
  • 5.1.4 French conclusion: The three most prominent French groups are NOUNs, VERBs, and ADJs, whose blending is motivated mostly by polyfunctionality and homonymy.Numerals form a separate group within the CONJs, ADPs, and PRONs cluster, indicating semantic behavior distinct from their sentence-level determiner or adjective functions.

5.2 Czech

The Czech embedding map forms prominent VERB, ADJ, NOUN, NUM, and smaller ADV tentacles, while PRON and DET form a separate, weakly divided group. Blending reflects homonymy, word-form transitions, idioms, and syntactic or semantic polyfunctionality, whereas functional words cluster less distinctly.

  • Czech embedding map: VERB, ADJ, NOUN, and NUM form clear tentacles, while ADV is smaller but visible.PRON and DET form a separate group at the NOUN–ADJ confluence, with DET closer to ADJ but without a pronounced split.
  • Transitions and blending: ADJ–NOUN blending mainly reflects collocations, ellipses, and formal homonymy, while ADJ–VERB blending centers on short adjectives and passive participles.The VERB–NOUN space contains parts of phrasal verbs and words used in vocatives, exclamations, or commands.
  • Functional and grammatical words: Functional and grammatical words form a less clear middle cluster, likely reflecting limited semantics and their limited number in Czech.Conjunctions, particles, and secondary prepositions are often polyfunctional or homonymous, while some primary prepositions approach NOUNs or VERBs through frequent co-occurrence.
  • Czech embedding map: The prominent VERB, ADJ, and NOUN regions reflect their syntactic, morphological, and semantic behavior.Their transitions correspond to homonymy, word-form transitions, and use within idioms or phrases.
  • Numerals: NUM forms a separate tentacle centered on cardinal numerals, including forms that also function as pronouns or names.This pattern supports treating NUM as a separate POS despite doubts in Czech.

5.3 Finnish

Finnish POS boundaries are blurred by lexicalization, inflectional overlap, and context-dependent category membership. The embedding visualizations reveal prominent NOUN and VERB regions, blending among NOUN, VERB, and ADJ, and dispersed or weakly separated minor categories.

  • Finnish POS system: Finnish POS boundaries are blurred because lexicalized forms can belong to multiple categories depending on context.The word loukkaantunut can function as part of a verb form, a noun, or an adjective.
  • Set A visualization: Set A shows four clear VERB, NOUN, ADJ, and NUM tentacles, with ADJ shorter and closest to NOUN.VERB-tagged participles occur in NOUN and ADJ regions, while some participles are tagged as ADJs; VERB and NOUN also blend through homonymy.
  • Set A visualization: ADVs lack a distinct tentacle, clustering mainly near VERB and NOUN, while ADPs blend more with NOUNs through lexicalized nominal forms.Examples include väli ‘space’ and välillä ‘in between’.
  • Set B visualization: Set B has denser, shorter, and closer VERB, NOUN, and ADJ tentacles, while NUM is thin and almost separated.Many TTAVA-participle forms are tagged as ADJ rather than VERB, possibly reflecting lexicalization.
  • Overall visualization: Across the Finnish data, NOUN and VERB are most prominent, ADJ is shorter, NUM is separate mainly through cardinals, and ADV and PRON are dispersed.Ordinals are closer to ADJs, while ADVs, PRONs, and ADPs spread across categories; CCONJs and SCONJs are nearly absent.

5.4 Russian

Russian shows distinct embedding regions for VERBs, NOUNs, ADJs, and NUMs, while ADVs are less separated. The overlaps and boundary positions correspond mainly to Russian morphosyntactic patterns, including short-form adjectives, substantivized adjectives, and the absence of a present-tense auxiliary verb.

  • Visualization: VERBs, NOUNs, ADJs, and NUMs form clear tentacles, whereas ADVs typically show the least separation.Direction- and location-expressing ADVs are the most distinct from other parts of speech, and the NUM tentacle is very isolated.
  • Visualization: ADJ–NOUN overlap involves adjectival forms established as nouns, while VERB–NOUN overlap is minimal and strongest for infinitives.The examples include chayevyye ‘tips’ for ADJ–NOUN overlap and infinitives such as zhenit’sya for movement toward NOUNs.
  • Visualization: Only forms of odin ‘one’ shift notably toward ADJs because this numeral agrees grammatically with the counted object.This agreement distinguishes odin from other numerals in the visualization.
  • Visualization: Short-form ADJs overlap with VERBs because they commonly function as nominal predicates rather than attributive modifiers.The visualization identifies the VERB–ADJ overlap as exclusively involving short-form adjectives.
  • Visualization: Predicatives shift toward VERBs but not more than typical full-meaning ADVs, despite their predicate function.The paper notes that few predicatives occur in the data and considers their position the main surprising aspect of the Russian visualization.

5.5 English

English visualisations reveal distinct NOUN, VERB, and ADJ tentacles, with smaller NUM and ADV tentacles. The main tentacles are relatively tight but blend substantially, suggesting that some POS are semantically closer than traditional grammar definitions imply.

  • English POS structure: English shows three main tentacles for NOUNs, VERBs, and ADJs, plus smaller tentacles for NUMs and ADVs.The visualisation also shows clear tentacles for the four main POS identified in the discussion, although the results paragraph names three main tentacles.
  • Blending and transitions: The three main tentacles occupy tighter positions than in other languages, with especially substantial blending between VERB and NOUN.This blending is mainly caused by polyfunctional words, including -ing verbal nouns, noun-like infinitives, and present-tense verb forms also usable as plurals.
  • Blending and transitions: NOUN–ADJ blending is also associated mainly with polyfunctional words and words characterized by typical collocations or monocollocability.Examples include secret, commercial, alternative, and vocational.
  • Interpretation: ADJ, NOUN, and VERB tentacles are unusually close, indicating that their semantic relationships may be tighter than English grammar definitions imply.The discussion still treats prototypical NOUNs that are well separated from VERBs as clear, while considering blended cases potentially controversial.
  • Exceptional patterns: NUMs forming their own separate tentacle is surprising because they do not have their own POS, while some ADVs blend into other POS spaces.Blending ADVs are mainly spatial adverbs or words with monocollocability or typical collocations.

6 Conclusions

The study’s main contribution is a visualization of the Universal Dependencies semantic latent space of parts of speech, based on corpus embeddings reduced by a neural-network POS classifier. The analyses illustrate how the visualization can guide interpretation across languages and traditions, while the method also applies beyond Universal Dependencies.

  • Main contribution: The visualization, rather than the POS classifier, is the study’s main output for navigating the semantic latent space of Universal Dependencies parts of speech.The classifier performs reasonably well and is necessary for the visual mapping, but optimizing classification would require a different neural-network topology.
  • Interpretation: The presented analyses are intended as a guide to reading the visualization and as encouragement for readers to discover their own meanings in the data.They are explicitly not a comprehensive description of the results.
  • Cross-linguistic findings: Across the selected European languages, Finnish did not produce results that appeared more chaotic than those of the other languages, despite reported uncertainty about its part-of-speech definitions.The language set included four Indo-European languages and one Uralic language, enabling comparisons across relatedness and linguistic traditions.
  • Future work: Systematic cross-linguistic comparisons and additional, particularly non-European, languages remain a suggested direction for future work.The authors leave this undertaking to readers because they lack the necessary expertise and resources.
  • Applicability: The method can be applied to frameworks beyond Universal Dependencies to compare different traditions both across languages and within a single language.This extends the method’s comparative use beyond the framework and multilingual setting studied here.

Data Availability

The study makes its data, code, visualizations, model analyses, and interactive charts publicly available online.

  • Data Availability: Data, scripts, visualizations, and further model analyses, including F1 scores and confusion matrices, are available at https://osf.io/gm6zd/.The interactive charts are also mirrored at a separate URL beginning https:.
Loading 2608.15443v1…