PaperlayerView exact revision on arXiv

Source-linked AI summary

Morphological Analyzer and Generator for Russian and Ukrainian Languages

Mikhail Korobov

arXiv:1503.07283v1cs.CL

TL;DR

Russian and Ukrainian morphological analysis requires broad lexical coverage and handling of out-of-vocabulary words. pymorphy2 combines compact lexicons with configurable linguistic rules and optional C++ extensions. For Russian, it achieves less than 1% analysis errors and runs at tens of thousands of words per second, while Ukrainian support remains experimental.

  • Problem

    Complete dictionary coverage is impractical because natural-language texts contain rare, new, derived, borrowed, named, and specialized words.

  • Method

    pymorphy2 combines efficiently encoded lexicons with a configurable pipeline of language-specific analyzer-unit rules for vocabulary and out-of-vocabulary words.

  • Results

    Less than 1% errors were reported for both pymorphy2 and Mystem on Russian analysis without disambiguation.

  • Takeaways & Limitations

    pymorphy2 provides a fast, extensible analyzer and generator for Russian and experimental Ukrainian support, with optional C++ acceleration.

  • Takeaways & Limitations

    Quantitative comparison is limited by the small corpus and evaluation choices, while Ukrainian support requires dictionary, rule, and annotated-corpus work.

Abstract

from arXiv · show

pymorphy2 is a morphological analyzer and generator for Russian and Ukrainian languages. It uses large efficiently encoded lexi- cons built from OpenCorpora and LanguageTool data. A set of linguistically motivated rules is developed to enable morphological analysis and generation of out-of-vocabulary words observed in real-world documents. For Russian pymorphy2 provides state-of-the-arts morphological analysis quality. The analyzer is implemented in Python programming language with optional C++ extensions. Emphasis is put on ease of use, documentation and extensibility. The package is distributed under a permissive open-source license, encouraging its use in both academic and commercial setting.

1 Introduction

pymorphy2 is a Russian and Ukrainian morphological analyzer and generator designed for practical natural-language processing. The paper introduces its uses, development context, organization, and open-source distribution.

  • 1 Introduction: Morphological analysis identifies words’ internal structure and grammatical possibilities such as part of speech and number.It is an important NLP pipeline step for richly inflected languages such as Russian and Ukrainian.
  • 1 Introduction: Morphological generation builds a word from its grammatical representation, including lemmatization, inflection, and finding lexemes.
  • 1 Introduction: pymorphy2 is a widely used Russian and Ukrainian analyzer and generator developed since 2012, with Ukrainian support added more recently.Its predecessor, pymorphy1, began development in 2009.
  • 1 Introduction: The paper covers architecture, dictionary-based analysis and generation, out-of-vocabulary rules, ambiguity selection, evaluation, and future improvements.

2 Software Architecture

pymorphy2 combines a simple Python interface with configurable analysis components and optional performance extensions. Its architecture supports analysis, generation, experimentation, and adaptation across languages.

  • 2 Software Architecture: pymorphy2 is implemented as a cross-platform Python3 library with a command-line utility and optional C++ extensions.It supports Python 2.x and Python 3.x, with 600+ unit tests and coverage above 90%.
  • 2 Software Architecture: With optional C++ extensions or PyPy, parsing usually reaches tens of thousands of words per second in a single thread.In some cases it exceeds 100000 words per second; memory consumption is about 15MB, or about 30MB including the interpreter.
  • 2 Software Architecture: Analysis and generation are exposed through a simple API for words, analyses, grammatical tags, inflection, lemmatization, and lexeme construction.The API also supports agreement operations and work with dictionaries, grammemes, and tags.
  • 2 Software Architecture: The analyzer is a configurable pipeline combining vocabulary-word analysis with rules for out-of-vocabulary words.Individual units can be customized, disabled, parameterized with language-specific data, or replaced by user-created units.

3 Analysis of Vocabulary Words

pymorphy2 stores large lexicons in compact structures that support fast vocabulary-word analysis, lemmatization, and inflection. Its representation factors lexemes into reusable paradigms and DAFSA entries.

  • 3 Analysis of Vocabulary Words: Russian dictionaries contain approximately 5 ∗106 word forms and 0.39 ∗106 lemmas, while the experimental Ukrainian dictionary contains approximately 2.5 ∗106 word forms.Russian data comes from OpenCorpora; Ukrainian data is based on LanguageTool data.
  • 3 Analysis of Vocabulary Words: DAFSA stores each word form with its paradigm ID and form index, while encoded information reconstructs lexemes on demand.The representation uses compact numeric encodings and stores word forms in a DAFSA.
  • 3 Analysis of Vocabulary Words: A paradigm is an inflection pattern represented by prefix, suffix, and tag triples for each word form.Lexeme stems are shared across forms, allowing lexemes to be factored into a stem and a paradigm.
  • 3 Analysis of Vocabulary Words: 5 ∗106 analyses and about 3 ∗106 unique Russian word forms occupy less than 7MB of RAM and support hundreds thousand lookups/sec from Python.The source XML is about 400MB on disk; shared analyses among words with similar endings improve compactness.
  • 3 Analysis of Vocabulary Words: Given a word’s paradigm and form index, pymorphy2 retrieves tags and reconstructs the lexeme for lemmatization or inflection.Analysis follows DAFSA transitions and traverses the remaining subtree to obtain all possible paradigm/form-index pairs.
  • 3 Analysis of Vocabulary Words: pymorphy2 preserves distinctions involving optional Russian ё and Ukrainian ґ by returning analyses according to the letters present or absent in input.DAFSA traversal follows alternative ё/е or ґ/г transitions without enumerating every replacement combination.

4 Analysis of Out-of-Vocabulary Words

Because dictionary coverage cannot include every rare, new, derived, borrowed, or specialized word, pymorphy2 applies ordered, language-specific rules to analyze out-of-vocabulary forms. These rules operate within a unified configurable analyzer pipeline.

  • 4 Analysis of Out-of-Vocabulary Words: A long tail of rare words, new formations, derivations, loanwords, names, locations, and special terms makes complete lexicon coverage impractical.Even doubling a lexicon may increase coverage only slightly.
  • 4 Analysis of Out-of-Vocabulary Words: Large tagsets make delaying unknown-word classification to later processing stages challenging for Slavic languages.OpenCorpora contains more than 4500 different tags.
  • 4 Analysis of Out-of-Vocabulary Words: pymorphy2 uses a novel combination of analyzer-unit rules for unknown words, applied in a language-specific order.
  • 4 Analysis of Out-of-Vocabulary Words: For listed immutable prefixes, pymorphy2 removes the prefix, analyzes the remainder, and reattaches the prefix.The remainder may itself be out of vocabulary, and prefix lists are encoded to DAFSAs for faster matching.
  • 4 Analysis of Out-of-Vocabulary Words: Suffix analysis assumes the whole word can be parsed like a dictionary suffix when length and open-class conditions are satisfied.The suffix must be longer than 3 characters, the remaining prefix no longer than 5, and the suffix must be an open-class word.
  • 4 Analysis of Out-of-Vocabulary Words: To find suffixes, pymorphy2 progressively treats initial characters as prefixes and looks up each remainder in the dictionary.

4.3 Endings Matching

pymorphy2 uses shared word endings to predict grammatical analyses, storing them compactly and retrieving them from longer to shorter suffixes. The resulting paradigm information also supports morphological generation, including for words outside the vocabulary.

  • Common endings in Russian and Ukrainian often share grammatical forms, making endings useful for predicting possible analyses.
  • The analyzer extracts endings of lengths 1 to 5, retains frequent analyses, and discards non-productive paradigms and rare endings.
  • The cleaned ending-to-analysis mapping is encoded in DAFSA using frequency, paradigmId, and formIndex information.
  • At prediction time, pymorphy2 checks endings from length 5 to 1, stops at the first match, and ranks analyses by frequency.
  • A word plus its paradigmId and formIndex can restore and inflect its lexeme, so generation works for out-of-vocabulary words.
  • The ending-based analyzer produces only open-class parts of speech and separately handles ё and paradigm prefixes.

4.4 Words with a Hyphen

pymorphy2 explicitly handles hyphenated words rather than uniformly splitting them into separate tokens. Its heuristics cover several hyphenated adverb patterns, including по- constructions.

  • Unlike some other morphological analyzers, pymorphy2 handles words containing a hyphen.
  • Other systems may split hyphenated compounds because of technical difficulty, whereas pymorphy2 analyzes them as supported forms.
  • The supported hyphenated categories include adverbs, particles, and compounds with left and right parts separated by a hyphen.
  • A Russian word is treated as a hyphenated adverb when it begins with по-, exceeds five characters, and becomes a full singular dative adjective after removing по-.
  • Examples of this adverb pattern include по-северному and по-хорошему.

Particles Separated by a Hyphen

pymorphy2 supports hyphen-separated particles and compound words by parsing their components and combining compatible analyses. Its compound-word heuristics are limited to words with a single hyphen.

  • Particles Separated by a Hyphen: Hyphen-separated particles are parsed by removing a recognized particle, analyzing the remaining word, and reattaching the particle.
  • Compound Words with a Hyphen: For hyphenated compounds, the main challenge is deciding whether the left part inflects with the right part or acts as a fixed prefix.
  • Compound Words with a Hyphen: pymorphy2 parses both parts separately, combines compatible analyses with joint inflection, and also adds an analysis treating the left part as fixed.
  • Compound Words with a Hyphen: Only words with a single hyphen use these heuristics; multiple-hyphen words may represent interjections or phrases.

4.5 Other Tokens

pymorphy2 adds specialized handling for initials and non-lexical tokens, while supporting inflection of out-of-vocabulary words through its analyzer-unit pipeline. The pipeline allows analyzer units to construct lexemes incrementally.

  • Initials: An initial abbreviates a person’s first or patronymic name and is parsed as a fixed singular noun with gender and case variants.
  • Initials: For first names, pymorphy2 builds separate male and female lexemes; for patronymics, it returns one lexeme.
  • Non-lexical Tokens: Non-lexical tokens receive tags for punctuation, Latin text, integers, real numbers, and Roman numerals.
  • Non-lexical Tokens: Handling non-lexical tokens during morphological analysis lets users use simpler tokenizers and keeps token information in a common format.
  • Out-of-vocabulary Inflection: Inflection is supported for out-of-vocabulary words by tracking analyzer units and asking the final unit to construct the resulting lexeme.
  • Out-of-vocabulary Inflection: A prefix-removal unit can obtain a previous lexeme and reattach the prefix to every form in the resulting lexeme.

5 Probability Estimation

pymorphy2 estimates P(analysis|word) from partially disambiguated OpenCorpora data, using single-analysis occurrences and Laplace smoothing. Ukrainian probabilities are assigned uniformly because no comparable freely available corpus existed.

  • pymorphy2 estimates P(analysis|word) as P(tag|word) using partially disambiguated OpenCorpora data.The estimate is applied to words with multiple analyzer analyses.
  • Single-analysis corpus occurrences supply maximum-likelihood estimates with Laplace add-one smoothing for multiply analyzable words.
  • For words without PMLE(tag|word) estimates, parsing assigns probabilities uniformly.
  • Ukrainian probabilities are assigned uniformly because no freely available Ukrainian corpus comparable to OpenCorpora was available.

6 Evaluation

The evaluation compares pymorphy2 with Mystem on manually disambiguated Russian text despite incompatible tagsets and corpus-dictionary dependencies. Both analyzers made less than 1% errors without disambiguation, but the small corpus and evaluation choices limit quantitative conclusions.

  • Evaluation quality is difficult to compare because analyzers and corpora use incompatible tagsets, while corpus-derived dictionaries can produce optimistic scores.
  • Both pymorphy2 and Mystem made less than 1% errors without disambiguation.The comparison used 2,498 manually disambiguated tokens from OpenCorpora and ruscorpora.ru after tag conversion to Mystem format.
  • Both analyzers had similar analysis quality, and reported numbers depended on choices about abbreviations, hyphenated words, and other evaluation minutiae.
  • Cross-corpus checking found 1 annotation error in OpenCorpora and 6 in ruscorpora data when parsed with the other analyzer.The paper presents this as evidence for using cross-corpora tools to check annotations.

7 Conclusion and Future Plans

pymorphy2 is openly developed and maintained, with ongoing work on analysis quality, language support, and performance. Future plans include Belarusian support, while Ukrainian support and analyzer comparison remain constrained by current resources and tooling.

  • The MIT license and permissively licensed dependencies encourage usage and contributions from volunteers working on dictionaries, corpora, related tools, and pymorphy2.
  • pymorphy2 analysis can be improved for names, locations, abbreviations, hyphenated words, and ordinal numbers, including digit notation.
  • A robust tagset conversion library may be needed for better comparisons among pymorphy, pymorphy2, Mystem, and other morphological analyzers.
  • Ukrainian support is experimental and needs dictionary work, Ukrainian-specific out-of-vocabulary rules, and an annotated corpus or frequency list for better P(tag|word) estimates.
  • Plans include adding Belarusian language support based on the Belarusian N-korpus grammar database.
  • pymorphy2 already processes tens of thousands of words per second in a single thread, but further speed improvements remain possible.
Loading 1503.07283v1…