PaperlayerView exact revision on arXiv

Source-linked AI summary

MorphTE: Injecting Morphology in Tensorized Embeddings

Guobing Gan, Peng Zhang, Sunzhu Li, Xiuqing Lu, Benyou Wang

arXiv:2210.15379v1cs.CLcs.AI

TL;DR

Large word-embedding matrices consume substantial storage and memory, limiting deployment on resource-constrained devices. MorphTE represents words as tensor products of morpheme vectors, and the paper reports over 20-fold compression without performance loss on translation datasets.

  • Problem

    Large vocabularies and embedding dimensions make word-embedding matrices costly to store and access, limiting deployment on resource-limited devices.

    Source: §1 · p. 1

  • Method

    MorphTE represents word embeddings as tensor products of morpheme vectors, using their semantic and grammatical information in compressed embeddings.

    Source: §1 · p. 2

  • Results

    More than 20 times embedding-parameter compression maintained original performance on four translation datasets, and MorphTE outperformed related compression methods.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    Meaningful morphemes and smaller morpheme vectors let MorphTE combine morphological knowledge with space-efficient embeddings.

    Source: §1 · p. 2

    Generating word embeddings introduces computational overhead, which the authors describe as light based on Appendix G analysis.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

In the era of deep learning, word embeddings are essential when dealing with text tasks. However, storing and accessing these embeddings requires a large amount of space. This is not conducive to the deployment of these models on resource-limited devices. Combining the powerful compression capability of tensor products, we propose a word embedding compression method with morphological augmentation, Morphologically-enhanced Tensorized Embeddings (MorphTE). A word consists of one or more morphemes, the smallest units that bear meaning or have a grammatical function. MorphTE represents a word embedding as an entangled form of its morpheme vectors via the tensor product, which injects prior semantic and grammatical knowledge into the learning of embeddings. Furthermore, the dimensionality of the morpheme vector and the number of morphemes are much smaller than those of words, which greatly reduces the parameters of the word embeddings. We conduct experiments on tasks such as machine translation and question answering. Experimental results on four translation datasets of different languages show that MorphTE can compress word embedding parameters by about 20 times without performance loss and significantly outperforms related embedding compression methods.

1 Introduction

MorphTE compresses word embeddings by representing them as tensor products of meaningful morpheme vectors, which also injects morphological knowledge. The paper reports that it preserves performance at over 20-fold embedding-parameter compression across translation datasets.

1 Introduction

  • Large vocabularies and embedding dimensions make word-embedding matrices costly to store and access, limiting deployment on resource-limited devices.Existing compression approaches include product quantization and matrix or tensor decomposition; small decomposition components lack specific meaning, and approximate substitutions can hurt performance on complex NLP tasks such as machine translation.Source: §1 · pp. 1–2
  • MorphTE represents each word embedding as an entangled tensor product of its morpheme vectors, combining morphology with tensor-based compression.The paper reports that fine-grained morphemes can improve embedding quality, citing prior work [4].Source: §1 · p. 2
  • Meaning-bearing roots and grammatical affixes inject semantic and grammatical knowledge, while shared morpheme vectors connect morphologically similar words.For example, “kind” is the root shared by “unkindly” and “unkindness”; “un,” “ly,” and “ness” mark negation, adverbs, and nouns, respectively.Source: §1 · p. 2
  • MorphTE trains and stores smaller morpheme vectors instead of full word vectors, reducing parameters; three 8-dimensional vectors can generate a 512-dimensional embedding.The morpheme vocabulary is also smaller than the word vocabulary.Source: §1 · p. 2
  • More than 20 times fewer embedding parameters preserved original performance on four translation datasets, while other compression methods hurt performance.MorphTE also improved on Word2ket by 0.7, 0.6, and 0.6 BLEU on De-En, En-It, and En-Ru, respectively.Source: §1 · pp. 2–3

2 Related Work

Related work covers morphological augmentation and decomposition-based compression. MorphTE combines these directions by using meaningful morphemes as the units for generating compressed embeddings.

Morphologically-augmented Embeddings

  • Morphological embedding approaches integrate morphological information to improve word representations, including methods that sum morpheme and surface-form vectors and Morphological RNNs that use morphemes as recursive units [4] [2] [10].MorphTE also uses morphemes and is a decomposition-based compression method, like Word2ket [27].Source: §2 · p. 3

Decomposition-based Compression Embeddings

  • Matrix-factorization methods approximate embedding matrices with smaller matrices, but their low-rank bottleneck limits expressive ability at high compression ratios [18] [9] [1] [35].Some methods first group words by frequency and factorize each block.Source: §2 · p. 3
  • Tensor methods use Tensor Train, entangled-tensor, or Kronecker decompositions; MorphTE differs by generating embeddings from meaningful morphemes rather than ungrounded vectors or tensors [15, 41, 27, 34].Word2ket and Kronecker Embeddings are described as consistent with Canonical Polyadic decomposition [27] [34].Source: §2 · p. 3

3 Preliminary

This preliminary section introduces tensor products and entangled tensors, then describes Word2ket as representing a word embedding with a rank- and order-parameterized tensor product.

3.1 Tensor Product Space and Entangled Tensors

  • The tensor-product basis combines basis vectors from the component spaces, with the resulting space having the product of their dimensions.Source: §3.1 · p. 3
  • A cumulative tensor product of n subspaces has order n and dimensionality equal to the product of their subspace dimensions.Source: §3.1 · p. 3
  • An entangled tensor is a sum of simple tensors, and its rank is the smallest number of simple tensors in that sum.Source: §3.1 · pp. 3–4

3.2 Tensorized Embeddings with Tensor Product

  • Word2ket represents a word embedding as an entangled tensor of rank r and order n.Its factors are small vectors combined by tensor product.Source: §3.2 · p. 4
  • With d = 512, q = 8, n = 3, and r = 2, Word2ket reduces a word embedding from 512 parameters to 48.The figure depicts the rank-2, order-3 setting.Source: §3.2 · p. 4, Figure 2 · p. 4

4 Methodology: Morphologically-enhanced Tensorized Embeddings

MorphTE responds to compressed embeddings’ lack of lexical knowledge by representing words through morpheme-based tensor products. It also shares morpheme vectors across related words and compares this design with Word2ket.

4.1 Motivation to Introduce Morphology

  • Decomposition-based compression can lack lexical knowledge and interpretability, motivating explicit morphology in resource-limited settings.The motivation argues that explicit knowledge can compensate for smaller parameter scales.Source: §4.1 · p. 4

4.2 Definition of MorphTE

  • MorphTE defines a word embedding as a sum of cumulative tensor products of morpheme vectors from multiple representation functions.The functions can be viewed as morpheme subspaces; their count is the rank, while the number of morphemes is the order.Source: §4.2 · p. 5

4.3 Workflow of MorphTE

  • The workflow segments words into fixed-length morpheme sequences, padding shorter words and concatenating excess morphemes when sequences are truncated.Each row of the resulting matrix represents one word's morpheme sequence.Source: §4.3 · pp. 5–6
  • Rank-one MorphTE tensor-products q-sized morpheme embeddings into a qn-sized vector; the general version sums r rank-one embeddings.Rank one is the special case where r = 1.Source: §4.3 · p. 6

4.4 Difference between MorphTE and Word2ket

  • MorphTE shares morpheme vectors across morphologically similar words, whereas Word2ket treats those words as unrelated.For example, “unkindly” and “unkindness” share “un” and “kind” vectors and are both related to “unkind.”Source: §4.4 · p. 6
  • MorphTE's parameter count is over 5× smaller than Word2ket's when rank, order, and subspace dimensionality match.The stated ratio is n|V|/|M|; with |V|/|M| > 2.5 and n ≥ 2, the parameter saving exceeds fivefold.Source: §4.4 · pp. 6–7

5 Experiments

Experiments compare MorphTE with embedding-compression and morphology-based baselines across translation, question answering, and natural language inference. They report strong performance at high compression and examine how rank, order, embedding size, and generation overhead affect the method.

5.1 Experimental Setup

  • The experiments compare compression and morphology baselines under roughly equal parameter counts, using shared morpheme segmentation for MorphTE, MorphSum, and MorphLSTM.They cover machine translation, WikiQA retrieval-based question answering, and SNLI natural language inference, with task-specific metrics and model setups.Source: §5.1 · p. 7

5.2 Main Results

  • At 20× compression, MorphTE maintains the original embedding performance on all four translation datasets, while no other compression method does.At this ratio, it improves BLEU by 0.4 on De-En and 0.3 on En-Ru, and matches the original scores on the other datasets.Source: §5.2 · p. 7, Table 2 · p. 8
  • MorphTE maintains original-model performance and outperforms compression baselines on WikiQA and SNLI, as well as across four translation datasets.The translation tests include separate source and target dictionaries for De-En and shared dictionaries for the other datasets.Source: §5.2 · p. 8, Table 3 · p. 8
  • MorphSum and MorphLSTM improve over original embeddings, while random vector sharing scores below Word2ket and MorphTE on translation datasets.The authors interpret these comparisons as evidence that morpheme-based morphological knowledge benefits embeddings.Source: §5.2 · p. 8, Table 4 · p. 8
  • At about 40× compression, MorphTE beats other compression baselines but scores below the original model; increasing compression from 20× to 40× reduces embedding parameters only from about 1M to 0.5M.At about 20×, embedding parameters fall from about 20M to 1M, and their share of total parameters falls from 20.1%-31.8% to 1.2%-2.3%.Source: §5.2 · pp. 8–9, Table 5 · p. 9

5.3 Ablation Study

  • At about 20× compression on De-En, order 3 achieves the best results among tested maximum morpheme counts of 2, 3, and 4.More than 90% of words have no more than three morphemes.Source: §5.3 · p. 9, Table 6 · p. 9
  • As MorphTE rank increases from 1 to 32, embedding parameters rise linearly while SNLI accuracy first increases and then stabilizes.The authors report that rank can be adjusted to balance compression and model performance.Source: §5.3 · p. 9, Figure 6 · p. 9
  • With word embedding sizes of 216, 512, and 1000, MorphTE compresses parameters by factors of 10, 19, and 30, respectively, while matching or exceeding the original model’s BLEU scores.The 512-dimensional case is 0.4 BLEU above the uncompressed model; at 1000 dimensions, the gain is 0.6 BLEU.Source: §5.3 · pp. 9–10, Table 7 · p. 10

6 Conclusion

The conclusion presents MorphTE as a combination of morphological knowledge and tensor-product compression for learning space-efficient word embeddings. The paper reports that it handles varied tasks and languages, outperforming related decomposition-based methods while compressing embeddings by dozens of times without hurting original-model performance.

6 Conclusion

  • MorphTE combines morphological prior knowledge with tensor-product compression, and the paper reports dozens-fold embedding compression without hurting original-model performance across varied tasks and languages.The conclusion specifically names Word2ket and Tensor Train as related methods that MorphTE outperforms.Source: §6 · p. 10

Checklist

The checklist reports that the paper describes its contributions and scope, but not its limitations or potential negative societal impacts. The additional material outlines word-formation phenomena, tensor-product definitions, and parameter accounting for embedding methods.

Checklist

  • The checklist marks contributions and scope, experimental reproducibility materials, training details, and compute resources as reported, but error bars as unreported.It also marks discussion of limitations and potential negative societal impacts as absent.Source: Checklist · p. 14

A Linguistic Phenomena in Word Formation

  • Inflection, derivation, and compounding create word forms with shared morphemes, motivating fine-grained word representations based on morpheme segmentation.The examples include cook/cooks/cooked/cooking, possible/impossible, and policeman as police plus man.Source: A Linguistic Phenomena in Word Formation · p. 15

B The Example of Tensor Products

  • The paper uses the flattened-vector definition of tensor product so the resulting word embeddings remain vectors, rather than increasing tensor dimensions.A tensor product of m- and n-sized vectors yields an mn-sized vector under this definition.Source: B The Example of Tensor Products · p. 15

C Analysis of Parameter

  • The appendix says it reports embedding-method parameter counts and settings, with tables covering translation, WikiQA, and SNLI experiments.It directs readers to Sections C.1 and C.2 for parameter complexity and method settings.Source: C Analysis of Parameter · p. 16, §C.1 · p. 16

C.1 Parameter Complexity of Embedding Methods

  • The parameter analysis expresses original embeddings as |V|d parameters, MorphSum as (|V| + |M|)d, and MorphLSTM as |M|d + 8d^2.It also states that Matrix Factor. requires r(|V| + d) parameters; |V| and d denote word vocabulary size and embedding size.Source: §C.1 · p. 16

C.2 Settings of Embedding Methods

The appendix maps its detailed embedding-method settings to the main results tables. It provides settings for translation, WikiQA, and SNLI experiments.

C.2 Settings of Embedding Methods

  • Tables 9–10 detail translation settings, while Tables 11–12 detail WikiQA and SNLI settings, corresponding to main-text Tables 2 and 3.The tables distinguish original, matrix-factorization, tensor-train, Word2ket, Word2ketXs, and MorphTE settings.Source: §C.2 · p. 17, Table 9 · p. 17, Table 10 · p. 17

D Implementation and Training Details

Implementation and training hyperparameters are reported separately for translation and for question answering and natural language inference.

D Implementation and Training Details

E Morpheme Statistics on IWSLT’14 De-En

On IWSLT’14 De-En, most German and English corpus words contain no more than three morphemes, motivating a maximum of three morphemes for MorphTE.

E Morpheme Statistics on IWSLT’14 De-En

  • 91% of German words and 95% of English words contain one, two, or three morphemes in the corpus.For German, the respective counts are 1,728, 3,623, and 2,627 of 8,848; for English, they are 1,724, 2,977, and 1,632.Source: E Morpheme Statistics on IWSLT’14 De-En · p. 18, Table 15 · p. 18
  • The authors set MorphTE’s order to 3 because most words contain no more than three morphemes.The chosen segmentation limits each word to a maximum of three morphemes.Source: E Morpheme Statistics on IWSLT’14 De-En · p. 18, Table 15 · p. 18
  • Table 15 reports segmentation statistics, including counts by number of morphemes and the resulting morpheme-vocabulary size.It compares unrestricted segmentation (mor∞) with schemes capped at a maximum of i morphemes (mori).Source: Table 15 · p. 18

F Effect of Morpheme Segmenters on MorphTE

Morpheme segmentation quality affects MorphTE’s translation performance, while its WikiQA memory analysis reports a large reduction in the embedding layer’s share of total model cost.

F Effect of Morpheme Segmenters on MorphTE

  • randomSeg leaves words of at most three characters unsplit and randomly divides longer words into three parts.The authors describe this as making little use of morpheme knowledge; Morfessor is used in the paper’s prior experiments.Source: F Effect of Morpheme Segmenters on… · p. 19
  • MorphTE with randomSeg performs worse than both original embeddings and MorphTE with Morfessor on four translation datasets.The authors note that improving segmentation may further enhance MorphTE; Table 16 compares the original method, Morfessor, and randomSeg.Source: F Effect of Morpheme Segmenters on… · p. 19, Table 16 · p. 19

G.1 Memory Analysis

  • On WikiQA, MorphTE reduces the word-embedding share of total model parameters and memory from about 62% to 2%.Table 17 separates embedding-layer cost (#Emb) from the cost of the remaining model (#Stuc).Source: §G.1 · p. 19, Table 17 · p. 19

G.2 Time Analysis

Embedding compression adds computation for generating embeddings, but MorphTE’s reported embedding time is a small share of total model time on WikiQA.

G.2 Time Analysis

  • Embedding-compression methods add time for preprocessing to generate embeddings, unlike direct word-vector lookup, which has almost no computational overhead.The WikiQA timing tests used an Intel E5-2698 CPU and Tesla V100 GPU, averaging 100 repetitions on one input sample.Source: §G.2 · p. 20
  • 5.4% on GPU and 3.2% on CPU: MorphTE’s embedding time as a share of total model time.Table 18 reports time costs for the embedding layer and the rest of the model separately.Source: §G.2 · p. 20, Table 18 · p. 20
  • Word2ketxs and tensor-train methods show notably increased time cost on CPU devices.The authors suggest this may relate to their having more computational operations at the same parameter count.Source: §G.2 · p. 20

Related papers

Loading 2210.15379v1…