PaperlayerView exact revision on arXiv

Source-linked AI summary

Encoding word order in complex embeddings

Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, Jakob Grue Simonsen

arXiv:1912.12333v2cs.CLcs.IR

TL;DR

Position embeddings represent absolute positions but do not model ordered relationships between positions. This paper extends word vectors into continuous, complex-valued functions of position and reports gains over baseline embeddings across text classification, machine translation, and language modeling.

  • Problem

    Vanilla position embeddings represent individual positions independently, leaving relationships such as adjacency, precedence, and relative distance unmodeled.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper represents each word as a continuous, complex-valued function of position and incorporates these embeddings into neural network architectures.

    Source: §1 · pp. 1–2

  • Results

    Complex-order embeddings outperform the reported baselines in text classification, machine translation, and language modeling.

    Source: §3.1 · p. 6, §3.2 · p. 8, §3.3 · p. 8

  • Takeaways & Limitations

    The experiments support using complex word embeddings to model word order across the evaluated tasks and neural architectures.

    Source: §1 · p. 2, §5 · p. 9

    The derivation assumes a position-free offset transformation and requires the embedding function to be bounded.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions. We present a novel and principled solution for modeling both the global absolute positions of words and their order relationships. Our solution generalizes word embeddings, previously defined as independent vectors, to continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can correlate with each other in a continuous function. The general solution of these functions is extended to complex-valued domain due to richer representations. We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding (we make all code available). Experiments on text classification, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings. To our knowledge, this is the first work in NLP to link imaginary numbers in complex-valued representations to concrete meanings (i.e., word order).

1 INTRODUCTION

The introduction identifies a gap in position embeddings: they mark absolute word positions but do not model the ordered relationships between positions. It proposes continuous, complex-valued word representations to capture both absolute position and word order, and reports gains across three NLP tasks.

1 INTRODUCTION

  • Position embeddings encode individual word positions but do not capture relationships such as adjacency or precedence between positions.The introduction argues that both global absolute positions and sequential relationships matter in language.Source: §1 · p. 1
  • Word vectors are extended into continuous functions of position, allowing representations to shift smoothly and correlate across positions.The functions are presented as a way to model both absolute positions and sequential or adjacent relationships.Source: §1 · pp. 1–2
  • Figure 1’s vertical slices represent a word’s embedding at each absolute position, with colors distinguishing dimensions.The plotted waves show the real part; the imaginary part has a π/2 phase difference and shares the curves of the real-valued counterpart.Source: §1 · p. 2, Figure 1 · p. 2
  • The complex-valued embedding uses amplitudes for semantic aspects and periods for positional sensitivity.The introduction describes this representation as a general solution for the position-dependent functions.Source: §1 · p. 2
  • The authors report gains over classical and position-enriched embeddings across text classification, machine translation, and language modeling.They integrate complex embeddings into CNN, RNN, and Transformer architectures and describe linking imaginary components to word order as novel in NLP.Source: §1 · p. 2

2 MODELLING WORD ORDER IN EMBEDDING SPACE

The section motivates replacing independent position vectors with position-dependent word functions that can represent relationships between positions. It develops bounded, linearly witnessed complex functions and uses them to form word embeddings whose positional sensitivity can vary by word and dimension.

2 MODELLING WORD ORDER IN EMBEDDING SPACE

  • Vanilla position embeddings leave the relative distance between words nearly unknown to position-insensitive neural networks because each position is represented independently.The section calls this the position independence problem: the embeddings indicate different positions but do not model their ordered relationship.Source: §2 · p. 2

2.1 EXTENDING VECTORS TO FUNCTIONS

  • Continuous functions over position represent each word across positions, so changing position changes the function input rather than requiring an independently trained vector.The intended smooth transformation lets representations of adjacent positions correlate and model word order.Source: §2.1 · p. 3

2.2 PROPERTIES FOR THE FUNCTIONS TO CAPTURE WORD ORDER

  • Under a position-free offset transformation and boundedness, the paper derives a unique solution when the transformations are linearly witnessed.Boundedness is required so the position embedding can handle arbitrarily large positions; the resulting form is expressed as g(pos) = r e^i(ωpos+θ).Source: §2.2 · pp. 3–4

2.3 COMPLEX-VALUED WORD EMBEDDING

  • Each complex-valued coordinate has a word-specific amplitude, period, and phase, and element-wise multiplication combines word and positional embeddings.Frequency controls positional sensitivity: very small frequency makes the representation approximately constant, while higher frequency makes it change more with position; Transformer positional encoding is described as a degraded special case.Source: §2.3 · pp. 4–5

3 EXPERIMENTAL EVALUATION

The evaluation tests complex-order embeddings across text classification, machine translation, and language modeling. Reported results show gains over evaluated alternatives, with text-classification performance on par with the state of the art.

3 EXPERIMENTAL EVALUATION

  • Evaluations used accuracy on six text-classification datasets, BLEU on WMT 2016 English-German (29,000 sentence pairs), and BPC on text8.The text8 dataset contains 100M characters, with 90M for training and 5M each for development and testing.Source: §3.1 · p. 5, §3.2 · p. 7, §3.3 · p. 8

3.1 TEXT CLASSIFICATION

  • Complex-order embeddings outperform all text-classification variations and all compared methods except InferSent on CR, where InferSent is marginally better.The authors report performance on par with the state of the art overall.Source: Table 2 · p. 6, §3.1 · p. 6, Table 3 · p. 6
  • On the Transformer ablation, period-sharing schemes slightly reduce performance; word-sharing still outperforms Vanilla Transformer at a comparable parameter count.The authors attribute the slight drops to reduced representational power from fewer parameters.Source: Table 4 · p. 7, §3.1 · p. 7

3.3 LANGUAGE MODELING

  • Transformer XL Complex-order outperforms its vanilla counterpart and strong recurrent baselines under the six-layer setting.The authors say this demonstrates generalization to tasks with long-term dependency; they note the six-layer limit reflects computing-resource constraints.Source: Table 6 · p. 8, §3.3 · p. 8

3.2 MACHINE TRANSLATION

  • By 1.3 BLEU over Vanilla Transformer and 1.1 BLEU over complex-vanilla Transformer, Transformer Complex-order leads the machine-translation baselines.The comparison is in absolute BLEU score.Source: Table 5 · p. 8, §3.2 · p. 8

4 RELATED WORK

Prior work applied complex-valued networks in areas including biology, signal processing, speech, vision, and audio. In text and knowledge-graph work, this paper distinguishes itself by linking complex phase to word position.

4 RELATED WORK

  • Complex-valued networks had been used for biological motivation, signal processing, speech recognition, RNNs, vision, and audio.The cited work includes complex-valued RNNs and gated recurrent cells.Source: §4 · p. 8
  • Prior text and knowledge-graph work used complex numbers, but the physical meaning of complex-valued entity and word embeddings remained unknown.The paper identifies its contribution as linking complex phase to word position.Source: §4 · p. 8

5 CONCLUSIONS

The paper generalizes word vectors to continuous functions over position and derives a complex-valued solution. Experiments across three task areas find the embeddings more effective than vanilla position embeddings.

5 CONCLUSIONS

  • Word vectors become position-dependent functions whose smooth shifts implicitly capture relative distances between words.The paper derives a general solution in complex-valued form.Source: §5 · p. 9
  • The position embedding of Vaswani et al. (2017) can be viewed as a simplified version of the proposed approach.The paper extends CNN, RNN, and Transformer networks to complex-valued versions for these embeddings.Source: §5 · p. 9
  • Experiments on text classification, machine translation, and language modeling find the embeddings more effective than vanilla position embeddings.Source: §5 · p. 9

A LINKING TO THE POSITION EMBEDDING IN (VASWANI ET AL., 2017)

The appendix relates the Transformer position embedding to a particular complex-valued embedding with shared, dimension-specific periods. It also notes that the rationale and general extension of the earlier position embedding were not well explained.

A LINKING TO THE POSITION EMBEDDING IN (VASWANI ET AL., 2017)

  • Vaswani et al. (2017) proposed a position-embedding initialization with performance comparable to the previous one, even without fine-tuning.The paper says its selection rationale was not well explained and its general extension was unknown, making improvement difficult.Source: A LINKING TO THE POSITION EMBEDDING IN… · p. 12
  • The complex position function maps the even and odd position-embedding dimensions to its imaginary and real parts, respectively.Source: A LINKING TO THE POSITION EMBEDDING IN… · p. 12
  • The Transformer position embedding is a particular case of the complex-valued embedding when words share each dimension's period.The paper describes it as equivalent, though not identical, to that complex-valued position embedding.Source: A LINKING TO THE POSITION EMBEDDING IN… · p. 12

B INTEGRATING COMPLEX-VALUED EMBEDDING TO GENERAL NEURAL

The paper adapts dense, convolutional, recurrent, and Transformer layers to accept and process complex-valued embeddings. Its adaptations carry complex values through feature layers and use a real-valued norm for final predictions or loss where described.

Complex-valued FastText

  • Complex-valued FastText uses a dense layer over summed word embeddings, with separate real- and imaginary-part computations.The real- and imaginary-part weights and biases may be shared to save parameters and compare fairly with real-valued baselines.Source: B INTEGRATING COMPLEX-VALUED EMBEDDING… · p. 12

Complex-valued CNN

Complex-valued RNN

  • The complex-valued RNN uses complex input and hidden-state transitions, with an L2 norm producing a real-valued loss for backpropagation.The paper also applies complex-valued gates in its LSTM.Source: B INTEGRATING COMPLEX-VALUED EMBEDDING… · p. 13

Complex-valued Transformer

C IMPLEMENTATION OF THE PROPOSED EMBEDDING

The embedding stores function parameters and constructs position-dependent values, requiring changes only to the embedding layer. Its real and imaginary components can also be concatenated for use in real-valued networks.

C IMPLEMENTATION OF THE PROPOSED EMBEDDING

  • The embedding stores amplitude, frequency, and initial-phase parameters, then constructs position-dependent real and imaginary values using cosine and sine.The implementation uses the position sequence to adjust phase before computing the two components.Source: C IMPLEMENTATION OF THE PROPOSED… · pp. 13–14
  • Frequency and initial-phase vectors can be shared between words or dimensions to save parameters.Source: C IMPLEMENTATION OF THE PROPOSED… · p. 14
  • Concatenating real and imaginary values yields a double-size real-valued vector usable in real-valued networks without complex-valued components.The embedding can be added by modifying the embedding layer.Source: C IMPLEMENTATION OF THE PROPOSED… · pp. 13–14

D VISUALIZATION OF FREQUENCIES/PERIODS

The section uses mean absolute frequency as a measure of positional sensitivity and examines frequency extremes in SST. Strong sentiment words are more common among the highest-frequency words than among the lowest-frequency words.

D VISUALIZATION OF FREQUENCIES/PERIODS

Related papers

Loading 1912.12333v2…