PaperlayerView exact revision on arXiv

Source-linked AI summary

FRAGE: Frequency-Agnostic Word Representation

Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, Tie-Yan Liu

arXiv:1809.06858v2cs.CLcs.LGstat.ML

TL;DR

Word embeddings can encode word frequency as well as semantics, leaving rare and popular words separated even when semantically related. FRAGE uses adversarial training to reduce this frequency information, and the paper reports improvements over baselines across four NLP task types.

  • Problem

    Learned embeddings can place rare and popular words in different regions and provide poor semantic neighbors for rare words, limiting their usefulness for downstream tasks.

    Source: §1 · p. 2

  • Method

    FRAGE jointly minimizes task-specific loss and trains embeddings to fool a discriminator that classifies words as rare or popular.

    Source: §1 · p. 2

  • Results

    FRAGE outperforms baselines across four NLP task types, including an improvement of about 5.4 points over the baseline on the rare-word dataset RW.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The approach is intended to make rare-word embeddings more semantically useful and help task-specific models perform better.

    Source: §1 · p. 2

    The authors leave theoretical study of word embedding learning and FRAGE, and applications beyond NLP, for future work.

    Source: §6 · p. 8

Abstract

from arXiv · show

Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks. Although it is widely accepted that words with similar semantics should be close to each other in the embedding space, we find that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high-frequency and low-frequency words lie in different subregions of the embedding space, and the embedding of a rare word and a popular word can be far from each other even if they are semantically similar. This makes learned word embeddings ineffective, especially for rare words, and consequently limits the performance of these neural network models. In this paper, we develop a neat, simple yet effective way to learn \emph{FRequency-AGnostic word Embedding} (FRAGE) using adversarial training. We conducted comprehensive studies on ten datasets across four natural language processing tasks, including word similarity, language modeling, machine translation and text classification. Results show that with FRAGE, we achieve higher performance than the baselines in all tasks.

1 Introduction

The introduction identifies frequency-biased embeddings as a problem for rare-word semantics and downstream NLP performance, then proposes adversarial training and reports gains across four task types.

1 Introduction

  • Rare and popular words occupy different embedding regions, and more than 85% of rare words have rare nearest neighbors.Rare-word neighbors can be semantically unrelated, while popular-word neighbors are often meaningful.Source: §1 · p. 2
  • These frequency-linked patterns weaken semantic understanding and can limit downstream task performance, including stability under popular/rare word substitutions.More than half of rare words are nouns or variants of popular words that should have similar meanings or topics.Source: §1 · p. 2
  • FRAGE adds a discriminator that classifies embeddings as rare or popular, while embeddings are trained to fool it and minimize task-specific loss.At equilibrium, the discriminator cannot distinguish the frequency classes, mixing rare and popular words in the embedding space.Source: §1 · p. 2
  • Across word similarity, language modeling, sentiment classification, and machine translation, FRAGE outperforms baselines and exceeds prior state-of-the-art results on PTB, WT2, and WMT14 English-German.The experiments span ten datasets across four task types.Source: §1 · p. 2

2 Background

The background reviews the difficulty of learning rare-word representations and positions FRAGE as adversarial training against frequency information, rather than domain adaptation.

2.1 Word Representation

  • Rare words are difficult to train because they occur infrequently, and rare subword units can still be underestimated in neural machine translation.During inference, models may favor popular words over rare alternatives.Source: §2.1 · p. 2

2.2 Adversarial Training

  • Adversarial training has been used in NLP with discriminators for cross-language semantics and for distinguishing human-written from generated sentences.The paper frames its method within this family but notes that it has no generator: its discriminator separates popular- and rare-word embeddings while the task model fools it and minimizes task loss.Source: §2.2 · p. 3
  • Unlike adversarial domain adaptation, FRAGE targets embedding effectiveness rather than a mismatch between training and testing domains.Source: §2.2 · p. 3

3 Empirical Study

The empirical study examines word embeddings from word2vec and machine translation models, finding frequency-linked neighborhoods and separated regions for rare and popular words.

3 Empirical Study

  • The study samples rare and popular words, compares cosine-based nearest neighbors with manually selected semantic neighbors, and uses the top 20% of vocabulary by frequency as popular.The authors report similar observations with 10% or 25% thresholds.Source: §3 · p. 3
  • For both studied tasks, more than 90% of model-predicted neighbors of rare words are rare words, and those neighbors are usually not semantically related.Chosen semantic neighbors are far away, while popular words’ predicted neighbors are usually reasonable.Source: §3 · p. 3
  • After SVD projection, rare and popular embeddings occupy different regions, and semantic neighbors are marked in the case-study panels.The visualization covers WMT14 Transformer and Google News word2vec embeddings; red and blue points denote rare and popular words.Source: §3 · pp. 3–4, Figure 1 · p. 4
  • A proposed explanation is that popular-word embeddings move farther during training: their average moving distance is twice that of rare-word embeddings.With small-variance initialization near the origin, rare embeddings remain near the origin while popular embeddings move farther away.Source: §3 · p. 4
  • The authors report that over half of rare words are nouns, around 10% contain hyphens, and over 30% are different parts of speech of popular words.They argue these word types should have mixed or similar semantics to popular words.Source: §3 · p. 4

4 Our Method

FRAGE trains task-specific embeddings to minimize task loss while an adversarial discriminator learns to identify rare words, and the embeddings learn to fool it.

4 Our Method

  • FRAGE uses adversarial training so embeddings minimize task-specific loss while making frequency information harder for a discriminator to detect.The method categorizes embeddings as popular or rare and aims to remove frequency information.Source: §4 · p. 5
  • The embedding matrix θemb contains the word vectors, while θmodel contains the other task-specific parameters.For language modeling, θmodel includes RNN or LSTM parameters; for translation, it includes encoder, attention, and decoder parameters.Source: §4 · p. 5
  • The task-specific loss LT is defined over dataset S; for language modeling, it is the negative log likelihood of the data.In that example, y is a sentence.Source: §4 · p. 5
  • The discriminator takes a word embedding and outputs a score for how likely its word is rare, with a loss defined over the vocabulary.Training alternates discriminator optimization for frequency classification with model and embedding optimization for task performance and fooling the discriminator.Source: §4 · p. 5

5 Experiment

The experiments evaluate FRAGE across four NLP task families against strong baselines, with reported improvements across all tasks and ten datasets. The results span word similarity, language modeling, machine translation, and text classification.

5 Experiment; 5.1 Settings

  • The experiments use state-of-the-art task architectures and training methods as baselines; FRAGE retains each baseline’s architecture and task loss, adding an adversarial loss.The evaluation covers word similarity, language modeling, machine translation, and text classification.Source: §5 · pp. 5–6, §5.1 · p. 6

5.2 Results

  • FRAGE consistently beats the baseline on all three word-similarity datasets, with an approximately 5.4-point gain on the rare-word dataset RW.The authors conclude that the method improves word representations, especially for rare words.Source: Table 1 · p. 7, §5.2 · p. 7
  • On PTB and WT2, FRAGE outperforms both language-modeling baselines at the tested checkpoints, with larger improvements on WT2, which contains more rare words.Reported test-perplexity improvements are 0.8/1.2/1.0 and 0.76/1.13/1.15 for the two baselines on PTB, and 2.3/2.4/2.7 and 1.15/1.72/1.54 on WT2.Source: Table 2 · p. 7, §5.2 · p. 7
  • For WMT14 English-German, FRAGE gains 1.06/0.71 BLEU in transformer_base/transformer_big; on IWSLT14 German-English, it gains 0.85 BLEU.The authors report these results as evidence that improved embeddings can accompany better performance on more complex tasks and larger datasets.Source: §5.2 · pp. 7–8, Table 3 · p. 8
  • For text classification, FRAGE outperforms the baseline by 1.26%/0.66%/0.44% across the three datasets.Source: Table 4 · p. 8, §5.2 · p. 8
  • Across four tasks and ten datasets, the experiments verify FRAGE’s effectiveness, with case studies showing more reasonable semantic similarities and better mixing of popular and rare words.The case studies and visualizations are in the supplementary material.Source: §5.2 · p. 8

6 Conclusion

The paper identifies frequency bias in learned embeddings, where high- and low-frequency words occupy different subregions, and proposes adversarial training for FRAGE. It reports improved performance across a wide range of tasks and names theoretical analysis and applications beyond NLP as future directions.

6 Conclusion

  • Learned word embeddings can be biased toward word frequency, placing high- and low-frequency words in different subregions of the embedding space.The paper proposes adversarial training to learn FRequency-AGnostic word Embedding (FRAGE).Source: §6 · p. 8
  • FRAGE’s performance is verified across a wide range of tasks.The paper describes the adversarial training method as improving model performance.Source: §6 · p. 8
  • The authors identify theoretical study of embeddings and FRAGE, and applications beyond NLP, as future directions.Source: §6 · p. 8

A.1 Dataset Description

The dataset descriptions cover word similarity, language modeling, machine translation, and text classification. They identify the datasets used and, where supplied, their salient statistics or preprocessing.

A.1 Dataset Description

  • Word similarity uses WordSim-353, rare-words, and RG65, comprising 353 common-word pairs, rarely used words, and 65 pairs, respectively.RG65 similarity values are means of judgments by 51 subjects.Source: §A.1 · p. 11
  • Language modeling uses Penn Treebank and WikiText-2, with out-of-vocabulary words replaced by <unk> during training and testing.Source: §A.1 · p. 11, Table 5 · p. 11
  • Machine translation uses WMT14 English-German and IWSLT14 German-English, with 4.5M and 160K training sentence pairs, respectively.WMT14 uses 37K shared BPE sub-word units; IWSLT14 uses a shared vocabulary of about 32K tokens.Source: §A.1 · pp. 11–12
  • Text classification uses AG’s News, IMDB, and 20NG; the 20 Newsgroups data is restricted to four categories: comp, politics, rec, and religion.Source: §A.1 · p. 12, Table 6 · p. 12

A.2 Hyper-parameter configurations

The appendix points to tables of task-specific hyperparameters and describes the baseline architectures used for language modeling, translation, and text classification.

A.2 Hyper-parameter configurations

  • Language-modeling, translation, and word-embedding hyperparameters are summarized in Tables 7–9; text classification uses convolutional kernels of sizes 2, 3, and 5 with batch normalization and shortcut connections.Translation uses Adam with β1 = 0.9, β2 = 0.98, and ε = 10−9; classification uses Adam with β1 = 0.9, β2 = 0.99, and ε = 10−8.Source: §A.2 · p. 12, Table 7 · p. 12, Table 8 · p. 13

A.3 Models Description

  • The described baselines include AWD-LSTM and AWD-LSTM-MoS for language modeling, Transformer for translation, and RCNN for text classification.AWD-LSTM uses Drop Connect and NT-ASGD; AWD-LSTM-MoS adds Mixture of Softmaxes, while Transformer uses self-attention in encoding and decoding.Source: §A.3 · p. 13
  • Word2vec is described as producing distributed word representations based on word co-occurrence.Source: §A.3 · p. 13

B Additional Comparisons

The additional machine-translation comparison evaluates FRAGE against frequency reweighting and weight decay, and reports that the simple alternatives do not work and can have negative effects.

B Additional Comparisons

  • The authors report that reweighting and L2 regularization do not work on the machine-translation tasks and can have negative effects.Source: B Additional Comparisons · p. 14
  • The comparison concerns BLEU scores on WMT14 English-German and IWSLT14 German-English test sets.Source: Table 10 · p. 14
  • Reweighting scales each word’s loss by reciprocal frequency, while Weight Decay applies a 0.2 rate to embeddings.Source: Table 10 · p. 14

C Case Study on Original Models and Qualitative Analysis of Our Method

The case studies report low rankings for semantic neighbors in original models, while FRAGE improves word similarities and mixes rare and popular words more closely.

C Case Study on Original Models and Qualitative Analysis of Our Method

Loading 1809.06858v2…