PaperlayerView exact revision on arXiv

Source-linked AI summary

All-but-the-Top: Simple and Effective Postprocessing for Word Representations

Jiaqi Mu, Suma Bhat, Pramod Viswanath

arXiv:1702.01417v2cs.CLstat.ML

TL;DR

The paper asks whether widely used word representations can be strengthened through simple postprocessing. It removes their common mean and a few top dominating directions, then evaluates the result across lexical and sentence-level tasks. The processed representations consistently perform better, while broader validation across additional downstream applications remains future work.

  • Problem

    Word2vec and GloVe capture linguistic regularities effectively, motivating investigation of whether their existing representations can be improved without replacing the underlying learning methods.

  • Method

    The method subtracts the common mean vector and projects word vectors away from the top D dominating directions, with D approximately d/100 as a rule of thumb.

  • Results

    Processed representations consistently outperform originals on word similarity, concept categorization, word analogy, semantic textual similarity, and text classification across the reported datasets and settings.

  • Takeaways & Limitations

    The simple postprocessing operation can be applied to published word embeddings for downstream tasks or used to initialize task-specific embeddings.

  • Takeaways & Limitations

    Systematic validation on additional applications such as named entity recognition, syntactic parsing, and machine translation, and on convergence and generalization, is left for future research.

Abstract

from arXiv · show

Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a {\em very simple}, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a few top dominating directions from the word vectors -- that renders off-the-shelf representations {\em even stronger}. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and { text classification}) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

1 INTRODUCTION

The paper proposes removing the common mean and a few dominant directions from pretrained word vectors to make them more isotropic and stronger across lexical and sentence-level evaluations. This counter-intuitive processing consistently improves representation quality across multiple tasks and settings.

  • Motivation: Off-the-shelf word representations are successful because their geometry captures linguistic regularities, but the paper finds that simple postprocessing can make them stronger.The studied representations include word2vec and GloVe, which support applications such as text classification, machine translation, and knowledge base completion.
  • Observation: Every tested representation has a non-zero mean and concentrates much of its energy in a very low-dimensional subspace, such as 8 of 300 dimensions.These shared components influence word vectors in similar ways.
  • Method: The algorithm removes the mean vector and projects word representations away from the top D dominating directions, with D approximately d/100 as a broadly effective rule of thumb.The suitable D also depends on representation properties and downstream applications.
  • Method: The approach is counter-intuitive because it removes dominant rather than weakest directions, aiming to produce more purified and isotropic representations.The authors posit that the operation improves self-normalization properties.
  • Lexical evaluations: Processed representations improve word similarity by 1.7% on average across seven datasets, concept categorization by 2.8%, 4.5%, and 4.3%, and word analogy by 0.5%, 0.2%, and 0.4%.The reported analogy gains are smaller because subtraction already cancels some dominant components during analogy solving.
  • Sentence-level evaluations: Postprocessing improves semantic textual similarity consistently and significantly across 21 datasets, with an average improvement of 4%.Sentences are represented by averaged word vectors and compared using cosine similarity.
  • Sentence-level evaluations: Across two pretrained vector sets, five datasets, and four CNN or RNN architectures, processing improves 34 of 40 text-classification instances by 2.85% on average.The remaining six instances have comparable processed and unprocessed performance.
  • Related work: The work differs from dataset-specific direction removal by estimating dominating directions over the entire language vocabulary.The paper contrasts its vocabulary-level directions with directions extracted from sentence representations within individual semantic textual similarity datasets.

2 POSTPROCESSING

The paper examines word representations across multiple embedding families and finds shared mean and low-dimensional structure that motivate removing common and dominant components. It then connects this postprocessing to frequency-related PCA directions and increased isotropy.

  • Across the tested representations, word vectors share a nonzero common mean, whose norm is approximately 1/6 to 1/2 of the average word-vector norm.
  • The proposed algorithm removes the mean vector and projects representations away from the top D principal directions, reducing shared energy and effective dimension.
  • The top PCA coefficients encode word frequency to a significant degree, with the relationship between the first two coefficients and unigram probability shown geometrically.
  • D is the only tuned hyperparameter; choosing D approximately as d/100 is reported as a useful rule of thumb, and improvements persist for higher-dimensional embeddings.
  • Postprocessing increases isotropy according to the reported measure and is evaluated through self-normalization using 1,000 randomly sampled unit vectors.

3 EXPERIMENTS

The experiments compare original and postprocessed word representations across lexical and sentence-level tasks. Postprocessing generally improves performance, with the smallest gains on analogy evaluation.

  • Word Similarity: Postprocessing improves word similarity consistently across seven datasets, averaging 2.3%.Similarity is measured by cosine similarity and evaluated against human judgments using Spearman correlation.
  • Concept Categorization: Postprocessing improves concept categorization consistently across three datasets, averaging 2.5%.Words are clustered with k-Means and evaluated by purity.
  • Word Analogy: Postprocessing improves word analogy performance, but the gain is less pronounced because subtraction can cancel common and dominant components.The analogy solver selects w4 by maximizing cosine similarity with v(w2) − v(w1) + v(w3).
  • Semantic Textual Similarity: Postprocessing improves semantic textual similarity consistently and significantly, averaging 4% across 21 datasets.Sentences are represented by averaged word vectors, and sentence similarity is computed with cosine similarity.

4 POSTPROCESSING AND SUPERVISED CLASSIFICATION

The paper evaluates postprocessed representations in supervised text classification with multiple neural architectures and datasets. It also reports that mean subtraction can be learned internally, while broader downstream validation remains future work.

  • Experimental Setting: Text classification is evaluated with CNN, vanilla RNN, GRU, and LSTM architectures on five benchmark datasets.The comparison uses word representations with and without postprocessing.
  • Future Work: Validation on additional downstream applications and classification methods, including named entity recognition and machine translation, is left for future research.The authors also identify convergence rate and classifier generalization as targets for further study.
  • Learned Postprocessing: The paper reports that subtracting the mean vector is effectively learned by neural networks within their nodes.This is a preliminary experiment addressing whether end-to-end models can reproduce part of the postprocessing.

5 CONCLUSION

The conclusion presents top-principal-component removal as a simple way to strengthen word representations. It reports success beyond the main English WORD2VEC and GLOVE experiments, including other representations and multilingual settings.

  • Conclusion: The proposed postprocessing strengthens word representations by eliminating their top principal components.The operation is intended for downstream word-embedding tasks or initialization of task-specific embeddings.

A ANGULAR ASYMMETRY OF REPRESENTATIONS

PMI-based word representations violate the zero-mean and isotropy assumptions used in earlier generative analyses. The paper relaxes those constraints to match observed geometry and shows the principal theoretical result still holds.

  • Observed geometry: PMI-based representations such as WORD2VEC and GLOVE are not zero-mean or isotropic, contradicting assumptions in the RAND-WALK model.The observed vectors share a common mean and concentrate energy in a few dominant directions.
  • Relaxed conditions: The revised model allows vectors to be isotropic around a point lying on a low-dimensional subspace.This directly accommodates a bounded mean vector and a few dominant directions.
  • Generative model: The paper interprets the revised geometry through a latent-discourse-variable log-linear generative model for words.The model retains the RAND-WALK framework while changing its constraints on word vectors.
  • Empirical geometry: After removing dominant components, the remaining WORD2VEC and GLOVE spectra are flat like a random Gaussian matrix, supporting greater isotropy.The comparison uses the singular values after excluding the top D components.
  • Theoretical result: Theorem A.1 shows that the RAND-WALK generative model and its properties extend to the relaxed geometric setting.The proof generalizes the earlier result while permitting broader word-vector geometry.

B NEURAL NETWORKS LEARN TO POSTPROCESS

Neural networks can in principle learn linear postprocessing internally, so the paper adds explicit linear layers to test whether they reproduce the proposed transformations. Mean removal is recovered closely, whereas learning the projection step remains unresolved.

  • Motivation: Neural networks can implement the proposed postprocessing through their linear operations, but internal learning is difficult to verify directly.The paper therefore uses explicit linear layers as a comparative test.
  • Mean removal: An appended bias layer tests whether networks learn mean subtraction while preserving the surrounding RNN architecture.The layer replaces each input v(w) with v(w) − b and is trained jointly with the network.
  • Mean removal: 0.66 average cosine similarity in 300 dimensions links the learned processed and original biases across five datasets and three architectures.The authors interpret this as near-exact implicit removal of the common mean.
  • Dominant-direction removal: The second step uses a projection P = I − Σ_i u_i u_i^⊤ to remove the top D principal directions.A matrix-based comparison tests whether networks learn this projection after preprocessing.
  • Dominant-direction removal: The projection-learning experiment cannot determine whether networks learn the second step, leaving a more careful test for future work.The ambiguity may reflect excessive parameters or an inability of network weights to learn the operation.
  • Generalization: Processed representations generalize across TSCCA, multiple languages, and four lexical or sentence-level tasks, with average gains of 2.23%, 2.39%, 0.11%, and 0.61%.German and Spanish similarity evaluations also consistently favor processed representations.

C.4 ROLE OF DIMENSIONS

Across dimensions from 300 to 1000, word representations retain a dominant low-dimensional spectral structure. Setting D around d/100 yields improvements on most evaluated tasks, though analogy gains remain marginal.

  • Representation geometry: The norm ratio of the mean vector to the average word-vector norm ranges from 1/3 to 1/4 across dimensions 300 through 1000.This confirms a substantial common component beyond the original 300-dimensional setting.
  • Representation geometry: The variance ratios decay near exponentially for small component indices and remain roughly constant later.The same spectral pattern appears in the higher-dimensional representations.
  • Choice of D: Choosing D around d/100 is validated for representations ranging from 300 to 1000 dimensions.The experiments use skip-gram representations trained on a Wikipedia snapshot.
  • Performance: 2.27%, 3.37%, 0.01, and 1.92% are the reported improvements on the four evaluated tasks under D = d/100.The analogy-task improvement is explicitly described as marginal.

D EXPERIMENTS ON WORD ANALOGY TASK

The paper reports detailed performance results for the word analogy task in Table 19.

  • Word analogy task: Table 19 provides the detailed performance results for the word analogy task.The supplied passage identifies the table but does not report its numerical values.

E EXPERIMENTS ON SEMANTIC TEXTUAL SIMILARITY TASK

This section presents before–after evaluation tables for word similarity, categorization, word analogy, and semantic textual similarity tasks. Detailed semantic textual similarity results are identified separately in Table 20.

  • Detailed semantic textual similarity performance is provided in Table 20.
  • Table 15 reports before–after results on the word similarity task across seven datasets.
  • Table 16 reports before–after results on the categorization task.
  • Table 17 reports before–after results on the word analogy task.

F STATISTICS OF TEXT CLASSIFICATION DATASETS

The text-classification evaluation uses four neural architectures across five benchmarks. The datasets cover review sentiment, subjectivity, question categories, multi-sentence reviews, and full-sentence sentiment data, with predefined or cross-validation splits.

  • Four architectures—CNN, vanilla-RNN, GRU-RNN, and LSTM-RNN—are used to evaluate representations with and without postprocessing.
  • The five benchmarks are MR, SUBJ, TREC, IMDb, and SST.
  • MR contains one-sentence reviews, while IMDb reviews consist of several sentences.
  • SUBJ classifies sentences as subjective or objective, and TREC partitions questions into six categories.
  • TREC, SST, and IMDb provide train/test splits, whereas MR and SUBJ use 10-fold cross validation.

G PROOF OF THEOREM A.1

The proof establishes concentration for the partition function Z(c), adapting an earlier proof to deterministic word vectors and additional asymmetric components. This supports the stated correlations involving p(w), p(w,w′), and word-vector norms.

  • The proof reuses much of Arora et al.’s argument but separately addresses concentration of Z(c).
  • Lemma G.1 states that for uniformly random unit-sphere c, Z(c)/|V| converges with high probability to a constant Z.
  • Unlike the earlier proof, this argument treats v(w) as deterministic and handles the asymmetric components µ and u1,...,uD.
  • The proof decomposes the concentration argument into multiple steps involving rotational symmetry, expectations, Gaussian variables, and concentration bounds.
  • After establishing Lemma G.1, the proof follows the earlier argument to show that p(w) and p(w,w′) correlate with ∥v(w)∥.
Loading 1702.01417v2…