PaperlayerView exact revision on arXiv

Source-linked AI summary

SPINE: SParse Interpretable Neural Embeddings

Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Eduard Hovy

arXiv:1711.08792v1cs.CL

TL;DR

Dense word embeddings are difficult to interpret because their dimensions do not clearly denote semantic concepts. SPINE transforms existing embeddings with a denoising k-sparse autoencoder; the paper reports greater interpretability and competitive or better performance across downstream benchmarks.

  • Problem

    Dense word embeddings are difficult to interpret because their dimensions may not correspond to semantically coherent concepts.

    Source: Introduction · p. 1

  • Method

    SPINE transforms existing word embeddings into sparse, non-negative representations using a denoising k-sparse autoencoder with a novel learning objective and activation function.

    Source: Introduction · p. 1, Methodology · p. 3

  • Results

    SPINE embeddings were more interpretable than original vectors and SPOWV, and performed competitively across benchmark tasks while significantly outperforming baselines on most.

    Source: Results and Discussion · p. 6, Performance on downstream tasks · p. 6

  • Takeaways & Limitations

    The retrofitting approach does not require access to a giant corpus and is agnostic to the method used to create the input embeddings.

    Source: Retroffiting vs joint learning · p. 7

    At 2,000 or more hidden dimensions, topic coherence and interpretability improved at severe downstream-performance cost; at 500 dimensions, both deteriorated.

    Source: Interpretability and downstream… · p. 7

Abstract

from arXiv · show

Prediction without justification has limited utility. Much of the success of neural models can be attributed to their ability to learn rich, dense and expressive representations. While these representations capture the underlying complexity and latent trends in the data, they are far from being interpretable. We propose a novel variant of denoising k-sparse autoencoders that generates highly efficient and interpretable distributed word representations (word embeddings), beginning with existing word representations from state-of-the-art methods like GloVe and word2vec. Through large scale human evaluation, we report that our resulting word embedddings are much more interpretable than the original GloVe and word2vec embeddings. Moreover, our embeddings outperform existing popular word embeddings on a diverse suite of benchmark downstream tasks.

Introduction

Dense word embeddings can be difficult to interpret because their dimensions lack clear semantic meaning. SPINE transforms existing embeddings with a denoising k-sparse autoencoder, and the paper reports improved interpretability and strong downstream-task performance.

Introduction

  • Dense GloVe and word2vec dimensions are difficult to interpret because their highest-participating words do not form semantically coherent groups.The paper defines interpretability here as each dimension denoting a semantic concept.Source: Introduction · p. 1
  • Feature-norming studies found that people describe words with a few sparse characteristic properties, motivating sparse, non-negative representations.The paper argues this design can make representations more interpretable.Source: Introduction · p. 1
  • The paper frames its central question as how to transform word representations into a space that is more interpretable.Source: Introduction · p. 1
  • SPINE applies a denoising k-sparse autoencoder with a novel learning objective and activation function to transform input word embeddings into interpretable, efficient representations.The paper evaluates the resulting embeddings with a crowdsourced intrusion-detection test and downstream tasks.Source: Introduction · p. 1
  • SPINE’s qualitative evaluation reports more interpretable embeddings than GloVe, word2vec, and SPOWV, with dimensions that can capture different word senses.The table examines top participating dimensions for randomly sampled words.Source: Table 1 · p. 2

Related Work

Prior work pursues interpretable word embeddings through sparse factorization, linguistically inspired dimensions, and vector rotations. SPINE instead uses a neural model that learns dimensions and permits graded word participation.

Interpretability in word embeddings

  • NNSE uses non-negative matrix factorization for sparse word representations, while Faruqui et al. (2015, a) use linguistically inspired binary-valued dimensions.The binary dimensions do not represent the extent of a word’s participation.Source: Interpretability in word embeddings · p. 2
  • SPINE differs from linear factorization and rotations by using neural models and learning dimensions during training, while allowing words to participate at varying levels.Source: Interpretability in word embeddings · p. 2

Sparse Overcomplete Word Vectors

  • SPOWV uses sparse coding in a dictionary-learning setting to produce sparse, non-negative word embeddings.The approach represents input vectors as sparse linear combinations of basis vectors.Source: Interpretability in word embeddings · pp. 2–3

k-sparse autoencoders

  • A k-sparse autoencoder has at most k active hidden units per input with high probability; matching expected activation alone need not ensure that sparsity.SPINE’s objective and activation function are intended to mitigate this issue.Source: k-sparse autoencoders · p. 3

Methodology

SPINE transforms input word embeddings into sparse, non-negative codes with a denoising k-sparse autoencoder trained to reconstruct its inputs. Its objective combines reconstruction, average sparsity, and partial sparsity losses; cap-ReLU bounds activations to support strict sparsity and the partial sparsity objective.

Model architecture

  • A denoising k-sparse autoencoder encodes each input word vector as a sparse, non-negative embedding and reconstructs the input from that code.Figure 1 depicts the code for “internet” as a few active hidden units and the remaining units inactive.Source: Methodology · p. 3, Figure 1 · p. 4

Training objective

  • The training objective combines reconstruction loss with average sparsity loss and partial sparsity loss, weighted by λ1 and λ2.Reconstruction loss measures the average error in reconstructing inputs from their learned representations.Source: Methodology · p. 3, Reconstruction Loss (RL) · p. 3

Denoising

  • Adding isotropic zero-mean Gaussian noise during autoencoder training is intended to produce more robust intermediate representations.The noise has variance σ2I.Source: Reconstruction Loss (RL) · p. 3

Sparsity losses

Activation function

  • Cap-ReLU produces non-negative activations in [0, 1], allowing exact zeros and bounded values suitable for partial sparsity loss.Sigmoid is ruled out because its asymptotic behavior does not permit strict sparsity.Source: Choice of activation function · p. 4

Experimental Setup

The authors trained SPINE on pretrained GloVe and word2vec vectors, selected hyperparameters using topic coherence subject to a reconstruction-loss cap, and compared against the original vectors and SPOWV. They report sparsity gains for the partial sparsity loss and compare it with an ℓ1 alternative.

Training data

  • The models use 300-dimensional pretrained GloVe and word2vec vectors, with 15,000 of 17,000 selected words for training and 2,000 for tuning.The GloVe vectors were trained on 6 billion tokens; word2vec used around 100 billion words.Source: Experimental Setup · p. 4

Hyperparameter tuning

  • Hyperparameters were selected by maximizing topic coherence under a reconstruction-loss cap, since high coherence otherwise came with poor downstream performance.The topic-coherence metric is reported to correlate positively with human evaluation.Source: Hyperparameter tuning · pp. 4–5
  • Denoising attained 6% more sparsity at similar reconstruction loss.The selected hidden layer has 1,000 units, mapping 15,000-by-300 inputs to 15,000-by-1,000 embeddings.Source: Hyperparameter tuning · p. 5

Partial sparsity loss

  • λ2=1 yields 22% more sparsity on GloVe and 67% more sparsity on word2vec than λ2=0 at comparable reconstruction loss.The comparison evaluates the utility of adding Partial Sparsity Loss.Source: Hyperparameter tuning · p. 5

ℓ1 comparison

  • At a reconstruction-loss threshold, ASL and PSL achieve higher automated topic coherence than an ℓ1 regularizer.The authors attribute the difference to ASL and PSL forcing activations toward 0 and 1, while ℓ1 drives values toward 0 and does not distinguish activation distributions.Source: Inducing sparsity through ℓ1 regularizer · p. 5

Baselines

  • SPINE is compared with its GloVe and word2vec starting embeddings and with 1,000-dimensional SPOWV embeddings generated using the authors’ recommended hyperparameters.The matched SPOWV output dimensionality supports a comparison with another method designed for sparse, effective, interpretable embeddings.Source: Baseline embeddings · p. 5

Interpretability

The paper evaluates embedding interpretability with word-intrusion tests and qualitative inspection of words active in sampled dimensions. In the intrusion test, people identify an intruder mixed with four highly active words.

Interpretability

  • Qualitative assessment checks whether top words from sampled active dimensions share observable similarities.The paper describes this as a second way to examine interpretability alongside intrusion detection.Source: Interpretability · p. 5
  • Each intrusion question mixes the four most active words in a dimension with a random intruder selected to be less active there but highly active in another dimension.The test uses the top four words from a dimension and selects an intruder from its bottom half that falls in the top 10 percentile of at least one other dimension.Source: Word Intrusion Detection Test · p. 5
  • A large-scale MTurk evaluation tested 300 of 1,000 dimensions per representation and initialization, with three workers per question and 5,400 questions overall.The majority vote determined the answer, with a random choice among annotators’ answers when all three differed.Source: Word Intrusion Detection Test · p. 5

Benchmark Downstream Tasks

The downstream evaluation tests embeddings across classification and word-similarity tasks, using averaged word vectors as text-classification features and tuned classifiers. The supplied table descriptions also cover intrusion-test precision and annotator agreement.

Benchmark Downstream Tasks

  • The benchmark suite includes sentiment analysis, news classification, noun-phrase bracketing, question classification, and word similarity; text-classification features average the words’ vectors.SVMs, logistic regression, and random forests are tuned on development data, with accuracy reported on test data.Source: Benchmark Downstream Tasks · pp. 5–6
  • The supplied tables describe intrusion-test precision, where higher scores indicate more interpretable dimensions, and inter-annotator agreement, including unanimous agreement.For SPINE, nearly half of the questions have agreement among all three annotators.Source: Table 4 · p. 6, Table 5 · p. 6
  • Sentiment analysis uses non-neutral Stanford Sentiment Treebank labels, while noun-phrase bracketing classifies three-word phrases using concatenated word vectors.The sentiment splits contain 8,337/1,081/2,166 sentences; noun-phrase bracketing uses 2,227 phrases in 10 folds.Source: Benchmark Downstream Tasks · p. 6
  • TREC classifies questions into six types, news classification covers three binary 20 Newsgroups tasks, and WS-353 evaluates cosine similarity with Spearman’s ρ.WS-353 excludes 59 of its 353 word pairs because both words are not present in the vocabulary.Source: Benchmark Downstream Tasks · p. 6

Results and Discussion

SPINE performs competitively across benchmark tasks and significantly better on a majority, while producing more interpretable representations than the original vectors and SPOWV. The discussion describes its retrofitting approach, performance trade-offs, and interpretation of embedding dimensions.

Results and Discussion

  • Word-intrusion precision is notably higher for SPINE than for original vectors and SPOWV, indicating more accurate intruder selection and greater agreement.The paper identifies this as its key result for producing interpretable representations.Source: Results and Discussion · p. 6
  • SPINE performs competitively on all benchmark tasks and significantly better on a majority, compared with original GloVe and word2vec vectors and SPOWV.Table 6 compares these embeddings across a suite of downstream tasks; all reported accuracies are percentages.Source: Performance on downstream tasks · p. 6, Table 6 · p. 7
  • Sampled SPINE dimensions contain coherent, relevant words and sometimes capture different interpretations of an ambiguous word such as “remote.”The authors characterize these examples as anecdotal evidence of higher interpretability.Source: Qualitative assessment · p. 6
  • SPINE retrofits existing embeddings, needs no giant corpus, and is agnostic to the method that generated the input vectors.The paper presents this as a practical advantage over joint learning approaches.Source: Retroffiting vs joint learning · p. 7
  • The authors hypothesize that sparse embeddings can improve some downstream results by directly capturing salient concept characteristics, helping task models converge to optimum weights.They give a location-name dimension as an example of a potentially useful feature for chunking models.Source: Interpretability and downstream… · p. 7
  • With 2,000 or more hidden dimensions, topic coherence and interpretability improve at a severe cost to downstream performance; 500 dimensions impair both.The paper notes that one-hot vectors would be highly interpretable but perform significantly worse on downstream tasks.Source: Interpretability and downstream… · p. 7
  • The authors attribute SPINE’s expressiveness to nonlinear neural transformations and argue that non-negativity and sparsity yield semantically coherent dimensions.They distinguish this dimension-level interpretability from GloVe and word2vec’s ability to preserve semantic neighborhoods and score well on word similarity.Source: General discussion · p. 7
  • Semantic dimensions matter for prediction explanations because softmax weights on the final representation can sway class probabilities.The authors argue that understanding dimensions with large positive or negative weights helps explain predictions.Source: General discussion · p. 7

Conclusion

SPINE uses denoising k-sparse autoencoders to generate interpretable word embeddings. The authors report greater interpretability than sparse-coding approaches, better performance than popular baselines across diverse downstream tasks, and potential for integration into neural models.

Conclusion

  • Crowdsourced experiments find SPINE embeddings more interpretable than embeddings from state-of-the-art sparse-coding approaches.Source: Conclusion · p. 7
  • SPINE embeddings outperform popular baseline representations on a diverse set of downstream tasks.Source: Conclusion · p. 7
  • The method generates interpretable word embeddings using denoising k-sparse autoencoders.Source: Conclusion · p. 7
  • Sub-differentiable loss functions and backpropagation may allow seamless integration into neural models and end-to-end training.Source: Conclusion · p. 7
  • Future work will investigate varying sparsity across multiple hidden layers in more sophisticated networks and study the resulting sparse activations.Source: Conclusion · p. 7
Loading 1711.08792v1…