PaperlayerView exact revision on arXiv

Source-linked AI summary

Charagram: Embedding Words and Sentences via Character n-grams

John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu

arXiv:1607.02789v1cs.CL

TL;DR

Textual sequence representations often rely on word units or more complex character architectures, motivating simpler subword-aware models. CHARAGRAM uses character n-gram counts followed by one nonlinear transformation, and it outperforms recurrent and convolutional character models across several similarity evaluations while converging quickly on tagging.

  • Problem

    Existing sequence models often use words as the smallest compositional units, while prior character-based approaches include more complex recurrent and convolutional architectures.

  • Method

    CHARAGRAM represents words and sentences with character n-gram count vectors followed by a single nonlinear transformation into low-dimensional embeddings.

  • Results

    CHARAGRAM outperforms character RNNs and CNNs on word and sentence similarity, while the three architectures achieve similar POS-tagging performance and CHARAGRAM converges fastest.

  • Takeaways & Limitations

    The results suggest practitioners should begin with simple architectures rather than immediately choosing RNNs or CNNs for character-based text representation.

  • Takeaways & Limitations

    The authors could not explore many training epochs because PPDB training sets were large and the compared architectures were computationally demanding.

Abstract

from arXiv · show

We present Charagram embeddings, a simple approach for learning character-based compositional models to embed textual sequences. A word or sentence is represented using a character n-gram count vector, followed by a single nonlinear transformation to yield a low-dimensional embedding. We use three tasks for evaluation: word similarity, sentence similarity, and part-of-speech tagging. We demonstrate that Charagram embeddings outperform more complex architectures based on character-level recurrent and convolutional neural networks, achieving new state-of-the-art performance on several similarity tasks.

1 Introduction

CHARAGRAM applies character-based compositional modeling to words and sentences using a simple n-gram architecture. Across similarity and tagging evaluations, it outperforms recurrent and convolutional character models on several tasks and converges quickly.

  • Motivation: Character-based models share subword information across words, potentially improving representations of rare words and morphological variants.Prior character RNNs and CNNs demonstrated improvements across several NLP tasks.
  • Approach: CHARAGRAM represents character sequences with character n-gram counts followed by a single nonlinear transformation into a low-dimensional embedding.The model can be interpreted as learning character n-gram embeddings whose summed vectors produce sequence representations.
  • Evaluation: CHARAGRAM is evaluated on word similarity, sentence similarity, and part-of-speech tagging.These evaluations compare its character-based representation across semantic and syntactic tasks.
  • Results: CHARAGRAM outperforms RNNs and CNNs on word similarity, achieving state-of-the-art performance on SimLex-999.The model also outperforms those architectures and PARAGRAM-PHRASE across a large suite of sentence-level semantic textual similarity tasks.
  • Results: The three architectures reach similar POS-tagging performance, though CHARAGRAM converges fastest to high accuracy.The introduction reports this convergence advantage while noting similar final performance across models.
  • Analysis: Nearest neighbors indicate that CHARAGRAM addresses spelling variation, morphology, and word choice while revealing etymological links among n-grams.The authors release CHARAGRAM resources as a subword-aware text-representation baseline.

2 Related Work

Related work ranges from subword features and additive word constructions to character-level recurrent and convolutional architectures. CHARAGRAM builds most directly on character n-gram count representations and applies them effectively to words and sentences.

  • Subword information: Early subword approaches appended morphological features or learned word and subword embeddings jointly, often composing them through addition.These methods incorporated subword information into word representations for task-specific models.
  • Character RNNs: Character-level RNNs have represented arbitrary word types and supported language modeling, POS tagging, dependency parsing, translation, tokenization, segmentation, and normalization.The related work describes both recurrent word encoders and character-level language models used as task features.
  • Character CNNs: Character CNNs with n-gram filters have been used for language modeling, POS tagging, named entity recognition, text classification, and machine translation.Combinations of character CNNs and RNNs have also been explored.
  • Closest precedent: CHARAGRAM is closely related to DSSM, which represents words with character n-gram count vectors for information retrieval.Sperr et al. applied a similar technique to neural language models for machine translation.
  • Closest precedent: The authors show that character n-gram count representations outperform character LSTMs and CNNs when applied to both words and sentences.This positions CHARAGRAM as a simple alternative to richer character-level architectures.

3 Models

CHARAGRAM embeds a character sequence by aggregating vectors for its matching character n-grams and applying an elementwise nonlinearity. The paper contrasts this exact-match mechanism with charLSTM and charCNN baselines and emphasizes its ability to learn precise n-gram vectors.

  • Sequence representation: A textual sequence includes characters, spaces between words, and special start-of-sequence and end-of-sequence markers.The sequence is denoted x = ⟨x_1, x_2, ..., x_m⟩.
  • CHARAGRAM: CHARAGRAM embeds a sequence by adding the vectors of its character n-grams and then applying an elementwise nonlinearity.This is the model’s core compositional operation.
  • CHARAGRAM: The model restricts contributing n-grams to a predefined vocabulary V and uses an indicator to select matching n-grams.Its parameter count is d + d|V|, and the formulation is equivalent to transforming a character n-gram count vector.
  • Baselines: charLSTM uses a forward LSTM over the character sequence and takes its final hidden vector as the sequence representation.This provides the recurrent baseline for comparison.
  • Baselines: charCNN uses a convolutional layer with varying filter lengths, optionally followed by a fully connected layer and max pooling.Each filter operates over the character n-grams in the sequence.
  • Design comparison: Unlike CNN filters, CHARAGRAM requires an exact n-gram match before a vector affects the representation, enabling precise vectors for particular n-grams.CNN filters instead affect sequences according to their learned responses and nonlinearity.

4 Experiments

The experiments evaluate CHARAGRAM on word similarity, sentence similarity, and POS tagging, comparing it with character-based recurrent and convolutional models and word-averaging baselines. CHARAGRAM performs strongly across semantic tasks, including rare-word and sentence similarity, while POS-tagging performance is similar across models and CHARAGRAM converges fastest.

  • Evaluation setup: The experiments target paraphrase similarity for words and sentences, plus English part-of-speech tagging.Word and sentence evaluations use cosine similarity, while POS tagging is a classification task.
  • Word similarity: The word-similarity evaluation covers WS353, SL999, and the Stanford Rare Word Similarity Dataset, training on PPDB word pairs.PPDB provides paraphrase pairs, with smaller subsets containing higher-confidence paraphrases.
  • Sentence similarity: The sentence-similarity evaluation spans 22 datasets, including SemEval STS tasks from 2012–2015, Twitter, and SICK.STS scores sentence similarity on a 0–5 scale across diverse domains.
  • Architecture comparison: CHARAGRAM outperforms charLSTM and charCNN on word similarity and outperforms both architectures and PARAGRAM-PHRASE on sentence similarity.For sentence similarity, CHARAGRAM-PHRASE outperforms PARAGRAM-PHRASE on 15 of 22 tasks.
  • Word-similarity results: 70.6 on SL999 is achieved by the largest CHARAGRAM model, while its rare-word score is 47.1 Spearman’s ρ versus 41.8 for Soricut and Och.The 70.6 result uses WS353 for model selection and 173,881 n-gram embeddings; the rare-word result uses SL999 for model selection.
  • Sentence-similarity results: 74.7 Pearson’s r is achieved by CHARAGRAM-PHRASE on the 2014 STS datasets, versus 61.3 for FastSent.CHARAGRAM-PHRASE also matches or exceeds top task-tuned systems on 5 tasks and comes within 0.003 on 2 more.

4.2 POS Tagging Experiments

The POS tagging experiments compare CHARAGRAM with character CNN and LSTM architectures under a replicated Penn Treebank setup. Performance is similar across models, but CHARAGRAM reaches high accuracy faster and remains strong with compact n-gram vocabularies.

  • Experimental Setup: The experiments use the Wall Street Journal portion of the Penn Treebank, with Sections 1–18 for training, 19–21 for tuning, and 22–24 for testing.Character embeddings have 50 dimensions and induced word representations have 150 dimensions; models train for 50 epochs.
  • Results: CHARAGRAM, charCNN, and charLSTM achieve similar POS tagging performance.Adding a second fully connected 150-dimensional layer slightly improves CHARAGRAM results.
  • Convergence: CHARAGRAM converges significantly faster than charCNN and charLSTM, reaching high performance in fewer training epochs.The comparison uses tagging accuracy on the validation set and shows only the first 10 tagging epochs.
  • Convergence: Longer training might allow charCNN or charLSTM to match or surpass CHARAGRAM on word similarity, but this regime was not explored because of data and computational demands.The authors conjecture that slow convergence may explain inferior LSTM performance on similarity tasks.
  • N-gram Vocabulary Size: Using 50,000 character n-grams performs best, while 1,000 outperforms 100 on semantic similarity; POS tagging remains relatively strong with only 100 per n-gram order.These experiments vary the top 100, 1,000, or 50,000 n-grams for each order.

5 Analysis

The analysis examines CHARAGRAM-PHRASE under unknown-word and sentence-length variation, then uses nearest neighbors to inspect its qualitative behavior. The model handles arbitrary character sequences, sentence length, spelling and morphological variation, and limited word-order phenomena such as negation.

  • Quantitative Analysis: CHARAGRAM-PHRASE performs better than PARAGRAM-PHRASE for every number of unknown words and can embed any character sequence.PARAGRAM-PHRASE degrades with more unknown words because it assigns them a shared unknown-word embedding.
  • Quantitative Analysis: Both models are robust to sentence length, achieve their highest correlations on the longest sentences, and CHARAGRAM-PHRASE outperforms PARAGRAM-PHRASE at all lengths.The analysis bins 12,108 sentence pairs from 20 SemEval STS datasets by maximum sentence length.
  • Qualitative Analysis: CHARAGRAM-PHRASE models negation, whereas PARAGRAM-PHRASE cannot model it because averaging makes the word “not” dominate its nearest neighbors.For CHARAGRAM-PHRASE, the nearest neighbor is a paraphrase in all but one tested case.
  • Qualitative Analysis: Nearest neighbors capture spelling variation, morphological variants, and synonymy for words both inside and outside the training data.Words in the training data generally have higher nearest-neighbor precision, though shared n-grams can also produce non-paraphrase errors.
  • Qualitative Analysis: Character n-gram nearest neighbors form thematic groups, including death, food, speed, and language-related terms.The analysis also identifies n-grams appearing in paraphrases and words such as french and vocabulary.

6 Conclusion

The paper compares character-based compositional architectures across three NLP tasks and finds that the simplest architecture converges fastest to high performance. It argues for starting with simple architectures and releases code and trained models for general-purpose character-based representation.

  • Contribution: The study provides a careful empirical comparison of character-based compositional architectures on three NLP tasks.The paper emphasizes semantic similarity tasks alongside prior applications in translation, language modeling, and syntactic analysis.
  • Conclusion: The simplest architecture consistently converges fastest to high performance.This conclusion is supported by the convergence comparisons across the evaluated models.
  • Conclusion: The results suggest that practitioners should begin with simple architectures rather than immediately using RNNs and CNNs.The conclusion connects this recommendation with the present results and findings from Wieting et al. (2016).
  • Resources: The authors release code and trained models for community use as general-purpose character-based text representations.The release is intended to support reuse of CHARAGRAM resources.

Appendix A Training

For similarity tasks, the models learn embeddings from PPDB paraphrase pairs using a margin-based objective that makes paraphrases more similar than selected negative examples.

  • Training Objective: Similarity training uses PPDB phrase pairs assumed to be paraphrases and optimizes an L2-regularized margin-based loss.Negative examples are selected from the mini-batch during optimization.
  • Training Objective: The objective requires each paraphrase pair to exceed its similarity to respective negative examples by at least margin δ.Similarity is measured with cosine similarity between the embedded phrases.

A.1 Selecting Negative Examples

The model selects negative examples either by choosing highly similar phrases from the mini-batch or by mixing this strategy with random sampling.

  • A.1 Selecting Negative Examples: MAX selects the most similar phrase in a chosen phrase set, excluding the given phrase pair.The authors use the current mini-batch as the phrase set to reduce tunable parameters.
  • A.1 Selecting Negative Examples: MAX can accidentally select a phrase that is actually a true paraphrase of the target phrase.
  • A.1 Selecting Negative Examples: MIX selects negatives with MAX half the time and randomly from the mini-batch otherwise.The sampling strategy is tuned in the experiments.

Appendix B Tuning Word Similarity Models

Word-similarity models use matched training and architectural settings, with CHARAGRAM additionally tuning its activation, regularization, and character n-gram vocabulary.

  • Appendix B Tuning Word Similarity Models: All architectures use 300-dimensional embeddings, tune mini-batch size and negative-sampling strategy, and set δ to 0.4.
  • Appendix B Tuning Word Similarity Models: CHARAGRAM tunes tanh versus linear activation and λ over {10^-4, 10^-5, 10^-6}, using 100,283 character n-grams of lengths 2–4.
  • Appendix B Tuning Word Similarity Models: charCNN and charLSTM use randomly initialized 300-dimensional character embeddings and tune architecture-specific gates, filters, and activations.
  • Appendix B Tuning Word Similarity Models: The SemEval results table reports Pearson’s r×100, with the highest score in each row boldfaced.
  • Appendix B Tuning Word Similarity Models: charCNN and charLSTM both tune λ over {10^-4, 10^-5, 10^-6}.

Appendix C Full Sentence Similarity Results

The appendix provides the complete sentence-similarity experiment results in Table 11.

  • Appendix C Full Sentence Similarity Results: Table 11 presents the full results of the sentence similarity experiments.
Loading 1607.02789v1…