Source-linked AI summary
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, Lawrence Carin
TL;DR
The paper asks whether sophisticated compositional functions add value over simple word-embedding pooling, given their greater parameter and computational costs. It compares parameter-free SWEMs with RNN- and CNN-based models across 17 datasets and introduces max and hierarchical pooling. SWEMs perform comparably or better in most cases, while hierarchical pooling retains spatial information and reaches comparable results to LSTM/CNN on word-order-sensitive tasks.
Problem
The paper addresses the lack of rigorous evidence about the added value of sophisticated compositional functions over simple pooling for text representation.
Method
The paper compares parameter-free SWEM pooling models with recurrent and convolutional networks across 17 datasets and proposes max and hierarchical pooling variants.
Results
SWEMs achieve comparable or superior performance in most evaluated cases, while hierarchical pooling is comparable to LSTM/CNN on tasks sensitive to word order.
Takeaways & Limitations
Simple pooling is a strong baseline, and compositional functions should be selected according to task requirements and the computation-versus-expressiveness tradeoff.
Takeaways & Limitations
SWEM can ignore word-order information, which is especially important for sentiment analysis, and results on small datasets are sensitive to regularization.
Abstract
from arXiv · showhide
Many deep learning architectures have been proposed to model the compositionality in text sequences, requiring a substantial number of parameters and expensive computations. However, there has not been a rigorous evaluation regarding the added value of sophisticated compositional functions. In this paper, we conduct a point-by-point comparative study between Simple Word-Embedding-based Models (SWEMs), consisting of parameter-free pooling operations, relative to word-embedding-based RNN/CNN models. Surprisingly, SWEMs exhibit comparable or even superior performance in the majority of cases considered. Based upon this understanding, we propose two additional pooling strategies over learned word embeddings: (i) a max-pooling operation for improved interpretability; and (ii) a hierarchical pooling operation, which preserves spatial (n-gram) information within text sequences. We present experiments on 17 datasets encompassing three tasks: (i) (long) document classification; (ii) text sequence matching; and (iii) short text tasks, including classification and tagging. The source code and datasets can be obtained from https:// github.com/dinghanshen/SWEM.
1 Introduction
The paper evaluates whether simple pooling over word embeddings can replace more expressive, expensive compositional functions across diverse NLP tasks. SWEMs often match or exceed recurrent and convolutional models, while max and hierarchical pooling add interpretability and spatial information.
- Motivation: SWEMs use parameter-free pooling over word embeddings, offering far fewer parameters and faster training than RNN- or CNN-based models.This creates a computation-versus-expressiveness tradeoff because simple pooling generally omits word-order information.
- Study design: The study compares SWEMs with recurrent and convolutional networks on 17 datasets spanning document classification, sequence matching, and short-text classification or tagging.The comparison is conducted point by point across three distinct NLP task groups.
- Main finding: SWEMs achieve comparable or superior performance in the majority of evaluated cases.This is the paper’s central empirical finding across the considered tasks and datasets.
- Proposed extensions: Max pooling selects salient embedding features for improved interpretability, while hierarchical pooling preserves spatial n-gram information.The proposed strategies address complementary limitations of basic pooling operations.
- Implication: The paper presents SWEMs as a strong baseline and emphasizes selecting compositional functions according to the computation-versus-expressiveness tradeoff.It also argues that simple models are preferred under Occam’s razor when task difficulty is similar across model classes.
2 Related Work
The paper situates SWEMs among simpler embedding-based architectures that can rival recurrence and convolution. It distinguishes its broader pooling exploration and its hierarchical strategy for incorporating spatial information.
- Prior work: Recent work suggests that simple word-embedding architectures can match or exceed recurrent or convolutional models on some NLP applications.These approaches avoid complex compositional functions but may add modules such as attention layers.
- Relation to prior models: SWEMs resemble DAN and fastText because all use average pooling successfully on selected NLP tasks.The paper’s comparison is broader because it examines multiple pooling operations rather than average pooling alone.
- Novelty: SWEM-hier incorporates spatial information and demonstrates superior sentiment-analysis results relative to average pooling.It learns fixed-length representations for n-grams rather than only recording their counts.
- Scope: The paper studies simple pooling across a wide range of NLP problems and derives rules for selecting models for different tasks.Its contribution extends beyond a single pooling operation or benchmark.
3 Models & training
The paper represents sequences by combining word embeddings with recurrent, convolutional, or parameter-free pooling functions. SWEM variants trade compositional expressiveness for efficiency, while hierarchical pooling retains local n-gram structure.
- Sequence representation: A sequence X contains L word tokens with embeddings v_i ∈ R^K, and a compositional function maps them to a fixed-length representation z for prediction.The representation can describe either a sentence or a document.
- Recurrent encoder: LSTM encoders update hidden states recurrently using each word vector and the preceding hidden state, incorporating word order but adding learned compositional parameters.The transition function controls how the current state combines the current input with prior information.
- Convolutional encoder: CNN encoders apply learned filters to windows of n consecutive words and aggregate the resulting feature maps to extract salient semantic features.Deep CNN text models are also included in some experiments.
- Average pooling: SWEMs use no additional compositional parameters; SWEM-aver computes element-wise means, producing a representation with the embedding dimension.Average pooling incorporates every sequence element into z.
- Max pooling: SWEM-max takes the maximum value in each embedding dimension, allowing task-irrelevant words to be ignored and producing more salient features.This operation is motivated by the observation that only a small number of keywords may drive predictions.
- Combined pooling: SWEM-concat combines average- and max-pooled features, which the paper treats as complementary information sources.All SWEM variants rely only on intrinsic word-embedding information.
- Hierarchical pooling: SWEM-hier averages each local n-word window and then applies global max pooling across windows, preserving local spatial n-gram information.It produces fixed-length n-gram representations rather than simple occurrence counts.
- Efficiency comparison: SWEM has no compositional parameters and is computationally more efficient than CNN and LSTM; when K = d, it is faster by factors of nd and d, respectively.The comparison considers parameters, computational complexity, and sequential operations.
4 Experiments
Across 17 datasets, SWEMs are evaluated against recurrent and convolutional models across document classification, sentence matching, and short-text tasks. Results show strong performance and efficiency on many tasks, while word order remains important for sentiment and short sentences.
- Experimental setup: Experiments cover 17 datasets spanning document categorization, natural-language sequence matching, and short sentence classification.The models use GloVe embeddings, either directly updated during training or transformed through a 300-dimensional ReLU MLP.
- Document categorization and text matching: SWEMs achieve comparable or superior results to CNN and LSTM models on many document-classification and sentence-matching tasks.They are stronger on topic prediction and generally perform best on most matching datasets except WikiQA.
- Efficiency: 61K parameters let SWEM-concat outperform CNN/LSTM on Yahoo! Answer while using one-tenth as many LSTM parameters and a fraction of their training time.The comparison excludes embeddings from the parameter counts.
- Pooling variants: SWEM-max generally trails SWEM-aver slightly but supplies complementary features, making SWEM-concat the strongest SWEM variant in most cases.SWEM-max embeddings are sparse, with values concentrated around zero, and its pooling selects the largest value in each embedding dimension.
- Text sequence matching: On SNLI, SWEM-max reaches 83.8% test accuracy with only 120K parameters, performing best among the SWEM variants.The result is described as competitive with sentence-encoding models in both performance and parameter count.
- Word order and sentiment: Word order matters most for sentiment: shuffled-training LSTMs remain comparable on Yahoo and SNLI but drop noticeably on Yelp, while SWEM is weaker on short sentences.Hierarchical pooling addresses this limitation by incorporating local n-gram information and achieves accuracies comparable to CNN or LSTM on document-level sentiment tasks.
5 Discussion
The discussion evaluates SWEM complexity and representation quality against CNN/LSTM models, finding strong parameter efficiency and competitive performance across tasks. Hierarchical pooling is especially effective when spatial information matters.
- Subspace Training: Subspace training constrains trainable parameters to a low-dimensional space, with intrinsic dimension d_int indicating the minimum dimension yielding a good solution.The study compares SWEM-max and CNN models under jointly optimized or frozen word embeddings.
- Subspace Training: SWEM retains similar accuracy to direct training at very small subspace dimensions, even when model parameters are not trained at all (d = 0).This result is reported for AG News with randomly initialized, jointly optimized word embeddings.
- Subspace Training: With frozen GloVe embeddings, SWEM achieves significantly higher accuracy than CNN across a large range of low subspace dimensions.On AG News, SWEM reaches an 80% testing-accuracy threshold at a lower d_int than CNN; on Yelp, CNN performs better when d is large.
- Representation Quality: A linear classifier causes only small drops on Yahoo! Answers, from 73.53% to 73.18%, and Yelp Polarity, from 93.76% to 93.66%.The result supports the informativeness of SWEM sentence representations without a nonlinear MLP classifier.
- Other Languages: On Sogou news, SWEM-hier reaches 96.2% accuracy, comparable to CNN at 95.6% and LSTM at 95.2%.SWEM-concat reaches 91.3%; the hierarchical model uses a local window size of 5 and incorporates spatial information.
6 Conclusions
The conclusions identify when simple pooling is sufficient and when order-sensitive composition is more useful. Across the evaluated tasks, document and matching problems favor SWEMs, while hierarchical pooling narrows the gap on sentiment tasks.
- Conclusions: Simple pooling is surprisingly effective for longer documents, whereas recurrent and convolutional functions are most effective for short sentences.The paper frames this as a task-dependent choice of compositional function.
- Conclusions: Sentiment analysis is more sensitive to word order than topic categorization, but hierarchical pooling achieves comparable results to LSTM/CNN on sentiment tasks.Hierarchical pooling preserves local spatial information while retaining a simple architecture.
- Conclusions: For sentence matching tasks such as textual entailment and answer selection, simple pooling achieves similar or superior results to CNN and LSTM.This conclusion covers the paper’s matching-task comparisons.
- Conclusions: In SWEM-max, each embedding dimension contains interpretable semantic patterns that group words sharing a theme or topic.Max pooling selects salient features directly from the word-embedding dimensions.
Appendix I: Experimental Setup
The appendix details the datasets, tagging setup, representation analyses, embedding dimensions, and limited-data experiments used to assess SWEMs. These experiments show strong low-data robustness and efficient semantic representation, while sequence tagging remains order-sensitive.
- Experimental Setup: The evaluation covers document categorization, text sequence matching, and short sentence classification, using established dataset splits and preprocessing procedures.Table 9 reports dataset statistics, including average words, class counts, and training-set sizes.
- Sequence Tagging: CNN-CRF and BI-LSTM-CRF consistently outperform SWEM-CRF on both sequence-tagging tasks, although BI-LSTM-CRF training takes around 4 to 5 times longer.The comparison indicates that contextual compositional functions are important for chunking and NER.
- Embedding Interpretability: SWEM-max identifies predictive topic-related words by selecting the top 10 words for each embedding dimension on Yahoo! Answers.The selected words are visualized in Figure 3.
- Embedding Dimensions: With only 10 embedding dimensions, SWEM achieves results comparable to the 1000-dimensional setting on Yahoo, despite generally improving with larger dimensions.The experiment varies embedding size from 3 to 1000 and suggests that lower-dimensional embeddings can reduce parameters while remaining competitive.
- Limited Training Data: Using 0.1% of Yahoo training data, SWEM reaches 56.10% accuracy, compared with 25.32% for CNN and 42.37% for LSTM.SWEM consistently outperforms CNN and LSTM across a wide range of training-data proportions on Yahoo and SNLI.