Source-linked AI summary
Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction
Hu Xu, Bing Liu, Lei Shu, Philip S. Yu
TL;DR
Aspect extraction must identify opinion targets in product reviews, while existing systems can depend on manual features or complex architectures. This paper proposes DE-CNN, combining general and exactly in-domain embeddings with a pure CNN sequence-labeling model, and reports that it outperforms state-of-the-art methods by a large margin.
Problem
Aspect extraction requires identifying product attributes in opinion text, but competitive supervised systems often use handcrafted features or complex architectures.
Method
DE-CNN combines general-purpose and exactly in-domain pretrained embeddings with a pure CNN sequence-labeling model, without extra supervision.
Results
DE-CNN outperforms state-of-the-art methods with a large margin, while ablations show that double embeddings and in-domain embeddings are important.
Takeaways & Limitations
The paper reports that a simple CNN with double embeddings can achieve strong aspect-extraction performance without additional supervision.
Takeaways & Limitations
DE-CNN errors include inconsistent labeling and unseen test aspects involving conjunctions, which the authors leave for future work.
Abstract
from arXiv · showhide
One key task of fine-grained sentiment analysis of product reviews is to extract product aspects or features that users have expressed opinions on. This paper focuses on supervised aspect extraction using deep learning. Unlike other highly sophisticated supervised deep learning models, this paper proposes a novel and yet simple CNN model employing two types of pre-trained embeddings for aspect extraction: general-purpose embeddings and domain-specific embeddings. Without using any additional supervision, this model achieves surprisingly good results, outperforming state-of-the-art sophisticated existing methods. To our knowledge, this paper is the first to report such double embeddings based CNN model for aspect extraction and achieve very good results.
1 Introduction
Aspect extraction targets product attributes in opinion text, but existing supervised deep models often rely on handcrafted features or complex architectures. The paper addresses this with double embeddings and a pure CNN sequence-labeling model.
- Motivation: Aspect extraction identifies product attributes or features that users express opinions about in review text.For example, “speed” is extracted from “Its speed is incredible” in a laptop review.
- Motivation: Existing supervised deep learning approaches often use handcrafted features, lexicons, and complicated neural network architectures.
- Motivation: Competitive performance with automated feature learning and a simple model is important, especially for real-life applications where complex models slow inference.
- Method: The proposed double embeddings mechanism combines general-purpose and domain-specific representations because aspect extraction requires fine-grained domain information.General embeddings may better represent common words, while in-domain embeddings can capture domain-specific meanings.
- Method: DE-CNN uses a pure CNN for sequence labeling instead of relying on sequentially dependent LSTM computation.The paper motivates CNNs as simpler and more parallelizable for this task.
2 Related Work
Prior aspect-extraction research includes supervised and unsupervised methods, with recent supervised systems using neural sequence models and attention alongside manual features. The paper situates its CNN approach within this landscape.
- Existing approaches: Aspect extraction has been studied through both unsupervised methods and supervised approaches.
- Supervised methods: Traditional supervised aspect extraction commonly uses Conditional Random Fields, while newer systems apply deep neural networks.
- Supervised methods: Recent neural approaches include LSTM-based models and attention mechanisms, often combined with manual features.
- CNN methods: CNNs have been adopted for named entity recognition and sentiment classification, and this paper applies them to aspect-extraction sequence labeling because they are simple and parallelized.
3 Model
DE-CNN processes each input sequence with general and exactly matched in-domain embeddings, then applies aligned CNN layers for position-wise aspect labeling. Its design also fixes embedding parameters to reduce problems from unseen test words.
- Architecture: DE-CNN contains two embedding layers, four CNN layers, a shared fully connected layer, and a position-wise softmax over B, I, and O labels.B and I mark the beginning and continuation of aspect phrases; O marks non-aspect words.
- Embeddings: Each input word sequence receives separate general and domain-specific representations from two embedding matrices.The general embeddings are pretrained on a very large corpus, while domain embeddings come from a small corpus matching the task domain.
- Embeddings: The domain embedding corpus must exactly match the domain of the training and testing data.For example, laptop reviews require laptop-domain rather than broader electronics-domain embeddings.
- Training design: The embedding layers are kept non-trainable because limited training data can produce unseen test words whose embeddings would retain mismatched features.
- CNN processing: The two embeddings are concatenated and passed through a stack of four CNN layers.
- CNN processing: CNN filters use odd kernel sizes, stride one, and zero padding so outputs remain aligned with original word positions.Each filter represents a word together with nearby contextual words.
4 Experiments
Experiments evaluate DE-CNN on laptop and restaurant aspect-extraction benchmarks against single-task, multi-task, and model-variation baselines. DE-CNN performs best, while analyses support double embeddings, exact in-domain data, position-preserving convolutions, and a simple softmax architecture.
- Datasets: Experiments use laptop and restaurant review datasets from SemEval aspect-extraction challenges.The datasets contain review sentences with aspect terms labeled as character spans.
- Baselines: DE-CNN is compared with single-task systems, multi-task systems using opinion supervision or sentiment lexicons, and several DE-CNN variants.The comparison includes CRF, LSTM, joint aspect-opinion, pooling, embedding, and output-layer variants.
- Results: DE-CNN performs the best, and using either general embeddings or domain embeddings alone gives inferior performance.The results support combining both embedding types rather than relying on one alone.
- Architecture analysis: Max pooling harms sequence labeling because it loses word positions, while replacing softmax with CRF does not substantially improve DE-CNN.CRF can improve laptop performance by 1–2% around 75%, but contributes little above 80%; its hyperparameters were not tuned because training is extremely slow.
- Embedding analysis: Out-of-domain embeddings perform poorly, indicating that the domain-embedding corpus must exactly match the aspect-extraction domain.A smaller corpus for general embeddings also produces poorer performance than DE-CNN.
- Cross-task comparison: BiLSTM-CNN-CRF is markedly worse than DE-CNN, suggesting that models developed for named entity recognition may need adaptation for opinion texts.The paper attributes this possibility to the baseline’s dependency-based embeddings and points to domain embeddings as one potential adaptation.
- Error analysis: DE-CNN errors arise from inconsistent labeling and unseen test aspects requiring conjunction semantics, such as extracting B from “A and B”; the latter is left for future work.These errors define important boundaries for the reported results.
5 Conclusion
The paper proposes DE-CNN, a CNN-based aspect-extraction model using double embeddings without extra supervision. Experimental results show that it outperforms state-of-the-art methods with a large margin.
- The paper proposes a CNN-based aspect-extraction model with a double-embeddings mechanism without extra supervision.