Source-linked AI summary
Unsupervised Word and Dependency Path Embeddings for Aspect Term Extraction
Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, Ming Zhou
TL;DR
Aspect term extraction requires identifying product or service properties in review sentences, while prior approaches include unsupervised rules and supervised CRF sequence labeling. The paper learns word and dependency-path embeddings jointly, models multi-hop paths with a recurrent neural network, and uses discretized embedding features in a CRF. On SemEval datasets, embedding-only features achieve state-of-the-art or comparable top-system performance, and syntactically informed WDEmb outperforms representative embedding baselines.
Problem
Aspect term extraction identifies product or service properties in reviews, but existing approaches rely on manually defined rules or supervised CRF sequence-labeling methods.
Method
The method jointly learns word and dependency-path embeddings with w1 + r ≈ w2, models multi-hop paths using a recurrent neural network, and supplies discretized context embeddings as CRF features.
Results
Embedding-only features achieve state-of-the-art or comparable top-system performance on SemEval datasets, while WDEmb outperforms representative embedding baselines.
Takeaways & Limitations
Explicitly representing syntactic information and multi-hop dependency paths supports aspect term extraction with embedding-based CRF features.
Takeaways & Limitations
The method includes only grammatical relations for dependency-path modeling because considering n-hop paths is computationally expensive, even when n = 2.
Abstract
from arXiv · showhide
In this paper, we develop a novel approach to aspect term extraction based on unsupervised learning of distributed representations of words and dependency paths. The basic idea is to connect two words (w1 and w2) with the dependency path (r) between them in the embedding space. Specifically, our method optimizes the objective w1 + r = w2 in the low-dimensional space, where the multi-hop dependency paths are treated as a sequence of grammatical relations and modeled by a recurrent neural network. Then, we design the embedding features that consider linear context and dependency context information, for the conditional random field (CRF) based aspect term extraction. Experimental results on the SemEval datasets show that, (1) with only embedding features, we can achieve state-of-the-art results; (2) our embedding method which incorporates the syntactic information among words yields better performance than other representative ones in aspect term extraction.
1 Introduction
Aspect term extraction identifies product or service properties in reviews and supports fine-grained sentiment analysis. This paper learns word and dependency-path representations unsupervised, then uses embedding-based CRF features for extraction.
- Motivation: Aspect term extraction identifies expressions referring to product or service properties from review sentences.It is described as a fundamental step toward fine-grained sentiment analysis of specific aspects.
- Motivation: Representation learning provides distributed word representations that reduce data sparsity compared with conventional one-hot representations.Such representations have been reported useful across many NLP tasks.
- Approach: The method learns word and dependency-path embeddings under an unsupervised framework, connecting words through dependency paths in a low-dimensional space.Multi-hop paths are modeled as sequences of grammatical relations, and a recurrent neural network learns their representations.
- Approach: The learned embeddings are discretized and combined into target-word, linear-context, and dependency-context features for a CRF extractor.Experiments are conducted on SemEval datasets and include comparisons with state-of-the-art models.
- Results: Using only derived embedding features, the approach achieves state-of-the-art results in aspect term extraction.The paper presents this as a contribution alongside explicitly encoding syntactic information and multi-hop dependency paths.
2 Related Work
Prior work includes unsupervised extraction based on association rules, opinion words, and dependency propagation, alongside supervised CRF systems with hand-crafted features. Representation-learning research supplies word, relation, and multi-hop path embeddings that motivate this paper’s unified dependency-path model.
- Unsupervised extraction: Unsupervised aspect extraction has used association-rule mining, opinion words, and iterative dependency-based propagation.Double propagation extracts aspect terms and opinion words using dependency relations as a crucial clue.
- Supervised extraction: Supervised aspect term extraction commonly treats the task as sequence labeling with CRF models and hand-crafted features.Top SemEval systems are described as relying on CRF and feature engineering.
- Representation learning: Word embeddings learn representations from raw text, while knowledge embeddings model relations as translation vectors between entity representations.Both are presented as successful applications of representation learning in NLP.
- Dependency representations: Related dependency-based models encode syntactic information into word or relation representations for NLP tasks.Prior work includes grammatical-relation vectors and dependency-based word embeddings, with dependency paths treated as dense vectors.
- Path representations: Knowledge-base research has modeled conjunctions and paths of multiple relations, including recurrent neural network composition and path-reliability measures.This paper instead learns the semantic composition of dependency paths over dependency trees.
3 Method
The method jointly learns word and dependency-path embeddings, enhances word representations with linear context, and converts these representations into CRF features for aspect term extraction.
- 3.1 Unsupervised Learning of Word and Dependency Path Embeddings: Dependency triples connect two words through their shortest dependency path, represented as a sequence of grammatical relations.The model extracts (w1, w2, r) from dependency trees and models r using grammatical relations.
- 3.1 Unsupervised Learning of Word and Dependency Path Embeddings: The learning objective encourages each dependency-path vector to approximate the difference between the endpoint word vectors.Ranking compares observed paths with randomly selected paths using the inner product between the path vector and w2 − w1.
- 3.1 Unsupervised Learning of Word and Dependency Path Embeddings: A recurrent neural network composes multi-hop dependency paths, recursively combining grammatical-relation embeddings into a final path representation.The model trains triples with hop numbers up to 3 because larger paths are time-consuming to learn.
- 3.2 Multi-Task Learning with Linear Context: Multi-task learning enhances word embeddings by predicting words in a fixed linear-context window around each target word.The linear-context window size is set to 5, and target and context roles use different word vectors.
- 3.4 Aspect Term Extraction with Embeddings: CRF features concatenate target-word, linear-context, and dependency-context embeddings after discretizing the continuous embedding matrix.Dependency-context features average representations of dependency-path and context-word pairs directly linked to the target, with paths limited to 3 hops.
4 Experiment
Experiments evaluate the method on SemEval datasets, compare embedding and feature configurations with established systems, and analyze embedding behavior and parameter choices. Results show benefits from contextual and syntactic information, with WDEmb outperforming representative embedding baselines and parameter-specific performance trade-offs.
- Dataset and Setting: Experiments use SemEval 2014 laptop and restaurant domains and SemEval 2015 restaurant data, with F1 as the evaluation metric.Unlabeled representations are learned from Yelp and Amazon review corpora, with additional similar-product reviews added for the laptop corpus.
- Compared Methods: The compared systems include naive dictionary tagging, baseline CRF features, top SemEval systems IHS RD, DLIREC, and EliXa, and several embedding methods.The top systems use richer lexical, syntactic, semantic, or statistical features, and some use additional-domain corpora.
- Result and Analysis: Combining target-word, linear-context, and dependency-context embeddings achieves comparable performance with the best SemEval 2015 systems and surpasses the best SemEval 2014 systems.Target-word embeddings alone perform worse, while W+L and W+D improve performance; adding baseline features improves results further.
- Comparison of Different Embedding Methods: WDEmb outperforms DepEmb because it separately embeds words and dependency paths, reducing sparsity, while multi-hop paths encode more syntactic information.DepEmb performs best among the other embedding baselines in both domains, indicating the value of syntactic information.
- Qualitative Analysis: WDEmb produces more topical similarity than DepEmb, whereas Skip-gram and CBOW produce words that are less similar in syntactic function; word-plus-path queries yield sensible results.The qualitative results are reported using nearest-word comparisons and dependency-path queries.
- Effect of Parameters: Performance increases as l rises below 15, peaks at l = 15 in two domains, then decreases; performance rises with d below 100 and remains stable for d ≥100.The authors interpret d = 100 as a trade-off between performance and training time.
5 Conclusion and Future Work
The method learns unsupervised word and dependency-path representations, discretizes them into CRF features, and achieves comparable results with top aspect-term extraction systems.
- The method learns distributed representations of words and dependency paths unsupervised, connecting words through dependency paths in embedding space.
- Dependency information helps distinguish words with similar contexts but different syntactic functions, which matter for aspect term extraction.
- Multi-hop dependency path representations are derived explicitly rather than being omitted from the embedding model.
- The learned embeddings are discretized and used as CRF features to extract aspect terms from review sentences.
- Using only distributed representation features, the method obtains comparable results with top systems on benchmark aspect term extraction datasets.