PaperlayerView exact revision on arXiv

Source-linked AI summary

SentiHood: Targeted Aspect Based Sentiment Analysis Dataset for Urban Neighbourhoods

Marzieh Saeidi, Guillaume Bouchard, Maria Liakata, Sebastian Riedel

arXiv:1610.03771v1cs.CL

TL;DR

Existing sentiment-analysis tasks do not adequately represent texts that express sentiments toward multiple aspects of multiple entities. The paper introduces targeted aspect-based sentiment analysis and the SentiHood QA-based neighbourhood dataset, evaluates logistic-regression and LSTM baselines, and reports that QA content is rich in information despite not being written as reviews.

  • Problem

    Existing aspect-based and targeted sentiment-analysis settings make restrictive assumptions about the number of entities or the sentiment structure in a text, although multiple entities and aspects can be discussed together.

  • Method

    The paper defines targeted aspect-based sentiment analysis and introduces SentiHood, a QA-derived urban-neighbourhood dataset, with logistic-regression and LSTM baselines.

  • Results

    QA and online-forum texts are reported to be rich in information despite not being written with the goal of writing a review.

  • Takeaways & Limitations

    The paper establishes a benchmark for extracting sentiments toward aspects of one or more urban-neighbourhood entities from less constrained QA text.

  • Takeaways & Limitations

    The authors identify data augmentation and more detailed analysis as ways to improve robustness, especially for LSTM models.

Abstract

from arXiv · show

In this paper, we introduce the task of targeted aspect-based sentiment analysis. The goal is to extract fine-grained information with respect to entities mentioned in user comments. This work extends both aspect-based sentiment analysis that assumes a single entity per document and targeted sentiment analysis that assumes a single sentiment towards a target entity. In particular, we identify the sentiment towards each aspect of one or more entities. As a testbed for this task, we introduce the SentiHood dataset, extracted from a question answering (QA) platform where urban neighbourhoods are discussed by users. In this context units of text often mention several aspects of one or more neighbourhoods. This is the first time that a generic social media platform in this case a QA platform, is used for fine-grained opinion mining. Text coming from QA platforms is far less constrained compared to text from review specific platforms which current datasets are based on. We develop several strong baselines, relying on logistic regression and state-of-the-art recurrent neural networks.

1 Introduction

The paper introduces targeted aspect-based sentiment analysis for texts discussing multiple entities and aspects, and presents SentiHood as a QA-based urban-neighbourhood benchmark with logistic-regression and LSTM baselines.

  • 1 Introduction: Targeted aspect-based sentiment analysis extracts each target entity, aspect, and associated sentiment from text mentioning multiple entities.It extends existing aspect-based and targeted sentiment-analysis settings, which make more restrictive assumptions about entities and sentiments.
  • 1 Introduction: SentiHood is a benchmark dataset annotated for targeted aspect-based sentiment analysis in urban neighbourhoods.The dataset uses discussions from a question-answering platform about city neighbourhoods.
  • 1 Introduction: QA and online-forum texts contain specific information about several aspects of several neighbourhoods, despite not being written as reviews.This motivates using less constrained social-media text for fine-grained opinion mining.
  • 1 Introduction: The paper provides strong baselines using logistic regression and Long Short Term Memory networks.These models are used to establish initial baselines for the new task.

2 SentiHood

SentiHood is a targeted aspect-based sentiment dataset built from Yahoo! Answers discussions about London neighbourhoods. Its annotation design limits sentences to one or two location mentions and assigns predefined aspect and polarity labels.

  • SentiHood uses Yahoo! Answers questions about London neighbourhoods as its source material for targeted aspect-based sentiment analysis.The dataset is constructed from question-answer content filtered for neighbourhood-related questions.
  • Sentences with one or two location mentions are retained and grouped as Single or Multi to compare annotation and modelling difficulty.Although sentences may contain more than 50 location mentions, the dataset annotates only those with one or two.
  • Annotators choose from a predefined aspect inventory including live, safety, price, quiet, dining, nightlife, transit-location, touristy, shopping, green-culture, multicultural, and general.The miscellaneous aspect was rejected because it reduced annotation decisiveness and agreement.
  • Each selected aspect receives either a Positive or Negative label because polarity-free aspect discussions were rare in the data.Unlike many related schemes, the annotation does not provide a Neutral sentiment category.

2.5 Target Entity

The annotation targets location entities for which an aspect and sentiment are expressed, while excluding irrelevant or uncertain sentences. Agreement was sufficient overall, with disagreements concentrated mainly on aspect identification.

  • A target entity is the location for which an opinion consisting of an aspect and sentiment is expressed.
  • Sentences are removed when a named entity is irrelevant, such as when a neighbourhood name refers to a film rather than a location.
  • Sentences are also removed when sentiments conflict for the same location and aspect or when the opinion lacks a directly mentioned area.
  • Aspect-category agreement reached Cohen’s Kappa values of 0.73, 0.78, and 0.70 across annotator pairs.These values were deemed sufficient quality.
  • Most annotator disagreements involved identifying the aspect rather than the sentiment, aspect expression, or target location.Examples included differing interpretations of whether “residential” implies quietness or desirability for living.
  • Target-location ambiguity arose when one area was geographically contained within another, such as Angel within Islington.

2.8 Dataset

SentiHood contains 5,215 annotated sentences and 5,920 opinions about neighbourhood aspects, with positive sentiment dominant for several aspects. Location names are masked for the task, while original names are also supplied.

  • 5215 sentences comprise SentiHood, including 3862 single-location sentences and 1353 sentences mentioning two locations.
  • Positive sentiment is dominant for aspects such as dining and shopping.
  • 5920 opinions exceed the sentence count because each sentence can contain one or more opinions.
  • The general aspect appears in over 2000 sentences, whereas touristy occurs in fewer than 100.
  • Location entity names are replaced with location1 and location2, while a version retaining original names is also provided.

3 Task

Targeted aspect-based sentiment analysis assigns sentiment to each aspect of each entity mentioned in a text unit. The task is framed as three-class classification for every entity–aspect pair, including a None label for no expressed opinion.

  • The task maps a text unit to a list of tuples (l, a, p), where p is the polarity for aspect a of entity l.
  • A sentence may contain zero to T labels, allowing multiple entity–aspect–polarity assignments.
  • Unlike existing ABSA formulations, the task jointly identifies each entity, its aspect, and its sentiment without requiring opinion-target-expression detection as a separate output.
  • For each (l, a) pair, classification uses Positive, Negative, or None, where None means no opinion is expressed for that aspect of that location.
  • Table 2 illustrates the task using an input sentence paired with its output labels.

4 Evaluation

The evaluation uses F1, accuracy, and AUC to assess aspect and sentiment detection, with F1 thresholds optimized on validation data and AUC avoiding threshold dependence.

  • F1 measures aspect detection, while accuracy measures sentiment classification, following common evaluation practice.
  • AUC evaluates ranking quality for both aspect and sentiment detection without relying on a threshold.

5 Baseline

The baselines formulate sentiment prediction as three-class classification for each aspect, using softmax over location representations derived from sparse features or bidirectional LSTMs.

  • Each baseline predicts a three-class sentiment label for every aspect using a softmax classifier.The classifier uses weights and biases specific to each sentiment class.
  • Location representations can use bag-of-words or distributional features, with each method defining its own representation.
  • Masked target-location n-grams represent each location in context while distinguishing multiple locations in one sentence.
  • Left-right n-grams concatenate separate n-gram representations of the contexts surrounding each location.
  • Left-right pooling combines max, minimum, average, and standard-deviation pooling of word embeddings from both contexts.
  • Bidirectional LSTMs learn aspect-specific classifiers using either the final sequence state or the state at the location index.The location-index representation is illustrated in Figure 2, where the selected state feeds a softmax sentiment classifier.

6 Experiments

Experiments focus on four frequent aspects and compare logistic-regression and LSTM baselines across single- and multiple-location sentences using fixed train, development, and test splits.

  • The experiments select price, safety, transit-location, and general as the four most frequent aspects.
  • The data are divided into 70% training, 10% development, and 20% test sets, with model selection based on development performance.
  • Results are reported separately for sentences containing one location, two locations, and all test sentences.
  • LSTM models use development loss for model selection and address class imbalance through batches containing equal numbers of examples from each sentiment class.
  • Logistic regression uses class-weighted training, with weights selected by cross-validation on the development set.

7 Results

The results compare logistic regression and LSTM systems across representations, location multiplicity, and aspects. Masked n-grams with POS information are strong overall, while LSTM performance varies by setting and does not consistently surpass logistic regression.

  • Table 3 compares best logistic-regression and LSTM models using F1, accuracy, and AUC averaged over selected aspects.
  • Location-masked n-grams slightly outperform left-right context representations, and POS information further improves performance.
  • Adding trigrams does not improve overall scores, while separating left and right bag-of-words contexts also fails to improve performance.
  • The best LSTM is not superior to logistic regression, especially on AUC; limited training data and absent POS information may contribute.
  • The two LSTM variants are not significantly different, although final-state embeddings perform slightly better than location-index embeddings.
  • Logistic regression performs slightly better on single-location sentences, whereas LSTM performs slightly better on multiple-location sentences.
  • Safety receives better AUC scores than the general aspect in the aspect-level breakdown.
  • The best logistic-regression system correctly identifies positive general sentiment for location2 while assigning no sentiment for that aspect to location1.

8 Related Work

Sentiment analysis progressed from overall-text polarity toward finer-grained analysis of aspects and target entities. Targeted aspect-based sentiment analysis addresses both limitations while supporting more generic text.

  • Early sentiment analysis mainly identified one overall polarity for an entire text unit.
  • Aspect-based sentiment analysis identifies sentiment toward multiple aspects but assumes a single entity per text unit.
  • Targeted sentiment analysis identifies polarity toward a target entity, but retains limitations that targeted aspect-based sentiment analysis addresses.
  • Targeted aspect-based sentiment analysis caters for more generic text by making fewer assumptions while extracting fine-grained information.

9 Conclusion

The paper introduces targeted aspect-based sentiment analysis, a new dataset, and logistic-regression and LSTM baselines. It identifies data augmentation and richer analysis as directions for improving robustness and understanding system capabilities.

  • The paper introduces targeted aspect-based sentiment analysis and a new dataset.
  • It provides two strong baselines using logistic regression and LSTM.
  • Data augmentation could make the models, especially LSTM, more robust to variations in the data.
  • The authors also identify more detailed analysis of each system's capabilities as future work.
Loading 1610.03771v1…