PaperlayerView exact revision on arXiv

Source-linked AI summary

Empath: Understanding Topic Signals in Large-Scale Text

Ethan Fast, Binbin Chen, Michael Bernstein

arXiv:1602.06979v1cs.CLcs.AI

TL;DR

Existing text-analysis tools cover relatively few lexical categories, limiting analysis of the broad topical and emotional signals in language. Empath generates and crowd-validates categories from seed terms using neural embeddings trained on 1.8 billion fiction words, while also providing 200 built-in categories. Its category analyses correlate highly with LIWC, although qualitative usability and seed-word selection remain open limitations.

  • Problem

    Existing lexicons cover few categories, omit useful topics, and may need manually curated updates for modern language.

  • Method

    Empath trains skip-gram neural embeddings on 1.8 billion words of modern amateur fiction, discovers terms from seed words, and validates categories with crowds.

  • Results

    0.906 correlation: Empath’s analyses closely match LIWC across a mixed-corpus dataset, remaining at 0.90 without the crowd filter.

  • Takeaways & Limitations

    Empath provides transparent, human-validated word lists for 200 built-in categories and for user-defined categories generated on demand.

  • Takeaways & Limitations

    Empath has not been evaluated on qualitative usability properties, and the choice of seed words can be important.

Abstract

from arXiv · show

Human language is colored by a broad range of topics, but existing text analysis tools only focus on a small number of them. We present Empath, a tool that can generate and validate new lexical categories on demand from a small set of seed terms (like "bleed" and "punch" to generate the category violence). Empath draws connotations between words and phrases by deep learning a neural embedding across more than 1.8 billion words of modern fiction. Given a small set of seed words that characterize a category, Empath uses its neural embedding to discover new related terms, then validates the category with a crowd-powered filter. Empath also analyzes text across 200 built-in, pre-validated categories we have generated from common topics in our web dataset, like neglect, government, and social media. We show that Empath's data-driven, human validated categories are highly correlated (r=0.906) with similar categories in LIWC.

INTRODUCTION

Empath addresses the limited breadth and costly curation of existing lexicons by generating and validating categories on demand while offering 200 pre-validated categories. Its neural embeddings learn word associations from 1.8 billion fiction words, and its analyses correlate highly with LIWC.

  • Motivation: Existing lexicons such as LIWC are limited in size, omit useful categories, and may require updates for modern terms.LIWC has 40 topical and emotional categories, many with fewer than 100 words.
  • Method: Empath learns a neural embedding from 1.8 billion words of modern amateur fiction and uses vector-space similarity to map words onto categories.The model captures associations between words and their contexts, providing a model of connotation.
  • Applications: Empath replicates and extends analyses of deceptive language, movie reviews, and Twitter mood.In deceptive reviews, 13 emotional categories are elevated among liars; positive movie reviews connect more strongly with philosophy, politics, and law.
  • Contribution: Empath lets users construct and validate new lexical categories from a few seed terms while providing 200 pre-validated emotional and topical categories.Examples include social media, neglect, government, strength, and technology.
  • Method: Empath combines deep learning and crowdsourcing to generate transparent, human-validated word-classification dictionaries.Its categories are word lists that can be extended and used for fast analysis.
  • Evaluation: 0.906 average Pearson correlation: Empath categories closely match hand-tuned, psychometrically validated categories, even without a crowd filter at 0.90.Using Google News instead produces 0.84 agreement with LIWC.

RELATED WORK

Empath builds on dictionary-based text analysis, unsupervised language modeling, and interactive text-analysis applications. It combines deep learning’s breadth with the transparency and interpretability of human-readable dictionaries.

  • Dictionary methods: Dictionary-based analysis has a long research history, with tools such as LIWC and the General Inquirer providing curated topical and emotional categories.Other lexicons target larger sets of emotional categories.
  • Empath’s position: Empath uses unsupervised deep learning to create human-interpretable feature sets while retaining dictionary transparency.This approach complements powerful sentiment-classification techniques with simpler, inspectable models.
  • Applications: Empath’s on-demand category generation could support interactive systems that respond to mood or other textual signals.Prior systems used text analysis to cue audio transitions, generate soundtracks, and map language to visual meaning.

Data Mining and Modeling

Empath applies unsupervised language modeling and mined textual knowledge to discover category signals, then demonstrates those signals in deception and review analyses. The examples show broad exploratory coverage alongside interpretable category-level comparisons.

  • Modeling: Empath discovers category terms by learning word representations from co-occurring words and mapping them into a vector space.The skip-gram model predicts nearby words and produces context-predictive neural embeddings.
  • Data sources: Empath draws on fiction mining and commonsense resources such as ConceptNet when constructing topical and emotional categories.Fiction is presented as a source for learning broad topical and emotional categories.
  • Deception analysis: Empath analyzes deceptive hotel reviews using normalized category counts, odds ratios, and Bonferroni-corrected significance tests.The dataset contains 3,200 truthful and deceptive reviews.
  • Deception results: 13 of Empath’s 32 emotional categories have significantly higher odds in deceptive reviews, while truthful reviews show higher odds for none of them.Liars also show higher odds for smell, while truth-tellers more often mention concrete phenomena such as the ocean and vehicles.

Spatial language in lies

Empath uses seeded, crowd-validated categories to test hypotheses about language patterns and compare its signals with established analyses. Its examples include spatial language in lies, sentiment in movie reviews, and hourly Twitter mood.

  • Spatial language in lies: Empath can generate a human-validated spatial category from the seeds “big,” “small,” and “circular” to test whether liars use less spatial language.The discovered cluster includes terms such as large, rectangular, oval, shape, and boxy.
  • Spatial language in lies: 1.2 odds: the new spatial category favors truthful reviews, whose language contains more concrete spatial details.The reported difference is significant at p < 0.001.
  • Movie reviews: 1.4 times more likely: ugliness appears in negative movie reviews; swear words and pain are each 1.3 times more likely.Positive reviews instead show higher likelihoods for beauty at 1.8, joy at 1.5, and pride at 1.4.
  • Movie reviews: No significant difference: Empath and LIWC classify positive versus negative movie reviews similarly using five shared emotional categories.The comparison uses logistic regression with 10-fold cross-validation and reports p = 0.43.
  • Twitter mood: 0.90 correlation: Empath and LIWC report strongly correlated hourly negative-mood signals, both reaching a low around 8am and a high around 11pm.Empath’s hourly negative-affect ANOVA is F(23, 591520) = 17.2, p < 0.001.

EMPATH

Empath broadens dictionary-based text analysis with 200 human-validated categories and a data-driven route to new categories from seed terms.

  • Empath analyzes text across hundreds of topical and emotional categories, including 200 human-validated categories.
  • ConceptNet dependency relationships supply category names and seed words, enabling a potentially scalable alternative to purely manual category creation.
  • Researchers can create and validate a clothing category from seeds such as “shirt” and “hat,” producing related terms including blazer, vest, and sweater.
  • Empath refines its base categories by selecting 200 common ConceptNet relationships, manually removing redundant or sparse categories, and using two to five seed terms per category.
  • The same seed-based term-generation process applies to topical and emotional categories, including 32 emotional categories bootstrapped from Parrott’s hierarchy.

Learning category terms from a text corpus

Empath learns category members from contextual relationships in a large text corpus, using vector-space similarity to expand small seed sets.

  • Empath uses seed words to generate candidate category members and validates those candidates through paid crowdsourcing.
  • A neural-network-trained vector space model compares words across many dimensions of meaning to find terms related to category seeds.
  • For this paper, Empath trains on more than 1.8 billion words of modern Wattpad fiction written by hundreds of thousands of authors.

Training a neural vector space model

Empath trains a skip-gram neural embedding on fiction, then uses vector representations and cosine similarity to retrieve terms near category queries.

  • The skip-gram network learns word representations by predicting co-occurring context words and mapping words into a vector space.
  • Training uses lemmatization, a 150-neuron hidden layer, a five-word sliding window, a thirty-occurrence minimum, negative sampling, and frequent-term down-sampling.
  • Empath’s vector space model uses cosine similarity to select nearby member terms for categories such as social media, violence, and shame.
  • The vector space has dimensions M(n × h), with vocabulary size n=40,000 and hidden-layer dimensionality h=150.
  • Category queries combine the category vector, when available, with vectors for its seed terms; clothing uses v(clothing) + v(shirt) + v(hat).
  • From a small seed, Empath can gather hundreds of category-related terms for textual analysis.

Refining categories with crowd validation

Empath filters automatically generated category terms through crowd judgments to improve precision and validate category contents.

  • Human validation is intended to prevent accidental terms from entering Empath’s lexicon while combining deep learning with human validation.
  • Crowdworkers rate each candidate’s relationship to a topic on a four-point scale from not related to strongly related.
  • Each task contains twenty words, is completed by three independent Masters workers, and costs $0.14 per task.
  • Empath keeps a word when at least two of three workers judge it at least weakly related; higher thresholds discard more terms.

Contents and Efficiency

Empath combines machine-generated category expansion with crowd validation to support broad, efficient lexical analysis. It offers 200 pre-validated categories and can be used through a web service or open-source library.

  • 95% of words generated by Empath’s unsupervised model were judged related to their categories, while workers agreed unanimously 81% of the time.
  • Validating 200 categories with more than 10,000 words cost $840, compared with an estimated $21,000 for naive crowdsourcing.
  • Crowd judgments remain subjective: 36% of rejected words received a minority vote for relevance.
  • Empath is available as a web service and open-source Python library for analyzing built-in categories and requesting validated new ones.

Comparing Empath and LIWC

The evaluation compares Empath with LIWC across shared categories using a mixed corpus and tuned seed and similarity parameters. It also benchmarks Empath against other validated lexicons and tests seed-term sensitivity.

  • The benchmark compares Empath and LIWC over 12 shared categories spanning emotion, money, death, achievement, home, religion, work, and health.
  • The mixed corpus contains more than 2 million words across 4,500 documents from tweets, StackExchange, reviews, and classic novels.
  • Empath categories were capped at 200 words, with up to five LIWC-approximating seed terms and a minimum cosine similarity of 0.5 selected on training data.
  • Pearson correlations between category word counts quantify agreement between Empath, LIWC, and benchmark lexicons.
  • GI and EmoLex correlated with LIWC at 0.876 and 0.899, respectively, establishing comparison benchmarks for Empath.
  • Seed sensitivity was tested by dropping one seed or replacing it with a similar alternative in each category.

Results

Empath closely matches LIWC overall, with crowd filtering producing a small average improvement, while seed choice and training data affect some categories. The results also expose ambiguity and scope limits in data-driven lexicons.

  • 0.906 is Empath’s overall average crowd-filtered Pearson correlation with LIWC, versus 0.90 for the unsupervised model.
  • Crowd filtering improved average correlation by 0.006, but the improvement was not statistically significant.
  • Removing a unique seed reduced agreement as far as 0.82; dropping “kill” from death removed adversarial terms such as “war,” “execute,” and “murder.”
  • Crowd filtering can decrease agreement in individual categories when workers remove words such as “semester” that LIWC includes.
  • Empath’s effectiveness depends on training data because fictional associations may perform less well on corner cases of connotation.
  • Fiction-trained embeddings averaged 0.90 unsupervised correlation with LIWC, while Google News embeddings decreased to 0.84.
  • Fiction was useful for training because it contains abundant concrete descriptors and emotional terms.

Limitations

Empath’s evaluation leaves important questions about qualitative usability, generalization beyond LIWC, seed selection, and corpus choice. Its expanded category space also requires correction for false positives from multiple comparisons.

  • Qualitative evaluation: Empath’s high Pearson correlations with LIWC do not establish interpretability, representative vocabulary, or low false-positive and false-negative rates.The authors have not evaluated these qualitative usability properties.
  • Generalization: Empath’s categories have not been tested for generalization beyond the core categories shared with LIWC.The authors specifically leave the practical performance of new categories as an open question.
  • Category construction: Seed-word selection can materially affect category quality, while the authors do not yet know what makes a good seed set.They identify seed discovery and selection as future-work questions.
  • Training data: Fiction is effective for generating categories, but other corpora may be better for particular topics.The authors cite Google News for death and propose fashion forums or Hacker News for topic-specific training.
  • Statistical testing: Expanding the number of analyzable categories increases the risk that random variation produces an apparently elevated category.This paper uses Bonferroni correction, while Holm’s method and FDR are identified as more mature alternatives.
Loading 1602.06979v1…