Source-linked AI summary
Reasoning Implicit Sentiment with Chain-of-Thought Prompting
Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, Tat-Seng Chua
TL;DR
Implicit sentiment analysis must recover sentiment from obscure cues using common-sense and multi-hop reasoning. The paper introduces THOR, which prompts language models to infer an aspect, opinion, and polarity sequentially, achieving improvements over existing baselines in supervised and zero-shot ISA settings.
Problem
Implicit sentiment analysis contains factual descriptions without explicit opinion cues, so accurate detection requires inferring latent intent through common-sense and multi-hop reasoning.
Method
THOR uses three chain-of-thought prompts to infer the fine-grained aspect, underlying opinion, and final sentiment polarity step by step.
Results
THOR-equipped language models outperform existing best-performing baselines on ISA datasets in both supervised and zero-shot setups, with larger models showing more significant improvements.
Takeaways & Limitations
THOR provides a multi-hop reasoning approach for implicit sentiment detection and is reported as broadly applicable to similar NLP problems.
Takeaways & Limitations
THOR’s improvements are limited on middle- or lower-size language models because its full benefits emerge only with sufficiently large models.
Abstract
from arXiv · showhide
While sentiment analysis systems try to determine the sentiment polarities of given targets based on the key opinion expressions in input texts, in implicit sentiment analysis (ISA) the opinion cues come in an implicit and obscure manner. Thus detecting implicit sentiment requires the common-sense and multi-hop reasoning ability to infer the latent intent of opinion. Inspired by the recent chain-of-thought (CoT) idea, in this work we introduce a Three-hop Reasoning (THOR) CoT framework to mimic the human-like reasoning process for ISA. We design a three-step prompting principle for THOR to step-by-step induce the implicit aspect, opinion, and finally the sentiment polarity. Our THOR+Flan-T5 (11B) pushes the state-of-the-art (SoTA) by over 6% F1 on supervised setup. More strikingly, THOR+GPT3 (175B) boosts the SoTA by over 50% F1 on zero-shot setting. Our code is open at https://github.com/scofield7419/THOR-ISA.
1 Introduction
Implicit sentiment analysis is difficult because texts provide factual descriptions without explicit opinion cues, requiring common-sense and multi-hop reasoning to recover hidden sentiment contexts. THOR addresses this challenge with three CoT prompts that infer the aspect, opinion, and final polarity step by step.
- Challenge: Implicit sentiment analysis must infer sentiment from factual descriptions that lack explicit opinion expressions, unlike mainstream explicit sentiment analysis.For example, existing classifiers may predict neutral for “Try the tandoori salmon!” despite humans inferring the intended sentiment.
- Challenge: Traditional sentiment-analysis methods are ineffective for ISA when they do not model how sentiment is aroused behind the text.
- Motivation: Accurate ISA requires discovering hidden opinion contexts, including latent aspects and opinions, before predicting the final polarity.The paper illustrates this with the latent aspect “taste” and opinion “good and worth trying.”
- Motivation: Common-sense reasoning identifies what a described target is, while multi-hop reasoning infers its aspect and then its underlying opinion.
- THOR framework: THOR uses three prompts to infer the target’s fine-grained aspect, underlying opinion, and final sentiment polarity in sequence.The framework uses easy-to-hard incremental reasoning to elicit hidden sentiment context step by step.
- THOR framework: Self-consistency selects intermediate aspect and opinion answers with high voting consistency, while reasoning revising uses supervised labels to improve intermediate reasoning.
- Contribution: The paper presents THOR as a multi-hop reasoning solution that improves over traditional non-reasoning methods and can extend to similar NLP problems.
2 Three-hop Reasoning Framework
THOR replaces direct polarity prediction with three prompted reasoning hops that infer an aspect, its implicit opinion, and the final sentiment polarity. It also uses self-consistency and, when supervision is available, reasoning revising to improve intermediate reasoning.
- Three-hop prompting: THOR prompts an LLM to infer the target’s aspect, the underlying opinion toward that aspect, and finally the sentiment polarity.The three hops are represented as A=argmax p(a|X,t), O=argmax p(o|X,t,a), and ŷ=argmax p(y|X,t,a,o).
- Three-hop prompting: The first hop asks which specific aspect of the target is mentioned in the sentence.Its output explicitly mentions the inferred aspect a.
- Three-hop prompting: The second hop uses the sentence, target, and inferred aspect to elicit the implicit opinion and its rationale.The resulting opinion answer becomes context for the final hop.
- Three-hop prompting: The third hop uses the complete sentiment skeleton to infer the final polarity toward the target.The context includes the sentence, target, aspect, and opinion.
- Reasoning consistency: Self-consistency generates multiple answers at each reasoning step and retains the highest-voting aspect, opinion, or polarity for the next step.This mechanism is used to consolidate reasoning correctness.
- Supervised reasoning revising: With supervised fine-tuning, reasoning revising feeds each intermediate answer with the final question to teach more correct reasoning from gold labels.The method predicts the sentiment label at each step instead of proceeding directly to the next reasoning step.
3 Experiments
Experiments evaluate THOR across supervised and zero-shot settings, model scales, GPT3 versus ChatGPT, and failure types on implicit sentiment datasets. THOR yields larger gains with larger language models and substantially improves ISA performance, especially in zero-shot GPT3.
- Experimental setup: Experiments use SemEval14 Laptop and Restaurant datasets, with Flan-T5, GPT3, and ChatGPT evaluated across supervised and zero-shot settings.Flan-T5 models range from 250M to 11B parameters, while GPT3 models range from 350M to 175B.
- Supervised fine-tuning: 7.45% F1 on Restaurant and 5.84% F1 on Laptop are the supervised ISA gains of Flan-T5-11B with THOR over the SoTA, averaging 6.65%.Ablations also indicate the importance of self-consistency and reasoning revising.
- Zero-shot reasoning: 51.94% F1 on Restaurant and 50.27% F1 on Laptop are the zero-shot SoTA gains of GPT3-175B with THOR, averaging a 51.10% F1 leap.Flan-T5-11B THOR also gives over 30% average F1 improvement over the best-performing baseline across two datasets.
- Model scale: Increasing LLM scale exponentially amplifies the efficacy of THOR’s multi-hop reasoning prompting.The paper attributes this pattern to stronger common-sense and multi-hop reasoning capabilities in sufficiently large LLMs.
- GPT3 and ChatGPT: THOR provides limited enhancement on ESA but considerably improves GPT3 and ChatGPT on ISA, where prompting-based systems still fail substantially.The comparison uses 50 randomly selected ESA and 50 ISA instances for each model.
- Failure analysis: 48.27% error rate for Flan-T5-11B in zero-shot falls to 12.79% with supervision, while unsupervised GPT3-175B has failures similar to supervised T5.Supervised T5 failures are mainly linked to reasoning incapability, whereas unsupervised GPT3 failures mainly involve problematic annotation.
4 Related Work
Related work frames ISA as a harder sentiment-analysis setting because opinion cues are hidden, and connects this paper to fine-grained sentiment decomposition and CoT prompting with large language models.
- Implicit sentiment analysis: ISA is more difficult than ESA because inputs contain factual descriptions without explicit opinion expressions, despite its prevalence in realistic scenarios.Traditional sentiment-analysis methods therefore struggle when the opinion characteristics are hidden.
- Fine-grained sentiment analysis: Fine-grained sentiment analysis decomposes sentiment into target, aspect, opinion, and polarity, and this work applies that stepwise reasoning spirit to ISA.The intended process infers sentiment elements incrementally before determining polarity.
- Chain-of-thought prompting: CoT prompting induces large language models to produce intermediate reasoning steps leading to a final answer, building on their demonstrated common-sense and multi-hop reasoning abilities.The paper implements implicit sentiment reasoning with language models using this gradient-free prompting approach.
5 Conclusion
The paper presents THOR as a three-step prompting framework for ISA, inferring aspect, opinion, and polarity. Across supervised and zero-shot settings, THOR outperforms existing baselines, with larger LLMs producing greater improvements.
- Framework: THOR uses three prompts to infer the fine-grained aspect, underlying opinion, and final polarity for implicit sentiment analysis.The framework applies chain-of-thought reasoning to ISA using existing language models.
- Conclusion: Across ISA datasets, LLMs equipped with THOR outperform the best-performing baselines in both supervised and zero-shot setups.The paper also reports that larger LLMs yield more significant improvements from THOR.
Limitations
THOR’s gains depend on model scale: it unleashes LLM capabilities only in sufficiently large models, while improvements are limited on middle- or lower-size models.
- THOR’s benefits are strongest when integrated into sufficiently large language models.The paper attributes this scale dependence to the emergence nature of LLMs.
- On middle- or lower-size language models, THOR’s improvement is limited to some extent.
A Appendix
The appendix compares THOR with vanilla prompting and zero-shot CoT on two testing cases. In both cases, THOR induces ChatGPT to reach the correct sentiment decision while the other methods fail.
- Testing setup: The appendix compares THOR with vanilla prompting and zero-shot CoT using ChatGPT testing examples.Zero-shot CoT is implemented by appending “Lets think step by step.”
- Figures: Figures 6–8 document case-I results, while Figures 9–11 document case-II results across the three prompting methods.
- Case-I: In case-I, the gold sentiment label is positive toward the metro station, and THOR succeeds where the other methods fail.The input describes walking downstairs to reach a metro station below the hotel.
- Case-II: In case-II, the gold sentiment label is positive toward Lunch, and THOR produces the correct polarity while the other methods fail.The case-II input states that lunch came with pickles and slaw at no extra charge.