PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Sentiment Classification and Irony Detection in Large Language Models through Advanced Prompt Engineering Techniques

Marvin Schmitt, Anne Schwerk, Sebastian Lempert

arXiv:2601.08302v1cs.CLcs.AI

TL;DR

The paper addresses limited evidence about how advanced prompting affects LLM performance on sentiment-analysis tasks, including nuanced tasks such as irony detection. It experimentally compares prompting strategies across GPT-4o-mini and gemini-1.5-flash, finding that performance depends on the model and task. The study concludes that prompt design should be evaluated across architectures and sentiment-analysis settings.

  • Problem

    Existing research has limited evidence isolating how advanced prompt engineering affects LLM sentiment-analysis performance, especially for subjective and context-dependent tasks.

  • Method

    The study compares baseline, few-shot, chain-of-thought, and self-consistency prompting across GPT-4o-mini and gemini-1.5-flash on classification, ABSA, and irony-detection tasks.

  • Results

    The experiments provide empirical evidence on prompt-engineering effects across sentiment analysis and irony detection, while baseline evaluations show both models struggle with neutral sentiments.

  • Takeaways & Limitations

    The paper positions targeted prompt comparisons as a way to study and improve the use of general-purpose LLMs for diverse sentiment-analysis problems.

  • Takeaways & Limitations

    Findings are model-specific to GPT-4o-mini and gemini-1.5-flash, while proprietary training data limits generalizability and deeper causal interpretation.

Abstract

from arXiv · show

This study investigates the use of prompt engineering to enhance large language models (LLMs), specifically GPT-4o-mini and gemini-1.5-flash, in sentiment analysis tasks. It evaluates advanced prompting techniques like few-shot learning, chain-of-thought prompting, and self-consistency against a baseline. Key tasks include sentiment classification, aspect-based sentiment analysis, and detecting subtle nuances such as irony. The research details the theoretical background, datasets, and methods used, assessing performance of LLMs as measured by accuracy, recall, precision, and F1 score. Findings reveal that advanced prompting significantly improves sentiment analysis, with the few-shot approach excelling in GPT-4o-mini and chain-of-thought prompting boosting irony detection in gemini-1.5-flash by up to 46%. Thus, while advanced prompting techniques overall improve performance, the fact that few-shot prompting works best for GPT-4o-mini and chain-of-thought excels in gemini-1.5-flash for irony detection suggests that prompting strategies must be tailored to both the model and the task. This highlights the importance of aligning prompt design with both the LLM's architecture and the semantic complexity of the task.

1. Introduction

The introduction frames a gap in evidence about how advanced prompt engineering affects LLM sentiment-analysis performance across models and task types. The study compares prompting strategies across standard and nuanced sentiment tasks to quantify their effects.

  • Study design: The evaluation compares GPT-4o-mini and gemini-1.5-flash using baseline, few-shot, chain-of-thought, and self-consistency prompts.
  • Research question: The study assesses prompting effects with accuracy and F1-score while considering differences across LLM architectures and sentiment-analysis tasks.
  • Study design: Experiments cover binary sentiment classification, German multilingual three-class sentiment, aspect-based sentiment analysis, and irony detection.
  • Motivation: Advanced prompting may help address sentiment-analysis tasks that require contextual reasoning beyond surface-level cues, such as aspect-based sentiment and irony detection.
  • Research question: The study examines whether prompt engineering improves LLM sentiment-analysis outcomes compared with generic or baseline prompting.

2. Related work

Related work shows that LLM sentiment performance depends strongly on prompting, input characteristics, and model adaptation. This study positions itself as a focused comparison of advanced prompting techniques across multiple sentiment tasks.

  • LLMs for sentiment analysis: Prior evaluations report that LLMs can perform strongly on aspect-based sentiment analysis with in-context demonstrations, sometimes rivaling or surpassing fine-tuned smaller models.
  • LLMs for sentiment analysis: Lightly fine-tuned medium-sized models can outperform in-context prompting on fine-grained aspect-based sentiment tasks.
  • Prompt sensitivity: Prompt wording, structure, and generation parameters can substantially change ChatGPT performance on sentiment, toxicity, and sarcasm detection.
  • Prompting for nuanced sentiment: Sarcasm-focused prompting frameworks and few-shot input/output prompting have been reported to improve sarcasm-detection performance relative to simpler prompting approaches.
  • Positioning of this work: The paper builds on prior broad evaluations by experimentally comparing few-shot, chain-of-thought, and self-consistency prompting across multiple sentiment tasks.

3. Background

The background defines prompts and prompt engineering while situating sentiment analysis as a broad field spanning polarity, aspect-level sentiment, and nuanced interpretation. It emphasizes that different tasks require different levels of contextual understanding.

  • Prompting: A prompt is an instruction, possibly accompanied by contextual information, given to an LLM to elicit a particular response.
  • Prompting: Prompts can include instructions, input data, context, output directives, style instructions, and supplementary information.
  • Prompt engineering: Prompt engineering concerns designing and conceptualizing prompts so that formulation rules and nuances guide model outputs toward higher performance and quality.
  • Sentiment analysis: Sentiment analysis includes binary classification, multiclass classification, aspect-based analysis, irony detection, temporal analysis, domain adaptation, and fine-grained opinion intensity.
  • Sentiment-analysis tasks: Classical sentiment classification assigns predefined categories, ABSA targets specific aspects, and MAST includes complex tasks such as irony detection requiring contextual interpretation.

4. Experimental design

The experimental design systematically compares advanced prompting strategies with a baseline across three sentiment-analysis areas and two LLMs. It uses multiple datasets, standardized metrics, language-matched prompts, and sampled evaluations.

  • Prompting strategies: The study compares baseline, few-shot, chain-of-thought, and self-consistency prompting to test their effects on LLM sentiment-analysis performance.
  • Hypotheses: The hypotheses predict improvements in classification, aspect-based sentiment detection, and irony detection from precisely designed or advanced prompts.
  • Evaluation procedure: Each dataset contributes 1,000 randomly sampled entries, and bootstrap resampling assesses whether differences between prompting strategies are statistically significant.
  • Evaluation procedure: Prompts are written in the same language as each dataset to reduce misinterpretations caused by language mismatches.
  • Datasets and tasks: The evaluation uses sentiment classification, ABSA, and irony detection datasets, including binary, multiclass, German-language, aspect-level, and ironic-tweet tasks.
  • Evaluation procedure: Performance is measured with accuracy, precision, recall, and F1-score, with attention to class-specific performance and systematic misclassifications.

5. Results

Results show that prompting effects vary by model and task: few-shot prompting produced the strongest broad gains for GPT-4o-mini, while CoT particularly improved gemini-1.5-flash irony detection. Neutral-class examples also reduced polarized prediction bias, whereas self-consistency introduced a cost and, in one setting, performance drawback.

  • 5.1. Shot prompting: 0.37 to 0.51: gemini-1.5-flash neutral-class recall increased on SB10k after adding a neutral one-shot example.The exemplar encouraged more neutral predictions and mitigated the models’ tendency toward polarized sentiment labels.
  • 5.1. Shot prompting: Few-shot prompting achieved the best performance across most subtasks, with GPT-4o-mini improving SB10k accuracy and weighted F1 by approximately 10 percentage points over baseline.The gains affected all three sentiment classes rather than only the neutral class.
  • 5.1. Shot prompting: GPT-4o-mini benefited more from few-shot prompting than gemini-1.5-flash, whose SB10k F1-score increased only marginally from 0.68 to 0.69.Most few-shot improvements were statistically significant, except for gemini-1.5-flash ABSA and GPT-4o-mini irony detection.
  • 5.2. Chain-of-thought: CoT slightly reduced GPT-4o-mini accuracy and F1-score from baseline without statistically significant differences, while one analyzed CoT example produced a plausible but incorrect irony rationale.The baseline correctly classified that example as no irony.
  • 5.2. Chain-of-thought: Gemini-1.5-flash achieved 0.95 accuracy and F1-score on SST2 with CoT, alongside statistically significant irony-detection gains.The improvement was notable because the model had generally performed poorly on irony detection before CoT.
  • 5.2. Chain-of-thought: For gemini-1.5-flash irony detection, CoT improved negative-class recall from 0.06 under baseline to 0.38.The approach shifted predictions away from the baseline tendency to classify almost all texts as ironic, and negative-class performance was critical.
  • 5.3. Self-consistency: Self-consistency was the only approach to decrease performance against baseline for GPT-4o-mini on SST2, with pronounced false negatives and low negative-class precision.Despite majority voting, it frequently returned an incorrect outcome for the same classification task.
  • 5.3. Self-consistency: Self-consistency increases monetary and temporal costs because each query is iterated n times.The method also involves an interaction between iteration frequency and temperature that affects response variability.

6. Synthesis of findings and answering the research question

The study finds that advanced prompting improves sentiment-analysis performance, but effectiveness varies by task, prompting strategy, and model architecture. Few-shot prompting is most reliable for standard classification, whereas CoT produces a strong irony-detection gain only for gemini-1.5-flash.

  • 6.1. H1: Prompt engineering boosts LLM accuracy in sentiment classification: Most advanced prompting techniques significantly improved binary and ternary sentiment classification over the zero-shot baseline.
  • 6.1. H1: Prompt engineering boosts LLM accuracy in sentiment classification: Few-shot prompting was the most reliable strategy, especially for GPT-4o-mini, improving accuracy and weighted F1-score while helping reduce polarity bias.
  • 6.2. H2: Targeted prompts enhance aspect-based sentiment detection in LLMs: ABSA gains were smaller than standard-classification gains, with zero-shot-CoT and self-consistency often failing to produce significant improvements.
  • 6.3. H3: Advanced prompt techniques sharpen LLM detection of ironic tweets: Under GPT-4o-mini, CoT and self-consistency degraded irony performance, while gemini-1.5-flash achieved a 46% increase in weighted F1-score with CoT.
  • 6.4. Synthesis of findings: Overall, prompt success depended on the interaction between prompt structure, task complexity, and model architecture rather than following a universal strategy.

7. Conclusion and future work

The study concludes that prompt engineering can improve LLM sentiment analysis, while its benefits remain model- and task-dependent. It recommends broader evaluation, adaptive prompt optimization, and deeper linguistic analysis to improve reliability and interpretability.

  • 7.1. Key insights: Few-shot prompting produced robust accuracy and F1-score improvements, while CoT and self-consistency showed notable model-dependent limitations.
  • 7.1. Key insights: Prompting strategies cannot be transferred across models without validation because performance diverged between GPT-4o-mini and gemini-1.5-flash.
  • 7.1. Key insights: The study’s findings support model-aware and task-specific prompt design for more reliable, transparent, and context-aware affective computing systems.
  • 7.2. Limitations: The evidence is limited to two proprietary models, downsampled datasets, manually designed prompts, fixed decoding parameters, and no detailed linguistic error analysis or multiple-hypothesis control.
  • 7.3. Directions for further research: Future work should compare architectures, test task-specific prompting, explore retrieval-augmented prompting, and assess robustness and self-consistency more systematically.

CRediT authorship contribution statement

The authors’ contributions span conceptualization, methodology, software, validation, analysis, investigation, writing, visualization, and supervision.

  • Marvin Schmitt contributed to conceptualization, methodology, software, validation, formal analysis, investigation, writing, and visualization.
  • Anne Schwerk contributed to conceptualization, writing, and supervision.
  • Sebastian Lempert contributed to conceptualization, writing, and supervision.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors used DeepL, GPT-4, and the Stanford Agentic Reviewer during manuscript preparation, with revisions reviewed and approved by the authors. The paper also states its conflict-of-interest, code, and data-availability information.

  • DeepL and GPT-4 were used to improve English-language quality because none of the authors are native English speakers.
  • The Stanford Agentic Reviewer provided feedback on clarity, methodology, and presentation before submission, and the authors reviewed and approved revisions.
  • The authors declared no known competing financial interests or personal relationships that could influence the reported work.
  • The code is available in the cited GitHub repository, while data will be made available on request.

Appendix A. All Prompts

Appendix A catalogs one-shot, few-shot, CoT, zero-shot CoT, and self-consistency prompting configurations, while Appendix B describes bootstrap-based significance testing for sentiment classification, ABSA, and irony.

  • A.1. One shot prompting: Table A.1 documents the one-shot prompting configuration.
  • A.2. Few shot prompting: Table A.2 documents the few-shot prompting configuration.
  • A.3. CoT prompting: Table A.3 documents the chain-of-thought prompting configuration.
  • A.4. Zero-shot CoT prompting: Table A.4 documents the zero-shot chain-of-thought prompting configuration.
  • A.5. Self consistency prompting: Table A.5 documents the self-consistency prompting configuration.
  • Appendix B. Statistical testing (Bootstrap): Bootstrap testing evaluates F1-score differences with 95% confidence intervals rather than p-values, treating intervals excluding zero as statistically significant.
  • Appendix B. Statistical testing (Bootstrap): Tables B.1 and B.2 classify comparisons as baseline-superior, prompt-superior, or not significantly different.

Appendix C. All results

Appendix C provides complete result tables for sentiment classification, aspect-based sentiment analysis, and irony detection, alongside confusion-matrix figures for one-shot performance.

  • Sentiment classification: Table C.1 contains all sentiment-classification results.
  • Aspect-based sentiment analysis: Table C.2 contains all aspect-based sentiment-analysis results.
  • Irony detection: Table C.3 contains all irony-detection results.
  • Irony detection: Figure D.1 presents the baseline-versus-one-shot confusion matrix for gemini-flash1.5 irony detection.
  • Irony detection: One-shot prompting improves detection of the no-irony class even when irony is the single positive-class example.
  • Sentiment classification: The confusion-matrix class labels include negative, positive, and neutral.
  • Sentiment classification: Figure D.2 presents the baseline-versus-one-shot confusion matrix for gemini-flash-1.5 on SB10k.
  • Sentiment classification: Providing one neutral-class example significantly improves neutral-class performance in one-shot prompting.

D.2. CoT prompting

The appendix compares confusion matrices for CoT irony detection and self-consistency on GPT-4o-mini, showing class-specific changes and persistent misclassification under majority voting.

  • D.2. CoT prompting: Figure D.3 compares baseline and CoT confusion matrices for gemini-1.5-flash irony detection.
  • D.2. CoT prompting: 0.06 baseline recall for the no-irony class rises to 0.38 with CoT prompting.
  • D.2. CoT prompting: Most baseline texts are falsely classified as ironic, whereas CoT improves distinction between ironic and non-ironic content.
  • D.3. Self-consistency prompting: Figure D.4 compares the best-performing few-shot and worst-performing self-consistency GPT-4o-mini approaches.
  • D.3. Self-consistency prompting: Self-consistency produces many false negatives and low precision for the negative class.
  • D.3. Self-consistency prompting: Majority voting does not prevent the model from consistently making apparently confident misclassifications.
Loading 2601.08302v1…