PaperlayerView exact revision on arXiv

Source-linked AI summary

Stance Detection with Collaborative Role-Infused LLM-Based Agents

Xiaochong Lan, Chen Gao, Depeng Jin, Yong Li

arXiv:2310.10467v2cs.CLcs.AI

TL;DR

Stance detection challenges LLMs to interpret multiple kinds of text cues and infer viewpoints that may be implicit. COLA organizes LLM agents into analysis, debate, and conclusion stages. The authors report superior performance over existing baselines, including a 19.2% absolute improvement over the best non-LLM zero-shot baseline on SEM16.

  • Problem

    Direct LLM stance detection must handle multi-aspect knowledge and infer implicit viewpoints, while collecting large labeled datasets is difficult.

    Source: Introduction · p. 1

  • Method

    COLA organizes multiple LLM agents into text-analysis, stance-specific debating, and stance-conclusion stages.

    Source: Introduction · p. 1, Conclusion and Future Work · p. 10

  • Results

    COLA reports superior performance over existing baselines, including a 19.2% absolute improvement over the best non-LLM zero-shot stance detection baseline on SEM16.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The authors report usability without extra annotated data or model training, explainability, and versatility across text-classification tasks.

    Source: Introduction · p. 2

    Without real-time training data for LLMs, performance on real-time topics might be slightly compromised.

    Source: Conclusion and Future Work · p. 10

Abstract

from arXiv · show

Stance detection automatically detects the stance in a text towards a target, vital for content analysis in web and social media research. Despite their promising capabilities, LLMs encounter challenges when directly applied to stance detection. First, stance detection demands multi-aspect knowledge, from deciphering event-related terminologies to understanding the expression styles in social media platforms. Second, stance detection requires advanced reasoning to infer authors' implicit viewpoints, as stance are often subtly embedded rather than overtly stated in the text. To address these challenges, we design a three-stage framework COLA (short for Collaborative rOle-infused LLM-based Agents) in which LLMs are designated distinct roles, creating a collaborative system where each role contributes uniquely. Initially, in the multidimensional text analysis stage, we configure the LLMs to act as a linguistic expert, a domain specialist, and a social media veteran to get a multifaceted analysis of texts, thus overcoming the first challenge. Next, in the reasoning-enhanced debating stage, for each potential stance, we designate a specific LLM-based agent to advocate for it, guiding the LLM to detect logical connections between text features and stance, tackling the second challenge. Finally, in the stance conclusion stage, a final decision maker agent consolidates prior insights to determine the stance. Our approach avoids extra annotated data and model training and is highly usable. We achieve state-of-the-art performance across multiple datasets. Ablation studies validate the effectiveness of each design role in handling stance detection. Further experiments have demonstrated the explainability and the versatility of our approach. Our approach excels in usability, accuracy, effectiveness, explainability and versatility, highlighting its value.

Introduction

Stance detection challenges LLMs to interpret varied linguistic and domain cues and reason about viewpoints that are often implicit. COLA assigns distinct roles to LLM agents to analyze text, debate possible stances, and reach a conclusion, without additional annotated data or model training.

Introduction

  • Stance detection requires multi-aspect knowledge and reasoning because social-media texts contain specialized cues and often imply rather than state their authors’ viewpoints.Relevant cues include domain-specific terms, cultural references, platform language, and attitudes toward related topics or events.Source: Introduction · p. 1
  • COLA uses linguistic, domain, and social-media analyses, stance-specific advocates, and a final decision stage to connect text features with a stance.The approach does not require annotated data or additional model training.Source: Introduction · pp. 1–2
  • The authors report that ablations support each module’s effectiveness and that experiments and case studies show explainability and versatility across text-classification tasks.They also describe the approach as highly usable and report superior performance over existing baselines.Source: Introduction · p. 2
  • 19.2% absolute improvement over the best non-LLM zero-shot stance detection baseline on SEM16 is reported for COLA.The authors also report that it requires no annotated data or additional model training.Source: Introduction · p. 2

Related Work

Prior stance-detection approaches include feature-based, deep-learning, cross-target, and zero-shot methods, but the latter approaches still rely on annotated training data. The authors position COLA as a multi-agent approach tailored to stance-specific knowledge and reasoning challenges.

Related Work

  • Earlier stance-detection work spans feature-based and deep-learning methods, alongside cross-target and zero-shot approaches that still require annotated datasets.The authors contrast these methods with COLA’s use of a pretrained LLM without additional annotated data or training.Source: Related Work · p. 2
  • The authors report that existing LLM stance-detection methods use relatively straightforward approaches and, in their replications, often underperform annotated-data-dependent baselines.They present COLA as tailored to the expert knowledge and reasoning stance detection often requires.Source: Related Work · p. 3
  • Multi-agent LLM research includes role-based collaboration and debate, which prior work has applied to software development, medical summarization, and text evaluation.The cited examples include METAGPT, DERA, ChatEval, and debate-based work by Du et al.Source: Related Work · p. 3
  • The authors describe their work as the pioneering use of multiple LLM-based agents for stance detection.They frame COLA as a response to limitations in direct LLM applications to stance detection.Source: Related Work · p. 3

Methods

COLA frames stance detection as assigning favor, against, or neutral to a document relative to a target. Its stages analyze text from distinct perspectives, have stance-specific agents argue from those analyses, and let a judge decide from the text and arguments.

Task Description and Model Overview

  • Stance detection assigns a document-target pair one of three labels: favor, against, or neutral.The overview describes a three-stage process: multidimensional analysis, stance-specific debate, and a final judge.Source: Task Description and Model Overview · p. 3

Multidimensional Text Analysis Stage

  • The multidimensional analysis stage uses linguistic, domain, and social-media perspectives to interpret cues that direct LLM application may miss.The perspectives cover grammatical and rhetorical features, domain entities and events, and social-media expressions such as hashtags, slang, and emotional tone.Source: Approach · pp. 3–4, Challenge · p. 3

Reasoning-Enhanced Debating Stage

  • The debating stage assigns an agent to each possible stance, asking it to use the expert analyses as evidence and argue for its assigned position.The framework uses a single debate round; the authors say the stance-aligned arguments encourage divergent thinking and establish links between text features and stance.Source: Reasoning-Enhanced Debating Stage · p. 5

Stance Conclusion Stage

  • The conclusion stage gives a judge the text and debaters’ evidence and reasoning, then outputs the identified stance.The judge evaluates the text’s inherent qualities alongside the arguments and their logical frameworks.Source: Stance Conclusion Stage · p. 5

Experiments

The experiments describe three stance-detection datasets, their evaluation conditions, the implementation and metrics, and comparisons with zero-shot and in-target methods. The supplied experimental passages provide setup details rather than comparative outcome values.

Datasets

  • The experiments use SEM16, P-Stance, and VAST, with different target and label configurations.SEM16 covers six targets and three stance categories; P-Stance covers three political targets with favor and against labels, while VAST includes varying targets and pro, con, or neutral labels.Source: Datasets · p. 5
  • SEM16 and P-Stance are evaluated on their test sets, while VAST is evaluated under a zero-shot condition.For fair comparison with LLM baselines, the authors sample the test set to reproduce those baselines under their prompts.Source: Datasets · p. 6

Implementation Details

Comparison Methods

  • The comparison includes zero-shot stance-detection methods and in-target methods trained for a given target.The in-target comparison covers SEM16 and P-Stance; the authors explain that VAST targets are mainly few-shot or zero-shot.Source: Comparison Methods · p. 6, Datasets · p. 6

Results and Discussions

COLA outperforms zero-shot stance-detection baselines, performs comparably to or better than in-target trained methods, and improves on direct LLM prompting. Ablations, explanation studies, and evaluations on other classification tasks support the reported effectiveness, explainability, and versatility.

Overall Performance (RQ1)

  • 16.9% and 26.6% absolute Favg gains on SEM16 CC and LA, and a 0.7% overall Macro-F1 gain on VAST, accompany COLA’s zero-shot gains over baselines.The authors report improvements over state-of-the-art zero-shot approaches across all metrics and statistically significant improvements on most metrics across three datasets.Source: Overall Performance (RQ1) · p. 6, Table 2 · p. 7
  • COLA’s zero-shot performance closely aligns with trained in-target baselines; it beats TPDG on SEM16 HC and CC and all baselines on P-Stance.The comparison covers SEM16 and P-Stance in-target stance detection.Source: Overall Performance (RQ1) · pp. 6–7, Table 3 · p. 7
  • Direct GPT-3.5 prompting scores only 9.1% Favg on SEM16 A and 31.1% on CC; even GPT-3.5+COT reaches only 13.3% and 34.0%.The paper reports that direct LLM application significantly lags behind COLA across almost all datasets and metrics, especially for abstract targets.Source: Overall Performance (RQ1) · p. 7
  • COLA exceeds both Flan-UL2 and ChatGLM2-6B on SEM16; with ChatGLM2-6B, it gains 23.9% Favg on CC and 13.4% on FM.Flan-UL2 outperforms GPT-3.5 Turbo, while ChatGLM2-6B underperforms it.Source: Overall Performance (RQ1) · p. 7, Table 4 · p. 7

Ablation Study (RQ2)

  • On SEM16 LA, removing the Linguistic Expert, Domain Specialist, or Social Media Veteran lowers Favg to 68.9%, 67.9%, or 64.1%; removing the full analysis stage lowers it to 63.8%.Removing any individual expert degrades performance in all reported cases except FM on SEM16; removing the whole stage causes a significant drop.Source: Ablation Study (RQ2) · p. 7, Study on multidimensional text… · p. 8, Table 5 · p. 8
  • Removing debate causes larger performance losses than removing text analysis, with Favg declines of 31.2% on LA, 14.1% on CC, and 11.2% on A.The authors report that debate is especially beneficial for relatively abstract targets.Source: Study on reasoning-enhanced debating… · p. 8

Study on Explainablity (RQ3)

  • In SEM16 explanation experiments, adding COLA-generated explanations improves GPT-3.5 Turbo Favg by 51.6 points on A and 29.3 on CC; HC and FM scores exceed COLA’s.The evaluation used the whole SEM16 dataset, rather than only its test set; case studies also illustrate explanations drawing on linguistic, domain, and social-media cues.Source: Case Studies · pp. 8–9, Table 6 · p. 8, Quantitative Experiments · p. 9

Study on Versatility (RQ4)

Conclusion and Future Work

COLA uses collaborating LLM-based agents in three stages for stance detection. The conclusion reports high accuracy, effectiveness, explainability, and versatility, while noting a boundary for real-time topics.

  • COLA combines multiple LLM-based agents through multidimensional text analysis, reasoning-enhanced debate, and stance conclusion.Source: Conclusion and Future Work · p. 10
  • Experiments demonstrate high accuracy, effectiveness, explainability, and versatility for COLA.Source: Conclusion and Future Work · p. 10
  • Without real-time training data, analysis of current topics might be slightly compromised; future work proposes a real-time knowledge base and broader web and social-media text analysis.Source: Conclusion and Future Work · p. 10

Ethics Statement

The ethics statement discusses dataset and service use, computational costs, and potential misuse of stance-detection technology.

  • The datasets are open-access and contain no personally identifiable information, though they may include offensive content.Source: Ethics Statement · p. 10
  • The study uses the GPT-3.5 Turbo API in accordance with OpenAI’s terms and policies.Source: Ethics Statement · p. 10
  • Closed-source LLMs entail significant financial costs, and running LLMs requires substantial energy.Source: Ethics Statement · p. 10
  • The authors identify risks of using the technology to silence critics or target dissenting voices on social media.Source: Ethics Statement · p. 10

Paper Checklist

The checklist affirms that the paper’s claims and scope are accurately represented and reports steps concerning reproducibility, repeated experiments, and statistical testing.

  • The checklist affirms that the paper’s main claims accurately reflect its contributions and scope.Source: Paper Checklist · p. 13
  • The checklist points to the experimental setup for reproducing the main results and states that code for COLA is provided.Source: Paper Checklist · p. 13
  • The authors report multiple repeated experiments and use paired t-tests when claiming their method outperformed the best baseline.Source: Paper Checklist · p. 13

Related papers

Loading 2310.10467v2…