Can Decision Models Understand Stance? Evaluating Jev Against General-Purpose LLMs

1 Introduction

• Stance detection distinguishes favor, against, and neutral toward a target, unlike sentiment classification, which need not determine whether an attitude is target-directed.In conversations, a reply can express agreement or disagreement with an earlier utterance, so its stance may not be clear from the reply alone.Source: §1 · p. 1
• Jev’s Choice interface selects from predefined options, making its structured decision format a fit for stance detection’s fixed label set.Whether it can identify stance accurately, especially when conversational judgments depend on earlier utterances, remains the question motivating the evaluation.Source: §1 · p. 1
• The evaluation covers English individual texts in VAST and Chinese conversations in ZS-CSD, comparing Jev with general-purpose LLMs and fine-tuned reference models.The compared general-purpose models are DeepSeek V4.1 Flash, Qwen3.8-Flash, GPT-5.6, and Claude Sonnet 5; the supervised references are LoRA-fine-tuned Qwen2.5-3B and Qwen2.5-7B.Source: §1 · p. 2
• Jev matches GPT-5.6 and outperforms the other compared general-purpose LLMs on VAST, but falls behind stronger LLMs on ZS-CSD.The authors suggest the conversational limitation may involve understanding reply relationships and stance direction rather than conversation length alone.Source: §1 · p. 2

2.1 Stance detection

• Cross-target and zero-shot approaches transfer stance knowledge to unseen targets, while VAST uses generalized topic representations and JointCL transfers stance features.Earlier in-target methods model target-text relationships; CrossNet extracts shared domain features through self-attention.Source: §2.1 · p. 2

2.2 Conversational stance detection

• Conversational stance methods model reply types and dialogue dependencies, and ZS-CSD combines speaker interaction with target-aware contrastive learning for unseen targets.These approaches address both interpreting a speaker’s stance in context and generalizing to unfamiliar targets.Source: §2.2 · p. 2

2.3 Evaluations of Jev

• Prior Jev evaluations study domain-specific judgments; this work examines decisions conditioned on a target and, for conversations, the history needed to interpret an utterance.Source: §2.3 · pp. 2–3

3 Method

• Each Jev Choice request combines one record’s structured input, a task instruction, and written criteria for the stance labels.Source: §3 · p. 3

3.1.1 Stance detection

• For individual-text stance detection, the method identifies an author’s stance toward a target as favor, against, or neutral.Favor and against indicate support for and opposition to the target; neutral follows benchmark annotations, including no expressed stance or text unrelated to the target.Source: §3.1.1 · p. 3

3.1.2 Conversational stance detection

• Conversational stance detection predicts the current utterance’s stance toward a target, using preceding dialogue and speaker information as context.Source: §3.1.2 · p. 3

3.2 Structured input representation

• Structured inputs represent the focus item, its history, and target type; ZS-CSD retains ordered turns and speaker identifiers and marks the last turn as the focus.VAST uses the text as its focus item with empty history and target type; ZS-CSD uses the current speaker-utterance pair and its history.Source: §3.2 · p. 3

3.3 Label-specific selection criteria

• Each label is paired with a written criterion: favor means support or approval, against means opposition or criticism, and neutral includes no discernible stance, unrelated or informational text, and ambiguity.Both datasets use these criteria, and no in-context demonstrations are supplied.Source: §3.3 · p. 3

3.4 Prediction from the Choice response

• The selected option identifier from Jev Choice becomes the prediction; the client validates it as a stance label and uses it directly without probability-based adjustment.Each request classifies one record without task-specific fine-tuning.Source: §3.4 · p. 3

4.1 Datasets and evaluation metrics

• 3,006 VAST test records cover seen- and unseen-target subsets, while 2,584 ZS-CSD records cover unseen noun-phrase and claim targets.VAST retains repeated records; ZS-CSD target strings are disjoint across training, development, and test splits.Source: §4.1 · p. 3

4.3 Experimental settings

• Table 1 reports full-test performance and resource use, Table 2 reports per-class F1, and Figure 1 shows Jev-minus-LLM macro-F1 differences with paired target-bootstrap intervals.Recorded resource measures include input and output tokens and cost for retained successful requests or completed batches.Source: Table 1 · p. 4, Table 2 · p. 5, Figure 1 · p. 5

4.1 Datasets and evaluation metrics

• The evaluation reports accuracy and three-class macro-F1, with paired target-bootstrap intervals based on 5,000 resamples and 95% percentile bounds.Each sampled target contributes all its records to both models; the intervals use 759 VAST and 45 ZS-CSD target clusters and have no multiple-comparison adjustment.Source: §4.1 · p. 4

4.2 Models and inputs

• Prompted systems receive the target and focus text, plus speaker-tagged history for ZS-CSD; Jev uses its Choice procedure and LLMs receive shorter prose label definitions.Here, prompted means no task-specific fine-tuning or in-context demonstrations.Source: §4.2 · p. 4
• DeepSeek V4.1 Flash and Qwen3.8-Flash use temperature zero, a 32-token output limit, and disabled thinking, while GPT-5.6 and Claude Sonnet 5 classify batches independently.GPT-5.6 receives up to 100 records per request and Claude Sonnet 5 up to 25; the flash models use schema-constrained or JSON output.Source: §4.2 · p. 4
• Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct are LoRA fine-tuned on each training split as supervised references, but only single-run scores are available.Paired and subgroup analyses therefore cover the five prompted systems.Source: §4.2 · p. 4

5.1 Main Results

• On VAST, Jev reaches 77.92% macro-F1, 0.16 points below GPT-5.6, while on ZS-CSD it scores 58.43%, 15.19 and 13.99 points behind GPT-5.6 and Claude Sonnet 5.The VAST difference from GPT-5.6 is not statistically significant; the ZS-CSD differences are supported by paired confidence intervals.Source: §5.1 · p. 5

5.2 Class-level Performance

• On VAST, Jev is close to GPT-5.6 across stance categories; on ZS-CSD, its largest weakness is distinguishing favor from against.On ZS-CSD, GPT-5.6 exceeds Jev by more than 22 points on favor and against, while their neutral F1 scores are 58.67% and 59.04%.Source: §5.2 · pp. 5–6

5.3 Performance across Target Types

• All five models score higher on noun-phrase than claim targets; Jev’s macro-F1 falls from 63.49% to 55.26%, yet remains behind GPT-5.6 on both types.The Jev–GPT-5.6 gaps are 14.53 points on noun phrases and 15.46 points on claims, showing that the deficit spans both target types.Source: §5.3 · p. 6, Table 3 · p. 6

5.4 Performance across Conversation Lengths

• Jev’s performance gap does not grow steadily with conversation length, falling from 22.69 points at two turns to 14.20 points at six or more turns.Jev scores highest on single-turn inputs, but that group contains only 45 samples; its limitations may also involve interpreting stance relationships between speakers.Source: §5.4 · p. 6, Table 4 · p. 6

• Inputs retain their original language and omit reference labels and sample-selection metadata.The study adds no translation, retrieval, manual text editing, or few-shot examples.Source: A Prompts and Input Construction · p. 8
• Models classify the focus text’s stance toward its target, using conversation context only to interpret that focus text.The label definitions distinguish support or approval (favor), opposition or criticism (against), and no discernible stance or ambiguity (neutral).Source: A Prompts and Input Construction · p. 8
• DeepSeek V4.1 Flash uses a stance-enum response schema, whereas Qwen3.8-Flash returns a single-key JSON object whose label is client-validated.Source: A Prompts and Input Construction · p. 8
• GPT-5.6 and Claude Sonnet 5 receive batched instructions to classify records independently and return one ordered JSON prediction per input ID.Earlier turns may be used only to interpret the focus text, and batches contain test IDs and inputs without reference labels.Source: A Prompts and Input Construction · p. 8

• Thinking is disabled for all four general-purpose LLMs.Source: B Reasoning Settings · p. 8

• GPT-5.6 and Claude Sonnet 5 make far fewer favor–against errors than Jev, while their errors involving neutral are closer to Jev’s.Source: C Pairwise Confusions on ZS-CSD · p. 8, Table 5 · p. 8