PaperlayerView exact revision on arXiv

Source-linked AI summary

MoPrO-SD: Modular Prompt Optimization for Stance Detection

Meysam Sabbaghan, Nikesh Gyawali, Arman Zareian Jahromi, Sebastian Loera-Lozano, Doina Caragea, Cornelia Caragea

arXiv:2610.13386v1cs.CL

TL;DR

LLM stance detection remains difficult because opinions may be implicit, sarcastic, or context-dependent, and existing prompt optimizers are not designed around stance-specific class structure. MoPrO-SD sequentially refines functional prompt modules with LLM feedback and class-conditioned errors. Across four datasets and multiple backbones it improves over the Base prompt in many settings, while class-conditioned feedback beats pooled feedback on SemEval-2016 for four of five backbones.

  • Problem

    LLM stance detection is challenged by implicit opinions, sarcasm, and context-dependent expressions, while existing optimization frameworks are largely task-general rather than organized around stance-specific class structure.

    Source: §1 · p. 1

  • Method

    MoPrO-SD decomposes stance prompts into functional modules and sequentially refines them with LLM feedback, including class-conditioned errors for stance-label modules.

    Source: §1 · pp. 1–2

  • Results

    MoPrO-SD improves over the Base prompt in many settings, and class-conditioned feedback achieves higher mean Macro-F1 than pooled feedback for four of five SemEval-2016 backbones.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Class-conditioned feedback supports label-specific refinement, while MoPrO-SD's performance varies by model and refinement strategy.

    Source: §6 · p. 8

    Generalizability to NLP tasks beyond stance detection remains unexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Stance detection remains challenging for large language models (LLMs) due to implicit opinions, sarcasm, and context-dependent expressions of stance. Although automatic prompt optimization offers an alternative to fine-tuning without updating model parameters, existing structured approaches are not specifically designed around stance-label semantics and class-specific ambiguities. We introduce MoPrO-SD, a stance-specific prompt optimization framework that decomposes prompts into functional semantic modules and refines them sequentially using LLM-based feedback. We study three refinement strategies: instruction-only optimization (I), instruction with reasoning (I+R), and instruction with per-example reasoning (I+R+E), using class-conditioned errors for stance-label refinement. We also introduce VaccineStance, a new human-annotated benchmark for vaccine stance detection. Experiments across four datasets and five open-weight LLMs show that MoPrO-SD improves upon the human-written Base prompt in many settings. On SemEval-2016, class-conditioned feedback achieves higher mean Macro-F1 than pooled feedback for four of five backbones. Both reasoning-based strategies, I+R and I+R+E, achieve strong performance, with their relative effectiveness varying across models and datasets, while remaining competitive with recent prompt optimization methods. The MoPrO-SD code and VaccineStance dataset are available on GitHub.

1 Introduction

Stance detection remains difficult for LLMs, while prompt-optimization methods are not organized around stance-specific class structure. MoPrO-SD addresses this gap with modular refinement and class-conditioned feedback, and is evaluated across four datasets and five open-weight LLMs.

1 Introduction

  • LLM stance detection is challenged by sarcasm, implicit opinions, and context-dependent expressions.Prompt wording, demonstrations, and example ordering also affect performance, motivating automated prompt optimization.Source: §1 · p. 1
  • Existing prompt optimizers are largely task-general, despite stance prompts assigning distinct roles to guidelines, label definitions, and output constraints.Prior work also explores structured prompt components and reasoning-based stance prompting.Source: §1 · p. 1
  • Across four datasets and five open-weight LLMs, MoPrO-SD improves over the human-written Base prompt in many model–dataset settings.The evaluation compares against established prompt-optimization baselines and introduces VaccineStance, a human-annotated vaccine-stance benchmark.Source: §1 · p. 2
  • MoPrO-SD combines stance-specific modular refinement with class-conditioned error feedback across three optimization strategies.The strategies are instruction-only, instruction with reasoning, and instruction with per-example reasoning; class-conditioned errors inform stance-label modules.Source: §1 · pp. 1–2

2 Related Work

Prior work covers task-general prompt optimization, structured prompt components, and reasoning for stance detection. MoPrO-SD specializes structured optimization to stance through fixed modules, sequential refinement, and class-conditioned feedback.

2 Related Work

  • Stance-prompting studies use explicit reasoning, rationale augmentation, or instruction and decoding strategies, while supervised adaptation requires additional training.MoPrO-SD instead optimizes the stance prompt without updating the target LLM.Source: §2 · p. 2
  • Automated prompt optimization spans instruction generation, feedback, search, reinforcement learning, and evolutionary approaches.Examples include OPRO, EvoPrompt, and CAPO, alongside other instruction-generation and search methods.Source: §2 · p. 2
  • Structured prompt research optimizes components such as prompt sections, content and format, or selected demonstrations.Examples include UniPrompt and SCULPT, as well as methods for demonstration selection and organization.Source: §2 · p. 2

3 Methodology

MoPrO-SD decomposes stance prompts into six functional modules and refines them sequentially using LLM-based feedback. It compares instruction-only and two reasoning-based strategies, selecting the prompt with the highest validation Macro-F1 for test evaluation.

3 Methodology

  • The framework studies instruction-only, instruction-with-reasoning, and instruction-with-per-example-reasoning refinement.The overview depicts sequential module refinement followed by validation Macro-F1 evaluation and selection of the highest-scoring prompt for final test evaluation.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Stance Detection Prompt Modules

  • The six base-prompt modules are task definition, guidelines, favor definition, against definition, neutral definition, and output format.Guidelines cover criteria such as sarcasm and implicit stance; label modules define each stance, and the output module enforces label-only predictions.Source: §3.1 · p. 3

3.2 Proposed MoPrO-SD Framework

  • MoPrO-SD evaluates the full prompt on validation Macro-F1, then rewrites one module at a time in a fixed order.After a full pass, it re-evaluates the prompt and selects the iteration with the highest validation Macro-F1; the procedure is described as coordinate descent in prompt space.Source: §3.2 · p. 3

3.3 Prompt Optimization Strategies

  • Instruction-only optimization proposes module variants using previous variants and their validation Macro-F1 scores, without analyzing misclassified examples.Source: §3.3 · p. 3
  • I+R uses aggregated reasoning over errors: label modules receive errors matching their gold class, while guidelines use errors from all classes.Task definition and output format retain instruction-only updates.Source: §3.3 · pp. 3–4
  • I+R+E analyzes label-module errors individually, summarizes their explanations, and uses that feedback to revise the corresponding module.Task definition, guidelines, and output format use instruction-only updates; a label module stays unchanged when no relevant errors are available.Source: §3.3 · p. 4

4 Experimental Setup

The evaluation uses three existing stance benchmarks and the newly introduced VaccineStance benchmark. The datasets cover controversial targets, gun-control debates, COVID-19 measures, and vaccine-related stance.

4.1 Stance Detection Datasets

  • The benchmark suite combines SemEval-2016, GunStance, COVID-19-Stance, and the new VaccineStance benchmark.VaccineStance is human-annotated; the other benchmarks cover tweets about controversial targets, gun-control debates, and COVID-19 measures.Source: §4.1 · p. 4
  • SemEval-2016 includes Favor, Against, and None labels across controversial targets, with Donald Trump excluded for having no training instances.None is mapped to Neutral for consistency across datasets.Source: §4.1 · p. 4
  • GunStance covers Banning Guns and Regulating Guns, while COVID-19-Stance covers measures including face masks, school closures, and stay-at-home orders.Source: §4.1 · p. 4

4. VaccineStance

VaccineStance is a human-annotated, three-class vaccine stance benchmark, and the experiments evaluate prompts with five open-weight LLMs, multiple baselines, and validation-based selection.

4. VaccineStance

  • VaccineStance contains 2,000 vaccine-relevant tweets annotated as Favor, Against, or Neutral, with Fleiss’ κ = 0.778.The split is 500 training, 500 validation, and 1,000 test instances; further construction and split details are in Appendix B.Source: §4 · p. 4

4.2 Large Language Models

  • The evaluation uses Phi-4-14B, GPT-OSS-20B, Mistral-Small-3.2-24B, Gemma-3-27B, and Llama-3.3-70B.Tables abbreviate these backbones as Phi, GPT-OSS, Mistral, Gemma, and Llama.Source: §4.2 · p. 4

4.3 Baselines

  • Baselines include zero-shot and few-shot prompting, CoT and CoS variants, automated optimizers, and structured methods on SemEval-2016.The structured-method comparison uses Mistral, Gemma, and Llama.Source: §4.3 · p. 4

4.4 Implementation Details

  • Prompts are optimized without fine-tuning, with six modules processed sequentially over 10 iterations and the highest-validation-Macro-F1 prompt selected for testing.Methods use fixed dataset splits and three independent runs; validation subsampling, training-example sampling, stochastic generation, and optimization budgets can vary.Source: §4.4 · pp. 4–5
  • Unless specified otherwise, each backbone serves as both stance classifier and optimizer or reasoner; GPT-5 is also tested as an external reasoner.Frozen-module ablations keep one module fixed while optimizing the others.Source: §4.4 · p. 5

4.5 Evaluation Metrics

  • Three-class Macro-F1 is the primary metric, averaging Favor, Against, and Neutral F1 equally, unlike the official SemEval-2016 metric, which excludes Neutral.Prompt selection uses validation Macro-F1; test results are mean ± standard deviation across three runs.Source: §4.5 · p. 5

5 Results and Discussion

The results examine optimization dynamics, benchmark performance, computational cost, ablations, and error cases. MoPrO-SD often improves on the Base prompt, while outcomes vary across models, datasets, and refinement strategies.

5.1 Optimization Dynamics

  • Across 10 I+R+E iterations, most models improve early, but later gains are smaller and fluctuating, so continued optimization does not guarantee monotonic improvement.The trajectories also show run-to-run variability, with some sharp drops followed by recovery, particularly on VaccineStance.Source: §5.1 · p. 5, Figure 2 · p. 6

5.2 Main Performance Results

  • I+R+E improves over the human-written Base prompt in 18 of 20 dataset–model combinations, but I+R outperforms it in several settings.The largest I+R+E gains include Phi on GunStance (39.55 →59.74, +20.19 Macro-F1) and COVID-19-Stance (37.78 →55.12, +17.34); CAPO with Llama leads SemEval at 74.87, while I+R with Mistral leads VaccineStance at 68.96.Source: §5.2 · p. 5, Table 1 · p. 6
  • On SemEval-2016, MoPrO-SD performs comparably to recent structured prompt optimizers, with I+R+E scoring highest numerically on Llama at 74.78 Macro-F1.Its performance is similar to both UniPrompt settings on Mistral and Gemma.Source: §5.2 · pp. 5, 7, Table 2 · p. 7
  • Using GPT-5 as the external reasoner yields gains in 17 of 20 dataset–model combinations, but Mistral on SemEval declines by 5.82 Macro-F1.The reported effects vary across model–dataset settings.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Token Cost Analysis

  • On SemEval-2016 with Mistral, I, I+R, and modular I+R+E use 8.43×, 18.10×, and 25.21× the Base token budget, respectively.Modular I+R+E uses fewer tokens than UniPrompt, CAPO, and EvoPrompt variants, but more than OPRO and evaluated full-prompt variants; the selected prompt requires no additional optimization calls at inference.Source: §5.3 · p. 7, Table 4 · p. 7

5.4 Ablation Studies

  • On SemEval-2016, modular I+R+E has higher mean Macro-F1 than full-prompt optimization for four of five backbones, with the largest gain for Mistral (+3.49).Llama is nearly unchanged (+0.05), while Gemma favors full-prompt optimization (−0.85).Source: §5.4 · p. 7, Table 5 · p. 8
  • Frozen-module ablations show model-dependent sensitivities, while class-conditioned error routing yields higher mean Macro-F1 than pooled routing for four of five backbones.Freezing Favor has the largest degradation for Phi (−9.34) and Gemma (−4.50), while freezing Against has the largest effect for GPT-OSS (−6.59) and Mistral (−9.66); class-conditioned routing decreases Llama by 0.15 points.Source: §5.4 · pp. 7–8, Table 6 · p. 8, Table 7 · p. 8

5.5 Error Analysis

  • Qualitative cases illustrate corrections involving implicit stance, sarcasm, and ambiguous expressions, including a sarcastic tweet where I and I+R+E recover the gold Favor label.For that example, Base and I+R predict Against.Source: §5.5 · p. 8

6 Conclusion

MoPrO-SD sequentially refines stance-prompt components without fine-tuning, improving over the Base prompt in many settings. Its outcomes vary by model and strategy, while class-conditioned feedback and modular optimization show benefits in reported comparisons.

6 Conclusion

  • Across four datasets and multiple LLM backbones, MoPrO-SD improves over the Base prompt in many settings, although performance varies by model and refinement strategy.The framework refines prompt components without model fine-tuning.Source: §6 · p. 8
  • Class-conditioned feedback achieves higher mean Macro-F1 than pooled feedback for four of five backbones on SemEval-2016.Modular optimization also outperforms full-prompt optimization for several models, with varying module contributions; optimization costs more during development but requires no additional inference-time optimization.Source: §6 · p. 8

Limitations

The paper identifies limits in task and transfer generalizability, modular structure, optimization cost, and reproducibility. Its supplementary material also provides prompt materials and describes VaccineStance construction and split distributions.

Limitations

  • Generalization beyond stance detection remains unexplored, and transfer across backbone models or datasets may require re-optimization.The fixed stance-specific modules may not extend readily to larger label spaces or more complex reasoning; repeated validation and optimizer calls add optimization-time cost, and API model updates may affect reproducibility.Source: Limitations · p. 9

A.1 Complete Prompt Template

  • Appendix A.1 provides the complete Base prompt template for stance detection.Source: §A.1 · p. 13

6. Output Format

  • The output format requests only a Favor, Against, or Neutral classification label, with no additional text or explanation.The stated response template is “Stance: [Favor/Against/Neutral].”Source: §6 · p. 13

A.2 Meta Prompts

  • The meta-prompts ask the optimizer to diagnose incorrect predictions and produce concise prompt revisions from feedback.The supplied examples include comparing wrong predictions with ground truth and summarizing problems and improvements.Source: §A.2 · p. 14

B.1 Dataset Statistics

  • Table T1 reports target-wise train, validation, and test distributions for the four experimental datasets.Source: §B.1 · p. 15

B.2 VaccineStance Construction and Annotation

  • VaccineStance was sampled from vaccine-related social-media posts and human-annotated as Favor, Against, or Neutral, with Fleiss’ κ = 0.778.It contains 2,000 instances split into 500 training, 500 validation, and 1,000 test examples, with no exact overlap across splits.Source: §B.2 · p. 15

B.3 Sample Annotated VaccineStance Examples

The supplementary analyses report optimization-budget gains on SemEval-2016, variable effects from retrieved demonstrations, and supplementary evaluation details. VaccineStance examples include both prediction matches and disagreements with human labels.

B.3 Sample Annotated VaccineStance Examples

  • The VaccineStance examples include both correct Llama-3.3 predictions and disagreements with final human annotations.Examples include neutral agreement and mismatches where Llama-3.3 predicted neutral or against but the human label was in-favor.Source: §B.3 · p. 15

C Effect of Optimization Budget

  • Increasing the I+R+E budget from 1 to 10 iterations improves mean Macro-F1 for all five SemEval-2016 backbones, with gains varying by model.The largest gains are Mistral (+12.55 points) and Phi (+4.23); gains are significant for Mistral, Gemma, and Phi, while no significant 5-versus-10-iteration difference is detected for any backbone.Source: Figure S1 · p. 15, C Effect of Optimization Budget · pp. 15–16

C.1 Implementation Details

  • Classification uses temperature 0.2, reasoning generation uses 0.7, and few-shot evaluation adds retrieved training demonstrations to frozen I+R+E prompts.The evaluation uses k = 1 or k = 3 demonstrations per stance class.Source: §C.1 · p. 16

D Additional Optimization Trajectories

  • Figures S2 and S3 plot validation Macro-F1 trajectories over ten iterations for I and I+R across four datasets, with Base marking the original prompt score.Curves show means across three runs and shaded standard-deviation regions.Source: D Additional Optimization Trajectories · p. 16

E Detailed Dataset-Specific Results

F Few-Shot Prompting Analysis

  • Retrieved demonstrations improve Macro-F1 in several settings, particularly on GunStance and VaccineStance, but gains are smaller or reverse elsewhere.Increasing k from 1 to 3 is not consistently beneficial after I+R+E optimization; Figure S4 compares zero-shot with k = 1 and k = 3 variants.Source: F Few-Shot Prompting Analysis · p. 16

G External Reasoner Results

The supplementary material describes an external-reasoner comparison and includes figures documenting optimization trajectories and prompting variants.

G External Reasoner Results

  • Figures S2 and S3 show I and I+R validation Macro-F1 trajectories, while Figure S4 compares zero-shot and few-shot I+R+E prompts.The figures describe four-dataset evaluations; S4 includes k = 1 and k = 3 retrieved demonstrations per stance class.Source: Figure S2 · p. 17, Figure S3 · p. 17, Figure S4 · p. 17
  • The external-reasoner experiment compares I+R+E using the classifier itself as reasoner with I+R+E using GPT-5 across four datasets and five backbones.Table T6 reports the full results of the experiment summarized in the main paper.Source: G External Reasoner Results · pp. 16–17

H.1 Qualitative Examples of Frozen-Module Ablation

The frozen-module examples associate failures with implicit stance cues and neutrality-related expressions. The case studies show optimized prompts correcting Base-prompt errors, with reasoning variants resolving implicit support and neutrality.

H.1 Qualitative Examples of Frozen-Module Ablation

  • In Mistral3.2 on SemEval-2016, frozen stance-definition modules are associated with failures on implicit cues and neutrality-related expressions despite correct full I+R+E predictions.The examples cover freezing Favor, Against, and Neutral definitions.Source: §H.1 · pp. 17–18
  • The included tables report per-target Macro-F1 results for SemEval-2016, GunStance, and COVID-19-Stance, plus a comparison of GPT-5 and the target model as reasoner.Results are reported as mean ± standard deviation across three runs for the per-target tables and three independent runs for the reasoner comparison.Source: Table T3 · p. 18, Table T4 · p. 19, Table T5 · p. 20

H.2 Qualitative Case Studies

  • For an implicitly supportive environmental tweet with negative tone, I, I+R, and I+R+E predict favor, correcting Base’s against prediction.The passage attributes the Base error to reliance on surface sentiment.Source: §H.2 · p. 19
  • On a climate-change sea-level tweet, I+R and I+R+E predict favor, while Base and I predict neutral.The gold label is favor.Source: §H.2 · p. 19
  • Reasoning-enhanced variants identify an implicit supportive stance from climate-change evidence, while Base and I predict neutral.The example illustrates a case where explicit advocacy is absent.Source: §H.2 · p. 19
Loading 2610.13386v1…