PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying Effective Strategies in Prompt Evolution for Stress and Emotion Prediction

SeungHyun Yi, Seungjun Yi, Sriram Sattiraju, Haeun Park, Joakim Nguyen, Terence Lim, Joseph Skrovan, Mehak Beri, Hitakshi Modi, Eloy Carrion, Andrew Well, Carlos M. Mery, Alexander Rasgon, Mia K Markey, Ying Ding

arXiv:2610.21407v1cs.CL

TL;DR

Social-media stress prediction can be obscured by false positives, while prompt revisions have shown inconsistent gains and their strategy links remain unclear. The paper compares six methods across stress and emotion tasks and two model sizes, finding leading results for GEPA variants and identifying strategies associated with development gains. Its task and evaluation observations also bound how those findings should be interpreted.

  • Problem

    Prompt revisions show inconsistent gains across models and tasks, and how instruction combinations relate to gains or declines remains unclear; accuracy can also obscure false positives.

    Source: §I · p. 1

  • Method

    The study compares six prompting methods on Dreaddit stress and GoEmotions emotion prediction across two model sizes, then traces prompt revisions and strategy combinations.

    Source: §I · pp. 1–2

  • Results

    GEPA had the highest mean Macro-F1 on Dreaddit with Qwen3-8B, and GEPA+merge on GoEmotions with Qwen3-32B; five strategies were associated with development gains.

    Source: §I · p. 2

  • Takeaways & Limitations

    Prompt evolution findings point to evaluating complete instruction combinations in context, since apparently useful revisions may reduce performance and isolated strategy effects remain unmeasured.

    Source: §VI · p. 10

    These benchmarks are not clinical cohorts and do not evaluate pre-onset detection, future disorders, or clinical validity.

    Source: A. Task Setup and Datasets · p. 3

Abstract

from arXiv · show

Early stress detection and risk prediction are crucial for timely mental health assessment and support. While large language models have recently shown promise for social media stress prediction, relying on accuracy can obscure false positives. Also, prompt revisions yield inconsistent gains across models and tasks, with unclear links to strategy combinations and instruction changes. By comparing six methods on social media stress and emotion data across two model sizes, we found that GEPA and its extension combining instructions from two prompts (GEPA+merge) improved prediction over initial prompts in several settings. We also identified five prompting strategies associated with development gains. These strategies included directing attention to emotional evidence, distinguishing current from past distress, illustrating indirect cues, requiring emotional context, and guiding label selection. Moreover, we documented four observations across tasks and evaluation settings. In a separate set of searches, binary stress prediction improved early, whereas multilabel emotion prediction initially declined and partly recovered. The other three concerned incorrect or conflicting instructions, gains on small comparison samples that did not hold on full development data, and changes in selected prompts' development and reported test rankings. These findings identify effective prompt evolution methods, strategy combinations associated with gains, and task and evaluation limits, providing guidance for applying prompt evolution methods to social media stress detection.

I. INTRODUCTION

The paper examines how prompt evolution affects social-media stress and emotion prediction, and how instruction changes relate to gains or declines. It compares six methods and analyzes strategy combinations and task- and evaluation-specific observations.

I. INTRODUCTION

  • Accuracy alone can obscure false positives, and prompt revisions have yielded inconsistent gains across models and tasks.The introduction notes that false positives accounted for 90% of errors in one initial Dreaddit prompt evaluation.Source: §I · p. 1
  • The study compares six methods on Dreaddit stress and GoEmotions emotion prediction across two model sizes, measuring accuracy, Macro-F1, and explanation support.It traces recorded instruction revisions and strategy combinations alongside development gains and declines.Source: §I · pp. 1–2
  • GEPA led on Dreaddit/Qwen3-8B at 67.81% mean Macro-F1, while GEPA+merge led on GoEmotions/Qwen3-32B at 25.14%.These were the highest mean Macro-F1 results among six evaluated methods.Source: §I · p. 2
  • Five strategies associated with development gains direct attention to emotional evidence, distinguish current from past distress, illustrate indirect cues, require emotional context, and guide label selection.The strategies occur in combinations along golden routes; complete prompts are measured, so individual strategy effects are not isolated.Source: §I · pp. 1–2
  • Four observations cover task differences, instruction errors, small-sample gains that did not hold on full development data, and selected-prompt ranking changes between development and test data.Separate searches showed early gains for binary stress prediction but an initial decline and partial recovery for multilabel emotion prediction.Source: §I · p. 2

II. RELATED WORK

The related work situates stress prediction in social-media text, prompt revision methods, and health-text interpretation. It also surveys clinical and patient-simulation contexts before describing this study’s comparisons and analyses.

A. Stress Detection in Social Media

  • Stress-detection work distinguishes an author’s expressed distress from stressful topics or others’ experiences, while this study compares prompt revisions without measuring false-positive or false-negative changes.Prior work includes stress labels on passages, user-level survey associations, emotion-informed stress detection, and LLM false positives.Source: A. Stress Detection in Social Media · p. 2

B. Prompt Revision and Health Text Interpretation

  • Prompt-revision methods vary in feedback and search, while health-text frameworks emphasize clinician oversight, traceable evidence, and validation beyond benchmark prediction.The cited methods include TextGrad, MIPROv2, GEPA, TAMA, SFT-TA, Auto-TA, and a clinical codebook framework, –.Source: B. Prompt Revision and Health Text… · p. 2

C. Clinical Context and Patient Simulation

  • The clinical-context literature spans models of suicidal ideation and behavior, therapy trials and theory, and evaluation of depression patient simulators.The cited work includes an integrated model, a randomized noninferiority trial, an exposure-therapy review, and PSI-Bench [23].Source: C. Clinical Context and Patient… · p. 2

III. METHODS

  • The study compares six methods against the initial instruction baseline, then analyzes 24 GEPA and GEPA+merge searches and supplements them with six historical searches.The analyses address strategy combinations, instruction changes, and differences across tasks and evaluation settings.Source: §III · p. 2

A. Task Setup and Datasets

The study uses English Reddit benchmarks for binary stress and multilabel emotion prediction, with fixed study samples and a test panel that limits claims about untouched-data generalization.

A. Task Setup and Datasets

  • Dreaddit labels whether a writer expresses stress in a passage, rather than whether the passage concerns a stressful topic.Neither benchmark is a clinical cohort; they do not evaluate detection before stress onset, future disorders, or clinical validity.Source: A. Task Setup and Datasets · pp. 2–3
  • GEPA retains revisions that improve on three training examples, while GEPA+merge also accepts merges scoring at least as high as the better parent on five development examples.Both evaluate accepted prompts on all 100 development examples and select the highest-scoring prompt at search end.Source: Figure 1 · p. 3
  • Each dataset supplied 100 training and 100 development items, but the test panel combined 100 historically exposed items with 100 selected from a fixed ordering.Results are exploratory and do not evaluate globally untouched data; prepared GoEmotions data lacked author or thread grouping for equivalent checks.Source: A. Task Setup and Datasets · p. 3
  • Qwen3-8B and Qwen3-32B ran two sequential instructions: an answer module generated labels and source-based explanation, then a source-review module checked and corrected the answer.The modules shared a 4,096-token budget for instructions and training demonstrations, excluding task definitions, output schemas, and the current source passage.Source: A. Task Setup and Datasets · p. 3

B. Compared Methods

The comparison includes prompt-search and prompt-evolution methods, with GEPA and GEPA+merge searches analyzed through linked instruction versions, parentage, retention decisions, and development evaluations.

B. Compared Methods

  • Baseline uses initial instructions without demonstrations or search; ICL adds valid training demonstrations, while TextGrad revises both instruction texts using textual feedback.MIPROv2 searches instruction and demonstration combinations for both modules, selecting the highest available full-development score.Source: B. Compared Methods · pp. 3–4
  • GEPA uses model reflection on task outputs and feedback; GEPA+merge adds instruction merging, while both keep model weights, task definitions, and reference data fixed.The methods retain two instruction modules and change only reusable prompt content.Source: B. Compared Methods · p. 4

C. Prompt Evolution

  • Searches link prompt revisions to parent instructions, changes, retention decisions, and evaluations; GEPA+merge records both parents for accepted combinations.Accepted prompts are selected by mean development scores and then frozen for testing; acceptance, full-development, and test scores remain distinct.Source: C. Prompt Evolution · p. 4

1) Link Search Records:

  • The 24 main-study searches saved 109 candidates, and linked records capture instruction pairs, parentage, decisions, development scores, and final selections.A separate interactive record added six Qwen3-8B searches with 135 saved candidates, though rejected revisions are unavailable and some development scores are incomplete.Source: C. Prompt Evolution · p. 4, 1) Link Search Records · p. 4

2) Code Instruction Content:

The analysis codes instruction content, compares parent–child revisions, traces selected prompt paths, and examines development-score patterns while distinguishing exploratory comparisons from full-development results.

2) Code Instruction Content:

  • Instruction coding used automated draft codes linked to source passages, without independent human coding or an agreement assessment.Seven overlapping functions cover task and decision rules, evidence and grounding, output structure, examples, verification, scope, and uncertainty; subcategories distinguish stress decisions from emotion-label selection.Source: 2) Code Instruction Content · p. 4
  • The analysis counted specified punctuation and formatting features in decoded instructions, counting retained revisions and their candidates once and excluding rejected revisions.Source spans identified fields, definitions, decision cues, and quoted labels.Source: 2) Code Instruction Content · p. 4
  • Coding records requested functions, not their correctness or evidence of reasoning, complete demonstrations, enforced decoding, or calibrated confidence.Negative wording and headings were treated as expression rather than additional functions; wording changes were separated from shared answer/review structure and GEPA operations.Source: 2) Code Instruction Content · p. 4

3) Compare Revisions and Trace Selected Paths:

  • Parent comparisons tracked additions, removals, elaborations, replacements, and refinements, while Table II linked five strategies to cases, instruction changes, gains, and counterexamples.The table reports neither strategy frequencies nor golden-route counts, and no new prevalence counts were calculated for the 24 main-study searches.Source: 3) Compare Revisions and Trace… · p. 4
  • Final test comparisons pair six methods across Dreaddit and GoEmotions and Qwen3-8B and Qwen3-32B, reporting means and standard deviations for each metric.The figure covers 200 attempted test items per seed, with five seeds except four for GoEmotions/Qwen3-32B ICL Conformability; compare gaps only within a metric.Source: Figure 2 · p. 5
  • Golden routes follow parent links from root to selected prompt, not revision-attempt order, and do not guarantee stepwise improvement or the best possible prompt.The additional searches trace 0→2→9→10 for Dreaddit and 0→1→3→15 for GoEmotions; the main-study GoEmotions/Qwen3-32B/seed 44 merge candidate 3 has two parent paths.Source: 3) Compare Revisions and Trace… · p. 5

4) Compare Development Score Patterns:

  • Development-score comparisons distinguish full-prompt scores from smaller retention comparisons; the additional searches’ F1 scores also differ from main-study exact-label agreement.ACC covers 57 Dreaddit and 78 GoEmotions candidates over 100 development items, versus 52 and 74 with complete F1; invalid or failed Dreaddit outputs count as errors.Source: 4) Compare Development Score Patterns · p. 5
  • Tree-level summaries average available candidates within each seed and level, then equally weight seed means; missing scores and absent levels are excluded.Exploratory paired tests compare F1 across levels with Holm correction across 21 pairs, but at most three searches and adaptive development selection prevent causal or generalization claims.Source: 4) Compare Development Score Patterns · p. 5

E. Evaluation

Evaluation combines exact label agreement, label-wise F1, and explanation grounding, while distinguishing what these measures capture from what the available summaries cannot establish.

1) Label Metrics:

  • Accuracy measures exact binary-label matches for Dreaddit and complete predicted-label-set matches for GoEmotions.Invalid outputs fail accuracy and contribute empty predicted sets to label-confusion counts.Source: 1) Label Metrics · p. 5
  • Macro-F1 averages label-level F1 across two Dreaddit labels or 28 GoEmotions labels, with undefined label F1 set to zero.It summarizes final testing, while prompt search optimizes exact label-set correctness.Source: 1) Label Metrics · p. 5
  • For valid Dreaddit outputs, precision, recall, specificity, and false-positive rate describe stress and no-stress classification errors.Macro-F1 weights both classes equally but does not encode unequal error costs or establish early detection or clinical utility; Dreaddit’s original evaluation also reports precision and recall.Source: 1) Label Metrics · p. 5
  • Main-study summaries lack invalid-output counts for false negatives and opposite-label false positives, preventing rankings by missed stress or false alarms.Conformability is distinct from label accuracy and human validation of an interpretation.Source: 1) Label Metrics · p. 5, 2) Conformability · p. 6
  • Conformability is the percentage of attempted predictions whose explanations pass a source-grounding rubric assessing factual support, evidence relevance, attribution, and unsupported assertions.A common Qwen3-32B judge evaluates the task, source, and answer without seeing the method or task model identity; invalid predictions fail, and unresolved judgments remain in the denominator.Source: 2) Conformability · pp. 5–6

3) Reporting Protocol:

The reporting materials preserve execution outcomes, prompt examples, and revision-score comparisons, while noting limits on attributing observed differences to a method or strategy.

3) Reporting Protocol:

  • Across 120 evaluated conditions, one GoEmotions/Qwen3-32B/ICL seed-44 Conformability judgment was unresolved, although its accuracy and Macro-F1 scores were retained.Execution records retain runtime revisions, earlier failures, and undefined metrics.Source: 3) Reporting Protocol · p. 6
  • The displayed Dreaddit and GoEmotions root candidates scored 66.56% and 29.30%, respectively.Their initial stress instruction asks for exactly one label: no stress or stress.Source: 3) Reporting Protocol · p. 7
  • Figure 4 compares prompt wording changes along historical GEPA golden routes, with development Macro-F1 or Micro-F1 and parent-to-child changes in percentage points.Shading marks new or rewritten wording relative to the full parent, not new strategy types.Source: Figure 4 · p. 7
  • Output-limit and failure-handling differences prevent isolating method or prompting-strategy effects.The reported execution conditions therefore do not provide a clean basis for attributing those effects.Source: 3) Reporting Protocol · p. 7

IV. RESULTS AND DISCUSSION

GEPA and GEPA+merge led on different tasks and model sizes, while prompt analyses associated development gains with combinations of strategies rather than isolated rules. The examples span emotional evidence, temporal distinctions, indirect cues, emotional context, and label selection.

and Emotion Prediction Tasks?

  • On Dreaddit, GEPA led Qwen3-8B Macro-F1 at 67.81%, while GEPA+merge led Qwen3-32B Macro-F1 at 71.80%; on GoEmotions, GEPA+merge led Qwen3-32B Macro-F1 at 25.14%.On GoEmotions, GEPA+merge exceeded Baseline in mean accuracy, Macro-F1, and Conformability with both model sizes, although metric leaders varied; on Dreaddit, GEPA had lower Conformability than Baseline with Qwen3-8B.Source: and Emotion Prediction Tasks? · p. 7

B. RQ2:

  • Table II’s examples show five strategies, but illustrate changes and exceptions rather than ranking their frequency.Dreaddit examples distinguish the writer’s emotion from topics or others’ experiences, current distress from past or resolved events, and distress from affection, help seeking, or coping.Source: and Emotion Prediction Tasks? · p. 8, Table II · p. 8
  • Directing attention to emotional evidence prioritizes the writer’s expressed state over difficult circumstances alone.Selected historical prompts share this rule, but their scores differ, so the rule alone does not explain performance differences.Source: and Emotion Prediction Tasks? · p. 8
  • Illustrating indirect emotional cues added examples such as putting oneself down, feeling trapped, and deserving negative treatment; Macro-F1 rose from 79.80% to 81.88%.These were shorter illustrations than complete demonstrations containing source text and a labeled answer.Source: and Emotion Prediction Tasks? · p. 8
  • Requiring emotional context distinguishes health concerns from distress about them, but retaining the rule alone did not preserve higher performance.A revision adding multiple changes raised Macro-F1 from 81.88% to 84.96%; a later child retaining the rule but changing other instructions scored 78.52%.Source: and Emotion Prediction Tasks? · p. 8
  • Guiding emotion-label selection specifies when neutral or multiple labels apply, including a procedure that checks emotionally charged text before matching it to the label inventory.The example’s parent already allowed overlapping emotions; Table II contrasts the procedure with a restrictive neutral rule.Source: and Emotion Prediction Tasks? · p. 8
  • A GEPA+merge candidate combined two parent prompts’ answer and source-review instructions unchanged, while evolved prompts’ formatting patterns did not establish a cause of performance differences.Common root instructions such as grounding and answer review were not discoveries; formatting observations included lists, emphasis, conditional rules, and restrictions.Source: and Emotion Prediction Tasks? · pp. 8–9

Evolution Vary Across Prediction Tasks and Evaluation Settings?

The separate searches showed different development trajectories for binary stress and multilabel emotion prediction, alongside instruction errors and evaluation discrepancies. Small comparison-sample gains did not always hold on the full development set, and development rankings could reverse on reported test results.

Evolution Vary Across Prediction Tasks and Evaluation Settings?

  • Across seeds 42 to 44, mean development F1 rose steadily across tree levels for binary stress but initially fell and partly recovered for 28-label emotion prediction.Dreaddit’s mean level-0-to-1 gain was 8.73 percentage points, but no comparison remained significant after Holm correction (adjusted p = 0.055 for the reported comparison).Source: Evolution Vary Across Prediction Tasks… · p. 9
  • All 19 scored Dreaddit root children gained 1.60 to 14.34 development Macro-F1 points, but only ten of 30 later scored revisions improved.By contrast, only two of eleven scored GoEmotions root children improved, each by less than one Micro-F1 point; seven fell by more than four.Source: Evolution Vary Across Prediction Tasks… · p. 9
  • Some revised GoEmotions prompts conflicted with the multilabel task by requesting a single primary emotion, and searches also contained incorrect label identifiers.One GoEmotions seed 43 candidate had only three correct identifiers among 28 entries and added labels absent from the dataset.Source: Evolution Vary Across Prediction Tasks… · p. 9
  • In GoEmotions/32B/GEPA/44, candidate 3 improved from zero to one correct item on a three-item comparison but scored 19% on full development data, below its parent’s 20%.The final frozen candidate scored 25% on development data.Source: Evolution Vary Across Prediction Tasks… · p. 9
  • Among three selected historical Dreaddit prompts, the highest-development prompt ranked lowest on reported test Macro-F1, while the lowest-development prompt ranked highest.Seed 42 candidate 10 scored 84.96% on development and 75.99% on test; seed 44 candidate 10 scored 78.95% and 79.99%, respectively.Source: Evolution Vary Across Prediction Tasks… · p. 9

V. CLINICAL IMPLICATIONS AND FUTURE WORK

The authors connect gain-associated prompting strategies to clinical reasoning but caution that the methods are not ready for clinical use. They propose clinical applications for future testing and conclude that results vary by task and evaluation, while isolated strategy effects remain unmeasured.

V. CLINICAL IMPLICATIONS AND FUTURE WORK

  • Gain-associated strategies echo clinical reasoning by separating expressed distress from difficult circumstances, distinguishing current from past distress, and attending to indirect cues such as feeling trapped.Because evolved prompts use plain language, clinicians can check them for errors; this is essential because some prompts contained wrong labels and contradictory rules.Source: §V · p. 9
  • The methods are not ready for clinical use because they were tested only on social media posts and the study did not measure false positives or missed stress.The conclusion also identifies validation on representative clinical data as a requirement for proposed clinical applications.Source: §V · p. 9, §VI · p. 10
  • The authors propose testing emotion detection in therapy narratives and simulated patients, and exploring text alongside speech and physiological signals.For written exposure therapy, they propose tracking change across repeated trauma narratives; change across sessions is a better target than intensity within a session, and prompts would need adapting because past events are central. Transfer from social media posts to clinical speech must be tested first.Source: §V · pp. 9–10

VI. CONCLUSION

  • GEPA led mean Macro-F1 on Dreaddit with Qwen3-8B, and GEPA+merge led on GoEmotions with Qwen3-32B, but accuracy gains did not always align with Macro-F1 or explanation support.Selected paths combined several identified strategies, including strategies also found in weaker prompts; complete instruction combinations require evaluation in context.Source: §VI · p. 10
  • Early gains in binary stress prediction contrasted with initial declines and partial recovery in multilabel emotion prediction; incorrect rules, failed full-development gains, and test-rank changes qualified progress.The conclusion calls for controlled tests of individual strategies and combinations, measuring false positives and missed stress, and validating on representative clinical data.Source: §VI · p. 10

Related papers

Loading 2610.21407v1…