PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification

Yufeng Wu, Meichun Liu

arXiv:2608.25579v1cs.CL

TL;DR

Metaphor-identification systems can behave unevenly across datasets, and evidence is limited on whether expert-informed procedures improve cross-dataset stability relative to parameter adaptation. This study compares four Chinese sentence-level regimes and finds that Skill-ZS has the most even observed profile, while fine-tuning retains native-data accuracy advantages.

  • Problem

    Metaphor-identification performance can change with dataset distribution and annotation policy, motivating comparison of cross-dataset stability beyond source-aligned accuracy.

  • Method

    The study compares BERT-FT, LLM-FT, direct LLM-ZS, and Skill-ZS across CMRE Test, CCIME, and CMC using Macro-F1-based stability summaries.

  • Results

    Skill-ZS has the highest external floor (82.64) and smallest three-dataset Macro-F1 range (4.08), while LLM-FT has the highest external mean (83.52).

  • Takeaways & Limitations

    Expert-informed procedural prompting offers a complementary route to more even observed cross-dataset performance, while fine-tuning remains strongest for native-data accuracy.

  • Takeaways & Limitations

    The empirical scope covers one base LLM, one Skill, four prespecified conditions, and three Chinese sentence-level datasets, so stability summaries do not establish behavior in unobserved settings.

Abstract

from arXiv · show

Metaphor-identification performance can change markedly across datasets that differ in text distribution and annotation policy. We examine whether a fixed expert-informed procedure produces a more even cross-dataset profile than task-specific parameter adaptation. Four prespecified conditions are compared for Chinese sentence-level metaphor identification: BERT fine-tuning (BERT-FT), QLoRA-based large language model fine-tuning (LLM-FT), direct zero-shot LLM prompting (LLM-ZS), and zero-shot prompting with a frozen procedural Skill (Skill-ZS). The Skill operationalizes established criteria involving contextual meaning, basic meaning, contrast, and comparison. Evaluation covers CMRE Test and two external datasets, CCIME and CMC. Fine-tuned scores are means over three seeds, whereas each zero-shot score comes from one deterministic configuration. Fine-tuning remains strongest on the native test set: BERT-FT reaches 91.76 Macro-F1. LLM-FT has the highest external mean (83.52), while Skill-ZS is close at 82.92 and has both the highest external floor (82.64) and the smallest observed range across all three datasets (4.08 points). In the matched zero-shot comparison, adding the Skill reduces metaphorical predictions on every dataset. This sharply lowers false positives on CCIME but increases false negatives on CMRE Test and CMC. The results position expert-informed Skill prompting as a complementary route to more even observed cross-dataset performance, while fine-tuning retains its advantage in native-data accuracy. To our knowledge, this is the first study to compare an expert-informed procedural Skill with task-specific fine-tuning in the same cross-dataset evaluation of Chinese sentence-level metaphor identification.

I. INTRODUCTION

The study asks whether a reusable expert-informed Skill can produce more even cross-dataset behavior than parameter adaptation for Chinese metaphor identification. It compares four knowledge-supply routes and evaluates stability through native performance, external performance, floors, gaps, and ranges.

  • Motivation: Metaphor-identification behavior can change across datasets because genre, construction mix, class prevalence, and annotation policy differ.These shifts affect both overall performance and the balance between false positives and false negatives.
  • Task background: The task requires judging whether contextual meaning contrasts with and relates to a more basic meaning.MIP and MIPVU organize this judgment into explicit procedural steps.
  • Proposed approach: Skill denotes a reusable multistep inference procedure supplied at inference time without changing model parameters.Its criteria may transfer differently from regularities learned through fine-tuning on one distribution.
  • Study design: The evaluation compares BERT-FT, LLM-FT, direct LLM-ZS, and Skill-ZS under a common Chinese sentence-level formulation.Skill-ZS uses a frozen expert-informed procedure, while LLM-ZS provides a matched no-Skill comparison.
  • Research questions: The study asks how the four conditions differ in stability, what dataset- and class-specific trade-offs underlie the summaries, and how the Skill shifts aggregate decisions.It presents the comparison as the first same-evaluation study of an expert-informed procedural Skill and task-specific fine-tuning for this task.

II. RELATED WORK

Prior work has embedded metaphor-identification expertise in models, prompts, and rules, while cross-dataset research highlights nonuniform transfer. This study extends that comparison by testing procedure-guided prompting against parameter adaptation in Chinese sentence classification.

  • Theory-informed identification: MIP and MIPVU structure metaphor judgments around contextual meaning, basic meaning, contrast, and comparison.These procedures aim to make metaphor decisions more reproducible.
  • Computational operationalizations: Computational systems have implemented these principles through contextual signals, contextualized interaction, basic-meaning modules, sense comparison, and contrast tests.Examples include Sequential metaphor identification, MelBERT, explicit basic-meaning modeling, and ContrastWSD.
  • Prompt-based approaches: Generative models express operational knowledge in prompts, motivating expert-informed procedures for Chinese metaphor identification.The present Skill adapts selected principles to sentence-level binary classification rather than reproducing the full lexical-unit MIPVU procedure.
  • Cross-dataset generalization: Metaphor-identification systems transfer nonuniformly across datasets, languages, genres, and other changed target distributions.This literature distinguishes source-aligned evaluation from performance under distribution shift.
  • LLM comparison: Recent LLM research compares parameter adaptation with direct or procedure-guided prompting, but this study focuses on Chinese sentence classification and observed cross-dataset evenness.It combines a four-regime comparison with a matched direct-versus-Skill comparison.

III. METHODS

The methods hold the sentence-level label space and evaluation rows constant across four knowledge-supply regimes and three Chinese datasets. Overlap removal and documented annotation-policy differences define the evaluation scope.

  • Design: The study compares four ways of supplying task knowledge while keeping the sentence-level label space and evaluation rows constant.CMRE Dev supports supervised model selection, and zero-shot conditions were finalized before formal evaluation.
  • Datasets and labels: Each sentence receives a metaphorical or non-metaphorical label across CMRE, CCIME, and CMC.CMRE provides the training, development, and native test partitions; CCIME and CMC provide external evaluation rows.
  • Overlap control: 38 CCIME rows and two CMC rows matching CMRE Train or Dev were removed before evaluation.Released duplicate rows within an evaluation view retained distinct identifiers, and the resulting rows were shared by all four conditions.
  • Annotation policy: CMRE retains released-positive similes as metaphorical, whereas CCIME assigns similes and hyperbole to the non-metaphorical class.Accordingly, CCIME reflects both dataset shift and annotation-policy shift.

B. Four Knowledge-Supply Regimes

The four regimes differ in where task knowledge is supplied: supervised fine-tuning learns from CMRE labels, while zero-shot prompting uses no labeled examples. The zero-shot pair isolates the Skill’s aggregate addition under matched runtime conditions.

  • Regimes: BERT-FT and LLM-FT learn from CMRE labels, whereas LLM-ZS and Skill-ZS receive no labeled examples.The four-way comparison therefore spans different architectures and adaptation costs.
  • Matched comparison: The controlled zero-shot comparison uses the same base model, runtime, prompt shell, output format, and evaluation rows.The frozen Skill is the only condition-level addition.
  • BERT-FT: BERT-FT uses a Chinese RoBERTa-based sequence-classification model with a two-label head and CMRE Dev Macro-F1 checkpoint selection.It uses AdamW, a 2 × 10^-5 learning rate, effective batch size 16, and a fixed 0.5 threshold.
  • LLM-FT: LLM-FT adapts the text-only Qwen/Qwen3.6-27B model with 4-bit NF4 QLoRA and CMRE Dev-based checkpoint selection.Its target outputs contain one JSON label and use loss restricted to target tokens.
  • Reproducibility: Both fine-tuned conditions use seeds 42, 43, and 44, selecting one checkpoint independently per seed before test prediction.The supplement reports the complete optimization and runtime configurations.

D. Zero-Shot Control and Expert-Informed Skill

The study compares direct zero-shot prompting with a frozen expert-informed Skill under deterministic, example-free conditions. The Skill encodes a multistep procedure for contextual meaning, candidate mapping, basic meaning, contrast, comparison, rejection, and final labeling.

  • Zero-Shot Control and Expert-Informed Skill: Skill-ZS adds a frozen expert-informed procedure to the same base LLM used by LLM-ZS.The matched zero-shot conditions use deterministic decoding and the same constrained JSON output format.
  • Zero-Shot Control and Expert-Informed Skill: The Skill operationalizes metaphor identification through six sequential operations ending in one sentence label.Its wording was refined through seven iterations on 60 balanced AI-assisted synthetic specification sentences.
  • Zero-Shot Control and Expert-Informed Skill: Macro-F1 is the primary cross-dataset measure because class prevalence differs across datasets.Stability is summarized using external mean, external floor, signed native–external mean gap, and three-dataset range.

IV. RESULTS

Results separate native peak performance from cross-dataset evenness. LLM-FT has the highest external mean, whereas Skill-ZS has the highest external floor and smallest observed range, with descriptive differences limited by unequal run designs.

  • Observed Cross-Dataset Stability: The stability analysis uses external mean, external floor, signed native–external mean gap, and three-dataset range.Macro-F1 is computed across CMRE Test, CCIME, and CMC, with the criteria based on corresponding unrounded values.
  • Observed Cross-Dataset Stability: 83.52 is LLM-FT’s highest external mean, while Skill-ZS reaches an external floor of 82.64 and a three-dataset range of 4.08 points.Skill-ZS is 0.60 points below LLM-FT on external mean and has the most even observed profile.
  • Observed Cross-Dataset Stability: BERT-FT has the highest native Macro-F1, whereas Skill-ZS’s external stability is coupled with a lower native peak.Skill-ZS scores 3.80 points higher externally than natively on average.
  • Observed Cross-Dataset Stability: Three-seed fine-tuned scores provide sample SDs, but deterministic zero-shot estimates provide no comparable between-run dispersion.The study therefore reports cross-regime rankings and differences descriptively without significance claims.

B. Dataset and Class-Specific Trade-offs

Dataset-specific precision–recall trade-offs explain the conditions’ differing profiles. Skill-ZS improves precision on CMRE Test and CCIME, but lowers recall on CMRE Test and CMC relative to LLM-ZS.

  • CMRE Test: 92.70 is BERT-FT’s CMRE Test metaphor precision, while LLM-FT has higher recall at 94.04.Skill-ZS raises precision over LLM-ZS from 83.62 to 88.58 but lowers recall from 79.29 to 67.53.
  • CCIME: 15.90 points is Skill-ZS’s CCIME Macro-F1 increase over LLM-ZS.Precision rises from 61.80 to 77.56 while recall remains similar at 90.83 versus 91.01.
  • CMC: On CMC, LLM-ZS has the highest accuracy, recall, metaphor F1, and Macro-F1, while LLM-FT has the highest precision.Skill-ZS raises precision from 87.06 to 88.48 but lowers recall from 90.88 to 70.07 relative to LLM-ZS.
  • Cross-Dataset Contrast: Skill-ZS scores 3.52 points higher on CCIME and 4.08 points higher on CMC than on CMRE Test.CCIME’s treatment of similes and hyperbole as non-metaphorical is offered as one plausible explanation.

C. Aggregate Decision Redistribution

Adding the Skill consistently reduces metaphorical predictions, with dataset-dependent effects on error types and a more even observed cross-dataset profile. The shift improves prevalence alignment on CCIME but worsens it on CMRE Test and CMC.

  • Aggregate decision shift: 57.14% versus 71.86%: Skill-ZS predicts fewer metaphorical cases than LLM-ZS on CCIME.False positives fall from 319 to 149, while false negatives change from 51 to 52.
  • Aggregate decision shift: 50 and 57: false negatives rise by these amounts on CMRE Test and CMC after adding the Skill.False positives also fall on both datasets.
  • Prevalence alignment: 23.06 versus 8.35 percentage points: Skill-ZS narrows CCIME’s excess metaphorical-prediction rate relative to gold prevalence.On CMRE Test and CMC, LLM-ZS is closer to gold prevalence than Skill-ZS.
  • Cross-dataset profile: The results trade off source-aligned peak performance against evenness across the observed datasets.The comparison distinguishes native peak, external average, and external floor with limited variation.
  • Cross-dataset profile: Skill-ZS combines a competitive external mean with the highest observed external floor and narrowest three-dataset range.This indicates a more even observed performance level, although native performance is not maximized.

B. Stability Through a Decision Trade-off

The Skill’s more even profile reflects a systematic shift in output decisions rather than uniform accuracy improvement. Annotation-policy differences help explain why the same stricter contrast criterion benefits CCIME but harms recall elsewhere.

  • Decision trade-off: The Skill lowers metaphorical predictions consistently, reducing CCIME false positives while increasing CMRE Test and CMC false negatives.Thus, aggregate stability coexists with dataset-specific precision–recall trade-offs.
  • Annotation-policy asymmetry: CCIME labels similes and hyperbole as non-metaphorical, whereas CMRE retains released-positive similes as metaphorical.A supported contextual–basic contrast can therefore align differently with the two annotation boundaries.

C. Choosing a Knowledge-Supply Regime

The preferred knowledge-supply regime depends on whether the objective prioritizes native accuracy or even external performance. The study’s scope and measurement design constrain how broadly these choices can be generalized.

  • Regime choice: Fine-tuning is favored when maximizing labeled-source performance is central, while LLM-FT provides the strongest external average in this study.Skill-ZS is instead attractive when external floor and cross-dataset variation receive greater weight.
  • Regime choice: Skill-ZS supports direct procedural revision and inspection without another parameter-update cycle.Its criteria remain explicit at inference time.
  • Future direction: Hybrid systems could combine supervised adaptation with procedural checks, evaluated using class-specific measures, decision costs, ablations, and prompt-sensitivity tests.These tests separate average-performance gains from cross-dataset-stability gains.
  • Scope: Four conditions, one base LLM, one Skill, and three Chinese sentence-level datasets limit claims about unobserved settings.The CCIME transfer partly reflects changed labeling policy rather than domain change alone.
  • Measurement limitations: One deterministic run for each zero-shot condition prevents comparable uncertainty estimates across all four conditions.Aggregate confusion counts also cannot identify instruction-level causes without controlled ablation and instance-level analysis.
  • Measurement limitations: Sentence-level binary evaluation does not measure exact metaphor-related words, explanations, procedural compliance, calibration, or human interpretability.External-dataset differences also prevent attributing performance gaps to a single domain-shift factor.

VII. CONCLUSION

The frozen expert-informed Skill yields the most even observed cross-dataset profile, while fine-tuning retains stronger native performance. Its matched zero-shot effects reveal dataset-specific trade-offs behind aggregate stability.

  • The frozen Skill achieves the highest external floor and smallest three-dataset Macro-F1 range.Its external mean remains within 0.60 points of LLM-FT.
  • Fine-tuning retains the strongest native performance, while LLM-FT attains the highest external average.
  • Adding the Skill consistently reduces metaphorical predictions relative to matched LLM-ZS.This substantially reduces false positives on CCIME but increases false negatives on CMRE Test and CMC.
  • A stable aggregate profile can contain different class-specific trade-offs across datasets.
  • The study uses existing research datasets and model-generated predictions, with synthetic specification sentences not presented as human annotation.Dataset access and redistribution remain subject to original release terms.
Loading 2608.25579v1…