PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Large Language Models for Detecting Satirical Code-Mixed Sentences

Răzvan-Alexandru Smădu, Ioana Buhnila, Daniela Tanase, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel

arXiv:2610.29907v1cs.CL

TL;DR

Code-mixed satire detection in Romanian remains underexplored, despite the importance of linguistic cues and a shortage of relevant data. The paper creates a synthetic Romanian satire benchmark and human-adjudicated subset, then evaluates LLMs across prompting and fine-tuning settings. Modern open models rival frontier systems zero-shot, and fine-tuning enables them to outperform frontier zero-shot baselines on human-adjudicated code-switched satire.

  • Problem

    The intersection of code-mixed text and satire detection remains underexplored, and Romanian code-mixed satire data are scarce.

    Source: §1 · p. 2

  • Method

    The paper creates CS-SeLeRoSa using four generation methods and four target languages, then benchmarks LLMs on the synthetic dataset and a human-adjudicated gold set.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Modern open models such as Gemma 4 31B rival frontier systems zero-shot and, when fine-tuned, outperform them on human-adjudicated code-switched satire.

    Source: §6 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    The benchmark analysis finds that code-mixing strategies strongly affect lexical naturalness and syntactic cohesion, while modern open models can perform competitively under code-switching when fine-tuned.

    Source: §7 · p. 9

    The study evaluates synthetic code-mixing without human assessment of the entire dataset, and naturalness remains constrained by the scarcity of satirical news that also contains code-switching.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Code-switching is a natural linguistic phenomenon, yet its impact on high-level reasoning tasks remains largely unexplored in under-resourced languages like Romanian. Natural language processing tasks, such as satire detection, rely heavily on nuanced cultural and syntactic cues that large language models struggle to capture. In this paper, we investigate four methods (i.e., random replacement, phrase translation, dependency parsing, and large language models) for generating synthetic code-mixed Romanian sentences with English, German, French, and Italian. We start with an existing dataset for satire detection, compiled from Romanian news articles and annotated at the sentence level by humans, and augment it with over 600k samples, namely CS-SeLeRoSa. We provide an in-depth analysis of the dataset generation process. Then, we evaluate several multilingual large language models across zero-shot, chain-of-thought, and fine-tuning regimes on both the synthetic corpus and a human-adjudicated gold benchmark, namely CS-SeLeRoSa-Gold. While older open models suffer substantial degradation (dropping to 37%-55% macro-F1), newer models like Gemma 4 31B achieve 68.1% zero-shot, rivaling frontier models like GPT-5 mini (69.5%). Furthermore, fine-tuning significantly reduces formatting errors and improves the results, increasing Gemma 4 31B to 77.5% on CS-SeLeRoSa and 72.6% on the gold standard, outperforming frontier zero-shot baselines.

1 Introduction

Code-mixed satire detection remains underexplored, particularly for Romanian, where code-switching behavior may differ in a morphologically rich language. This work introduces a synthetic sentence-level Romanian satire dataset and gold subset, analyzes the generated data, and benchmarks LLMs.

1 Introduction

  • Romanian is paired with English, German, French, and Italian, reflecting diaspora ties and the scarcity of code-mixed satire data.The paper notes that code-switching behavior can shift in morphologically rich languages.Source: §1 · pp. 1–2
  • The intersection of satire detection and code-mixed text remains underexplored, with data scarcity leading many studies to rely on synthetic generation.Prior work on satire, irony, and humor in code-mixed text has focused mainly on Hindi-English and Spanish-English.Source: §1 · pp. 1–2
  • CS-SeLeRoSa is the first synthetic sentence-level code-mixed Romanian satirical news dataset, covering four generation methods and four language pairs.The work also provides linguistic and semantic analyses of the generated data.Source: §1 · p. 2
  • The authors curate CS-SeLeRoSa-Gold, a human-adjudicated subset verified by expert annotators.They benchmark LLMs on both datasets across zero-shot, chain-of-thought, and fine-tuning settings.Source: §1 · p. 2

2 Related Work

Related work spans code-switching methods and datasets, satire and humor detection, and reasoning-oriented satire models. Prior satire-focused code-mixing studies cover other language combinations, while existing code-switching resources do not include Romanian.

2 Related Work

  • Code-switching research covers tasks such as slot filling, intent prediction, and distinguishing natural from manipulated sentences using syntactic or graph-based methods.Studies also compare manipulated variants with fluent bilingual preferences and analyze their syntactic differences.Source: §2 · p. 2
  • Prior satire, irony, and sarcasm work spans Hindi-English code-mixed social media, multilingual irony detection, and more recent Arabic code-mixing studies.The cited approaches include sub-word embeddings, feature-based and neural models, and work addressing Arabic-language challenges.Source: §2 · p. 2
  • Code-mixed dataset work includes synthetic text for several language pairs and multilingual audio resources, but SwitchLingua does not include Romanian.The reviewed work also includes LLM-based code-mixed tasks and an agentic data-synthesis method.Source: §2 · p. 3
  • Satire detection research ranges from lexical and structural cues to adversarial debiasing and multimodal inputs, with prior work arguing for common-sense reasoning.Later studies frame satire through incongruity and examine chain-of-thought and structured reasoning for LLM detection.Source: §2 · p. 3

3 Method

The method builds on SeLeRoSa and generates Romanian code-mixed variants through four strategies, then constructs a human-adjudicated evaluation subset. The dataset covers four target languages and up to three variants per combination, with generation counts declining when attempts fail or variants cannot be produced.

3.1 Dataset

  • The method uses SeLeRoSa’s approximately 13.8k manually annotated Romanian satire sentences and preserves its anonymization and train/validation/test split.Named entities remain generic tags, and the same official split is used in experiments.Source: §3.1 · p. 3

3.2 Translation

  • Gemma 3 27B supplies machine translation for generation methods that translate words or phrases before reintegrating them into Romanian sentences.End-to-end LLM generation does not require this separate translation step.Source: §3.2 · p. 3, §3.3 · p. 5

3.3 Synthetic Code-Mix Generation

  • Random replacement splits sentences at word boundaries and randomly replaces 50% of selected words with translations, without filtering degraded outputs.The method follows the prior approach while using LLM-based machine translation.Source: §3.3 · p. 4
  • Phrase translation identifies keyphrases with multi-RAKE, translates them, and replaces randomly selected phrases in the original sentence.The selected phrases are intended to be self-contained so they can be translated coherently.Source: §3.3 · pp. 4–5
  • Dependency-tree generation translates a phrase spanned by a dependency sub-tree, using the tree structure to select a self-contained span with more context.Its output depends on the quality of the dependency parser and translation system.Source: §3.3 · p. 5
  • The end-to-end LLM method prompts Gemma 3 27B to produce a meaning-preserving code-mixed sentence directly, at temperature 0.8.This method bypasses the separate translation step.Source: §3.3 · p. 5

3.4 Dataset Statistics

  • The four methods across four target languages yield at most 665,904 samples over three iterations; counts fall at higher iteration levels when variants fail or cannot be extracted.For k=1, the maximum is 221,956; the totals are 221,716 for k=2 and 219,306 for k=3.Source: §3.4 · p. 5
  • The evaluation-only gold set begins with 715 samples from 50 Romanian source sentences and retains 391 high-quality samples after human adjudication.The selection spans variants from the four methods and four languages.Source: Figure 1 · p. 4, §3.4 · p. 5

4 Experimental Setup

The task is binary sentence-level satire classification, with factual and satirical labels. The experiments compare open-weight and proprietary models across prompting and fine-tuning settings, using macro-F1 as the primary metric.

4.1 Classification Task

  • The classification task labels each sentence as factual or satirical, with code-switched sentences retaining Romanian grammatical structure and translated segments inserted as well-formed, semantically self-contained islands.The authors follow the Matrix Language Frame model; random code-switching may instead produce syntactically or semantically unnatural samples.Source: §4.1 · p. 6
  • Zero-shot and chain-of-thought prompts specify the task, language pair, classes, and input; fine-tuning uses a minimal prompt with the task, input, and target answer.For zero-shot and chain-of-thought, models produce XML-like responses; unparseable outputs count as classification errors.Source: §4.1 · p. 6

4.2 Models

  • Models include open-weight and proprietary systems, but fine-tuning is evaluated only for open-weight models, while proprietary reasoning models are evaluated zero-shot only.The setup also includes frontier models on the CS-SeLeRoSa-Gold test set.Source: §4.2 · p. 6

4.3 Evaluation Metrics

  • Macro-averaged F1 is the primary metric, with accuracy reported in the appendix; open-weight results average three seeded runs, while proprietary results are generally single-run.Proprietary zero-shot results on CS-SeLeRoSa-Gold average three runs; proprietary chain-of-thought results on Gold are single-run.Source: §4.3 · p. 6, Table 2 · p. 7

5 Results

Fine-tuning delivers the strongest reported benchmark results, while zero-shot performance varies by model generation and code-switching method. The section also identifies elevated false positives and limitations in the synthetic data and evaluation scope.

5 Results

  • Gemma 3 27B drops 12.9 macro-F1 points (49.9 to 37.0) under zero-shot code-switching, while Gemma 4 31B drops 2.1 points and matches the resilience of proprietary models.On the gold benchmark, capable models lose 5.5–9.9 points relative to synthetic results; Gemma 4 31B scores 61.8 ± 0.2, statistically on par with GPT-5 at 61.1 ± 1.6.Source: §5 · p. 7
  • Chain-of-thought prompting helps weaker models and reduces run-to-run variance, but slightly lowers Gemma 4 31B’s code-switched performance by 1.6 points.For RoLlama 3.1 8B, chain-of-thought lowers performance by 6.7 points on CS-SeLeRoSa and 7.9 points on CS-SeLeRoSa-Gold; the reported errors mostly flip factual labels to satirical.Source: §5 · p. 8
  • 77.5% F1 for Gemma 4 31B after fine-tuning on CS-SeLeRoSa marks the benchmark’s highest score, with fine-tuning also outperforming zero-shot proprietary models on the gold set by 11.5 points.Fine-tuning brings most architectures to within −0.8 to +2.3 F1 points of the monolingual baseline, and Llama 3.1 8B improves from 69.0% to 74.9%.Source: §5 · p. 8
  • Code-switching often raises false-positive rates, random replacement causes the largest model-performance drops, and dependency-based insertion is the most resilient method.Romance-language insertions generally produce lower false-positive rates than Germanic ones, though code-mixing intensity varies across languages.Source: §5 · p. 8

6 Discussion

  • Older and smaller models struggle with code-switched satire, while lexical substitutions and instruction-following errors are reported failure modes that fine-tuning can mitigate.Some smaller models fail to produce the required XML format, translate tags, use synonyms, or output corrupted labels.Source: §6 · pp. 8–9

7 Conclusions

  • CS-SeLeRoSa is the first large-scale code-mixed Romanian satire benchmark, and modern open models can rival frontier systems zero-shot and outperform them after fine-tuning.The analysis also finds that code-mixing strategies affect lexical naturalness and syntactic cohesion.Source: §7 · p. 9

Limitations

  • The dataset is synthetic and not fully human-evaluated; natural code-switched satirical news is scarce, and the study uses Romanian as its base language and evaluates a limited set of small LLMs.A human-validated gold set and a small human evaluation address some dataset-quality concerns, but do not constitute human evaluation of the entire dataset.Source: Limitations · p. 9

Ethical Considerations

  • Preserving SeLeRoSa’s anonymization is intended to focus predictions on linguistic features and context rather than memorized entity-level stereotypes.The dataset retains original news-source URLs so future work can retrieve entity contexts when studying model biases.Source: Ethical Considerations · p. 9

B Translation Evaluation

The authors benchmarked candidate translation models on FLORES+ for Romanian-to-English, German, French, and Italian translation, selecting Gemma 3 27B for dataset generation.

B Translation Evaluation

  • Gemma 3 27B outperforms the other models on all language pairs and metrics, so the authors select it for code-mixed dataset generation.Best scores are obtained for English and French.Source: B Translation Evaluation · p. 14

C Code-Mixing Evaluation

The authors assess synthetic code-mixing with automated measures of mixing, diversity, semantic similarity, and readability, then create a human-adjudicated subset for model evaluation.

C Code-Mixing Evaluation

  • Random replacement produces more frequent language switching, while the other methods yield more balanced code-mixing patterns.The evaluation uses CMI, M-Index, I-Index, language entropy, and switch-point average, computed with CodeMixedNLP; language identification combines character-based heuristics and lingua.Source: C Code-Mixing Evaluation · pp. 14–15, Table 4 · p. 15
  • Random replacement generates the most dissimilar sentences; LLM-based mixing is slightly more dissimilar than phrase and dependency methods for EN, FR, and IT, but not German.Dissimilarity is measured at token level using Jaccard and at semantic level using BERTScore with XLM-RoBERTa-large embeddings.Source: C Code-Mixing Evaluation · p. 15, Table 5 · p. 16
  • After filtering 715 samples to remove unnatural code-switching or significant grammatical errors, 391 high-quality code-mixed sentences form CS-SeLeRoSa-Gold, intended only for model evaluation.Source: C Code-Mixing Evaluation · p. 16

D Dataset Structure

CS-SeLeRoSa adds fields that identify each sample’s target language, generation method, and variant index, enabling researchers to isolate variants or deduplicate splits.

D Dataset Structure

  • Each sample records its code-switch target language, generation method, and generation index, with k = 0 marking the original sentence.The method field distinguishes none, random, dependency, LLM, and phrase generation.Source: D Dataset Structure · p. 16
  • Table 6 provides examples across target languages, generation methods, and satire and regular labels.The examples include original Romanian sentences, generated code-switched counterparts, and English translations.Source: D Dataset Structure · p. 16

E.1 Models

The paper describes the evaluated models and training setup, then reports a human evaluation of generated samples using multiple criteria and adjudication.

E.1 Models

  • Romanian-pretrained RoGemma 2 and RoLlama 3.1 outperform their base counterparts on the gold set in zero-shot evaluation; newer open models and closed-source models attain the highest scores.The authors suggest pretraining data volume and composition may contribute to these differences.Source: §E.1 · p. 18

E.2 Hyperparameters

  • Open-source models were fine-tuned with 4-bit QLoRA for three epochs, while proprietary models were not fine-tuned because of budget constraints.The setup used a 10^-4 starting learning rate, batch size 32, and context length 1,024 tokens.Source: §E.2 · p. 18

E.3 Computational Resources and Costs

  • The proprietary-model experiments cost approximately 120 USD.Inference and fine-tuning used NVIDIA A100 80GB and H100 80GB GPUs; proprietary models were accessed through the OpenAI API.Source: §E.3 · pp. 18–19

F.1 Annotation Protocol

  • Four Romanian-proficient annotators evaluated synthetic samples using class preservation, linguistic acceptability, meaning preservation, code-switching quality, and a keep-or-reject verdict.The process used calibration and batch annotation, with consensus discussion for disagreements and final human verification.Source: F Human Evaluation · p. 19, §F.1 · p. 19

F.2 Data Selection

  • The annotated evaluation set began with 715 samples drawn from 50 Romanian source sentences and their generated counterparts across four methods and four languages.The source set contained 37 satirical and 13 regular sentences; 150–200 samples per target language were labeled.Source: §F.2 · p. 20

F.3 Annotation Statistics

  • The final benchmark contains 391 samples, or 54.7% of the annotated set; acceptance exceeded 60% for LLM and dependency methods but stayed below 20% for random replacement.Poor code-switching quality was the most common rejection reason (63.9%), followed by linguistic errors (34%) and meaning loss (26.9%).Source: §F.3 · p. 20, Figure 3 · p. 20, Figure 4 · p. 20
  • Logistic regression models using the four annotation criteria achieved over 90% in-sample accuracy in predicting the final verdict.Meaning preservation was the most important feature; linguistic acceptability and code-switching quality were similarly important to each other but weaker, and class preservation provided the lowest signal.Source: §F.3 · pp. 21–22, Table 9 · p. 22
  • Final-verdict agreement was 75.9% with Krippendorff’s α = 0.518; meaning preservation was the strongest criterion signal, while class preservation was weakest.The class-preservation alpha was 0.178 despite 85.0% raw agreement, attributed to skew toward the yes label; disagreement most often concerned code-switching quality, especially phrase-level switches.Source: §F.3 · pp. 21–23

F.4 Gold Standard Dataset Statistics

The adjudicated gold set contains 391 retained samples, while monolingual baselines show Gemma 4 31B leading open-weight models and fine-tuning outperforming in-context evaluation for most models.

F.4 Gold Standard Dataset Statistics

  • 391 samples remained in the gold set after adjudication, representing 54.7% of samples with a final “keep” verdict.The retained set contained 218 satirical and 173 regular samples, with an average sentence length of 21 words.Source: §F.4 · p. 23, Table 13 · p. 23
  • The gold set covered 43 of 48 method-language-k combinations (89.6%), with German-random represented by one sample.English and French each had approximately 120 mixed samples; German and Italian had 70–86 each.Source: §F.4 · p. 23, Table 12 · p. 23, Table 13 · p. 23
  • On the gold set, the data were 55.8% satirical and 44.2% regular, and code-switching statistics showed human annotators preferred fewer switching points for random replacement.Other metrics for phrase-, dependency-, and LLM-based generation generally remained unchanged.Source: §F.4 · p. 23, Table 14 · p. 23

G.1 Monolingual Baseline Evaluation

  • 70.2% F1 for Gemma 4 31B was the best monolingual zero-shot result, while fine-tuning brought most models above 70% accuracy and F1.The 31B model also exceeded 78% in the fine-tuning setting.Source: §G.1 · pp. 24–25, Table 15 · p. 24
  • On the original 50 Romanian sentences used to derive the CS-SeLeRoSa-Gold test set, Gemma 4 31B remains the top-performing model in the fine-tuning regime.Source: Table 16 · p. 24, §G.1 · p. 25

G.2 Code-Switching Benchmark Results

On the full code-switched test set, Gemma 4 31B is competitive with proprietary models in zero-shot evaluation and exceeds 77% in both reported metrics after fine-tuning. On the gold benchmark, it remains the best-performing model, including against frontier proprietary models.

G.2 Code-Switching Benchmark Results

  • 68.1% F1 for Gemma 4 31B was comparable to GPT-5 mini at 69.5% on the full test set; fine-tuning raised Gemma 4 31B above 77% in both metrics.Its chain-of-thought scores were 66.5% on both metrics.Source: §G.2 · p. 25, Table 19 · p. 26
  • A Romanian-adapted model reached 75.4% under fine-tuning, outperforming other Romanian-adapted architectures such as RoLlama and RoMistral.Source: §G.2 · p. 25
  • Around 72–73% in both metrics was achieved by fine-tuned Gemma 4 31B on the gold set, where frontier models did not surpass it.In zero-shot and chain-of-thought settings, Gemma 4 31B achieved around 62–63% in both metrics.Source: §G.2 · p. 25
  • 52.6% vs 44.8% on CS-SeLeRoSa and 54.8% vs 43.0% on CS-SeLeRoSa-Gold were RoLlama versus base Llama zero-shot F1 results.Romanian-pretrained models continued to outperform their base-model counterparts in the zero-shot setting.Source: §G.2 · p. 25

G.3 Detailed Method and Language Breakdown

Across code-switching methods and language pairs, fine-tuning improves results, while random replacement is generally the most disruptive and dependency-based generation the most resilient. Degradation varies by model and language, with Germanic pairs more challenging than Romance pairs in the reported comparisons.

G.3 Detailed Method and Language Breakdown

  • Fine-tuning significantly improved results across languages and generation methods over zero-shot and chain-of-thought prompting.Gemma 4 31B achieved top performance across nearly all language-method combinations under fine-tuning.Source: §G.3 · pp. 26, 28, Table 20 · p. 27
  • Random replacement often reduced performance by more than 10% relative to syntactically informed methods, though GPT-5 mini’s variation across methods was at most 5%.The method was applied without additional filtering; fine-tuning on its data yielded results competitive with other generation methods.Source: §G.3 · p. 26
  • French and Italian code-mixing yielded higher F1-scores and less degradation than English and German; German reached approximately 35% for zero-shot RoMistral 7B.The supplied comparison reports the steepest degradation for German among open-weight models in zero-shot settings.Source: §G.3 · pp. 27–28
  • RoMistral 7B gained 8–13 F1 points with chain-of-thought, whereas RoLlama 3.1 8B lost about 5–10 macro-F1 points across methods and languages.Gemma 4 31B was the top performer across nearly all combinations under fine-tuning, including noisy random code-mixing where models reached over 70%.Source: §G.3 · p. 28
  • On the gold set, phrase- and LLM-based generation had the highest performance; GPT-5 mini reached approximately 70% F1 and exceeded GPT-5 by 2%.Gemma 4 31B achieved the highest performance when fine-tuned, while models still tended to underperform on random-method samples after filtering.Source: Table 20 · p. 27, §G.3 · p. 28

G.4 Method-Level Degradation Analysis

  • Up to 18.8% degradation occurred with random switching for Gemma 3 27B, while dependency-based switching stayed under 5% for almost all models.Dependency-based degradation was near zero for Gemma 4 31B, GPT-5 mini, and GPT-6 Luna; GPT-5 mini’s maximum degradation was 4.4% under random switching.Source: Table 21 · p. 28, §G.4 · p. 28

G.5 Failure Modes Analysis

Code-switching noise is associated with more false-positive satire predictions, while fine-tuning reduces false positives at the cost of more false negatives. The analysis also documents a chain-of-thought failure case, instruction-following errors, and representation differences among generation methods.

G.5 Failure Modes Analysis

  • Most models showed high false-positive and low false-negative rates in zero-shot settings, with the lowest false-positive rates on dependency-based switching and the highest on random switching.GPT-5 mini was an exception, maintaining a statistically invariant 26–30% false-positive rate across switching methods.Source: §G.5 · p. 29, Table 22 · p. 29
  • Romance-language mixing had lower false-positive rates than Germanic-language mixing across almost all models; the reported differences were statistically significant for Gemma models and both GPT models.The passage suggests cognate sharing and syntactic compatibility may mitigate false-positive satire detection.Source: §G.5 · pp. 29–30, Table 23 · p. 30
  • 92.0% to 26.1% was Gemma 3 27B’s false-positive rate after fine-tuning; RoLlama 3.1 8B fell from 70.5% to 26.1%.Fine-tuning also increased false-negative rates as the models became more selective and missed some harder satire examples.Source: Table 24 · p. 30, §G.5 · p. 30

G.6 Baseline RoLlama 3.1 8B Chain-of-Thought Degradation

  • About 10 percentage points was RoLlama 3.1 8B’s monolingual F1 decline with chain-of-thought, as predicted satire increased by 15.6 points and 336 factual samples became false positives.The reasoning traces over-associated optimism, positivity, or formal phrasing with satire; 39.5% of affected sentences belonged to a cluster about people leaving and changing positions.Source: §G.6 · pp. 30–31

G.7 Instruction Following Error Rate

  • 23%–25% were the highest zero-shot instruction-following error rates, for Llama 3.1 8B and RoMistral 7B; fine-tuning reduced error rates to almost 0%.Gemma 3 27B, Gemma 4 31B, and GPT models had formatting error rates below 0.1%.Source: §G.7 · p. 31, Table 27 · p. 32, Table 28 · p. 32

H t-SNE Visualization

  • Random code-mixing produced separable sentence-embedding representations, while other methods produced examples closer to the original distribution; the figure compares languages and generation methods.The authors report no meaningful representation differences across languages.Source: H t-SNE Visualization · p. 33, Figure 8 · p. 33

Related papers

Loading 2610.29907v1…