PaperlayerView exact revision on arXiv

Source-linked AI summary

Introducing TrGLUE and SentiTurca: A Comprehensive Benchmark for Turkish General Language Understanding and Sentiment Analysis

Duygu Altinok

arXiv:2512.22100v1cs.CLcs.AI

TL;DR

Turkish lacks a comprehensive, standardized NLU benchmark despite the need to evaluate models across tasks. The paper introduces TrGLUE and SentiTurca, built with Turkish-focused data and semi-automated annotation. TrGLUE captures substantial morphological variety, while Turkish Hate Map remains challenging even for robust models.

  • Problem

    Turkish has scarce evaluation resources and no comprehensive standard benchmark, limiting coverage and comparability across models.

    Source: §1 · p. 2

  • Method

    The paper introduces TrGLUE and SentiTurca, combining Turkish-focused datasets with semi-automated annotation and human validation.

    Source: §1 · p. 3

  • Results

    2.25 morphemes/token and 1,095 distinct inflectional feature bundles characterize TrGLUE, while Turkish Hate Map challenges even robust language models.

    Source: §3.4.1 · p. 23, §6.3 · p. 45

  • Takeaways & Limitations

    Turkish Hate Map offers researchers a challenging resource for hate-speech classification.

    Source: §6.3 · p. 45

    Turkish morphology, pronoun omission, and word order can eliminate WNLI’s intended ambiguity or make it unnatural, undermining validity.

    Source: §3.2.7 · pp. 19, 21

Abstract

from arXiv · show

Evaluating the performance of various model architectures, such as transformers, large language models (LLMs), and other NLP systems, requires comprehensive benchmarks that measure performance across multiple dimensions. Among these, the evaluation of natural language understanding (NLU) is particularly critical as it serves as a fundamental criterion for assessing model capabilities. Thus, it is essential to establish benchmarks that enable thorough evaluation and analysis of NLU abilities from diverse perspectives. While the GLUE benchmark has set a standard for evaluating English NLU, similar benchmarks have been developed for other languages, such as CLUE for Chinese, FLUE for French, and JGLUE for Japanese. However, no comparable benchmark currently exists for the Turkish language. To address this gap, we introduce TrGLUE, a comprehensive benchmark encompassing a variety of NLU tasks for Turkish. In addition, we present SentiTurca, a specialized benchmark for sentiment analysis. To support researchers, we also provide fine-tuning and evaluation code for transformer-based models, facilitating the effective use of these benchmarks. TrGLUE comprises Turkish-native corpora curated to mirror the domains and task formulations of GLUE-style evaluations, with labels obtained through a semi-automated pipeline that combines strong LLM-based annotation, cross-model agreement checks, and subsequent human validation. This design prioritizes linguistic naturalness, minimizes direct translation artifacts, and yields a scalable, reproducible workflow. With TrGLUE, our goal is to establish a robust evaluation framework for Turkish NLU, empower researchers with valuable resources, and provide insights into generating high-quality semi-automated datasets.

1 Introduction

Turkish NLU evaluation lacks a comprehensive, standardized benchmark, and existing Turkish datasets are scattered across tasks. The paper introduces TrGLUE and SentiTurca, with Turkish-focused datasets and a semi-automated annotation workflow.

1 Introduction

  • Turkish resources are scarce for evaluation, and existing evaluations lack comprehensive coverage, reproducibility, and a shared standard for comparing models.The authors also identify translation-related risks, including unnatural text, cultural bias, and failure to reflect Turkish agglutination.Source: §1 · p. 2
  • TrGLUE provides a standardized Turkish benchmark spanning text classification and sentence-pair classification, with STS-B handled as regression.TrSTS-B has no released test set because of limited dataset size; evaluations use its development split.Source: §1 · p. 2, Table 1 · p. 4
  • The benchmark mirrors GLUE-style task designs and domains while sourcing Turkish-native text from resources such as Wikipedia, news, social platforms, and forums.TrSTS-B is the exception: it is translated and reviewed by humans; other datasets avoid direct translation.Source: §1 · pp. 2–3
  • A disagreement-driven workflow combines classifier and LLM label hypotheses, directs disagreements to human annotation, and spot-checks agreement cases.The paper also provides documentation and prompts to support reproducible dataset creation and work on benchmarks for other languages.Source: §1 · p. 3
  • SentiTurca brings together sentiment datasets across domains and includes the Turkish Hate Map, described as the most extensive Turkish hate-speech dataset.The contributions also include fine-tuning and evaluation scripts for using the benchmarks.Source: §1 · p. 3

2 Related Work

Existing multilingual benchmarks include Turkish only in limited or relatively small datasets, while Turkish NLI efforts have relied substantially on translated data. The paper positions its work as a native-first, semi-automated alternative with documented provenance and licensing.

2 Related Work

  • Multilingual benchmarks such as XGLUE, XTREME, and XTREME-R include Turkish only through limited and relatively small datasets.Earlier GLUE-inspired benchmarks target multiple languages, including Chinese, French, Korean, Japanese, and others.Source: §2 · p. 4

2.1 Constructing NLI datasets for Turkish

  • XNLI includes Turkish among 15 languages using translated MultiNLI instances, while another Turkish NLI effort translated SNLI and MultiNLI with Amazon Translate and human review.The latter study reported high quality and treated machine translation as a feasible, efficient construction approach.Source: §2.1 · p. 5
  • The authors’ stated advances are native Turkish sourcing except for TrSTS-B, LLM-assisted annotation, and full provenance and licensing information.Their workflow uses classifier–LLM disagreement to route items to human annotators, with spot checks for agreement; TrCoLA is hand-crafted, TrRTE hypotheses are human-written, and TrWNLI is omitted.Source: §2.1 · pp. 5–6
  • The paper says its approach produces high-quality datasets quickly and at lower cost while preserving linguistic naturalness, and covers GLUE-style classification and TrSTS-B regression.Table 2 compares dataset sizes between GLUE and translated datasets and reports the percentage retained after translation, but the supplied passages give no cell values.Source: §2.1 · p. 6, Table 2 · p. 6

3 TrGLUE Benchmark

This section presents TrGLUE’s Turkish-native GLUE-style classification and inference datasets, describing their task-specific construction and adaptations, including the omission of a WNLI-style split. It also characterizes their linguistic and corpus properties, including morphology, syntax, entity profiles, and greater subword fragmentation than in English GLUE.

3 TrGLUE Benchmark

  • TrGLUE covers Turkish counterparts to GLUE tasks except WNLI, using native human-written sources and a generally LLM-assisted, human-verified workflow.TrRTE hypotheses were human-written under linguist-designed guidelines, TrCoLA was hand-crafted by linguists, TrSST-2 labels came from review ratings with spot-checks, and TrSTS-B used translate-then-edit.Source: §3 · p. 6

3.1.1 TrCoLA

  • TrCoLA achieved Krippendorff’s Alpha of 0.91, with each instance judged by four annotators and accepted only with at least three agreeing votes.The Turkish textbook- and linguistics-book-based dataset is described as the first of its kind for Turkish, with a standalone release retaining sentence, variation, violation type, and label.Source: §3.1.1 · pp. 7–8

3.1.2 TrSST-2

  • TrSST-2 contains about 78K film reviews, excluding six-star ratings and labeling ratings below five negative and above five positive.Its star distribution is skewed, motivating binary accuracy and F1-score; most reviews are 0–50 words and average 48 words.Source: §3.1.2 · p. 8, Figure 1 · p. 8

3.2.1 TrMRPC

  • ≈5.18k Tr-MRPC pairs comprise ≈1.6k paraphrases and ≈3.5k non-paraphrases, with labels finalized through human review and expert adjudication.Pairs count as paraphrases when they assert the same main claim despite non-contradictory detail; changes to truth conditions, polarity, modality strength, or key quantities remain non-paraphrases.Source: §3.2.1 · pp. 9–10, Labeling policy difference from… · p. 11

3.2.2 TrSTS-B

  • TrSTS-B contains 3.06K pairs and withholds a test set, reporting evaluation on the development split.It uses human-verified translate-then-edit with cultural substitutions, while removing pairs that collapse to identical Turkish sentences and filtering duplicates.Source: §3.2.2 · pp. 11–12

3.2.3 TrQQP

  • TrQQP has 219,329 non-duplicate and 149,465 duplicate pairs, retaining informal orthography and emojis from heterogeneous Turkish Q&A and opinion platforms.Annotators judged about 35K uncertain pairs and audited another 5K high-confidence pairs; question clusters are kept within individual splits to prevent leakage.Source: §3.2.3 · pp. 12–14

Dataset compilation

  • TrMNLI uses eight BellaTurca-sourced genres: four matched genres for training and in-domain evaluation, and four mismatched genres for out-of-domain evaluation.The eight-genre design preserves matched/mismatched generalization testing while spanning narrative, evaluative, expository, and conversational registers.Source: Dataset compilation · pp. 14–15

Construction methodology

  • TrMNLI uses Snowflake Arctic to generate entailment, neutral, and contradiction hypotheses, followed by human validation, double annotation on a subset, and expert adjudication.Construction balances lexical overlap across labels, caps overt-negation contradictions, and deduplicates near-duplicates; the described prompts use factuality-focused, linguistic-variation, and free styles.Source: Construction methodology · p. 15, Diversity of linguistic phenomenon · p. 15

Diversity of linguistic phenomenon

  • TrMNLI spans factual reasoning, paraphrasing, and structural variation beyond lexical overlap, reflecting Turkish expressive morphology and flexible word order.Examples include entailment, contradiction, and neutral relations as well as paraphrasing, summarization, syntactic restructuring, negation, and added details.Source: Diversity of linguistic phenomenon · p. 16

3.2.5 TrQNLI

  • About 140K validated Turkish question-answer-context triplets form TrQNLI, converted into sentence-pair entailment classification with matched negatives and balanced splits.The triplets were generated from approximately 24K Turkish Wikipedia paragraphs, then filtered and human-reviewed; the final sentence-pair splits avoid paragraph leakage.Source: §3.2.5 · p. 17

3.2.6 TrRTE

  • TrRTE uses native-speaker-written hypotheses in linguistic, free-paraphrase, and factuality styles, with entailment/not-entailment labels balanced across splits.A separate annotator validation and adjudication process is used; artifact controls include balancing lexical overlap, diversifying contradiction mechanisms, and deduplicating near-duplicates.Source: §3.2.6 · p. 18

3.2.7 Absence of TrWNLI: Why WNLI Does Not Transfer to Turkish

  • TrWNLI is omitted because Turkish pro-drop, case and agreement morphology, and flexible information structure remove or distort Winograd-style ambiguity and undermine construct validity.The paper documents this linguistic rationale and says Turkish reference-related reasoning is assessed through more appropriate tasks elsewhere.Source: §3.2.7 · pp. 19, 21

3.3 Comparison of corpus statistics

  • Turkish datasets generally have fewer words per instance but more unique surface forms relative to lemma counts; these patterns motivate lemma-aware filtering and morphology-aware processing.Pairwise tasks compress more in Turkish, while subword tokenization may still increase sequence lengths; the authors release scripts and settings for replication.Source: §3.3 · p. 22, Table 6 · p. 22

3.4.1 Morphological richness and coverage

  • 2.25 morphemes per token is TrGLUE’s mean (median 2; p95 5; p99 8), alongside 1,095 distinct inflectional feature bundles.Task profiles range from a 2.14 mean in CoLA to 2.50–2.58 in MNLI, STS-B, and RTE; bundle counts range from 340 in CoLA to 1,095 overall.Source: §3.4.1 · p. 23, §3.4.4 · pp. 24–25

3.4.2 Syntax and word order variation

  • 73.64% of TrGLUE finite clauses omit subjects, versus 0.5% in English GLUE; Turkish also has 3.07% noncanonical S/O/V order and 3.26-token dependency distance.The English comparison is 1.1% noncanonical order and 2.5-token dependency distance, consistent with the reported contrast between Turkish flexibility and English’s more rigid profile.Source: §3.4.2 · pp. 23–24, Figure 2 · p. 25

3.4.3 Named entities

  • 9.2% of TrGLUE tokens are named-entity tokens; morphology profiles also vary by task, from 2.14 mean morphemes per token in CoLA to 2.50–2.58 in MNLI, STS-B, and RTE.Inflectional feature bundles range from 340 in CoLA to 1,095 overall, while named-entity counts use a unified NER pipeline and tokenization regime.Source: §3.4.3 · p. 24, §3.4.4 · pp. 24–25

3.4.4 Task-relevant phenomena

  • Task-specific syntax ranges from canonical SOV and high pro-drop in CoLA to more variable word orders in RTE and STS-B.QNLI has elevated partial orders and slightly longer dependencies, whereas QQP has the shortest dependencies; overall, SOV dominates with occasional flexible alternations.Source: §3.4.4 · p. 26
  • Named-entity density is highest in TrQNLI (16.3%) and TrMRPC (15.0%), supporting greater potential payoff from entity-aware modeling in those tasks.RTE has 9.2% entity tokens, MNLI 8.4%, QQP 6.4%, CoLA 6.3%, and STS-B 2%; QQP and CoLA primarily test generalization beyond explicit named references.Source: §3.4.4 · pp. 26–28
  • TrGLUE retains an agglutinative backbone, while subsets vary in morpheme-tail weight and functional morphology.The pooled “Overall” aggregate has 11.75% negation and 9.21% evidentiality among finite verbs; genre mixture and corpus breadth also expand inflectional combinations. Heavier tails can lengthen effective contexts, while lighter subsets emphasize lexical paraphrase and local morphosyntactic cues.Source: §3.4.4 · p. 26, Figure 3 · p. 27

3.4.5 Tokenization behavior

  • 1.58 subwords per token for BERTurk on TrGLUE compares with 1.29 for BERT on GLUE, reflecting greater Turkish wordpiece fragmentation.Higher fragmentation increases effective sequence length and compute under the same maximum length, while improving coverage of rare forms.Source: §3.4.5 · p. 28

4 Evaluation using TrGLUE

TrGLUE evaluation found a broadly balanced BERTurk profile across tasks, with stronger performance than BERT on RTE but weaker performance than GLUE baselines on STS-B. Fractional-data curves show early plateaus on several tasks, while TrSTS-B continues improving; CoLA results also expose differences in how models handle Turkish grammar.

4 Evaluation using TrGLUE

  • BERTurk outperforms BERT on RTE but underperforms the corresponding GLUE baselines on STS-B, producing a broadly balanced cross-task profile.Table 7 compares BERT-base and RoBERTa-base on GLUE with BERTurk on TrGLUE; SST-2 reports both accuracy and F1 because of class imbalance.Source: §4 · p. 28, Table 7 · p. 28
  • Classification tasks reach at least 95% of full-data scores by 0.6–0.8× training data, while TrSTS-B continues rising at 1.0×.TrRTE plateaus around 0.8×; Figure 4 plots fraction of training data against fraction of full-data score.Source: §4 · p. 29, Figure 4 · p. 29

TrRTE vs RTE

  • 0.92 at full data for TrRTE compares with roughly 0.67 for English RTE under the same recipe, with TrRTE plateauing by 0.6–0.8×.English RTE’s mean accuracy from 0.6–1.0× varies by roughly 2–3 points, with overlapping confidence bands; the authors attribute the gap to dataset adequacy rather than size or translation effects.Source: TrRTE vs RTE · pp. 29–30, §4 · p. 29

STS-B

  • TrSTS-B has lower absolute correlation than English STS-B and keeps improving through 1.0×, unlike the earlier saturation of classification tasks.The authors cite its roughly one-third size, Turkish surface variation, cultural substitutions, and filtering of near-identical pairs as factors; they describe the gap as measurement granularity and scale, not task mismatch.Source: STS-B · p. 30

4.1.1 CoLA

  • LLaMa 3 70B and Qwen2-72B are the TrCoLA frontrunners, while Claude 3 Sonnet, GPT-4, and Gemini Pro perform poorly.The task was challenging for human evaluators as well as models.Source: §4.1.1 · p. 31, Table 8 · p. 31
  • Gemini failed to distinguish acceptable from unacceptable sentences, and Claude 3 Sonnet had the poorest performance.Gemini often labeled sentences acceptable and struggled with semantic, morphological, and some syntactic variations; GPT-4 Turbo showed some ability to identify unacceptable sentences.Source: §4.1.1 · p. 31, Figure 5 · p. 32
  • LLaMa 3 70B rose from an MCC of 0.05 to 0.35 with chain-of-thought prompting, while Qwen2-72B explained a Turkish suffix error correctly.For the same test sentence, LLaMa relied on word statistics, whereas Qwen2 supplied a grammatical explanation.Source: §4.1.1 · p. 32, Figure 6 · p. 33
  • Qwen2-72B was the only LLM described as handling intricate Turkish morphology and syntax, while BERTurk identified morphological errors but struggled more with syntactic and semantic variation.Claude 3 Sonnet’s rationale relied on corpus statistics and failed to identify -(s)I as a possessive suffix when asked directly.Source: §4.1.1 · p. 34

5 SentiTurca

SentiTurca combines movie reviews, e-commerce customer reviews, and a Turkish hate-speech dataset. Its components span varied targets and text styles, with the Hate Map annotation process revised after poor initial agreement.

5 SentiTurca

  • SentiTurca includes movie reviews, e-commerce customer reviews, and a human-annotated hate-speech dataset, the Turkish Hate Map.The movie-review component is the same dataset as TrSTS-2; the reviews were initially rated by reviewers, while Co-one annotators labeled the hate-speech data.Source: §5 · p. 34, §5.1 · p. 35

5.2 Customer reviews

  • The customer-review corpus contains 103K reviews from Hepsiburada and Trendyol, with mismatched star-rating reviews filtered and 60% of ratings at four or five stars.Reviews average 9.39 words and 70.26 characters; the authors describe the collection as the most substantial published Turkish customer-review dataset, noting prior scale, class, or release limitations.Source: §5.2 · pp. 35–36

5.3 Turkish Hate Map

  • The Turkish Hate Map contains 52K Ekşi Sözlük entries across 13 target categories, averaging 57 words and 453 characters per instance.Most texts are offensive or neutral; refugee-category entries are an exception, with hate speech predominant and offensive language close behind.Source: §5.3 · pp. 36–38

5.4.1 Data compilation

  • Hate Map texts were gathered by curating topical headlines, scraping associated entries, then randomizing and sampling the collected material.Ekşi Sözlük’s unmoderated platform and colloquial style provide profanity, slang, everyday language, and often lengthy arguments.Source: §5.4.1 · p. 39

5.4.2 Choice of target groups

  • The target categories cover culturally significant groups and underexplored targets, including animals and occupational groups, to represent varied hate-speech contexts.The authors connect category breadth to culturally relevant hate detection and issues such as animal hate and occupational discrimination.Source: §5.4.2 · pp. 39–40

5.4.3 Choice of labels

  • The authors add “civilized” as a neutral subgroup for texts that promote constructive dialogue and provide insights.They identify the subgroup through sentence-encoder clustering and observations that its texts tend to be longer and more structured.Source: §5.4.3 · p. 40

5.4.4 Data annotation

  • After first-round annotations had an ICC of 61.2 and were discarded, revised guidelines raised agreement to an ICC of 91.2.The initial labels often misclassified offensive texts as neutral; Snowflake Arctic was also found unsatisfactory for annotation automation.Source: §5.4.4 · pp. 40–41

6 Evaluation using SentiTurca

SentiTurca evaluations show varied model performance across movie reviews, customer reviews, and Turkish Hate Map classification. Errors on the Hate Map often involve subtle offensive content and culturally nuanced language.

6.1 Movie reviews

  • BERTurk scored 87.4 binary accuracy and 91.3 F1 on movie reviews, without a specific intervention for class imbalance.Improving this baseline is left as an open challenge.Source: §6.1 · p. 41

6.2 Customer reviews

  • On customer reviews, BERTurk achieved 0.66 accuracy and 0.64 macro F1, with errors concentrated between adjacent star ratings.The reported confusions include 1-star versus 2-star and 4-star versus 5-star reviews.Source: §6.2 · p. 41
  • Gemini Pro classified all customer reviews correctly, while some prominent LLMs performed below BERTurk in the single-shot setting.The authors suggest Gemini Pro may have encountered examples from the review website during pretraining.Source: §6.2 · pp. 41–42

6.3 Turkish Hate Map

  • Hate Map classification is difficult because hateful content may lack explicit insults, while curse words can appear in non-offensive text.Cultural references add another source of complexity for interpretation.Source: §6.3 · p. 42
  • BERTurk struggled more with offensive than neutral Hate Map items, including subtle “polite hate” misclassified as neutral or civilized.Civilized examples could also be labeled neutral or offensive when lengthy texts contained an offensive-toned sentence.Source: §6.3 · pp. 43–44
  • GPT-4 and LLaMa 3 improved across offensive and neutral Hate Map categories, whereas Qwen2 often mislabeled neutral instances as offensive.Gemini Pro and Claude also confused some neutral instances with offensive or hate labels and sometimes labeled offensive items as hate.Source: §6.3 · pp. 44–45

7 Conclusion

The paper presents TrGLUE as a Turkish general language understanding benchmark and SentiTurca as a Turkish sentiment analysis benchmark. It aims for TrGLUE to support pretrained-model evaluation and future dataset development.

7 Conclusion

  • The paper details the creation of TrGLUE for Turkish language understanding and SentiTurca for Turkish sentiment analysis.The authors aim for TrGLUE to serve as a comprehensive evaluation tool for pretrained models and facilitate development of more challenging NLU datasets, such as Adversarial GLUE [68].Source: §7 · p. 45
  • Future work includes expanding Turkish datasets, particularly instruction fine-tuning datasets, and encourages researchers to share Turkish data.Source: §7 · p. 46

8 Legal and Ethical Concerns

The authors describe licensing and data-acquisition choices intended to keep the datasets commercially usable and to limit the impact of web crawling.

8 Legal and Ethical Concerns

  • Snowflake Arctic was selected as commercially usable for derivative works, and translated datasets are available under permissive licensing terms.Source: §8 · p. 46
  • The authors followed websites’ robots.txt files and scraped limited data to avoid overwhelming servers.They describe their collection approach as aligned with legal and ethical standards.Source: §8 · p. 46
  • Translations and LLM assessments used Poe.com under a two-month subscription costing $40, with inputs kept within platform and model length limits.Source: §8 · p. 46

A.1 Prompting and human annotations

The appendix documents prompts and annotation guidance for constructing Turkish task data, including generation, translation, label preservation, and validation procedures.

A.1.1 TrCoLA

  • TrCoLA prompting asks Arctic for morphological, syntactic, and semantic violation variants of Turkish sentences.Source: §A.1.1 · p. 47

A.1.2 Translation prompting for sentence-pair tasks

  • Sentence-pair translation prompting formats each pair with a vertical bar and separates pairs with double hashes.Source: §A.1.2 · p. 47

A.1.3 TrMRPC

  • TrMRPC guidelines label pairs as paraphrases when they preserve the same main claim, and as non-paraphrases when substitution changes that claim.The prompt requires one binary label per pair and no extra text.Source: §A.1.3 · pp. 47–49

A.1.4 TrSTS-B

  • TrSTS-B guidance uses minimal edits to make pairs idiomatic and culturally appropriate while preserving their supplied similarity score.The guidance maps score ranges to near-identical, closely related, loosely related, or unrelated meanings.Source: §A.1.4 · pp. 49–50

A.1.5 TrQQP

  • TrQQP prompting generates a second Turkish question for each labeled question, favoring idioms and cultural references for duplicates.Source: §A.1.5 · p. 51

A.1.6 TrMNLI

  • TrMNLI uses prompts for factuality and linguistic variation, with hypotheses designed to match entailment, contradiction, or neutral labels.The guidance also calls for matching premise tone and style.Source: §A.1.6 · pp. 52, 53, 55

A.1.7 TrQNLI

  • TrQNLI construction first generates SQuAD-style context-question-answer triplets, with answers drawn from text spans and indexed by their start positions.Source: §A.1.7 · p. 56

A.1.8 RTE annotation guidelines

  • RTE guidance calls for multiple generation styles and independent labels, with no-consensus items excluded from evaluation.Styles include linguistic phenomena, free paraphrasing, and factual edits; the guidance also specifies majority voting and split-level label balance.Source: §A.1.8 · pp. 56–57

A.2.1 CoLA

The examples illustrate Turkish naturalness judgments, sentiment labels, and sentence-pair equivalence judgments. They include both positive and negative sentiment and both equivalent and non-equivalent pairs.

A.2.1 CoLA

  • CoLA examples contrast an acceptable vowel-harmony sentence with unacceptable statements about Ali Şir Nevayi and Ottoman Turkish.The labels are presented directly after each sentence.Source: §A.2.1 · p. 59

A.2.2 TrSST-2

  • TrSST-2 examples pair Turkish film reviews with star ratings, ranging from 4 to 9 stars.The supplied reviews include favorable and critical opinions.Source: §A.2.2 · p. 59

A.2.3 TrMRPC

  • TrMRPC examples label a report about Iran and Hamas as equivalent, but a first-person account of a petition as not equivalent.Source: §A.2.3 · p. 59

A.2.4 TrSTS-B

The examples show how Turkish sentence pairs are labeled for similarity, duplication, entailment, contradiction, and relatedness. They span scores and categorical judgments across five tasks.

A.2.4 TrSTS-B

  • TrSTS-B assigns 5.0 to two paraphrases of a plane taking off and 3.0 to a pair contrasting a man bicycling with a monkey riding a bicycle.Source: §A.2.4 · p. 59

A.2.5 TrQQP

  • TrQQP marks two differently phrased questions about getting a sweetheart to buy a rose as duplicates, unlike unrelated tax-revenue and investment questions.Source: §A.2.5 · p. 60

A.2.6 TrMNLI

  • TrMNLI labels a paraphrased account of Israel's pandemic tourist restrictions as entailment, a conflicting Steam-price claim as contradiction, and an unrelated pair as neutral.Source: §A.2.6 · p. 60

A.2.7 TrQNLI

  • TrQNLI labels a question about French intentions as not entailed by a passage describing a 1945 parliamentary raid, while a battery-type question is entailed by its passage.Source: §A.2.7 · p. 60

A.2.8 TrRTE

  • TrRTE examples distinguish an entailed paraphrase about an author receiving many awards from an unrelated statement about Lake Van.Source: §A.2.8 · p. 60

Evaluation of CoLA dataset with LLMs

LLM evaluation used GPT-4 Turbo on Poe.com, with test instances sent in batches of ten and a plainly described WNLI prompt. The included SentiTurca examples span customer reviews and socially sensitive topics.

Evaluation of CoLA dataset with LLMs

C.1 Customer reviews

  • SentiTurca customer-review examples pair product opinions with star ratings from 1 to 5.Source: §C.1 · p. 61

C.2.1 Animals; C.2.2 Cities

  • Animal and city examples include neutral pet-care advice and personal recollection, alongside offensive or hateful language about animals and Erzurum.Source: §C.2.1 · p. 62, §C.2.2 · p. 62

C.2.3 Ethnic groups; C.2.4 LGBT

C.2.5 Misogyny; C.2.6 Occupations

  • Misogyny and occupation examples include neutral descriptions as well as offensive remarks about women and hateful generalizations about real-estate agents.Source: §C.2.5 · p. 63, §C.2.6 · p. 63

C.2.7 Politics; C.2.8 Political orientation; Cultural reference including instances

C.2.9 Refugees

  • Refugee examples range from neutral concern about future consequences to hateful statements about Syrian refugees and landlords.Source: §C.2.9 · p. 64

C.2.10 Sects; C.2.11 Vegans-vegetarians

  • Sect and vegan examples include neutral reflections or self-description alongside offensive stereotypes and criticism of veganism.Source: §C.2.10 · p. 64, §C.2.11 · p. 64

D.1 Turkish Hate Map

The Turkish Hate Map appendix presents misclassified examples across true-label categories and compares model predictions. The examples show disagreements as well as cases where a model is described as correct.

D.1 Turkish Hate Map

  • The appendix presents misclassified instances per model, with predicted labels shown in bold.Source: §D.1 · p. 65

True label: offensive

  • Examples with true label offensive include a neutral prediction for political wordplay and an offensive prediction for a comment about a television host.Source: True label: offensive · p. 65

True label: hate

True label: neutral

  • True-neutral examples include hateful remarks about deportation and an offensive label for a long account of kindness on public transport.Source: True label: neutral · p. 66

True label: civilized

D.1.2 Gemini 1.0 Pro and Claude 3.5 Sonnet

  • On a Turkish statement labeled hate about Syrian refugees taking jobs, Gemini Pro labeled it neutral, while Claude labeled it offensive, illustrating a direct disagreement.Source: §D.1.2 · p. 67

D.1.3 GPT-4 Turbo and LLaMa 3 70B

  • GPT-4 and LLaMa 3 also differ: a true-civilized tax post is labeled hate by GPT-4 and offensive by LLaMa 3, while another instance is marked correctly by LLaMa 3.Source: §D.1.3 · p. 69

Misclassified neutral instances

Related papers

Loading 2512.22100v1…