Source-linked AI summary
TurEngMix: A Text Corpus and Benchmark for Turkish-English Code-Mixed Language Identification and Named Entity Recognition
Ilayda Dogan, Phuong-Anh Nguyen-Le, Julia Mendelsohn
TL;DR
Code-mixed NLP remains underexplored for lower-resourced language pairs, and Turkish-English adds mixed tokens with English roots and Turkish suffixes. This paper introduces a corpus and annotated LID/NER benchmark, then finds that morphologically mixed tokens remain a persistent challenge for multilingual NLP systems. It suggests morphology-aware approaches as a direction for progress on code-mixed NLP for agglutinative languages.
Problem
Code-mixing remains underexplored in NLP for lower-resourced language pairs, and Turkish-English suffixation creates mixed-language tokens that challenge LID and NER.
Source: §1 · pp. 1–2Method
The paper introduces TurEngMix and an expert-annotated Turkish-English LID and NER benchmark, and evaluates decoder LLM and fine-tuned encoder baselines.
Source: §1 · p. 2, §3.2 · p. 3Results
Morphologically mixed tokens remain a persistent challenge for multilingual NLP systems, while Organization and Product are frequently confused NER categories.
Source: §6.2 · p. 7, §7 · p. 8Takeaways & Limitations
The findings suggest morphology-aware approaches, such as sub-token language attribution or morphological segmentation as preprocessing, may be needed for code-mixed NLP on agglutinative languages.
Source: §7 · p. 8Because GPT-4o helped filter the corpus for English material, that selection may favor its LID performance relative to models uninvolved in dataset creation.
Source: §8 · p. 9
Abstract
from arXiv · showhide
Natural language processing systems underperform on code-mixed text, particularly for low-resource language pairs. Turkish-English poses a further challenge: it lets English stems combine with Turkish suffixes to form single mixed-language tokens. We introduce TurEngMix, a corpus of 5.5K noisy, naturally occurring social media posts (486,974 tokens) rich in Turkish-English code-mixing. From this corpus, we construct a new Turkish-English benchmark for code-mixed language identification (LID) and named entity recognition (NER), comprising 15K expert-annotated tokens. Evaluating both decoder LLM and fine-tuned encoder baselines, we find that monolingual Turkish and English tokens are labeled reliably, but all models have high error rates on mixed-language tokens for both LID and NER. For morphologically integrated tokens, NER error rates were 5.2x and 6.3x higher for GPT-4o and Qwen, respectively. This highlights how morphological integration remains a challenge. We release the corpus, annotations, and code to support future computational and sociolinguistic research on Turkish-English code-mixing.
1 Introduction
Code-mixing is widespread and socially functional, yet remains underexplored in NLP, especially for lower-resourced language pairs. Turkish-English adds a distinctive challenge: English roots can take Turkish suffixes within mixed-language tokens, while annotated resources for evaluating these forms remain scarce.
1 Introduction
- NLP research on code-mixing remains limited, particularly for lower-resourced language pairs, despite code-mixing’s linguistic and social functions.The passage names identity signaling, tone-setting, and communication of culturally or technically specific concepts.Source: §1 · p. 1
- English roots combined with Turkish suffixes create mixed-language tokens that challenge LID and NER sequence labeling.This morphology distinguishes Turkish-English from code-mixing that typically occurs at word boundaries.Source: §1 · p. 1, Table 1 · p. 1
- Only one publicly available Turkish-English code-mixed dataset is identified, and the authors know of no resources for NER or LID with mixed-language tokens.The identified dataset comprises 391 naturally occurring examples with word-level language labels.Source: §1 · pp. 1–2
- The paper contributes a naturally occurring social-media corpus, a 15K-token annotated benchmark, and systematic LLM and encoder baseline evaluations.The corpus, annotations, and code are publicly available for noncommercial use.Source: §1 · p. 2
2 Background
Turkish-English code-mixing often combines English content stems with Turkish suffixes, creating forms that remain challenging for NLP. Existing code-mixing resources and evaluations cover few language pairs, and TurEngMix adds a benchmark for Turkish-English LID and NER.
2.1 Turkish-English Code-Mixing
- English stems commonly take Turkish inflectional or derivational suffixes, producing mixed-language tokens that challenge existing NLP systems.The passage describes Turkish as providing the matrix frame and English as supplying content stems.Source: §2.1 · p. 2
2.2 Code-Mixing Resources and Evaluation
- Most code-mixing resources focus on Spanish-English and Hindi-English, while TurEngMix jointly annotates Turkish-English text for token-level LID and NER.The paper calls it the first such benchmark on naturally occurring forum text and the largest such resource with metadata for English-origin stems bearing Turkish morphology.Source: §2.2 · pp. 2–3
3 TurEngMix Corpus
TurEngMix is built from public posts on Ekşi Sözlük, filtered to enrich Turkish-English code-mixing, and released with annotations and preprocessing scripts. The resulting corpus spans thousands of posts, hundreds of thousands of tokens, and a range of online topics and post lengths.
3.1 Data Collection
- The corpus draws on public Ekşi Sözlük posts collected across topic tags selected to surface code-mixing, without retaining personally identifying metadata.The collected posts were text strings; the filtered dataset, token-level annotations, and preprocessing scripts are publicly available.Source: §3.1 · p. 3
3.2 Data Filtering
- The three-stage filtering pipeline uses langdetect, two GPT-4o checks for English material and false positives, then excludes posts under three or over 500 words.The second GPT-4o check excludes posts whose apparent English consists solely of URLs, brand names, or other non-linguistic strings.Source: §3.2 · p. 3
- Manual inspection found code-mixing in 71 of 100 retained posts versus 22 of 100 discarded posts, showing enrichment rather than exhaustive capture.Retained posts could still be monolingual, while some code-mixed posts were discarded; monolingual posts provide negative examples.Source: §3.2 · p. 3
3.3 Corpus Statistics
- 5,549 posts and 486,974 tokens make up the final corpus; most posts are under 100 words, with lengths extending to 500.Mean post length is 87.75 words and median length is 57 words; the corpus spans 71 topic tags, with technology and social-media topics among the most frequent.Source: §3.3 · pp. 3–4, Figure 2 · p. 4, Figure 3 · p. 4
4 LID and NER Annotated Benchmark
The paper builds a token-annotated Turkish-English LID and NER benchmark from naturally occurring social media posts. It captures varied code-mixing, including integrated tokens that are relatively rare by token count but appear across many posts.
4 LID and NER Annotated Benchmark
- The benchmark contains 15,012 annotated tokens drawn from naturally occurring Turkish-English social media posts.The authors say the benchmark preserves characteristics of authentic code-mixed discourse.Source: §4 · p. 4
4.1 Benchmark Construction
- Posts were sampled and tokenized with whitespace and punctuation boundaries while preserving alphanumeric strings and morphologically mixed tokens.Apostrophes and quotation marks were retained because they help form morphologically complex words.Source: §4.1 · p. 4
4.1.1 Annotation Scheme
- LID uses six labels, while NER uses nine entity types in BIO format; named entities receive the LID label NE regardless of language or Turkish suffixes.The benchmark also marks English-origin words, including named entities and brands, that bear Turkish suffixes as INTEGRATED.Source: §4.1.1 · p. 4
4.1.2 Annotation Process
- Agreement on a 203-token sample was κ = 0.96 for LID and κ = 0.94 for NER, with the lowest per-class agreement on MIXED (κ = 0.75).Disagreements included lexicalized English loanwords and related entity categories; discussion resolved them for the final annotations.Source: §4.1.2 · p. 5
4.2 Benchmark Statistics
- Integrated tokens make up 2.57% of annotated tokens but occur in 66.0% of posts; 87.2% of posts contain more than one language.The mean document-level Code-Mixing Index is 13.25 (SD = 13.12); the benchmark also includes isolated English insertions and longer embedded English phrases.Source: §4.2 · p. 5
- Encoder splits use an 80/10/10 post-level partition to limit information leakage, though some rare NER labels are absent from the test set.The split distributions are similar across LID and NER labels.Source: §4.2 · pp. 5–6
5 Baseline Evaluations
The baseline evaluation compares decoder LLMs and fine-tuned transformer encoders on LID and NER. Decoder models receive complete posts with pre-tokenized, indexed tokens, while encoders are trained for token classification.
5 Baseline Evaluations
- The baseline evaluation covers two sequence-labeling tasks, LID and NER, using pretrained encoder and decoder models.Source: §5 · p. 6
5.1 Decoder Setup
- For LID and NER, models received complete posts as pre-tokenized indexed-token sequences and predicted one label per token in zero-shot and three-shot settings, with predictions aligned by token index before scoring.Predictions covered the 250-post benchmark and were compared with gold-standard annotations; results were reported as macroaveraged and per-label F1 for both tasks.Source: §5.1 · p. 6
5.2 Encoder Setup
- XLM-RoBERTa, BERTurk, and TurkishBERTweet are fine-tuned on the training split, with a majority-class baseline for comparison.Tokens are segmented into subwords, and labels apply only to the first subword; reported test results average four random-seed runs.Source: §5.2 · pp. 6–7
6 Results
Across LID and NER, models handle monolingual Turkish and English better than mixed-language tokens, while NER outcomes vary with prompting and entity type. Error analyses identify Turkish suffix integration as a recurring challenge for both tasks.
6.1 Language Identification
- All models label Turkish and English tokens well (F1 > 0.84), but MIXED-token F1 is 0.45 for GPT-4o and 0.22 for the best encoder, TurkishBERTweet.GPT-4o’s three-shot LID macro-F1 is 0.5238 versus 0.5170 zero-shot; Qwen3-8B’s is 0.4788 versus 0.4920.Source: §6.1 · p. 7, Table 8 · p. 8, Table 9 · p. 8
6.2 Named Entity Recognition
- Three-shot prompting raises GPT-4o’s BIO-level NER macro F1 from 0.5669 to 0.6140, while encoder models have lower macro F1 despite high token accuracy.Non-entity tokens account for approximately 93% of the dataset.Source: §6.2 · p. 7, Table 10 · p. 8
- GPT-4o reaches an NE-class F1 of 0.80 with three-shot prompting, compared with 0.70 for Qwen; encoder models range from 0.47 to 0.67.Collapsing entity categories into NE and Non-NE assesses detection separately from entity classification and boundary prediction.Source: §6.2 · p. 7
- Person, Location, and Title generally score higher than Organization, Other, Event, and Time; Organization and Product are the most frequently confused categories.Technology-related entities such as Google, Netflix, and Microsoft are particularly prone to this confusion.Source: §6.2 · p. 7
6.3.1 LID
- In LID, integrated English borrowings are often mislabeled as Turkish, while suffixed named entities are often labeled MIXED rather than NE.English hyphenated compounds without Turkish morphology are also sometimes labeled MIXED.Source: §6.3.1 · p. 7
6.3.2 NER
- NER errors include Product–Organization confusion for technology entities and missed integrated entities such as reactlerle and python’daki.Source: §6.3.2 · p. 7
6.3.3 Morphological Integration in Named Entities
- GPT-4o errors occur on 19.7% of mixed tokens versus 3.8% of other tokens; Qwen’s rates are 33.5% versus 5.3%.Source: §6.3.3 · pp. 7–8
7 Conclusion
TurEngMix addresses gaps in Turkish-English code-mixed NLP with a corpus, an annotated LID and NER benchmark, and model evaluations. The conclusion identifies morphologically mixed tokens as a persistent challenge and suggests morphology-aware approaches for agglutinative languages.
7 Conclusion
- Morphologically mixed tokens remain a persistent challenge for multilingual NLP systems in code-mixed settings.The authors suggest morphology-aware approaches, such as sub-token language attribution or morphological segmentation, rather than scale alone.Source: §7 · p. 8
- The work introduces a Turkish-English code-mixed corpus, an expert-annotated LID and NER benchmark, and systematic evaluations of LLM and encoder-based models.Source: §7 · p. 8
- The cited supplemental analyses cover binary named-entity detection and compare NER error rates for morphologically integrated English borrowings versus other evaluated tokens.Source: Table 11 · p. 8, Table 12 · p. 8
- The conclusion suggests morphology-aware approaches, including sub-token language attribution or morphological segmentation, as directions for agglutinative-language code-mixed NLP.Source: §7 · p. 8
8 Limitations
The authors identify limits involving the corpus's source and selection, possible pretraining contamination, annotation and prompting choices, and class imbalance.
8 Limitations
- The corpus most accurately represents code-mixing on Ekşi Sözlük and is not intended to estimate its prevalence in Turkish online communication.Topic tags likely to contain Turkish-English code-mixing were prioritized during construction.Source: §8 · p. 8
- Because the site is public, pretraining contamination is possible; the authors suggest extending future work to other communities or settings.Source: §8 · p. 9
- GPT-4o's corpus-filtering role may favor its LID performance, despite a langdetect first stage and manual inspection of retained and discarded posts.The authors say GPT-4o results should be interpreted with this selection effect.Source: §8 · p. 9
- A separate NE category can obscure named entities with Turkish morphological integration, and prompts lacked demonstrations for every annotation category.The authors partially address the first issue by separately annotating INTEGRATED tokens.Source: §8 · p. 9
- Class imbalance leaves some LID and NER categories with few instances, limiting the reliability of per-category evaluation for rare labels.TR tokens dominate LID, while O (non-entity) tokens dominate NER.Source: §8 · p. 9
A.1 Dataset Collection
The corpus was collected from publicly available Ekşi Sözlük posts using topic tags chosen to increase the likelihood of Turkish-English code-mixing.
A.1 Dataset Collection
- TurEngMix draws on publicly available Ekşi Sözlük posts collected during November 2025, with candidate posts retrieved using topic tags selected to increase the likelihood of code-mixing.Source: §A.1 · p. 11
A.1.1 Topic Tags
The appendix describes topic-tag selection, corpus filtering and annotation prompts, agreement and per-class evaluation tables, representative errors, and released resources.
A.1.1 Topic Tags
- Topic tags span areas including technology, lifestyle, daily life, social media, entertainment, and sports, with examples such as “spotify,” “tatil,” and “netflix.”Source: §A.1.1 · p. 11
A.2 Corpus Filtering
- A multi-stage filtering pipeline identifies posts containing Turkish-English code-mixing while removing noise, beginning with English-word detection and following with code-mixing verification.The verification prompt checks the whole post and excludes URLs and brand names as English words.Source: §A.2 · p. 11, §A.2.1 · p. 12, §A.2.2 · p. 12
A.2.3 Language Identification Prompt
- The LID prompt assigns token labels TR, EN, MIXED, OTHER, NE, or AMBIGUOUS and requires an index-label output for each numbered token.Source: §A.2.3 · p. 12
A.2.4 Named Entity Recognition Prompt
- The NER prompt uses BIO tags and category-specific definitions, including physical places for locations and sports teams or bands for groups.Source: §A.2.4 · pp. 12–13
A.3 Per-Class Inter-Annotator Agreement
- The appendix provides per-class inter-annotator agreement tables for LID and NER and states that all disagreements were resolved.Source: §A.3 · p. 13, Table A1 · p. 13, Table A2 · p. 13
A.4.1 Per-Class Named Entity Recognition
- Per-class NER F1 tables are provided separately for decoder LLMs and encoder models.Source: §A.4.1 · p. 13, Table A3 · p. 14, Table A4 · p. 14
A.5 Error Analysis Examples
- A table presents representative model errors involving Turkish morphological integration and named-entity ambiguity.Source: §A.5 · p. 13, Table A5 · p. 15
A.6 Released Resources
- The appendix identifies publicly available dataset-construction and training-and-evaluation scripts among the released resources.Source: §A.6 · pp. 13–14