Source-linked AI summary
A Hybrid Protocol for Large-Scale Semantic Dataset Generation in Low-Resource Languages: The Turkish Semantic Relations Corpus
Ebubekir Tosun, Mehmet Emin Buldur, Özay Ezerceli, Mahmoud ElHussieni
TL;DR
Turkish lacks comprehensive semantic relationship datasets, while existing resources have limited scale and domain coverage. The paper proposes a hybrid protocol combining large-scale automation with quality control and dictionary validation, producing 843,000 semantic pairs and demonstrating utility in downstream models.
Problem
Turkish lacks comprehensive semantic relationship datasets, and existing resources have limited scale and domain coverage for modern neural NLP systems.
Method
The paper uses a hybrid protocol combining large-scale automation, explicit quality control, expert-curated terminology, and filtered dictionary integration.
Results
843,000 semantic pairs were produced, with downstream models achieving 90% top-1 retrieval accuracy and 90% F1-macro.
Takeaways & Limitations
The corpus demonstrates practical utility for training semantic systems and provides a large Turkish semantic resource.
Takeaways & Limitations
The dataset is primarily grounded in legal-domain vocabulary, which may cause models trained on it to underperform on casual or conversational Turkish.
Abstract
from arXiv · showhide
We present a hybrid methodology for generating large-scale semantic relationship datasets in low-resource languages, demonstrated through a comprehensive Turkish semantic relations corpus. Our approach integrates three phases: (1) FastText embeddings with Agglomerative Clustering to identify semantic clusters, (2) Gemini 2.5-Flash for automated semantic relationship classification, and (3) integration with curated dictionary sources. The resulting dataset comprises 843,000 unique Turkish semantic pairs across three relationship types (synonyms, antonyms, co-hyponyms) representing a 10x scale increase over existing resources at minimal cost ($65). We validate the dataset through two downstream tasks: an embedding model achieving 90% top-1 retrieval accuracy and a classification model attaining 90% F1-macro. Our scalable protocol addresses critical data scarcity in Turkish NLP and demonstrates applicability to other low-resource languages. We publicly release the dataset and models.
1 Introduction
Turkish NLP lacks large, comprehensive semantic relationship resources, especially at the scale required by modern neural systems. The paper proposes a hybrid protocol combining automated semantic structure, LLM classification, and dictionary validation to produce a large corpus with downstream validation.
- Research Gap: Turkish lacks comprehensive semantic relationship datasets comparable to those available for resource-rich languages.This scarcity affects semantic parsing, word sense disambiguation, and semantic similarity development.
- Research Gap: Existing Turkish resources have limited scale and domain coverage, creating a bottleneck for systems requiring millions of training examples.Morphological richness, projected or manually curated resources, and limited coverage of specialized terminology contribute to the challenge.
- Approach: The proposed hybrid protocol targets both computational efficiency and linguistic quality through three phases.The phases combine clustering-based context preparation, automated relationship classification, and dictionary integration.
- Corpus: 843,000 unique semantic pairs are produced, with dictionary-based validation helping preserve semantic coherence across induced relations.The corpus covers synonyms, antonyms, and co-hyponyms.
- Validation: 90% top-1 retrieval accuracy is achieved for synonym pairs, while classification reaches 90% F1-macro.These downstream models provide the paper’s validation of corpus utility.
2 Related Work
Prior Turkish semantic resources are constrained by labor-intensive construction, translation-based projections, limited coverage, and insufficient specialized-domain depth. This work extends LLM-based semantic data generation by adding clustering-based context and dictionary validation.
- Lexical Resources: Manual WordNet-style curation has limited the scale and coverage achievable for under-resourced languages.WordNet organizes lexical knowledge through synsets connected by semantic relations.
- Turkish Resources: Turkish Tree Bank derives from translated English WordNet synsets, inheriting the source resource’s conceptual organization and potential cultural biases.The passage identifies translation-based projection as the construction approach.
- Turkish Resources: KeNet uses Turkish dictionaries and corpora but remains limited to approximately 80,000 synsets and lacks depth in specialized domains.The cited resources primarily cover general vocabulary.
- LLM-Based Generation: LLMs have been used to generate training data for semantic similarity, natural language inference, and question answering.Their multilingual training provides knowledge that can be extracted for NLP data generation.
- Positioning: This work combines LLM generation with clustering-based context provision and dictionary validation for semantic relationship data.It extends prior studies of LLM-generated synonyms, antonyms, and paraphrases.
- Distributional Semantics: FastText supports semantic organization in morphologically rich languages through subword representations of rare forms and compounds.Embedding methods capture relatedness from co-occurrence patterns.
- Distributional Semantics: Clustering is used as contextual input for LLM-based relationship classification rather than as the final semantic output.This provides semantic scaffolding for relationship generation.
- Semantic Relations: Antonyms and synonyms can be close in embedding space because they occur in similar contexts, complicating distributional discrimination.Counter-fitting separates them but depends on precompiled lexical constraints.
3 Methodology
The methodology builds semantic context from an expanded Turkish term lexicon, uses FastText and agglomerative clustering to form clusters, applies Gemini 2.5-Flash for three-way relation labeling, and adds filtered dictionary pairs. The resulting pipeline combines scalable generation with quality controls and standardized output.
- Phase I: Context Preparation: The starting vocabulary contains 77,000 expert-curated legal and domain-specific terms assembled by legal specialists.The lexicon emphasizes Turkish legal codes, court decisions, and regulatory documents.
- Phase I: Context Preparation: NER-based augmentation expands the lexicon to approximately 110,000 unique concept-terms.The added terms include legal concepts, technical terms, and domain-specific entities.
- Phase I: Context Preparation: FastText embeddings represent the terms, with mean constituent vectors used for multi-word expressions.FastText’s subword modeling supports morphological variants, compounds, and specialized terminology.
- Phase I: Context Preparation: Agglomerative clustering with cosine distance and a 0.4 threshold yields approximately 13,000 clusters for downstream contextualization.The threshold targets broad thematic relevance rather than strict synonymy, allowing synonyms, antonyms, and co-hyponyms to co-occur.
- Phase II: LLM Enrichment: Gemini 2.5-Flash classifies intra-cluster relations into synonyms, antonyms, and co-hyponyms at an overall cost of approximately $65.The model uses cluster context and multilingual knowledge to generate structured relationship labels.
- Phase II: LLM Enrichment: Batch processing and post-processing remove self-synonyms, deduplicate pairs, normalize Unicode, and validate JSON structure.This phase produces approximately 827,000 labeled semantic pairs.
- Phase III: Dictionary Integration: Strict dictionary filtering retains high-precision synonym entries with at most two candidates and yields 16,000 validated pairs.Entries with ambiguous meanings or overlap with LLM-generated data are excluded.
- Output Format: The final JSONL format uses sentence pairs and labels for synonym, antonym, and co-hyponym relations.The format supports standard sentence-pair classification and conversion to contrastive-learning formats.
4 Dataset Analysis
The corpus contains 842,946 pairs with a strongly imbalanced distribution dominated by co-hyponyms, alongside mostly synthetic generation and broad domain coverage. Its low Type–Token Ratio indicates dense reuse of anchor terms across relations.
- Distribution Statistics: 842,946 total pairs comprise the corpus, with co-hyponyms at 71.96%, synonyms at 17.60%, and antonyms at 10.44%.Co-hyponyms account for 606,612 pairs, synonyms for 148,367, and antonyms for 87,967.
- Data Sources: Approximately 98% of the dataset is synthetically generated through LLMs, while dictionary-derived pairs account for 1.90%.The reported proportions summarize the corpus’s generation sources.
- Distribution Statistics: Co-hyponyms account for 72% of all pairs, creating the corpus’s dominant class.The distribution is described as reflecting the greater frequency of broad semantic relatedness than strict synonymy or antonymy.
- Structural Statistics: A Type–Token Ratio of 0.02 indicates a highly interconnected pairwise structure with repeated anchor terms across distinct relations.Tokenization statistics report a maximum input length of 37 tokens and a mean of 11.04.
- Domain Coverage: The dataset covers legal, financial, technical, medical, administrative, and regulatory vocabulary.The listed domains include banking, insurance, taxation, corporate finance, information technology, engineering, and medicine.
5 Experiments and Results
The experiments validate the corpus through contrastive embedding learning and relationship classification, using selected transformer encoders and optimized final training. The resulting models achieve strong held-out performance, while co-hyponyms are not useful as embedding hard negatives.
- Embedding Model Training: Approximately 55,000 embedding samples are organized as query-positive-hard_negatives triplets, with antonyms and co-hyponyms serving as hard negatives.Positive examples are true synonyms, while hard negatives are labeled antonyms or co-hyponyms.
- Embedding Model Results: Including co-hyponyms as hard negatives worsened embedding performance compared with excluding them.The result suggests co-hyponyms were not crucial for distinguishing strict synonymy from broader thematic similarity in this setup.
- Embedding Model Results: 90% top-1 retrieval accuracy is achieved for synonym pairs on a held-out embedding-model test set.Accuracy measures the proportion of queries whose true synonym appears among the top-k retrieved results.
- Model Selection: Six transformer-based sentence embedding models were benchmarked under identical conditions before optimized retraining of the selected model.The two-phase framework comprised candidate selection followed by final training on upgraded hardware.
- Model Selection: 0.87 F1-macro was achieved by turkish-e5-large during candidate selection, with stable convergence.The selected model was subsequently retrained with an upgraded configuration on NVIDIA L40S hardware.
- Classification Model Results: 90% F1-macro was achieved by the final classification model on the held-out test set.Per-class results were 0.83 F1 for synonyms, 0.92 for antonyms, and 0.94 for co-hyponyms; macro precision and recall were 0.88 and 0.92.
- Classification Model Results: Weighted loss addresses class imbalance, with minority classes retaining competitive performance despite limited training representation.One reported formulation states that synonym and antonym minority classes comprise only 28% of training data.
6 Discussion and Limitations
The hybrid protocol combines clustering, LLM enrichment, and dictionary validation to produce scalable semantic resources, while several domain, synthetic-data, temporal, and morphological limits remain. Its cross-linguistic transfer depends on access to embeddings, a target-language LLM, and dictionary resources.
- Strengths of the Hybrid Approach: FastText clustering, LLM enrichment, and curated dictionaries provide complementary scalability, semantic nuance, and quality anchoring.The protocol combines unlabeled semantic organization, nuanced relationship generation, and high-confidence validation.
- Strengths of the Hybrid Approach: The three-way scheme distinguishes synonyms, antonyms, and co-hyponyms, supporting models that represent both semantic similarity and specificity.Co-hyponyms capture shared semantic space without synonymy.
- Limitations: Legal-domain grounding may introduce systematic bias, and models trained on the dataset may underperform on casual or conversational Turkish.The limitation concerns domain transfer beyond formal legal vocabulary.
- Limitations: Approximately 98% of the data is synthetically generated via LLM, so LLM-specific biases may propagate to downstream models.Human evaluation is reported as showing high quality, but synthetic generation remains a stated limitation.
- Limitations: The dataset is a 2025 terminology snapshot, while evolving legal and technical vocabulary requires periodic updates.Its validity therefore depends on continued maintenance as terminology changes.
- Limitations: Terms include some morphological variants but are not systematically expanded across the full Turkish suffixation paradigm.For example, all inflected forms of a term may not be represented.
- Generalizability: Cross-linguistic transfer requires FastText embeddings, a target-language-capable LLM, and a dictionary resource, with estimated LLM costs of $50–100.The authors estimate comparable costs for languages meeting these requirements.
7 Conclusion
The paper presents a scalable hybrid protocol and an 843,000-pair Turkish semantic corpus combining LLM enrichment with dictionary validation. Downstream embedding and classification results demonstrate practical utility, while the protocol and trained models are released for low-resource-language research.
- Conclusion: 843,000 annotated semantic pairs make the Turkish Semantic Relations Corpus the largest native Turkish semantic resource to date.The corpus combines LLM enrichment with dictionary validation.
- Conclusion: 90% retrieval accuracy and 90% F1-macro from downstream embedding and classification models demonstrate practical utility.The two metrics correspond to embedding retrieval and classification models, respectively.
- Conclusion: The protocol’s independence from extensive manual annotation supports generalization to other low-resource languages facing similar data scarcity.The dataset and trained models are publicly released to facilitate research in under-resourced languages.
A Representative Dataset Examples
Table 6 presents representative Turkish semantic pairs spanning synonyms, antonyms, and co-hyponyms. The examples cover common vocabulary alongside specialized legal and financial terminology.
- Representative Dataset Examples: Examples include legal, financial, and domain-specific co-hyponym pairs, with English translations provided for international readers.The cited examples include contract, buyer/seller, and civil-law/criminal-law terminology.
- Representative Dataset Examples: Table 6 demonstrates three relationship types: synonyms, antonyms, and co-hyponyms.The table provides representative semantic-pair examples from the corpus.
B LLM Semantic Enrichment Prompt Template
Figure 3 provides the complete prompt used for Gemini 2.5-Flash semantic enrichment. The prompt directs classification into three relationship categories under strict categorization and output-format rules.
- LLM Semantic Enrichment Prompt Template: Figure 3’s prompt instructs Gemini 2.5-Flash to classify cluster-internal relationships as synonyms, antonyms, or co-hyponyms.It also specifies strict categorization rules and structured JSON output requirements.
C NER-Based Term Augmentation Prompt
The prompts define a structured workflow for augmenting the Turkish lexicon and classifying semantic relationships. They specify systematic extraction categories, strict relationship criteria, and structured outputs for consistent processing.
- Semantic Relationship Classification: Semantic classification distinguishes synonyms, antonyms, and co-hyponyms using explicit definitions and examples.The prompt treats synonyms as identical or nearly identical meanings, antonyms as exact semantic opposites, and co-hyponyms as distinct terms sharing a hypernym.
- Semantic Relationship Classification: Synonyms must be 100% substitutable in context, including abbreviations and their expansions.Examples include sözleşme–mukavele, mahkeme–yargı, and VUK–Vergi Usul Kanunu.
- Semantic Relationship Classification: The classification prompt enforces strict categorization rules and structured JSON output across 13,000 processed clusters.It frames the task as analyzing clusters of related Turkish terms and assigning semantic relationships.
- NER-Based Term Augmentation: The NER system is instructed to detect all concepts, terms, events, and facts completely, systematically, and consistently.Concepts include abstract qualities, values, principles, states, and other philosophical, ethical, social, or scientific meanings.
- NER-Based Term Augmentation: 33,000 additional domain-specific terms expanded the initial lexicon from 77,000 to 110,000 entries.The NER-based prompt extracts concepts, terms, events, and facts from legal documents.