PaperlayerView exact revision on arXiv

Source-linked AI summary

Turk-LettuceDetect: A Hallucination Detection Models for Turkish RAG Applications

Selva Taş, Mahmut El Huseyni, Özay Ezerceli, Reyhan Bayraktar, Fatma Betül Terzioğlu

arXiv:2509.17671v1cs.CLcs.AI

TL;DR

Hallucinations remain a challenge for RAG, and Turkish detection is underexplored given the language’s low-resource status and complex morphology. Turk-LettuceDetect adapts token-level classification to three encoder architectures using translated RAGTruth data. ModernBERT performs best on summary, QA, and the whole dataset, while EuroBERT leads on data-to-text; summarization remains most challenging.

  • Problem

    Turkish hallucination detection is underexplored, and RAG systems can still hallucinate despite grounding responses in retrieved context.

    Source: §I · pp. 1–2

  • Method

    Turk-LettuceDetect fine-tunes three encoder architectures for binary token-level detection using a machine-translated RAGTruth dataset.

    Source: §I · p. 2

  • Results

    ModernBERT performs best on summary, QA, and the whole dataset, EuroBERT excels in data-to-text, and summarization is the most challenging task.

    Source: C. Token-Level Hallucination Detection · p. 6, §V · p. 7

  • Takeaways & Limitations

    The models provide a foundational step toward Turkish RAG hallucination detection and support future research in low-resource, morphologically complex languages.

    Source: §V · p. 7

    All encoder models have reduced summarization effectiveness, with hallucinated-token F1 scores below 0.65.

    Source: B. Comparative Model Performance and… · p. 5

Abstract

from arXiv · show

The widespread adoption of Large Language Models (LLMs) has been hindered by their tendency to hallucinate, generating plausible but factually incorrect information. While Retrieval-Augmented Generation (RAG) systems attempt to address this issue by grounding responses in external knowledge, hallucination remains a persistent challenge, particularly for morphologically complex, low-resource languages like Turkish. This paper introduces Turk-LettuceDetect, the first suite of hallucination detection models specifically designed for Turkish RAG applications. Building on the LettuceDetect framework, we formulate hallucination detection as a token-level classification task and fine-tune three distinct encoder architectures: a Turkish-specific ModernBERT, TurkEmbed4STS, and multilingual EuroBERT. These models were trained on a machine-translated version of the RAGTruth benchmark dataset containing 17,790 instances across question answering, data-to-text generation, and summarization tasks. Our experimental results show that the ModernBERT-based model achieves an F1-score of 0.7266 on the complete test set, with particularly strong performance on structured tasks. The models maintain computational efficiency while supporting long contexts up to 8,192 tokens, making them suitable for real-time deployment. Comparative analysis reveals that while state-of-the-art LLMs demonstrate high recall, they suffer from low precision due to over-generation of hallucinated content, underscoring the necessity of specialized detection mechanisms. By releasing our models and translated dataset, this work addresses a critical gap in multilingual NLP and establishes a foundation for developing more reliable and trustworthy AI applications for Turkish and other languages.

I. INTRODUCTION

Hallucinations remain a challenge for RAG, and Turkish’s low-resource status and complex morphology make detection especially underexplored. Turk-LettuceDetect adapts token-level LettuceDetect detection to Turkish using three encoder architectures.

I. INTRODUCTION

  • RAG systems can still hallucinate on complex queries or when retrieved context is incomplete or contradictory.RAG aims to ground responses in external knowledge, but empirical studies report persistent hallucination rates.Source: §I · p. 1
  • Turkish hallucination detection is underexplored, with scarce training data and limited benchmarks compounded by agglutinative morphology and rich inflection.The paper notes that Turkish is spoken by over 80 million people worldwide.Source: §I · p. 1
  • Despite LettuceDetect’s reported 79.22% F1 on RAGTruth and expansions to several languages, a gap remains for morphologically complex, low-resource Turkish.The reported score marked a 14.8% enhancement over earlier English encoder-based methods.Source: §I · pp. 1–2
  • Turk-LettuceDetect adapts LettuceDetect to Turkish with ModernBERT, TurkEmbed4STS, and EuroBERT models trained for binary token classification.Tokens in generated answers are labeled supported or hallucinated using the context and question.Source: §I · p. 2
  • The work presents Turkish RAG hallucination models, reports competitive and efficient ModernBERT adaptation, and releases the translated dataset and models under an open-source license.The reported validation spans question answering, data-to-text generation, and summarization.Source: §I · p. 2

II. RELATED WORK

Prior hallucination mitigation includes prompt-based LLM judging and fine-tuned encoder classifiers, each with practical trade-offs. This work adapts encoder-based detection to Turkish and extends the dataset for multilingual evaluation.

II. RELATED WORK

  • Prompt-based LLM judges are flexible but costly, less interpretable, and inconsistent across model versions, while encoder models offer efficiency and reproducibility.Encoder approaches classify unsupported spans directly and are described as suitable for real-time deployment.Source: §II · p. 2
  • ModernBERT’s local-global attention and rotary positional embeddings support sequences up to 8,192 tokens, addressing long-context demands in RAG verification.The passage links long-context understanding to verification against source documents.Source: §II · p. 2
  • Turkish embedding work on STS and retrieval motivates adapting encoder architectures to hallucination detection while accounting for Turkish morphology and multilingual-transfer limitations.The passage names TurkEmbed4STS and TurkEmbed4Retrieval.Source: §II · p. 2
  • This work trains ModernBERT, GTE-multilingual-base, and EuroBERT on machine-translated RAGTruth and adds Gemma3-27b-it translations for cross-lingual generalization.The authors state that efficient encoder-based detection can rival prompt-based methods while maintaining inference speed and scalability.Source: §II · p. 2

III. METHODOLOGY

The methodology applies LettuceDetect token-level hallucination detection across three encoder backbones. It trains and evaluates on RAGTruth, whose human-annotated examples span three generation tasks and are recast as binary labels.

III. METHODOLOGY

  • The study evaluates three backbones: Turkish-fine-tuned ModernBERT-base-tr, TurkEmbed4STS based on gte-multilingual, and a pre-existing EuroBERT encoder.ModernBERT-base-tr was fine-tuned for Turkish NLI and STS downstream tasks.Source: §III · pp. 2–3

A. Dataset

  • RAGTruth provides 17,790 training and 2,700 test instances across question answering, data-to-text generation, and summarization.The paper describes it as the first large-scale benchmark specifically constructed for hallucination phenomena in RAG settings.Source: A. Dataset · p. 3
  • The dataset draws QA from MS MARCO, business reviews from Yelp, and news summarization documents from CNN/Daily Mail.LLMs generated responses from matched contexts, reviews, or sampled news documents, respectively.Source: A. Dataset · p. 3
  • Human experts marked hallucinated spans and rationales across six-model responses; training simplifies four hallucination categories into binary detection.The categories are evident and subtle conflict, and evident and subtle introduction of baseless information.Source: A. Dataset · p. 3
  • Input lengths averaged 801 tokens, with a 741-token median and a 194–2,632-token range, motivating context windows above 4,096 tokens.The passage connects long-context models to identifying hallucinations in lengthy, context-rich inputs.Source: A. Dataset · p. 3

B. Multilingual Extension of RAGTruth Dataset

The dataset extension translates RAGTruth content into multiple target languages. Its described translation pipeline uses Gemma 3 27B IT on an A100 GPU with parallel processing.

B. Multilingual Extension of RAGTruth Dataset

  • The multilingual extension translates RAGTruth content into multiple target languages to address the dataset’s monolingual limitation.The passage says the approach follows translation to other European languages.Source: B. Multilingual Extension of RAGTruth… · p. 3

1) Translation Methodology:

  • The translation pipeline used google/gemma-3-27b-it via vLLM on one NVIDIA A100 GPU, processing about 30 examples in parallel and taking roughly 12 hours per target language.Source: 1) Translation Methodology · p. 3

2) Translation Protocols:

The translation protocols aim to preserve content, annotations, and structure while adapting both answers and prompts for Turkish. Answer translation preserves hallucination tags; prompt translation prioritizes functional equivalence and clean output.

2) Translation Protocols:

  • Answer translation preserves the number and positions of existing <HAL> tags while translating their contents, and does not add or remove tags.Source: 2) Translation Protocols · p. 3
  • Answer segments without <HAL> tags are translated directly while maintaining their semantic meaning and contextual nuances.Source: 2) Translation Protocols · p. 3
  • The prompt protocol translates both natural-language instructions and structured elements, including JSON keys and values.Source: 2) Translation Protocols · p. 4
  • Prompt translation prioritizes functional equivalence over literal wording and produces only the target-language prompt, without explanatory text.Source: 2) Translation Protocols · p. 4

3) Sample Translated Data:

The paper presents an example of translated RAGTruth data and describes a token-level hallucination detection pipeline using three encoder architectures. Its design avoids auxiliary-task transfer and supports fine-grained localization.

3) Sample Translated Data:

C. Model Architecture

  • The pipeline classifies each generated-response token as supported or unsupported using ModernBERT, TurkEmbed4STS, or EuroBERT.Source: C. Model Architecture · p. 4
  • The architecture uses base transformer representations directly, avoiding NLI pretraining, auxiliary-task transfer, and multi-stage training.Source: C. Model Architecture · p. 4
  • Token-level outputs localize hallucinated content for downstream applications that need precise identification of factual inconsistencies.Source: C. Model Architecture · p. 4

D. Training Configuration

Three transformer models were fine-tuned with identical token-level supervision, then evaluated on the RAGTruth test split. The evaluation covers multiple tasks and examines decoder-only models and encoders.

D. Training Configuration

  • Three models received identical token-level supervision, with question and context tokens masked and answer tokens labeled supported or hallucinated.Source: D. Training Configuration · p. 4
  • Training ran for six epochs at a learning rate of 1 × e−5 and batch size 4 on an NVIDIA A100 40GB GPU.Source: D. Training Configuration · p. 4

IV. EVALUATION

  • The evaluation assesses the RAGTruth test split and analyzes hallucination behavior across decoder-only models and encoders.Source: §IV · pp. 4–5

A. Experimental Setup

The evaluation covers question answering, data-to-text generation, and summarization, assessing performance at example and token levels. It uses precision, recall, macro F1, and AUROC to examine complementary aspects of detection.

  • The test split covers question answering, data-to-text generation, and summarization, with assessment at both example and token levels.Source: A. Experimental Setup · p. 5
  • Precision measures the correctness of predicted hallucinated or supported instances, while recall measures identification of actual instances.Source: A. Experimental Setup · p. 5
  • Macro F1 balances precision and recall across hallucinated and supported classes, while AUROC summarizes discrimination across classification thresholds.Source: A. Experimental Setup · p. 5

B. Comparative Model Performance and LLM Hallucination Behavior

The compared encoders show different strengths across tasks, while the evaluated LLMs combine high recall with low precision. The reported results also identify weaker encoder performance in summarization.

B. Comparative Model Performance and LLM Hallucination Behavior

C. Token-Level Hallucination Detection

The token-level results show that model strengths vary by class and task. ModernBERT is balanced in QA, TurkEmbed4STS is consistent across tasks, and summarization remains challenging for hallucinated-token detection.

C. Token-Level Hallucination Detection

  • Across models, supported-token precision exceeds 0.63 and recall exceeds 0.72, but hallucinated-token detection drops particularly in summarization.ModernBERT's summarization precision is 0.6935.Source: C. Token-Level Hallucination Detection · p. 6
  • ModernBERT achieves QA precision of 0.7642 and recall of 0.7536, while its data-to-text hallucination detection is weaker than the other models'.In data-to-text, EuroBERT has the best F1-score, while TurkEmbed4STS has the best recall and AUROC.Source: C. Token-Level Hallucination Detection · p. 6
  • TurkEmbed4STS keeps precision and recall relatively close across tasks, avoiding extreme imbalances between supported- and hallucinated-class detection.Source: C. Token-Level Hallucination Detection · p. 6
  • The reported overall token-level scores indicate that encoder models detect hallucinations and discriminate between supported and hallucinated content despite class imbalance.Source: C. Token-Level Hallucination Detection · p. 6

V. CONCLUSION

Turk-LettuceDetect adapts token-level hallucination detection to Turkish RAG applications. The conclusion reports task-dependent model performance and positions the suite as a foundation for further work on Turkish RAG reliability.

V. CONCLUSION

  • Turk-LettuceDetect adapts LettuceDetect's token-level classification methodology to Turkish using three encoder architectures fine-tuned on RAGTruth.Source: §V · pp. 6–7
  • The ModernBERT-based model leads on summarization, question answering, and the whole dataset, while the EuroBERT-based model excels on data-to-text.The conclusion identifies summarization as the most challenging task.Source: §V · p. 7
  • EuroBERT-based models show robust cross-lingual generalization without full in-language retraining.Source: §V · p. 7
  • GPT-4.1 and Mistral show high recall but consistently low precision, while the encoder models offer an efficiency-accuracy trade-off.The encoder models span sizes of 135M, 210M, and 305M and support inputs up to 8,192 tokens.Source: §V · p. 7
  • The authors present the models as a foundational step toward robust Turkish RAG hallucination detection and plan to release them publicly.They describe the work as filling a gap in multilingual RAG evaluation and supporting future research in low-resource, morphologically complex languages.Source: §V · p. 7
Loading 2509.17671v1…