Source-linked AI summary
IndicDetect: Evaluating Cross-Lingual LLM-Generated Text Detection for Hindi, Telugu, and Tamil
Bhaskar Ganesh Devalla, Junchao Wu, Nilesh Dokuparthi, Greeshma Yaluru, Tatiana Muniz Rodriguez, Lidia S. Chao, Derek F. Wong
TL;DR
AI-generated text detection benchmarks largely overlook Indic languages and realistic distribution shifts. IndicDetect addresses this gap with a standardized benchmark for Hindi, Telugu, and Tamil spanning domains, generators, and adversarial attacks, finding that supervised neural detectors generally outperform zero-shot methods while robustness failures remain substantial, especially for Hindi under attacks.
Problem
Existing AI-generated text detection benchmarks give limited attention to Indic languages and often evaluate under idealized conditions that omit realistic distribution shifts.
Method
IndicDetect pairs curated human-written and LLM-generated texts in Hindi, Telugu, and Tamil across domains and generators, then evaluates statistical and neural detectors under standardized domain, generator, and adversarial-shift settings.
Results
Supervised neural detectors generally outperform zero-shot alternatives, while detector performance degrades under unseen generators and attacks, with Hindi showing the sharpest adversarial F1 drops.
Takeaways & Limitations
The benchmark establishes a language-aware foundation for assessing detector robustness in Indic scripts rather than relying only on peak matched-condition accuracy.
Takeaways & Limitations
Coverage is limited to Hindi, Telugu, and Tamil, newer generators may be absent, and the attack suite excludes adaptive attacks and human post-editing.
Abstract
from arXiv · showhide
The rapid proliferation of LLMs has further heightened the need to develop dependable AI-generated text detection, especially beyond English. Nevertheless, current benchmarks pay little attention to Indic languages and test detectors in idealized settings that do not represent the real world. We present a generalized benchmark for AI-generated text detection in Hindi, Telugu, and Tamil, which we call IndicDetect, designed to assess the robustness of detectors under realistic distribution shifts. IndicDetect comprises highly curated human-written texts matched with LLM-generated counterparts across various domains and generators, and systematically evaluates detectors in the presence of domain shift, generator shift, and adversarial perturbation. Using a single and repeatable evaluation scheme, we evaluate a wide range of statistical and neural detectors. We find substantial robustness failures: supervised neural detectors perform well in-distribution, while training-free methods degrade considerably under unseen generators and adversarial attacks. The severity of these failures varies across languages, with Hindi exhibiting the largest overall degradation under adversarial perturbations. These results highlight that the primary weakness of existing detectors in Indic settings lies in their robustness, not in their peak accuracy. IndicDetect provides standard data splits, an evaluation protocol, and baselines to establish a robust, language-aware foundation for AI-generated text detection in Indic scripts.
1 Introduction
IndicDetect addresses the lack of realistic, non-English benchmarks for AI-generated text detection, focusing on Hindi, Telugu, and Tamil. It targets robustness failures under distribution shifts and adversarial perturbations.
- Existing detection benchmarks are predominantly English-centric and often use idealized conditions unlike real-world applications.
- Indic languages such as Hindi, Telugu, and Tamil are especially underrepresented because of their distinct linguistic characteristics.
- Detectors that perform well under matched conditions can fail under domain shifts, unseen generators, or adversarial perturbations.
- IndicDetect evaluates detector robustness in Hindi, Telugu, and Tamil under realistic distribution shifts.
2 Related Work
Prior work has established the importance of detecting LLM-generated text but remains limited in language coverage and realism. IndicDetect extends robustness evaluation to multiple Indic languages and Brahmic-script perturbations across all major evaluation axes.
- LLM-generated text creates risks including plagiarism, factual misinformation, and automated propaganda, increasing the need for automated detection.
- Existing benchmarks commonly separate human and machine text into clean pools without modeling human editing, noise, or adversarial interference.
- RAID expands robustness testing with over six million generated texts spanning multiple models, topics, and decoding strategies.
- IndicDetect covers three Indic languages, four domains, Brahmic-script adversarial perturbations, and identical cross-language evaluation settings.
- IndicDetect is distinguished from DetectRL by script-specific perturbations, identical multi-language settings, and monolingual script-constant evaluation.
3 IndicDetect
IndicDetect is a reproducible benchmark built from curated human and LLM-generated texts in Hindi, Telugu, and Tamil across domains and generators. It evaluates detectors under matched conditions, distribution shifts, and meaning-preserving adversarial attacks.
- Dataset Construction: The benchmark covers Hindi, Telugu, and Tamil across academic, news, creative, and movie-review domains using multiple LLM generators.Human-written texts are paired with generated counterparts under controlled, reproducible preprocessing and scoring.
- Dataset Construction: Standardized preprocessing, tokenization, fixed splits, and uniform scoring support reproducible comparisons across detectors and evaluation settings.
- Dataset Construction: Dynamic prompts use four keywords from each human-written sample to ground a corresponding generated text and reduce artificial template regularity.
- Dataset Construction: Adversarial attacks preserve semantics and labels while targeting orthographic and morphological properties of Brahmic scripts.
- Dataset Construction: The attack suite includes paraphrasing, character deletions, whitespace changes, and neutral paragraph insertion.
- Experiment Settings: The generalization score G_d is the uniformly weighted mean Macro-F1 across Multi-Domain, Multi-Generator, and Multi-Attack conditions.In-distribution settings and AUROC are excluded from this scalar score.
4 Detectors
IndicDetect compares training-free statistical detectors with supervised neural detectors using a unified preprocessing and evaluation framework. The detector set includes five statistical methods and three supervised neural baselines.
- Training-free detectors include Log-Likelihood, Log-Rank, LRR, Fast-DetectGPT, and Binoculars.
- Supervised baselines comprise XLM-RoBERTa-Base, XLM-RoBERTa-Large, and task-specifically fine-tuned Qwen 2.5-7B.
- All detectors use a unified preprocessing pipeline with identical training conditions where applicable.
5 Results and Discussion
Supervised XLM-RoBERTa detectors lead overall and remain robust across domain, generator, and attack shifts, while zero-shot methods show substantial instability and degradation, especially in Hindi.
- XLM-RoBERTa-Large achieves the highest average score in Telugu (96.06) and Hindi (85.17), while XLM-RoBERTa-Base leads Tamil (94.98).
- LRR is especially unstable, collapsing in Telugu (48.60) and Tamil (63.43), while Qwen 2.5-7B is weak under domain shift.
- Domain-Robustness: XLM-RoBERTa-Large maintains the strongest cross-domain performance, with Multi-Domain F1 of 95.70, 97.04, and 97.59 in Telugu, Hindi, and Tamil.
- Generator-Robustness: XLM-RoBERTa-Large generalizes best to unseen generators, with Multi-Generator F1 of 96.25, 97.15, and 97.60 across the three languages.
- Attack-Robustness: XLM-RoBERTa-Large maintains M-Attack AUROC above 99.37 across all languages, whereas zero-shot methods degrade sharply under adversarial attacks.
- Table 3 ranks detectors by language and reports AUROC, Macro-F1, and domain, generator, and attack task-setting scores.
- Attack-Robustness: Hindi is the most vulnerable language overall, with fine-tuned XLM-RoBERTa-Large reaching Multi-Attack F1 of 61.32.
6 Conclusion
IndicDetect benchmarks LLM-generated text detection in Hindi, Telugu, and Tamil under realistic shifts and Brahmic-script attacks. Results favor supervised neural detectors overall, while robustness remains language- and setting-dependent.
- IndicDetect standardizes data collection, splitting, and evaluation across domain shifts, generator shifts, and seven Brahmic-script attack types.
- Fine-tuned XLM-RoBERTa models achieve the best average performance across all three languages, with XLM-RoBERTa-Large strongest overall.
- Qwen 2.5-7B transfers strongly across generators and adversarial attacks but degrades sharply under domain shift.
- Zero-shot methods degrade rapidly under unseen generators and adversarial perturbations, with LRR collapsing on Telugu and Tamil.
- Hindi is the most challenging setting, showing the lowest average scores and greatest vulnerability to adversarial attacks.
Limitations
The benchmark is limited to Hindi, Telugu, and Tamil because broader Indic coverage lacks large-scale, domain-balanced corpora, while computational constraints restrict evaluation to smaller models.
- IndicDetect covers only Hindi, Telugu, and Tamil because large-scale, domain-balanced corpora were unavailable for broader Indic language coverage.
- Computational constraints limited evaluation to smaller models.
- Newer generators may be absent, and seven attack types do not exhaust adaptive attacks or human post-editing.
- Binary human/machine labels and pipeline-bound data may not capture code-mixed, transliterated, partially edited, or co-authored text.
A.1 Computational Experiments
All experiments ran on NVIDIA A100 GPUs with 80GB memory in the university high-performance computing cluster, supporting large-scale detector evaluation and reproducibility.
- The university HPCC ran all experiments on NVIDIA A100 GPUs with 80GB of memory.
- The GPU memory enabled evaluation across many detectors, domains, generators, and adversarial environments without resource-induced bottlenecks.
- The computational arrangement was intended to promote reproducibility and scalability of the reported results.
A.2 Ethics Statement
IndicDetect's transparent construction and attack suite carry dual-use risks because they may expose detector-evasion techniques. The authors nevertheless argue that openness can accelerate less evasive detection systems, while residual privacy and offensive-content risks require academic-only use and caution.
- IndicDetect's transparent construction pipeline and attack suite could reveal techniques for evading existing detectors.
- The authors argue that openness will accelerate detection systems less susceptible to evasion.
- Residual risks include unintentional personally identifiable information and offensive content despite manual review.
- The resource is intended for academic use only, and users should exercise caution.
A.3 Data Collection
Human-written data for Hindi, Telugu, and Tamil are collected from domain-representative online, academic, literary, news, and entertainment sources. Tamil samples additionally use a pre-2022 publication cutoff to avoid temporal contamination.
- Telugu data cover news, movie reviews, academic writing, and public-domain creative texts from contemporary and specialized sources.
- Hindi data combine Google News, an academic journal, public-domain poetry collections, and Bollywood Hungama opinion texts.
- Tamil data span news, cinema reviews, academic texts, and contemporary or classical creative writing from representative portals and archives.
- All Tamil samples were published before 2022 to prevent temporal contamination with LLM-generated counterparts.
A.4 Generative models
IndicDetect evaluates diverse generators and realistic adversarial perturbations to test whether Indic-language detectors remain robust beyond clean, matched conditions.
- Generative models: IndicDetect evaluates widely used LLMs as machine-text generators for cross-generator robustness testing.Models are accessed through official APIs or local checkpoints, with identifiers and access paths recorded for reproducibility.
- Generation procedure: Generation uses interactive, domain-specific prompts to produce stylistically varied text across academic, news, creative, and movie-review domains.Decoding temperature is fixed at 0.9 and generation length is constrained to 400–450 tokens.
- Adversarial attacks: Back-translation paraphrase attacks preserve semantic content while introducing lexical and syntactic variation.Hindi, Telugu, and Tamil samples are translated through Chinese and back using the official Google Translate API.
- Adversarial attacks: Character-removal attacks independently delete each character with probability p = 0.5 to test sensitivity to realistic surface noise.The perturbation targets tokenization and local distributional statistics while approximately preserving semantic content.
- Adversarial attacks: Whitespace attacks modify selected spacing segments without changing characters or lexical meaning, targeting whitespace-sensitive tokenization.The resulting text preserves readability while deliberately violating token boundaries and local formatting cues.
- Adversarial attacks: Synonym-Swap attacks replace dictionary-covered words with meaning-preserving synonyms, altering lexical cues and subword tokenization patterns.The attack is designed to stress detectors that rely on brittle word-choice statistics.