Source-linked AI summary
Automatic detection of Gen-AI texts: A comparative framework of neural models
Cristian Buttaro, Irene Amerini
TL;DR
The paper addresses the growing difficulty of distinguishing fluent AI-generated text from human writing and its associated reliability and accountability concerns. It develops and compares four supervised neural architectures with commercial detectors across multilingual, balanced, single-class, and thematic settings. The results indicate that detector behavior varies with architecture, language, domain, calibration, and thresholding, so no universally optimal detector exists.
Problem
Fluent LLM outputs make Human-versus-GenAI discrimination difficult, while detection errors raise reliability, transparency, accountability, social, ethical, and legal concerns.
Method
The study compares MLP, CNN 1D, MobileNet CNN, and Transformer detectors with commercial tools across English and Italian COLING data and an Art-and-Mental-Health thematic dataset.
Results
No universally optimal detector exists; model behavior varies across languages and domains and depends on architecture, decision thresholds, probability calibration, and regularization.
Takeaways & Limitations
Reliable assessment requires heterogeneous controlled experiments and stress tests beyond standard balanced in-domain evaluations.
Takeaways & Limitations
Robustness under distributional shift cannot be reliably assessed using standard balanced evaluations alone.
Abstract
from arXiv · showhide
The rapid proliferation of Large Language Models has significantly increased the difficulty of distinguishing between human-written and AI generated texts, raising critical issues across academic, editorial, and social domains. This paper investigates the problem of AI generated text detection through the design, implementation, and comparative evaluation of multiple machine learning based detectors. Four neural architectures are developed and analyzed: a Multilayer Perceptron, a one-dimensional Convolutional Neural Network, a MobileNet-based CNN, and a Transformer model. The proposed models are benchmarked against widely used online detectors, including ZeroGPT, GPTZero, QuillBot, Originality.AI, Sapling, IsGen, Rephrase, and Writer. Experiments are conducted on the COLING Multilingual Dataset, considering both English and Italian configurations, as well as on an original thematic dataset focused on Art and Mental Health. Results show that supervised detectors achieve more stable and robust performance than commercial tools across different languages and domains, highlighting key strengths and limitations of current detection strategies.
1 Introduction
AI-generated text is increasingly difficult to distinguish from human writing, creating reliability, transparency, accountability, and fairness concerns across multiple domains. Existing detection approaches remain limited in generalization, cross-model robustness, and susceptibility to false positives and false negatives.
- LLMs produce fluent, coherent text that is increasingly difficult to distinguish from human writing.
- AI-generated content has expanded across educational, journalistic, administrative, and legal domains.
- Detection errors can cause false accusations, discrimination, and reduced trust in educational, media, and judicial institutions.
- Current approaches face limitations in generalization, cross-model robustness, and false-positive and false-negative susceptibility.
- The paper frames AI-text detection as a social, ethical, and legal issue rather than only an automated classification task.
2 Related Works
AI-generated text detection has developed across stylistic, probabilistic, supervised, watermarking, and cognitive approaches, alongside commercial tools. However, existing research provides limited evidence on multilingual, cross-domain, and consistently comparative reliability.
- The field includes stylistic, linguistic, probabilistic, supervised, watermarking, and cognitive approaches to AI-text detection.
- Token predictability, distributional irregularities, and contextual representations have been studied as indicators of artificial generation.
- Many studies use single-language, often English-only, and balanced benchmarks, limiting insight into multilingual behavior and domain variability.
- Academic models and commercial detectors are often evaluated separately, limiting reliability comparisons under consistent conditions.
- This work compares heterogeneous supervised neural detectors and commercial tools across English and Italian, general-purpose and thematic settings.
3 Methodology
The methodology uses a shared end-to-end Human-versus-GenAI pipeline while varying the neural feature extractor. It compares four model families with pooling, regularization, calibration, and threshold-based classification choices.
- The modular framework converts raw text into binary Human-versus-GenAI predictions while keeping the end-to-end pipeline consistent across detectors.
- The pipeline tokenizes and normalizes sequences, embeds tokens, extracts features, pools them, applies dropout, and produces a thresholded probability decision.
- The comparison includes an MLP baseline, a one-dimensional CNN, a MobileNet-style CNN, and a Transformer-based detector.
- The MLP uses masked pooled token embeddings, whereas the one-dimensional CNN captures local contiguous token patterns with convolutional filters.
- The Transformer combines positional information, stacked attention and feed-forward encoder blocks, normalization, dropout, and pooling strategies.
- Hyperparameter comparison includes dropout, label smoothing, weight decay, embedding and hidden dimensions, filter counts, kernel configurations, and validation-based threshold tuning.
4 Overview Dataset
The study evaluates AI-text detectors across multilingual benchmark data and thematic datasets, comparing supervised neural architectures with online tools under varied language and domain conditions. Results expose trade-offs between human-text preservation, GenAI sensitivity, and robustness under distributional shift.
- Datasets: The evaluation uses English and Italian subsets of the COLING multilingual dataset alongside original Art and Mental Health thematic data.The English subset is a balanced binary benchmark, whereas the Italian subset contains only GenAI texts for single-class evaluation.
- dtEN results: The dtEN benchmark reveals a trade-off between minimizing false positives on human texts and detecting GenAI content.MobileNet offers the best overall trade-off; MLP and Transformer preserve human accuracy at 97.1% and 97.3%, while CNN 1D detects all GenAI texts but fails on human texts.
- dtEN results: Online detectors generally preserve human-text accuracy but show lower GenAI sensitivity on dtEN, indicating a tendency to avoid false positives.Because they were not trained on dtEN, these results also indicate cross-dataset generalization capability.
- dtITA results: On the single-class dtITA evaluation, implemented detectors remain stable, whereas several online tools misclassify substantial portions of GenAI texts as Human.The MobileNet and Transformer models are excluded because the limited, single-class dataset does not support meaningful evaluation of high-capacity architectures.
- Cross-domain evaluation: Models trained on the heterogeneous Italian ART&MH dataset transfer better to dtITA than models trained on the English dtEN dataset.The comparison associates stronger single-class generalization with stylistic diversity and linguistic alignment, while English-trained deeper models degrade more under cross-lingual shift.
- ART&MH results: On ART&MH, proposed detectors maintain high performance but make different errors, while some commercial tools produce seemingly perfect yet opaque results.MLP favors human-text preservation, CNN 1D emphasizes GenAI detection, and Writer predicts only the Human class.
5 Conclusions
The study finds that Human–GenAI detection has no universally optimal detector: performance and error profiles depend on architecture, thresholds, calibration, regularization, language, and domain. It concludes that balanced in-domain evaluations are insufficient for assessing robustness under distributional shift.
- No universally optimal detector exists across the evaluated Human–GenAI detection settings.Model behavior depends critically on architectural choices, decision thresholds, probability calibration, and regularization strategies.
- Balanced datasets expose trade-offs between preserving human texts and detecting GenAI content.Conservative models minimize false positives, whereas more sensitive models emphasize GenAI detection.
- Thematic Art and Mental Health data reveal the difficulty of distinguishing expressive human writing from synthetic text.
- Standard balanced in-domain evaluations cannot reliably assess robustness under distributional shift.Models performing well on balanced in-domain tests may degrade when applied to different languages or domains.
- Meaningful evaluation requires heterogeneous conditions and stress tests that probe robustness beyond standard in-domain settings.The conclusion identifies multilingual coverage, language-shift analysis, and adaptive calibration as directions for improving evaluation and practical reliability.