A Lightweight Approach to Detection of AI-Generated Texts Using Stylometric Features

• The introduction describes hallucinations as generated content made up by LLMs and frames text detection as an initial step before addressing other AI-generated content.Source: §1 · pp. 1–2
• Manual identification is nearly impossible for fluent LLM prose; an early detector achieved 26% accuracy and mislabeled human text as AI in 9% of cases [3].Source: §1 · p. 2
• Transformer classifiers can reach 91%-99% accuracy and F1 in controlled settings [7] [10] [11], but require hundreds of millions of parameters and may degrade on unseen domains [13].Source: §1 · p. 2

Gap & Contribution

• 97% accuracy on the Kaggle AI-vs-Human dataset lets NEULIF match or exceed heavyweight transformer ensembles while using fewer than 10^5 parameters.The framework combines about 68 stylometric and readability features with compact CNN and Random Forest classifiers.Source: §1 · p. 2
• NEULIF addresses the need for a detector that uses rich stylometric features, approaches ensemble performance, and runs efficiently on standard CPUs.Source: §1 · p. 2

2.1 Solution vision:

• The task is binary classification: separate AI-generated from human-written text using stylometric vectors and independently evaluated CNN and Random Forest classifiers.Source: §2.1 · p. 3

2.1.1 Variable-Length Text Input and Sequence Models

• Rather than processing raw variable-length text sequences, the approach extracts predefined linguistic features, reducing computational overhead relative to resource-intensive sequence models.Source: §2.1.1 · p. 3

2.1.2 Linguistic Feature Extraction using spaCy and TextDescriptives

• spaCy and TextDescriptives map each text to a fixed-length feature vector, which is sent to a CNN or Random Forest for classification.Source: §2.1.2 · pp. 3–4, Figure 1 · p. 4

2.2 Dataset and Training

• The study randomly sampled 20,000 essays from the roughly 500,000-text Kaggle corpus, balanced between AI-generated and human-written examples.Source: §2.2 · p. 4

2.2.1 Feature Extraction

• The 68 extracted features cover text statistics, readability, syntax and dependencies, lexical diversity, cohesion, and text-complexity heuristics.Source: §2.2.1 · p. 4

2.2.2 Data Splitting and Preprocessing

• After removing rows with missing feature values, the data were split into stratified training, validation, and test sets, standardized to zero mean and unit variance.The partitions contained 15,981 training, 1,993 validation, and 1,997 test samples, with equal class proportions.Source: §2.2.2 · pp. 4–5

• The model uses fixed-size feature vectors from spaCy and TextDescriptives instead of processing raw text sequences.Source: §3 · p. 5
• The feature-vector approach is presented as enabling efficient computation while maintaining high classification accuracy.Source: §3 · p. 5
• The neural network processes linguistic features extracted before classification, rather than learning representations directly from raw text sequences.Source: §3 · p. 5

3.1 Neural network architecture

• The CNN departs from token-sequence processing by using extracted linguistic features, a choice the authors associate with efficiency, interpretability, and expert-crafted patterns.Source: §3.1 · p. 5
• A convolutional layer identifies local patterns among the 68 features, followed by three dense layers that learn interactions between linguistic characteristics.Source: §3.1 · p. 5

Architecture Overview

• The architecture processes 68-dimensional inputs through convolution, normalization, flattening, and dense layers with regularization, then uses a sigmoid neuron to produce an AI-class probability.The convolution uses 128 filters with kernel size 3; the dense layers contain 256, 128, and 64 neurons.Source: Architecture Overview · pp. 5–6

3.2 Neural Network Implementation

• A one-dimensional convolution with 128 kernels of size 3 captures local patterns among the 68 extracted features.Batch normalization follows the convolution to stabilize and accelerate training.Source: §3.2 · p. 6
• Three ReLU-activated dense layers with 256, 128, and 64 neurons process the flattened convolution output, with dropout rates of 0.4, 0.3, and 0.2.The dropout is interleaved between dense layers to prevent overfitting.Source: §3.2 · p. 6
• A single sigmoid output neuron produces the model’s confidence that a text was AI-generated rather than human-written.Source: §3.2 · p. 6

3.3 Classification Process and Implementation

• Raw text is processed with spaCy-TextDescriptives to extract 68 features spanning readability, syntax, lexical diversity, dependency relations, and stylometry.Examples include Flesch Reading Ease, dependency-tree depth, function-word ratios, and punctuation density.Source: §3.3 · pp. 6–7
• The feature vector is standardized using training-set means and standard deviations, yielding zero mean and unit variance for each feature.Source: §3.3 · p. 7
• The normalized vector is reshaped for Conv1D input, and the network’s sigmoid score is classified at a typical 0.5 threshold.Scores above 0.5 indicate AI-generated text; scores below it indicate human-written text.Source: §3.3 · p. 7
• Under 100 milliseconds is the typical inference time on standard CPU hardware for the full classification pipeline.The pipeline includes feature extraction, standardization, reshaping, neural-network inference, and threshold-based decision-making.Source: §3.3 · p. 7

Evaluation Methodology

• The test set contained 1,997 samples, comprising 10% of the dataset and held out during training and validation.Source: Evaluation Methodology · p. 7

Metrics and Evaluation

• 97% accuracy was achieved on the held-out test set, with 1,932 of 1,997 samples correctly classified and 65 misclassified.Source: Metrics and Evaluation · pp. 7–8
• 97% F1-score, 97% precision, 96% recall, 99.51% ROC-AUC, and 0.0915 logarithmic loss were reported for the neural network.Source: Metrics and Evaluation · p. 8

Confusion Matrix Analysis

• Figure 2 shows 969 true negatives and 963 true positives, alongside 37 false negatives and 28 false positives; the authors describe this as a slight conservative bias.The test set total was 1,997 samples.Source: Confusion Matrix Analysis · pp. 8–9, Figure 2 · p. 9

4.1 Random Forest Architecture

• TextDescriptives stylometric processing supplies readability, syntactic-complexity, and lexical-diversity features to the Random Forest classifier.The paper attributes improved discriminative capability to this broader range of textual characteristics [33].Source: §4.1 · p. 9

Architecture Overview

• A Random Forest comprises decision trees trained on different data subsets, with the trees’ predictions combined by majority vote.Random feature subsets per split and bootstrap sampling provide the described randomness.Source: Architecture Overview · p. 9
• The architecture uses spaCy’s en_core_web_sm model as its English-language base and adds the TextDescriptives all-pipeline component to extract features.Source: Architecture Overview · pp. 9–10
• The RandomForestClassifier hyperparameters include the number of trees, the split-quality criterion, and the random-state seed.Source: Architecture Overview · p. 10

4.2 Random Forest Implementation

• The Random Forest uses 100 decision trees, Gini impurity, and random state 42.The implementation uses spaCy with TextDescriptives for feature extraction.Source: §4.2 · p. 11

4.3 Classification Process and Implementation

• The classification pipeline balances AI-generated and human-written samples, extracts linguistic features, and splits data into training, validation, and test sets.The workflow uses Kaggle data, spaCy, TextDescriptives, and scikit-learn.Source: §4.3 · p. 11
• GridSearchCV tunes hyperparameters; performance is assessed on validation and test sets, and the trained model is serialized with joblib.The implementation also benchmarks resource use with cProfile, memory profiler, and psutil.Source: §4.3 · p. 11

Metrics and Evaluation

• 94% accuracy was reported on the held-out test set; AI-generated-text precision was 94%, recall 95%, F1-score 94%, and ROC-AUC 95%.The models were trained and evaluated on identical datasets for direct comparison.Source: Evaluation Methodology · p. 11, Metrics and Evaluation · p. 12
• The Random Forest log loss was 1.6039, which the passage interprets as indicating well-calibrated probability estimates.Log loss assesses probabilistic prediction quality.Source: Metrics and Evaluation · p. 12

Confusion Matrix Analysis

• 2,000 samples yielded 962 true negatives, 949 true positives, 38 false negatives, and 51 false positives in the Random Forest confusion matrix.The matrix summarizes predictions against ground truth on test data.Source: Figure 5 · p. 12, Confusion Matrix Analysis · p. 12

Performance Validation

• Independent validation data is presented as supporting confidence in performance on unseen data; the authors recommend regular monitoring and possible recalibration.The passage frames monitoring as a way to maintain performance over time.Source: Performance Validation · p. 13

Accuracy

• The validation accuracy was 0.94 (94%), and validation data was used during model development and hyperparameter tuning.Validation accuracy guides tuning, while test accuracy provides a final estimate on unseen data.Source: Accuracy · p. 13
• Test accuracy is also reported as 0.96 (96%) on the completely unseen test set.The passage describes this as a final, unbiased performance estimate.Source: Accuracy · p. 13

Neural Network Approach

• The neural network achieved 97% accuracy, 97% precision, 96% recall, 97% F1-score, and 99.5% ROC-AUC.Its log loss was 0.0915, and the passage describes its precision and recall as nearly balanced.Source: Table 2 · p. 13, Neural Network Approach · p. 14

Random Forest Approach

• The Random Forest achieved 95% accuracy and 94% F1-score, with ROC-AUC of 95.55% and log loss of 1.6039.The passage says it trails the neural network on every metric; model sizes are approximately 10.6 MB for RF and 25 MB for the neural network.Source: Random Forest Approach · pp. 14–15
• Figure 6 compares test confusion matrices side by side; both models show high true-positive and true-negative counts, while the Random Forest slightly reduces false positives.The neural network is on the left and the Random Forest on the right.Source: Figure 6 · p. 15, Random Forest Approach · p. 15

5.2 Benchmarking with other Existing solutions

• 97% CNN accuracy was 16 percentage points above StyloAI’s 81% on large-scale evaluation; the Random Forest achieved 95%, with NEULIF results reported in-domain.The benchmark compares lightweight detectors; Table 3 presents methods, model classes, and test accuracy.Source: Table 3 · p. 15, §5.2 · pp. 15–16

6 Conclusions

• NEULIF’s CNN and Random Forest achieved 97% and 95% accuracy, respectively, outperforming existing lightweight methods.The conclusion identifies the framework as state-of-the-art in the lightweight detector class.Source: §6 · p. 16
• The models are described as on par with or exceeding computationally intensive transformer methods, with ROC-AUC up to 0.995 and log loss 0.0915.The conclusion also characterizes their precision–recall balance as robust.Source: §6 · p. 17
• Future work will investigate generalization across domains, languages, and adversarial settings, as well as detection in other modalities.The conclusion also proposes studying texts spanning multiple centuries.Source: §6 · p. 17