Source-linked AI summary
Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Serguei Barannikov, Irina Piontkovskaya, Sergey Nikolenko, Evgeny Burnaev
TL;DR
The paper addresses the difficulty of detecting increasingly human-like AI text when existing detectors may not transfer across generators and domains. It estimates each text sample’s intrinsic dimension from token embeddings using persistent homology and uses that value for detection. The resulting detector is reported to be robust across domains, models, languages, and adversarial conditions, while the authors identify stochasticity, fluent-generator assumptions, and limited language coverage as boundaries.
Problem
Existing artificial-text detectors often generalize poorly across generation models and domains, motivating a more general detection signal.
Method
The method computes persistent-homology intrinsic dimension from contextualized token embeddings for each text sample and uses it as a detection score.
Results
The detector is reported to be robust to domain shift, model shift, and adversarial attacks, with performance across considered languages and transfer settings.
Takeaways & Limitations
Intrinsic dimension provides a general-purpose score for separating natural and artificial texts without access to the generator.
Takeaways & Limitations
The method is stochastic, directly detects only fluent low-temperature generators, and has been evaluated only on several relatively high-resource languages.
Abstract
from arXiv · showhide
Rapidly increasing quality of AI-generated content makes it difficult to distinguish between human and AI-generated texts, which may lead to undesirable consequences for society. Therefore, it becomes increasingly important to study the properties of human texts that are invariant over different text domains and varying proficiency of human writers, can be easily calculated for any language, and can robustly separate natural and AI-generated texts regardless of the generation model and sampling method. In this work, we propose such an invariant for human-written texts, namely the intrinsic dimensionality of the manifold underlying the set of embeddings for a given text sample. We show that the average intrinsic dimensionality of fluent texts in a natural language is hovering around the value $9$ for several alphabet-based languages and around $7$ for Chinese, while the average intrinsic dimensionality of AI-generated texts for each language is $\approx 1.5$ lower, with a clear statistical separation between human-generated and AI-generated distributions. This property allows us to build a score-based artificial text detector. The proposed detector's accuracy is stable over text domains, generator models, and human writer proficiency levels, outperforming SOTA detectors in model-agnostic and cross-domain scenarios by a significant margin.
1 Introduction
AI-generated text is increasingly human-like, making general-purpose detection difficult. The paper proposes intrinsic dimension as a generator-independent signal and develops a persistent-homology-based detector for broad settings.
- LLM-generated texts increasingly resemble human writing, creating a need for artificial-text detection across social domains.
- Model-specific classifiers often generalize poorly to new generators and domains, while watermark detectors can be vulnerable to perturbation and paraphrasing.
- Intrinsic dimension separates artificial from natural texts without requiring additional knowledge about the generator.
- The method estimates each text sample’s geometry with persistent homology and evaluates classification across languages and settings.
- The contributions include a multilingual generation dataset and robustness across cross-domain, cross-model, adversarial, and writer-proficiency conditions.
2 Related work
Related work includes supervised and likelihood-based detectors, model-specific approaches, watermarking, and geometric methods for intrinsic-dimensionality analysis. These approaches motivate a detector designed to transfer across unseen generators and domains.
- Supervised artificial-text detectors can perform well in matched settings but often fail to generalize across domains, generation models, and sampling strategies.
- Model-based discriminators and likelihood scores commonly work best when the detector and generator are closely matched.
- Prior geometric and topological studies estimate intrinsic dimensionality in data manifolds, neural representations, and model parameters.
3 Intrinsic dimension and persistent homology dimension
Intrinsic dimension measures the degrees of freedom of a data manifold, but noise, mixed structures, and approximation choices complicate its estimation. The paper uses persistent homology dimension, whose estimator is designed to remain accurate with scarce and noisy samples.
- Intrinsic dimension: Intrinsic dimension is the number of local degrees of freedom needed to describe a subset within a higher-dimensional space.
- Estimation challenges: Real datasets may combine different dimensions, contain noise and outliers, and yield scale-dependent estimates because of hierarchical structure.
- Estimator choice: Global linear or nonlinear mappings can lose information important for artificial-text detection, motivating a fractal-dimension approach.
- Persistent homology dimension: PHD combines local and global dataset properties, is sample-efficient and noise-tolerant in the authors’ experiments, and has theoretical links to topology and fractal geometry.
- Persistent homology dimension: Persistent homology tracks topological features as gaps between points are filled at increasing distance thresholds.
- Estimator construction: The estimator weights persistent-feature lifespans, where each lifespan is the difference between a feature’s death and birth scales.
- Estimator evaluation: PHD is the only compared estimator reported as noise-tolerant, remaining stable when samples are scarce and outperforming alternatives in noisy settings.
4 Methodology
The method represents each medium-length text as a point cloud of contextualized token embeddings and estimates its intrinsic dimension with persistent homology. It stabilizes the estimator through repeated sampling and averaging, then uses PHD as the single feature in a logistic-regression detector.
- Text representation: ≈300-token texts are treated as separate manifolds, with contextualized token embeddings produced by RoBERTa-base for English and XLM-R for other languages.Artificial boundary tokens are removed before estimating the point cloud’s persistent homology dimension.
- PHD estimation: The estimator samples subsets of the embedding point cloud, computes persistent scores, and fits a line to pairs of log subset sizes and log persistent scores.The fitted slope κ is used to estimate dimension d as 1/(1−κ).
- PHD estimation: Repeated sampling and averaging replace single-subset calculations to improve the stability of dimension estimates.This addresses instability arising from small, non-uniform point sets.
- Detection model: A logistic-regression classifier uses PHD as a single feature to distinguish real and generated texts.The classifier is trained on a dataset containing real and generated samples.
5 Experiments
Experiments show that intrinsic dimension separates natural from generated texts across languages, embedding models, domains, and generation models. The resulting PHD detector remains robust under cross-domain transfer, paraphrasing, prompt-based style changes, and non-native writing bias, while short or unusual texts remain edge cases.
- Intrinsic dimensionality: Natural texts cluster around intrinsic dimension 9–10, whereas generated texts are approximately 8, with a clear separation across genres and languages.Most European languages are approximately 9±1; Italian and Spanish are approximately 10±1, while Chinese and Japanese are approximately 7±1.
- Embedding models: PHD varies only slightly with the embedding language model, decreasing for smaller models; RoBERTa-base and same-sized XLM-R are used thereafter.RoBERTa-base provides the best variance for English PHD estimation, while XLM-R is used for multilingual experiments.
- Universal detection: PHD classifiers outperform general-purpose detectors at 1% FPR, with margins of +10% for GPT-3.5 and +14% for OPT.The evaluation uses open-ended generation from Wikipedia prompts; PHD also remains effective after DIPPER paraphrasing, unlike most detectors.
- Cross-domain and cross-model performance: PHD remains stable under domain transfer, whereas supervised classification is nearly perfect in-domain but fails across domains.Across Reddit, Wikipedia-style, and StackExchange texts, PHD slightly outperforms the supervised baseline on average while being much more stable; the experiments also assess cross-model transfer.
- Multilingual detection: PHD achieves solid multilingual detection, averaging ROC-AUC 0.78 across ten languages, compared with 0.8 for MLE.The weakest PHD results are reported for Chinese and Japanese, at 0.71 and 0.74 respectively.
- Bias and robustness: PHD reduces non-native-speaker false positives to 26%, then to 7.7% after GPT-4 polishing, while remaining robust to prompt-based complexity changes.Baseline false-positive rates reach 58% for OpenAI and 52% for GPTZero on non-native essays; both ID classifiers retain high detection rates when generation prompts request simpler or more complex wording.
- Edge cases: Misclassifications concentrate among short texts and passages rich in addresses, geographical names, proper nouns, rare tokens, or numbers.The authors hypothesize that persistent-homology dimension estimation is less accurate on short texts and leave fuller failure-case analysis for future work.
6 Limitations and broader impact
The method has three main limitations: stochastic estimation, restricted detection of fluent low-temperature generators, and uncertain transfer to low-resource languages.
- The estimator is stochastic because same-generator texts vary widely in persistent-homology dimension and repeated estimation adds noise or slows computation.
- Out of the box, the detector targets fluent generators using relatively low sampling temperatures.Bad or high-temperature generators can have higher persistent-homology dimension than real texts, requiring recalibration.
- The method has been evaluated only on several relatively high-resource languages, leaving transfer to low-resource languages unknown.The authors identify this transfer question as future work.
- Despite these limits, the method is reported as more robust to model and domain changes than known tools and avoids discriminating against non-native speakers.
7 Conclusion
The paper introduces intrinsic-dimension estimation for text samples and uses persistent homology to compute a detector score. Its computational procedure estimates dimension from persistence-barcode statistics across sample sizes.
- 7 Conclusion: The paper introduces intrinsic dimension as a feature of text samples that separates human-written texts from modern LLM outputs.The conclusion states that human samples in a language have approximately similar dimension, while LLM texts have lower average dimension.
- Persistent homology: Persistent homology represents topological features across filtration values as homology groups and maps, summarized through persistence barcodes.
- 0th persistence barcode: The 0th persistence barcode tracks connected-component lifespans as edges are added in increasing length order.When an edge joins two components, the first component dies and a lifespan (0, λ) is recorded.
- 0th persistence barcode: The 0th-barcode algorithm is equivalent to Prim’s minimum-spanning-tree construction, with each new bar corresponding to an MST edge.
- PHD computation: The PHD computation evaluates persistence-barcode sums over multiple subset sizes, fits a regression to log-transformed values, averages three slopes, and converts the final slope into dimension.The procedure uses k subset sizes, J random subsets per size, and three random seeds before producing the final estimate.
C Additional experiments
Additional experiments examine how the PHD estimate depends on α. The authors report that α = 1.0 is reasonable, while the exact value changes with this parameter.
- The PHD estimate is obtained from the regression slope between log E0_α(X_n_i) and log n_i.
- α must be smaller than the text’s intrinsic dimension, and the experiments fix α = 1.0.
- α = 1.0 provides reasonable performance, although the exact PHD values for natural and generated texts depend on α and require further investigation.
- For α ∈ [0.5; 2.5], the results generally align with prior experiments studying dimension-estimator performance across α values.
C.2 Effect of paraphrasing on intrinsic dimension
The paper evaluates how paraphrasing changes intrinsic-dimension measures and detector performance. Increased lexical diversity slightly raises PHD and MLE, whereas sentence reordering has almost no noticeable impact on the reported detector behavior.
- Paraphrasing does not substantially harm PHD-based detection, while MLE-based detector performance drops significantly, especially for GPT-3.5.
- Increased lexical diversity slightly raises mean PHD and mean MLE, which theoretically could reduce detector efficiency.
- The PHD detector shows no observed performance decrease after lexical-diversity changes, likely because the mean shift is small and concentrated in texts already likely to evade detection.
- Increasing the re-ordering rate from 0 to 60 has almost no noticeable impact.
C.4 Intrinsic dimension of texts in different languages
The section presents multilingual PHD comparisons on Wikipedia data using XLM-RoBERTa-base embeddings, alongside related detector and text-example materials.
- Figure 10 compares PHD distributions across languages for Wikipedia texts using XLM-RoBERTa-base embeddings.
- The multilingual comparison uses the same XLM-RoBERTa-base model to obtain embeddings for the texts.
- The surrounding materials include a comparison of GPT detectors and examples of original and ChatGPT-generated text.
F Various Intrinsic Dimension estimators
This section documents intrinsic-dimension estimates across genres and illustrates text examples, misclassification outliers, and estimator-related experimental materials.
- Table 6 provides paired examples of original text and ChatGPT-generated text with common prompt parts highlighted.
- Figure 11 contrasts PHD for random-token, human Wikipedia, text-davinci-003, and repeated-token texts.The figure notes that no additional outlier restarts were performed, so it contains more outliers than Figure 3.
- Table 9 reports intrinsic dimensions of English texts from different genres estimated by different methods.
- Table 10 lists data-split sizes used for cross-model, cross-domain, and universal-detector comparisons.
G Data description
The experiments combine existing Wikipedia and Reddit data with a newly assembled StackExchange dataset, then use controlled splits and length filtering.
- Table 10 reports the sizes of the data splits used for cross-model, cross-domain, and universal-detector evaluations.
- Wikipedia and Reddit data come from Krishna et al. (2023), while the StackExchange dataset was assembled using the same scheme.
- For GPT2-XL, OPT, and GPT-3.5, the study randomly selected 2700 natural/generated text pairs per model.
- Data were split into train, validation, and test sets in a 7:1:1 proportion.The split prevents human texts or texts generated from the same prompt from appearing simultaneously in training and testing.
- Texts shorter than 50 tokens were removed for stable intrinsic-dimensionality estimation, along with an equal number from the opposite class.