DeBERTa-ConPara: Attack-Aware and Deployment-Realistic Detection of AI-Generated Text

1 Introduction

• The deployment problem includes domain and generator shifts, adversarial perturbations, and evaluation protocols that tune thresholds using target-domain labels unavailable in practice.Surface-form attacks such as homoglyph substitutions and zero-width characters can disrupt tokenisation-sensitive pipelines while preserving semantic content.Source: §1 · pp. 1–2
• DeBERTa-ConPara combines attack-aware preprocessing with a multi-dataset adversarial training corpus to address deployment-oriented detection.The paper also reports two negative results: ConPara augmentation does not improve the best configuration, and feature fusion is inert in distribution and harmful out of it.Source: §1 · p. 2
• 99.01% TPR@5% FPR on the official RAID hidden test accompanies 93.14% average balanced accuracy across HC3 Plus and MAGE under a fixed threshold.The reported evaluation is restricted to the English domains, generators, and attacks assessed in the paper.Source: §1 · p. 2
• A controlled factorial varies preprocessing placement and finds that training-time normalization removes adversarial supervision, whereas inference-time normalization is an effective defence.The introduction frames this comparison as a deployment-relevant test of the two placements.Source: §1 · p. 2
• The robustness gain is confined to homoglyph and zero-width-space attacks, and the same attack-specific signature appears in a zero-shot detector of a different architecture.The introduction attributes the effect to the attacks rather than to DeBERTa-ConPara itself.Source: §1 · p. 2

2 Related Work

• The section surveys zero-shot, supervised, watermarking, and adversarially robust detection paradigms, with realistic deployment involving unseen generators, domain shifts, and adversarial perturbations.These approaches are presented as complementary strands of AI-generated text detection.Source: §2 · p. 2

2.1 Zero-Shot and Statistical Detection

• Zero-shot statistical detectors use token distributions, probability curvature, or cross-model likelihood patterns, but prior work reports degradation under shifts, decoding variation, and adversarial rewriting, especially at fixed thresholds.Examples include GLTR, DetectGPT, Fast-DetectGPT, and Binoculars.Source: §2.1 · p. 2

2.2 Supervised Detection Methods

• Supervised detectors fine-tune pretrained encoders and use adversarial training or representation learning, yet prior evaluations find degradation under cross-generator and cross-domain shifts.Paraphrasing has also been reported to evade many detectors, motivating adversarial training and semantic-invariance augmentation.Source: §2.2 · p. 3

2.3 Watermarking-Based Detection

• Watermarking embeds detectable signals during generation, but requires generator cooperation and its robustness degrades under paraphrasing and semantic-preserving rewriting.The paper therefore distinguishes watermarking from post-hoc detection of already-generated text.Source: §2.3 · p. 3

2.4 Domain Generalisation and Adversarial Robustness

• Domain-generalisation methods target generator- or domain-invariant representations, while RAID benchmarks attacks including homoglyphs and zero-width Unicode characters.The related-work section notes that comparatively little prior attention addresses explicit mitigation of surface-form perturbations before tokenisation.Source: §2.4 · p. 3

3 Datasets

• HC3 Plus, M4, MAGE, and RAID provide complementary coverage of semantic invariance, cross-domain generalisation, fluent multi-domain writing, and adversarial robustness.Together, the benchmarks expose variability across generators, domains, writing styles, and attacks.Source: §3 · p. 3
• The training corpus contains 1,549,358 rows balanced exactly between human and AI-generated text, with 172,826 additional rows held out for validation.PeerRead and WikiHow, generated by the authors, are the corpus’s only exposure to newer models beyond the four public benchmarks’ 2023-era generators.Source: §3 · p. 3
• Table 1 reports corpus composition; its RAID contribution includes 398,834 AI-generated rows and 131,700 human rows, while M4 supplies human filler without AI text.The table caption also distinguishes benchmark-wide domain and generator counts from the sampled subset.Source: Table 1 · p. 4

4.1 Overall Architecture

• The reported architecture encodes inference-normalized text with DeBERTa-v3-large and feeds its classification embedding directly to a two-layer head, without the feature branch.The feature-fusion branch was included in the submitted version but is not part of the reported configuration.Source: §4.1 · pp. 4–5, §4 · p. 4

4.2 Attack-Aware Preprocessing Pipeline

• The preprocessing pipeline targets homoglyphs, typographic variants, combining marks, Unicode format characters, and whitespace before tokenisation.It uses curated homoglyph substitutions, typographic normalization, NFKD decomposition with mark removal and NFC recomposition, removes category Cf characters, and collapses whitespace; homoglyphs require explicit handling because decomposition alone does not repair them.Source: §4.2 · p. 5
• The method applies raw-text training and inference-only normalization because training-corpus normalization removes adversarial supervision while inference normalization is an effective defence.The reported placement differs from the submitted version, which applied the pipeline at both stages.Source: §4.2 · p. 5

4.3 Semantic-Invariance Augmentation

• Semantic-invariance augmentation is investigated through filtered paraphrases and supervised contrastive learning with paraphrased positive pairs and opposite-class negatives.Paraphrases are generated with flan-t5-base and retained using a cosine-similarity threshold of at least 0.85; the contrastive loss is weighted by a selected λ from {0.05, 0.1, 0.2}.Source: §4.3 · pp. 5–6

4.4 Training Configuration

• Training uses AdamW with weight decay 0.01, a peak learning rate of 2 × 10^-5, batch size 32, and maximum sequence length 512.The reported checkpoint is the epoch with the best validation balanced accuracy, using a OneCycleLR schedule with cosine annealing.Source: §4.4 · p. 6

5.1 Evaluation Protocol

• A single threshold calibrated on source validation data is held constant across target distributions, exposing operating-point trade-offs when target-domain labels are unavailable.For RAID, thresholds are calibrated to achieve exactly 5 % FPR on human-written text within each domain.Source: §5.1 · p. 6

5.2 Ablation Study: Preprocessing at Training Time and at Inference Time

• Normalising training collapses 35.4% of RAID rows and removes adversarial supervision, while inference normalisation improves TPR@1 % FPR by +1.08 to +15.04 pp.Raw-corpus training also helps by +2.05 to +18.05 pp; the factorial varies training and inference preprocessing independently.Source: §5.2 · pp. 6–7
• The preprocessing gain is concentrated in homoglyph and zero-width-space attacks; the same two-attack signature appears in a different-architecture zero-shot detector.Without inference normalisation, detection is 11.05 % for homoglyph and 1.12 % for zero-width-space attacks; with it, both reach 96.98 %.Source: §5.2 · pp. 7–8
• The feature branch costs 23.4 pp of TPR@1 % FPR on SemEval-2024 and 7.7 pp on HC3-SI, with its largest RAID penalty on poetry.On poetry the cost is 49.8 pp in the fully normalised configuration, falling to 7.8 pp with raw training.Source: §5.2 · pp. 7–8

5.3 Backbone Comparison

• DeBERTa leads BERT by six points of TPR@1 % FPR and RoBERTa by twenty-five, while the validation-pool backbone ranking does not survive the hidden test.On the hidden test, AUROC is 99.61, 99.07 and 94.32 % for DeBERTa, BERT and RoBERTa, respectively.Source: §5.3 · p. 8

5.4 Impact of the Training Data Scaling and Validation Design

• Scaling to 9.38M RAID rows improves hidden-test TPR@1 % FPR by 0.55 pp but lowers the cross-dataset average from 93.14 to 92.45 %.Correcting a MAGE-overrepresented validation split restored three points of HC3-SI performance.Source: §5.4 · p. 8

5.5 Cross-Dataset Generalisation

• Across external benchmarks, the four preprocessing cells’ averages span only 93.14 to 93.55 %, and inference normalisation is inert where Unicode perturbations are absent.Raw-corpus training costs about two points on HC3-SI and one on M4 while gaining one on MAGE; the feature branch helps on M4GT-Bench but has the opposite effect on SemEval-2024 and HC3-SI.Source: §5.5 · pp. 8–9

5.6 Comparison Against Strong Baselines

• Our configuration leads the cross-dataset average by 7.3 points over MELD, although competitors’ training sources favor our system on those external benchmarks.MELD is the only open system above ours on RAID alone; the nearest RAID systems fall to near chance on HC3-SI.Source: §5.6 · p. 9, Table 3 · p. 9

• Among systems with public checkpoints, DeBERTa-ConPara is within a point of the best RAID score and first on the cross-dataset average.Source: §6 · p. 9
• Inference-time normalization neutralizes zero-width and homoglyph attacks, whereas training-corpus normalization deletes that adversarial supervision; the feature branch reduces TPR@1% FPR in every condition.The feature-branch reduction is 49.8 pp on poetry alone.Source: §6 · p. 9
• Inference-time normalization is deterministic and parameter-free, and its signature appears in a different-architecture zero-shot detector, though it cannot replace adversarial exposure during training.Source: §6 · p. 9

• Each configuration is a single training run, and the factorial's paired variance could not be quantified without repeating it.Three seeds of one configuration spanned 10.68 points of hidden-test TPR@1% FPR; small cell differences are therefore indicative only.Source: Limitations · p. 10
• All robustness claims apply only to the English domains, generators, and attacks evaluated; transfer to non-Latin scripts remains open.Source: Limitations · p. 10
• Attack-aware preprocessing improves surface-perturbation robustness, but semantically preserving transformations such as paraphrasing remain challenging.Source: Limitations · p. 10
• Human-written training data comes from pre-2022 sources, while AI-assisted composition is an intermediate category that binary detection does not address.Source: Limitations · p. 10
• Robustness evaluation is bounded by available benchmarks and attacks, while emerging humanisation filters and personally adapted systems pose further challenges.Source: Limitations · p. 10

Ethical Considerations

• Detectors risk false positives for non-native speakers, minority dialects, and atypical human writing, so the paper recommends human oversight and appeal mechanisms in high-stakes use.The authors recommend decision support rather than treating detector output as definitive evidence.Source: Ethical Considerations · p. 10

A.1.1 HC3 and HC3 Plus

• HC3 Plus extends question-answering data with summarisation, translation, and paraphrasing; its semantic-invariant tasks preserve source meaning and are harder to detect.The work uses the English subset, reserving its semantic-invariant split for robustness evaluation.Source: §A.1.1 · p. 14

A.1.3 MAGE

• MAGE spans seven writing tasks and 27 language models, with eight testbeds progressing to unseen-domain and unseen-generator settings under paraphrasing attacks.The benchmark has 447,674 samples; the work follows its official split and uses 319,071 training samples.Source: §A.1.3 · p. 15

A.1.4 RAID

• RAID’s 2024 release covers 11 generators and 11 attack categories, but newer humanisation filters may accelerate distribution shift and motivate benchmark refreshment.The benchmark’s official hidden test is withheld without labels.Source: §A.1.4 · pp. 15–16

A.2.1 Attack Taxonomy and Preprocessing Pipeline

• The preprocessing pipeline targets Unicode and formatting perturbations before tokenisation and feature extraction, while normalisation also stabilises linguistic-feature estimates inflated by Unicode attacks.The pipeline is deterministic, idempotent, and has no learned parameters.Source: §A.2.1 · p. 17, Table 7 · p. 17

A.2.2 Linguistic Features

• The feature branch uses 62 handcrafted linguistic and statistical features, selecting 30 by mutual information using source-side training data only.Feature normalisation is also fitted on source training data to avoid target-domain leakage.Source: §A.2.2 · pp. 17–18, Table 9 · p. 17

A.2.3 Contrastive Learning and Paraphrase Augmentation

• Paraphrase augmentation uses offline Flan-T5-base outputs filtered at cosine similarity ≥0.85, while supervised contrastive learning pairs paraphrases and originals from the same class.The contrastive objective is combined with weighted cross-entropy; λ is tuned over 0.05, 0.1, and 0.2.Source: §A.2.3 · p. 18

A.3.1 Extended Out-of-Distribution Evaluation

• 94.69% balanced accuracy and 99.90% AUROC on the 669K academic benchmark; Gemini 2.0 Flash HC3-style responses had 98.64% recall.Evaluation used a fixed threshold without target-domain recalibration.Source: §A.3.1 · p. 18

A.3.2 ConPara Ablations

• The combined SupCon-plus-paraphrasing configuration reduces RAID TPR@5% FPR from 97.65% to 95.88%.SupCon alone modestly improves HC3-SI but consistently degrades MAGE and RAID; paraphrasing also slightly improves HC3-SI while substantially decreasing cross-domain adversarial robustness.Source: §A.3.2 · p. 20, Table 13 · p. 20
• Semantic-invariance objectives may conflict with AI-versus-human discrimination in heterogeneous multi-generator settings by reducing sensitivity to informative stylistic and statistical signals.SupCon’s strongest HC3-SI score is 85.45% at λ = 0.05, while that configuration reduces MAGE performance by more than three percentage points.Source: §A.3.2 · p. 20

A.3.3 Multi-Seed Stability

• Across five seeds, the observed robustness improvements remain stable, with balanced-accuracy variance of ±0.75 pp and AUROC variance of ±1.20 pp.The authors attribute the gains to architectural and data-centric choices rather than favourable random initialisation.Source: §A.3.3 · p. 20, Table 14 · p. 20

B Factorial Ablation: Extended Results

• +13.45 pp from inference-time normalisation after normalised-corpus training, versus +1.94 pp after raw training; raw training adds +4.66 pp under normalised inference.The two preprocessing placements substitute for one another, and normalising only at training removes the adversarial supervision without an inference-time defence to compensate.Source: B Factorial Ablation: Extended Results · p. 21
• The reported configuration’s unattacked TPR@1% FPR is 96.98%, below the 97.17% for raw training with raw inference; its aggregate advantage comes with the smallest attack penalty, 0.41 pp.The eight cells span 88.38% to 97.17% on unattacked rows.Source: B Factorial Ablation: Extended Results · p. 21
• The feature branch reduces TPR@1% FPR in all four train/inference combinations, by 7.07, 3.89, 1.85, and 0.14 pp.Its cost is largest in the fully normalised configuration and shrinks as raw training is introduced.Source: B Factorial Ablation: Extended Results · p. 21
• 96.98% detection for homoglyph and zero-width-space attacks with inference normalisation, versus 11.05% and 1.12% with no defence.Raw-corpus training alone reaches 85.78% for homoglyphs and 73.80% for zero-width spaces; inference normalisation slightly reduces performance on nine other attack classes by 0.09–0.24 pp.Source: B Factorial Ablation: Extended Results · p. 21
• The same two-attack signature appears with Fast-DetectGPT: inference normalisation raises zero-width-space AUROC from 0.604 to 0.875 and homoglyph AUROC from 0.683 to 0.873.The remaining ten classes move between −0.025 and +0.005, and the authors attribute the effect to the attacks and tokenisation failures rather than their architecture.Source: B Factorial Ablation: Extended Results · p. 21

C Backbone Comparison: Extended Results

• Homoglyph-substituted words fragment into subword pieces across BERT, RoBERTa, and DeBERTa, with token inflation of 3.40×, 4.00×, and 4.12×, respectively, and zero unknown tokens.The authors identify this as a property of subword tokenisation rather than one architecture.Source: C Backbone Comparison: Extended Results · p. 22
• On RAID validation, RoBERTa leads BERT by nine points in TPR@1% FPR, but on the hidden test BERT leads by nineteen points.Validation-based backbone selection would have chosen the worse hidden-test system.Source: C Backbone Comparison: Extended Results · p. 22

D Threshold Sensitivity

• The fixed-threshold oracle gap averages 4.15 pp on HC3-SI and at most 0.35 pp on HC3-QA, M4, and MAGE; the worst HC3-SI case is 13.58 pp.Across the 24 cell-benchmark pairs, all 18 pairs outside HC3-SI fall between 0.02 and 0.99 pp.Source: D Threshold Sensitivity · p. 22, Figure 2 · p. 23
• For the released configuration, the oracle gap is 0.01 pp on HC3-QA, 0.10 pp on MAGE, and 0.25 pp on M4, but 2.97 pp on HC3-SI.Threshold-sweep curves show broad plateaux for HC3-QA and M4, while HC3-SI is sharply peaked for every cell.Source: D Threshold Sensitivity · p. 22, Figure 3 · p. 23

E Version History

• +2.82 pp TPR@5 % FPR from v1 to v2 followed changes to preprocessing, corpus composition, and adversarial coverage together.Later changes also affected data or loss while the backbone stayed fixed.Source: E Version History · p. 23
• v2.2 had the sequence’s highest AUROC, but v2.4-3k had the best TPR at both operating points; v2.3’s weighted loss cut TPR@1 % FPR by 2.35 pp.The v2.4-5k row showed that proportional validation improved HC3-SI at the cost of RAID operating-point robustness.Source: E Version History · p. 23
• The 13 pp A1–A3 gap reflects train–inference mismatch, not preprocessing’s contribution; factorial cells with normalised training and raw inference lost 12.79 and 15.80 pp under attack.A1 and A3 were both trained on normalised text, and A1 was evaluated on raw text.Source: E Version History · p. 23

F Validation-Set Composition

• Correcting validation from MAGE-heavy sampling restored HC3-SI from 81.18% to 85.33% while preserving RAID robustness.The initial validation set contained 62.6% MAGE samples, versus approximately 34% of training data; v2.2 HC3-SI was 84.04%.Source: F Validation-Set Composition · p. 24

G Factorial Ablation: Complete Breakdowns

• The feature branch’s poor poetry performance may reflect surface-statistic features whose distributions poetry systematically violates, yielding a confidently wrong contribution.The cited passage presents this mechanism as plausible, not established.Source: G Factorial Ablation: Complete… · p. 24
• Tables report TPR@1 % FPR by attack, domain, and generator across all eight factorial cells, with the best value per row marked.The attack breakdown is in Table 19; the domain and generator breakdowns are in Tables 20 and 21.Source: Table 19 · p. 25, Table 20 · p. 25, Table 21 · p. 25

G.1 Per-Cell Detail

• Table 18 gives the reported configuration at full precision; the other seven cells appear as columns in Tables 19–25.The reported configuration is without the feature branch, with raw training and normalised inference.Source: §G.1 · p. 25

H Per-Cell Performance Matrices

• The matrices show generator×domain, generator×attack, and domain×attack breakdowns on a shared 0–100 colour scale for cross-page comparison.Panels distinguish factorial cells, with TPR@1 % FPR or TPR@5 % FPR shown depending on the figure or table.Source: H Per-Cell Performance Matrices · p. 25, Table 22 · p. 26, Table 23 · p. 26