PaperlayerView exact revision on arXiv

Source-linked AI summary

SIM: Subspace Interaction-based Method for Token-Level Text Anomaly Detection

Kehan Yan, Yue Tan, Qingfeng Chen, Shiyuan Li, Yu Zheng, Yixin Liu

arXiv:2609.08200v1cs.LG

TL;DR

Token-level anomaly detection must localize anomalies despite global-distance dilution and PLM over-smoothing. SIM addresses these issues with interacting subspaces, hard pseudo-anomalies, and probabilistic boundary learning, and experiments report strong performance, robustness, efficiency, and interpretability.

  • Problem

    Existing methods dilute localized anomaly signals through global distance calculations, while PLMs smooth surface anomalies in token representations.

  • Method

    SIM decomposes token embeddings into interacting subspaces, generates hard pseudo-anomalies, and uses probabilistic boundary loss to separate anomalous scores statistically.

  • Results

    SIM achieves the best document-level AUROC and AUPRC across three datasets, best average token-level AUROC and AUPRC, and 98.44 token-level AUROC on SMS Spam versus GPT-4.1-nano's 92.19.

  • Takeaways & Limitations

    SIM provides fine-grained token localization alongside document-level detection and maintains a clear robustness margin under 10% training-data contamination.

Abstract

from arXiv · show

Token-level text anomaly detection, as an emerging trend of text anomaly detection, moves beyond coarse-grained document-level detection by localizing anomalous tokens within text. By providing fine-grained abnormality prediction, token-level text anomaly detection plays a critical role in various real-world applications, such as spam filtering and fake news detection. However, existing methods still rely on the global distance calculation for scoring, during which the local anomaly signals are severely diluted by numerous redundant normal feature dimensions. Moreover, pre-trained language models used in these methods inevitably smooth out surface anomalies, further limiting their effectiveness in token-level anomaly detection. To address these limitations, we propose a Subspace Interaction-based Method (SIM for short) for token-level text anomaly detection. To prevent local signal dilution, SIM adopts a subspace interaction-based anomaly detector, which decouples high-dimensional token embeddings into multiple low-dimensional ones, amplifying localized anomaly signals hidden within specific dimensions. To counteract the over-smoothing effect, we design a hard pseudo-anomaly generation module to construct pseudo-anomalous tokens, simulating the subtle anomalies obscured by semantic smoothing. Also, a probabilistic boundary loss is developed to standardize anomaly scores into statistical distances, effectively enforcing anomalous instances to deviate significantly from the normal distribution center. Extensive experiments on multiple benchmark datasets verify the effectiveness of SIM and demonstrate its remarkable efficiency, robustness, and interpretability. The source code is available at: https://github.com/yankehan/SIM-TAD.

I. INTRODUCTION

Token-level detection localizes anomalous tokens, addressing the limits of document-level scoring and global embedding distances. SIM combines subspace interaction, hard pseudo-anomalies, and probabilistic boundary learning to improve detection.

  • Text anomaly detection supports applications including spam filtering, fake news detection, and machine-generated content identification.
  • Document-level methods assign one score per document, making anomaly reasoning and problematic-segment localization difficult.
  • TokenCore extends anomaly scoring to tokens but global distance calculations dilute subtle anomalies concentrated in a few embedding dimensions.
  • SIM decouples token embeddings into low-dimensional subspaces and models their interactions to amplify localized anomaly signals.
  • SIM adds hard pseudo-anomaly generation and probabilistic boundary loss to address surface-anomaly smoothing and guide anomaly-score learning.
  • Extensive experiments on three real-world datasets report superior token- and document-level performance, efficiency, robustness, and interpretability.

II. RELATED WORK

Related work progresses from global-space anomaly detectors toward text-specific representations, but document-level methods still cannot localize anomalous tokens. This motivates fine-grained token-level detection.

  • Classical and representation-learning anomaly detectors primarily evaluate anomalies in the global feature space.
  • Global evaluation dilutes sparse, localized anomaly signals with redundant normal feature dimensions.
  • Document-level text anomaly methods include end-to-end and two-stage routes that assign anomaly scores to entire texts.
  • Because document-level scores cannot localize anomalous tokens, fine-grained token-level detection is needed for tasks such as grammatical error correction.

C. Token-Level Text Anomaly Detection

Token-level text anomaly detection learns token scores from normal data while identifying diverse, unknown anomalies. TokenCore provides an early baseline, but PLM smoothing and global distances remain limiting factors.

  • The task uses a one-class setting, learning normal token distributions without labels for actual anomalies.
  • It assesses document abnormality while precisely localizing anomalous tokens and must handle diverse, unknown anomaly variants.
  • TokenCore maps tokens into high-dimensional PLM representations and scores them by nearest-neighbor distance to normal tokens.
  • SIM targets global-distance dilution and PLM over-smoothing with subspace interaction, hard pseudo-anomaly generation, and probabilistic boundary loss.
  • Token-level scores can be aggregated into document-level scores, while also identifying triggering tokens and their anomaly intensities.

IV. METHOD

SIM detects localized anomalies by decomposing token embeddings into interacting subspaces rather than relying only on global representations. Cross-subspace attention emphasizes anomaly-related information before producing a token score.

  • Global embeddings can mask local anomalies because their anomalous features often occupy only a few dimensions.
  • SIM uniformly partitions each d-dimensional token embedding into m subspaces of dimension d_sub = d/m.
  • Learnable Query, Key, and Value projections provide a common basis for modeling relationships among subspaces.
  • Scaled dot-product cross-subspace attention lets each subspace aggregate information from all other subspaces.
  • The model adaptively emphasizes anomaly-related subspaces and suppresses redundant information, while accounting for cross-subspace structural inconsistencies.
  • After updated subspaces are concatenated, a two-layer MLP converts their representation into the final token-level anomaly score.

B. Hard Pseudo-Anomaly Generation

SIM generates hard pseudo-anomalies by perturbing normal token features according to local density, then projecting them back toward the global feature manifold. These samples preserve similarity to normal data while lacking meaningful semantic or syntactic structure, helping counter PLM over-smoothing.

  • SIM addresses PLM over-smoothing by synthesizing pseudo-anomalous tokens that are semantically close to normal tokens but lack meaningful semantic structure.
  • The method estimates each normal token’s local density from average distances to its K nearest neighbors and uses this density to determine perturbation direction and magnitude.
  • A repulsion-strength parameter β controls perturbation magnitude, with larger values pushing pseudo-anomalies farther from their original semantic anchors.
  • Hypersphere projection constrains perturbed features to the global manifold while preserving their original radial distance from the batch center.
  • The generated samples are physical feature-space displacements without semantic or syntactic structure, yet remain non-extreme outliers that provide discriminative training signals.

C. Probabilistic Boundary Loss

SIM standardizes token anomaly scores relative to a reference normal distribution and optimizes them with a probabilistic boundary loss. The objective clusters normal scores near the distribution center and separates pseudo-anomalies by a confidence boundary.

  • SIM samples 5000 standard-normal instances to estimate reference mean µref and standard deviation σref for normal anomaly scores.
  • Raw token anomaly scores are converted into standardized Z-scores, making deviations from the normal data center statistically interpretable.
  • The loss avoids overfitting to specific pseudo-anomaly patterns by optimizing statistically meaningful deviations from the normal distribution.
  • The probabilistic boundary loss drives normal-token scores toward |dev(s)| →0 while requiring pseudo-anomalous scores to remain at least α standard deviations from the normal center.
  • For document-level detection, SIM uses max pooling so the document score preserves the strongest token-level anomaly evidence instead of diluting it through averaging.

D. Computational Complexity

SIM’s complexity analysis excludes the cached pretrained-language-model forward pass and characterizes the detector using document length, embedding dimension, subspace count, and batch size.

  • With cached token embeddings, SIM’s subspace detector partitions embeddings into m subspaces and applies cross-subspace attention at cost O(Td^2/m + Tmd).

A. Experiment Setup

Experiments evaluate SIM on three benchmark datasets with document- and token-level AUROC and AUPRC, using BERT-base embeddings and three random seeds. SIM leads the reported comparisons, while ablations show that each component contributes to performance.

  • Experiment Setup: Experiments cover Grammar, Review, and SMS Spam, representing grammatical errors, negative sentiment, and text corruption.
  • Experiment Setup: AUROC and AUPRC are averaged across three random seeds, with BERT-base-uncased embeddings and max pooling for subword-to-word alignment.
  • Main Results: SIM achieves the best document-level AUROC and AUPRC across all three datasets, with over 10% relative gain over the strongest baseline.
  • Main Results: At token level, SIM obtains the best average AUROC and AUPRC, reaching 98.44 AUROC on SMS Spam versus 92.19 for GPT-4.1-nano.
  • Ablation Study: Removing subspace interaction, hard pseudo-anomaly generation, or probabilistic boundary loss consistently reduces performance.
  • Ablation Study: Replacing hard pseudo-anomaly generation with Gaussian noise produces the largest drop in most settings, including 72.14 →48.66 on Grammar at token level.

D. Hyperparameter Analysis

SIM’s hyperparameter analysis examines pseudo-anomaly generation and compares efficiency against representative baselines. Performance depends on anomaly type, while SIM achieves strong accuracy with low runtime on Grammar.

  • Hyperparameter Sensitivity: Across datasets, relatively large anomaly ratios α consistently achieve optimal performance, demonstrating the benefit of pseudo-anomaly samples during training.The preferred perturbation scale β varies by anomaly type.
  • Hyperparameter Sensitivity: Grammar and Review favor larger perturbation scales β because small perturbations fail to disrupt the original semantic and syntactic structures.
  • Hyperparameter Sensitivity: SMS Spam favors a smaller β because excessive perturbation creates trivial outliers and degrades detection precision.
  • Efficiency Analysis: SIM achieves the highest AUROC with consistently low runtime on Grammar compared with representative methods.
  • Efficiency Analysis: 0.77s: SIM runs faster than ECOD at 0.86s despite using a deep learning-based model.
  • Efficiency Analysis: 8.14%: SIM outperforms TokenCore in token-level AUROC with negligible efficiency trade-off.

F. Robustness Analysis

SIM remains effective under contaminated training data and provides interpretable token-level anomaly attributions. Its visualizations identify localized surface, semantic, and structural anomalies across datasets.

  • Robustness Analysis: At 10% contamination, SIM’s degraded performance still significantly surpasses every baseline’s peak performance under clean training data.
  • Robustness Analysis: As contamination increases, all models’ detection performance declines, but SIM preserves strong performance and maintains a clear margin over strong baselines.
  • Interpretability Analysis: Darker shading in Figure 7 represents higher token-level anomaly scores across datasets.
  • Interpretability Analysis: SIM assigns the highest score, 0.61, to the anomalous SMS Spam string “Kjjjjggjgjytfd,” pinpointing local structural corruption.
  • Interpretability Analysis: SIM identifies the Grammar anomaly “at” with score 0.48 and the Review descriptor “Tasteless” with score 0.47 while ignoring neutral tokens.
  • Conclusion: SIM’s experiments demonstrate state-of-the-art performance at token and document levels, alongside efficiency, robustness, and interpretability.
Loading 2609.08200v1…