PaperlayerView exact revision on arXiv

Source-linked AI summary

PersianAnonymizer: Evaluating LLM-Labeled Training for Efficient NER-based Anonymization in Persian

Mohammad Hossein Shalchian, Mostafa Amiri, Amir Mahdi Sadeghzadeh

arXiv:2609.00958v1cs.CL

TL;DR

Persian customer chats contain sensitive information, motivating anonymization before downstream processing. The paper compares LLM labelers as supervision sources for compact NER models and evaluates quality, agreement, and efficiency. OSS_ZeroShot provides the most learnable supervision, while its NER model labels 40K test messages in about 2 minutes on one RTX 3090.

  • Problem

    Persian customer chats contain PII and PHI, making privacy-preserving anonymization important before downstream processing.

  • Method

    The paper compares three instruction-tuned LLM labelers, trains a compact NER model per resulting corpus, and evaluates token metrics, coverage, agreement, and efficiency.

  • Results

    OSS_ZeroShot supervision yields the most learnable Persian anonymization NER, balancing macro-F1 with broad entity-token coverage.

  • Takeaways & Limitations

    A compact NER trained on this supervision labels a 40K-message test set in 2 minutes on a single RTX 3090.

  • Takeaways & Limitations

    The study is limited by imperfect recall, sparse entity classes, limited human adjudication, aggregate Venn coverage, and data from one organizational setting.

Abstract

from arXiv · show

We target practical anonymization of Persian customer chats by training a compact NER model from LLM-labeled supervision and selecting the best labeler for deployment. We compare three instruction-tuned LLMs: DeepSeek-V3-0324, GPT-OSS-120B, and Qwen3-235B-A22B-Instruct-2507, to produce span annotations under a shared JSON protocol, yielding four corpora (OSS_ZeroShot, Qwen_ZeroShot, Qwen_FewShot, DeepSeek_FewShot). A MatinaRoberta-based token-classifier is trained per corpus and evaluated with token-level Precision/Recall/F1 (overall and per-class). We also report Label Coverage Recall (LCR), the proportion of gold non-O tokens predicted as non-O, and quantify cross-labeler behavior via a token-level Venn on test annotations. Finally, we contrast test-set annotation latency of the LLMs on H200 nodes with the trained NER's test-time labeling on a single RTX 3090. Results show that supervision from OSS_ZeroShot yields the strongest macro-F1 and LCR, while the resulting NER labels an entire 40K-message test set in approximately 2 minutes on one consumer GPU. This establishes a practical path to high-quality, low-cost anonymization for Persian industrial data.

1. Introduction

The paper addresses privacy risks in Persian customer chats by comparing LLM-generated supervision for compact NER anonymization. It evaluates labeler learnability and deployment efficiency to identify a practical labeling strategy.

  • Motivation: Raw customer chats can contain PII and PHI, creating privacy and compliance concerns when shared with external or closed-source models.Anonymization is proposed before downstream processing.
  • Motivation: Compact token-classification NER offers predictable, low-cost inference after reliable domain supervision is obtained.This contrasts with the multi-GPU resources and vendor exposure associated with high-capacity LLMs.
  • Research Gap: Persian NER resources are comparatively limited and are not tailored to industrial chat anonymization.Existing resources therefore provide limited direct coverage for this setting.
  • Research Gap: The study targets the underexplored choice of LLM labeler for learnable Persian anonymization NER and deployment-efficient performance.It also responds to documented performance gaps for LLMs in Persian.
  • Approach: Three instruction-tuned LLMs label Persian data, separate compact NER models are trained, and evaluation covers quality, agreement, and efficiency.The paper reports overall and per-class token-level metrics alongside cross-labeler comparisons and end-to-end timing.

2. Related Work

Prior work combines NER-based de-identification with LLM-assisted labeling, but Persian anonymization from industrial LLM-labeled data remains insufficiently studied. This paper positions unified labeler comparison and downstream learnability as the missing evaluation focus.

  • NER-Based De-identification: De-identification commonly treats PII and PHI as named entities and masks them through sequence labeling before downstream processing.Recent pipelines also combine high-recall NER with conservative post-processing to reduce residual leakage.
  • LLMs as Labelers: LLM-as-annotator pipelines use prompt design, retrieval, and active acquisition to reduce supervision noise and cost.Related strategies include active learning and robust training for weakly supervised NER.
  • Paper Positioning: The paper focuses on Persian chat anonymization, comparing multiple LLM labelers under one pipeline and evaluating downstream learnability and coverage.This links label generation directly to the performance of compact task models.
  • Paper Positioning: The paper addresses the gap between general Persian NER resources and LLM-labeled industrial data for anonymization.Its framing connects existing de-identification and weak-supervision methods to a Persian domain-specific setting.
  • Persian NER: Persian NER has established corpora and models, including PERSONER, PEYMA, and ParsBERT-based systems.Most prior efforts target generic entity extraction rather than industrial anonymization.

3. Dataset Construction

The study constructs a large Persian organizational-chat corpus, obtains span labels from multiple instruction-tuned LLMs, and normalizes them for compact NER training. It also examines annotation validity, alignment, efficiency, and cross-labeler behavior.

  • Corpus: The corpus contains 265,000 short, informal organizational-chat messages with PII-bearing and PII-adjacent content.Examples include names, phone numbers, emails, URLs, payment identifiers, dates, addresses, and organizations.
  • Label Schema: The annotation schema uses BIO tags over 14 PII-oriented entity types, plus the outside tag O.Each entity type receives B- and I- prefixes, including PERSON, URL, IBAN, and ORGANIZATION.
  • LLM Annotation: Three instruction-tuned LLMs receive the same prompt template and parsed JSON contract to produce phrase spans and canonical entity types.The standardized response contains the original text and a named_entities list.
  • LLM Annotation: Four usable corpora result from model- and prompting-specific format stability: OSS_ZeroShot, Qwen_ZeroShot, Qwen_FewShot, and DeepSeek_FewShot.GPT-OSS few-shot and DeepSeek zero-shot variants were discarded because outputs frequently violated the required format.
  • Alignment and Recovery: All datasets are normalized into a common token-level BIO format for consistent NER training and evaluation.The resulting corpora are suitable for fine-tuning token-classification models.
  • Alignment and Recovery: Span outputs are deterministically aligned to token boundaries and converted into BIO tags, with fuzzy recovery for missing phrase matches.Recovered fuzzy matches used a similarity threshold of 0.6 and represented fewer than 5% of annotations.
  • Efficiency: Each dataset has 225,000 training and 40,000 test messages, while LLM annotation timing is measured on identical 8×H200 nodes.The timing table includes parsed-API retries, which remained under 5% for all labelers.
  • Cross-Labeler Comparison: Qualitative comparisons show near-perfect agreement on structured identifiers but divergent treatment of titles, organizations, and locations.One example shows DeepSeek merging a locative phrase into an organization span, highlighting boundary sensitivity.

4. Training

The study fine-tunes identical MatinaRoberta-based token classifiers on four LLM-labeled corpora, using controlled splits and shared training settings. Training data comprise 225K messages per corpus, with a held-out development split and an untouched 40K-message test set.

  • Model and data: Each corpus trains a MatinaRoberta-based token classifier with a BIO-matched linear head and identical hyperparameters.The backbone derives from XLM-RoBERTa Large, and all four corpora are trained under the same configuration for comparability.
  • Model and data: Each corpus uses 225K training messages, a stratified 10% development holdout, and an untouched 40K-message test set.
  • Optimization: Training uses AdamW, a 2×10−5 learning rate, linear warmup, FP16, gradient clipping, constant batch size, and early stopping through six epochs.Each run executes on one RTX 3090 and takes about four hours.
  • Evaluation setup: Table 3 reports macro-averaged token-level Precision, Recall, F1, and Label Coverage Recall across the four trained models.
  • Evaluation setup: Figure 2 compares per-class token-level F1 across models, while Figure 3 compares token-level non-O annotation overlap among three LLM-labeled test sets.

5. Evaluation

Evaluation compares the four trained NER models using token-level quality, cross-dataset transfer, labeler overlap, and throughput. OSS_ZeroShot provides the strongest overall supervision, while the compact NER delivers rapid single-GPU inference.

  • Comparison of trained NER models: The OSS_ZeroShot-trained NER achieves the highest macro-F1 and strongest LCR, indicating better class-balanced quality and broader entity-token coverage.
  • Comparison of trained NER models: High-frequency URL, IP_ADDRESS, EMAIL, and PHONENUMBER categories perform consistently well, whereas LOCATION and ORGANIZATION remain comparatively challenging.The latter difficulty is associated with intertwined institutional and locative cues in informal Persian.
  • Comparison of trained NER models: CREDIT_CARD and IBAN occur only a few dozen times across datasets, so their per-class F1 values are statistically unreliable and variance-prone.These entities are uncommon in the target organizational support-ticket domain.
  • Cross-dataset validation: On PEYMA, ParsBERT-NER leads most categories and macro averages, while NER_OSS achieves stronger COST recall and F1.
  • Cross-dataset validation: On the OSS-labeled test set, NER_OSS dominates overall, with particularly large recall gaps on DATETIME and ORGANIZATION.ParsBERT-NER retains a slight PERSON-precision advantage, but its overall performance is constrained by low recall.
  • Labeler agreement: The cross-labeler analysis treats every non-O token as labeled and computes token-level overlap across GPT-OSS-120B, Qwen3 zero-shot, and DeepSeek few-shot annotations.
  • Practical throughput: Each trained NER labels the entire 40K-message test set in roughly 2 minutes on one RTX 3090, versus tens of minutes for the original LLMs on multi-node H200 setups.This supports lightweight anonymization at a fraction of the computational cost.

6. Conclusion

OSS_ZeroShot supervision produced the most learnable Persian anonymization NER, combining strong macro-F1 with broad entity-token coverage. The resulting compact model supports fast deployment and motivates targeted future improvements.

  • OSS_ZeroShot supervision yields the most learnable Persian anonymization NER, balancing macro-F1 with broad entity-token coverage measured by LCR.
  • A compact NER trained on this supervision labels a 40K-message test set in 2 minutes on a single RTX 3090.
  • Lightweight heuristics are recommended for challenging entity types such as ORG/LOC boundary cues where recall matters most.
  • Future work includes human audits, agreement-aware label consolidation, and transfer evaluation across adjacent Persian domains.

7. Ethics Statement

The study used authorized internal chat data for privacy-preserving anonymization research and released no raw customer messages. Examples were anonymized, while task-specific model checkpoints may be releasable under confidentiality constraints.

  • All chat data were processed under explicit organizational authorization for non-commercial, internal privacy-preserving anonymization research.
  • No raw customer messages are released, and examples replace identifying strings with semantically similar surrogates while removing or perturbing sensitive details.
  • The underlying chat datasets cannot be shared because of confidentiality and data-protection commitments.
  • Task-specific token-classification checkpoints may be released because they do not generate text, reducing the risk of reproducing original messages.

8. Limitations

The study reports imperfect recall, sparse-class instability, limited human adjudication, and domain-specific evaluation. These constraints bound interpretation of class-level metrics and transfer beyond the source organizational setting.

  • Recall is lower for ORGANIZATION and LOCATION, with persistent boundary errors in informal, code-mixed contexts.
  • CREDIT_CARD and IBAN have few instances, making their per-class F1 values variance-prone and difficult to interpret reliably.
  • LLM-generated supervision is evaluated through downstream learnability without large-scale human adjudication.
  • The token-level Venn aggregates non-O labels and measures coverage overlap rather than class agreement.
  • Data from a single organizational setting may limit performance under shifts in industries, styles, or policies.

10. Language Resource References

The paper’s language-resource references span Persian NER corpora and models, multilingual foundations, annotation LLMs, and the confidential industrial chat corpus used in the study.

  • Qwen3-235B-A22B-Instruct-2507 and GPT-OSS-120B are listed as models used for dataset annotation.
  • XLM-RoBERTa is referenced as the multilingual transformer base underlying MatinaRoberta.
  • ParsBERT is cited as a Persian language-understanding model used for evaluation.
  • The Persian Industrial Chat Corpus is identified as a confidential, non-public industrial dataset used under authorization.
  • PERSONER and PEYMA are cited as Persian named-entity corpora used in the resource lineage.
Loading 2609.00958v1…