PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification

Aleesha Zainab, Muhammad Ahmed Khalid, Faheem Ullah Khan, Asifullah Khan

arXiv:2608.16928v1cs.LG

TL;DR

Reliable sensitivity classification requires benchmarks free of residual label leakage, which can let models exploit document artefacts instead of genuine content signals. This paper introduces the leakage-controlled Strategic 16K corpus and evaluates six model architectures, finding BERT performs best while TF-IDF with Logistic Regression offers a lower-cost baseline.

  • Problem

    Prior PlusD sensitivity-classification studies rarely document leakage removal, limiting whether their scores reflect genuine sensitivity understanding or support fair, reproducible comparison.

  • Method

    The paper constructs Strategic 16K from 16,000 PlusD cables, removes three categories of embedded classification markers, and benchmarks six model architectures under identical conditions.

  • Results

    BERT achieves the strongest performance across all six models, with F1 = 89.33%.

  • Takeaways & Limitations

    TF-IDF with Logistic Regression provides a strong practical baseline at substantially lower computational cost than transformer fine-tuning.

  • Takeaways & Limitations

    A monolithic model has structural limitations because sensitivity can depend on multiple evidence types, including entities, topics, and document structure.

Abstract

from arXiv · show

Automatic sensitivity classification of organizational documents is a critical yet underserved problem, where the consequences of misclassification range from regulatory violations to security breaches. While AI-based approaches offer a scalable alternative to manual review, their reliability depends fundamentally on the integrity of training data. A pervasive but underreported problem in this domain is label leakage: residual classification markers embedded within document bodies that allow models to exploit surface shortcuts rather than learning genuine content-based sensitivity signals, producing performance estimates that are inflated and unreliable. This paper addresses this problem by introducing Strategic 16K, a carefully constructed, leakage-controlled corpus of 16,000 diplomatic cables sourced from the WikiLeaks Public Library of US Diplomacy (PlusD), and presents a systematic benchmark evaluating six model architectures spanning classical machine learning and transformer-based approaches. We document an extended leakage removal protocol that identifies and eliminates three categories of residual classification markers embedded within document bodies. On the clean benchmark, BERT achieves the strongest performance (Accuracy = 89.14%, F1 = 89.33%), followed by ELECTRA (Accuracy = 88.57%, F1 = 88.90%). Among classical models, TF-IDF with Logistic Regression achieves the strongest performance at significantly lower computational cost. These results constitute the first fully reproducible sensitivity classification benchmark constructed under explicit leakage-controlled conditions from WikiLeaks PlusD.

I. INTRODUCTION

Document sensitivity classification remains largely manual and difficult to scale, while label leakage lets models exploit embedded classification markers instead of learning genuine sensitivity signals. This paper introduces Strategic 16K and a reproducible, leakage-controlled benchmark for comparing classical and transformer-based models under identical conditions.

  • Problem: Manual document-level labeling is time consuming, inconsistent across reviewers, and unable to scale with organizations’ growing document volumes.Sensitivity classification is typically performed by trained reviewers before documents are routed, stored, or transmitted.
  • Label leakage: Embedded classification codes, phrases, and distribution notices can expose ground-truth labels, allowing classifiers to rely on markers rather than document sensitivity.These artefacts create a direct, machine-readable link between document bodies and labels.
  • Prior work: Prior PlusD studies generally do not disclose explicit leakage-removal procedures, limiting interpretation, fair comparison, and reproducibility of their reported scores.Without knowing which artefacts remained in training data, performance cannot establish genuine sensitivity understanding.
  • Contributions: Strategic 16K is a 16K-document WikiLeaks PlusD benchmark with documented leakage removal, sensitive-focused sampling, and identical-condition evaluation across six model architectures.The protocol targets three categories of residual classification markers embedded within document bodies and compares classical TF-IDF classifiers with transformer-based models.

II. RELATED WORK … C. Transformer-Based Models

Prior work established strong classical and deep-learning baselines for text and sensitivity classification, while transformer models introduced contextual representations that capture bidirectional token context. Applications included WikiLeaks PlusD sensitivity detection and privacy-sensitive classification.

  • A. Traditional Machine Learning Approaches: SVM achieved 97% accuracy on document categorisation, demonstrating that regularized linear models can perform strongly with lexically rich text.The evaluation used TF-IDF and bag-of-words representations alongside SVM, Random Forest, k-Nearest Neighbours, and Naive Bayes.
  • II. RELATED WORK: TF-IDF and bag-of-words representations provided solid baselines for text classification across domains.
  • B. Deep Learning Approaches: A CNN-based system detected sensitivity in WikiLeaks diplomatic cables with F1 score of 0.91 by segmenting long documents into shorter paragraphs.The segmentation redirected attention toward locally sensitive passages.
  • II. RELATED WORK: The CNN system represented the first reported deep-learning application to WikiLeaks PlusD sensitivity classification.
  • C. Transformer-Based Models: BERT uses self-attention across layers to model left and right token context simultaneously, unlike TF-IDF or recurrent models.Its representations are learned through bidirectional pre-training on the masked language-modelling task.
  • C. Transformer-Based Models: BERT fine-tuning achieved F1 = 0.95 on a privacy-sensitive classification task.This result came from Petrolini et al.'s work on automatic sensitive-data detection.

D. Summary of Research Gaps · III. DATASET AND PREPROCESSING · A. Source: WikiLeaks Public Library of US Diplomacy

The paper identifies undisclosed leakage-removal protocols in prior WikiLeaks PlusD studies and establishes PlusD as a large labeled source requiring custom collection and preprocessing. The dataset contains 251,287 diplomatic cables across seven official classification labels, with more than 100,000 unique documents deduplicated into structured CSV format.

  • D. Summary of Research Gaps: Prior WikiLeaks PlusD studies explicitly disclose no leakage removal protocol.This gap motivates a leakage-controlled benchmark.
  • A. Source: WikiLeaks Public Library of US Diplomacy: PlusD contains 251,287 diplomatic cables produced by US embassies worldwide.It is described as the largest publicly available collection of real government documents with official sensitivity labels.
  • A. Source: WikiLeaks Public Library of US Diplomacy: The dataset uses seven official classification labels assigned by trained government officials.The labels include UNCLASSIFIED, CONFIDENTIAL, LIMITED OFFICIAL USE, SECRET, UNCLASSIFIED//FOR OFFICIAL USE ONLY (FOUO), CONFIDENTIAL//NOFORN, and SECRET//NOFORN.
  • III. DATASET AND PREPROCESSING: No pre-built version of the data was available for direct use.The researchers therefore collected every document through a custom automated extraction pipeline.
  • III. DATASET AND PREPROCESSING: The extraction pipeline fetched HTML cable pages, extracted content and original labels, and removed leftover HTML artefacts.It repeatedly processed each classification category through the WikiLeaks PlusD API.
  • A. Source: WikiLeaks Public Library of US Diplomacy: More than 100,000 unique documents were deduplicated and stored in structured CSV format.Deduplication was part of the automated collection and preprocessing pipeline.

B. Binary Label Mapping · C. Leakage Removal Protocol

The corpus reduces seven source labels to binary Sensitive and Non-Sensitive classes, then removes three categories of embedded classification artefacts while preserving genuine contextual sensitivity terms. Clean-corpus analyses show that models must rely on document content rather than classification markers, document length, or other surface shortcuts.

  • B. Binary Label Mapping: UNCLASSIFIED and UNCLASSIFIED//FOUO were mapped to Non-Sensitive, while the five formally restricted labels were mapped to Sensitive.The mapping represents the basic access-restriction decision.
  • C. Leakage Removal Protocol: The preprocessing protocol identified and removed three types of leakage embedded in document bodies.These artefacts could let classifiers exploit surface shortcuts instead of learning genuine sensitivity signals.
  • C. Leakage Removal Protocol: Inline paragraph markers such as (C), (S), (U), and (SBU) were removed because they directly indicated paragraph-level classification.The markers were inserted at paragraph beginnings in specific cables.
  • C. Leakage Removal Protocol: Repeated classification phrases and distribution notices were eliminated from document bodies, including explicit SECRET statements and repetitive distribution boilerplate.Classification terms in natural, semi-natural contexts were retained when they reflected genuine content rather than artefacts.
  • C. Leakage Removal Protocol: No letter-spaced classification banners were found in the sampled subset, making the three identified categories the complete set of observed leakage artefacts.The corpus was therefore considered fully leakage-controlled with respect to the examined patterns.
  • C. Leakage Removal Protocol: The top TF-IDF features for both classes consisted entirely of generic English vocabulary, with no SECRET, CONFIDENTIAL, or NOFORN tokens among the highest-weighted features.This provides empirical confirmation that cleaned-corpus classifiers must use content-based signals.
  • C. Leakage Removal Protocol: TF-IDF with Logistic Regression achieved nearly perfect performance on raw documents but declined substantially after leakage removal, indicating raw-corpus inflation from residual artefacts.The comparison empirically confirms the severity of label leakage.
  • C. Leakage Removal Protocol: The length-only baseline achieved substantially lower accuracy than every evaluated model, showing that document-length differences cannot reliably proxy sensitivity.Strategic 16K therefore requires models to learn from document content rather than structural shortcuts.

D. Strategic 16K Benchmark

Strategic 16K is a publicly available, leakage-cleaned corpus of 16,000 randomly sampled documents with a near-balanced sensitivity distribution. Its broad length variation limits simple length-thresholding as a reliable classification strategy.

  • Corpus Composition: 16,000 documents comprise Strategic 16K, with 52.4% Sensitive and 47.6% Non-Sensitive documents.The corpus was assembled after leakage removal using sensitive-focused sampling and class-adjusted random sampling.
  • Corpus Composition: Near-balanced classes avoid structurally privileging one class during cross-validation evaluation.The complete dataset is publicly available to support reproducibility and future benchmarking.
  • Document Length: 276 words is the median Sensitive-document length, compared with 64 words for Non-Sensitive documents.Sensitive documents span short cables to documents exceeding 1,400 words, indicating substantial within-class length variation.
  • Document Length: Simple length-thresholding is unreliable because sensitive documents vary widely in length, while longer buckets contain more Sensitive documents.Non-Sensitive documents concentrate in the 0–100 word range.

IV. MODELS AND EXPERIMENTAL SETUP · A. Evaluation Protocol · B. Classical TF-IDF Models

The study evaluates six models under reproducible, leakage-controlled conditions using stratified five-fold cross-validation and weighted F1-score and accuracy as primary metrics. Its classical baselines use TF-IDF representations with sublinear term-frequency scaling, with hyperparameters tuned through nested randomized search.

  • A. Evaluation Protocol: Six models were evaluated using 5-fold stratified cross-validation on Strategic 16K with the 52.4%/47.6% class ratio preserved across folds.A fixed random seed of 42 was used throughout for reproducibility.
  • A. Evaluation Protocol: Weighted F1-score and accuracy were reported as the primary evaluation metrics for all models.Weighted F1 accounts for slight class imbalance while jointly capturing precision and recall.
  • A. Evaluation Protocol: Sensitive-class recall was treated as a secondary criterion because false negatives can cause sensitive documents to bypass appropriate access controls.The passage identifies potentially severe operational and regulatory consequences for undetected sensitive documents.
  • B. Classical TF-IDF Models: Three classical models were evaluated using TF-IDF feature representations with sublinear term-frequency scaling.The passage identifies Logistic Regression, Linear SVM, and a third classical model within this evaluation group.
  • B. Classical TF-IDF Models: Logistic Regression used L2 regularization and a linear decision boundary, with C = 25.13 and balanced class weights selected through randomized hyperparameter search.RandomizedSearchCV used 25 candidates and 5-fold inner cross-validation.
  • B. Classical TF-IDF Models: Linear SVM used a maximum-margin separating hyperplane in TF-IDF space, with C = 14.53 and balanced class weights selected as the best parameters.The cited passage reports these settings as the selected configuration for Linear SVM.

C. Transformer-Based Models · V. RESULTS AND DISCUSSION · A. Overall Performance

The benchmark fine-tuned three transformer architectures under stratified cross-validation and found BERT strongest overall, with ELECTRA close behind and classical TF-IDF models forming a lower-performing tier. RoBERTa occupied an intermediate position, while the reported metrics include sensitive recall and precision.

  • C. Transformer-Based Models: BERT uses masked language modelling with bidirectional context and a linear classification head attached to the [CLS] token.During pre-training, 15% of tokens are randomly masked and reconstructed.
  • C. Transformer-Based Models: RoBERTa modifies BERT through dynamic masking, removal of next-sentence prediction, and training on substantially larger data.These changes are intended to produce stronger general-purpose representations.
  • A. Overall Performance: BERT achieved the strongest performance across both accuracy and F1 on Strategic 16K.The complete results were reported using 5-fold cross-validation across all six models.
  • A. Overall Performance: 2.25 pp lower accuracy and 2.39 pp lower F1 placed ELECTRA behind BERT but clearly ahead of the classical models.These are the reported gaps between ELECTRA and BERT.
  • A. Overall Performance: The three classical TF-IDF models formed a distinct lower-performance tier, with SVM marginally leading on F1 and recall.RoBERTa occupied an intermediate position, underperforming both BERT and ELECTRA.
  • A. Overall Performance: S-Recall denotes Sensitive Recall, while S-Prec denotes Sensitive Precision.These abbreviations appear in the reported results table.

B. Transformer vs. Classical Gap

Transformers consistently outperform classical models on Strategic 16K, with BERT leading and statistically significant advantages over several classical baselines. This advantage is attributed to bidirectional document-wide context modeling that supports semantic rather than frequency-based discrimination.

  • Statistical significance: p < 0.001: paired t-tests confirm significant BERT advantages over SVM, Logistic Regression, and Naive Bayes.The reported comparisons are BERT vs. SVM (t = 15.19), BERT vs. LR (t = 12.52), and BERT vs. Naive Bayes (t = 13.80).
  • Model tiers: 86.51% F1: RoBERTa occupies an intermediate position between the transformer and classical tiers and performs comparably to the best classical model.The figure separates BERT, ELECTRA, and RoBERTa from LR, SVM, and Naive Bayes into transformer and classical tiers.
  • Mechanism: Self-attention considers bidirectional context between every word in the document, enabling semantic, context-sensitive discrimination less dependent on lexical frequency.This distinction is relevant when diplomatic cables share administrative wording but differ in operational sensitivity.

C. BERT vs. ELECTRA · D. Sensitive Class Recall Analysis · E. Efficiency Tradeoff: Classical vs. Transformer

BERT leads ELECTRA on overall accuracy and F1 with a statistically significant advantage, while ELECTRA achieves higher sensitive-class recall. Sensitive recall also highlights Naive Bayes’s conservative detection, whereas classical TF-IDF models offer strong performance and practical efficiency with CPU-based deployment.

  • C. BERT vs. ELECTRA: BERT outperforms ELECTRA by 0.57 pp on accuracy and 0.43 pp on F1.Across five fold-level F1 scores, the advantage is statistically significant (t = 8.20, p = 0.0012), with a mean F1 difference of 2.40 pp and a 95% confidence interval of (1.58 pp, 3.21 pp).
  • C. BERT vs. ELECTRA: BERT’s masked language modelling is sensitive to broad semantic patterns, whereas ELECTRA’s replaced token detection may fit PlusD vocabulary less well.The comparison attributes the difference to how each objective represents context and token plausibility in diplomatic cables.
  • D. Sensitive Class Recall Analysis: Sensitive class recall is the most operationally critical metric because it measures correctly identified truly sensitive documents.The analysis frames recall as especially important from a security standpoint.
  • D. Sensitive Class Recall Analysis: Naive Bayes achieves a sensitive recall of 86.40%, comparable to Logistic Regression at 86.41% and SVM at 86.74%, despite lower F1 and accuracy.Its recall profile indicates relatively conservative classification, although the model is less precise.
  • E. Efficiency Tradeoff: Classical vs. Transformer: SVM leads the classical models with F1 = 86.95%, using standard CPU resources without GPU fine-tuning.TF-IDF-based models are described as practical when inference latency and rapid model updates matter.
  • E. Efficiency Tradeoff: Classical vs. Transformer: Logistic Regression follows SVM with F1 = 86.83% and tunes in 362.6 seconds versus SVM’s 382.2 seconds.The comparison presents Logistic Regression as slightly quicker to tune while remaining close in classical-model performance.

VI. CONCLUSION · Limitations and Future Directions

The paper benchmarks six sensitivity-classification architectures on Strategic 16K, a 16,000-document leakage-controlled corpus, using a three-category protocol targeting residual classification markers. BERT performs best, while the authors identify structural limitations of monolithic models and propose a Multi-Agent System architecture for future work.

  • VI. CONCLUSION: Strategic 16K contains 16,000 WikiLeaks PlusD diplomatic cables and supports a systematic benchmark of six document sensitivity-classification models.The corpus was constructed under leakage-controlled conditions.
  • VI. CONCLUSION: The leakage-removal protocol targets inline paragraph markers, embedded classification phrases, and distribution notices in raw WikiLeaks PlusD cables.These artefact categories were not explicitly addressed by prior studies, according to the supplied passage.
  • VI. CONCLUSION: 89.14% Accuracy and 89.33% F1 make BERT the strongest and most stable evaluated architecture, with ±0.46% F1 standard deviation.ELECTRA achieves 88.90% F1 and may suit resource-constrained settings because of parameter efficiency.
  • VI. CONCLUSION: 86.95% F1 and 86.74% sensitive recall make SVM the strongest classical model, while Logistic Regression is described as comparable and slightly faster.RoBERTa achieves 86.51% F1 but underperforms both tran…
  • Limitations and Future Directions: A monolithic model has structural limitations for security-critical deployment because sensitivity can depend on who is mentioned, the topic, or phrasing.The supplied passage presents these as distinct signal types that a single model must handle.
  • Limitations and Future Directions: A shared attention mechanism cannot independently isolate, prioritize, or explain these signals, and policy changes require full model retraining.These limitations motivate the proposed architectural shift.
  • Limitations and Future Directions: Future work proposes a Multi-Agent System whose semantic, named-entity, and structural agents evaluate topical sensitivity, restricted references, and subject-line or routing cues.The named-entity agent targets restricted personnel, facilities, and programmes.
  • Limitations and Future Directions: A learned fusion mechanism will combine the specialized agents’ confidence estimates to produce the final decision and provide built-in explai…The supplied passage truncates the remainder of the explanation-related claim.
Loading 2608.16928v1…