PaperlayerView exact revision on arXiv

Source-linked AI summary

Shedding New Light on the Language of the Dark Web

Youngjin Jin, Eugene Jang, Yongjae Lee, Seungwon Shin, Jin-Woo Chung

arXiv:2204.06885v2cs.CLcs.IRcs.LG

TL;DR

Limited access to the Dark Web and a scarcity of public datasets constrain study of its language. The paper introduces CoDA, a 10,000-document corpus, and uses it to analyze linguistic properties and evaluate classification. The analysis finds lexical differences from the Surface Web, while SVM and BERT perform well in the Dark Web domain.

  • Problem

    Limited access to the Dark Web and a lack of public datasets leave its linguistic properties relatively underexplored.

    Source: §1 · p. 1

  • Method

    The paper introduces CoDA, a 10,000-document Dark Web corpus, and uses it for linguistic analysis and classification evaluation.

    Source: §1 · p. 2, §3.1 · p. 2

  • Results

    CoDA analysis finds clear lexical differences from the Surface Web, and SVM and BERT perform well in the Dark Web domain despite those differences.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors demonstrate CoDA’s practicality in two NLP use cases and speculate that its lower duplicate content than DUTA may aid application performance.

    Source: §7 · pp. 8–9

    The corpus excludes multi-topic webpages, leaving multi-label datasets and classification for future work.

    Source: §3.5 · p. 4

Abstract

from arXiv · show

The hidden nature and the limited accessibility of the Dark Web, combined with the lack of public datasets in this domain, make it difficult to study its inherent characteristics such as linguistic properties. Previous works on text classification of Dark Web domain have suggested that the use of deep neural models may be ineffective, potentially due to the linguistic differences between the Dark and Surface Webs. However, not much work has been done to uncover the linguistic characteristics of the Dark Web. This paper introduces CoDA, a publicly available Dark Web dataset consisting of 10000 web documents tailored towards text-based Dark Web analysis. By leveraging CoDA, we conduct a thorough linguistic analysis of the Dark Web and examine the textual differences between the Dark Web and the Surface Web. We also assess the performance of various methods of Dark Web page classification. Finally, we compare CoDA with an existing public Dark Web dataset and evaluate their suitability for various use cases.

1 Introduction

The Dark Web’s limited accessibility and volatile onion services make its topology and content difficult to characterize, while its language remains underexamined and public datasets are scarce. The paper introduces CoDA to support linguistic analysis, classification, and comparison with DUTA.

1 Introduction

  • Limited access and volatile onion services make the Dark Web’s topology and overall content difficult to grasp.Prior work has studied page classification and content analysis, but linguistic analysis has received relatively little attention (Choshen et al., 2019).Source: §1 · p. 1
  • The authors argue that a public Dark Web dataset could serve NLP and cybersecurity research, including page classification and malicious-activity detection.They attribute the lack of comprehensive NLP work on Dark Web language mainly to the scarcity of publicly available datasets.Source: §1 · p. 1
  • DUTA has a highly skewed category distribution, with human trafficking represented by only 3 of 10,367 services, and just 51% of its texts are unique.The paper identifies these shortcomings as relevant to using DUTA for language analysis.Source: §1 · p. 1
  • CoDA is introduced as a corpus for linguistic analysis of the Dark Web, classification, and comparison with DUTA.The authors connect the lack of publicly available Dark Web datasets with limited linguistic research and potential value to NLP and cybersecurity research.Source: §1 · pp. 1–2

2 Related Work

Prior Dark Web research has used recursive crawling, public dataset DUTA, and a range of text-classification methods. Earlier findings raised concerns about deep representations, while transformer models have shown promise but may not readily adapt to the domain.

2 Related Work

  • Crawling from Dark Web search engines and directories can work because the visible Dark Web is suggested to be well connected by hyperlinks.Prior work uses services such as Ahmia and The Hidden Wiki to find content recursively (Guitton, 2013; Al Nabki et al., 2017; He et al., 2019).Source: §2 · p. 2
  • DUTA is a public, manually labeled text dataset collected from active Tor domains and extended to 10,367 unique domains (Al Nabki et al., 2017; Al-Nabki et al., 2019).The paper describes it as the first and, to its knowledge, only publicly available Dark Web text dataset.Source: §2 · p. 2
  • Prior classification work used SVM, Naïve Bayes, and Logistic Regression with TF-IDF or Bag-of-Words representations (Moore and Rid, 2016; Al Nabki et al., 2017; Ghosh et al., 2017; Avarikioti et al., 2018; He et al., 2019).Choshen et al. (2019) reported subpar performance from GloVe and ELMo relative to traditional methods, and suggested specialized vocabulary and small training data may be factors.Source: §2 · p. 2
  • BERT has shown promise in text classification, although transformer models do not always adapt easily to the Dark Web domain (Ranaldi et al., 2022).Source: §2 · p. 2

3 The CoDA Corpus

CoDA is constructed as a manually annotated, topic-based corpus of Dark Web documents, with a category scheme and processing choices intended to support text analysis. The corpus covers 10,000 documents from 7,101 onion services, while its scope excludes some page types and multi-topic documents.

3.1 CoDA Category Set

  • CoDA uses ten topical categories and does not divide them into legal and illegal labels because legality can be difficult to determine from website surface information.It also excludes structural categories such as forums and marketplaces, which can overlap with topical labels; about 30% of documents are assigned to “others,” including blogs, news, search, and wiki pages.Source: §3.1 · pp. 2–3

3.2 Data Collection

  • The corpus collection expands onion-address seeds from Ahmia and onion-domain lists by finding unseen addresses in crawled pages, while retaining only one copy of redundant webpages.The deduplication step is intended to help construct a quality corpus.Source: §3.2 · p. 3

3.3 Data Size and Language Distribution

  • CoDA contains exactly 10,000 documents from 7,101 onion services, and about 88% of its documents are in English.The authors caution that the dataset reflects various biases of the Dark Web that future research should take into account.Source: §3.3 · pp. 3–4

3.4 Annotation

  • Ten annotators from a cyber threat analytics company manually assigned one category per page, achieving Fleiss’ Kappa of 0.887.Page-level labeling was chosen because pages within wikis and forums can cover a wide range of topics.Source: §3.4 · p. 4

3.5 Annotation Guidelines

  • Annotation guidelines classify each document by page-content topic rather than by page type, such as a marketplace, forum, or blog.The guidelines include examples and methods for handling borderline cases.Source: §3.5 · p. 4
  • CoDA excludes multi-topic pages and pages containing malicious information about personally identifiable individuals.The authors leave multi-label datasets and classification for future work.Source: §3.5 · p. 4

3.6 Additional Processing & Text Masking

  • Documents are language-labeled and processed with 18 mask-identifier types to anonymize sensitive information; punctuation and words longer than 50 letters are removed, and text is lemmatized and lowercased.Masking uses keywords, regular expressions, and a cryptocurrency-address detection library.Source: §3.6 · p. 4, Table 3 · p. 4

4 Data Analysis

The analysis compares CoDA with DUTA and English Surface Web datasets to characterize Dark Web language and document content. It examines vocabulary, word frequencies, document similarity, and category-specific masks and terms.

4 Data Analysis

  • The analysis assesses Dark Web linguistic properties and compares CoDA with other datasets, including the Surface Web.Some analysis methods and results are presented in the appendix.Source: §4 · p. 4, Table 4 · p. 5

4.1 Datasets for Comparison

  • CoDA and DUTA are compared using English-labeled documents, alongside an aggregate Surface Web corpus of reviews, wiki articles, and forum discussions.The Surface Web corpus is randomly sampled to match the size of the Dark Web counterparts; raw text is used for some comparisons to reduce masking bias.Source: §4.1 · p. 5

4.2.1 In-vocab / Out-of-vocab Word Analysis

  • Dark Web out-of-vocabulary words include explicit slang, malicious-activity abbreviations, and misspellings, while Surface Web examples include common abbreviations, companies, and celebrities.Out-of-vocabulary status is based on a spellchecker dictionary and does not necessarily mean a word is incorrect or nonexistent.Source: §4.2.1 · p. 5

4.2.2 Word Frequency Distribution

  • Word frequency distributions show no significant difference between the Dark Web and Surface Web, despite concern that near-identical phishing pages might inflate some word counts.The authors conclude that the distribution is largely domain-invariant.Source: §4.2.2 · p. 6

4.2.3 Document Similarity

  • More than half of DUTA documents share almost completely overlapping content, whereas CoDA documents show much lower maximum similarity.The authors interpret CoDA as more uniquely varied and higher in information density, despite DUTA covering more onion services.Source: §4.2.3 · p. 6, Figure 1 · p. 6

4.2.4 Mask ID Distribution and TF-IDF

  • Mask-ID frequencies and TF-IDF reveal category-specific signals, including a high proportion of ID_WEIGHT in drug listings and Apple-related terms in electronics.ID_NUMBER is excluded from the mask analysis because it dominates across categories.Source: §4.2.4 · pp. 6–7

5 Classification Experiments

The experiments compare SVM, CNN, and BERT classifiers trained on CoDA. BERT performs best overall, SVM is comparable, and performance varies across categories.

5.1 Setup

  • The study evaluates SVM, CNN, and BERT using a 70:30 stratified CoDA train-test split and preprocessing selected for performance across models.SVM uses TF-IDF features; CNN uses GloVe embeddings, and BERT uses a pretrained model.Source: §5.1 · p. 7

5.2 Results

  • BERT achieves the best weighted-average performance on CoDA, SVM is comparable, and CNN fares relatively worse.The authors suggest BERT may benefit from contextual information and unknown-word modeling, while CNN embeddings may poorly cover specialized vocabulary.Source: Table 7 · p. 7, §5.2 · p. 7
  • BERT performs relatively well on arms, electronics, and gambling, but worse on diverse-topic cryptocurrency and financial categories.It often confuses hacking with cryptocurrency or financial when phrases such as “hacked PayPal” occur, because those cases do not meet the paper’s hacking-category guidelines.Source: §5.2 · p. 7

6 Use Cases

The use cases test CoDA and DUTA for Dark Web synonym inference and document classification. On a separate benchmark, the CoDA-trained classifier consistently outperforms the DUTA-trained classifier.

6 Use Cases

  • Word2vec models trained on CoDA and DUTA associate Tesla and Toyota with drug-related terms; CoDA also links Wasabi to cryptocurrency terms, while DUTA lacks Wasabi in its vocabulary.The CoDA model’s top Wasabi neighbors include mustard, electrum, samourai, and mycelium.Source: §6 · pp. 7–8
  • The CoDA-trained BERT classifier consistently outperforms the DUTA-trained classifier on a 2,236-page benchmark spanning drugs, weapons, and finance.The benchmark excludes documents from matching URLs or mentioning benchmark website names; the authors conjecture CoDA’s lower duplication and more diverse domain-specific words support generalization.Source: Table 9 · p. 8, §6 · p. 8

7 Conclusion

The paper introduces CoDA and uses it to analyze Dark Web language and evaluate text classification. It reports lexical differences from the Surface Web, strong SVM and BERT performance, and two NLP use cases for CoDA.

7 Conclusion

  • The analysis finds clear lexical differences between the Dark Web and Surface Web, including abbreviations and differences in lexical structure such as PoS distribution.It also identifies category-level lexical characteristics through mask ID distribution and TF-IDF.Source: §7 · p. 8
  • SVM and BERT perform well on Dark Web classification despite the domain’s language differences from the Surface Web.Source: §7 · p. 8
  • The authors demonstrate CoDA’s practicality through two NLP use cases and speculate that its lower duplication than DUTA may aid application performance.Source: §7 · pp. 8–9
  • The authors hope the dataset and analysis will motivate further research in language-based Dark Web analysis.Source: §7 · p. 9

Ethical Considerations

The paper masks potentially sensitive identifiers, limits access to illegal media by collecting text and masking URLs, and retains text describing illegal activities. It also describes safeguards for annotators and research-only access to CoDA, alongside a staged annotation training process.

Masking Sensitive Information

  • Potentially sensitive data types, including email, IP, URL, crypto addresses, and social security numbers, are masked using regular expressions and manually checked.The authors report finding no sensitive content outside the masked information during these checks.Source: Masking Sensitive Information · p. 9

Handling Illegal Content

  • Pages are collected as HTML and parsed for text, while URL and onion addresses are masked to prevent access to illegal media.The authors state that they and annotators therefore do not access media that is illegal by law.Source: Handling Illegal Content · p. 9

Ethics on Annotation

  • Experienced Dark Web specialists consented to exposure to potentially unsettling content, and all ten annotators received equal page counts and compensation.Source: Ethics on Annotation · p. 9

Preventative Measures to Discourage Non-Academic Use of CoDA

A Inter-Annotator Agreement

  • Across three 50-document training intervals, Fleiss’ Kappa rose from 0.67 to 0.88 as annotators discussed disagreements and guidelines were revised.The remaining documents were assigned to single annotators, and the full process took about three months.Source: A Inter-Annotator Agreement · p. 12

B Experimental Details

The paper reports the computing environments used for analysis and classification, and provides a language-distribution table for CoDA.

B Experimental Details

  • Classification experiments used an Intel Xeon Gold 6258R CPU and Nvidia GeForce RTX 3090, while data analysis used an Intel Xeon E5-2630 v4 CPU without a GPU.Source: B Experimental Details · p. 12
  • The CNN used GloVe embeddings, three 2D convolution layers, and a fully connected layer, with 10 epochs of SGD training.The convolution kernels were 3, 4, and 5; learning rate was 1.5 and batch size was 32.Source: B Experimental Details · p. 12
  • BERT training used Adam for 10 epochs, with a 2e-5 learning rate, batch size 32, and maximum sequence length of 256 tokens.The sequence-length limit assumes the leading text part is indicative of topics.Source: B Experimental Details · p. 12

C Language Distribution

D Additional Data Analysis Methods & Results

The analysis compares PoS and content/function-word distributions across Dark and Surface Web categories. It reports higher content/function ratios and more proper nouns and numerals in Dark Web text, while noting limits in the Surface Web aggregate’s representativeness.

D.1 PoS Distribution & Content Word / Function Word Ratio

  • The analysis averages document-level PoS and content/function-word ratios by category because document lengths vary widely.Content words are tagged ADJ, ADV, NOUN, PROPN, VERB, X, or NUM; other words are treated as function words.Source: §D.1 · p. 13
  • Dark Web categories generally have higher content/function-word ratios, with higher proper-noun and numeral proportions; Surface Web text has more determiners.The authors suggest this may indicate a higher proportion of non-sentence structures in Dark Web language, giving drug-and-price lists as an example.Source: §D.1 · p. 13, Figure 3 · p. 14, Figure 2 · p. 14

D.2 Discussion

  • The authors note that spaCy was not pretrained on Dark Web content, but argue that either outcome of this potential error still indicates lexical differences between the domains.Source: §D.2 · p. 13
  • The aggregate Surface Web dataset may not represent the full domain: IMDb and Wikitext contribute many complete sentences, while marketplace content could affect the comparison.The authors propose broader Surface Web coverage and boilerplate removal as future analyses.Source: §D.2 · p. 13

D Additional Data Analysis Methods & Results

  • Figure 4 plots category-level word-frequency distributions on log-log axes and excludes DUTA because it has too many categories for the figure.Source: Figure 4 · p. 15

E TF-IDF Measurements

F Forum & Marketplace Benchmark Dataset

The benchmark dataset’s source is described in Section 6, while website titles and onion addresses are masked for ethical reasons.

F Forum & Marketplace Benchmark Dataset

  • Website titles and onion addresses are masked in the forum and marketplace benchmark dataset to follow ethical guidelines.Table 12 identifies the dataset source, with details described in Section 6.Source: Table 12 · p. 17

Related papers

Loading 2204.06885v2…