PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Automated Cyber Threat Intelligence Elicitation in Underground Forums

Lorenzo Bossi, Federico Saccani, Francesco Panebianco, Antonio Maci, Stefano Zanero, Stefano Longari, Michele Carminati

arXiv:2608.23185v1cs.CR

TL;DR

As operationally relevant information becomes less visible in open forums and may emerge through dialogue, DarkBot uses specialized agents for active CTI elicitation. In simulation, it recovered 72.8% of MITRE ATT&CK techniques from initial posts.

  • Problem

    As relevant information shifts toward private spaces and may emerge through dialogue, passive collection may miss insights that require active engagement.

    Source: §I · p. 1

  • Method

    DarkBot uses eleven specialized agents to gate engagement, plan and generate ATT&CK-guided questions, handle adversarial input, and adapt language to forum communities.

    Source: §I · p. 2

  • Results

    72.8% of MITRE ATT&CK techniques were recovered in simulated CrimeBB conversations from initial posts; across 104 live conversations, CTI disclosures occurred without observed suspensions, moderator interventions, or accusations.

    Source: §I · p. 2, §VIII · p. 13

  • Takeaways & Limitations

    The findings support controlled interaction with publicly accessible underground communities as a CTI collection paradigm distinct from passive monitoring.

    Source: §I · p. 1

    Live deployment covered specific forums, and ethical constraints limited its scale and duration, reducing statistical power and generalizability.

    Source: §VII · p. 13

Abstract

from arXiv · show

Cyber threat intelligence from underground forums has traditionally relied on passive monitoring. However, as users have become more aware of large-scale data collection, valuable intelligence has become increasingly rare in open forums, often migrating instead to private or harder-to-reach spaces, making passive approaches inadequate. Building on the intuition that relevant information can be obtained through active elicitation, this paper presents DarkBot, to the best of our knowledge, the first multi-agent LLM-based system for active CTI elicitation in underground forums. DarkBot decomposes the interaction task across eleven specialized agents organized into three functional blocks: engagement gating for relevance and safety filtering, context-aware question generation driven by MITRE ATT&CK tactics, and linguistic style adaptation to better align with real forum users. In a controlled evaluation across 100 CrimeBB conversations, the system recovered 72.8% of the validated MITRE ATT&CK techniques present in the original discussions by observing only the initial post at the start of each interaction, and it consistently outperformed a monolithic baseline. The proposed layered safety design contained all injected jailbreak attempts at the pipeline level. These results were further supported by real-world experiments: in a prospective matched deployment, threads assigned to DarkBot accumulated an average of 3.85 more CTI entities than their controls over seven days, and across 104 live forum conversations, the system elicited CTI-relevant disclosures without observed account suspensions, moderator interventions, or explicit accusations of automated participation.

I. INTRODUCTION · II. BACKGROUND · III. RELATED WORK

DarkBot addresses the limits of passive CTI collection by using specialized LLM agents to elicit threat intelligence through interactive engagement in underground forums. The paper situates this approach within OSINT and CTI research and distinguishes it from prior chatbot and multi-agent systems that do not actively elicit CTI through recurring forum conversations.

I. INTRODUCTION

  • Because users’ awareness of large-scale data collection is reducing operationally relevant information in open forums, passive collection may miss intelligence that emerges through dialogue and targeted questions.CTI turns heterogeneous threat data into contextualized insights, including tactics, techniques, procedures, motivations, and likely targets; online communities can anticipate vulnerabilities, leaks, and attack campaigns.Source: §I · p. 1
  • DarkBot recovered 72.8% of MITRE ATT&CK techniques across 100 simulated CrimeBB conversations from only the initial post; threads in 20 matched pairs averaged 3.85 more CTI entities than controls.Ablations also showed that multi-agent decomposition consistently outperformed a monolithic baseline.Source: §I · p. 2
  • Across 104 live conversations and 213 published messages, DarkBot elicited CTI disclosures without observed bans, moderator actions, or AI accusations.Disclosures included details of active data breaches and newly published attack techniques already being tested in the wild.Source: §I · p. 2
  • Its layered safety architecture contained 100% of tested jailbreak attempts at pipeline level, while filtering blocked most policy-violating outgoing messages.In separate evaluations, DEFENSIVEAGENT detected 92.8% of jailbreak prompts, and FILTERAGENT blocked 85.6% of slang-rewritten policy-violating messages; undetected questions were borderline content.Source: §I · p. 2
  • DarkBot is presented as the first multi-agent LLM system for active CTI elicitation in public underground forums, decomposing interaction across eleven specialized agents.The agents support engagement gating, ATT&CK-guided elicitation planning, question generation, adversarial-input handling, and community-specific linguistic adaptation.Source: §I · p. 2

II. BACKGROUND

  • OSINT gathers information from public or semi-open sources, while CTI structures it for defensive decisions; MITRE ATT&CK organizes adversarial behavior into TTPs –.Prior work incorporates LLMs into CTI extraction, including benchmark evaluation, report summarization and data augmentation, entity and relation extraction with retrieval-augmented reasoning, and prompt chaining or multi-agent workflows –.Source: §II · p. 2

III. RELATED WORK

  • Prior investigative chatbots include conceptual dark-web evidence collectors, scam-engagement systems [25], [26], and tools for analyzing collected forensic evidence, but not sustained CTI elicitation in underground forums.Adaptive LLM profiling can also threaten user privacy by stealthily eliciting personal information [28].Source: §III · pp. 2–3
  • Although multi-agent LLMs have been applied to passive forum analysis and other security tasks, the paper identifies no prior system for recurring interactive CTI elicitation from underground-forum users.It investigates whether agent role decomposition can improve effectiveness, robustness, and control in adversarial conversations.Source: §III · p. 3

IV. MOTIVATION AND PROBLEM STATEMENT · V. PROPOSED METHODOLOGY · A. Design Principles

As valuable CTI has become scarcer in public forums, this work investigates shifting from passive monitoring to active elicitation and presents DarkBot, a multi-agent system designed for safe, credible participation. Its design addresses relevance, adversarial safety, targeted questioning, and linguistic fit.

IV. MOTIVATION AND PROBLEM STATEMENT

  • As public-forum CTI has declined while relevant discussions migrate to harder-to-reach spaces, success means eliciting human-contributed CTI absent from the pre-intervention thread.The CTI delta compares human contributions in the final conversation with CTI already present before intervention; bot messages are excluded.Source: §IV · p. 3
  • The system must avoid detectable or overly persistent participation while resisting adversarial manipulation and terminating conversations when appropriate.The problem requires bounding interaction frequency and turns per thread; excessive insistence raises suspicion and detection risk.Source: §IV · p. 3
  • Because CTI is fragmented and implicit in informal forum language, the system must ask strategic, targeted questions rather than generic conversational replies.Questions should increase the chance of obtaining relevant information while minimizing detection risk.Source: §IV · p. 3

V. PROPOSED METHODOLOGY

  • DarkBot coordinates 11 specialized LLM agents in iterative rounds to gate engagement, generate context-consistent questions, and continue or terminate after inspecting replies.Separate agents and explicit decision logic divide relevance, safety, planning, discourse style, and question generation rather than assigning them to one prompt.Source: §V · p. 4

A. Design Principles

  • DarkBot uses specialized agents because CTI elicitation combines multiple parallel challenges, and multi-agent approaches have shown gains on complex security tasks,.The architecture decomposes the task into specialized agents.Source: A. Design Principles · p. 4
  • A dedicated engagement gate limits interaction to CTI-relevant threads, detects jailbreak attempts or bot accusations, and enforces ethical guidelines.It terminates conversations involving those adversarial signals.Source: A. Design Principles · p. 4
  • Questions organized around 14 MITRE ATT&CK tactics make elicitation more tractable than tracking over 200 techniques while allowing specific techniques to arise organically.High-level attacker goals align with forum-thread topics and are expected to maximize response probability; tactic-level tracking is more manageable for an LLM.Source: A. Design Principles · p. 4
  • Credible participation requires matching thread tone and community slang, which general-purpose aligned models may suppress or fail to produce [30]–.Tone matching may help messages blend into discussions and improve elicitation, as perceived similarity favors information sharing,.Source: §IV · p. 3, A. Design Principles · p. 4

B. Functional Blocks

DarkBot organizes its pipeline into three functional blocks for engagement gating, context-aware question generation, and linguistic style adaptation. Together, these blocks filter unsafe or irrelevant interactions, guide questions toward CTI-relevant tactics, and adapt messages to forum conversation styles.

Engagement Gating

  • Engagement gating filters threads and replies for CTI relevance and safety, terminating interactions that violate constraints and discarding policy-breaking generated responses.Separate analysis and gating agents distinguish topical relevance from adversarial manipulation; the approach follows Shah et al. and addresses context-dependent jailbreak risks [36],. Prompt-based gating was chosen because LlamaGuard3 [38] can suppress hacking-related CTI discussions.Source: B. Functional Blocks · p. 4

Context Modeling and Question Generation

  • Context modeling scores candidate MITRE tactics and uses the highest-scoring effective tactic to guide follow-up questions grounded in extracted attack behaviors.A dynamic map covers all fourteen tactics; scores weigh evidence strength, response probability, intelligence value, conversational flow, and ATT&CK applicability. If no tactic is sufficiently effective, the pipeline terminates. Question generation avoids procedural instructions or encouraging specific actions. The design adapts Yuancheng Liu’s project.Source: B. Functional Blocks · pp. 4–5

Linguistic Style and Credibility

  • Linguistic style adaptation aligns questions with a conversation’s tone and stylistic patterns while preserving their semantic intent and reducing AI-suspicious phrasing.A reviewer checks style, safety, and natural single-question structure, sending revisions back to the question agent; an approved question then receives community-specific slang cues.Source: B. Functional Blocks · p. 5

VI. EXPERIMENTAL SETUP · A. Evaluation Framework · B. Intelligence Probing Evaluation (EQ1)

The evaluation tests DarkBot’s ability to identify productive questions from an initial forum post, alongside safety, architectural, intelligence-yield, and live-operation questions. In the controlled EQ1 simulation, DarkBot recovered most techniques present in the full conversations by probing from the initial post alone.

VI. EXPERIMENTAL SETUP

  • To protect potentially sensitive information, the evaluation used open-weight models on offline hardware: abliterated Gemma3-27B for uncensored attack reasoning, Qwen3-32B for reasoning and classification, and a fine-tuned 4B model for slang conversion.The paper assigns models according to agent needs and the narrowness of the slang-conversion task.Source: §VI · p. 5

A. Evaluation Framework

  • The framework asks whether DarkBot can identify productive inquiry, remain safe against adversarial inputs, and improve performance through its architectural choices.EQ1 uses simulated conversations; EQ2 examines prohibited topics, jailbreak detection, and blocking policy-violating content; EQ3 compares the full pipeline with a single-agent baseline and ablates components, using simulated extraction and a stylistic-credibility survey.Source: A. Evaluation Framework · pp. 5–6
  • The framework also tests whether participation raises intelligence yield and whether DarkBot can elicit information in real threads without detection or intervention.EQ4 uses a prospective matched experiment with identical initial posts; EQ5 deploys DarkBot in real underground forums and assesses conversations, bans, moderator interventions, and AI detection.Source: A. Evaluation Framework · p. 6

B. Intelligence Probing Evaluation (EQ1)

  • The controlled simulation used 100 CrimeBB threads: DarkBot saw only each initial post, while an LLM user proxy could answer from the complete original discussion.Live experiments were avoided to prevent iterative interactions from revealing the study, and domain experts were considered infeasible for consistent, unbiased simulation at scale.Source: B. Intelligence Probing Evaluation (EQ1) · p. 6
  • The proxy returned only directly targeted source replies, and technique recovery was manually annotated rather than scored from the extractor’s raw output.The extractor proposed candidate mappings for domain-informed verification; annotators also reread threads to find missed techniques. Evaluation excluded initial-post techniques, repeated mentions, and techniques appearing only in DarkBot’s questions; efficiency measured newly evidenced techniques per published question, with unique reply URLs as a secondary yield measure.Source: B. Intelligence Probing Evaluation (EQ1) · p. 7
  • DarkBot recovered 72.8% of the techniques in complete conversations, with a probing efficiency of 0.171 techniques per question.The initial-post baseline was 32.3%; across the experiment, DarkBot published 928 questions, recovered 159 new techniques, and received 122 source-matching URLs.Source: Figure 2 · p. 7, B. Intelligence Probing Evaluation (EQ1) · pp. 7–8
  • Message limits characterized recovery after the first x questions, but served as caps rather than targets because the pipeline could stop when no tactic met its effectiveness threshold.Recovery was non-decreasing by construction, and configurations shared the same initial-post baseline.Source: B. Intelligence Probing Evaluation (EQ1) · p. 7, Figure 2 · p. 7

C. Robustness Evaluation (EQ2)

The robustness evaluation found that the full pipeline rejected all tested jailbreaks, while individual safeguards had measurable misses. Engagement and message-filter tests also assessed safe targeting and outgoing-content blocking.

Engagement Gating Validation

  • The system classified 91.5% of Hacking texts as Relevant; among five relevant Arms, Porn, and Violence texts, it blocked the Porn question and kept the other four focused on CTI.The four questions did not engage with or address the harmful aspects of the Arms and Violence discussions.Source: C. Robustness Evaluation (EQ2) · p. 8, Figure 3 · p. 8

Robustness Against Adversarial Inputs

  • The standalone DEFENSIVEAGENT detected 92.8% of 845 jailbreak prompts, though it was not a perfect classifier.The overall pipeline also uses the REPLYANALYZER to detect attacks that deviate from the conversation topic and the FILTERAGENT as a final safeguard.Source: C. Robustness Evaluation (EQ2) · p. 8
  • The full pipeline rejected 100% of tested jailbreak prompts across three representative conversations, with reply analysis detecting off-topic attacks and the final filter blocking unsafe messages.The DefensiveAgent alone missed 7.2% of prompts, but the end-to-end scenarios produced no response to any injected jailbreak.Source: C. Robustness Evaluation (EQ2) · pp. 8–9

Message Filter Validation

  • The FILTERAGENT blocked 311 of 347 harmful questions (89.6%), falling to 85.6% after slang rewriting; assuming independent, transferable failures, combined-layer failure was estimated at 1.04%.Most undetected questions were borderline rather than explicitly harmful.Source: C. Robustness Evaluation (EQ2) · pp. 8–9

D. Design Validation (EQ3)

Design validation found that the multi-agent pipeline improved technique recovery and probing efficiency over a single-agent baseline, while stylistic ablations showed stronger support for slang adaptation than for the TONEAGENT. The evaluation also found higher computational costs and compared the TTP extractor with LADDER on labeled reports and forum threads.

Multi-Agent Architecture Evaluation

  • 72.8% vs. 51.4% technique overlap and 0.171 vs. 0.131 new techniques per question favored the multi-agent pipeline over the single-agent baseline.The multi-agent pipeline retrieved 122 unique external references versus 47 for the single-agent baseline.Source: D. Design Validation (EQ3) · pp. 9–10

Stylistic Credibility

  • 44.41% of stylistic-credibility judgments favored the full-pipeline messages as more human-like, versus 35.25% favoring the single-agent output and 20.34% perceiving no difference.The survey recruited 118 participants with a Computer Science background and no incentives.Source: D. Design Validation (EQ3) · p. 9

Computational Cost

  • The multi-agent system used 37,485 vs. 20,701 completion tokens and 422,280 vs. 340,006 prompt tokens, more than the single-agent system.The authors describe this as aligning with prior literature on the benefits and costs of modular reasoning in complex intelligence extraction tasks.Source: D. Design Validation (EQ3) · p. 9

Agent Ablations

  • 60.51% preferred full-pipeline messages over outputs without the SLANGCONVERTER, while 17.97% preferred the ablated variant; only 32.20% preferred full-pipeline output over the no-TONEAGENT variant, and 46.10% rated it worse.The ablation survey covered five representative threads; the authors retained the TONEAGENT and left its improvement as future work.Source: D. Design Validation (EQ3) · p. 9

Intelligence Probing

  • 72.8% final recovery for the full pipeline fell to 67.4% when tactic selection was integrated into the QUESTIONAGENT.The ablation yielded 0.165 vs. 0.171 new techniques per published question and 0.108 vs. 0.131 new URLs per question; explicit planning helped sustain recovery by tracking covered tactics and prioritizing underexplored dimensions.Source: D. Design Validation (EQ3) · pp. 9–10

TTP Extraction Validation

  • On 105 labeled AnnoCTR reports, the LLM extractor achieved 46.4% recall and 19% precision, versus LADDER’s 15.8% recall and 18.8% precision.On 100 CrimeBB threads, it extracted 1,996 techniques (19.96/thread), versus LADDER’s 50 (0.50/thread); CrimeBB results indicate candidate yield, not accuracy against expert labels.Source: D. Design Validation (EQ3) · p. 10
  • On AnnoCTR, the LLM system achieved 74.4% tactic recall and 50.9% precision, versus LADDER’s 45.9% recall and 46.7% precision.On CrimeBB, it identified 902 tactics (9.02 per thread), while LADDER’s inferred tactics numbered 46 (0.46 per thread).Source: D. Design Validation (EQ3) · p. 10

E. Intelligence Yield Effect Evaluation (EQ4) · F. Automated Forum Deployment (EQ5)

In a matched deployment, DarkBot threads yielded more CTI entities than controls; in live forum conversations, it elicited new operational disclosures and richer user activity without observed enforcement actions or explicit automation accusations.

E. Intelligence Yield Effect Evaluation (EQ4)

  • Across 20 matched pairs, DarkBot threads yielded 3.85 more CTI entities on average than controls (95% CI 0.37–7.38; p=0.033).The pairs used identical posts published at the same time, with bot assignment counterbalanced across forums and sections; counts covered human-authored messages over seven days. Bot-interacted threads had 223 entities in 90 human messages, versus 154 entities in 84 messages on controls. A matched CAPTCHA-bypass thread also elicited details about using Gemini to solve audio CAPTCHAs and mass-create GitHub accounts, with no comparable disclosure in its control.Source: E. Intelligence Yield Effect… · pp. 10–11, Figure 5 · p. 11

F. Automated Forum Deployment (EQ5)

  • On Dread, bot-interacted threads had a median of 9 user messages versus 6 for passive-baseline threads; on BHW, the medians were 18 versus 7.The comparison used relevant threads from the same forum sections and creation window, but topics could differ, so the results show association rather than isolating a causal effect. Message counts are a proxy for activity and may not capture CTI value.Source: F. Automated Forum Deployment (EQ5) · pp. 11–12, Figure 6 · p. 12
  • BHW yielded 1.09 qualifying replies per question and 3.39 per thread, versus 0.44 and 0.74 on Dread; 66.7% of BHW threads and 28.1% of Dread threads received replies.Among responding threads, the averages were 5.09 replies on BHW and 2.63 on Dread. User replies averaged 221.5 characters versus 130.0 for system messages on BHW, while on Dread they averaged 141.1 versus 152.4.Source: F. Automated Forum Deployment (EQ5) · p. 12
  • Median time to first reply was 2.8 h on BHW versus 7.2 h on Dread, and median conversation duration was 48.5 h versus 82.8 h.Deployments spanned approximately three weeks on BHW and one month on Dread; conversation duration ran from the bot’s first message to the thread’s last message that received no further replies.Source: Figure 8 · p. 12, F. Automated Forum Deployment (EQ5) · p. 12
  • Neutral tones dominated replies on both platforms, with negative sentiment limited and never escalating into adversarial interactions.Anger and Fear appeared in higher proportions on Dread, particularly in discussions involving exploits, fraud, and operational security; the analysis used Hartmann’s English emotion classification model.Source: F. Automated Forum Deployment (EQ5) · p. 12
  • Across 104 live conversations, DarkBot elicited newly tested attack techniques, details about an active database breach, and additional information through private channels.Examples included a user reporting tests of the recently published EDR-Freeze technique and a threat actor confirming continued access to a compromised database. Some users shared further information or contact and service links in private messages, where DarkBot did not operate.Source: F. Automated Forum Deployment (EQ5) · pp. 11, 13

VII. THREATS TO VALIDITY

The authors identify internal and external validity threats that limit confidence in DarkBot’s evaluation and the generalizability of its live-deployment findings.

Internal Threats

  • The intelligence-probing evaluation uses a simulated user and manually validated technique ground truth, while general-purpose jailbreak tests may not capture system-aware attacks.Manual annotation reduces dependence on the extractor but introduces judgment for implicit or ambiguous ATT&CK mappings; penalizing questions about topics absent from the original conversation may also underestimate elicitation capability.Source: §VII · p. 13

External Threats

  • External validity is limited by participant backgrounds, deployment on specific forums, changing community norms and language, and ethical limits on deployment scale and duration.Replication is difficult as forum content, moderation, and user behavior change; public evaluation threads may also change, be removed, or migrate.Source: §VII · p. 13

VIII. CONCLUSION · IX. ETHICAL CONSIDERATIONS

DarkBot uses specialized agents to elicit CTI from underground forums, with simulated and live evaluations reporting CTI recovery and no observed deployment interventions. The paper also outlines future extensions and ethical safeguards for the work.

VIII. CONCLUSION

  • DarkBot recovered 72.8% of MITRE ATT&CK techniques from initial posts in simulated CrimeBB conversations and added an average 3.85 CTI entities over seven days in 20 matched topic pairs.Across 104 live forum threads, it elicited CTI-relevant information unavailable through passive crawling.Source: §VIII · p. 13
  • DarkBot decomposes CTI elicitation into 11 agents for engagement gating, question generation, and linguistic style adaptation, outperforming a single-agent baseline in tactical coverage and stylistic credibility.The system is designed for targeted conversational engagement in public underground forums, unlike passive crawling or single-agent approaches limited to specific tasks.Source: §VIII · p. 13
  • The pipeline rejected all tested jailbreak prompts and filtered most policy-violating content, with the overall architecture compensating for limitations of individual components.The system also effectively detects harmful content and is robust against adversarial inputs.Source: §VIII · p. 13
  • Across 104 live forum threads, the deployment triggered no account bans, moderator interventions, or AI accusations while eliciting CTI-relevant information unavailable through passive crawling.The prospective experiment used 20 matched topic pairs and found an average increase of 3.85 CTI entities in DarkBot-assigned threads over seven days.Source: §VIII · p. 13
  • Future work includes extending the pipeline to multilingual forums and broader platforms, adapting elicitation to community feedback, and longitudinally assessing sustained detection risk.These directions are proposed as future work, not evaluated findings.Source: §VIII · p. 13
  • Further proposed directions include targeting emerging threats such as new CVEs or threat actors, supporting migration to private channels, and automating discovery of hidden underground communities.These are future-work directions.Source: §VIII · pp. 13–14

IX. ETHICAL CONSIDERATIONS

  • The study received institutional ethical review and data protection impact assessment approval and uses safety guardrails, content filters, data minimization, and self-hosted models.The authors identify risks of facilitating criminal behavior, unsafe interactions, and misuse beyond defensive purposes; they state that procedures follow human-subjects research standards and data protection principles, including purpose limitation and proportionality.Source: §IX · p. 14

A. Institutional Evaluation Process … F. Engagement Guidelines

The study received favorable privacy and ethics reviews and used safeguards to limit privacy, interaction, and dual-use risks. Its research was restricted to publicly accessible forums and defensive cybersecurity purposes.

A. Institutional Evaluation Process

  • Privacy officers issued a favorable GDPR-compliance opinion, and the institutional Research Ethics Committee approved the study after reviewing its methods, ethics, and safeguards.The review included data-processing documentation, a DPIA, an ethics request, and a commitment to enforce data non-disclosure agreements.Source: A. Institutional Evaluation Process · p. 14

B. Human Subjects and Consent

  • Because individual consent was impractical and could compromise validity, the study used openly accessible forums without identifying, contacting, or profiling users, and published a general privacy notice.Source: B. Human Subjects and Consent · p. 14

C. Personal Data and Privacy

  • Treating potentially reidentifiable pseudonyms and text as personal data, the team salted-hashed usernames, secured and restricted data access, and deleted raw data after analysis.Only aggregated and anonymized results were preserved.Source: C. Personal Data and Privacy · p. 14

D. Deception and Interaction Risks

  • The agent concealed its automated nature but did not impersonate real people or organizations, and safeguards, automatic termination, and manual monitoring constrained interaction risks.It asked only cybersecurity-related clarifying or exploratory questions, detected sensitive domains and jailbreak attempts, and terminated suspected automation or unexpected interaction patterns; the study posed no reasonably foreseeable psychological or material harm.Source: D. Deception and Interaction Risks · pp. 14–15

E. Dual-Use and Misuse Concerns

  • To limit dual-use risks, the study focused on defensive cybersecurity and restricted DarkBot code access to scientific use with author approval and an intended-use request plus ethical justification.The intended contribution was to improve threat intelligence and understanding of emerging cyber threats for defensive and preventive security.Source: E. Dual-Use and Misuse Concerns · p. 15

F. Engagement Guidelines

  • Engagement guidelines excluded private, paid, invitation-only, or affiliation-gated spaces, prohibited attachment downloads and discussions of sensitive or harmful topics, and required responsible disclosure of imminent harm.Interactions were limited to public threads; restricted topics included drugs, weapons, pornography, violence, and judicial, medical, or health-related personal information.Source: F. Engagement Guidelines · p. 15

GENERATIVE AI USAGE … APPENDIX C STYLISTIC CREDIBILITY SURVEY

The appendices document AI-assisted preparation, artifact and dataset access constraints, implementation choices, and a survey assessing stylistic credibility and tone alignment.

GENERATIVE AI USAGE

  • Claude assisted with writing, experimental code, and visualization drafts, while TrinkAI/WriteFull and Wordtune supported grammar checking and text revision.Authors reviewed AI-assisted content, manually validated code, and checked visualization plots against underlying experimental data.Source: GENERATIVE AI USAGE · p. 17

APPENDIX A OPEN SCIENCE

  • DarkBot’s codebase, prompts, orchestration logic, and evaluation scripts are available upon request, subject to an intended-use description and ethical discussion.The fine-tuned SLANGCONVERTER checkpoint is also to be released under the same access conditions.Source: Appendix A · p. 17
  • AnnoCTR, BeaverTails, CoDA, and jailbreak datasets are public, but CrimeBB and its derived synthetic slang dataset cannot be redistributed.The code describes how to extract experimental subsets for researchers with access to the original datasets; live-experiment thread URLs are provided but may change or disappear.Source: Appendix A · p. 17

APPENDIX B IMPLEMENTATION DETAILS

  • The system used AutoGen for agent communication and Temporal.io for persistent workflows and retries, with offline open-weight models chosen to avoid sharing sensitive data with providers.Uncensored attack-reasoning agents used an abliterated Gemma 3 27B variant, while logical reasoning and classification agents used Qwen3-32B.Source: Appendix B · p. 17
  • A fine-tuned 4B-parameter SLANGCONVERTER turns formally generated questions into slang using paired CrimeBB messages and LLM-generated formal rewrites.The synthetic dataset was generated by SYNTHETICAGENT; SLANGCONVERTER was trained to reproduce dark web users’ terminology and stylistic patterns.Source: Appendix B · p. 17

APPENDIX C STYLISTIC CREDIBILITY SURVEY

  • The survey collected 118 valid responses from uncompensated participants with a Computer Science background and used five representative CrimeBB threads.Participants received consent information, a content warning, task instructions, and a visual example; reply positions were randomized.Source: Appendix C · pp. 17–18
  • The survey compared DarkBot with a single-agent baseline and versions without SLANGCONVERTER or TONEAGENT on credibility and tone alignment.Style and Credibility covered the baseline and SLANGCONVERTER comparisons; Tone Alignment covered the TONEAGENT comparison, with neutral response options in both sections.Source: Appendix C · p. 18

APPENDIX D DATASETS DETAILS · APPENDIX E LLAMAGUARD3 COMPARISON

The appendices document the datasets used to evaluate DarkBot’s relevance, extraction, safety, and style components, and compare its relevance approach with LLaMA Guard 3. The default LLaMA Guard policy suppresses many hacking posts, while a custom CTI category improves hacking recall but increases false positives.

APPENDIX D DATASETS DETAILS

  • CoDA [54] contains 10,000 documents in ten high-level topic categories; the evaluation subset uses 470 texts, including 200 hacking samples, for relevance and prohibited-topic testing.Because its labels are not specific to CTI relevance and no dark-web forum conversation dataset was available, the authors use CoDA; duplicates and texts exceeding 15,000 tokens were excluded.Source: Appendix D · p. 18
  • CrimeBB contains over 99 million posts from 34 forums; the study selects and manually reviews 100 relevant English threads from its September 2024 snapshot.Threads had 15–150 messages and at least one predefined relevant keyword; the subset supports information-extraction and design-validation experiments and SLANGCONVERTER training data construction.Source: Appendix D · p. 18
  • AnnoCTR [56] provides expert-annotated cyber threat reports, and the evaluation uses its 105 reports with at least one ground-truth MITRE ATT&CK technique annotation.Annotations cover explicit and implicit tactic and technique mentions and serve as ground truth for evaluating the TTP Extractor.Source: Appendix D · p. 18
  • The safety evaluations use 845 deduplicated jailbreak prompts from two public datasets and BeaverTails [55] questions from eight harmful-content categories.The jailbreak data supports robustness and pipeline-safety testing; BeaverTails tests the final message filter against harmful content that bypassed earlier components.Source: Appendix D · p. 18
  • The Slang Conversion Dataset pairs real CrimeBB slang messages with LLM-generated formal rewrites to fine-tune SLANGCONVERTER to convert formal questions into forum-style slang.The generated slang incorporates terminology and stylistic patterns of real dark-web users.Source: Appendix D · p. 18

APPENDIX E LLAMAGUARD3 COMPARISON

  • The comparison evaluates LLaMA Guard 3 8B as a single-model baseline on the CoDA subset against THREADANALYZER combined with RELEVANCEAGENT.The relevance, defensive, and filter agents use a general-purpose reasoning LLM with a structured prompt encoding exclusion rules and positive CTI indicators.Source: Appendix E · p. 19
  • The default LLaMA Guard policy blocks a far larger share of hacking documents than DarkBot, making it unsuitable as a CTI relevance filter.The out-of-the-box policy treats safe posts as relevant and unsafe categories as irrelevant; its S2 category conflates cybercrime with drug trafficking and financial fraud.Source: Appendix E · p. 19
  • The custom S14 Cyber Threat Intelligence category improves recall on hacking posts but produces a much higher false-positive rate across policy-violating content.Posts count as relevant only when classified unsafe under S14, defined with positive CTI indicators and no exclusion rules.Source: Appendix E · p. 19

APPENDIX F STYLE EMBEDDINGS ANALYSIS · APPENDIX G CTI ENTITIES SUBSET · APPENDIX H WEEKLY TIMING PATTERNS

The appendices assess DarkBot’s stylistic similarity to forum users, define and annotate CTI entities for a matched-pair experiment, and present weekday message distributions.

APPENDIX F STYLE EMBEDDINGS ANALYSIS

  • StyleDistance [67] yielded average cosine similarities of 0.73 on Dread and 0.68 on BHW, versus 0.23/0.02 for Style-Embedding.The comparison used style-focused sentence transformers on messages from live interactions.Source: Appendix F · p. 19
  • UMAP projections showed substantial overlap between DarkBot and user-message clusters under both style representations, with no cluster containing only system messages.The authors say this suggests DarkBot’s style is indistinguishable from regular users’ style.Source: Appendix F · p. 19, Figure 12 · p. 20

APPENDIX G CTI ENTITIES SUBSET

  • CTI mentions were annotated using STIX 2.1 object types: eight STIX Domain Objects and nine STIX Cyber-observable Objects.The authors used this external threat-intelligence specification to define the unit of intelligence for the matched-pair experiment.Source: Appendix G · p. 19
  • Defence (course-of-action) was the dominant entity type and covers controls or detection signals described as catching or stopping an operation; evasive manoeuvres were annotated as Technique.Evasion has no separate STIX type because ATT&CK treats defence evasion as a tactic over techniques.Source: Appendix G · pp. 19–20
  • The annotation covered 174 human-authored messages in 20 matched pairs during the seven-day window: 90 in DarkBot-assigned threads and 84 in controls.Initial posts, bot messages, and moderator or automated messages were excluded so the annotated population matched the activity measure.Source: Appendix G · p. 20
  • A regular-expression extractor pre-annotated six candidates across the cohort’s 310 messages: three URLs and three images.The extractor was used for indicators with fixed formats to help reduce omissions.Source: Appendix G · p. 20

APPENDIX H WEEKLY TIMING PATTERNS

  • Appendix H presents message counts by day of week on Dread and BHW during experiment VI-F.Figure 13 shows the message distribution by weekday; the supplied passages do not report specific daily values.Source: Appendix H · p. 20, Figure 13 · p. 20

APPENDIX I INTERACTION EXAMPLES

The appendix presents anonymized examples of DarkBot interactions in the Dread forum, showing the bot participating as a regular user without disclosing its automated nature.

APPENDIX I INTERACTION EXAMPLES

  • The examples show DarkBot exchanges in Dread, with unrelated or less interesting messages omitted.The interactions are anonymized.Source: Appendix I · p. 20
  • Each example places the thread’s initial post above the exchange, highlights CTI-relevant information extracted by DarkBot, and censors harmful language.The bot intervenes as a regular participant without disclosing its automated nature.Source: Appendix I · p. 20
  • Captions identify Figures 14, 15, 16, 18, and 19 as DarkBot conversations on the Dread forum, while Figure 17 is identified as a DarkBot interaction under experiment §VI-E.Source: Figure 14 · p. 21, Figure 15 · p. 21, Figure 16 · p. 21

Related papers

Loading 2608.23185v1…