PaperlayerView exact revision on arXiv

Source-linked AI summary

DarKnight: Retrieval-Guided Agentic Framework for Breaking Dark Web CAPTCHA Gates

Binghui Wu, Yeo Zi Yi, Darren Tan, Manit Baser, Levente Csikor, Dinil Mon Divakaran

arXiv:2610.19830v1cs.CRcs.CV

TL;DR

Dark-web CTI CAPTCHA solving must work across diverse marketplace challenges without per-site training or dependence on commercial APIs whose availability and behavior investigators do not control. DARKNIGHT uses a locally hosted open-weight VLM, retrieves past successes and failures, and adapts preprocessing from site verdicts. On D1, an equipped 27B model reaches 83.2%, compared with 44% for Claude Opus 5.

  • Problem

    Dark-web CAPTCHA solvers either depend on per-style labeled training or commercial APIs, while CTI deployments need local control over solver access, behavior, and recurring costs.

    Source: §I · p. 2

  • Method

    DARKNIGHT uses a small open-weight VLM to retrieve visually similar successful and failed attempts, compose preprocessing, and update from each site's pass/fail verdict without retraining.

    Source: §I · pp. 2–3

  • Results

    83.2% on D1 for a DARKNIGHT-equipped general-purpose 27B model, versus 44% for Claude Opus 5.

    Source: §I · p. 3

  • Takeaways & Limitations

    An efficient open-weight backbone paired with DARKNIGHT can provide a favorable accuracy–latency trade-off for local CAPTCHA solving without third-party VLM APIs.

    Source: H. Discussion · p. 15

Abstract

from arXiv · show

Cyber Threat Intelligence (CTI) investigators who monitor dark web marketplaces for leaked credentials and breach listings must pass the diverse CAPTCHAs that operators deploy to block automated collection. Existing solvers either need unscalable per-site fine-tuning or rely on commercial Vision-Language Model (VLM) APIs, which surrender operational sovereignty: provider policies, Tor-exit blocking, government regulation, or model updates outside the investigator's control can halt or alter the solver overnight, and per-call costs recur with every session and retry. We present DarKnight, a self-improving agentic framework designed to run on a single workstation GPU inside an investigator's Tor crawler and solve CAPTCHAs with small, locally hosted open-weight VLMs that no provider decision can withdraw, without per-market training or third-party services. For each CAPTCHA, DarKnight retrieves successful and failed past attempts on similar images (dual-signal RAG), composes an image-specific preprocessing pipeline, and learns from the site's pass/fail verdict. In a case study on four real-world dark web marketplaces, agentic tool selection outperformed parameter scaling. A deployable open-weight VLM with DarKnight reached 83.2\% on the most heavily defended market, 37.2 percentage points above its base model, outperforming even the frontier Claude Opus~5 (44%). DarKnight recovered 48.2% of baseline failures at a 7.6% degeneration cost, indicating that selective tool use can avoid much of the degradation caused by uniform preprocessing.

I. INTRODUCTION

CAPTCHAs are a chokepoint for scalable dark-web marketplace monitoring, while existing solvers do not fit the varied styles and operational constraints CTI investigators face. The section introduces DARKNIGHT, which uses a small open-weight VLM, retrieved solve experience, and site verdicts to adapt preprocessing without per-site retraining.

I. INTRODUCTION

  • CAPTCHAs are the exception among marketplace defenses: overcoming this gate cheaply and at scale makes automated dark-web measurement feasible, while failure stalls collection behind manual intervention.The section also describes provider policies, regulation, Tor blocking, model changes, and recurring per-call costs as reasons commercial APIs are not a practical CTI option.Source: §I · pp. 1–2
  • The section identifies gaps in evaluations of small open-weight VLMs under dark-web operational constraints, deployable enhancement methods, and non-Latin heterogeneous CAPTCHA styles.Prior dark-web work targets English-language, Western markets and Latin-script inputs, while clear-web VLM evaluations predominantly use frontier-closed or large models.Source: §I · p. 2
  • DARKNIGHT retrieves successful and failed attempts on visually similar images, composes preprocessing, and learns from site verdicts using the same open-weight VLM for planning and solving.The closed verdict-feedback loop supports stream-level improvement without retraining, and the framework combines visual analysis, dual-signal RAG, five preprocessing operations, and automated ground-truth feedback.Source: §I · pp. 2–3
  • On D1, DARKNIGHT-equipped 27B reaches 83.2%, above raw Step3-VL-10B (63.6%), Qwen3-VL-30B-A3B (42.4%), and Claude Opus 5 (44%).The comparison is reported for a general-purpose 27B model equipped with DARKNIGHT against the cited raw-model baselines.Source: §I · p. 3
  • DARKNIGHT recovers 48.2% of baseline failures while degrading 7.6% of correct predictions, a 5.8× recovery-to-degeneration ratio.The Planner selects no preprocessing in 86.9% of D2 cases but only 26.6% on heavily defended D1.Source: §I · p. 3
  • Across 2,000 CAPTCHAs from four marketplaces and nine backbones, DARKNIGHT improves accuracy in 30 of 32 configurations, with a 13.33% average absolute improvement.The reported average improvement across all backbones on D1 is +30.48%; one configuration handles alphanumeric, math, and Chinese CAPTCHAs in the same mixed-style market without per-site retraining.Source: §I · p. 3

II. RELATED WORK

The related work spans bespoke dark-web CAPTCHA defenses, style-specific trained solvers, and generalized VLM agents, but does not address the paper's deployment setting with an adaptive open-weight solver. The threat model and design rationale therefore emphasize local, resource-bounded operation and modular, experience-driven solving.

A. Dark Web Analytics and Anti-Crawling Mechanisms

  • Dark-web marketplaces rely on bespoke, locally hosted, server-rendered CAPTCHAs, with noise, hyper-kerning, variable lengths, and math puzzles aimed at conventional scrapers.Tor constraints and avoidance of third-party JavaScript underpin this contrast with clear-web behavioral validation systems.Source: A. Dark Web Analytics and… · p. 3

B. Classical ML-Based CAPTCHA Solvers

  • Per-style CNN and GAN solvers can score highly on trained styles, but require labeled data and retraining and do not generalize across churning, heterogeneous markets.The paper targets operation without per-style supervision; it does not compare these trained solvers on accuracy.Source: B. Classical ML-Based CAPTCHA Solvers · p. 3

C. Generalized VLM Solvers and Agentic Frameworks

  • Prior generalized VLM and agentic CAPTCHA solvers are evaluated on clear-web challenges or rely on commercial cloud APIs, which the authors exclude under dark-web CTI deployment constraints.Prior dark-web use of a VLM is described as one component of a single-market navigation tool, not a study of CAPTCHA-solving behavior across scales, preprocessing, or agent designs.Source: C. Generalized VLM Solvers and Agentic… · p. 3

III. THREAT MODEL

  • The threat model casts the investigator's Tor crawler as the attacker and the marketplace CAPTCHA system as the defender, with no initial labeled pairs but feedback-derived experience available.The study targets text-based Latin alphanumeric, arithmetic, and non-Latin challenges; it excludes sliders, image-selection grids, audio, and challenges requiring JavaScript beyond text rendering.Source: §III · pp. 3–4

IV. DESIGN RATIONALE

  • The design rationale separates perception, planning, and execution, assigning visual analysis to VLMs and deterministic transformations and arithmetic to specialized tools.This decomposition is intended to improve component controllability and targeted error diagnosis while avoiding unnecessary stochastic error in deterministic tasks.Source: §IV · p. 4
  • The framework retains both successful and failed attempts and retrieves visually similar records to reuse effective preprocessing and avoid strategies that degraded performance.This principle motivates the dual-signal RAG knowledge base.Source: §IV · p. 4
  • New preprocessing tools can be added without changing model weights, but empirical validation of adding tools is left to future work.The modular toolbox is intended to support adaptation without continual retraining.Source: §IV · pp. 4–5

V. THE DARKNIGHT FRAMEWORK

DARKNIGHT runs a plan–execute–record loop: it tailors preprocessing using visual analysis and retrieved experience, submits an OCR-derived answer, and stores the site's verdict. Its Planner analyzes image properties, retrieves visually and semantically similar records, and plans a processing pipeline.

A. System Overview

  • The system plans a CAPTCHA-specific pipeline, executes preprocessing and OCR before submission, then records the site's binary verdict with the attempt in RAG.The record includes the image, visual analysis, pipeline, OCR output, and outcome, whether the answer succeeds or fails.Source: A. System Overview · p. 5, Figure 2 · p. 5
  • Accumulated site-specific experience supports better tool selection without model retraining; the same loaded VLM handles visual analysis, planning, and OCR without additional model memory.The agentic loop adds local inference time, while routing planning through the already loaded model keeps memory requirements the same as a single-VLM pipeline.Source: A. System Overview · p. 5

B. Planner Agent

  • The Planner's first stage converts the raw CAPTCHA into structured JSON describing eight visual properties, which informs retrieval and provides explicit context for planning.The properties are presented as the output of Stage 1 visual analysis.Source: B. Planner Agent · p. 5, Table I · p. 5
  • Experience retrieval combines text similarity of the image description with perceptual-hash similarity of the image, then supplies top-k records annotated with pipelines and pass/fail outcomes to planning.The automated retrieval uses no additional VLM inference; the resulting plan specifies an ordered preprocessing pipeline and whether to activate the math expression solver after OCR.Source: B. Planner Agent · p. 5

C. Dual-signal RAG Knowledge Base

The dual-signal RAG store retains successful and failed CAPTCHA-solving attempts so the Planner can use past outcomes when choosing preprocessing. Its similarity, storage, and eviction rules balance relevant experience with a diverse, bounded memory.

C. Dual-signal RAG Knowledge Base

  • Successful records provide positive pipeline examples, while failed records warn the Planner against repeating destructive tool sequences; the store starts empty and accumulates records sequentially.The first image is solved without retrieved experience; subsequent images can use the history accumulated so far.Source: C. Dual-signal RAG Knowledge Base · p. 6

1) Storage Layout:

  • Each configuration has a separate store with up to 100 successful and 100 failed records; cross-dataset retrieval is excluded from primary per-dataset experiments.A mixed-site setting separately evaluates one shared store per backbone across marketplaces.Source: 1) Storage Layout · p. 6

2) Quality-Based Eviction:

  • When a slot is full, least-frequently-used eviction removes records with the fewest retrievals, allowing frequently matched experience to persist and the memory to specialize.Records that are never retrieved are replaced by more relevant newcomers.Source: 2) Quality-Based Eviction · p. 6

3) Deduplication:

  • An incoming record is suppressed when an existing record scores at least 0.99 in weighted similarity, preventing near-duplicates from crowding out diverse examples.The stated purpose is to preserve the breadth of memory.Source: 3) Deduplication · p. 6

4) Similarity Metric:

  • Similarity weights RGB perceptual-hash cosine at 0.7 and description-embedding cosine at 0.3, and records below τ = 0.7 are excluded from retrieval.The image hash captures structural and color-gradient patterns; sentence-transformer embeddings add semantic context from the free-text description.Source: 4) Similarity Metric · p. 6

5) Split Retrieval:

The system retrieves both successful and failed examples for the Planner, then executes image-specific preprocessing under ordering constraints. The same VLM performs OCR, with validation and fallback handling for math CAPTCHAs.

5) Split Retrieval:

  • Retrieval returns up to five records, drawing from successful and failed slots to ensure balanced positive and negative examples for the Planner.The method retrieves up to two from each slot, with one additional record from whichever slot has the higher top match.Source: 5) Split Retrieval · p. 6
  • Failed records act as negative signals: the Planner avoids a tool repeatedly associated with failures, while successful records are adopted, verified, or ignored according to similarity.The retrieved records include similarity, description, applied tools, outcome, and—on failed records—raw OCR output.Source: 5) Split Retrieval · p. 6

D. Executor and Toolbox

  • The Executor applies selected preprocessing tools, then invokes the same VLM with a CAPTCHA-type-specific OCR prompt; math output is passed to a rule-based solver.The toolbox provides five operations aimed at visual degradations encountered in CAPTCHA images.Source: D. Executor and Toolbox · p. 7
  • Color extraction must precede sharpening or binarization, denoising follows color extraction, and binarization is reserved for clean backgrounds.The Planner may choose no tools or several, but tool sequences must respect these ordering constraints.Source: D. Executor and Toolbox · p. 7
  • For a clean-background CAPTCHA with colored lines and residual dots, color extraction → denoising → binarization is a valid pipeline; misapplied operations can increase visual obfuscation.The same tool can help one CAPTCHA archetype and harm another, motivating selection informed by retrieved successes and failures.Source: D. Executor and Toolbox · p. 7
  • If math-expression validation fails, the Executor reruns OCR with an alphanumeric prompt; if an alphanumeric result contains = or ?, it also attempts math solving.Math output must include an arithmetic operator and equals sign or question mark, and contain no alphabetic characters.Source: D. Executor and Toolbox · p. 8

E. Ground-truth Feedback Loop and Self-Improvement

DarKnight records each attempt, whether it passes or fails, and uses those outcomes to guide later solves. The loop begins without site-specific history and can also share experience across marketplaces with visually similar CAPTCHAs.

E. Ground-truth Feedback Loop and Self-Improvement

  • Every solve attempt is written to the RAG store with its image hash, description, pipeline, parameters, OCR output, outcome, and timestamp.Both passed and failed attempts are recorded.Source: E. Ground-truth Feedback Loop and… · p. 8
  • Successful pipelines become high-confidence references for similar images, while failed pipelines are surfaced to discourage repeating strategies that degraded similar CAPTCHAs.The store starts empty for a new marketplace; the authors observe retrieval becoming effective typically within 5–10 images as site-specific history accumulates.Source: E. Ground-truth Feedback Loop and… · p. 8
  • Because retrieval uses visual similarity rather than site identity, experience can be shared across marketplaces with similar CAPTCHA characteristics.The paper evaluates this capability in a mixed-site experiment.Source: E. Ground-truth Feedback Loop and… · p. 8

VI. EXPERIMENT SETUP

The evaluation covers nine model backbones on four real-marketplace CAPTCHA datasets, using a single GPU under a capped-memory setup. The datasets contain 2,000 images, with the most diverse marketplace serving as the primary test of tool selection across heterogeneous inputs.

A. Models and Hardware Configurations

  • Nine models span an OCR specialist, general-purpose instruct models, and general-purpose thinking models.The set includes a 1B OCR specialist, Qwen3-VL instruct models at 4B, 8B, and 30B-A3B, plus thinking-model families.Source: A. Models and Hardware Configurations · p. 8
  • All models were evaluated unmodified on one RTX Pro 6000 GPU, with GPU memory capped at 29 GB to simulate a resource-constrained production environment.No third-party quantization, fine-tuning, or other modifications were applied.Source: A. Models and Hardware Configurations · p. 8

B. Datasets

  • The four marketplace datasets contain 500 CAPTCHA images each, totaling 2,000 instances collected from login pages.The marketplaces are ChangAn (D1), Freecity (D2), Russian market (D3), and Mobilestore (D4).Source: B. Datasets · p. 8
  • D1 is the most diverse dataset, spanning alphanumeric, math, and Chinese-character CAPTCHAs with varied visual defenses, making it the primary benchmark for tool selection on heterogeneous inputs.D4 contains alphanumeric CAPTCHAs with thick characters on a patterned mesh background.Source: B. Datasets · pp. 8–9
  • Ground-truth labels were prepared independently of model predictions, with three annotators resolving disagreements through review and consensus.Inter-annotator reliability was measured using pairwise exact-string agreement and character error rate before reconciliation.Source: B. Datasets · p. 9

C. Baselines and Models

The evaluation compares nine open-weight VLMs in raw-baseline and DARKNIGHT configurations, using website pass/fail exact-match accuracy and local inference time as primary measures. Baseline performance varies across datasets, while agentic assistance can yield larger gains than scaling within a model family.

D. Evaluation Metrics

  • Exact-match accuracy measures the fraction of CAPTCHAs accepted by the website; inference time covers local pipeline computation but excludes Tor round-trip time.Accuracy is case-insensitive for alphabetic characters, while math CAPTCHA answers must match exactly.Source: D. Evaluation Metrics · p. 9

A. CAPTCHA-solving Accuracy

  • On D1, Qwen3-VL-4B with DARKNIGHT reaches 55.6%, a +27.6 pp gain over its baseline, exceeding the 8B model’s 39.6% baseline.Doubling Qwen3-VL parameters from 4B to 8B raises D1 accuracy by 11.6 pp (28.0%→39.6%).Source: A. CAPTCHA-solving Accuracy · p. 10
  • D1 baseline accuracy ranges from 6% with Gemma4-e4b-it to 64% with Step3-VL-10B; D4 is less defended, with several models exceeding 80%.Step3-VL-10B reaches 98.6% on D2, while D3 baselines fall between 27–67%.Source: A. CAPTCHA-solving Accuracy · p. 10
  • DARKNIGHT raises Qwen3-VL-30B-A3B from 42.4% to 80.6% on D1, while Gemma4-E4B-IT with DARKNIGHT reaches 46.8%.The latter slightly exceeds the raw Qwen3-VL-30B-A3B result of 42.4%.Source: A. CAPTCHA-solving Accuracy · p. 10

B. The Degeneration-Recovery Effect

DARKNIGHT selectively preprocesses CAPTCHA images, using retrieval memory and pass/fail outcomes to guide tool choices. Across planner decisions it recovers more baseline failures than it causes degenerations, and retrieval and shared cross-site memory provide further benefits.

B. The Degeneration-Recovery Effect

  • 48.2% of baseline failures are recovered at a 7.6% degeneration cost, yielding 1,390 recoveries against 241 degenerations.The recovery-to-degeneration ratio is 5.8×.Source: B. The Degeneration-Recovery Effect · p. 10
  • Preprocessing risk varies by tool: remove_straight_line has a +75.3 net effect, while denoise yields only +1.5.The former recovers 82.8% of failures while degrading 7.5% of correct predictions; binarize was selected only three times, insufficient for a meaningful estimate.Source: B. The Degeneration-Recovery Effect · pp. 10–11, Table VII · p. 10
  • The Planner selects no preprocessing in 97.0% of D3 cases and intervenes in 73.4% of D1 cases, consistent with image-dependent rather than uniform tool use.Withholding tools is an explicit decision intended to avoid potential degeneration when intervention is unnecessary.Source: B. The Degeneration-Recovery Effect · p. 11, Table VIII · p. 11

C. Effect of Retrieval Memory

  • Retrieval consistently improves accuracy over the no-retrieval pipeline across evaluated configurations, adding benefit beyond preprocessing and prompting.Fig. 7 shows the retrieval-memory ablation across Qwen3-VL-4B and Qwen3-VL-8B on D1–D4.Source: Figure 7 · p. 11, C. Effect of Retrieval Memory · p. 11

D. Cross-Site Adaptation

  • In a 500-image mixed-site stream, shared-memory DARKNIGHT raises Qwen3-VL-8B from 65.0% to 75.0% and Qwen3.6-27B from 65.0% to 78.0%.Accuracy improves for all four evaluated backbones in this setting.Source: Table IX · p. 11, D. Cross-Site Adaptation · p. 11

E. Inference Time and Computational Overhead

DARKNIGHT adds local inference overhead, but that cost differs substantially across backbones. Among evaluated models, the mixture-of-experts Qwen3-VL-30B-A3B-Instruct combines the highest average accuracy with the lowest latency among the cited comparisons.

E. Inference Time and Computational Overhead

VIII. CONCLUSION

  • For evaluated backbones, effective CAPTCHA solving did not depend on parameter scaling alone; the best accuracy–latency trade-off combined an efficient backbone with selective orchestration.The conclusion reports a +36.8% gain for a small 9B VLM on the most heavily defended dataset, with a 5.38 s local loop.Source: E. Inference Time and Computational… · p. 12, §VIII · p. 12

F. Cross-Model Accuracy-Latency Analysis

APPENDIX

The appendix reports annotation reliability, reference-baseline results, memory-size sensitivity, and the scope of released research artifacts. It also specifies that operational solver assets and deployment tooling are withheld.

A. Ethical Considerations

  • The study frames dark web CAPTCHA solving as research for dark web threat-intelligence collection and releases study artifacts without marketplace access information.Released datasets contain no onion addresses, credentials, or other marketplace access information.Source: A. Ethical Considerations · p. 14

B. Data Availability and Responsible Release

  • The authors release code, reproduction materials, and four annotated datasets, while withholding operational solver assets and deployment tooling that could facilitate CAPTCHA bypass at scale.The released data contain no onion addresses or other marketplace access information.Source: B. Data Availability and Responsible… · p. 14

C. Inter-Annotator Agreement

  • D2 has the highest transcription agreement at 99.4% ESA and 0.16% CER, while D1 has the lowest at 90.4% ESA and 2.90% CER.Statistics were computed before disagreement resolution; D1’s character-level disagreement remains low.Source: Table XII · p. 14, C. Inter-Annotator Agreement · p. 14

D. Additional Baseline Comparison

E. Sensitivity to RAG Memory Size

F. API parameters for VLM calls

The appendix documents the two VLM calls and the detailed evaluation results, then discusses local CAPTCHA solving, negative memory, and limitations for live deployment.

F. API parameters for VLM calls

G. Detailed results

  • Table XIII gives exact accuracy percentages and absolute changes for evaluated VLMs across four dark web datasets, providing the numerical basis for Fig. 6.Source: G. Detailed results · p. 15

H. Discussion

  • DARKNIGHT pairs an efficient open-weight VLM with adaptive tool orchestration for a favorable accuracy–latency trade-off without third-party VLM APIs.The discussion also states that model size alone does not determine accuracy or inference cost.Source: H. Discussion · p. 15
  • Historical failures serve as negative constraints in dual-signal RAG, helping the Planner avoid degradation from uniformly applied aggressive preprocessing.The discussion says aggressive preprocessing consistently degrades clean inputs when applied uniformly.Source: H. Discussion · p. 15
  • Live deployment remains outside the offline evaluation’s scope; targeted CAPTCHA perturbations, new mechanisms, Tor delays, and crawler pacing remain unassessed.Formal robustness to targeted adversarial examples remains an open question, and latency measurements cover local computation only.Source: H. Discussion · p. 15
Loading 2610.19830v1…