PaperlayerView exact revision on arXiv

Source-linked AI summary

CAPTCHAs in the Agentic Era: Solvers That Learn from Every Encounter

Oguzhan Salman, Kemal Bicakci

arXiv:2609.02393v1cs.CR

TL;DR

This paper addresses the gap between costly VLM-based CAPTCHA solving and fast detectors limited to familiar categories. It combines a confidence-routed YOLOv8 detector with an open-weight VLM, using VLM answers as training labels to expand and repair the detector. The hybrid reaches 85.4% overall and 84.2% macro accuracy across 16 classes, while recovering throughout a year-long simulated arms race.

  • Problem

    VLMs can reason across visual CAPTCHAs but incur high inference costs, whereas specialized detectors are fast yet limited to trained categories and do not improve with exposure.

  • Method

    The system cascades a fine-tuned YOLOv8 detector with an open-weight VLM and distills VLM-generated answers into YOLO training data without browser automation or DOM access.

  • Results

    85.4% overall and 84.2% macro accuracy across 16 classes, with VLM fallback enabling category expansion and detector recovery from adversarial attacks.

  • Takeaways & Limitations

    The solver improves with use: it absorbs unseen classes after one or two encounters and recovers each round in a year-long grey-box arms race, even with a roughly 70%-accurate teacher.

Abstract

from arXiv · show

Vision-language models (VLMs) can solve visual CAPTCHAs without task-specific training, but the agents built on them approach every challenge from scratch. For such an agent, the hundredth instance of a familiar puzzle costs as much time and compute as the first. Specialized detectors invert the trade-off, answering in milliseconds but only for categories they were trained on. Neither improves with exposure. We study what changes when a solver improves with use. Our system pairs a fine-tuned YOLOv8 detector with an open-weight VLM behind a confidence-based router, and runs entirely from screenshots and operating-system input events, with no browser automation or DOM access. It reaches 85.4% overall and 84.2% macro accuracy across 16 classes, exceeding either component alone. Every answer VLM produces also serves as a training label, so the detector absorbs categories it was never trained for, typically after one or two encounters and without human annotation. The same loop also repairs it. A CAPTCHA operator can perturb images against the publicly released detector and drive its accuracy to 0%, but the perturbations leave VLM untouched, and its labels let the detector recover. Under a year-long simulated arms race in which the CAPTCHA operator re-crafts its perturbations each month, the solver recovers every round, and a cheap ~70%-accurate open-weight teacher hardens it as effectively as a perfect oracle. Visual CAPTCHA defenses that assume a failing bot stays failing therefore understate how quickly an adaptive solver returns.

I. INTRODUCTION

Existing CAPTCHA solvers trade breadth, speed, and stealth: specialized detectors are fast but narrow and often DOM-dependent, while VLM agents are broad but costly. The paper introduces a DOM-free hybrid that routes familiar challenges to YOLO and novel or uncertain cases to a VLM, then learns from those encounters.

  • Specialized CAPTCHA detectors are fast on trained classes but commonly depend on browser automation and DOM access, exposing detectable artifacts.
  • VLM-based agents can reason across diverse visual tasks, but their inference overhead prevents real-time CAPTCHA solving.
  • 70% of challenges are resolved by YOLO at 4 ms per cell, while VLM fallback at 225 ms reduces average latency 3 times versus VLM-only inference.
  • 85.4% overall and 84.2% macro accuracy are achieved across 16 classes, exceeding either component alone.
  • VLM-generated answers become training data, enabling new object classes after one or two encounters and recovery after PGD reduces YOLO accuracy from 83% to 0%.
  • Across a year-long adaptive arms race, the solver re-hardens every round, while a roughly 70%-accurate teacher performs as effectively as a perfect oracle.

III. METHOD

The method combines screenshot-based UI localization, deterministic process control, and complementary YOLO and VLM solving paths. A confidence-aware cascade handles familiar cells with specialized models and delegates uncertain or novel cases to Qwen-7B-VL.

  • The pipeline captures screenshots and sends OS-level mouse and keyboard events, avoiding DOM access and browser automation.
  • YOLO performs single-pass discriminative detection, while Qwen-7B-VL generates constrained class indices or bounding-box coordinates through open-ended visual reasoning.
  • Three separately trained YOLO models handle UI element detection, tile classification, and instance segmentation.
  • A deterministic finite-state machine coordinates initialization, puzzle analysis, solving, verification, recovery, and iterative re-analysis of dynamic grids.
  • EasyOCR extracts the target instruction, after which cropped cells are routed to YOLO classification or segmentation models or the Qwen-7B-VL agent.
  • For segmentation, predictions over concatenated 4×4 cells are mapped back using a 1% cell-area overlap threshold.

A. Dataset Composition

The evaluation uses held-out CAPTCHA data spanning 16 object classes and compares YOLO, VLM, and hybrid solvers under shared classification and segmentation metrics. The hybrid is evaluated against both single-model baselines on identical images.

  • Dataset Composition: 15,172 held-out samples cover 16 object classes, including Boat, Taxi, and Tractor, which are unsupported by the published YOLO classifier.The evaluation combines public reCAPTCHA data with experimentally collected samples and avoids images seen during training.
  • Evaluation Metrics: Recall is prioritized for 3×3 classification because missing a required target tile increases CAPTCHA risk.For 4×4 segmentation, F1-score balances under-selection and over-selection during boundary localization.
  • Evaluation Metrics: Macro-averaged accuracy is treated as security-critical because defenders can bias puzzle distributions toward classes where solvers fail.Overall accuracy reflects the natural class distribution, whereas macro metrics weight classes equally.
  • Evaluation Metrics: The comparison requires one top-1 label per tile because VLM outputs do not provide reliable class probabilities comparable to YOLO softmax scores.Top-k and probability-threshold metrics are therefore discarded to preserve a shared evaluation procedure.
  • Classification Experiments: 85.4% overall and 84.2% macro accuracy make the hybrid stronger than YOLO-only and VLM-only baselines on the classification test set.With τ = 0.70, VLM is invoked for 30% of images; the hybrid improves over YOLO-only by +10.0 pp overall and +24.2 pp macro, and over VLM-only by +4.9 pp overall and +3.2 pp macro.
  • Classification Experiments: 85.0% overall and 81.6% macro accuracy remain highest for the hybrid on the 12-class subset supported by YOLO.The confidence-based fallback corrects low-confidence YOLO errors even when unsupported classes are excluded.

B. Segmentation

The hybrid architecture extends to 4×4 segmentation by combining YOLO’s geometric precision with VLM’s coverage of unsupported classes. Its performance advantage is constrained by the VLM’s lower segmentation precision and much higher generation latency.

  • Segmentation: 88.6% accuracy (F1=0.850) is achieved by the YOLOv8x-mask + VLM hybrid on 4×4 segmentation puzzles.The system combines YOLO’s geometric precision on supported classes with VLM’s zero-shot coverage of unsupported ones.
  • Segmentation: The 7B VLM is not an adequate segmentation teacher: its F1=0.769 trails YOLOv8m’s F1=0.811, so distillation would degrade YOLO.Teacher distillation is therefore restricted rather than applied to segmentation predictions.
  • Latency: 2251 ms segmentation latency reflects autoregressive generation of long coordinate sequences, versus 225 ms for classification’s single class-index token.YOLO maintains approximately 26 ms for segmentation because its single-pass architecture is optimized for real-time inference.
  • Recognition Strategies: YOLO’s learned silhouettes and VLM’s semantic reasoning produce complementary class behavior, including VLM’s 93.2% motorcycle recall versus YOLO’s 68.9%.YOLO performs better on pattern-based Crosswalks, with 90.5% recall, while VLM records 41.9%.
  • Recognition Strategies: VLM records 21.6% recall on Other because object-centric bias and semantic over-interpretation lead it to hallucinate specific labels for generic scenes.YOLO instead learns a decision boundary for images containing none of the target concepts.

VI. TEACHER-STUDENT DISTILLATION

VLM predictions are used as training labels to extend YOLO to unsupported CAPTCHA classes without human annotation, while Experience Replay limits forgetting of existing classes. New-class recall varies with visual diversity, and difficult classes can impose substantial retention costs.

  • Teacher-student distillation: VLM outputs provide labels for fine-tuning YOLO on unsupported classes, closing a deployment-time reasoning-to-recognition loop without human annotation or a pre-collected dataset.The approach converts expensive VLM predictions into lightweight detector responses.
  • Experience Replay: Experience Replay combines exemplars from existing classes with new-class data to reduce catastrophic forgetting during class extension.The experiments maintain balanced representation by sampling N exemplars per existing class.
  • New-class recall: 96% mean recall is reached for Taxi after 10 training samples, while Tractor reaches 88% at N=15 and Boat plateaus near 48% with all six available images.Taxi typically requires about two puzzles, whereas Boat remains difficult because its available examples cover diverse visual concepts.
  • New-class recall: Intra-class visual diversity relative to sample size is identified as the primary predictor of few-shot class-extension difficulty.Taxi images are visually consistent, Tractor images share coherent structure, and Boat images span substantially different concepts.
  • Catastrophic forgetting: Taxi forgetting drops from 6.8 pp at N=5 to 3.3 pp at N=23, while Tractor drops from 7.2 pp to 5.3 pp at N=15.Boat causes 17.2 percentage points of forgetting at N=5 and remains at 15.0 pp at its maximum N=6.

VII. ADVERSARIAL ROBUSTNESS AND AUTONOMOUS RECOVERY

The study tests whether adversarial perturbations can defeat the detector and whether VLM-guided retraining restores performance. Although PGD drives YOLO accuracy to zero, VLM remains largely effective and supplies labels for recovery across fixed and recurring attacks.

  • Adversarial attack: PGD reduces YOLO validation accuracy from 83.1% to 0.0%, while the same perturbations reduce VLM accuracy by only 3.7 percentage points.The perturbations are optimized against YOLO gradients, so their effect on VLM is unintended transfer.
  • Autonomous recovery: 66.8% adversarial accuracy is recovered with ground-truth labels using approximately 1,800 samples, with most gains occurring within the first approximately 2,000 samples.This corresponds to roughly 200 puzzle encounters in the reported setup.
  • Autonomous recovery: VLM-labeled retraining trails ground-truth retraining by 6–10 pp in adversarial accuracy and lowers clean accuracy to approximately 72–73%.At approximately 11,000 samples, VLM-labeled and ground-truth models reach 65.0% and 71.0% adversarial accuracy, respectively.
  • Iterative grey-box escalation: Across a 12-round, year-long arms race, every solver variant recovers after each newly crafted attack.The experiment varies teacher quality and whether the defender can reproduce the solver’s training process.
  • Iterative grey-box escalation: Pre-recovery accuracy is 28.4% for a coupled perfect-teacher solver, versus 52.9% with independent batch order and 58.2% for the VLM-taught solver.The reported comparison attributes improved resistance to decorrelated training trajectories, including noisy VLM labels.
  • Cascade under attack: The confidence cascade routes attacked cells to VLM when YOLO confidence falls below the gate, but label smoothing is needed to preserve escalation under attack.Without smoothing, only 4% of attacked cells escalate because the softmax saturates and the τ = 0.70 gate rarely fires.

VIII. LIVE RECAPTCHA EVALUATION

The hybrid solver transfers its screenshot-based cascade to live reCAPTCHA v2, passing every session while recovering puzzle-level failures through retries. Its deployment behavior and conclusion reinforce a complementary YOLO–VLM design that supports autonomous recovery and continued hardening.

  • Live performance: 100 of 100 live sessions passed, covering 206 puzzles across static and dynamic challenges.181 puzzles were solved on the first attempt, while 25 failures were recovered through reload-and-retry within the same session.
  • Live performance: 87.9% of puzzles were solved on the first attempt, with all remaining failures recovered during the same session.
  • Cascade transfer: 37.1% was the aggregate VLM call rate, within 7 percentage points of the 30% static-benchmark rate.Per-session routing ranged from 0% to 75%, reflecting different challenge difficulty.
  • Deployment costs: Dynamic puzzles impose deployment-only iterative re-analysis costs because selected tiles re-render after each click.The median time was 17.1 seconds per puzzle and 35.2 seconds per session.
  • Conclusion: The DOM-free hybrid combines millisecond-scale YOLO inference with VLM fallback and distills VLM outputs into YOLO over time.The paper frames this collaboration as a teacher–student mechanism for adding classes and recovering from attacks.
  • Adaptive robustness: A year-long grey-box arms race showed recovery every round, with a roughly 70%-accurate teacher hardening the solver at least as well as a perfect oracle.Noisy labels were reported to reduce transferability from the defender’s surrogate.

APPENDIX A OPEN SCIENCE

The open-science appendix documents the released application, models, datasets, notebooks, evaluation artifacts, and deployment requirements. It emphasizes reproducibility while noting that some datasets are generated or only partially included because of storage constraints.

  • Artifacts: The released CAPTCHA Solver Application includes the finite-state-machine controller, hybrid YOLO/VLM backend, and pretrained weights.
  • Datasets: The UI detector dataset is reproducible from generation scripts using 220 CAPTCHA source images to create 1,200 annotated training samples.Generated data and background screenshots are omitted because of storage constraints.
  • Datasets: The evaluation release includes samples for Taxi, Boat, and Tractor, while remaining classes can be obtained from the public dataset.Raw prediction JSON files from the original experiments are included for verification.
  • Reproduction: Evaluation scripts cover classification metrics, cascade routing, segmentation evaluation, and per-class metric computation.
  • Reproduction: Distillation and adversarial-robustness notebooks provide reproducible Experience Replay and end-to-end attack experiments.
  • Artifacts: Researchers receive scripts and notebooks for PGD attacks, adversarial VLM inference, sample-efficiency analysis, and grey-box escalation.
  • Requirements: The minimum environment requires 16GB RAM and 20GB disk, while hybrid VLM deployment recommends 32GB RAM, 16GB-plus GPU VRAM, and 100GB disk.
  • Requirements: The system supports Windows, macOS, and Linux and uses a DOM-independent screenshot-based browser approach.Optimal operation is reported at 1920×1080 resolution, with browser zoom adjustments potentially needed elsewhere.

APPENDIX B CAPTCHA SOLVER IMPLEMENTATION

The implementation uses a human-inspired dual-process architecture: YOLO handles familiar patterns quickly, while VLM reasoning handles novel or difficult cases. A finite-state machine coordinates screenshot monitoring, puzzle solving, verification, and recovery across supported backends.

  • Architecture: YOLO provides fast reflexes for frequent routine patterns, while the VLM intervenes on novel or semantically complex challenges.
  • Architecture: The finite-state machine coordinates detection, solving, verification, and recovery loops.
  • Monitoring: The application continuously monitors the screen at two-second intervals and invokes the state machine when CAPTCHA elements appear.
  • Backends: The processor supports YOLO-only, VLM-classification, and confidence-based hybrid backends.
  • Models: The model stack includes a YOLOv8 UI detector for five element classes and a YOLOv8x segmentation model for 4×4 puzzles.
  • Usage: The command-line interface selects yolo, llm, or hybrid backends and can disable mouse animations for faster execution.Mouse movements did not influence solving success rates in the reported experiments.
  • Workflow: The workflow detects the checkbox, analyzes the puzzle, routes to a backend, clicks target cells, verifies completion, and reloads failures.It uses OCR for the target object and distinguishes classification from segmentation puzzles.

APPENDIX C YOLOV8 UI DETECTOR TRAINING

The UI detector is trained on synthetically assembled webpage screenshots and evaluated with standard classification and detection metrics. Training converges near epoch 148 with strong validation scores and approximately 1.1 ms GPU inference per image.

  • Dataset construction: The system separates a 1,200-sample UI-detection dataset from a 11,908-sample classification and segmentation evaluation dataset.The evaluation data spans 16 object classes.
  • Dataset construction: The UI detector is trained for five classes: CAPTCHA area, cell, reload button, submit button, and robot checkbox.The dataset contains 880 CAPTCHA screenshots, 200 checkbox samples, and 120 negative samples, split 80%/20% for training and validation.
  • Synthetic generation: 220 CAPTCHA screenshots are overlaid on four backgrounds at varied positions and scales to generate annotated training images.The procedure detects 3×3 or 4×4 grids, creates YOLO annotations, and supports manual verification.
  • Dataset construction: 120 negative samples use empty annotations to reduce false positives on non-CAPTCHA webpages.
  • Training: The training configuration uses AdamW, cosine learning-rate annealing, batch size 16, 512×512 images, 150 epochs, and augmentation.
  • Results: Precision reached 0.999, recall 1.000, mAP@0.5 0.995, and mAP@0.5:0.95 0.990 at final validation, with approximately 1.1 ms GPU inference.Training converged at epoch 148.
  • Evaluation: The reported metric workflow collects top-1 predictions and computes confusion matrices and per-class metrics from JSON outputs.The classification notebook filters to an 893-image test set for these calculations.
  • Evaluation: The appendix records F1-score as 2 × (Precision × Recall) / (Precision + Recall).

B. Segmentation Metrics Generation

Segmentation evaluation aggregates cell-level decisions across puzzles and target classes, comparing YOLO variants, VLM, and a hybrid system. The hybrid achieves the strongest overall result by combining YOLO’s geometric precision with VLM’s coverage of rare and unsupported categories.

  • Evaluation procedure: 3,264 cell decisions from 204 puzzles were evaluated using confusion matrices mapped from predicted regions to 4×4 grid cells.Predictions used bounding boxes or masks, with a 1% overlap threshold for selecting cells.
  • Evaluation procedure: Per-class metrics aggregate TP, FP, FN, and TN across puzzles targeting each object class to compute class-level F1-scores.The same aggregation procedure supports the per-class results reported in Table X.
  • Overall results: 88.6% accuracy and F1=0.850 make YOLOv8x-mask + VLM the strongest segmentation configuration.The hybrid outperforms YOLOv8x-mask, which reaches 86.7% accuracy and F1=0.814.
  • Overall results: 80.1% recall and 73.9% precision show VLM favors coverage over precision relative to YOLO masks.YOLOv8x-mask is described as the strongest single model, while VLM attains higher recall but lower precision.
  • Per-class results: F1=0.000 on Crosswalk, Stairs, and Taxi occurs for all YOLO models, whereas VLM covers all 8 classes without CAPTCHA-specific training.VLM reaches F1=0.857 on Taxi, 0.752 on Stairs, and 0.685 on Crosswalk.
  • Aggregate results: 87.2% precision, 76.3% recall and F1=0.814 give YOLOv8x-Mask higher weighted aggregate performance than VLM’s 73.9% precision, 80.1% recall and F1=0.769.The weighted aggregation reflects puzzle frequency, while macro-averaged metrics favor VLM’s unsupported-class coverage.

APPENDIX E VLM BACKEND INFRASTRUCTURE

The VLM backend exposes fine-tuned Qwen-7B-VL through a REST API, allowing the CAPTCHA solver to send screenshot-derived crops or concatenated cells for inference. The documented infrastructure supports local and remotely tunneled deployment while returning model outputs and diagnostic dimensions and timing.

  • Backend architecture: The backend serves a fine-tuned Qwen-7B-VL GUI agent through a FastAPI REST API.The service is designed specifically to enable VLM inference in the hybrid CAPTCHA solver.
  • Model loading: The model is loaded with HuggingFace Transformers, downloaded once from HuggingFace Hub at approximately 16GB and cached locally for later runs.Subsequent executions load the cached model rather than downloading it again.
  • Image processing: The /generate endpoint accepts an image and OpenAI-compatible chat messages, resizes and tokenizes them, runs inference, and returns decoded text.The endpoint records original dimensions before resizing to model input requirements.
  • Response format: API responses include raw model output, inference duration, and original and resized image dimensions.These fields support both result handling and runtime diagnostics.
  • Deployment: Local deployment runs on http://0.0.0.0:8000, while ngrok can expose a public HTTPS tunnel for remote inference.The tunneled setup separates GPU-intensive VLM execution from the main solver process.
  • Solver integration: For classification, the solver sends cropped grid cells to the VLM; for segmentation, it sends concatenated cells to retrieve bounding boxes.The backend code, deployment instructions, and API examples are included in the artifact repository.

APPENDIX H VLM PROMPT TEMPLATES

The appendix specifies constrained VLM prompts and the end-to-end processing pipelines for classification and segmentation CAPTCHAs. Collected unsupported-class samples illustrate the contrast between YOLO’s fixed class support and VLM’s broader recognition coverage.

  • APPENDIX H VLM PROMPT TEMPLATES: Classification prompts require the VLM to return only one numeric class index from the paper’s 16-class label list.The prompt asks for the class seen with highest confidence and forbids explanatory text.
  • APPENDIX H VLM PROMPT TEMPLATES: Segmentation prompts ask for all instances of the target object and complete bounding-box coordinates for each instance.The requirements emphasize including every visible part and enclosing the entire object.
  • APPENDIX I PIPELINE DIAGRAMS: Classification pipelines extract 9 cells from a detected 3×3 CAPTCHA, classify each independently with YOLOv8 or VLM, and convert labels into binary selections.Both models use top-1 prediction and commit to exactly one class per cell.
  • APPENDIX I PIPELINE DIAGRAMS: Segmentation pipelines concatenate 16 cells from a 4×4 CAPTCHA, predict masks or boxes, and map regions back to cells using a ≥1% overlap rule.YOLO can produce pixel-accurate masks or boxes, whereas VLM produces boxes only.
  • APPENDIX J COLLECTED CLASS SAMPLES: VLM recall on unsupported collected classes is 82.1% for Taxi, 100% for Tractor, and 100% for Boat, while YOLO predictions are always incorrect.The samples are annotated with green for correct predictions and red for misclassifications.
  • APPENDIX J COLLECTED CLASS SAMPLES: VLM correctly identifies 23 of 28 Taxi samples, while YOLO misclassifies all 28, predominantly as Car.The five VLM errors are three Car predictions and two Motorcycle predictions.
  • APPENDIX J COLLECTED CLASS SAMPLES: YOLO misclassifies all 20 Tractor samples and all 11 Boat samples, whereas VLM achieves 100% recall on both sets.These figures are shown in the collected-class sample figures.
Loading 2609.02393v1…