Source-linked AI summary
CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs
Zhenhao Zhang, Zhaoyu Fan, Haohan Ying, Jingwen Hu, Hancen Fan, Junhao Zhou, Zitian Chen, Linchao Zhu
TL;DR
Interactive CAPTCHAs remain difficult for computer-use agents, while existing datasets trade off type coverage, interaction fidelity, and trajectory supervision. CaptchaArena supplies execution-verified interactive puzzles and trajectories, and CaptchaAgent trains one policy across 20 types. Its Pass@1 rises from 70.5 after supervised fine-tuning to 71.7 after reinforcement learning.
Problem
Existing CAPTCHA datasets trade off type coverage, interaction fidelity, and trajectory supervision, while interactive CAPTCHA failures can block computer-use workflows.
Source: §1 · p. 1Method
CaptchaArena provides execution-verified interactive puzzles, screenshot-action trajectories with reasoning annotations, and pixel masks; CaptchaAgent trains one 9B policy with supervised fine-tuning followed by verifier-rewarded reinforcement learning.
Source: §4 · p. 6, §3.2 · p. 6, §6 · p. 10Results
70.5 Pass@1 after supervised fine-tuning and 71.7 after reinforcement learning, versus 11.4 for the untrained backbone; reinforcement learning also improves scores on two external benchmarks.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The results point to training data and interaction supervision, rather than model scale alone, as a key factor in CAPTCHA performance.
Source: §6 · p. 10Experiments use hosted environments and do not interact with live CAPTCHA services or deployed anti-bot systems.
Source: Ethics statement · p. 10
Abstract
from arXiv · showhide
Interactive CAPTCHAs remain challenging for computer-use agents, while existing datasets face trade-offs among type coverage, interaction fidelity, and trajectory supervision. To address these gaps, we present CaptchaArena, the first large-scale, fine-grained training dataset for interactive CAPTCHA solving. It contains 50K puzzles across 20 CAPTCHA types and 5 interaction modes, with every solution verified through execution. CaptchaArena provides 50K screenshot-action trajectories, including 46K with step-by-step reasoning annotations. It also includes fine-grained pixel-mask annotations for irregular targets. Using CaptchaArena, we train CaptchaAgent, a single 9B policy for all 20 CAPTCHA types, with supervised fine-tuning followed by reinforcement learning. The environment verifier directly provides the RL reward. Supervised fine-tuning reaches 70.5 Pass@1, and reinforcement learning further improves it to 71.7, while also improving performance on two external benchmarks. These results demonstrate the value of large-scale, fine-grained computer-use supervision for training interactive CAPTCHA agents. We release CaptchaArena and CaptchaAgent at https://github.com/X0X0X00/CaptchaArena.
1 Introduction
Interactive CAPTCHAs can interrupt computer-use workflows, yet existing benchmarks and training datasets vary in whether and how they cover CAPTCHA interaction. CaptchaArena supplies verified interactive trajectories across many CAPTCHA types, and CaptchaAgent improves Pass@1 through supervised fine-tuning and reinforcement learning.
1 Introduction
- Existing resources trade off type coverage, interaction fidelity, and trajectory supervision, while benchmark interfaces may omit fine-grained interaction or step-by-step execution.ReCAP and CaptchaMind provide reasoning trajectories across 7 and 8 types, while MirrorCAPTCHA evaluates 18 types but supplies answer and click-coordinate supervision.Source: §1 · pp. 1–2
- Figure 1 samples puzzles from 20 CAPTCHA types; border colors mark interaction modes and corner icons mark cognitive skills.Source: Figure 1 · p. 2
- CaptchaArena contains 50K interactive puzzles across 20 types and 5 modes, with execution-verified solutions, screenshot-action trajectories, 46K reasoning annotations, and pixel masks for irregular targets.The five modes include single-click, multi-click, arrow-cycle, real-time, and text-entry.Source: §1 · pp. 2–3
- 70.5 Pass@1 after supervised fine-tuning and 71.7 after reinforcement learning, up from 11.4 for the untrained backbone; humans reach 94.1 on the same 4,000 puzzles.The policy also improves on two external benchmarks after reinforcement learning.Source: §1 · pp. 2–3
2 Related Work
Prior CAPTCHA resources differ in type coverage, interaction interfaces, and the supervision they provide. The section positions CaptchaArena against those trade-offs without supplying a single uniform comparison protocol.
2 Related Work
- Training resources vary in type coverage and supervision: ReCAP and CaptchaMind cover 7 and 8 types, while MirrorCAPTCHA evaluates 18 types but provides answer and click-coordinate labels.Open CaptchaWorld covers 20 types; MCA-Bench offers over 180K training data for 20 types but uses task-specific LoRA adapters.Source: §2 · p. 3
- CAPTCHA-specific benchmarks differ in interface fidelity: Open CaptchaWorld uses browser-use actions, whereas MCA-Bench predicts structured outputs instead of executing step-by-step in a live browser.Source: §2 · p. 3
- Trajectory supervision ranges from reasoning-action and self-correction traces to synthetic labels, and broad type coverage with reasoning-annotated screenshot-action trajectories remains absent in the cited resources.ReCAP reports improved overall zero-shot transfer but underperformance versus base models on some unseen types.Source: §2 · p. 3
3 CaptchaArena
CaptchaArena is a live, closed-loop CAPTCHA dataset designed to train computer-use agents. It combines controlled puzzle construction, multiple forms of answer annotation, and browser replay that both verifies solutions and records trajectories.
3 CaptchaArena
- CaptchaArena contains 50K puzzles across 20 types, split into 42K training, 4K validation, and 4K test puzzles; each type requires visual recognition and some require grounding, reasoning, or computation.Agents observe a fixed 1280 × 1080 viewport and act through low-level pointer and keyboard inputs.Source: §3 · p. 4
3.1 Puzzle Construction
- The interaction schema provides screenshot, click, text-entry, drag, and timed-hold actions, using the same format for replay, annotation, training, and evaluation.Submission is automatic for four types; otherwise the agent submits the puzzle.Source: §3.1 · p. 4
- Puzzle construction combines procedurally generated geometry tasks and generated or openly licensed imagery; authors inspect generated images, and answer positions and distractors are designed to limit systematic guessing.Dice Count and Dart Count images receive additional manual checking and correct-result labels.Source: §3.1 · pp. 4–5
3.2 Answer Annotation
- Ground truth comes from generation-time labels, manual annotations, and renderer-derived masks; all 7,500 puzzles with irregular targets carry masks for shape-based click evaluation.Misleading Click uses a complementary mask rule, while single-point targets use a stored coordinate and tolerance.Source: §3.2 · p. 6
3.3 Replay Verification
- Each stored answer is compiled into browser actions and replayed; only puzzles accepted by the page verifier are retained, and each released puzzle has one screenshot-action trajectory.A puzzle may admit multiple valid solutions and action orderings, including trajectories that detour and recover.Source: §3.3 · p. 6
4 CaptchaAgent
CaptchaAgent uses one shared policy across all 20 CAPTCHA types, trained first on reasoning-annotated verified trajectories and then with reinforcement learning. Its annotation, supervised fine-tuning, and reinforcement-learning stages use a shared interaction format and the live environment verifier.
4 CaptchaAgent
- CaptchaAgent is a single policy over all 20 types, with shared parameters, tools, and observations rather than task-specific adapters or switching.Source: §4 · p. 6
4.1 Reasoning Annotation
- Replay-verified actions are augmented with per-step reasoning, with 46K trajectories annotated across training and validation puzzles and test puzzles held out from annotation.A separate judge filters annotations, and each turn becomes a training sample in the inference format.Source: §4.1 · p. 7
4.2 Supervised Fine-Tuning
- Supervised fine-tuning uses 12,000 puzzles spanning all 20 types, expanded into 37,621 per-turn samples; 1,500 puzzles per type are reserved for reinforcement learning.The vision encoder and vision-language aligner remain frozen.Source: §4.2 · p. 7
4.3 Reinforcement Learning
- Reinforcement learning continues training against the live environment verifier, which scores rollouts without a learned reward model, preference data, or step-level reward annotations.Episodes are capped at 15 turns; difficulty is mined per puzzle, and reward increases monotonically with submission and closeness to the answer.Source: §4.3 · pp. 7–8
5 Experiments
CaptchaAgent substantially outperforms the evaluated untrained and GUI-agent baselines, with reinforcement learning adding further gains. Its remaining errors reflect non-submission, weak grounding, and unstable execution, while the evaluation uses a common screenshot-and-action protocol across 20 types.
5.1 Evaluation Setup
- Evaluation uses screenshots and pixel-level actions across 4,000 test puzzles, with 200 puzzles per type and server-side verification of submitted states.CaptchaAgent is sampled five times per puzzle, other systems once, and episodes end at submission or a 15-step cap; overall scores are unweighted means across types.Source: §5.1 · p. 8
5.2 Results
- After SFT, the 9B backbone’s 11.4 Pass@1 average masks a procedural split: 56.6 on four auto-submit types but at most 0.5 on 16 requiring explicit submission.The untrained backbone submits on 20.5% of rollouts, while CaptchaAgent submits on 96.8%.Source: §5.2 · p. 8
- RL improves average Pass@1 by +1.2 over the supervised policy (+1.5 excluding Click Order), with 16 of 20 types improving and gains on two external benchmarks.Pass@1 improves more than Pass@5 (+1.2 versus +0.8); Click Order is the only substantial regression, decreasing by 3.1 points.Source: §5.2 · pp. 8–9
- 70.5 average Pass@1 and 86.0 Pass@5 place CaptchaAgent above the strongest closed-source model at 69.2 and open-weight GUI agents at up to 35.2.Eight types reach at least 90 Pass@1, and four match or exceed the 94.1 human average; the remaining 23.6-point human gap is concentrated in four types.Source: §5.2 · p. 8, Table 3 · p. 9
5.3 Error Analysis
- Non-submission, weak grounding, and unstable execution characterize distinct remaining errors; the four lowest post-SFT types all use all-or-nothing grading.The supervised policy submits on only 54.4% of Dice Count episodes, while humans reach 83.0 and the best evaluated system reaches 22.5 on that task.Source: §5.3 · p. 9
- Place Dot isolates a CaptchaAgent grounding weakness: it submits on 95.3% of rollouts but scores 16.1 Pass@1, while GPT-5.4 reaches 88.5 on the same puzzles.Patch Select is difficult across systems: no model exceeds 12.5, compared with 73.5 for humans.Source: §5.3 · pp. 9–10
- The Pass@5-minus-Pass@1 gap reaches 42.2 to 91.0 in Slide Puzzle, 56.8 to 92.5 in Bingo, and 59.8 to 93.0 in Pick Area, indicating first-rollout instability.The gap stays under 10 points in the eight types already above 90 Pass@1.Source: §5.3 · p. 10
6 Conclusion
CaptchaArena supplies executable, replay-verified CAPTCHA tasks with trajectories and fine-grained supervision, and CaptchaAgent reaches 71.7 Pass@1 after reinforcement learning. The paper also identifies limits in its annotations, encoder, action representation, and self-hosted evaluation setting.
6 Conclusion
- CaptchaArena contains 50K executable browser puzzles across 20 types, with 50K screenshot-action trajectories, 46K judge-filtered reasoning annotations, and pixel-mask supervision for irregular targets.Solutions are admitted only after replay is accepted by the site’s verifier.Source: §6 · p. 10
- 71.7 Pass@1 after reinforcement learning improves on 70.5 after supervised fine-tuning and 11.4 for the untrained backbone.CaptchaAgent is a single 9B policy across all 20 types, and the environment verifier supplies its RL reward.Source: §6 · p. 10
- The authors identify commercial-model dependence for reasoning annotations, a frozen vision encoder, and discrete actions that cannot reproduce behavioral detectors’ mouse-trajectory signals.The largest gap to the human reference appears on a task requiring precise grounding.Source: §6 · p. 10
- Experiments use self-hosted environments and locally served external suites; none interacts with a live CAPTCHA service or deployed anti-bot system.The paper frames the work as studying CAPTCHA solving as a computer-use capability, not bypassing protections on services the authors do not own.Source: Ethics statement · p. 10
- Released puzzles include executable browser-action solutions that third parties can replay against the environment verifier without access to the generation pipeline.The authors do not claim that rerunning stochastic generation reconstructs the dataset exactly.Source: Reproducibility statement · pp. 10–11
A Action Space and Coordinate Frames
- Agents use screenshot, click, text-entry, drag, and hold tools; episodes end at submission or the 15-turn limit, with some tasks submitting automatically.Coordinate frames are stored per record: some targets use natural-image coordinates mapped to viewport clicks, while fixed HTML controls use viewport coordinates only.Source: A Action Space and Coordinate Frames · p. 15
B Task Catalogue
The catalogue groups CAPTCHA tasks by page layout and specifies their interaction patterns, required skills, reference trajectories, and grading predicates. It covers candidate cycling, localization, grid selection, and structured singletons, alongside construction, verification, and evaluation details.
B.1 Family A: Arrow-Cycle Single Choice
- Family A contains seven arrow-cycle tasks that match candidates to references through semantic recognition, counting, coordinate or path transfer, and rotation.Other catalogue examples include matching icon connections and summing dartboard regions.Source: §B.1 · pp. 15–16
B.2 Family B: Single-Point Localization
- Family B uses single-image clicks for four localization tasks, graded by target or forbidden-region masks or, for Place Dot, distance to a target point.Pixel masks handle irregular target regions that bounding boxes cannot represent accurately.Source: §B.2 · p. 16
B.3 Family C: Grid Multi-Select
- Family C tasks select one or more grid tiles and require exact set equality, so any extra or missing selection fails.The tasks include class recognition, object-occupied patches, single-tile selection, and unusual-animal detection.Source: §B.3 · p. 17
B.4 Family D: Structured Singletons
- Family D combines distinct interaction patterns, including swaps, ordered clicks, slider dragging, timed holds, and text entry.Four tasks end on submit; Hold Button instead ends when the page timer completes.Source: §B.4 · p. 17
C Generation, Compositing, and Quality Control
- Construction uses multiple asset sources, manual inspection and regeneration, answer labels from generation or annotation, fixed layouts, and balanced answer positions where applicable.Two authors independently inspect sampled rendered puzzles; invalid or duplicate samples are discarded and regenerated.Source: C Generation, Compositing, and Quality… · p. 18
D Certificate Schema and Acceptance Regions
- Puzzle certificates store executable action sequences, while localization grading uses masks for three tasks and a distance rule for Place Dot.Geometry Click’s tolerance is encoded by dilating its stored mask; Pick Area uses an undilated target mask and Misleading Click accepts clicks outside its forbidden-region mask.Source: D Certificate Schema and Acceptance… · pp. 18–19
E Full Pass@k Breakdown
- Pass@1–Pass@5 gaps distinguish policies that find answers within five attempts from those that reliably discriminate them, while other systems have only Pass@1 results.The full curves cover the SFT and RL policies on the same test split using five rollouts per puzzle.Source: E Full Pass@k Breakdown · p. 19
F Submit Rates
- The untrained 9B model submits on only 0.5% of Place Dot episodes, versus 95.3% for the supervised policy on the same puzzles.Across non-automatic tasks, untrained models reach a graded submission in only 0–3% of episodes.Source: F Submit Rates · p. 19
G Human Reference
The human comparison shows that the policy’s largest deficits concentrate in a few task types, while some tasks remain ambiguous even for annotators.
G Human Reference
- The policy trails humans by an average of 23.6 points; Place Dot, Dice Count, Patch Select, and Slide Puzzle account for 14.1 points of that gap.RL reduces the average gap to 22.4 points but leaves the same four largest deficits; annotators solve Place Dot perfectly and Slide Puzzle in 199 of 200 cases.Source: G Human Reference · p. 20
- Image Recognition has the lowest human accuracy at 65.5, suggesting ambiguity about which cells contain the target.By contrast, humans solve Place Dot perfectly and Slide Puzzle in 199 of 200 cases.Source: G Human Reference · p. 20
H Source-Asset Reuse Across Splits
The paper audits source-image overlap between training and test splits and cautions that reuse does not itself mean puzzles are duplicates.
H Source-Asset Reuse Across Splits
- Twelve of 20 types share no source images across training and test, while six types have reported source-image overlap.Overlap ranges from 32 of 232 Rotation Match test images to 276 of 282 Image Matching test images.Source: H Source-Asset Reuse Across Splits · p. 20
- The audit reports no overlap figure for Image Recognition, whose cell subdirectories were outside the scan, and omits Hold Button because only its button image is reused.Hold Button’s task is determined by the required hold duration.Source: H Source-Asset Reuse Across Splits · pp. 20–21
- Reused images do not imply duplicate puzzles because targets, distractors, layouts, and answers are sampled per puzzle.The overlap measurements are provided so readers can account for source reuse when interpreting generalization results.Source: H Source-Asset Reuse Across Splits · p. 21
I Generalization Analysis
The paper evaluates transfer on two independently built CAPTCHA benchmarks and examines performance variation across their task types. RL improves aggregate scores on both external benchmarks, while transfer and some system results have important scope boundaries.
I Generalization Analysis
- Published Open CaptchaWorld scores are not directly comparable because this evaluation replaces browser-use element actions with pixel-coordinate actions.Both external benchmarks use the paper’s screenshot-based interface, 15-turn cap, and viewport.Source: I Generalization Analysis · p. 21
- RL improves Pass@1 by +3.8 on Open CaptchaWorld and +6.4 on Halligan, compared with +1.2 on the paper’s test split.Submit rates also increase and mean executed actions decrease on both external benchmarks.Source: I Generalization Analysis · p. 22
- On Open CaptchaWorld, RL improves on 9 types, declines on 5, and leaves 6 unchanged; on Halligan, it improves on 14 of 26, declines on 8, and leaves 4 unchanged.Halligan’s absolute transfer remains limited at 20.0, and none of its 26 types appears in the training corpus.Source: I Generalization Analysis · p. 22
- UI-TARS’s low results on some external tasks reflect interface constraints: it reaches the Halligan puzzle in only 11 of 1,200 episodes and lacks timed-hold support for Open CaptchaWorld.It scores 0.0 on the 12 corresponding Halligan types and on Open CaptchaWorld Hold Button.Source: I Generalization Analysis · p. 22
J.1 Rectangle Grading Inflates Click Accuracy
Rectangle grading can substantially inflate measured click accuracy on irregular targets, while comparisons across interaction types reveal distinct procedural failures. The section also reports the supervised fine-tuning configuration used for CaptchaAgent.
J.1 Rectangle Grading Inflates Click Accuracy
- Mask regrading reproduces the recorded server verdict on 99.5–100.0% of rollouts, with residual differences attributed to one-pixel boundary rounding.The setup holds clicks fixed and changes only the grading rule; it covers Geometry Click and Pick Area, not Misleading Click.Source: §J.1 · p. 23
- A Geometry Click target’s rectangle covers 1.1% of the image, while a Pick Area rectangle covers 65.1%, admitting substantially more background.The comparison measures the same clicks under mask and bounding-rectangle grading.Source: §J.1 · p. 23, Table 12 · p. 25
J.2 Where CaptchaAgent Succeeds and UI-TARS Fails
- CaptchaAgent scores 100.0, 43.9, 94.5, and 56.3 Pass@1 on Hold Button, Slide Puzzle, Rotation Match, and Bingo, versus UI-TARS’s 0.0, 2.5, 3.5, and 3.5.UI-TARS lacks a timed hold action for Hold Button.Source: §J.2 · p. 25
J.1 Rectangle Grading Inflates Click Accuracy
- Rectangle grading lifts UI-TARS Pick Area accuracy from 33.0 under the verifier to 77.5, versus 61.6 for a uniform-in-rectangle click under mask grading.Across policies, rectangle grading raises Geometry Click accuracy by 5.0–8.0 points and Pick Area accuracy by 21.7–21.8 points for trained policies; the increase is 36.0 points for the untrained backbone and 44.5 points for UI-TARS.Source: §J.1 · pp. 23–24, Table 13 · p. 26
J.2 Where CaptchaAgent Succeeds and UI-TARS Fails
- The examples expose procedural and interface limitations: UI-TARS cannot hold, stops short on a slide puzzle, misses the target orientation, and submits an invalid Bingo swap.CaptchaAgent completes the corresponding interactions; the comparison is in-distribution for CaptchaAgent and zero-shot for UI-TARS.Source: §J.2 · p. 26
K Supervised Fine-Tuning Configuration
- LoRA adapters update only the language model while the vision encoder and aligner stay frozen, and the reported checkpoint follows the full schedule rather than best-epoch selection.Per-turn expansion turns 12,000 training puzzles into 37,621 samples, averaging 3.1 supervised turns per puzzle.Source: K Supervised Fine-Tuning Configuration · p. 28
L Reinforcement Learning Details
The reinforcement-learning details describe puzzle-level selection, reward shaping, and on-policy optimization and evaluation. The training pool retains puzzles with variable outcomes across rollouts, and the reward is designed to discourage shortcuts and prolonged unsuccessful trajectories.
L Reinforcement Learning Details
- Puzzles solved in some but not all of five supervised-policy rollouts are retained for GRPO, because always-solved or always-failed puzzles offer little or no within-group reward variance.Mining at the puzzle level retains trainable examples from all 20 types rather than pruning types by mean success rate.Source: L Reinforcement Learning Details · p. 28
- The reward gives 1 for a correct submission and uses partial credit and turn-dependent terms to discourage premature submission and prolonged unsuccessful probing.Submission bonuses require sufficient progress, while flooring reward at zero prevents episode-length differences among zero-progress rollouts from creating artificial within-group variance.Source: L Reinforcement Learning Details · pp. 28–29
- On-policy updates use batches of 128 trajectories from eight puzzles with 16 rollouts each, and each training step takes approximately 6–10 hours.Checkpoints are monitored on validation data, shortlisted checkpoints are evaluated on all 200 validation puzzles per type, and the selected checkpoint is tested once.Source: L Reinforcement Learning Details · pp. 29–30