Source-linked AI summary
The Race between Agentic AI Capabilities and Data Quality Control in Online Surveys
Sourav Panda, Hillmer Chona, Rupak Kumar Das, Shreyash Kale, Shikha Soneji, Jonathan Dodge
TL;DR
Online surveys depend on attention checks, but agentic AI may complete them autonomously and exploit structural cues exposed by web interfaces. This paper evaluates a single-agent system in a controlled survey sandbox and tests DOM metadata obfuscation as a defense. The results show that metadata can enable attention-check resolution, while obfuscation substantially reduces image-answer accuracy without disrupting completion.
Problem
The paper asks whether agentic AI can autonomously complete web surveys and pass attention checks that are intended to protect survey data quality.
Method
The study evaluates a single-agent, multimodal-capable architecture with tool-based web interaction on a controlled survey sandbox, comparing exposed and obfuscated DOM metadata.
Results
DOM metadata obfuscation substantially reduces image-page accuracy while agents continue navigating and completing the survey workflow.
Takeaways & Limitations
Survey researchers should limit DOM-exposed information and maintain separate answer keys, while recognizing that improving VLMs may weaken current defenses.
Takeaways & Limitations
The evaluation uses a controlled survey sandbox, and the authors report that autonomous failures arise from both planning instability and answer-generation errors.
Abstract
from arXiv · showhide
Online surveys are a foundational data collection instrument in a variety of fields, with attention checks serving as critical guardians of response quality. However, the rapid emergence of agentic AI (goal directed systems powered by a large language model (LLM) brain and/or a multimodal processing unit with tool-augmented capabilities) raises new questions about the robustness of these safeguards. We investigate how well agentic AI architectures can complete web-based surveys and pass standard attention checks. We evaluate a single-agent architecture capable of multimodal input processing and tool-based web interaction on a controlled survey sandbox. We analyze the problem from two perspectives. From an attack perspective, we demonstrate how structural vulnerabilities such as exposed DOM metadata and predictable option encoding allow agents to resolve attention checks through structured parsing only. From a defense perspective, we implement a mitigation strategy of DOM metadata obfuscation to remove semantic cues in text-based questions. We evaluate multiple open-source language and multimodal models to study capability and orchestration effectiveness. Based on our evaluations, we offer perspectives on how to simultaneously meet the needs of empiricists and agentic AI researchers.
1. Introduction
Online surveys rely on attention checks to protect response quality, but agentic AI may complete surveys autonomously and exploit structural cues rather than genuine reasoning. The paper frames this as an attack-and-defense problem spanning agent capability and survey data quality.
- Attention checks are intended to filter inattentive, automated, or strategically manipulated online survey responses.
- Modern LLMs can achieve near-perfect attention-check pass rates, challenging the distinction between human and machine-generated submissions.
- Agentic systems independently perceive, plan within, and act on web interfaces, enabling autonomous completion of multi-step survey workflows.
- The attack question asks how well agentic AI solves a broader class of attention checks, while the defense question asks how to prevent that success.
- The study evaluates agentic AI on web surveys, demonstrates metadata-driven attention-check resolution, evaluates a lightweight defense, and analyzes model-specific failures.
2. Survey Sandbox
The survey sandbox is a controlled, realistic multi-page workflow combining heterogeneous interaction primitives, visual tasks, attention checks, and an explicit submission gate. Ground-truth answers are stored outside the application to prevent trivial static extraction.
- The sandbox implements a fixed multi-page participant flow in a controlled Next.js web survey environment.
- The text page contains 10 items—9 substantive questions and 1 attention check—using numeric, dropdown, slider, Likert-scale, and text interactions.
- The image page contains binary image selections, CAPTCHA transcription, and an image attention check with randomized left/right option ordering.
- The thank-you page gates submission with a final AI-acknowledgment attention check and a mandatory Submit Responses action.
- Ground-truth answers are kept outside the survey codebase and client-accessible components, preventing correctness labels from being obtained through static inspection.
3. Agent Architecture
The agent uses a single-agent observe–plan–act architecture that converts browser state into valid actions and answer constraints, generates structured plans and responses, and replans after execution failures. Its loop combines model reasoning, web tools, persistent state, and interface validation.
- Agent Architecture Components: The single-agent architecture combines a model-based brain, prompt policy, web-tool interface, and persistent state memory.
- Agent Architecture Components: Text-only brains plan from structured page metadata, whereas multimodal brains additionally incorporate visual evidence such as screenshots.
- Agent Workflow: The control loop repeatedly observes the browser, plans action intents, executes browser operations, and updates internal state.
- Agent Workflow: The interface adapter converts page context C_t into a valid action space A_t and answer constraints Γ_t, restricting planning to feasible, semantically valid operations.
- Agent Workflow: At each step, the agent emits ranked candidate actions and constrained answer proposals, after which execution produces the next page context.
- Agent Workflow: The workflow terminates at the /done page and can use step, repeated-failure, and no-progress safeguards to prevent infinite loops.
4. Experiment 1: Exploitability Analysis
Experiment 1 shows that agentic AI can reliably complete multi-page surveys and pass attention checks, with exposed DOM structure making interaction and answer inference easier.
- Results: All models passed embedded attention checks on both text and image pages in 100% of runs under both behavioral modes.The evaluation also measured end-to-end completion, page-level accuracy, and failure decomposition.
- Results: End-to-end success remained strong across models and policies, indicating reliable traversal of the multi-page survey to terminal submission.Successful submissions were measured over 100 independent runs per model and mode.
- Results: Completion-mode agents nearly answered all nine text-page questions, while unconstrained agents sometimes advanced without answering every visible item.The behavioral policy therefore influenced page-level completeness.
- Results: Image-page accuracy stayed consistently high across both policies, showing that visual attention checks were not inherently robust in this configuration.The minimal constrained–unconstrained gap indicates policy changes did not materially alter image accuracy.
- Failure Analysis: Residual failures were split between planning and answer errors, indicating remaining problems arose in reasoning or answer generation rather than interaction-pipeline instability.Answer errors occurred despite successful page navigation.
- Exploitability Mechanisms: Exposed DOM metadata encoded input types, constraints, options, and image descriptors, converting intended interaction and visual reasoning into structured parsing and language-based inference.Normalized action primitives then made selecting inferred answers straightforward.
5. Experiment 2: Metadata Obfuscation
Experiment 2 tests DOM metadata obfuscation as a defense. Obfuscation preserves survey navigation and completion but substantially reduces image-answer and attention-check accuracy by removing semantic cues.
- Defense Setup: Survey v1 replaces image labels, alt-text, filenames, and answer descriptions with generic or non-descriptive tokens while preserving the workflow and interaction primitives.The same agent pipeline, models, policies, and hardware were used as in survey v0.
- Workflow Execution: Per-page completion remains near 100% across models and policies under metadata obfuscation.End-to-end completion rates also remain high, with agents reaching terminal submission states.
- Answer Accuracy: Image accuracy drops substantially across all models and modes in survey v1 compared with survey v0.Agents continue executing interface actions but fail to align instructions with the correct image when descriptive cues are removed.
- Attention Checks: Attention accuracy also drops substantially under obfuscation, with selection becoming closer to chance-level at 50% in all questions.The result supports reliance on semantic DOM cues for satisfying visual attention checks.
- Failure Analysis: Survey v1 errors shift primarily toward answer-generation failures rather than planning errors.The intervention disrupts semantic interpretation while leaving planning intact.
- Interpretation: Obfuscating machine-readable cues degrades objectively scored tasks while leaving navigation and interaction planning unaffected.The paper identifies semantic metadata as a key driver of autonomous answer accuracy.
6. Experiment 3: Multimodal Evaluation
The multimodal evaluation adds rendered screenshots to the unchanged DOM-based agent pipeline, testing whether visual grounding improves survey completion under metadata obfuscation. Results vary substantially across VLMs: planning errors dominate failures, while Qwen retains stronger image-task performance than text-only agents under obfuscation.
- Evaluation setup: The multimodal pipeline supplies a rendered screenshot alongside the structured DOM at every interaction step.This lets multimodal models ground decisions in visual content rather than relying only on machine-readable metadata.
- Overall performance: Only one evaluated VLM achieves moderate completion rates, while the other models fail to produce successful end-to-end runs.The tested open-weight models were llava-1.6-7b, minicpm-v-2.6, and qwen2.5-vl-7b.
- Failure analysis: Planning errors, rather than answer-generation errors, dominate unsuccessful VLM runs across both survey versions.This differs from text-only LLM experiments, where planning remained highly reliable across models.
- Image-page performance: Per-page completion remains near 100% within successful runs, but image-level metrics are reported only for qwen2.5-vl-7b because other VLMs had no successful end-to-end runs.The reporting restriction prevents per-page metric computation for the other models.
- Image-page performance: On survey v0, Qwen achieves near-perfect image attention performance and moderate image-answer accuracy, while survey v1 preserves high attention performance with reduced image-answer accuracy.The survey v1 degradation is substantially less severe than for text-only LLM agents.
7. Conclusion
The conclusion frames survey protection and agent behavior as a continuing tension. It recommends reducing DOM-based answer leakage while recognizing that increasingly capable multimodal agents may challenge current defenses.
- For Agentic AI Researchers: Agent failures arise from both planning instability and answer-generation errors, motivating systematic analysis of perception, state encoding, and action selection.The conclusion identifies multiple interacting failure sources rather than a single failure mode.
- For Survey Researchers: Survey designers should maintain separate answer keys and limit information exposed through the DOM to reduce data leakage.The recommendation responds to language-only models solving vision-based tasks when answers are leaked through DOM metadata.
- For Survey Researchers: As VLMs improve, current defenses may become less reliable, complicating the distinction between human respondents and capable agents.The paper presents this as an evolving arms race between survey designers and increasingly capable agents.
A. Code Availability
The paper makes its agent and survey files available on GitHub, accompanied by a detailed README with setup instructions.
- Code availability: All agent and survey files are available on GitHub.
- Code availability: The repository includes a detailed README with setup instructions.
B. Related Work
Related work positions attention checks as safeguards against inattentive or automated submissions and describes agentic AI systems that can increasingly imitate survey respondents. Prior findings also show that agents exploit webpage structure and metadata to answer diverse survey tasks.
- Attention checks: Attention checks are designed to verify active reading and reduce inattentive or careless responses in online data collection.They also serve as a first-line safeguard against automated or non-human submissions.
- Attention checks: Prior studies report that AI agents can pass embedded attention checks, challenging safeguards designed around inattentive humans or simple automation.The emerging risk is qualitatively different from merely inattentive responding.
- Agentic AI: Agentic AI systems combine language models with tool use to perceive, reason about, and act within web environments, including navigating sites and filling forms.
- Agentic AI: A reported autonomous synthetic respondent passes around 99.8% of standard attention checks and can systematically skew online survey estimates.
- Agent interaction: Agents often use HTML labels, ARIA tags, and other webpage metadata rather than visual cues to answer closed-ended, open-ended, and multi-question survey tasks.Prototype systems have also handled some types of CAPTCHAs.
C. Full-page Screenshots
Figure 7 presents the survey sandbox through full-page screenshots.
- The figure shows full-page screenshots.
- The screenshots depict the survey sandbox.
- Figure 7 is titled “Survey Sandbox.”