Building the Web for Agents: A Declarative Framework for Agent-Web Interaction

1 Introduction

• Human-oriented pages make agents rediscover affordances and leave unintended actions or unauthorized access to sensitive data possible.Agents scrape HTML, heuristically parse the DOM, or analyze screenshots; even minor state changes can disrupt workflows.Source: §1 · p. 1
• VOIX replaces brittle inference from human-oriented interfaces with an explicit, auditable contract in which developers declare agent actions and relevant state.The framework is presented as a web-native, declarative substrate for robust, privacy-preserving agent–web interaction.Source: §1 · pp. 1–2
• VOIX shifts control from external agent providers to website developers while formalizing auditable interaction contracts among developers, inference providers, and users.The proposed governance model distributes responsibility rather than centralizing control with inference providers or site operators.Source: §1 · p. 2
• A three-day hackathon with 16 developers provided initial evidence that VOIX can be rapidly adopted to build diverse agent-enabled web applications.The study was intended to validate practicality and accessibility, including learnability and expressiveness.Source: §1 · p. 2
• VOIX operationalizes the Agentic Web vision as a concrete implementation, which the authors describe as empirically validated.The cited position papers [7] [21] describe the paradigm as a vision not yet realized in practice.Source: §1 · p. 2

2.1 Web Agents and Environments

• Agentic Web proposals call for standardized machine-native affordances, while web-agent environments and studies document current limits and gains from API access.WebArena finds strong models underperform humans on long-horizon tasks; hybrid browsing plus APIs improves over browsing alone, with outcomes moderated by API quality [15].Source: §2.1 · p. 2

2.2 Multimodal Interaction

• Multimodal systems combine speech or text with pointing, selection, or visual cues, while maintaining parity across modalities adds implementation complexity.Prior work reports reduced ambiguity and stronger user preference for spatial or object-specific tasks, and notes that modality alignment and discoverability burden developers [38].Source: §2.2 · p. 2

2.3 Human-in-the-Loop Agentic Systems

• Human-in-the-loop agent systems add oversight features, but approval requirements vary by application domain, supporting a central role for website maintainers.Magentic-UI includes co-planning, action approval, and verification [9]; banking and news applications may require different approval processes.Source: §2.3 · p. 3

2.4 Positioning VOIX

• VOIX is guided by requirements for privacy and safety, efficient task-scoped representation, and client-side hosting that avoids inference costs for website owners.The principles place data exposure and safety-critical tool definitions with developers, while users retain control over conversational data and model choice.Source: §2.4 · p. 3
• VOIX is also intended to use familiar, standardized web patterns and express simultaneous or sequential, abstract, complementary, and redundant multimodal interactions.The expressive requirements build on multimodal interaction research.Source: §2.4 · p. 3

3 VOIX: A Web-Native Interface for Agents

• VOIX adds <tool> declarations for typed actions and <context> declarations for task-relevant state, creating a standardized interface for agents.Websites declare and implement the contract; browser agents discover declarations and dispatch calls, while inference providers choose tools and parameters from user objectives.Source: §3 · p. 3

3.1 Trust Boundaries

• VOIX keeps user commands between the browser agent and chosen inference provider, while exposing only website-selected context and actions to the agent.Users can disable specific contexts, and the design contrasts with site-operated LLMs or provider agents inferring state from raw HTML and screenshots.Source: §3.1 · p. 4

3.2 Reference Implementation

• The reference Chrome extension discovers declared tools and contexts, monitors DOM changes, and offers chat and voice interaction through a side panel.Users can configure inference and transcription providers, and tool results can be returned to the agent to inform its response.Source: §3.2 · p. 5

3.3 Developing VOIX-supported Websites

• Developers expose application state with plain-text <context> and define actions with <tool>, linking calls to existing JavaScript logic.A tool may return structured feedback asynchronously; the agent waits for a return event before its next response.Source: §3.3 · p. 5

4.1 Study Setup

• Six teams built six content-creation applications during the three-day, in-person hackathon; participants had varied web-development experience and received VOIX documentation and code examples.The experience distribution included 18% advanced, 41% solid, 29% basics, and 12% no experience.Source: §4 · p. 5, §4.1 · p. 5

4.2 Data Collection

• The mixed-methods evaluation combined usability and trust surveys with observations, team interviews, and analysis of all six projects’ source code.The surveys included SUS and TOAST; each team took part in a scheduled one-hour interview.Source: §4.2 · p. 5, Survey · p. 6, Observational Mentorship · p. 6

4.3 Expressiveness of the Applications

• The six applications supported abstract requests, complementary voice-and-canvas input, redundant voice and graphical actions, and context-sensitive tool availability.Examples included generating a workout plan from a high-level request, creating ambient audio from “make it sound like a rainforest,” and exposing editing tools after an element was selected.Source: §4.3 · pp. 6–7
• Developers scoped tools and context within React or Vue components, including dynamically exposing tools according to application state.Examples included showing image generation only within a flashcard category and exposing editing tools after an element was selected.Source: §4.3 · p. 7

4.4 Learnability of VOIX

• 72.34 (SD=14.82) was the mean SUS score, above the industry average of 68; TOAST means were 5.81 (SD = 0.85) for System Understanding and 5.14 (SD = 0.87) for System Performance.The TOAST scores use a 1–7 scale.Source: §4.4 · p. 7
• Although participants quickly mastered VOIX’s mechanics, deciding which tools would provide meaningful benefits required more reflection.Teams sometimes defaulted to simple CRUD-style tools.Source: §4.4 · p. 7

5 Discussion

• VOIX separates conversational data from the website while exposing only explicitly declared context, protecting sensitive or proprietary data from indiscriminate scraping.The Browser Agent sends prompts to the user’s chosen LLM provider, so the website operator does not see the conversation.Source: §5 · p. 7
• Tool and context primitives provide structured affordances, and teams dynamically scoped them to match the interface’s real-time state.This component-based pattern was used with React and Vue.Source: §5 · p. 7
• VOIX’s client-side architecture places LLM inference costs on the user’s agent and provider rather than the website owner.The paper presents this decentralization as lowering a barrier to adoption.Source: §5 · p. 7
• All teams built functional agent-enabled applications within three days, and participants found VOIX’s two additional tags easy to use like HTML elements.The framework uses familiar declarative HTML and standard JavaScript event listeners.Source: §5 · p. 8
• The hackathon projects demonstrated abstract commands, click-plus-voice interactions, and redundant GUI-or-voice actions.These patterns illustrate the framework’s support for multimodal interaction.Source: §5 · p. 8

5.1 Latency Comparison

• 4.25 seconds to over 21 minutes: vision-based agents’ task latencies contrast with VOIX’s efficient single-step solutions and immediate feedback.The comparison concerns end-to-end latency; immediate feedback avoids long loops of inferring affordances, acting, and checking success.Source: §5.1 · p. 8, Table 1

5.2 Limitations and Future Work

• In legacy codebases, VOIX tools can fall out of sync with evolving GUIs, requiring development discipline and testing methods beyond this short-term study.A GUI feature without a corresponding VOIX implementation can make multimodal interaction inconsistent.Source: §5.2 · p. 8
• Tool design trades maintainability against performance: atomic GUI-mirroring tools offer little benefit, while high-level tools are harder to design comprehensively and can be brittle outside anticipated goals.The paper identifies finding a middle ground as a substantial design burden.Source: §5.2 · p. 8

6 Conclusion

• VOIX lets websites expose reliable, auditable, privacy-preserving agent affordances through declarative HTML, addressing brittleness, inefficiency, and security risks in scraping and DOM parsing.The framework shifts responsibility for defining agent capabilities from the agent provider to the website developer.Source: §6 · p. 8
• In a multi-team hackathon, developers with varying experience rapidly built functional agent-enabled applications and demonstrated synergistic multimodal interaction patterns.The study’s applications supported combining direct manipulation and natural language.Source: §6 · p. 8
• VOIX represents a deployable step toward an Agentic Web architecture that respects user privacy and developer control.Source: §6 · p. 8

A Hackathon Applications

• The Soundscape Creator uses VOIX tools to change multiple sound-mixing sliders from user instructions while generating matching images with a diffusion model.The application is built with vanilla HTML.Source: A Hackathon Applications · p. 10
• The Fitness App supports hands-free operation of workout functions and LLM-based exercise planning, with timers, set tracking, dashboards, and PDF report export.It is a React app for creating multi-week training plans with customizable days and exercises.Source: A Hackathon Applications · p. 10
• The Character Creator uses VOIX tools to fill out a fantasy-character quiz from instructions instead of selecting each question individually.The vanilla HTML site also supports LLM-guided role playing with created characters.Source: A Hackathon Applications · p. 10
• The Creative Studio exposes canvas design functions as VOIX tools, combining object selection with natural-language instructions for multimodal tasks.Users can modify shape and text properties including color, size, rotation, and position.Source: A Hackathon Applications · p. 10
• The Project Management Tool lets users create Kanban-board tasks through natural language in a Vue.js application.Tasks move through stages such as ‘To Do’, ‘In Progress’, and ‘Done’.Source: A Hackathon Applications · p. 10
• The Anki Creator controls category creation, image generation, and deck export through natural-language commands in a Vue.js application.The application generates image-based flashcard decks compatible with Anki.Source: A Hackathon Applications · p. 10

B Latency Benchmark

• The appendix details the methodology for the latency benchmark comparison presented in Table 1 and Section 5.1.Source: B Latency Benchmark · p. 11

B.1 Benchmark Objective

• The benchmark measures end-to-end task completion latency for VOIX versus Perplexity Comet and BrowserGym, contrasting declarative machine-readable interaction with visual-UI and raw-DOM interpretation.Source: §B.1 · p. 11

B.2 Systems Under Test

• VOIX tests use participant-built applications, a reference Chrome extension, and the Qwen3-235B-A22B model.Source: §B.2 · p. 11
• BrowserGym uses GPT-5-mini on the same applications as VOIX and receives the same high-level objective for each task.BrowserGym is identified as an open-source research framework for web agents.Source: §B.2 · p. 11
• Perplexity Comet is tested as a commercial vision-based agent that infers actions from screenshots and HTML, using its publicly available version as of September 4, 2025.Source: §B.2 · p. 11

• Latency ran from prompt submission until the requested change was fully rendered and visually confirmed.Source: §B.3 · p. 11
• Each task was executed once per platform for a single-shot completion-time measurement, with unsuccessful tasks retried up to three total attempts.VOIX required no retries for any task.Source: §B.3 · p. 11
• A task counted as failed if it remained incomplete after three attempts or exceeded the 25-minute timeout.Source: §B.3 · p. 11

• Canvas tasks prompted agents to add a blue triangle, rotate a green triangle 90 degrees left, delete a selected object, or export an image.Source: Creative Studio · p. 11
• Fitness tasks covered creating a full-week HIIT plan, starting Day 1, exporting Days 2 and 5 as a PDF, and showing workout statistics.Source: Creative Studio · p. 11
• Project-management tasks asked agents to create a database-optimization task, report in-progress ecommerce tasks, and copy the newest Website Redesign task to another project.Source: Creative Studio · p. 11