PaperlayerView exact revision on arXiv

Source-linked AI summary

PageGuide: Browser extension to assist users in navigating a webpage and locating information

Tin Nguyen, Thang T. Truong, Runtao Zhou, Trung Bui, Chirag Agarwal, Anh Totti Nguyen

arXiv:2604.23772v3cs.HC

TL;DR

Web agents can answer questions and automate browser tasks, but often omit visible page evidence and intermediate decisions needed for verification and control. PageGuide grounds Find, Guide, and Hide interactions in the HTML DOM, and a 94-participant study reports consistent gains over unaided browsing across all three features.

  • Problem

    Existing web agents return answers and execute actions without showing their page sources or pausing for user verification, leaving users to cross-check results and trust opaque steps.

  • Method

    PageGuide routes queries to Find, Guide, or Hide handlers that read and directly modify the HTML DOM, using highlights, stepwise instructions, or reviewable hiding decisions.

  • Results

    PageGuide consistently outperformed unaided browsing across all three features; Hide task time fell 70%, Guide completion rose 30 percentage points, and Find task time fell 19%.

  • Takeaways & Limitations

    Grounded, in-situ interaction supports tasks requiring users to understand answer evidence, follow procedures, or inspect what content has been changed.

  • Takeaways & Limitations

    The single-mode router cannot handle composite queries spanning multiple features, while Hide requires users to respecify hiding preferences across sessions.

Abstract

from arXiv · show

Users browsing the web daily struggle to quickly locate relevant information in cluttered pages, complete unfamiliar multi-step tasks, and stay focused amid distracting content. State-of-the-art AI assistants (e.g., ChatGPT, Gemini, Claude) and browser agents (e.g., OpenAI Operator, Browser Use) can answer questions and automate actions, yet they return answers without showing where the information comes from on the page, forcing users to manually verify results and blindly trust every automated steps. We present PageGuide, a browser extension that grounds LLM answers directly in the HTML DOM via visual overlays, addressing three core user needs: (a) Find-locating and highlighting relevant evidence in-situ so users can instantly verify answers on the page; (b) Guide-showing step-by-step instructions (e.g. how to change password) one at a time so users can follow and perform actions by themselves; and (c) Hide-hiding distracting content-giving users a chance to decide to hide an element or not. In a user study (N=94), PageGuide outperform unaided browsing across all modes: Hide accuracy improve by 26 percentage points (86.7% relative gain) and task completion time drops by 70%; Guide completion rate increases by 30 percentage points; and Find reduces manual search effort, with Ctrl+F usage falling by 80% and task time decreasing by 19%. Code and demo is at: pageguide.github.io.

1 Introduction

PageGuide addresses verification, step-level task support, and distraction control by grounding LLM outputs directly in webpage HTML through visual overlays. A controlled study evaluates these features against standard browsing and reports improvements across interaction modes.

  • Existing web agents return answers and perform actions without showing their page sources or exposing intermediate decisions, limiting user verification and intervention.
  • PageGuide grounds LLM outputs in HTML DOM elements through three features: Find highlights evidence, Guide delivers confirmed steps, and Hide presents reviewable hiding decisions.
  • The study compares PageGuide with standard browser tools in a within-subject evaluation of 94 participants across six tasks and three interaction modes.
  • The framework prioritizes in-situ inspectability over opaque end-to-end automation and is designed to keep users involved in verifying answers and actions.
  • PageGuide consistently outperforms unaided browsing and autonomous agents across all three interaction features, with effects on perceived ease of use, confidence, and control also examined.

2 Related Work

Related work frames web-agent design as a trade-off between autonomy and grounding, while existing reasoning and filtering systems leave important evidence and control on the webpage unresolved. PageGuide combines DOM-grounded outputs, mixed-initiative interaction, and intent-driven filtering in one extension.

  • Most browser agents emphasize autonomous end-to-end task completion but provide low grounding, leaving intermediate decisions opaque and page sources undisclosed.
  • Autonomous-agent opacity can produce incorrect high-stakes actions, motivating copilot designs that let users pause, reject, or choose alternatives.
  • Grounded reasoning methods link outputs to input spans, but webpage evidence remains unchanged, so users still locate supporting content themselves.
  • Existing filtering tools rely on static rules and generally lack natural-language intent handling, per-element explanations, and review or override controls.
  • PageGuide combines DOM-grounded answers, mixed-initiative control, and intent-driven filtering, treating the browser as a shared workspace for agent-user collaboration.

3 Method

PageGuide is a Chrome extension that reads and mutates the HTML DOM to ground Find, Guide, and Hide interactions directly on webpages. Its handlers highlight evidence, deliver user-confirmed procedural steps, or hide semantically matched content while preserving user control.

  • System Overview: PageGuide injects content scripts into webpages, enabling real-time read and write access to the HTML DOM.The system produces final page states by highlighting evidence, surfacing instructions, or hiding distracting elements.
  • Reading the HTML DOM: A structured element index records visible, text-bearing, or interactive DOM nodes with content, tag type, and bounding box so handlers can target page elements by index.Guide re-reads the DOM after each user-confirmed action so subsequent steps reflect the current page state.
  • Intent Router: The extension routes each query to Find, Guide, or Hide using a structured LLM call based on the query and compact page context.The router returns a mode label and one-sentence justification, allowing each pathway to use a task-specific handler.
  • Find Mode: Find generates answers whose factual claims cite exact text spans and DOM indices, then renders those spans as in-page highlights linked from the side-panel answer.Users can verify claims by viewing highlighted evidence without leaving the page.
  • Guide Mode: Guide generates an ordered action plan and presents one highlighted target at a time, while Next confirms and advances each step and Stop ends the session.The confirmation gate lets users inspect the target, preview the outcome, and retain veto power over every action.
  • Hide Mode: Hide uses an LLM to match natural-language hiding intents to DOM elements, adapting beyond fixed selectors and letting users review each decision before hiding content.This approach supports semantically specified criteria and preserves user control over hiding decisions.

4 User Study

In a within-subject study of 94 participants, PageGuide improved task outcomes and completion speed versus standard browsing, while reducing manual search effort and increasing perceived ease across Find, Guide, and Hide. Guide introduced a distinct behavioral pattern because users navigated to new pages as part of the workflow, and full completion was not guaranteed.

  • Accuracy and task success: 86% Find accuracy with PageGuide versus 81% control (+5 pp), although the difference was not statistically significant.The paired test reported p=0.32, with limited room for improvement from the high baseline.
  • Accuracy and task success: 56% Hide recall with PageGuide versus 30% control (+26 pp), a statistically significant gain.The comparison was significant at p<10^-5.
  • Accuracy and task success: 53% of Guide trials reached the correct terminal state with PageGuide versus 23% under control (+30 pp).Three raters evaluated final screenshots, with moderate inter-rater agreement.
  • Behavioral effort: Ctrl+F usage fell from 0.26 to 0.05 (-80%), while text selections, mouse clicks, scrolls, and mouse distance also decreased.After Benjamini–Hochberg correction, Ctrl+F usage, scroll count, and mouse clicks remained statistically significant.
  • Behavioral effort: Find and Hide reduced exploratory interactions, whereas Guide increased page visits by 35% and mouse movement distance because it directs users across pages.Page visits overall remained similar across conditions (0.64 versus 0.68), indicating efficiency gains within pages rather than fewer page visits.
  • Subjective experience: Participants generally rated PageGuide as easier, less effortful, and better for locating, verifying, and filtering information.Agreement that tasks were easier reached 83% for Find, 77% for Guide, and 89% for Hide.
  • Subjective experience: PageGuide increased partial completions, suggesting it helps users persist through difficult tasks without guaranteeing full success.This distinction motivates future work on adaptive step granularity and multi-turn correction.

5 Limitations and Future Work

The current system has limitations in query composition, cross-page evidence persistence, guided-step control, and persistent content-hiding preferences. Future work proposes multi-step planning, cross-page highlights, adaptive guidance, correction controls, and saved hiding preferences.

  • Limitations and Future Work: Single-mode dispatch cannot handle composite queries that require multiple features, such as Find followed by Guide.A multi-step planner is proposed to decompose such requests into ordered mode invocations.
  • Limitations and Future Work: Find highlights can overwhelm users on dense pages and currently persist only within a single page.Proposed remedies include re-ranking evidence and aggregating highlights across page visits.
  • Limitations and Future Work: Guide requires per-step confirmation, while future work proposes adaptive step granularity and multi-turn correction controls.Examples include skipping a step or undoing the last action.
  • Limitations and Future Work: Hide has no memory across sessions, and confirmation dialogs can add overhead for long lists of matched elements.A persistent, reviewable user preference history is proposed.

6 Conclusion

PageGuide embeds grounded LLM assistance directly in webpages across Find, Guide, and Hide, keeping users involved in verification and action. A controlled study found consistent gains across all three features compared with unaided browsing.

  • PageGuide grounds LLM answers in the HTML DOM across Find, Guide, and Hide, embedding outputs in the page rather than a sidebar.Find highlights evidence, Guide presents one confirmed action at a time, and Hide provides per-element justifications before hiding content.
  • Find reduced task completion time by 19% (65.2s →52.8s) and Ctrl+F usage by 80% (0.26 →0.05 presses per task).
  • Guide improved navigation completion by 30 % (23% →53%) while keeping users in full control of every action.
  • Hide reduced task completion time by 70% (104s →31.7s) and improved hide recall by 26 % (30% →56%, an 86.7% relative gain).
  • The study used unified task interfaces with instruction panels, timers, and standardized answer interfaces across Find, Guide, and Hide tasks.Post-study questionnaires captured participants’ subjective impressions after completing the tasks.

D.1 Intent Router Reliably Classifies User Intent

PageGuide’s intent router classifies user queries into Find, Guide, and Hide modes before downstream execution. On a labeled set of 1,208 queries, it achieved high overall accuracy and perfect Guide classification.

  • The intent router achieved 97.68% overall accuracy on 1,208 labeled queries spanning Find, Guide, and Hide.The evaluation set contained 969 Find, 132 Guide, and 107 Hide queries.
  • Guide queries were classified with 100% accuracy.

Gemini-3-Flash is Prefered for the Find task

The Find evaluation compares evidence-grounded retrieval across Natural Questions, QASPER, and a self-collected webpage subset using multiple model backbones. Gemini-3-Flash provides the strongest overall performance across the reported Find evaluations.

  • Find evaluation used Natural Questions and QASPER with human-annotated evidence spans, plus a 10-sample self-collected subset of heterogeneous webpages.
  • Gemini-3-Flash improved NQ Recall by +9.08 and Answer Correctness by +12.91, while achieving the highest QASPER Answer F1 with +5.90.
  • Gemini-3-Flash led QASPER Evidence F1 at 43.06 versus 29.85 for LED-base, a +13.21 difference.
  • Gemini-3-Flash was recommended as PageGuide’s default Find backbone because it achieved higher average QASPER performance and greater overall NQ improvements.

D.3 Guide Successfully Navigates Procedural Web Tasks

Guide is evaluated as a step-by-step navigator on benchmark and self-collected procedural web tasks. PageGuide’s one-step grounding approach outperformed the SeeAct baseline across reported settings.

  • PageGuide directly generates executable actions with grounded references, coupling reasoning and interaction in one step rather than separating them into two stages.
  • PageGuide exceeded the comparison backbone across all three Online-Mind2Web difficulty levels by +4.75.The benchmark reports task success rates for Easy, Medium, and Hard splits.
  • PageGuide surpassed SeeAct by +5.17 on average, achieving 35.17 versus 30.00.
  • On the self-collected Guide subset, PageGuide outperformed SeeAct with both backbones: 38.46% versus 26.57% and 54.55% versus 42.66%.The subset contained 143 samples and was designed to represent diverse real-world tasks beyond the academic benchmark.

D.4 Hide Effectively Hides Irrelevant Content

PageGuide’s Hide mode evaluates webpage elements against a user’s query and hides those judged irrelevant. On a 312-sample dataset, it achieves stronger recall and F1 than the comparison system, while precision remains a limitation.

  • Evaluation: PageGuide evaluates Hide using Precision, Recall, and F1 on ground-truth annotations of query-irrelevant webpage elements.The self-collected dataset contains 312 samples, and Avg is the mean across the three metrics.
  • Results: 59.36 vs. 46.84, +12.52 recall, indicating that PageGuide identifies more elements matching the user’s hiding intent.The comparison is between PageGuide and the evaluated alternative reported in Table 5.
  • Results: 47.42 vs. 41.56, +5.86 F1, while the average across metrics leads by +7.18 (50.92 vs. 43.74).The reported average combines Precision, Recall, and F1.
  • Limitation: Low precision is a meaningful risk because roughly half of PageGuide’s selected elements fall outside the ground-truth irrelevant set.The review dialog lets users uncheck individual matches, but it does not currently provide per-element confidence scores.

F Data Generation Pipeline

PageGuide constructs separate Find, Guide, and Hide datasets through task-specific collection, annotation, and validation pipelines. The pipelines produce grounded examples tailored to locating information, following instructions, and hiding page elements.

  • Hide: The Hide pipeline annotates webpage elements that should be hidden for specific requests and grounds each task to the corresponding page content.Annotators inspect the webpage and identify the target elements to remove.
  • Pipeline overview: The three PageGuide subsets use separate Find, Guide, and Hide data-generation pipelines.Each pipeline is designed around the corresponding feature’s task requirements.
  • Find: The Find pipeline collects webpages, filters for long content-dense pages, extracts requests, and manually validates examples before export.The subset targets pages where answers exist but are not immediately obvious.
  • Guide: The Guide pipeline selects official documentation and extracts, rewrites, and validates guide requests.Its sources include documentation, help-center articles, and tutorial pages for common user-facing tasks.
  • Validation: Manual review removes trivial, unclear, or duplicate examples and verifies that each request is feasible and supported by the visible page.Final Hide examples include a webpage, crawled HTML, a hide request, and grounded target elements.

G Beyond Basic Functionality: Enhancing Web Extensions with PDF Reading, Visual Question Answering, and Page-Off Support

PageGuide extends its core web features to support questions involving PDFs, uploaded images, and information beyond the active tab. These additions ground responses in document, webpage, or image evidence as appropriate.

  • Additional features: PageGuide adds PDF reading, visual question answering, and Page-Off support beyond Find, Hide, and Guide.These features address needs involving documents, uploaded images, and questions unrelated to the active webpage.
  • PDF reading: The PDF upload feature lets users ask questions about an uploaded document and receive answers grounded in evidence from that document.The example uses the Attention Is All You Need paper and asks about its main contributions.
  • Visual question answering: Visual question answering combines an uploaded image with webpage information and provides textual answers grounded in webpage evidence and highlighted image regions.The example compares a tiger in the uploaded image with a tiger described on the webpage.
  • Page-Off: Page-Off lets users ask questions unrelated to the active tab by gathering information from other webpages and providing supporting internet evidence.The feature uses information from other webpages rather than limiting the question to the currently visible page.

H Limitations of Existing Extensions

Existing browser-integrated agents can answer webpage questions but generally do not connect their outputs to specific visible evidence. PageGuide’s examples instead highlight relevant text spans, while its extension also requires browser-level testing for diverse interaction states.

  • Existing extensions: Gemini Chatbot, Browser Use, MolmoWeb, and Gemini Agent return answers without linking claims to specific HTML elements or highlighting supporting page spans.This makes it difficult for users to verify where the information came from.
  • Existing extensions: Gemini Chatbot provides only partial references and may highlight unrelated text, including male cast members for a query about female Stranger Things cast members.The example illustrates how dense text and incomplete highlighting reduce verification clarity.
  • Existing extensions: Browser Use correctly answers a Stranger Things query but cannot highlight the corresponding evidence with bounding boxes.The missing visual linkage makes verification difficult.
  • Existing extensions: MolmoWeb and Gemini Agent similarly answer queries without highlighting the corresponding evidence on Wikipedia and Megabus pages.Their examples show the same grounding limitation across different webpages.
  • PageGuide comparison: PageGuide generates a step-by-step plan and highlights relevant Megabus text spans for locating the timeframe for finding a lost item.The combined plan and page highlighting support locating and verifying the answer.
  • Testing: PageGuide’s browser-extension testing uses Playwright and GitHub Actions to cover extension loading, service workers, side panels, options pages, PDF viewing, and edge cases.The suite addresses failure modes that unit tests alone cannot fully cover in browser-extension development.
Loading 2604.23772v3…