PaperlayerView exact revision on arXiv

Source-linked AI summary

Facilitating Proactive and Reactive Guidance for Decision Making on the Web: A Design Probe with WebSeek

Yanwei Huang, Arpit Narechania

arXiv:2601.15100v1cs.HC

TL;DR

Web agents lack transparent, iterative support for data-driven web decision making because they rely on text-centered interaction and often hide or freeze intermediate artifacts. WebSeek addresses this gap with a mixed-initiative, data-first browser extension whose interactive canvas supports tangible artifacts, direct manipulation, and proactive or reactive AI guidance. An exploratory study found successful complex-task completion with high confidence, agency, and control, while the paper notes limitations in context scalability and study scope.

  • Problem

    Web agents often hide intermediate data artifacts and make complex, customizable analysis depend on text, limiting transparent and iterative data-driven decision making.

  • Method

    WebSeek is a mixed-initiative browser extension with an interactive canvas where users and AI build, manipulate, transform, and refine tangible data artifacts.

  • Results

    Participants using WebSeek completed complex decision-making tasks successfully and reported high confidence, agency, and control while valuing direct manipulation and proactive and reactive guidance.

  • Takeaways & Limitations

    The findings support a shift from opaque conversational delegation toward transparent, data-centric human-AI collaboration for web-based analysis and decision making.

  • Takeaways & Limitations

    The study tasks cover only a limited portion of data analysis and may not reflect rigorous or large-scale activities, while raw-HTML context is not optimized for scale.

Abstract

from arXiv · show

Web AI agents such as ChatGPT Agent and GenSpark are increasingly used for routine web-based tasks, yet they still rely on text-based input prompts, lack proactive detection of user intent, and offer no support for interactive data analysis and decision making. We present WebSeek, a mixed-initiative browser extension that enables users to discover and extract information from webpages to then flexibly build, transform, and refine tangible data artifacts-such as tables, lists, and visualizations-all within an interactive canvas. Within this environment, users can perform analysis-including data transformations such as joining tables or creating visualizations-while an in-built AI both proactively offers context-aware guidance and automation, and reactively responds to explicit user requests. An exploratory user study (N=15) with WebSeek as a probe reveals participants' diverse analysis strategies, underscoring their desire for transparency and control during human-AI collaboration.

1 Introduction

WebSeek addresses fragmented, text-centered web workflows by making data artifacts directly manipulable and combining proactive and reactive AI guidance. An exploratory study found successful task completion alongside high confidence, agency, and control.

  • Motivation: Multi-step web tasks require users to gather, synthesize, and analyze information across sources, often through fragmented browser-tab and copy-paste workflows.This fragmentation increases cognitive load and friction during decision-making tasks.
  • Motivation: Existing web agents automate web actions through natural-language commands but often hide intermediate artifacts and provide static, non-customizable outputs.Their chatbot-like interfaces make text the primary interaction medium rather than shared, directly usable data representations.
  • WebSeek: WebSeek is a mixed-initiative browser extension that treats data as a first-class citizen through tangible artifacts, direct manipulation, and an interactive canvas.Users can build and refine extracted snippets and intermediate tables while performing edits and transformations.
  • WebSeek: WebSeek’s design space formalizes proactive and reactive guidance for data tasks while addressing when, what, and how AI should intervene.The system targets both low-level task support and the efficiency–control trade-off in open-ended decision making.
  • Study: N=15 participants completed complex decision-making tasks successfully and reported high confidence, agency, and control, while valuing direct manipulation and both guidance modes.The study covered news fact-checking and product research and revealed diverse workflows with varying reliance on AI guidance.

2 Related Work

Prior web agents streamline automation but can limit agency, hide manipulable intermediate data, and make complex transformations depend on text. WebSeek responds with integrated, iterative data work and a principled approach to proactive guidance.

  • AI-powered Web Agents: Web agents commonly use chatbot interfaces, browser extensions, or forked browsers to execute web tasks through natural-language interaction.These classes differ in how they access page context and expose suggestions or actions.
  • AI-powered Web Agents: Autonomous web-agent workflows can hide intermediate data and manipulation steps, while static snapshots limit users’ ability to inspect and modify artifacts directly.These limitations are especially relevant when users need agency for data-driven or high-stakes decisions.
  • AI-powered Web Agents: Text becomes a bottleneck when users must express nuanced data-transformation changes solely through language rather than direct manipulation.Direct manipulation can express some transformation nuances more directly and unambiguously.
  • Web Data Extraction and Analysis: Web data extraction and analysis remain separated in many systems, limiting seamless transitions and iterative exploration across the workflow.Real-time proactive guidance throughout the analysis process also remains challenging.
  • Proactive Guidance: Proactive guidance can reduce cognitive load and improve efficiency, but ill-timed or irrelevant suggestions may disrupt workflows or feel intrusive to users seeking control.The resulting design tension concerns what to suggest, when to trigger it, and where to present it.
  • Proactive Guidance: WebSeek contributes a detailed design space for proactive assistance in web-based data-driven sensemaking.The framework addresses the content, timing, and presentation of guidance within a specific domain.

3 A framework for mixed-initiative assistance in data tasks on the web

WebSeek’s framework adapts data-analysis stages and interaction modalities to web-based decision making, then organizes proactive, reactive, and direct-manipulation support around tangible data instances.

  • Research Questions: The framework asks which low-level data tasks comprise web decision making and how AI can assist across those tasks proactively and reactively.These questions adapt general data-analysis models to the specific web environment.
  • Framework Development: The authors developed the design space through an initial draft and multiple discussion-based revisions focused on when and what AI guidance should provide.The process was informed by the authors’ experience as data analysts.
  • Framework Structure: The framework organizes web data work into Discovery, Data Extraction & Wrangling, Data Profiling & Cleaning, and Data Modeling & Visualization.It adapts an empirically grounded enterprise data-analysis taxonomy to web tasks.
  • Guidance Design: Guidance presentation is divided between in-situ micro suggestions and peripheral macro suggestions designed to avoid disrupting the user’s flow.The framework also specifies proactive actions and triggering signals heuristically, with validity examined iteratively during design.
  • Interaction Modalities: The design space surveys interaction modalities spanning coding, DSLs, demonstration, examples, textual prompts, visual prompts, and direct manipulation.This maps the available ways users can express intent during web data tasks.
  • Interaction Modalities: Textual prompts and direct manipulation provide the broadest and most intuitive task coverage, motivating data instances as WebSeek’s universal interaction unit.Data instances can be manipulated directly while remaining manageable through other modalities.

4 Design Goals

WebSeek’s design goals center mixed-initiative assistance on tangible data instances, transparent plans, reliable tools, and user control. Its editors support direct data manipulation alongside in-situ and peripheral suggestions.

  • Design Goals: WebSeek’s design goals translate the framework into implementation principles for mixed-initiative data tools.The goals were distilled from applying the framework to real-world web-based sensemaking.
  • DG1: Holistic Context: DG1 requires AI to infer intent from system state, interaction history, webpage and view focus, and conversational history.Combining these signals is intended to support more relevant and timely assistance.
  • DG2: Interaction Modalities: WebSeek prioritizes direct manipulation and textual prompts as complementary modalities around data instances for everyday users and data analysts.Direct manipulation can construct instances or demonstrate examples for AI inference, while text remains broadly accessible.
  • DG3: Composite Suggestions: DG3 presents composite suggestions as transparent multi-step plans when a proposed action depends on prerequisite transformations.For example, visualizing price data may require formatting a Price column before visualization.
  • DG4: Tool-Based Execution: DG4 grounds AI actions in deterministic, concrete tools because allowing an LLM to generate final data states is unreliable and prone to hallucination.WebSeek executes tool calls through its own tested code so actions remain predictable and reversible.
  • Editors: The table editor supports spreadsheet-style editing and in-situ row-completion suggestions, while the visualization editor maps attributes to chart shelves.Visualization output includes interactive features such as zooming, panning, tooltips, and filtering.
  • DG5: User-Centered AI: DG5 makes AI deferential to user authority by yielding during active interaction, respecting viewport focus, showing system status, and clearing stale suggestions.These policies aim to prevent interference and keep guidance relevant to the user’s current goal.

5 WebSeek

WebSeek is a mixed-initiative browser extension that combines webpage data capture, tangible artifacts, direct manipulation, and proactive or reactive AI guidance in one workspace.

  • 5.1 Interface design: WebSeek provides an AI guidance view alongside a canvas workspace for creating data tables and visualizations.The guidance view supports peripheral suggestions and chat, while the workspace supports tangible data artifacts.
  • 5.1.1 Manual Instance Creation and Editing.: Users can manually edit tables and visualizations, including spreadsheet-like operations and advanced transformations such as joins, reshaping, and aggregation.The table editor supports cell editing, formulas, copying, selection, and adding or deleting rows and columns.
  • 5.1.1 Manual Instance Creation and Editing.: WebSeek supports webpage capture and source tracing by saving selected DOM elements into cells and navigating users to highlighted source content.The capture interaction enters selection mode, while the source interaction locates the originating DOM on the webpage.
  • 5.1.2 Tool Design and Guidance.: The system offers in-situ completions, peripheral suggestions, and chat-based guidance for proactive and reactive assistance.In-situ suggestions appear in editors, peripheral suggestions appear after more than 5 seconds of inactivity, and chat supports explicit requests.
  • 5.1.2 Tool Design and Guidance.: WebSeek uses webpage, instance, focus, conversation, and interaction contexts for guidance generation, but incorporating all contexts can cause context explosion and high latency.The authors identify more efficient generative models and agent frameworks as future work.
  • 5.2 Usage scenario: In the product-research scenario, users capture products from Amazon and eBay, enrich and join tables, and request a visualization through direct manipulation and chat.The system also proposes currency formatting before joining the tables.

6 Technical Evaluation

The technical evaluation tested WebSeek’s guidance generation across web data tasks of varying difficulty using a custom benchmark and simulated user interactions.

  • 6 Technical Evaluation: The evaluation assessed in-situ and peripheral suggestions across diverse web domains and task difficulties.Its methodology was adapted from prior evaluation of proactive guidance for visual analytics systems.
  • 6.1 Method: The benchmark contained 50 tasks spanning data extraction, wrangling, and visualization, with some tasks combining multiple activities.Tasks were designed because few existing benchmarks target web-based data wrangling or data-driven decision making.
  • 6.1 Method: Tasks were labeled Easy, Medium, or Hard according to whether they required multiple webpages, more than five transformations, or visualization.Tasks meeting 0 criteria were Easy, 1–2 were Medium, and all 3 were Hard.
  • 6.1 Method: The evaluation used Grok Code Fast 1 as a virtual user, with an author manually executing the model’s browser actions in WebSeek.The model received real-time system state, including HTML context, data instances, and prior guidance.
  • 6.2 Results: 97.2% overall accuracy was achieved, with guidance generated within 20 seconds of interactions on average.Remaining errors mainly involved data-type determination, tool-call formatting, and hallucinations; regeneration and alternative interactions enabled task completion.

7 User Study

The authors conducted an exploratory user study with WebSeek as a design probe to examine transparent human–AI collaboration in web data analysis and decision making.

  • 7 User Study: The study examined how participants perceived and used proactive guidance, reactive guidance, and tangible data artifacts through different interaction modalities.Confirmatory measures were included to evaluate usability and effectiveness despite the absence of suitable comparison baselines.

7.1 Study Design

The study recruited participants with data-analysis experience to complete fact-checking and product-comparison tasks using WebSeek under controlled study procedures.

  • 7.1.1 Participants: 15 participants were recruited from two academic institutions, including undergraduates, Master’s students, and Ph.D. students with experience in visual analytics or programming.Their average self-reported data-analysis experience was 3.67 on a 5-point Likert scale, with SD=0.724.
  • 7.1.3 Tasks: The two tasks combined an artificial fact-checking scenario with ground truth and an open-ended real-world product-comparison scenario.Both tasks covered the framework’s task categories and features.
  • 7.1.3 Tasks: The fact-checking task required cleaning and relating two 30-row tables containing Olympic medals, GDP, and population data with inconsistent entities, missing values, and redundant characters.Participants could clean the data manually or with AI assistance.
  • 7.1.3 Tasks: The product-comparison task asked participants to select a monitor under a HKD 1000 budget using criteria such as price, reviews or ratings, and resolution, then justify the choice with supporting data.Participants could define their own criteria and consult any number of shopping platforms.
  • 7.1.2 Procedure and Measures: Each session included a tutorial, a 10-minute warm-up, both tasks, a post-study questionnaire, and a semi-structured interview, lasting approximately 90 minutes.The study protocol was approved by the institution’s IRB committee.
  • 7.1.2 Procedure and Measures: Participants rated WebSeek’s usability and feature helpfulness, and evaluated guidance for intent alignment, reliability, error avoidance, and regret.Manual functionality reliability was also assessed.

7.2 Findings

WebSeek supported successful decision-making through a combination of manual control, direct manipulation, and multiple forms of AI assistance. Participants adopted diverse workflows, favoring manual interaction for control and accuracy while using AI for repetitive or complex operations.

  • Task outcomes: 12 of 15 participants reached the correct fact-checking conclusion, while all participants completed both tasks.Three fact-checking errors resulted from a data-cleaning error or misinterpretation of correct charts.
  • Interaction patterns: 67.1% of task time involved manual interactions, and every participant performed more than 70% of interactions manually.Chatting with AI and in-situ suggestions appeared in 86.2% and 82.8% of sessions, respectively, whereas peripheral suggestions appeared in 13.8%.
  • Interaction patterns: Participants used diverse workflows, including iterative extraction, chat-assisted wrangling, manual visualization, lengthy prompts, and custom scoring without visualizations.These patterns included both gradual refinement and more autonomous or manually controlled approaches.
  • Manual control and AI assistance: Direct manipulation was indispensable for small fixes, accuracy-critical tasks, thoughtful exploration, and correcting AI errors.Participants regarded manual actions as more reliable and predictable, and some used examples to communicate intended operations to the AI.
  • Manual control and AI assistance: In-situ suggestions were praised for reducing repetitive extraction work, but participants wanted control over completion granularity and local versus global intent.Participants considered in-situ suggestions accurate and intuitive while requesting mechanisms such as adjustable overhead and differentiated placement.
  • Manual control and AI assistance: Participants used chatting for complex or laborious operations, while peripheral suggestions were more selective because of attention costs and perceived misalignment.Chat supported tasks such as semantic extraction and table joining; peripheral suggestions were sometimes used for inspiration but often ignored or reproduced manually.

8 Discussion

WebSeek frames web-agent collaboration around tangible, user-controlled data artifacts rather than opaque conversational delegation. The discussion identifies benefits in transparency and control, while noting limitations in cue-driven guidance, scalability, native-web integration, and study validity.

  • WebSeek shifts agentic web automation from conversational delegation toward transparent, data-centric collaboration with direct user control over data.
  • 8.2 Infering Guidance Without an Evident Cue: Cue-driven proactive assistance improves relevance by responding to observable user actions, but may miss useful guidance about unrecognized data-quality problems.Examples include joining product reviews or normalizing mixed measurement units.
  • 8.3 Bridging the Gap Between a Data-Driven web extension and Native Web Interactions: Users valued a unified data space for confidence and control, but WebSeek paid less attention to integrating with native web interactions.Participants often began with familiar site interactions such as e-commerce filtering before extracting data into WebSeek.
  • 8.4 Improving the Traceability of Data Artifacts: Persistent visual links between transformed data artifacts and their web sources could strengthen transparency and user confidence.The current source link is not always visible after complex transformations.
  • 8.5 Limitations and Future Work: WebSeek’s raw-HTML and full-canvas context approach is effective for the study tasks but can become computationally expensive and inefficient at scale.Suggested directions include semantic search, dynamic context pruning, and smaller task-specific models.
  • 8.5 Limitations and Future Work: Canvas usability may degrade as data artifacts grow, and the study’s synthetic pages may not represent the DOM complexity of live websites.Future work proposes organizational structures such as hierarchical grouping, portals, and relationship graphs.

9 Conclusion

WebSeek presents a data-first browser extension for human-AI collaboration on web-based, data-driven tasks, combining tangible artifacts with proactive and reactive guidance. A user study found successful task completion alongside high confidence, agency, and control, motivating a shift toward transparent, data-centric web agents.

  • WebSeek combines direct manipulation of tangible data artifacts on an interactive canvas with an LLM-powered assistant and a design space for proactive and reactive guidance.
  • The user study showed that participants successfully completed complex tasks while reporting high confidence, agency, and control.
  • The paper argues for moving web-agent design from opaque conversational delegation toward transparent, data-centric collaboration.
Loading 2601.15100v1…