Source-linked AI summary
UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design
Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, Dakuo Wang
TL;DR
UX usability testing is difficult to iterate and recruit for, motivating support before studies with real participants. UXAgent generates persona-based LLM Agents that interact with websites and provide behavioral, qualitative, and video data. UX researchers found the system helpful for refining study designs, while noting that its simulated behavior is not fully human-like and should not replace human participants.
Problem
Usability testing faces challenges in experiment design and recruiting enough qualified participants, especially for narrowly defined user groups.
Method
UXAgent generates large-scale persona-based LLM Agents, connects them to web environments, and produces behavioral traces, interviews, and video recordings for UX researchers.
Results
A heuristic evaluation with 5 UX researchers found UXAgent-generated data helpful for iterating experiment designs, although participants judged it unlike real human behavior.
Takeaways & Limitations
UXAgent can support researchers in designing and refining usability studies and reducing risk for real human subjects through human-AI collaboration.
Takeaways & Limitations
The evaluation used few participants, assessed perspectives qualitatively, and did not quantitatively compare LLM-Agent behavior with real human shopping behavior.
Abstract
from arXiv · showhide
Usability testing is a fundamental yet challenging (e.g., inflexible to iterate the study design flaws and hard to recruit study participants) research method for user experience (UX) researchers to evaluate a web design. Recent advances in Large Language Model-simulated Agent (LLM-Agent) research inspired us to design UXAgent to support UX researchers in evaluating and reiterating their usability testing study design before they conduct the real human subject study. Our system features an LLM-Agent module and a universal browser connector module so that UX researchers can automatically generate thousands of simulated users to test the target website. The results are shown in qualitative (e.g., interviewing how an agent thinks ), quantitative (e.g., # of actions), and video recording formats for UX researchers to analyze. Through a heuristic user evaluation with five UX researchers, participants praised the innovation of our system but also expressed concerns about the future of LLM Agent-assisted UX study.
1 Introduction
UXAgent addresses usability-testing challenges in experiment design and participant recruitment by using LLM Agents to simulate web users before real human studies. It combines scalable behavioral simulation with qualitative, quantitative, and video-based outputs, while evaluation found the generated data useful but not fully human-like.
- Challenges: Usability testing is difficult to iterate during experiment design and difficult to recruit when target users form a narrow group.Poorly designed studies may fail to produce useful results for feature iteration.
- Caveat: LLM Agents can differ from human users because optimized agents may take efficient paths whereas humans may use seemingly wasted actions.This creates a direct concern about interpreting simulated behavior as human behavior.
- System motivation: UXAgent is designed to generate LLM-Agent participants at scale and simulate interactions with web environments for UX research.The system is intended to support researchers before conducting studies with human participants.
- System motivation: The system provides behavioral traces, qualitative interviews, and video recordings for researchers to analyze simulated usability-testing data.These outputs support multiple familiar analysis methods.
- Evaluation: In a study with 5 UX researchers reviewing 60 agents on a shopping website, participants found the generated data useful for iterating experiment designs but not like real human behavior.Participants described the data as very detailed and unlike how real users think.
2 Related Work
Usability testing evaluates how users interact with products, but web-design studies face experiment-design and recruitment challenges. Prior agent systems addressed constrained environments, while LLM Web Agents motivate broader web-based usability-testing simulation.
- Usability testing: Usability testing observes users completing tasks to evaluate design effectiveness, identify usability issues, and measure satisfaction.It also helps validate design choices and reduce internal bias.
- Challenges in usability testing: Web-design usability testing remains challenging in experiment design and participant recruitment, while earlier agent tools mainly targeted predefined GUIs and games.Traditional web-automation constraints have limited broader web usability-testing applications.
- LLM Agents: Persona-based LLM Agents simulate diverse roles, preferences, and habits rather than focusing only on completing specific tasks.Prior work demonstrated believable behavior through interactions among agents with unique personas.
- LLM Web Agents: LLM Web Agents have performed complex web tasks such as searching, browsing, and shopping, motivating UXAgent for web-environment usability testing.UXAgent applies this capability to automating usability testing in web environments.
3 UXAgent: LLM-Agent-Based System for Usability Testing
UXAgent combines persona generation, an LLM Agent, browser interaction, simulation environments, and researcher-facing qualitative feedback tools. Its architecture supports diverse web interactions while producing traces, recordings, and outcomes for usability analysis.
- System overview: UXAgent generates LLM-Agent participants at scale and runs simulated interactions with web environments to collect behavioral data.The system is proposed as an LLM-Agent-based approach to usability testing.
- System architecture: The Persona Generator creates diverse personas, while the Universal Browser Connector converts browser content and agent actions into executable interactions.The connector parses raw HTML into simplified representations and translates actions such as clicking into browser operations.
- Persona generation: Users specify demographic distributions and example personas so the system can generate the desired number of personas with matching randomized demographics.This supports large-scale simulation across different user backgrounds.
- LLM Agent: The LLM Agent uses a Memory Stream and Fast and Slow loops, with the Fast Loop handling responsive interaction and the Slow Loop providing higher-level reasoning.The Fast Loop includes perception, planning, and action modules; the Slow Loop includes wondering and reflection.
- Universal Browser Connector Module: The Universal Browser Connector simplifies visible HTML into a tree and generates task-agnostic actions such as click, type, and back for cross-website interaction.It trims unnecessary CSS and JavaScript while preserving visible page information.
- Researcher interface: Researchers can interview simulated personas through a chat interface using their action and memory traces after each simulation.The system also produces video recordings, traces, and final outcomes such as purchases or session exits.
- Simulation environment and task: UXAgent was deployed on WebArena and Google Flights, and one WebArena study simulated 60 personas across gender and income groups.The study used the shopping intent “buy a jacket” and found average purchased amounts increasing across income groups.
4 User Study
Five UX researchers analyzed UXAgent-generated usability-testing data through trust assessment, sense-making, hypothesis formation, and conclusion drawing. They found the data helpful for iterating study designs, while raising concerns about realism, bias, and the readability of raw traces.
- Participant Recruitment and Participant Task: Five UX researchers analyzed UXAgent-generated video recordings, action traces, memory logs, and final outcomes from simulated shopping sessions.Participants navigated the data while thinking aloud, interacted with a chatbot, completed surveys, and joined post-study interviews.
- How Did UX Researchers Use Our UXAgent System?: Participants first evaluated the trustworthiness of each data type before conducting deeper analysis, although some who skipped verification encountered later challenges.Trust assessment was the predominant initial activity in the study.
- How Did UX Researchers Use Our UXAgent System?: After establishing trust, participants interpreted relationships between variables, calculated group-level statistics, and sometimes misread randomly generated persona names as meaningful behavioral categories.One participant analyzed shopping behavior by names such as “Jake,” suggesting that understanding data generation is important for adoption.
- How Did UX Researchers Use Our UXAgent System?: Participants formed hypotheses and checked raw action logs, including explaining lower non-binary purchase rates through product availability and a failed unisex-product search.The action trace supplied evidence for the proposed explanation.
- How Did UX Researchers Use Our UXAgent System?: Participants organized findings and discussed revising and rerunning their study designs, with one participant saying rapid simulation could encourage more iterative design.The cited participant described the system as quick and easy for repeated design iteration.
- Participants’ Perceptions and Concerns of UXAgent: Although simulated behavior was rated not very realistic (M=3, SD=0.7), participants rated the data helpful (M=3.4, SD=0.89), insightful (M=3.6, SD=0.89), and trustworthy (M=3.6, SD=1.51).They especially valued interviewing agents (M=4.8, SD=0.44), final outcomes and statistics (M=4.6, SD=0.54), and action traces (M=4.2, SD=0.84).
- Participants’ Perceptions and Concerns of UXAgent: Participants found raw memory traces difficult to read and analyze, while video recordings helped them gain trust but were hard and time-consuming to analyze.Participants suggested presenting thoughts through highlights or summaries rather than unprocessed traces.
- Participants’ Perceptions and Concerns of UXAgent: Participants raised concerns about representation bias, algorithmic decision bias, and simulated behavior that may be too detailed or unlike real human behavior.Despite these concerns, they still valued the additional insights unavailable from experiments with real human participants.
5 Discussion
UXAgent offers faster, lower-cost UX study iteration through simulated user behavior, but the paper emphasizes that these outputs remain supplementary and raise privacy, ethical, and validity concerns. The authors identify future needs for automated insight generation, stronger human-versus-agent comparisons, and multimodal processing.
- UXAgent can support faster study-design iteration while reducing costs and improving study-design quality.The authors frame these benefits as potential enabled by simulated user behavior.
- Researchers requested automated high-level summaries because raw agent memories were difficult to analyze.The paper presents insight summarization as a future system-design direction.
- LLM-assisted UX studies introduce privacy and ethical risks, especially when sensitive data are involved.Participants treated agent data as supplementary rather than definitive because LLMs may not accurately represent outcomes and can contain bias.
- Using simulated data as a substitute for human participants could produce flawed conclusions because it cannot fully reproduce complex human behavior.The authors call for clearer limits, ethical guidelines, and privacy safeguards.
- The evaluation used few participants, lacked quantitative comparisons with real human shopping behavior, and excluded visual webpage information.The authors propose larger studies, systematic human-versus-agent analysis, and multimodal models as responses.
- Future work could extend UXAgent beyond web applications to desktop, mobile, and mixed-reality interfaces.This expansion is presented as a possible direction for broader user-research contexts.
6 Conclusion
UXAgent enables researchers to run simulated user studies and iteratively refine UX study designs. The evaluation suggests that it can improve study quality and reduce risk for real human subjects, positioning the work as an initial step toward human-AI collaboration in UX research.
- UXAgent enables simulated user studies that facilitate iterative refinement of UX study designs.
- The evaluation suggests UXAgent can improve UX study quality and reduce risk for real human subjects.
- The authors position the system as a first step toward human-AI collaboration in UX research.
A Browser Environment
The browser environment converts webpages into simplified, semantically informative observations and exposes a structured action space for agents. Parsing policies remove redundant structure while preserving useful text, semantic attributes, state, and interaction context.
- Observation Space: The observation space is represented as a JSON object containing webpage URL, simplified HTML, clickables, inputs, and error messages.
- Handling of Text: The parser extracts text through default, CSS-selector, or JavaScript methods, with JavaScript supporting rare cases such as sibling-label text.
- Handling of Text: Text formatting can add context when visual and textual information differ, such as converting a review score into “Rating: 4.2 out of 5 stars”.
- Handling of Tag Name and Attributes: The parser removes most visual attributes while retaining semantic and state information, and can override tag names for readability and accessibility.
- Handling of Tag Name and Attributes: Recipe options preserve or override selected attributes when class values carry semantic information despite usually describing visual styling.
- Handling of Tag Name and Attributes: JavaScript state such as selected options and input values is extracted into the simplified HTML for agent access.
- Handling of Child Elements: Nested styling elements are flattened by selecting relevant deep children or removing child elements when no children field is specified.
- Action Space: The simplified HTML preserves webpage semantics while reducing redundant tags, and the browser action space includes click, type, type_and_submit, clear, back, and terminate.
B Prompt
The persona prompt instructs the system to generate diverse personas using specified demographic attributes and a consistent example-based format. It also constrains the output to the persona alone.
- B.1 Prompt for Persona Generatoin: The prompt asks the model to generate a persona different from previous personas to ensure diversity.
- B.1 Prompt for Persona Generatoin: Each generated persona must have the specified age.
- B.1 Prompt for Persona Generatoin: Each generated persona must have the specified gender.
- B.1 Prompt for Persona Generatoin: Each generated persona must have income within the specified range.
- B.1 Prompt for Persona Generatoin: The persona must follow the same format as the examples.
- B.1 Prompt for Persona Generatoin: The prompt requires outputting only the persona and no additional text.
C.1 Action Trace
The action trace records a shopping task as a sequence of concrete browser actions, from searching for a product to configuring and adding it to a cart.
- The trace begins by typing “woman's jacket” into the search field and submitting the form.
- The agent then opens a women’s jacket product and selects the Navy color option.
- The trace continues by selecting the Medium size and adding the jacket to the cart.
C.2 Memory Trace
The memory trace combines task actions with the agent’s intermediate thoughts and observations, including product requirements, budget concerns, and search-result interpretation.
- The agent’s thoughts include personal context about networking, employment, budgeting, and financial stress.
- The trace records the agent’s intention to search for a navy blue, medium-sized women’s jacket after submitting the query.
- The memory includes observations confirming that the search succeeded and identifying the page title for the query.