Source-linked AI summary
Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
William Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright, Chris Russell
TL;DR
Websites can ask whether a browsing agent’s underlying model is identifiable from passive UI traces, a question with security implications for targeted attacks. The paper formalises this as behavioural classification using JavaScript-collected actions and timings across models and web tasks. It reports identification up to 96% F1 and finds that timing randomisation is not robust against retraining. The study also releases a harness and labelled trace corpus, while remaining bounded by its single harness, closed 14-model set, and imperfect open-set detection.
Problem
Websites may be able to identify which underlying model operates an LLM-based browser agent, creating a security-relevant model-attribution question.
Method
The paper trains supervised classifiers on passive JavaScript-collected sequences and timings of agent UI actions, excluding network-layer and model-internal information.
Results
Up to 96% F1 is achieved across 14 frontier models, while randomised timing delays remain insufficient when classifiers are retrained on delayed traces.
Takeaways & Limitations
Passive page observation can make ordinary agent navigation an attribution event that supports conditioning attacks on the identified model.
Takeaways & Limitations
The study uses a single Midscene.js harness, a closed set of 14 frontier models, and imperfect open-set detection.
Abstract
from arXiv · showhide
As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96\% F1. We formalise this attack surface by demonstrating that classifiers trained on agent actions generalise across model sizes and families. We further show that strong classifiers can be trained from few interaction traces and that agent identity can be inferred early within an episode. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our harness and a labelled corpus of agent traces \href{https://github.com/KabakaWilliam/known_actions}{here}.
1 Introduction
The paper asks whether websites can identify an agent’s underlying model from passive browsing traces and presents this as a security risk. It uses JavaScript-collected UI behaviour to fingerprint models, formalises defences and releases research resources.
- Motivation: Passive observation of clicks, scrolls, keypresses, and timings can reveal which foundation model controls a browser agent.The threat arises because model identification can support model-specific jailbreaks or narrower white-box attack searches.
- Contributions: A trace-collection pipeline logs agent actions and feeds the resulting trace to a model-identity classifier.The overview consists of agent actions, tracker logging, trace formation, and classification.
- Contributions: Up to 96% Macro F1 is achieved across 14 frontier models using behavioural traces collected through passive JavaScript injection.The classifier uses temporal and structural interaction dynamics rather than browser attributes or headers.
- Contributions: Randomised timing delays do not reliably remove the identifying signal when an adversary retrains on delayed traces.The paper frames this as defence analysis under a passive co-located adversary.
- Contributions: The authors release a labelled corpus spanning four web environments and a browser harness compatible with closed and open-source LLMs.These resources are intended to support reproducible behavioural attribution research.
2 Related Work
Prior work distinguishes humans from bots, re-identifies users through browser attributes, and fingerprints applications through active queries. This paper instead attributes the underlying model from ordinary UI events observed by the visited page.
- Prior work: Earlier behavioural-trace research primarily separates human users from automated clients, while browser attributes can re-identify humans across sessions.These approaches frame detection as binary human-versus-bot classification or client fingerprinting.
- Research gap: The paper asks which model produced an interaction trace once the client is known to be an LLM-based agent.This is a finer-grained attribution problem than human-versus-bot detection.
- Related security work: Side-channel and traffic-fingerprinting studies show that correlated observables can reveal hidden system properties without exposing outputs.Related web-traffic classifiers have identified visited websites from encrypted packet sequences.
- Novelty: Prior LLM application fingerprinting uses crafted queries and responses, whereas this work relies on passive UI events during ordinary browsing.Because agents share the same harness and action space, the target is the underlying model rather than an application or tool configuration.
- Novelty: The attribution surface therefore extends from network observers and active probers to the website visited by the agent.The claim follows from observing model-generated UI behaviour directly on the page.
3 Problem Formulation
The paper formulates model attribution as supervised classification over action sequences and timings collected by a passive, co-located site operator. It distinguishes closed-set identification from open-set unknown-agent detection.
- 3.1 Agent Identification as a Classification Problem: A browsing agent combines model m, environment E, and fixed harness h, repeatedly mapping observations to actions over planning steps.The environment executes each action and returns the next rendered observation.
- 3.1 Agent Identification as a Classification Problem: The classifier receives client-side interaction signals consisting of action sequence and inter-action timing, excluding headers, IP addresses, and TLS fingerprints.Traces can span multiple pages on one host but are restricted to a single domain to control environmental variability.
- 3.1 Agent Identification as a Classification Problem: The task is to predict the originating model from an unknown agent trace using a classifier trained on labelled traces.This differs from human-versus-bot discrimination because automation is assumed in advance.
- 3.1 Agent Identification as a Classification Problem: Features encode inter-event intervals, navigation structure, and interaction-pattern statistics such as action-type distributions.These mappings are designed to capture behavioural regularities associated with the underlying model.
- 3.2 Threat Model: The adversary is a passive co-located site operator that injects lightweight JavaScript and observes on-page actions without model internals, generated text, or network traffic.After identifying the model, the adversary may consult model-specific jailbreaks or initialise targeted optimisation.
- 3.2 Threat Model: Closed-set fingerprinting assumes every trace belongs to one of K known agents and performs standard multi-class classification.The classifier can be updated by routing a small number of sessions from a new model through the instrumented site.
- 3.2 Threat Model: Open-set fingerprinting includes unknown agents by training on known agents and flagging held-out agents as unknown.The leave-one-agent-out protocol evaluates binary known-versus-unknown discrimination using AUROC.
4 Experimental Setup
Experiments cover live information-seeking and shopping tasks across 14 multimodal LLMs using a shared browser harness and JavaScript event observer. Multiple classifiers are evaluated on closed- and open-set metrics.
- Data: Four live-web benchmarks cover information seeking and online shopping, including 2WikiMultiHop, FRAMES, Webshop, and Deepshop.Standard train, validation, and test splits are used across the environments.
- Models: The study evaluates 14 multimodal LLMs at both model-family and specific-variant granularity.The set includes open-source and proprietary frontier models spanning multiple families.
- Agent Harness: All agents use the Midscene.js computer-use harness with identical configuration, so behavioural differences are attributed to the underlying model rather than the harness.The harness translates model actions into Playwright commands for browser interaction.
- Trace collection: A lightweight JavaScript observer records DOM interaction events with millisecond-resolution timestamps and converts them into structured traces.A fresh browser context is used per session, and model identity supplies the class label.
- Classifiers: Five classifier families are trained, with XGBoost reported primarily because it achieves the strongest performance across datasets.The evaluated families include Lasso, logistic regression, random forest, XGBoost, and LSTM.
- Metrics: Closed-set evaluation reports per-LLM and macro F1, whereas open-set evaluation reports AUROC.These metrics correspond to multi-class identification and known-versus-unknown discrimination respectively.
- Hardware: Open-source models are served via vLLM on a node equipped with two NVIDIA H100 GPUs.This specifies the hardware used for local model serving.
5 Fingerprinting Results
Action traces reliably identify known browsing agents across four benchmarks, while open-set detection is above chance but exact attribution remains challenging. Closed-set and open-set identifiability can diverge substantially.
- Closed-set fingerprinting: 96.1% F1 on 2WikiMultiHopQA and 92.1% F1 on WebShop are achieved by the strongest identified agents.
- Closed-set fingerprinting: Per-agent F1 exceeds 70% for most models on every dataset, with the weakest reported result still 63.7% versus an approximately 7% random baseline.
- Generalisation: Single-task transfer is weak, but pooling traces from multiple tasks on the same site recovers strong attribution.
- Open-set fingerprinting: Most agents exceed AUROC 0.60 for unknown-agent detection, but Seed-2-lite falls below chance on three datasets despite leading closed-set identification.
- Robustness to timing perturbation: Retrained classifiers shift from timing features toward action-centred features such as structural key ratio and click position after five-second delays.
- Open-set fingerprinting: Open-set detection remains useful when attribution fails because unrecognized traces can trigger collection and later enrollment.
6 Analysis
Timing dominates clean-trace fingerprinting, but action features provide a more robust signal after delays. Fingerprints can be learned from limited data and inferred early during an episode.
- Timing and action signals: Timing features, including IEI variability, mean click IEI, and time to first action, receive the largest clean-trace attributions.
- Timing and action signals: Unadapted classifiers lose performance as random delays increase, whereas retraining on delayed traces largely restores performance across all four datasets.
- Timing and action signals: After timing disruption, classifiers rely more on action-centred features, indicating that behavioural fingerprints persist beyond temporal signatures.
- Training efficiency: Fewer than one third of traces suffice to approach peak classification performance across all four datasets.
- Identification speed: Macro F1 rises sharply within the first 40% of observed actions and then plateaus near full-trace performance across all datasets.
7 Implications
Passive model identification expands the web-agent attack surface by enabling model-conditioned exploitation, resource exhaustion, and access control. These uses allow site operators to tailor content to inferred agent identities.
- Attack Surface: Agent-Targeted Exploits: Knowing a target model reduces the search space from all possible attacks to those known to affect that model.
- Attack Surface: Agent-Targeted Exploits: AI Agent Traps can be conditioned on model identity, converting generic exploits into targeted ones across multiple attack surfaces.
- Attack Surface: Agent-Targeted Exploits: Model-specific prompt injection can use known jailbreaks or targeted optimisation instead of generic injections or costly black-box searches.
- Attack Surface: Agent-Targeted Exploits: Identity-conditioned sponge attacks can serve pages designed to maximise the visiting model's token consumption while showing normal content to others.
- Attack Surface: Agent-Targeted Exploits: Site operators can blacklist specific models or serve misleading model-conditioned content while appearing benign to other visitors.
8 Conclusion
UI actions can fingerprint an LLM web agent’s underlying model with up to 96% F1, while attribution can occur passively during ordinary page visits. The study also identifies scope limits and releases resources for further research.
- Up to 96% F1 identifies the underlying model from UI actions across 14 frontier models.The signal is typically recoverable from fewer than 15 observed events without browser attributes, network-layer visibility, or active probing.
- Passive attribution turns ordinary page visits into potential events for model-specific injections, content poisoning, or budget exhaustion.
- The paper reframes agent-aware web infrastructure around identifying which model produces behaviour rather than only detecting human versus automated clients.The authors release a trace corpus and evaluation harness to make this attribution axis measurable.
- The study is limited to one agent harness, a closed set of 14 frontier models, and imperfect open-set detection.Future work is proposed on harness-invariant signals, adaptive obfuscation defences, and broader browsing modalities.
- The behavioural attribution primitive is available to both attackers and operators as LLM agents become standard web clients.
A Additional Experimental Details
The appendix documents the evaluated models, browser tasks, harness execution, event collection, trace format, and behavioural features used for classification.
- A.1 LLMs Used: The experiments evaluate models from multiple families, including GLM, Qwen, UI-TARS, Gemma, Seed, GPT, Gemini, and Claude.The model table notes UI-specialist fine-tuning and reports active parameter counts for mixture-of-experts models.
- A.2 Tasks: The benchmark includes Wikipedia question answering and Amazon product selection tasks with browser-specific instructions.Amazon agents may search, filter, inspect, add, and remove products without logging in or checking out.
- A.3 Classifiers: Four classifier families are trained on behavioural features, with validation-based hyperparameter selection and refitting on the training split.The families are Random Forest, XGBoost, two Logistic Regression variants, and a hybrid LSTM.
- A.4 Harness and Execution: Each episode runs in an isolated container containing Chromium, Playwright, and the MidScene PlaywrightAgent action loop.The setup prevents browser state, cookies, or cached content from persisting across episodes or agents.
- A.4 Harness and Execution: Events are collected through an injected JavaScript push bridge, an end-of-episode backstop harvest, and separate Playwright navigation listeners.Completed episodes store metadata, answers, action logs, DOM traces, and timestamped browser interactions as JSON files.
- A.5 Behavioural Features Collected: The feature set contains 41 scalar measurements computed from client-side browser events and Midscene action logs.Feature families cover temporal dynamics, scrolling, clicking, navigation and action volume, and page-level normalised statistics.
- A.5 Behavioural Features Collected: Features include event-type counts, page and domain counts, scroll depth, and inter-event intervals derived from ordered event sequences.
B.1 Does Task Capability Predict Identifiability?
Task capability does not meaningfully predict agent identifiability: agents with high and low FRAMES accuracy can both have distinctive behavioural fingerprints.
- Capability is evaluated over 75 FRAMES questions with a maximum of 40 turns, and incomplete responses are marked incorrect.
- Neither Pearson’s r = 0.14 (p = 0.626) nor Spearman’s ρ = 0.05 (p = 0.852) shows a significant capability-identifiability relationship.Capability is measured by FRAMES accuracy, while identifiability is XGBoost F1 in the closed-set setting.
- Claude Opus 4.6 achieves 0.88 accuracy with F1 = 0.69, whereas UITars-7B records 0.03 accuracy with F1 = 0.90.
- Agents span the full identifiability range regardless of accuracy, supporting behavioural identity as distinct from task performance.The paper attributes fingerprints to action sequencing, tool-call management, and page navigation.
B.2 How well do our classifiers transfer across tasks and websites?
Behavioural fingerprints transfer weakly across individual tasks and websites but become strong when diverse traces from the same website are pooled, indicating site-conditioned attribution.
- Cross-task macro F1 falls from 79.4/75.3 in-domain to 41.1 and 49.8 when training on one Wikipedia task and testing on the other.
- Pooling 2WikiMultiHopQA and FRAMES traces recovers macro F1 of 81.3 on 2WikiMultiHopQA and 77.2 on FRAMES.
- Cross-site transfer remains much weaker, showing that behavioural fingerprints are shaped by the model, task distribution, harness, and website interface.The table caption likewise characterises fingerprints as site-conditioned rather than universal.
- These results suggest that site operators can learn robust identifiers from diverse traces collected on the same website rather than requiring universal task-invariant fingerprints.