PaperlayerView exact revision on arXiv

Source-linked AI summary

Penny for Your Prompts: A Measurement Study of Contextual Leakage in AI Chat Platforms

Paschalis Bekos, Billy Tsouvalas, Nick Nikiforakis, Michalis Polychronakis

arXiv:2610.37467v1cs.CRcs.CY

TL;DR

Chatbot interactions can include sensitive disclosures, while third-party tracking scripts may collect page content and personal information; the scale and mechanics of this leakage across chatbot platforms had not been fully traced. The study audits 42 platforms and uses JavaScript-level instrumentation to connect leakage to user actions and extraction mechanisms. It finds widespread leakage of conversation titles and prompts, with email exposure also enabling identity linkage.

  • Problem

    The scale and mechanics of third-party conversational-context leakage across general-purpose and sensitive-domain chatbots remain underexplored.

    Source: §1 · p. 1

  • Method

    The study audits 42 chatbot platforms using network capture and JavaScript-engine instrumentation to attribute leakage flows to user actions and extraction mechanisms.

    Source: §1 · p. 2

  • Results

    21 platforms leaked conversation titles and 9 leaked verbatim user prompts, while email exposure on context-leaking general-purpose platforms could link context to user identity.

    Source: §1 · p. 2, §8 · p. 14

  • Takeaways & Limitations

    Chatbot page structures can encode user intent in ways that let standard tracking integrations inadvertently capture sensitive disclosures, so privacy audits should cover varied interaction events.

    Source: §8 · p. 14

    The study covers web-based platforms offering free account registration, excluding paid tiers and chatbots primarily distributed as mobile or desktop applications.

    Source: Platform Coverage · p. 12

Abstract

from arXiv · show

AI chatbot platforms have become one of the most popular web interfaces users interact with online, routinely receiving personal and intimate information shared under an expectation of privacy. Yet, on the web application side, these platforms have the ability to embed third-party analytics, advertising, and session replay scripts that operate silently alongside the chatbot, creating an underexamined channel for conversational data leakage towards third parties. We conduct the first systematic large-scale measurement study of contextual leakage in AI chatbots, auditing the web interfaces of 42 general-purpose and domain-specific (health, legal, and education) platforms. Beyond standard network interception, we instrument the browser at the JavaScript engine level to attribute each data flow to the exact user action that triggered it, and the DOM capture mechanism used to retrieve its content, a granularity invisible to network-layer analysis alone. We find that 21 of these platforms leak conversation titles to third-party trackers, while nine leak verbatim user prompts. Beyond content, we observe a substantial amount of chatbot platforms sending user emails to third-party trackers, enabling the association of conversational context with a user's identity. Our findings cast traditional web privacy issues under a new light, highlighting the privacy issues that are inherent in how users interact with modern AI chatbots.

1. Introduction

Chatbot prompts can contain sensitive personal disclosures, while third-party tracking scripts operate on these platforms much as they do on other websites. This study measures contextual and identity leakage across chatbot platforms and traces how user actions trigger JavaScript-level extraction.

1. Introduction

  • Prior studies used limited-scale network capture and largely overlooked multi-interaction leaks and task-specific chatbots in sensitive domains.Conversation titles may be generated only after an initial prompt, making interaction-driven leakage easy to miss.Source: §1 · p. 1
  • The study combined network capture with JavaScript-engine instrumentation to attribute data flows to user interactions and reconstruct how scripts extract conversational context.The method also examined five interaction scenarios per platform.Source: §1 · p. 2
  • 21 platforms leaked conversation titles and 9 leaked verbatim prompts to third-party trackers; among general-purpose platforms, 11 of 13 context-leaking platforms also exposed user email.The email exposure can link conversational context to a named individual.Source: §1 · p. 2
  • The audit covered 42 platforms, including general-purpose chatbots and task-specific agents in health, legal, and education.Source: §1 · p. 2
  • The study identifies conversational-context leakage alongside identity leakage, allowing trackers receiving context to link it to a user.Source: §1 · p. 2
  • Third-party scripts can capture chatbot context through familiar tracking mechanisms because dynamic page attributes encode user intent rather than generic metadata.The same mechanisms used to read descriptive page content on traditional websites can capture conversational context on chatbot interfaces.Source: §1 · pp. 1–2

2. Background

The background describes how chatbot interfaces make detailed, potentially sensitive input and conversation history part of the page. It also outlines how third-party scripts can read page content and user information, motivating investigation of these flows.

2. Background

  • Some popular chatbots reference many third-party domains in their Content Security Policies, indicating potential for external data flows but not proving active collection.You.com, QuillBot, and Claude list 66, 57, and 56 distinct domains, respectively.Source: §2 · p. 2, Figure 1 · p. 2

2.1. AI Chatbots

  • Chatbot prompts can contain detailed sensitive disclosures, and preserved chat histories plus automatically generated titles retain semantic context from prior questions.Source: §2.1 · p. 2
  • The leakage model describes a tracker capturing an account email and later reading a conversation title or prompt, enabling it to associate identity with inquiry context.The email establishes a persistent identifier before the tracker collects conversational content.Source: §2.1 · pp. 2–3, Figure 2 · p. 3
  • Prior work measured chatbot tracking or examined app ecosystems and malicious extensions, but had not traced the full user-interaction-to-JavaScript-extraction chain or covered task-specific sensitive-domain bots.Source: §2.1 · p. 3

2.2. Tracking Pixels

  • Tracking pixels and analytics scripts can read webpage context such as URLs, titles, and referrers, and form listeners can capture personal information such as email addresses.Source: §2.2 · p. 3
  • Because chatbot prompts, responses, and generated titles appear in the DOM, scripts that read page content can capture users’ private query context.This exposes not only who a user is, but also what they ask about or disclose.Source: §2.2 · p. 3

3. Methodology

The study combines network capture with JavaScript-level instrumentation to measure third-party collection from chatbot interfaces and attribute leakage to user actions and content-extraction mechanisms. It audits 42 platforms across general-purpose and sensitive-domain chatbots using structured sessions and isolated interaction scenarios.

3. Methodology

  • The methodology pairs network capture with JavaScript-level instrumentation to identify third-party data collection and trace each flow to its triggering interaction and extraction mechanism.Network interception captures outgoing traffic; VisibleV8 logs JavaScript API calls and DOM reads that reveal how scripts retrieve context.Source: §3 · p. 3, §3.3 · p. 5

3.1. AI Chat Platform Selection

  • The audit covers 42 web-based platforms: 25 general-purpose chatbots and 17 task-specific platforms in education, health, and legal assistance.Selection requires free public access and a web interface accepting free-form prompts; mobile apps are excluded.Source: §3.1 · p. 4

3.2. Session Capture

  • Structured sessions capture requests containing seeded prompts, generated titles, or personal information, while multi-interaction sessions cannot isolate which action triggered a flow.General-purpose platforms receive generic and sensitive prompts; task-specific platforms receive domain-relevant prompts, with one Gmail account used for those tests.Source: §3.2 · pp. 4–5

3.3. Interaction Decomposition

  • Five isolated scenarios are compared against prompt submission alone, while VisibleV8 traces identify the script, DOM reads, and network API behind each observed leak.Scenarios include navigation, deletion, scrolling, and an unrelated button click; prompt and generated-title values are checked in original and URL-encoded forms.Source: §3.3 · p. 5

4. Findings

The findings examine how third-party trackers on AI chat platforms access chatbot page content and receive conversation titles, verbatim prompts, and user emails. They characterize the actions and script reads associated with these flows, while noting that privacy notices may not acknowledge the informational payload.

4.1.2. Conversational Context Leakage.

  • Conversation titles reach trackers from 13 of 25 general-purpose platforms and 8 of 17 task-specific platforms; four and five platforms, respectively, also leak verbatim prompts.You.com averages 92 prompt-bearing requests to Google per prompt, while task-specific leaks can include medical or legal details.Source: §4.1.2 · p. 7, §4.2.2 · pp. 8–9

4.1.3. Context Leakage per Tracker.

  • Google receives 230 distinct context-bearing requests across eight general-purpose platforms, while Meta receives about 55 across eight; tracker traffic also concentrates among a few owners.Google receives more than 40% of generic-chatbot tracker traffic, and Google, Meta, and Microsoft together account for more than 70% of task-specific tracker requests.Source: §4.1.1 · pp. 6–7, §4.1.3 · p. 7, §4.2.1 · p. 8

4.1.4. Email Leakage and Identity Correlation.

  • Of 13 general-purpose platforms leaking context, eight send both context and email to one company; seven of eight task-specific context-leaking platforms do likewise, enabling identity linkage.For two general-purpose platforms, email and context share the same request; four others share a persistent identifier across requests. In the task-specific set, AI Lawyer sends title and email in the same request to Meta and ByteDance.Source: §4.1.4 · p. 8, §4.2.2 · p. 9, Figure 9 · p. 9

4.2.1. Tracker Landscape.

  • Task-specific platforms send 39.7% of outbound requests to trackers, versus 22% for general-purpose platforms, despite generating fewer requests per prompt (353 vs. 540).The task-specific tracker share is similar across education (38.5%), health (41.6%), and legal (41.1%).Source: §4.1 · p. 6, §4.2.1 · p. 8

4.2.3. Leakage Across Task-Specific Chatbot Categories.

  • Education platforms average roughly 367 tracker requests and leak content in about 24%; legal platforms average 317 and leak 9%, while health platforms average 124 and leak 3%.Leakage is concentrated in individual services: SciSpace accounts for roughly 80% of education leaks and AI Lawyer for nearly all legal leaks. Content is sent in plaintext; health platforms that leak email send it raw, unlike predominantly hashed email in education and legal.Source: §4.2.3 · pp. 9–10, §4.3 · p. 10, Figure 10 · p. 10

4.3. Interaction Decomposition.

  • Six platforms leak context upon prompt submission, all nine expose conversation titles during navigation, and scrolling or clicking adds no new destinations; deletion triggers title leakage on Writesonic alone.The analysis isolates five interaction scenarios in separate browser sessions, comparing observed third-party destinations with the prompt-submission baseline.Source: §4.3 · pp. 10–11, Table 3 · p. 11
  • Trackers commonly obtain titles through document.title reads, while Meta Pixel selects the title element and Clarity reads DOM text and values, including prompts before a title exists.A document.title read occurs on eight of nine platforms; Meta Pixel uses querySelector("title") and textContent on all five platforms where present.Source: §4.3 · p. 11
  • Manus tracking captures clicked conversation titles through element text and attributes, while You.com places prompts in the page URL for trackers reading location.href.Conversation sub-pages with dynamically generated, context-derived titles turn conventional page-view and title-tracking mechanisms into routes for transmitting conversational context.Source: §4.3 · p. 11

4.4. Privacy Policies

  • Privacy notices may disclaim selling or sharing data for targeted advertising while advertising and analytics pixels collect conversational context; notices describe browsing metrics but not this payload.The notices describe collection of page views, click events, or session duration.Source: §4.4 · p. 11
  • Chat interfaces dynamically derive structural elements from user input, so trackers’ capture of page components can retrieve prompts and semantically compressed conversation titles rather than generic page metadata.Policies acknowledge pixels for page views and interaction metrics without acknowledging that these signals carry a different informational payload on chat platforms.Source: §4.4 · p. 12

Key Takeaways

  • 21 platforms leaked conversation titles and nine leaked verbatim user prompts to third-party trackers, showing contextual leakage across the audited platforms.The audit covered 42 platforms.Source: Key Takeaways · p. 12
  • Trackers receiving conversational context frequently also obtained users’ emails, allowing sensitive disclosures to be tied to named individuals.Source: Key Takeaways · p. 12
  • Traditional-web analytics tools on AI chat platforms should be treated as a privacy risk because they can capture conversational context users expect to remain private.Source: Key Takeaways · p. 12

5. Limitations

The measurements cover platforms under fixed conditions during a defined period, from one US-East vantage point with tracking consent accepted. The study is limited to free-registration web platforms, excluding paid tiers and app-based chatbots.

Experimental Repetitions.

  • Platforms were measured under identical conditions from May 6 to May 20, 2026, so the results describe deployments during that period.Tracker deployments can change as platforms add or remove vendors.Source: Experimental Repetitions · p. 12

Location and Consent.

  • Measurements used one US-East vantage point with tracking consent accepted, so they do not assess consent compliance or tracker variation across jurisdictions.Consent state and region determine which trackers a platform loads.Source: Location and Consent · p. 12

Platform Coverage.

  • The study excludes paid tiers and chatbots distributed primarily as mobile or desktop applications, covering only web platforms with free account registration.Extending the analysis to application-based clients could provide additional insights into the ecosystem.Source: Platform Coverage · p. 12

6. Related Work

Prior work studies privacy risks in models, applications, and web tracking, while chatbot-interface tracking remains less examined. This study treats tracking as interaction-driven and uses multi-interaction sessions and JavaScript instrumentation to attribute leakage to triggers and extraction mechanisms.

Model and Application Leakage.

  • Model- and application-layer studies examine PII memorization, jailbreaks, prompt regurgitation, file exfiltration, and cross-session disclosure, but not embedded web-interface trackers.These studies address risks originating from the LLM or its application layer.Source: Model and Application Leakage · pp. 12–13

Third-party Tracking on the Web.

  • Web-tracking research has measured third-party presence, sensitive-site tracking, PII collection, and contextual collection by Meta Pixel and its deployment structures.Examples include automated registration-form submission and artificial forms to measure PII leakage.Source: Third-party Tracking on the Web · p. 13

Third-party Tracking on AI Chatbots.

  • Prior work measured content and identity exposure across 20 general-purpose AI chatbots via single-prompt network capture and third-party tracking across four chatbot platforms via HAR analysis.Single-prompt network capture is insufficient to understand when and how contextual leakage occurs.Source: Third-party Tracking on AI Chatbots · p. 13
  • Unlike model- and application-layer studies, this work examines web-interface tracking and decomposes interactions to attribute leakage to causal triggers.Its JavaScript analysis also identifies generic page-metadata primitives used to capture context, which network-only measurements cannot establish.Source: Third-party Tracking on AI Chatbots · p. 13, Model and Application Leakage · p. 13

7. Discussion

Conventional tracking scripts can collect chatbot prompts and dynamically generated conversation titles through ordinary page-structure mechanisms. The discussion describes privacy-conscious analytics and broader interaction audits as responses, and reports platform remediation following disclosure.

Discussion

  • Chatbot page titles and other structural components can encode disclosed conversational content, so conventional tracking mechanisms may capture more than generic page metadata.The discussion argues that chatbot platforms should not be treated like traditional websites when configuring third-party tracking integrations.Source: §7 · p. 13
  • The proposed long-term direction is AI-specific analytics that decouple usage measurement from conversational content, alongside less specific titles or stricter vendor constraints.The paper also notes that blocking tracking scripts would prevent the observed leakage.Source: §7 · pp. 13–14
  • Privacy audits should cover navigation, button clicks, and chat deletion as well as page loads and prompt submissions, because these interactions can independently trigger leakage.Tracking pixels are event-driven, so narrower audits may miss such triggers.Source: §7 · p. 14
  • After responsible disclosure, ChatGPT and Grok had removed trackers that had collected conversational context, and TheLawGPT remediated its leaking tracker configuration.Other named platforms acknowledged or forwarded the researchers’ notifications to appropriate teams.Source: §7 · p. 14

8. Conclusion

The study audits 42 general-purpose and task-specific chatbot platforms and finds widespread leakage of conversation titles, prompts, and user emails. Its interaction decomposition attributes flows to triggers and reconstructs the JavaScript mechanisms used to extract context.

Conclusion

  • 21 platforms transmitted conversation titles and 9 transmitted verbatim prompts to third-party trackers; most context-leaking platforms also exposed user emails, enabling linkage to named individuals.The audit covered 42 platforms, including task-specific chatbots in sensitive domains.Source: §8 · p. 14
  • Chatbot interfaces dynamically encode user intent in structural page components, and leakage may occur under different interaction events, motivating audits across the full range of user interactions.The conclusion contrasts this with traditional websites and calls for careful evaluation of third-party tracking deployments.Source: §8 · p. 14

Appendix A. Extended Findings

The extended findings detail how tracker ownership and leakage vary across platforms, accounts, and chatbot types. The authors also provide the collected traffic logs, JavaScript traces, and reproduction scripts.

Appendix A. Extended Findings

  • Under Gmail, Kimi, ZenoChat, ChatGPT, and Manus leak about 12–24× more whole-string content than under non-Gmail accounts.You.com, Writesonic, and Genspark show the opposite direction; each comparison uses one session per configuration.Source: Appendix A · p. 16, Figure 12 · p. 16
  • Google’s context-bearing requests span multiple endpoints, including google.com, doubleclick.net, and googleadservices.com.Each endpoint independently accumulates substantial requests, as broken down in Figure 13.Source: Appendix A · p. 16, Figure 13 · p. 17
  • Google, Meta, and Microsoft account for 70% of tracking-related requests, and Google appears in every network trace.The reported company-level flow is shown in Figure 14.Source: Appendix A · p. 17, Figure 14 · p. 17
  • Eight task-specific chatbots share conversation titles with seven tracker-owning companies, while five instances of full-prompt leakage reach four companies.Title leakage is heavier in academic and educational chatbots, with SciSpace sending many title-bearing requests to Google.Source: Appendix A · p. 17, Figure 15 · p. 17, Figure 16 · p. 17

Appendix B. Open Science

  • The accompanying artifact includes traffic logs and VisibleV8 execution traces for all 42 platforms, plus scripts to reproduce the findings.Source: Appendix B · p. 17

Appendix C. AI Usage

The authors describe their use of AI in conducting the study and preparing the manuscript, alongside the study’s controlled, observational ethics approach. The supplied meta-review passage identifies the committee and review process.

Appendix C. AI Usage

  • Claude Opus 4.7 generated chatbot prompts and helped develop and refine the analysis pipeline; authors manually verified outputs.Claude was also used for editorial tightening, spelling, grammar, and syntax, with suggestions inspected by the authors.Source: Appendix C · p. 18

Appendix D. Ethics Considerations

  • The audit used two dedicated accounts and minimal prompts in low-volume sessions, collecting no real-user data and recording only traffic and execution traces from controlled sessions.The authors characterize the method as strictly observational and report avoiding disruption to platform operations.Source: Appendix D · p. 18

Appendix E. Meta-Review

  • The program committee prepared the meta-review as part of the 2027 IEEE Symposium on Security and Privacy review process.Source: Appendix E · p. 18

E.1. Summary

The meta-review summary describes a measurement study of contextual leakage across general-purpose and domain-specific chatbot platforms. It highlights leaked conversational content, user identifiers, and the JavaScript mechanisms involved.

E.1. Summary

  • The study combines Puppeteer network interception and VisibleV8 instrumentation to link leaks to user interactions and JavaScript retrieval mechanisms across general-purpose and domain-specific platforms.Source: §E.1 · p. 18
  • The analysis finds leaked conversation titles, verbatim prompts, and registration email addresses, which can associate conversational content with user identities.Source: §E.1 · p. 18
  • Many leaks arise through generic metadata collection, including dynamically generated page titles containing conversation summaries.Source: §E.1 · p. 18

E.2. Scientific Contributions

The meta-review characterizes the paper as identifying an impactful vulnerability and advancing an established field.

E.2. Scientific Contributions

  • The meta-review identifies the paper’s contribution as an impactful vulnerability.Source: §E.2 · p. 18
  • The meta-review describes the work as a valuable step forward in an established field.Source: §E.2 · p. 18

E.3. Reasons for Acceptance

The reviewers valued the paper’s characterization of contextual leakage in chatbot interfaces, including sensitive-domain platforms, and its account of when and how leaks occur. They also raised concerns about the limited number of measurements and sessions.

E.3. Reasons for Acceptance

  • Attributing leaks to specific user interactions and identifying trackers’ DOM-read primitives provides insight that network-only analyses cannot.The reviewers described this as a novel characterization of contextual leakage in AI chatbots.Source: §E.3 · p. 18
  • The study of health, legal, and education platforms offers insight into privacy implications for sensitive-domain chatbot applications.Source: §E.3 · p. 18

E.4. Noteworthy Concerns

  • Reviewers raised reproducibility concerns because the paper relies on a limited number of measurements and sessions.Source: §E.4 · p. 18
Loading 2610.37467v1…