PaperlayerView exact revision on arXiv

Source-linked AI summary

PrivAudit: A Dual-Lens Auditing Framework for Website Privacy Practices under the CCPA

Mohamed Moustafa Dawoud, Riya Aggarwal, Likith Rahul Krishnamurthy, Ram Sundara Raman

arXiv:2609.09697v1cs.CR

TL;DR

Prior enforcement and research are largely case-specific, and scalable methods to assess CCPA policy disclosures alongside cookie practices remain limited. PrivAudit uses expert-guided LLM analysis of CCPA-grounded policies and browser measurements of cookie behavior under varied privacy configurations, linking both at the website level. CCPA-Subject websites more often describe opt-out mechanisms (77% vs. 57%); almost half of cookies involved third-party writes.

  • Problem

    Prior enforcement and research are largely case-specific, and scalable methods to assess CCPA policy disclosures alongside cookie practices remain limited.

    Source: §1 · pp. 1–2

  • Method

    PrivAudit uses expert-guided LLM analysis of CCPA-grounded policies and browser measurements of cookie behavior under varied privacy configurations, linking both at the website level.

    Source: §1 · p. 2

  • Results

    CCPA-Subject websites more often describe opt-out mechanisms (77% vs. 57%); almost half of cookies involved third-party writes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Linked disclosure and cookie evidence yields scalable signals to identify websites and practices for deeper manual review, but the signals alone do not establish legal violations.

    Source: §1 · p. 2

    Measurements exclude cookies observable only through authenticated or interaction-gated flows, such as login or scrolling.

    Source: §5 · p. 13

Abstract

from arXiv · show

Five years after the enforcement of the California Consumer Privacy Act (CCPA), understanding how website privacy practices evolve at scale in response to regulation remains a key challenge for both researchers and regulators. Prior work and regulatory efforts have focused on manual and case-specific enforcement, but there remain no scalable approaches to systematically audit two key user-facing facets of websites that are crucial signals for the CCPA: privacy disclosures and front-end user tracking behavior. In this paper, we present PrivAudit, an automated auditing framework that adopts a dual-lens approach to capture: (1) privacy disclosures through large language model-based analysis of privacy policies grounded in CCPA provisions, and (2) user-observable data collection behavior through automated browser measurements of cookie writes under diverse privacy configurations. We apply PrivAudit to 998 websites and report two broad findings. The law is associated with stronger privacy disclosures: CCPA-subject policies are more likely to disclose opt-out mechanisms, data-sharing practices, and user rights. On the other hand, cookie-based tracking remains pervasive, with both CCPA-subject and not-subject websites setting a total of 6,392 targeting cookies, 49% of which are third-party writes. Moreover, cookies show limited-to-moderate responsiveness to privacy signals and consent choices, even when websites claim to honor them in their disclosures. Our results highlight the need for multi-layered and scalable auditing approaches that combine policy analysis with behavioral evidence. PrivAudit can support these auditing workflows at scale by generating actionable signals and patterns for further manual review. We open-source PrivAudit and are engaging with regulators to support auditing in practice.

1 Introduction

Scalable CCPA auditing remains difficult because enforcement and prior studies are case-specific, while policy disclosures and cookie behavior have not been systematically assessed together. PrivAudit combines these lenses and finds stronger disclosures alongside persistent tracking and disclosure–behavior gaps.

1 Introduction

  • Prior enforcement and academic work examine specific companies or compliance questions, leaving scalable assessment of policy disclosures and cookie practices under the CCPA limited.The two dimensions offer complementary, externally observable signals relevant to regulators’ multi-layered audits.Source: §1 · pp. 1–2
  • PrivAudit combines expert-guided LLM evaluation of CCPA-grounded policy disclosures with browser-based cookie measurements across privacy configurations, then links both views at the website level.The policy analysis was validated against human labels with 97% agreement, and Monte Carlo analysis tested robustness to LLM error.Source: §1 · p. 2
  • CCPA-Subject websites more often disclose opt-out mechanisms (77% vs. 57%), access rights (85% vs. 68%), deletion rights (86% vs. 69%), and GPC support (29% vs. 13%).Only a small fraction of policies detail specific practices such as cookie consent behavior or honoring privacy signals.Source: §1 · p. 2
  • 6,392 Targeting cookies were set across CCPA-Subject and CCPA-Not-Subject websites, almost half through third-party writes, with privacy signals and consent choices reducing but not eliminating tracking.Among CCPA-Subject websites, GPC reduced total tracking cookies by 55% and third-party tracking cookies by 41%.Source: §1 · p. 2
  • Among CCPA-Subject websites whose policies claim to honor GPC, 39% show no reduction in Targeting cookies, while 47% show no reduction in third-party Targeting cookies.These patterns are scalable auditing signals for further manual review, not evidence of legal violations.Source: §1 · p. 2
  • PrivAudit’s outputs can help identify websites, sectors, and practices for deeper investigation, and the framework is extensible to state laws with similar disclosure and opt-out requirements.The authors open-sourced the framework and datasets and are engaging with California regulators about practical auditing uses.Source: §1 · pp. 2–3

2 Background and Related Work

Prior CCPA research has examined separate aspects of compliance, while work on policy analysis and tracking has developed complementary tools and findings. PrivAudit distinguishes itself by jointly comparing CCPA-grounded disclosures with observed cookie behavior at the website level.

Background and Related Work

  • The CCPA uses an opt-out model and applies only to businesses meeting specified California, consumer-volume, revenue, or data-sharing thresholds.Prior CCPA studies examined opt-out link deployment and dark patterns, policy ambiguity, and related compliance questions.Source: §2 · p. 3
  • Earlier CCPA studies generally assessed one dimension, whereas PrivAudit jointly audits policy disclosures and front-end cookie behavior to surface potential mismatches.Examples include a policy denying data sales alongside third-party Targeting cookies at page load, or claimed GPC compliance without reduced tracking.Source: §2 · pp. 3–4
  • Audits of opt-out signal propagation do not establish that data collection stops; PrivAudit measures front-end cookies and scripts across six privacy configurations.The configurations include GPC, Do Not Track, third-party cookie blocking, ad blocking, and consent-banner interaction.Source: §2 · p. 3
  • Prior policy-analysis systems summarize or classify policies, test statutory clauses, or compare claims with behavior; PrivAudit uses a CCPA-specific rubric and links disclosures to cookie behavior.Related tools include PoliCheck, OVRSeen, PoliGraph, and DiffAudit.Source: §2 · p. 3
  • Research documents tracking that persists despite consent choices and interface designs that steer users toward acceptance or make rejection difficult.Cookie-based tracking also sits within a broader ecosystem that includes fingerprinting, evercookies, cookie synchronization, and identifier sharing.Source: §2 · pp. 3–4

3 Methods

PrivAudit combines policy analysis and browser-based cookie measurement to audit user-facing privacy practices at scale. Its workflow selects websites, evaluates CCPA-grounded policy disclosures, measures cookie behavior across privacy configurations, and applies statistical tests to group differences.

3 Methods

  • PrivAudit combines CCPA-grounded LLM policy analysis with browser-based cookie detection, using six privacy configurations to capture disclosures and cookie-setting practices.Measurements were collected from a California-based client in mid-2025.Source: §3 · p. 4

3.1 Website Selection

  • The study analyzes 998 sites with verified CCPA subjectivity—602 subject and 396 not subject—and separately runs the pipeline on 1,000 unlabeled popular websites.Only 915 known-subjectivity sites set cookies on homepage visits and enter the cookie analysis.Source: §3.1 · p. 5

3.2.1 Privacy Policy URL Extraction.

  • A two-stage automated pipeline locates privacy-policy pages using high-precision link keywords first, then a broader privacy-text and URL fallback, obtaining policy-related pages for all 998 sites.The analysis uses 933 combined policies; manual review found one additional international-focused page outside the study’s focus.Source: §3.2.1 · p. 5

3.2.2 LLM-Based Analysis of Privacy Policies.

  • Policies are segmented into passages and scored by LLMs on CCPA-grounded dimensions, with statutory mappings reviewed by an external privacy-law expert.The rubric covers rights completeness, mechanism usability, accuracy, disclosures, privacy claims, and mentions of laws and practices.Source: §3.2.2 · pp. 5–6

3.2.3 LLM Implementation and Validation.

  • GPT-4o outputs receive schema checks and manual verification; its scores matched human labels on 97% of a 100-policy sample, and 9 of 10 significant findings survived error perturbations in at least 99.8% of simulations.Across three models, boolean outputs had Fleiss’ κ=0.74 and ordinal scores had pairwise Spearman ρ=0.69–0.83.Source: §3.2.3 · p. 6

3.3.1 Automated Browsing and Cookie Extraction.

  • The crawler isolates sessions in clean Chromium instances, records cookie state before and after consent acceptance or rejection, and tests six privacy configurations.It detects consent banners automatically and manually checks unmatched sites; 239 websites displayed a banner, including 183 CCPA-Subject sites.Source: §3.3.1 · p. 7

3.3.2 Cookie Functionality Classification.

  • Multi-source name- and domain-based categorization labels 70.6% of cookies; validation matched the reference for 94% of 171 cookies, while errors skew toward false negatives, making tracking rates conservative lower bounds.At least one cookie was categorized for 877 websites and at least three-quarters for 658.Source: §3.3.2 · pp. 7–8

3.3.3 Third-Party Cookie Classification.

  • Script attribution identifies third-party writes by the initiating script’s eTLD+1, including scripts that write cookies scoped to the visited site; self-hosted or proxied scripts can therefore be missed.The conventional cookie-domain definition classified 4.3% as third-party, whereas script attribution captures the writing origin.Source: §3.3.3 · p. 8

4 Results

CCPA-Subject policies more often disclose user rights, but cookie tracking remains widespread across both subjectivity groups. Comparing disclosures with observed behavior reveals gaps even where policies claim to honor GPC or deny selling data.

4.1 Privacy Policy Findings

  • CCPA-Subject policies more often disclosed access rights (85.4% vs. 68.3%), deletion rights (85.9% vs. 69.4%), and opt-out instructions (77.1% vs. 57.1%).They also more often disclosed data retention details (64.6% vs. 49.2%), while offline collection disclosures were uncommon across both groups (26.0% overall).Source: §4.1 · pp. 9–10
  • Although 86.0% of policies mention cookies, fewer than 2% detail cookie consent behavior; more than 80% disclose third-party sharing, but explicit data-selling disclosures are less common.Explicit selling disclosures were 33.0% for CCPA-Subject policies versus 23.2% for CCPA-Not-Subject policies; the authors note sensitivity to residual LLM labeling error.Source: §4.1 · p. 10

4.2 Cookies Analysis

  • Of 19,119 analyzed cookies, 7,943 (41.5%) involved third-party writes, and 72.8% of websites set at least one; script attribution identified 7,729 cookies versus 823 by domain classification.Scripts can write cookies scoped to first-party domains, a pattern that can evade conventional browser-level third-party cookie blocking [11] [38] [83].Source: §4.2 · pp. 10–11
  • At default, both groups set a median of 3 Targeting and 3 Performance cookies per website; with GPC enabled, about 45% of CCPA-Subject sites still set Targeting cookies.GPC reduced Targeting cookies by 55.8% overall, while uBlock reduced them by 87%.Source: §4.2 · p. 10, Figure 3 · p. 11, Table 2 · p. 11
  • Ten third-party scripts account for 58.4% of script-set Targeting cookies; Google’s advertising and tag-management infrastructure alone accounts for 576 across 386 websites.Source: Table 3 · p. 11, §4.2 · p. 11

4.3 Auditing Disclosures vs. Tracking

  • Among 110 CCPA-Subject sites claiming to honor GPC and setting Targeting cookies in both configurations, 52.7% reduced them by less than 20%, and 39.1% showed no reduction.Third-party Targeting cookie reductions were also limited: 66.2% of sites showed less than a 20% reduction and 47.5% showed none.Source: §4.3 · p. 12
  • Among sites whose policies said they do not sell personal data, 50.8% nonetheless set third-party Targeting cookies at initial load, and 23.5% set five or more.Cookies alone do not constitute a “sale” under the CCPA, so this co-occurrence is a signal for manual review rather than proof of a sale.Source: §4.3 · p. 12
  • Of CCPA-Subject sites setting third-party Targeting cookies at initial load, 9.7% disclosed no data collection and 8.2% disclosed neither sale nor sharing; high-quality policies also did not correspond to lower tracking.Among 394 sites with high-quality policies, 43.7% set five or more Targeting cookies on first visit, with a median count identical to the population median.Source: §4.3 · p. 12

5 Discussion

The discussion interprets stronger disclosures beyond the CCPA’s formal scope alongside persistent cookie tracking. It argues for broader collection regulation and presents paired audits as a way to surface review signals, not replace legal analysis or enforcement.

5 Discussion

  • The authors hypothesize that CCPA-Not-Subject sites’ high-quality disclosures and CCPA references partly reflect uncertainty about applicability, which may encourage over-compliance and impose costs.They note that multiple state privacy laws with differing applicability criteria further fragment the compliance landscape.Source: §5 · p. 12
  • Because cookie writes are a precondition for practices the CCPA regulates, the authors argue that the law should cover granular collection such as cookies, as the GDPR does.They argue this could prevent initial collection, encourage transparency, and enable faster third-party auditing.Source: §5 · p. 12
  • Paired disclosure and tracking audits can surface signals for audit selection, public reporting, rulemaking, and enforcement priorities, but do not replace legal analysis or enforcement.The authors also describe PrivAudit as a potential self-auditing tool for website developers and privacy teams.Source: §5 · p. 12
  • Measurements cover cookies observable without authenticated or interaction-gated flows, and reflect a single mid-2025 crawl; the authors leave other tracking techniques and repeated crawls to future work.They also note that manual prompt engineering and validation remain necessary to reduce LLM hallucinations.Source: §5 · p. 13

A Generative AI Usage

  • The authors report using generative AI tools for grammar and readability and LLMs for policy evaluation, with all manuscript content written, reviewed, and verified by the authors.Source: A Generative AI Usage · p. 15

B Open Science

  • PrivAudit’s repository includes its crawler, policy scraper, LLM analysis materials, cookie categorization pipeline, datasets, and analysis scripts; some raw classification databases are excluded due to licensing.Source: B Open Science · pp. 15–16

C Ethical Considerations

  • Crawling used California-based clients, fewer than 10 requests per site, public website responses, and no interaction with user data beyond the researchers’ own.Internal pages were selected from site sitemaps or common public paths, and configuration crawls were round-robin.Source: C Ethical Considerations · p. 16

D Website Categorization

  • Website categorization treats only for-profits clearly meeting CCPA applicability criteria as CCPA-Subject; all other categories are CCPA-Not-Subject.Source: D Website Categorization · p. 16

E Scaling to Unlabeled Websites

PrivAudit’s unlabeled-site case study applies the policy-scoring and cookie-audit pipelines to a stratified sample of 1,000 popular sites. Disclosures are weaker than in the Known-Subjectivity list, while tracking and disclosure–behavior gaps remain substantial.

E Scaling to Unlabeled Websites

  • The case study draws 1,000 sites at random across four Tranco rank bands, after filtering for selected TLDs, active homepages, and exclusion from the Known-Subjectivity list.The same GPT-4o policy-scoring pipeline and six browser-native privacy configurations are used; unretrievable policies are outside the analysis.Source: E Scaling to Unlabeled Websites · p. 16
  • Figure 4 compares per-website cookie-count distributions across Initial, Accept, and Reject stages, separating CCPA-subject from not-subject sites by line style.Source: Figure 4 · p. 16
  • 43.6% of sites claiming not to sell data still set third-party Targeting cookies at initial load; 54% of sites presenting an opt-out mechanism also do so.The case study reports 7,244 cookies across 740 sites that set cookies, including 1,650 Targeting cookies; third-party scripts set 67% of all cookies and 79% of Targeting cookies.Source: E Scaling to Unlabeled Websites · p. 16
  • Only 72% of sites expose a usable privacy policy, and their completeness, usability, and accuracy scores are 1.65, 1.78, and 1.94 out of 3.Only 28% present a machine-detectable CCPA opt-out mechanism; access, deletion, and opt-out disclosures appear on 40%, 36%, and 52% of sites, respectively.Source: E Scaling to Unlabeled Websites · p. 16
  • uBlock Origin removes 96% of third-party Targeting cookies and GPC reduces them by 22% (p<0.001), while DNT, third-party-cookie blocking, and Consent-O-Matic show no measurable effect.Source: E Scaling to Unlabeled Websites · p. 16

F Cookie Writing by Consent Flow

Consent interactions produce only limited changes in cookie counts. Accepting increases counts, while rejecting yields a small decrease for subject sites and no change for not-subject sites.

F Cookie Writing by Consent Flow

G Cookie Security and Privacy Characteristics

Cookie security attributes are uncommon, and many cookies are persistent or accessible to JavaScript. The audit also finds plaintext location and network data that often persists under privacy-enhancing configurations.

G Cookie Security and Privacy Characteristics

  • Only 25.4% of cookies use Secure, including 29.1% on CCPA-Subject and 19.8% on CCPA-Not-Subject sites; Targeting cookies are at 20.2%.The group difference for Secure adoption is significant (p<0.001, V=0.10).Source: G Cookie Security and Privacy… · p. 17
  • 57.1% of cookies are persistent, while Targeting cookies have a median lifespan of almost a year and only 39.5% are session cookies.HttpOnly adoption is 5.1% overall, and SameSite adoption is 16.1% with no practically meaningful group difference.Source: G Cookie Security and Privacy… · p. 17
  • The audit found location-related names across 171 unique cookie identifiers, with 7 latitude/longitude instances, 6 ZIP codes, and 226 IP addresses; plaintext-only analysis makes this a lower bound.Among the IP addresses, 162 appear in Functional cookies and 28 in Strictly Necessary cookies.Source: G Cookie Security and Privacy… · p. 17
  • GPC leaves ZIP-code instances at 6 and coordinates rise from 7 to 8; IP-address instances fall 15% under GPC and 14% under uBlock Origin.The authors report 226 baseline IP-address instances, compared with 193 under GPC and 195 under uBlock Origin.Source: G Cookie Security and Privacy… · p. 17

H Delegation Through Nested Scripts

Cookie writes can pass through deeply nested third-party script chains, including chains that place cookies under a publisher’s domain. A small set of platforms accounts for much of script-set targeting activity.

H Delegation Through Nested Scripts

  • Cookie-setting traces average 8.2 function frames and typically reach depths of 40–50, with extreme delegation chains exceeding 500 layers.A representative _ga write begins with a site’s tag manager, loads Google Tag Manager and Google Analytics scripts, and ends with a first-party-scoped cookie.Source: H Delegation Through Nested Scripts · p. 17
  • Just 10 scripts from Google, Microsoft, Meta, and Adobe account for 58% of script-set targeting cookies.The authors note that more consistent opt-out handling by major tag managers, advertising scripts, or analytics SDKs could propagate across many websites.Source: H Delegation Through Nested Scripts · p. 17
  • Script-set first-party tracking cookies are reported as more resistant to GPC than in the prior work discussed by the authors.Source: H Delegation Through Nested Scripts · pp. 17–18
  • Figure 5 plots ECDFs of non-session cookie longevity by cookie type under the Default configuration, distinguishing subject and not-subject sites by line style.Source: Figure 5 · p. 18

I Cookie Longevity

Targeting and Performance cookies tend to persist longer than other cookie types, and cookie lifetimes are similar across CCPA-subject and not-subject websites.

I Cookie Longevity

  • About 50% of targeting cookies have lifetimes of almost a year.Targeting and Performance cookies generally tend to be more long-lived.Source: I Cookie Longevity · p. 18
  • 23.5% of cookies last 0–1 days, while 32.7% last 1–2 years.Short-lived cookies are dominated by Strictly Necessary and Functional cookies.Source: I Cookie Longevity · p. 18
  • Both CCPA-Subject and CCPA-Not-Subject websites have a median cookie lifetime of roughly one year.Source: I Cookie Longevity · p. 18

J Policy and Cookie Analysis by Industry Sector

Industry patterns differ in both privacy disclosures and cookie deployment: for-profit websites show more disclosure and targeting activity, while Government and nonprofit websites tend to show less.

J Policy and Cookie Analysis by Industry Sector

K Internal-Page vs. Homepage Tracking

The internal-page validation finds fewer third-party Targeting cookies on internal pages than on homepages, while homepages capture most internal-page trackers.

K Internal-Page vs. Homepage Tracking

  • Across all six privacy configurations, internal pages set significantly fewer third-party Targeting cookies than homepages.The crawl sampled up to five same-site internal pages per website, loading each in a fresh, isolated browser context.Source: K Internal-Page vs. Homepage Tracking · pp. 18–19
  • Homepages capture 87–90% of a typical internal page’s distinct third-party Targeting trackers and 78–83% of the union across internal pages.Source: K Internal-Page vs. Homepage Tracking · p. 19
  • The stricter union measure includes a tail of additional trackers across internal pages, but those third parties are largely non-deterministic between visits.The per-page measure better reflects representativeness; under uBlock Origin, targeting cookies are nearly eliminated on both page types, making the paired difference near zero by construction.Source: K Internal-Page vs. Homepage Tracking · p. 19

L Firm-Size Controls and Multiple-Testing Correction

Disclosure differences persist after adjusting for firm revenue and ad-revenue reliance and after restricting comparisons to large organizations; the main policy findings also survive stricter multiple-testing correction.

L Firm-Size Controls and Multiple-Testing Correction

  • CCPA subjectivity remains a strong positive predictor of opt-out, access, and deletion disclosures after adjusting for revenue and ad-revenue reliance (all p<0.001).Company revenue is not a significant predictor of opt-out disclosure (p=0.55) and is at most weakly associated with the other disclosures.Source: L Firm-Size Controls and… · p. 19
  • Among organizations above the $25M threshold, CCPA-Subject firms disclose opt-out mechanisms more often than large exempt organizations (77% vs. 46%), with similar gaps for access and deletion rights (85% vs. 60%).The observational comparison cannot establish strict causation.Source: L Firm-Size Controls and… · p. 19

M Multiple Hypothesis Corrections

N Legal Basis and Mapping

The authors map each LLM prompt field to CCPA/CPRA provisions and validate that legal alignment through expert review. The prompt turns that mapping into a structured, evidence-grounded schema for policy analysis.

N Legal Basis and Mapping

  • Each schema field is mapped to a CCPA/CPRA provision, and an external privacy-law expert reviewed the mapping and suggested interpretive refinements.The review validates the rubric’s legal alignment, not the LLM’s application of it.Source: N Legal Basis and Mapping · pp. 19–20
  • Table 8 reports raw and adjusted p-values across 25 Subject-versus-Not-Subject tests, with conclusions based on Holm-adjusted significance.The tests cover 16 policy and 9 cookie comparisons.Source: Table 8 · p. 20

O Full LLM Prompt

  • The prompt returns a fixed JSON schema with statute-aligned fields and requires textual justifications for non-trivial flags and scores.When a policy does not explicitly support a behavioral flag, the prompt requires an “unspecified” value with an explanation rather than a silent default.Source: O Full LLM Prompt · pp. 20–21
  • The schema distinguishes online from offline collection, including web and app tracking versus in-store purchases and other physical-world collection.Absent practices are recorded as “Not mentioned.”Source: O Full LLM Prompt · p. 20
  • Policy quality is scored from 0–3 on completeness, usability, and accuracy, covering rights, actionable mechanisms, and legal terminology.The rubric assesses whether core rights are explained, how easy their mechanisms are to use, and whether statutory terms are accurate and current.Source: O Full LLM Prompt · p. 20
  • Disclosure fields record whether policies explicitly cover statutory items, while behavioral claims separately capture stated GPC recognition and cookie-consent practices.Disclosure fields include data collection, sharing, purposes, retention, and consumer rights; behavioral claims use true, false, or “unspecified.”Source: O Full LLM Prompt · p. 21
  • The schema records data-transfer practices, cookie and CCPA/CPRA mentions, and contact methods for opt-out, access, and deletion requests.It also distinguishes California-specific legal acknowledgments from policies that do not name those frameworks.Source: O Full LLM Prompt · pp. 21, 22, 25
Loading 2609.09697v1…