PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

Jun Tang, Aleksandra Korolova, Xiaolong Bai, Xueqiang Wang, Xiaofeng Wang

arXiv:1709.02753v2cs.CRcs.CYcs.LG

TL;DR

Apple’s public disclosures left its privacy-budget parameters and management unclear, despite those choices being central to evaluating the deployment’s privacy protections. This paper systematically studies Apple’s configuration and budget-maintenance mechanisms, finding that daily and cumulative permitted privacy loss can greatly exceed per-datum loss.

  • Problem

    Apple’s public materials did not explain the privacy parameters governing permitted privacy loss, leaving a crucial aspect of its deployment unclear.

  • Method

    The paper systematically examines Apple’s configuration files, database tables, report generation, and privacy-budget maintenance functionality.

  • Results

    The deployment limits privacy loss per datum but permits daily loss above commonly acceptable values, with overall per-device loss potentially unbounded.

  • Takeaways & Limitations

    The authors call for Apple to make its privacy-preserving implementation public and make privacy-loss rates transparent and tunable by users.

  • Takeaways & Limitations

    Apple could weaken privacy guarantees without users’ knowledge or consent by changing privacy-loss parameters, rates, or budget keys.

Abstract

from arXiv · show

In June 2016, Apple announced that it will deploy differential privacy for some user data collection in order to ensure privacy of user data, even from Apple. The details of Apple's approach remained sparse. Although several patents have since appeared hinting at the algorithms that may be used to achieve differential privacy, they did not include a precise explanation of the approach taken to privacy parameter choice. Such choice and the overall approach to privacy budget use and management are key questions for understanding the privacy protections provided by any deployment of differential privacy. In this work, through a combination of experiments, static and dynamic code analysis of macOS Sierra (Version 10.12) implementation, we shed light on the choices Apple made for privacy budget management. We discover and describe Apple's set-up for differentially private data processing, including the overall data pipeline, the parameters used for differentially private perturbation of each piece of data, and the frequency with which such data is sent to Apple's servers. We find that although Apple's deployment ensures that the (differential) privacy loss per each datum submitted to its servers is $1$ or $2$, the overall privacy loss permitted by the system is significantly higher, as high as $16$ per day for the four initially announced applications of Emojis, New words, Deeplinks and Lookup Hints. Furthermore, Apple renews the privacy budget available every day, which leads to a possible privacy loss of 16 times the number of days since user opt-in to differentially private data collection for those four applications. We advocate that in order to claim the full benefits of differentially private data collection, Apple must give full transparency of its implementation, enable user choice in areas related to privacy loss, and set meaningful defaults on the privacy loss permitted.

1 INTRODUCTION

Apple’s differential-privacy deployment attracted interest because its privacy protections and budget management were largely undisclosed. This study systematically examines those choices and finds that per-datum limits do not prevent substantially larger cumulative privacy loss.

  • Research gap: Apple’s public materials disclosed little about its differential-privacy technology or privacy parameters, leaving permitted privacy loss unresolved.The authors identify privacy-parameter choice as their primary focus.
  • Contribution: The study is presented as the first systematic examination of privacy-budget use in Apple’s deployment, following unanswered calls for greater disclosure.EFF and academics had previously asked Apple to detail its privacy-budget use.
  • Privacy budget: Differential privacy quantifies privacy risk through privacy loss, with larger epsilon providing less privacy protection.The risk increase is tied to an individual’s data inclusion in algorithm inputs.
  • Privacy budget: Composition means multiple submissions can accumulate privacy loss, so deployment analysis must consider both per-datum loss and submission frequency.The paper connects this need to proposals for an “Epsilon Registry.”
  • Findings: The authors report that per-datum loss is limited to values typically used in the literature, while daily permitted loss exceeds values commonly considered acceptable and total device loss may be unbounded.This finding concerns the deployment’s cumulative privacy budget rather than only individual perturbations.

2 OVERVIEW

The study reconstructs macOS’s differential-privacy system by examining its daemon, framework, database, reports, and maintenance tasks. It focuses on how data are privatized, stored, selected, reported, and budgeted over time.

  • System components: The macOS system includes a differential-privacy framework, the dprivacyd daemon, database storage, report files, and Console messages.The framework handles privatization, budget management, database updates, and report creation; the daemon coordinates the system.
  • System components: After opt-in, dprivacyd creates the DifferentialPrivacy database and accepts work for differentially private collection.The database is created under /var/db/DifferentialPrivacy.
  • Scope: The study examines Apple’s announced applications, but its experiments reliably trigger differential-privacy activity mainly for new words and emojis.The authors therefore base their conclusions on those applications.
  • Data pipeline: Emoji and previously unseen new-word data are privatized and added, with metadata, to corresponding database tables.Every emoji is stored, whereas only previously unseen words are privatized and stored.
  • Data pipeline: A periodic ReportGenerator selects database records, writes them to report files, transmits them to Apple, and marks selected rows for later deletion.Other maintenance tasks also delete database records and report files.
  • Study questions: The analysis asks which privacy parameters are used, how records and reports are scheduled, how many records are submitted, and whether total loss is bounded or alterable.The authors report that the first three answers depend on configuration files and their use by the framework.

3 SYSTEM’S DETAILS

macOS maps data-specific keys to shared properties and budget keys, then applies configured privatization algorithms and privacy parameters. Separate budget properties determine replenishment timing and amounts.

  • Database: ZOBHRECORD stores privatized emoji records, while ZCMSRECORD stores privatized new-word records.Every typed emoji is privatized; only words not previously typed are privatized and stored.
  • Database: The ZPRIVACYBUDGETRECORD table contains seven entries, and ZBALANCE stores each application’s currently available integer privacy budget.Entries cover NewWords, Emoji, AppDeepLink, Search, health, default, and testBudget.
  • Configuration overview: The key-configuration examples and tables expose mappings among data types, properties, algorithms, privacy parameters, and budget names.The figures and tables document the configuration structure used by the daemon.
  • Configuration mappings: Each KeyName maps to a PropertiesName, and multiple language- or region-specific keys can share one property category.New-word keys map to NewWords, while emoji keys map to TermFrequency.
  • Privatization: Each property specifies a privatization algorithm and privacy parameter, including OneBitHistogram with 1 for TermFrequency and CountMedianSketch with 2 for NewWords.Additional algorithm settings are specified separately in algorithmparameters.plist.
  • Budget mappings: Properties also map to BudgetKeyNames, grouping related data such as all new words or all emojis under shared budget keys.The configuration contains seven distinct BudgetKeyNames.
  • Budget properties: Budget properties specify SessionSeconds and SessionAmount for each BudgetKeyName, with intervals generally measured in days and amounts ranging from 1 to 10.Health uses 604800 seconds; other listed budget keys use 86400 seconds.

4 PRIVACY LOSS FINDINGS

Apple’s implementation limits privacy loss per datum through configured parameters, but budget replenishment and rollover permit substantially larger cumulative losses. For the four initially announced applications, the permitted loss can reach 16 per day and 16 times the days since opt-in.

  • 4.1 Each Datum’s Privatization: PrivacyParameter specifies the privacy loss used to privatize each datum before database insertion.
  • 4.1 Each Datum’s Privatization: Emoji privatization uses OneBitHistogram with PrivacyParameter 1, while English and Russian new words use CountMedianSketch with PrivacyParameter 2.
  • 4.2 Privacy Budget Management: Every 18 hours, ReportGenerator selects records subject to per-KeyName limits and the currently available BudgetKeyName balance.
  • 4.2 Privacy Budget Management: PrivacyBudgetMaintenance increases each applicable budget balance by SessionAmount every 24 hours, multiplying the increase by elapsed days after sleep.
  • 4.2.3 Total Privacy Loss Permitted: After 20 unused days, typing 20 emojis across 10 languages produces privacy losses of 10 on each of two reporting days.
  • 4.2.3 Total Privacy Loss Permitted: 16 is the permitted daily privacy loss for NewWords, AppDeepLink, Search, and Emoji combined, while rollover permits 16 times the days since opt-in.DeepLink functionality appears absent on MacOS 10.12.3, where the actual permitted loss is as high as 6 per day; iOS 10.1.1 reaches 14 per day.

5 DISCUSSION

The discussion describes storage and report maintenance, safeguards against configuration abuse, and changes between MacOS 10.12.1 and 10.12.3. It also notes related differences between the MacOS and iOS implementations.

  • 5.1 Database and Report File Management: StorageCulling, StorageMaintenance, and ReportFilesMaintenance delete submitted, stale, oversized, and older report data on periodic schedules.The schedules are 24 hours, 12 hours, and 24 hours respectively; report files older than a month are removed.
  • 5.2 Ease of Altering System’s Performance: Apple makes configuration changes difficult by requiring System Integrity Protection to be disabled on MacOS, with no configuration-change method found on iOS.
  • 5.2 Ease of Altering System’s Performance: Runtime safeguards cap PrivacyParameter at 2, default higher values to 1, and cap SessionAmount at 40.Records affected by an over-limit PrivacyParameter receive Submission Priority 99999 and are effectively excluded from reports.
  • 5.3 Configuration Differences between MacOS versions: The study observed configuration-file changes between MacOS 10.12.1 and 10.12.3.
  • 5.3 Configuration Differences between MacOS versions: MacOS 10.12.3 raises NewWords SessionAmount from 1 to 2 and introduces health-related budgets and properties.The update also adopts SubmissionPriority and introduces budget or property entries associated with new applications and abuse protections.
  • 5.4 Comparison with iOS: The iOS implementation follows the same principles as MacOS, but iOS reports contain more metrics and report files are deleted faster.DeepLink records were abundant on iOS but could not be triggered on MacOS; deletion was observed after 7 versus 30 days.

6 CONCLUSIONS AND FUTURE WORK

The authors identify substantial shortcomings in Apple’s differential privacy deployment, especially opaque and potentially cumulative privacy loss, and call for greater transparency and user control.

  • Apple’s privacy loss permitted by the system is not explained publicly and requires significant reverse-engineering.The authors argue this prevents users from making informed opt-in decisions based on quantifiable risk.
  • The implementation may allow Apple to weaken privacy guarantees by changing per-datum loss, loss rates, or BudgetKeyName assignments without user knowledge or consent.
  • A daily privacy loss of 16 is considered significantly higher than commonly reasonable values, while daily replenishment can make total device-level loss grow by orders of magnitude.
  • The database and report structure can reveal users’ features, language, and potentially geographic or keyboard preferences, while new-word updates may expose whether a particular word was used.
  • The authors call for public implementation details and privacy-loss rates that are transparent and tunable by users.
  • Future Work: Future work includes studying source–aggregator decoupling, verifying claimed differential privacy implementations, and improving public communication of privacy guarantees.

A.1 Code Support for Findings

The appendix describes the Objective-C naming and syntax conventions used when interpreting Hopper-disassembled framework code.

  • Objective-C analysis uses leading - or + to distinguish instance and class methods, with brackets enclosing function and argument names.Arguments are separated by colons in the disassembled code.

A.1.1 Checking PrivacyParameter corresponds to the privacy parameter used in a datum’s privatization.

The authors validate that PrivacyParameter is the parameter used in Apple’s per-datum privatization code for new words and emojis.

  • PrivacyParameter determines the _DPBiasedCoin probability for new-word privatization as 1.0/(exp(PrivacyParameter) + 1.0).Runtime LLDB observations connect the parameter to the privatization function, with analogous analysis applied to emoji randomization.

A.1.2 Checking that SessionAmount controls the daily budget increase value.

The appendix verifies that SessionAmount controls the daily increase of Apple’s privacy budget, alongside hardcoded time-interval constants used by the framework.

  • SessionAmount is used as the daily budget increase value in _DPPrivacyBudget updates.The update function multiplies the interval budget value by the number of days since the last update.

A.1.3 Hard-coded values.

The implementation contains hard-coded values and exhibits several puzzling behaviors, including unexplained budget changes and occasional automatic termination of data collection.

  • Observed implementation behavior: The privacy budget increased dramatically twice during six months and was reset to a date-based value, for an unknown reason.The authors suggest that an Apple server may have triggered the change remotely.
  • Observed implementation behavior: The daemon occasionally stopped data collection and unchecked the Analytics Sharing opt-in boxes.This effect occurred several times during six months but could not be reliably reproduced.
  • Hard-coded values: The implementation includes hard-coded variables and code for fetching, randomizing, and reporting differentially private records.Figures show report files, record-fetch code, new-word and emoji randomization code, and other hard-coded variables.
  • Hard-coded values: The code hard-codes epsilonMAX as 2 and the number of seconds in one day.These values appear in the implementation code.
  • Hard-coded values: Code evidence links SessionAmount to intervalBudgetValue and intervalBudgetValue to the daily budget increase value.The cited figures document these implementation relationships.
Loading 1709.02753v2…