PaperlayerView exact revision on arXiv

Source-linked AI summary

Knock Knock, Who's There? Membership Inference on Aggregate Location Data

Apostolos Pyrgelis, Carmela Troncoso, Emiliano De Cristofaro

arXiv:1708.06145v2cs.CR

TL;DR

Aggregate location data can reveal whether a target user contributed to statistics associated with sensitive groups, locations, or timeframes. The paper formalizes membership inference as a distinguishability game, implements it with machine learning, and evaluates raw and differentially private aggregates. Attacks can be highly effective under informative prior knowledge, while differential privacy trades utility for protection and remains vulnerable to adversaries trained on noisy aggregates.

  • Problem

    Aggregate location statistics may expose whether individuals belong to sensitive groups or contributed during sensitive locations and timeframes.

  • Method

    The paper formalizes membership inference as a distinguishability game and uses a machine-learning classifier to infer target membership from aggregate location data.

  • Results

    Membership inference can reach up to 0.83 AUC with 100 users per group and up to 1.0 AUC with 9,500 users per group; differential privacy reduces attacks but noisy-training adversaries can reduce protection by up to 83%.

  • Takeaways & Limitations

    Attack effectiveness varies with adversarial knowledge, data characteristics, group size, and aggregation timeframe, while defenses trade privacy protection against utility.

Abstract

from arXiv · show

Aggregate location data is often used to support smart services and applications, e.g., generating live traffic maps or predicting visits to businesses. In this paper, we present the first study on the feasibility of membership inference attacks on aggregate location time-series. We introduce a game-based definition of the adversarial task, and cast it as a classification problem where machine learning can be used to distinguish whether or not a target user is part of the aggregates. We empirically evaluate the power of these attacks on both raw and differentially private aggregates using two mobility datasets. We find that membership inference is a serious privacy threat, and show how its effectiveness depends on the adversary's prior knowledge, the characteristics of the underlying location data, as well as the number of users and the timeframe on which aggregation is performed. Although differentially private mechanisms can indeed reduce the extent of the attacks, they also yield a significant loss in utility. Moreover, a strategic adversary mimicking the behavior of the defense mechanism can greatly limit the protection they provide. Overall, our work presents a novel methodology geared to evaluate membership inference on aggregate location data in real-world settings and can be used by providers to assess the quality of privacy protection before data release or by regulators to detect violations.

I. INTRODUCTION

The paper studies whether aggregate location time-series can reveal a target user’s membership, addressing a previously unexamined privacy problem. It formalizes and evaluates this threat, including under differential privacy.

  • Motivation: Aggregate location statistics can still threaten the privacy of individuals included in them.Membership inference asks whether a target user’s location data contributed to the released aggregates.
  • Motivation: Membership inference is especially sensitive when aggregates concern groups, places, or timeframes associated with private characteristics.Inferring inclusion in aggregates of Alzheimer’s patients, sensitive locations, or sensitive periods can reveal private information.
  • Implications: The methodology can help providers assess privacy protection before release and regulators investigate possible violations.The paper positions membership inference as an evaluation tool for aggregate location data practices.
  • Approach: The paper introduces a distinguishability game and a machine-learning classifier to infer whether a target user is included in unseen aggregates.The classifier is trained using adversarial prior knowledge, including past locations and aggregates from groups that include or exclude the target.
  • Results: Up to 0.83 AUC is achieved with 100 users per aggregation group, while prior information about groups enables up to 1.0 AUC with 9,500 users per group.Weaker adversarial knowledge reduces effectiveness but still produces non-negligible privacy leakage.
  • Results: Differentially private defenses generally reduce inference effectiveness but incur a non-negligible utility loss, and noisy-training adversaries can reduce protection by up to 83%.The study evaluates these trade-offs using two real-world mobility datasets.

III. DEFINING MEMBERSHIP INFERENCE ON AGGREGATE LOCATIONS

The paper defines membership inference on aggregate location time-series as distinguishing whether a target user contributed to released counts across regions and time intervals. A parameterized game formalizes the challenge faced by an adversary with prior knowledge.

  • Aggregate data: Aggregate location data records the number of users in regions of interest during specified time intervals.For example, an aggregate may report the number of taxis in a region during one hour.
  • Representation: Each user’s location time-series is represented as a binary region-by-time matrix, while all users’ data form a three-dimensional matrix.The aggregate over a user subset is a region-by-time matrix whose entries count users present in each region and interval.
  • Game definition: The distinguishability game asks whether the target user’s data is included in an aggregate formed from a group of size m over inference period TI.The game is parameterized by the universe of users U, group size m, and inference period TI.
  • Game definition: The challenger randomly creates either a target-inclusive or target-exclusive aggregate, and the adversary guesses which case occurred.The resulting aggregate matrix over TI is returned to the adversary, who wins when her guess matches the hidden bit.
  • Adversary interface: The distinguishing function receives the target, challenge aggregate, game parameters, and adversarial prior knowledge, without requiring knowledge of every user in U.The adversary only needs to know that the target belongs to the possible-user universe.

IV. METHODOLOGY

The methodology varies adversarial prior knowledge to evaluate membership inference in the distinguishability game. It considers direct location knowledge and knowledge of target participation in past aggregate groups.

  • Framework: The evaluation builds adversarial prior knowledge, instantiates the distinguishing function, and measures inference performance.The framework supports adversaries of variable strength through different priors.
  • Subset of Locations: Under the subset-of-locations prior, the adversary knows the actual locations of a subset of users including the target during the inference period.The observation and inference periods coincide, and the known-user fraction is modeled as |Y| = α · |U|.
  • Participation in Past Groups: Under the past-groups prior, the adversary knows aggregates from β groups observed during a period disjoint from inference and knows whether the target belonged to each group.For each group, the prior includes the observation-period aggregate and the target’s membership indicator.
  • Participation in Past Groups: The past-groups prior includes same-groups and different-groups variants depending on whether observed groups are reused during inference.The same-groups variant models stable groups with continuous release, whereas the different-groups variant is less restrictive.

B. Distinguishing Function

The paper implements the adversary’s distinguishing function as a supervised binary classifier and evaluates it with AUC-based privacy loss. Experiments use two mobility datasets with distinct characteristics.

  • Distinguishing Function: The distinguishing classifier predicts whether the target’s data is present in a challenge aggregate.It is trained on observations contained in the adversary’s prior knowledge.
  • Evaluation: AUC summarizes classifier performance across discrimination thresholds in the distinguishability game.The ROC curve is derived from true-positive and false-positive rates computed from the adversary’s guesses.
  • Evaluation: Privacy loss measures improvement over random guessing, whose baseline is AUC = 0.5.The resulting privacy-loss value captures the adversary’s advantage in distinguishing target membership.
  • Datasets: The experiments use TFL and SFC mobility datasets with different mobility characteristics.TFL data comes from public transport, while SFC data comes from San Francisco taxis; both support aggregate location evaluation.

B. Experimental Setup

The experiments instantiate membership inference as a machine-learning classification pipeline that aggregates user groups, extracts time-series statistics, and classifies unseen aggregates as including or excluding the target user. They evaluate multiple classifiers and attack users across mobility profiles.

  • Classification: The attack trains on balanced aggregates from groups that include or exclude the target user, then predicts membership on previously unseen balanced aggregates.Training data comes from the adversary’s prior knowledge.
  • Pipeline: The pipeline comprises aggregation, feature extraction, and classification, with phase details varying according to the adversary’s prior knowledge.The dataset contains labeled aggregates for groups including and excluding the target user.
  • Feature Extraction: Feature extraction computes variance, minimum, maximum, median, mean, standard deviation, and sums for each region of interest’s time series.These statistics are supplied as classifier inputs.
  • Classifiers: The evaluation uses Logistic Regression, k-nearest neighbors, Random Forests, and a Multi-Layer Perceptron as distinguishing classifiers.The Random Forest uses 30 trees, while the MLP has one hidden layer with 200 nodes.
  • Feature Processing: Feature extraction yields 4081 features for TFL and 707 for SFC before Recursive Feature Elimination reduces dimensionality to the number of samples.MLP inputs are standardized to mean 0 and variance 1; other classifiers receive features in their original form.
  • Evaluation: Attacks are performed against 150 users sampled equally from high, mild, and somewhat mobility profiles.The raw-aggregate evaluation compares classifier performance across the different adversarial priors.

A. Subset of Locations

Membership inference remains effective on aggregate location data, especially with small groups, regular mobility, or strong adversarial priors. Larger groups and weaker priors generally reduce attack performance and privacy loss, but do not eliminate leakage.

  • Subset of Locations: With m = 10, LR and RF achieve mean AUC scores of 0.97 and 0.99 on TFL, showing highly effective inference for small groups.Users’ contributions are significant when groups are small.
  • Subset of Locations: RF achieves 0.94 mean AUC for groups of 50 TFL users and 0.83 for 100, while groups of 500 or 1,000 approach the random-guess baseline.Even with 1,000 users, 60% of targets have AUC above 0.6.
  • Subset of Locations: TFL mean privacy loss is 0.99 for m = 10, 0.89 for 50, and 0.68 for 100, decreasing as aggregation groups grow.Larger groups improve privacy, but loss remains significant.
  • Subset of Locations: On SFC, mean AUC falls from 0.9 for groups of 10 cabs to 0.72 or 0.7 for groups of 50, depending on the classifier.For groups of 100, the corresponding mean AUC scores are 0.68 and 0.67.
  • Subset of Locations: At m = 100, privacy loss is 0.68 on TFL versus 0.44 on SFC, indicating that membership inference is easier on sparse data.SFC privacy loss is 0.86 at m = 10 and remains 0.44 at m = 100.
  • Different Groups than Released: When groups differ between observation and inference, TFL RF mean AUC is 0.89 for 50 commuters and 0.78 for 100, but performance approaches baseline at 1,000.Privacy loss follows the same broad pattern, reaching 0.17 at 1,000 and 0.22 at 9,500 commuters.

C. Length of Inference Period

Shorter inference periods make membership inference harder because they provide less mobility information, while temporal regularity can substantially affect attack performance. These effects differ between regular commuter traces and demand-driven cab traces.

  • Experimental setup: The experiments examine one-week, one-day, and eight-hour inference periods, with weekday-versus-weekend comparisons for the shorter periods.The analysis uses the same-groups prior and fixes group sizes to 1,000 TFL commuters and 100 SFC cabs.
  • Inference-period length and temporal semantics: Mean AUC on TFL is 0.97 for Monday training/testing and 0.8 for Saturday, reflecting stronger inference on regular weekday mobility.The difference is attributed to commuters’ regular patterns during the week.
  • Inference-period length and temporal semantics: AUC decreases from 0.91 on Monday 8am–4pm to 0.72 on Saturday during the same timeframe.The comparison uses an 8-hour inference period.
  • Inference-period length and temporal semantics: On SFC, mean AUC drops from 0.62 to 0.54 when the inference period shrinks from one week to one day.Cab movements lack regularity because they are driven by client demand.
  • Privacy loss: On TFL, privacy loss averages 0.98 for a one-week inference period and declines as the period becomes shorter.Mean privacy loss is also higher on Mondays than Saturdays for 24-hour and 8-hour periods.

D. Raw Aggregates Evaluation – Take-Aways

Raw aggregate location time-series enable substantial membership inference, especially when adversaries possess informative prior knowledge. Attack power varies with data regularity, sparsity, aggregation-group size, and inference-period characteristics.

  • Overall findings: A machine-learning classifier can infer membership in previously unseen aggregates from features extracted from known aggregate location time-series.The evaluation reports significant risks across different datasets and adversarial priors.
  • Adversarial prior knowledge: Membership inference is very successful when the adversary knows target-containing user locations or past aggregates for the same released groups.With different prior groups, privacy leakage is relatively small but remains non-negligible.
  • Data characteristics: Privacy leakage is generally larger for commuter mobility data than for cab traces because regularity and sparse location signals ease inference.The datasets are TFL and SFC, respectively.
  • Aggregation and inference periods: Small aggregation groups make membership inference highly successful, while larger groups generally provide more privacy.TFL remains vulnerable at large group sizes because commuter mobility is regular.
  • Aggregation and inference periods: Longer inference periods and favorable time semantics make attacks easier because more information is available for extracting mobility patterns.The paper also reports that inference-period timing plays an important role.

VII. EVALUATING DP DEFENSES

The paper evaluates differential-privacy defenses for aggregate location time-series, including standard noise mechanisms and Fourier-based perturbation methods. These mechanisms limit dependence on any one user’s data, with privacy guarantees determined by sensitivity and noise calibration.

  • Differential privacy foundations: Differential privacy limits disclosure by making computation outcomes insensitive to including any one user’s record.This indistinguishability-based property can mitigate membership inference attacks.
  • Differential privacy foundations: Sensitivity measures how much one record can affect a function’s output, using the maximum p-norm difference between neighboring datasets.Neighboring datasets differ in at most one element.
  • Noise mechanisms: The Laplacian mechanism adds independent Laplace noise to aggregate statistics and provides ε-DP when calibrated to function sensitivity.A weaker time-series variant provides only event-level privacy and is included as a baseline.
  • Noise mechanisms: The Gaussian mechanism perturbs statistics with Gaussian noise and offers a weaker privacy guarantee than the Laplacian mechanism.Its calibration depends on ε, δ, and the function’s L2 sensitivity.
  • Fourier mechanisms: FPA compresses a time-series with the DFT, perturbs the first κ coefficients with Laplace noise, then reconstructs the series and guarantees ε-DP.The method pads perturbed coefficients with zeros before applying the inverse DFT.
  • Fourier mechanisms: EFPAG probabilistically selects coefficients, uses DCT and Gaussian noise, and targets better accuracy while preserving differential privacy.It assigns higher probability to choices minimizing root-sum-squared error.

B. Experimental Design

The defense evaluation measures privacy gain and utility loss when differential-privacy mechanisms perturb large-group aggregates under a strong adversarial prior. It compares passive classifiers trained on raw aggregates with strategic classifiers trained to mimic the defense.

  • Evaluation assumptions: The evaluation targets large groups because large group size gives defenses a utility advantage, while testing a worst-case adversary with perfect prior knowledge.The adversary knows released-group aggregates and target membership, enabling an accurate classifier on raw statistics.
  • Adversary models: The experiment applies a differential-privacy mechanism before releasing challenge aggregates and evaluates privacy gain in two training scenarios.Testing always uses freshly perturbed released aggregates.
  • Adversary models: The passive adversary trains on raw aggregates, whereas the strategic active adversary trains on noisy aggregates using the defense mechanism’s parameters.The strategic setting models an adversary that mimics the defender during training.
  • Experimental settings: The experiments use 9,500-user TFL groups and 500-user SFC groups, with one-week observation and inference periods.The attack datasets contain 200 TFL and 400 SFC sampled groups, half including the target.
  • Mechanism configuration: The study varies ε across {0.01, 0.1, 1.0, 10.0} and calibrates mechanisms using dataset-specific sensitivities.Sensitivity is 207 for TFL and 2,685 for SFC; δ is 0.1 for GSM and EFPAG.
  • Metrics: Privacy gain is the relative decrease in classifier AUC on perturbed aggregates compared with raw aggregates, while MRE measures utility loss.MRE compares raw aggregate time-series with their perturbed versions and averages across regions of interest.

C. Results

Differentially private mechanisms trade privacy protection against utility, and their effectiveness varies with the privacy parameter, dataset, and adversary training strategy.

  • Utility: As ϵ increases, utility improves, but LPA(∆/ϵ) remains least accurate while LPA(1/ϵ) achieves the best utility with poor privacy protection.On TFL and SFC, LPA(∆/ϵ) is 8 and 41 times less accurate than raw aggregates at ϵ = 10; LPA(1/ϵ) has the best utility but poor membership-inference protection.
  • Train on Raw / Test on Noisy Aggregates: For TFL, all mechanisms achieve PG close to 1 up to ϵ = 0.1, while higher ϵ reduces privacy for mechanisms with MRE < 1.At ϵ = 1, mean PG is 0.9 for FPA, 0.75 for EFPAG, and 0.38 for LPA(1/ϵ); at ϵ = 10, the decreases are 0.45 and 0.3 for FPA and EFPAG.
  • Train on Raw / Test on Noisy Aggregates: For SFC, PG stays high through ϵ = 1, and at ϵ = 10 remains almost 1 for LPA(∆/ϵ) and GSM but falls for FPA and EFPAG.At ϵ = 10, mean PG is 0.96 for FPA and 0.92 for EFPAG, while LPA(1/ϵ) provides negligible privacy gain.
  • Utility: The SFC results are reported in Table III, which measures MRE for aggregate location time-series under different mechanisms and ϵ values.The table provides the utility context for the SFC privacy-gain comparisons.
  • Train on Noisy / Test on Noisy Aggregates: Training the classifier on noisy aggregates sharply weakens protection as ϵ increases, especially for FPA and EFPAG.For TFL at ϵ = 10, FPA falls to 0.2 mean PG and EFPAG provides negligible protection; for SFC, GSM, FPA, and EFPAG reach 0.8, 0.32, and 0.15 mean PG, respectively.

D. DP Evaluation – Take-Aways

The evaluation finds that differential privacy can reduce membership inference, but utility costs and adaptive adversaries limit protection; dataset sparsity also shapes privacy gain.

  • DP Evaluation – Take-Aways: Differentially private mechanisms can prevent membership inference, but the privacy–utility trade-off remains difficult to optimize.LPA and GSM provide poor utility, whereas FPA and EFPAG provide better utility at the cost of reduced privacy.
  • DP Evaluation – Take-Aways: A sparser dataset loses privacy gain faster as ϵ grows than a denser dataset.The reported aggregation sensitivities are ∆ = 207 for TFL and ∆ = 2,685 for SFC.
  • DP Evaluation – Take-Aways: The paper addresses a research gap by evaluating privacy threats to individuals contributing to aggregate location time-series.Prior location-privacy work had focused less on this specific threat.
  • DP Evaluation – Take-Aways: Membership inference is accurate for small groups and for users with regular habits, establishing a practical privacy risk in aggregate location data.The evaluation uses a distinguishability game and machine-learning classifier across two real datasets.
  • DP Evaluation – Take-Aways: Differential privacy is more effective against classifiers trained on raw aggregates than against adversaries trained on noisy aggregates.The latter strategy mimics the perturbation mechanism and substantially reduces protection.
  • DP Evaluation – Take-Aways: The methodology can help providers test privacy before release and regulators detect possible violations.These uses are proposed for real-world evaluation of attacks and defenses.

APPENDIX A MACHINE LEARNING CLASSIFIERS

The appendix reviews four classifiers used throughout the paper, spanning linear, neighbor-based, ensemble-tree, and neural-network approaches.

  • Logistic Regression: Logistic Regression models outcome probabilities with a logistic function and estimates parameters by maximum likelihood.Its fitting procedure uses an iterative algorithm.
  • Nearest Neighbors (k-NN): k-NN assigns each query point the class receiving the majority vote among its nearest neighbors.The method is based on local neighbor composition.
  • Random Forest (RF): Random Forest combines decision trees whose majority vote determines the prediction.Each tree uses a bootstrap sample and randomized feature subsets when splitting nodes.
  • Multi-Layer Perceptron (MLP): MLP is a multilayer neural network trained by back propagation to distinguish nonlinearly separable data.Its nodes use nonlinear activation functions outside the input layer.
Loading 1708.06145v2…