Source-linked AI summary
Conformal Prediction for Offensive Security
Giovanni Cherubin
TL;DR
Offensive security has scarcely used CP despite CP’s defensive adoption, leaving a gap in attacks based on conformal inference. The paper introduces CP-based attacks in privacy-preserving ML and network traffic analysis, finding useful prediction-set behavior while documenting marginal-validity and deployment limitations.
Problem
CP has been widely applied defensively but remains scarcely employed for offensive security, despite offensive research’s role in understanding attacks.
Method
The paper develops and examines CP-based attacks in privacy-preserving ML and network traffic analysis, using prediction sets and conformal validity guarantees.
Results
In reconstruction attacks, CP achieves small prediction sets and effective point predictions without privacy, while the attack is unsuccessful under Differential Privacy with ε = 5; CP attains coverage α = 0.1.
Takeaways & Limitations
CP’s set predictions and validity guarantees fit security settings offering abundant IID training data, partial label spaces, or multiple valid predictions.
Takeaways & Limitations
CP validity is marginal rather than conditional, and network traffic attacks face real-world concerns including concept drift and changing websites.
Abstract
from arXiv · showhide
Despite its introduction more than a quarter century ago, Conformal Prediction (CP) has seen surprisingly few applications to the cyber security world thus far. In particular, we observe that, while CP has been employed as a defensive measure in many recent works, its use for carrying out attacks (i.e., for offensive security) is hard to trace in the literature. We explore this gap, by presenting initial findings in two key areas of offensive security: Privacy-Preserving Machine Learning, and network traffic analysis.
1 Introduction
Offensive security research improves attacks, strengthens defenses, and informs users and vendors about attack techniques. Although CP is widely used defensively, it remains largely unexplored for offensive security, motivating CP-based attacks in privacy-preserving ML and network traffic analysis.
- Offensive research strengthens defenses because an attack need bypass only one defense, while a defense must address most attacks in a threat scenario.
- Offensive research also informs the public and product vendors about usable attack techniques.
- CP has supported defensive applications including intrusion detection, malware detection, and protection against evasion and poisoning attacks.
- Traditional ML is widely adopted for offensive security, but CP has been largely unexploited and no known attack had employed it offensively.
- The paper introduces CP-based attacks in two cybersecurity areas and identifies unlimited data generation and prediction sets as advantages for offensive settings.
2 Preliminaries
Offensive security attacks can be framed as statistical inference problems in which an attacker predicts secrets from observations and examples. CP reframes traditional point prediction into dynamically sized prediction sets with validity guarantees, enabling different attack strategies while preserving an average error target.
- Many attacks predict secret information s from side information o using their joint distribution P(s, o) and observed training examples.
- Traditional ML trains a predictor f from sampled pairs (o_i, s_i) and uses f(o) to guess the true secret.
- CP adapts these attacks by reframing inference problems rather than simply replacing traditional ML models.
- A conformal predictor returns a subset of plausible secrets for an observation under exchangeability assumptions and a significance level α.
- CP changes attacks from single guesses to candidate sets whose error risk is controlled through α, potentially reducing search effort.
- Unlike fixed top-k or thresholded multi-label methods, CP determines prediction-set size dynamically from the underlying probability distribution.
- The validity guarantee is marginal: it holds on average for new samples, not conditionally for every individual example.
3 Privacy-preserving ML
Privacy-preserving ML studies what private training information can leak from released models, with reconstruction attacks targeting individual training records. The paper investigates CP for this high-leakage attack type and illustrates the attack setting through a model trained with one private record.
- Privacy-preserving ML asks what information about private training data can be inferred from a released model or its predictions.
- The illustrated attack targets features and a label of one training record while the attacker has the model and the remaining training dataset.
- Reconstruction attacks infer the values of a training data point from access to the trained target model.
- Reconstruction attacks can provide high information leakage but are among the hardest privacy-preserving ML attacks to perform.
- The target model is trained on a private dataset Z ∈ (X × Y)^N, with the attack reconstructing a record from the trained model.
Problem setup and threat model.
The reconstruction threat model gives the attacker the target model and all training data except one record, then uses a learned regressor to infer that record. Arbitrarily many synthetic training examples can be generated, while the target and attacker models serve distinct roles.
- Threat model: The attacker infers a training record from the target model’s parameters and the remaining training dataset, under informed-attacker assumptions.The attacker is also assumed to know model hyperparameters and randomness, enabling identical retraining if the full dataset were available.
- Attack model: The attacker trains a regressor that maps the available target-model information to a guessed record and evaluates reconstruction using distance from the true record.The described goodness measure includes L2 distance between the guessed and true records.
- Attack model: Synthetic records can be sampled and used to train corresponding models, allowing the attacker to generate arbitrarily many training examples.This expands the available training data for the attacker’s regressor in principle.
- Threat model: The target model is trained by a data curator, whereas the attacker’s model is used to perform the attack.The two models have different owners and functions despite both being machine-learning models.
CP for reconstruction.
The paper replaces reconstruction’s point prediction with CP regression, producing prediction bands for private records. Its experiment trains separate CQR-based regressors for each record feature and label using synthetic target models.
- CP formulation: CP reformulates reconstruction from a point prediction into a prediction region, giving attackers bands that can be more informative for inferring private information.The underlying reconstruction attack uses a traditional ML regressor for point guesses before this CP reformulation.
- Target model training: The experiment trains 5,000 Logistic Regression target models on the Adult dataset to reconstruct held-out records and their labels.Each target model adds one test-split point to the common training set.
- Target model training: The target task predicts whether income exceeds 50k from five Adult-dataset features.The listed features are age, education-num, capital-gain, capital-loss, and hours-per-week.
- Attacker’s model training: CQR with LightGBM supplies separate regressors for each feature and label, whose outputs are joined into the reconstructed record.The attacker’s training set contains 100k synthetic target models generated from sampled records.
- Evaluation: The evaluation measures CP coverage and prediction-region size, while point-prediction success uses L2 distance against a median-value baseline.The baseline ignores the attacker’s additional information by outputting each feature’s median in the remaining data.
Results.
The reconstruction attack attains the stated CP coverage and is effective without privacy protection, but fails under differential privacy. Individual examples can still violate validity because CP guarantees hold on average rather than conditionally.
- Quantitative results: In the no-privacy setting, CP produces small prediction sets and point predictions that outperform the naive baseline, indicating an effective reconstruction attack.The metrics are reported separately for each feature and the label.
- Quantitative results: With DP ε = 5 applied, the reconstruction attack is unsuccessful as expected.This contrasts with the successful attack against the unprotected Logistic Regression model.
- Coverage: CP coverage of α = 0.1 is attained in both privacy settings.The table covers standard and differentially private Logistic Regression models over 5k test examples.
- Validity behavior: Some individual reconstruction examples violate validity, including a case where the Capital Gain coverage interval falls outside the prediction region.The examples are cherry-picked to show perfect validity, edge-aligned values, and the largest observed deviation for Capital Gain.
- Implications: CP is presented as suitable for reconstruction attacks because informative coverage bands can be combined with information from other attacks.The discussion frames this as support for considering CP in offensive-security threat models.
- Implications: Reconstruction attacks can use virtually unlimited synthetic training examples, enabling split CP and CQR despite their lower data efficiency.These methods are described as more computationally efficient than full CP.
3.2 Research direction: Membership Inference attacks
The paper identifies membership inference as another PPML attack area where CP may be directly applicable or may motivate new attack designs. Traditional membership inference itself is a binary classification problem.
- Research direction: The paper proposes two new membership inference attacks that leverage CP, alongside straightforward adaptations for attacks such as attribute inference.It presents CP as potentially significant across varied PPML attacks.
- Membership inference: Traditional membership inference asks whether a record belongs to a model’s private training dataset.The attacker has access to the trained model and the candidate record.
- Membership inference: Because membership inference predicts the indicator I(z ∈Z), applying CP classification is straightforward but may be uninsightful.The paper distinguishes this direct adaptation from other CP-based membership-inference designs.
Membership inference from first principles.
Membership inference can exploit shadow models and conformal prediction to identify whether queried records resemble training members, with coverage guarantees supporting more reliable true-positive detection.
- Membership inference prioritizes reliable true-positive predictions because non-members vastly outnumber members.
- Attackers train shadow models on mock datasets, labeling training points as members and held-out points as non-members.
- Applying CP to shadow-model outputs tests whether a queried point behaves typically of training data.
Subset membership inference.
Subset membership inference asks attackers to identify which records in a challenge set belong to a model’s training data, using CP-inspired procedures to reduce the candidate space with error guarantees.
- The attack targets the subset S = Z ∩ A of challenge records that belong to the model’s training data.
- Attack success is measured by the precision and recall of an output subset S′ relative to the true member set S.
- Two approaches apply CP record-by-record or formulate records as labels in a CP classification problem.
- Subset membership inference can help attackers identify multiple users in large datasets while reducing the search space with error guarantees.
4 Traffic analysis attacks
The paper extends CP to offensive traffic analysis, demonstrating a Website Fingerprinting attack against Tor and outlining CP-based approaches for SSH keystrokes and chatbot-topic inference.
- Traffic analysis infers information from encrypted communications by exploiting patterns such as message volume and timing.
- Website Fingerprinting: Website Fingerprinting infers visited webpages from encrypted Tor or VPN traffic, traditionally using an ML classifier.
- Website Fingerprinting: The WF dataset contains 95 monitored websites with 1,000 traces each and 40,716 open-world traces.
- Website Fingerprinting: The CP attack uses split CP, reserving 10% of training data for calibration and defining nonconformity as one minus CNN softmax probability.
- Website Fingerprinting: For open-world traces, empty prediction sets indicate that traffic likely comes from an unmonitored website without explicitly modeling that class.
- Limitations and monitoring: WF deployment remains bounded by concerns that laboratory attacks may not succeed under real-world changes such as concept drift.
- Further traffic-analysis directions: CP could control error rates and candidate-sequence sizes in SSH keystroke attacks and evaluate multiple chatbot topics independently from token-size sequences.
5 Discussion
The discussion argues that CP is underused in offensive security despite matching threat models with open-world labels, multiple valid answers, and abundant IID examples.
- CP has been scarcely applied offensively, despite security threat models often offering abundant IID training examples and set-valued prediction opportunities.
- Set prediction can help attackers narrow candidate secrets before pursuing more costly attacks such as brute-forcing.
- CP’s validity guarantee lets attackers quantify and control the risk of excluding the correct secret through α.
- The paper presents CP as having significant potential for future offensive-security attacks.