PaperlayerView exact revision on arXiv

Source-linked AI summary

Mockingbird: Defending Against Deep-Learning-Based Website Fingerprinting Attacks with Adversarial Traces

Mohammad Saidur Rahman, Mohsen Imani, Nate Mathews, Matthew Wright

arXiv:1902.06626v5cs.CRcs.LG

TL;DR

Website fingerprinting lets a local eavesdropper identify visited sites from traffic, and an attacker can train on adversarial traces generated by a defense. Mockingbird varies its target traces rather than following predictable gradients. It lowers hardened attack accuracy with moderate bandwidth overhead.

  • Problem

    Website fingerprinting lets a local eavesdropper identify visited sites, and an attacker can reproduce a defense's adversarial traces to train a more robust classifier.

    Source: §I · pp. 1–2

  • Method

    Mockingbird generates adversarial traffic traces by randomly selecting and switching target traces, using classifier confidence without accessing gradients or logits.

    Source: §I · p. 2

  • Results

    Attack accuracy falls from 98% to 38%-58% depending on scenario, with 58% full-duplex overhead; Top-2 accuracy is at most 72% versus 97% on W-T and 95% on WTF-PAD.

    Source: §I · p. 2

  • Takeaways & Limitations

    WeFDE analysis finds less information leakage for many feature types than W-T or WTF-PAD, a promising first step toward protecting privacy.

    Source: §I · p. 2

Abstract

from arXiv · show

Website Fingerprinting (WF) is a type of traffic analysis attack that enables a local passive eavesdropper to infer the victim's activity, even when the traffic is protected by a VPN or an anonymity system like Tor. Leveraging a deep-learning classifier, a WF attacker can gain over 98% accuracy on Tor traffic. In this paper, we explore a novel defense, Mockingbird, based on the idea of adversarial examples that have been shown to undermine machine-learning classifiers in other domains. Since the attacker gets to design and train his attack classifier based on the defense, we first demonstrate that at a straightforward technique for generating adversarial-example based traces fails to protect against an attacker using adversarial training for robust classification. We then propose Mockingbird, a technique for generating traces that resists adversarial training by moving randomly in the space of viable traces and not following more predictable gradients. The technique drops the accuracy of the state-of-the-art attack hardened with adversarial training from 98% to 42-58% while incurring only 58% bandwidth overhead. The attack accuracy is generally lower than state-of-the-art defenses, and much lower when considering Top-2 accuracy, while incurring lower bandwidth overheads.

I. INTRODUCTION

Deep-learning website fingerprinting threatens Tor users, while adversarial training can undermine straightforward adversarial-example defenses. Mockingbird uses randomized trace generation to resist this adaptation, reducing attack accuracy with moderate overhead.

I. INTRODUCTION

  • A local eavesdropper can use deep-learning WF attacks that exceed 98% accuracy on undefended traffic in a closed-world setting, while existing lightweight defenses remain vulnerable.The cited attacks include Deep Fingerprinting and Var-CNN; the paper motivates defenses that can withstand improved attacker capabilities.Source: §I · p. 1
  • Straightforward adversarial-example techniques can fail when an attacker reproduces the defense's traces for adversarial training.The paper identifies this attacker adaptation as a reason to seek a more robust WF defense.Source: §I · p. 2
  • Mockingbird searches by moving toward randomly selected target traces and switching targets, using classifier confidence rather than gradients or logits.This produces varied paths from a source trace and can end near different targets across traces generated from the same source, helping reduce Top-k accuracy.Source: §I · p. 2
  • Mockingbird reduces state-of-the-art adversarially trained attack accuracy from 98% to 38%-58%, depending on scenario, with 58% full-duplex bandwidth overhead.The reported overhead is better than both W-T and WTF-PAD; results also hold against a more powerful classifier than the one used to generate examples.Source: §I · p. 2
  • Against Mockingbird, the best attack reaches at most 72% Top-2 accuracy, versus 97% on W-T and 95% on WTF-PAD.The paper presents this as making it difficult for an attacker to narrow the user's possible sites to a small set.Source: §I · p. 2
  • WeFDE analysis finds less information leakage for many feature types with Mockingbird than with W-T or WTF-PAD.The authors describe this investigation as a promising first step toward using adversarial examples to undermine WF attacks and protect privacy.Source: §I · p. 2

A. Threat Model

The threat model is a local passive WF attacker who classifies Tor traffic to identify monitored sites, while the defense manipulates traffic with a cooperating bridge. The section reviews deep-learning attacks and defenses, highlighting their accuracy and overhead trade-offs.

A. Threat Model

  • A local passive attacker observes traffic between the client and Tor guard, then classifies traces to identify visits to monitored sites; open-world testing also includes unmonitored sites.The attacker cannot drop, delay, or modify real packets, and the open-world test uses unmonitored sites not included in training.Source: A. Threat Model · pp. 2–3

B. Defense Model

  • A WF defense manipulates traffic and requires participation by the client and a cooperating Tor node called a bridge, preferably the middle node.A middle-node bridge cannot directly identify the client, unlike the guard node.Source: B. Defense Model · p. 3

A. WF Attacks

  • 98.8% closed-world accuracy was reported for Var-CNN, while DF reached 98% Top-2 accuracy against Walkie-Talkie.DF also reached 90% accuracy against WTF-PAD; the attacks use deep-learning classifiers to identify traffic patterns.Source: A. WF Attacks · p. 3

B. WF Defenses

  • More than 100% bandwidth overhead and two to three times Tor’s fetch time characterize constant-rate defenses; Walkie-Talkie reports 31% bandwidth and 34% latency overhead.Walkie-Talkie’s two-site anonymity sets have a theoretical maximum accuracy of 50%, while DF achieved 49.7% accuracy and 98.4% Top-2 accuracy against it.Source: B. WF Defenses · pp. 3–4
  • WTF-PAD drops k-NN accuracy from 92% to 17% at 60% bandwidth overhead, but DF achieves up to 90% accuracy against it.Application-level defenses also report trade-offs: ALPaCA reduces CUMUL accuracy from 56% to 33% with 41% latency and 44% bandwidth overhead, while LLaMA reduces it to 34% with 9% latency and 7% bandwidth overhead.Source: B. WF Defenses · p. 4

IV. PRELIMINARIES

The preliminaries introduce adversarial examples and explain why small perturbations and transferability make them relevant to WF defense. They also show that straightforward adversarial traces fail against adversarial training, motivating Mockingbird’s more randomized design.

A. Adversarial Examples

  • An adversarial example modifies an input to be classified as a different target class; an untargeted example may be classified as any class other than its true class.Source: A. Adversarial Examples · p. 4

B. Properties of Adversarial Examples

  • Adversarial examples offer three potentially useful WF-defense properties: robust misclassification, small perturbations that can limit padding overhead, and transferability across at least some classifiers.Transferability is relevant because the attacker’s classifier cannot be predicted in advance.Source: B. Properties of Adversarial Examples · p. 4

C. Adversarial Training

  • Adversarial training incorporates adversarial examples into training so the classifier learns to classify them correctly, but it may not adapt to generation techniques it has not seen.In WF, an attacker can know the trace-generation method through open-source Tor code, making adversarial training a significant concern.Source: C. Adversarial Training · pp. 4–5

D. Data Representation

  • Traffic is represented as consecutive incoming and outgoing bursts, which can be lengthened with padding but not shortened by dropping real packets.Dropping real packets would cause retransmissions, alter traffic patterns, and add user delays.Source: D. Data Representation · p. 5

A. Applying Existing Methods in WF Defense

  • With adversarial training, the Carlini–Wagner technique fails as a WF defense: DF reaches 97% accuracy.Without adversarial training, the same approach reduces DF accuracy from 98% to 3% and CUMUL accuracy from 92% to 31%.Source: A. Applying Existing Methods in WF… · p. 5
  • The attacker can train after learning the defense’s trace-generation method, so techniques effective in ordinary adversarial-example tests may be poorly suited to WF defense.The modified Carlini–Wagner method was adapted from images to one-dimensional traffic traces.Source: A. Applying Existing Methods in WF… · p. 5

B. Generating Adversarial Traces

Mockingbird generates adversarial traces by moving a source trace toward selected traces from other site classes, using the detector to check when the source class changes. It restricts perturbations to burst increases and changes targets when progress stalls, aiming to reduce dependence on classifier-specific gradients.

B. Generating Adversarial Traces

  • Mockingbird selects potential target traces randomly from other classes, moves the source toward a selected target, and stops when the detector determines that the source class has changed.The resulting trace need not be classified as the target class; the added change governs bandwidth overhead and should be minimized.Source: B. Generating Adversarial Traces · p. 6
  • Mockingbird moves toward the target trace rather than optimizing detector loss or logits, using the detector only to estimate misclassification confidence.The authors say this reduces reliance on detector architecture and helps resist adversarial training; an optimization-based variant was much less robust.Source: B. Generating Adversarial Traces · p. 6
  • The source trace is represented as a burst sequence, and the goal is to alter it so the detector assigns a class other than its original class.Source: B. Generating Adversarial Traces · p. 6
  • Only nonnegative burst-length changes are allowed, because reducing a burst would require dropping real packets and cause retransmissions.The target is chosen from a random pool of traces from other classes, selecting the nearest by l2 distance to help minimize overhead.Source: B. Generating Adversarial Traces · p. 6
  • The perturbation adds nonnegative changes to burst lengths, and the method seeks the least change that moves the source closer to its target.The gradient of negative distance supplies the direction for changing the source toward the target.Source: B. Generating Adversarial Traces · pp. 6–7
  • When progress stalls while the sample remains in its source class, Mockingbird selects a new target pool and continues from the already modified trace.The authors used λ = 10 consecutive iterations for the stall condition; confusion in the final trace is proportional to the number of targets changed.Source: B. Generating Adversarial Traces · p. 7

A. Datasets

The evaluation uses full-duplex and half-duplex traffic traces in closed-world and open-world settings, with separate data for generating adversarial traces and training the detector. Both duplex datasets use 750-burst inputs to limit computational cost.

A. Datasets

  • Mockingbird is evaluated on burst sequences from both full-duplex and half-duplex communication modes, with Walkie-Talkie using half-duplex communication.Source: A. Datasets · p. 7
  • The datasets comprise closed-world and open-world traces from Sirinam et al.; monitored websites come from the top Alexa sites.Source: A. Datasets · p. 7
  • After preprocessing, full-duplex closed-world data contain 518 instances for each of 95 classes, while half-duplex data contain 720 instances for each of 83 classes.The open-world sets contain 40,716 full-duplex and 40,000 half-duplex sites, with one instance per site.Source: A. Datasets · p. 7
  • 750 bursts are used as the fixed input size for both duplex datasets to reduce adversarial-example generation cost; using 1,500 bursts improved DF accuracy by just 1%.More than 80% of half-duplex closed-world traces have fewer than 750 bursts, and more than 80% of full-duplex closed-world traces have fewer than 500.Source: A. Datasets · p. 7, Figure 4 · p. 7
  • The data are split into non-overlapping Adv and Detector sets, each containing half the instances of every class.Each set has 259 instances per class for 95 full-duplex classes and 360 per class for 83 half-duplex classes.Source: A. Datasets · p. 7, Table I · p. 7

B. Experimental Method

The experiments evaluate Mockingbird against adversarially trained deep-learning attackers and traditional machine-learning attacks, using distinct detector and attacker settings. They also examine Top-2 accuracy and how the target pool affects evaluation.

B. Experimental Method

  • The detector is trained on Detector Set traces, while Adv Set sources generate adversarial traces; defended burst traces are evaluated as packet directions.The detector uses AWF or DF and 10,000-packet inputs; +1 and −1 encode outgoing and incoming packets.Source: B. Experimental Method · p. 8
  • Black-box tests use AWF as detector and DF or Var-CNN as stronger attackers; adversarial training gives attackers access to defense-generated samples.White-box DF against DF is used only for parameter search; black-box tests assess transfer to different classifiers.Source: B. Experimental Method · p. 8
  • Attack data are split 80%/10%/10% for training, validation, and testing, and represented as 5,000-packet vectors padded or truncated as needed.Outgoing and incoming packets are encoded as +1 and −1.Source: B. Experimental Method · p. 8
  • Hyperparameters are tuned sequentially for DF and Var-CNN; tuned models perform better on full-duplex traces, while default models perform better on half-duplex traces.Table II covers hyperparameter tuning for black-box attacks.Source: B. Experimental Method · p. 8, Table II · p. 8
  • Top-2 accuracy supplements Top-1 because an attacker narrowing a trace to two websites can still reveal sensitive-site visits.The paper also evaluates CUMUL, k-FP, and k-NN, which exceed 90% accuracy on undefended Tor in closed-world datasets with 100 samples per class.Source: B. Experimental Method · p. 8
  • The target pool uses detector-trained classes in Case I and unmonitored sites in Case II, a more conservative security estimate.Case I may fit settings where defender and attacker care about the same sites; Case II assumes target samples are outside detector training.Source: B. Experimental Method · pp. 8–9

C. Tuning on Full-Duplex Data

The full-duplex experiments tune confidence and perturbation thresholds, α, and iteration count to balance attack accuracy against bandwidth overhead. The selected settings are τc = 0.01, τD = 0.0001, α = 5, and 500 iterations.

C. Tuning on Full-Duplex Data

  • τc = 0.01 is selected because lower confidence thresholds reduce attack accuracy but increase bandwidth overhead.The experiments hold τD = 0.0001, α = 5, and iteration count at 500.Source: C. Tuning on Full-Duplex Data · p. 9
  • τD = 0.0001 is selected because lower perturbation thresholds reduce overhead but increase attack accuracy.The experiments hold τc at the selected value while varying τD.Source: C. Tuning on Full-Duplex Data · p. 9
  • 35% attack accuracy occurs for Case I at both α=5 and α=7, with 56% bandwidth overhead at α=5 versus 59% at α=7.The experiments therefore fix α=5; in Case II, accuracy and overhead are also slightly lower at α=5 than α=7.Source: C. Tuning on Full-Duplex Data · pp. 9–10
  • Case I attack accuracy is at least 20% lower than Case II for α=5 and α=7, with comparable bandwidth overhead.Case I uses targets from classes represented in detector training.Source: C. Tuning on Full-Duplex Data · p. 10
  • 35% attack accuracy at α=5 requires 54% bandwidth overhead with 400 iterations and 56% with 500 in Case I.In Case II at α=5, 400 iterations yield 57% accuracy and 53% overhead; 500 yield 55% accuracy and 56% overhead, so 500 iterations are selected.Source: C. Tuning on Full-Duplex Data · p. 10

D. Tuning on Half-Duplex Data

Mockingbird’s half-duplex tuning balances attack accuracy against bandwidth overhead, and its evaluation compares the defense across black-box and white-box attacks, Top-k identification, and repeated observations. The reported results show reduced attack accuracy, with repeated-round intersection attacks still leaving uncertainty about the visited site.

D. Tuning on Half-Duplex Data

  • For half-duplex tuning, τc = 0.01 and τD = 0.0001 yield 35% and 29% attack accuracy with 73% and 63% overhead in Cases I and II; α = 7 and 500 iterations are selected for subsequent experiments.At α = 7, the reported accuracies are 35.5% and 28.8%, with overheads of 62.7% and 73.5%, respectively.Source: D. Tuning on Half-Duplex Data · p. 10, Figure 9 · p. 10

E. Results Analysis

  • In the white-box setting, Case I attack accuracy is below 40% for both full- and half-duplex traffic, while Case II reaches 29% for half-duplex and 55% for full-duplex.Table III evaluates Mockingbird against DF.Source: Table III · p. 11, E. Results Analysis · p. 11
  • 58% bandwidth overhead in full-duplex Mockingbird is 6% and 14% lower than WTF-PAD and W-T, respectively; half-duplex overhead is 62% for Case I and 70% for Case II.The paper attributes packet-addition control to keeping trace changes below a 0.0001 threshold.Source: E. Results Analysis · p. 11, Table IV · p. 12
  • The highest accuracy among CUMUL, k-FP, and k-NN against Mockingbird is 32% in Case II full-duplex, and the defense is competitive with or better than both comparators on all three attacks.Source: E. Results Analysis · p. 11
  • Mockingbird’s worst-case Top-2 accuracy is 72%, compared with 97% for W-T and 95% for WTF-PAD; its worst-case Top-10 accuracy is 92%.The Top-10 figure is for full-duplex Case II; half-duplex Top-10 values are 90% for Case I and 88% for Case II.Source: E. Results Analysis · p. 11, Figure 10 · p. 11
  • 42% (Case I) and 62% (Case II) are the respective maximum black-box attack accuracies against Mockingbird; Var-CNN reaches 90% against WTF-PAD and 44% against W-T.The paper concludes Mockingbird’s Top-1 effectiveness falls between the two comparison defenses.Source: E. Results Analysis · p. 11, Table IV · p. 12

F. Intersection Attacks

  • After five intersection rounds, absolute successes are 27% and 36% for full-duplex Cases I and II and below 20% for both half-duplex cases; mean intersections range from 2.60 to 4.12.The authors conclude this attack is only moderately helpful, but Mockingbird does not assure protection against multiple rounds of observation.Source: Table V · p. 12, F. Intersection Attacks · p. 12

G. Information Leakage Analysis

The information-leakage analysis finds Mockingbird comparable to other defenses overall, with lower maximum per-feature leakage than WTF-PAD and W-T. The comparison also reports a trade-off: Mockingbird has lower Top-2 accuracy than W-T but higher Top-1 accuracy than W-T, and it can use full-duplex communication.

G. Information Leakage Analysis

  • Mockingbird leaks at most 1.9 bits per feature, versus at most 2.2 bits for W-T and 2.6 bits for WTF-PAD; its maximum leakage is comparable to the other defenses.The analysis covers 3,043 manually defined features across 14 categories; Timing and Pkt. per Second measurements exclude W-T and Mockingbird.Source: G. Information Leakage Analysis · p. 12
  • Mockingbird improves over W-T on Transposition, First 30, Last 30, and Burst features, while W-T performs better on Pkt. Distribution features.Mockingbird appears most vulnerable to N-gram features and is effective against packet-count features.Source: G. Information Leakage Analysis · p. 13

A. Comparison

  • Compared with W-T, Mockingbird has lower Top-2 accuracy by at least 27% for DF and 22% for Var-CNN, but its Top-1 accuracy is relatively higher.The authors characterize W-T’s 97% Top-2 accuracy as unacceptable given its deployment and operating costs.Source: A. Comparison · p. 13
  • The black-box comparison uses AWF, a detector weaker than DF and Var-CNN, so a more powerful detector could potentially improve robustness against future models.Source: A. Comparison · p. 13
  • Mockingbird can use full-duplex communication, unlike W-T, and therefore has lower latency overhead; its full-duplex bandwidth overhead is modestly better than both compared defenses.The authors hypothesize that random target selection creates unpredictable patterns that confuse the attacker among more possible sites.Source: A. Comparison · p. 13

B. Implementation & Deployment

Mockingbird’s deployment requires addressing trace-generation, padding, and computational challenges, while server-side deployment is also proposed. The conclusion reports stronger protection with lower bandwidth overhead than prior lightweight defenses, but notes implementation challenges before practical Tor deployment.

B. Implementation & Deployment

  • Mockingbird needs the full burst sequence before generating a trace, requiring a database of recent reference bursts; burst molding would queue packets until a timeout before forwarding them.The burst size can be communicated from client to bridge in prior-burst dummy packets, but setting the timeout to minimize delay remains future work.Source: B. Implementation & Deployment · pp. 13–14
  • Most sites’ traces can be generated in around 0.5 second, but some half-duplex sites take up to 3 seconds, and generation currently requires significant computational resources such as a dedicated GPU.The generation process currently involves hundreds of iterations; the paper describes a few seconds before each visit as the deployment target.Source: B. Implementation & Deployment · p. 14
  • Relaxing the bandwidth constraint on a first visit or precomputing traces for previously visited sites could reduce first-visit computation; pruning, quantization, distillation, and SLIDE are also discussed as possible efficiency approaches.The authors leave further investigation of these techniques to future work.Source: B. Implementation & Deployment · p. 14

C. Server-Side Defense

  • The paper proposes server-side Mockingbird for site operators, who could generate traces during downtime and periodically update them to raise attackers’ costs.The proposal modifies traces by adding dummy web objects and padding existing objects, following Cherubin et al..Source: C. Server-Side Defense · p. 14

VIII. CONCLUSION

  • Top-1 accuracy falls from over 98% to at most 62%, and Top-2 accuracy is at most 72% versus 95% for WTF-PAD and 97% for Walkie-Talkie.Mockingbird’s full-duplex bandwidth overhead is 58%, lower than either prior defense; the evaluation is closed-world, and the conclusion says 58% accuracy in an open-world setting is very likely to produce many false positives for the attacker.Source: §VIII · p. 14
  • Mockingbird has implementation challenges to resolve before practical Tor deployment, although the paper identifies potential for a server-side defense.The paper also points to distribution-agnostic adversarial patches as a possible future direction for network traces.Source: B. Implementation & Deployment · p. 13, §VIII · p. 14

APPENDIX A PERFORMANCE OF BASE C&W METHOD

The appendix evaluates C&W-based adversarial traces in settings where attackers train on undefended or defended traffic. The traces transfer against undefended classifiers but remain detectable when attackers use adversarial training.

APPENDIX A PERFORMANCE OF BASE C&W METHOD

  • C&W searches for a perturbation that makes a trace resemble its source while causing misclassification, using a distance metric and an objective function.The appendix considers both targeted and non-targeted objectives.Source: Appendix A · p. 16
  • The appendix distinguishes attackers trained on undefended traces from those trained on defended traces, using a half-duplex closed-world dataset for both scenarios.Without adversarial training, the classifier is trained on undefended traces before testing adversarial traces; with adversarial training, training and testing use traces protected by the same defense.Source: Appendix A · p. 16, A. Without-Adversarial-Training · p. 16, B. With-Adversarial-Training · p. 16

A. Without-Adversarial-Training

  • 62% bandwidth overhead accompanies 98% misclassification of the target simple CNN, while DF and CUMUL accuracies are 3% and 31% on adversarial traces.The authors interpret these results as transferability to other models, with stronger transfer to the deep-learning model than to the traditional machine-learning model.Source: A. Without-Adversarial-Training · p. 16, Table VI · p. 16

B. With-Adversarial-Training

  • 91% accuracy for the simple CNN, 97% for DF, and 91% for CUMUL show that C&W-generated traces remain detectable when attackers train on defended traces.The authors conclude that this approach is ineffective against adversarial training and motivates a generation technique designed specifically for website fingerprinting.Source: B. With-Adversarial-Training · p. 17, Table VII · p. 17

APPENDIX B Mockingbird WITH C&W

The appendix tests replacing Mockingbird’s movement toward a selected target with C&W optimization while retaining random target changes. The tuned C&W alternative has higher DF attack accuracy than Mockingbird HD at both reported overhead levels.

APPENDIX B Mockingbird WITH C&W

  • The variant replaces movement toward a randomly selected target with C&W loss optimization, changing targets when the current trace has not reached the target class and limiting changes to T.The experiments set T = 8 and vary the maximum iteration count k.Source: Appendix B · p. 17
  • At 70% bandwidth overhead, the C&W alternative has 65% attack accuracy, at least 24% higher than Mockingbird HD; at 50% overhead, its accuracy is at least 35% higher than any Mockingbird HD case.The figure compares DF attack accuracy across iteration counts for C&W, Mockingbird HD, and W-T.Source: Appendix B · pp. 17–18
  • The authors suggest that reducing gradients may follow more predictable search paths than moving directly toward randomly selected target traces, making the optimized variant less effective.They judge the overhead cost of finding an adversarial example worthwhile for preventing adversarial training from finding the same kinds of examples.Source: Appendix B · p. 18

Related papers

Loading 1902.06626v5…