PaperlayerView exact revision on arXiv

Source-linked AI summary

Var-CNN: A Data-Efficient Website Fingerprinting Attack Based on Deep Learning

Sanjit Bhat, David Lu, Albert Kwon, Srinivas Devadas

arXiv:1802.10215v2cs.CRcs.LG

TL;DR

Website fingerprinting can let a local adversary infer which website a Tor user visits, but prior deep-learning attacks offer little advantage with small training sets. Var-CNN combines deep learning with packet-sequence-specific insights, achieving stronger open-world results and marked gains in low-data settings. Its architecture-independent insights reduce the training data needed for effective attacks.

  • Problem

    Prior deep-learning WF attacks outperform earlier methods with medium to large datasets but provide marginal to no accuracy improvements with small training sets.

  • Method

    Var-CNN combines a ResNet CNN with dilated causal convolutions, automatically extracted direction and timing features, and manually extracted cumulative features.

  • Results

    In open-world settings, Var-CNN achieves over 1% better TPR and 4× lower FPR than prior art, while low-data gains reach a 3.12% FPR reduction and 13% TPR increase.

  • Takeaways & Limitations

    Var-CNN’s model insights can apply to future neural-network attacks, reducing training-data needs and supporting effective attacks by weaker adversaries.

  • Takeaways & Limitations

    Var-CNN trains longer than DF because of its more complex model, additional cumulative-feature layer, and timing model.

Abstract

from arXiv · show

In recent years, there have been several works that use website fingerprinting techniques to enable a local adversary to determine which website a Tor user visits. While the current state-of-the-art attack, which uses deep learning, outperforms prior art with medium to large amounts of data, it attains marginal to no accuracy improvements when both use small amounts of training data. In this work, we propose Var-CNN, a website fingerprinting attack that leverages deep learning techniques along with novel insights specific to packet sequence classification. In open-world settings with large amounts of data, Var-CNN attains over $1\%$ higher true positive rate (TPR) than state-of-the-art attacks while achieving $4\times$ lower false positive rate (FPR). Var-CNN's improvements are especially notable in low-data scenarios, where it reduces the FPR of prior art by $3.12\%$ while increasing the TPR by $13\%$. Overall, insights used to develop Var-CNN can be applied to future deep learning based attacks, and substantially reduce the amount of training data needed to perform a successful website fingerprinting attack. This shortens the time needed for data collection and lowers the likelihood of having data staleness issues.

1 Introduction

Website fingerprinting threatens Tor users because traffic patterns can reveal visited sites, while deep-learning attacks have struggled with small training sets. Var-CNN addresses this gap with packet-sequence-specific design insights and improves results across data regimes.

  • Motivation: Small-data deep-learning WF attacks can match, but not substantially outperform, manual-feature attacks, despite stronger performance with more data.Trace changes force attackers to refresh databases, making data efficiency practically important.
  • Approach: Var-CNN is a semi-automated deep-learning WF attack tailored specifically to network packet sequence classification.It combines manual and automated feature extraction.
  • Key insights: Dilated causal convolutions enlarge feature detectors exponentially without increasing runtime, accommodating packet sequences’ global relationships.The design draws on techniques from audio synthesis and computational biology.
  • Key insights: Var-CNN combines automatically extracted direction and timing features with manually extracted cumulative features during training.Timing information remains useful under a domain change from direction data to packet timing data.
  • Results: Over 1% higher TPR and 4× lower FPR than prior art were achieved in the largest open-world setting tested.The attack also improved closed-world accuracy from 96.5% to 98.8% in the largest closed-world experiment.
  • Results: With limited training data, Var-CNN reduced prior-art FPR by 3.12% while increasing TPR by 13%.In a closed-world example, it reached 97.8% accuracy with 100 traces per site, while prior art needed 5× as much data for 98.1%.

2 Background and threat model

The WF threat model considers a passive adversary who builds trace databases and classifies monitored versus unmonitored websites. Evaluation uses closed- and open-world settings under assumptions about representative data and practical applicability that remain contested.

  • Threat model: A passive adversary monitors the user-to-Tor connection and uses packet traces to identify monitored websites.The adversary neither drops, modifies, nor inserts packets.
  • Threat model: The adversary creates a database of packet sequences and timestamps, then classifies collected user traces as monitored or unmonitored.Direction data records packet direction, while time data records delays between consecutive packets.
  • Attacker settings: Closed-world evaluation classifies traces among a known set of monitored sites and measures accuracy.This setting is less realistic but isolates the classifier’s ability to distinguish websites.
  • Attacker settings: Open-world evaluation includes monitored and unmonitored websites and permits training with unmonitored traces.Its metrics are Two-TPR, Multi-TPR, and FPR.
  • Assumptions: WF attacks assume training traces represent actual Tor traffic and that the attack can be applied effectively in practice.These are called replicability and applicability assumptions.
  • Assumptions: Dynamic content, browser differences, latency variation, mismatched site populations, unknown trace boundaries, concurrent visits, and background noise challenge those assumptions.The paper notes that realistic user datasets are difficult to collect because of privacy concerns.

3 Related work

Earlier WF attacks manually designed protocol-specific features, while later deep-learning attacks automated feature extraction from raw sequences. Deep Fingerprinting improved on prior art with sufficient data but showed little advantage in low-data settings.

  • Manual feature extraction attacks: Manual WF attacks target protocol-specific leakage, including resource lengths, packet lengths, orderings, concentrations, bursts, and cumulative features.Examples include Wang et al.’s k-NN, CUMUL’s SVM, and k-FP’s Random Forest.
  • Manual feature extraction attacks: Manual feature attacks are restricted by their chosen feature sets and can lose accuracy when defenses perturb the information they primarily use.CUMUL’s reliance on packet ordering is one cited example.
  • Automated feature extraction attacks: Automated feature-extraction attacks use deep neural networks on raw input sequences, removing the need for manual feature design.Early SDAE approaches underperformed Wang et al.’s manually engineered k-NN, while later models slightly surpassed prior art.
  • Automated feature extraction attacks: Deep Fingerprinting, the state-of-the-art WF attack, uses a CNN architecture similar to VGG and improves over prior art in closed- and open-world settings.Its low-data advantage was limited: with 50 traces, both attacks achieved around 90% accuracy.

4 Var-CNN: Model variations on CNN

Var-CNN builds on a ResNet-18 baseline and adapts it for packet-sequence classification with dilated causal convolutions, residual connections, and cumulative features. These choices expand temporal context efficiently while combining automated and manually extracted information.

  • Baseline architecture: The baseline uses an 18-layer ResNet with four stages of two convolutional blocks, each containing two convolutional layers with batch normalization and ReLU.Residual skip connections add each block’s input to its output, easing optimization and supporting higher-level feature extraction.
  • Dilated causal convolutions: Regular convolutions provide receptive fields that grow linearly with depth, which is poorly matched to packet sequences’ long-term, intertwined structure.Packet activity can trigger cascading effects across several hundred streams, requiring broader temporal context.
  • Dilated causal convolutions: Dilated causal convolutions skip inputs at selected rates, giving the CNN a coarse, wide view without increasing filter parameters or training cost.The model doubles dilation rates through {1, 2, 4, 8} before cycling back to 1, producing exponential receptive-field growth over a limited number of steps.
  • Dilated causal convolutions: Causal convolutions restrict each neuron to previous timesteps, while dilation expands the temporal context used to model packet-sequence dependencies.Using dilated causal convolutions increased accuracy from 94% to 96% in a closed world of 100 monitored sites and 90 monitored instances.
  • Cumulative statistical features: Var-CNN supplements automatically extracted ResNet-18 features with cumulative manually extracted features, making it a semi-automated feature-extraction model.Cumulative features alone perform poorly, so the paper combines them with the ResNet during training rather than post-training.
  • Inter-packet timing: Timing data shows that the ResNet can use sequence-like inputs under domain shifts, although its accuracy is lower with timing than with direction data.The paper combines separately trained timing and direction models by averaging their softmax outputs after training.

5 Var-CNN evaluation

Var-CNN is evaluated across closed- and open-world settings, variants, confidence thresholds, training-data sizes, and WF defenses. It improves prior art broadly, with especially large gains in low-data settings and lower false-positive rates.

  • Experimental setup: The evaluation uses a publicly available Rimmer et al. dataset because Sirinam et al.’s dataset was unavailable and lacked sufficient timing and metadata information.The processed dataset contains 900 monitored sites with 2,500 traces each and 500,000 additional unmonitored sites with one trace each.
  • Var-CNN variants: The direction and time models achieve comparable closed-world accuracy, while the time model remains slightly less accurate than the direction model.With 1,000 traces, their accuracy difference is 0.9%, demonstrating that Var-CNN can classify websites using timing data alone.
  • Var-CNN variants: The ensemble combining direction and time always outperforms either constituent, indicating that the models’ predictions complement one another.Averaging their outputs improves predictions when one model is more confident than the other on some traces.
  • TPR-FPR trade-off: At a confidence threshold of 0.5, Var-CNN achieves 98.01% Multi-TPR and 0.36% FPR; at 0.9, these become 93.68% and 0.10%.With 100,000 unmonitored testing sites, false positives fall from 360 to 100 as the threshold increases.
  • Open-world performance: In open-world tests, Var-CNN achieves over 1% higher Multi-TPR than prior art and a 4× lower FPR, with low-data gaps reaching 13% TPR and 3.12% FPR.Against DF, Var-CNN also attains similar Multi-TPR while reducing FPR nearly eightfold in undefended and defended settings.

6 Discussion

The discussion explains Var-CNN’s stronger performance in harder, data-scarce settings through ensemble-based generalization, while acknowledging higher training cost.

  • Var-CNN takes approximately 25 minutes to train versus around 4 minutes for DF on 100 sites with 100 traces per site.
  • DF would need approximately 5× more traces to reach accuracy comparable to Var-CNN in the stated closed-world setting.
  • Var-CNN’s accuracy improvements over DF increase as monitored traces decrease, monitored sites increase, or unmonitored training data decreases.
  • The evaluated settings enlarge and obscure the hypothesis space, making it harder for optimization to identify models with low generalization error.
  • Ensembles help filter hypotheses that overfit training data when the hypothesis space becomes unclear.

7 Future work

Future work considers stronger architectures, data augmentation, real-world validation, and adversarial defenses, while noting constraints in training and packet-sequence perturbations.

  • More powerful baseline models: Larger ResNets or DenseNets could improve results, although preliminary tests did not justify their increased computational costs.
  • More powerful baseline models: RNNs with synthetic gradients might model long-term packet interactions better than dilated causal convolutions.
  • Data augmentation: Packet-sequence data augmentation, such as randomly shifting packets, could expand training data and improve performance.
  • User-sourced datasets: User-sourced trials could test replicability and applicability assumptions using real users, visited sites, background traffic, and metadata.
  • Adversarial machine learning: Specialized adversarial perturbations might reduce classification ability with less overhead than traditional WF defenses.
  • Adversarial machine learning: Designing useful adversarial constraints is difficult because packet inputs must preserve temporal orderings.
  • Adversarial machine learning: New attacks could be trained to become robust against adversarial machine-learning defenses.

8 Conclusion

Var-CNN combines several packet-sequence classification insights with a ResNet CNN and improves both accuracy and data efficiency over prior attacks.

  • Var-CNN combines a ResNet CNN with dilated causal convolutions, automatically extracted direction and timing features, and manually extracted cumulative features.
  • Over 1% better TPR and 4× lower FPR are achieved than prior-art attacks in open-world settings with large amounts of data.
  • A 3.12% reduction in prior-art FPR accompanies a 13% increase in TPR in low-data scenarios.
  • Var-CNN’s model insights can apply to most future neural network models and allow less training data than prior art.
  • Using less training data lowers the likelihood of data-staleness performance issues and supports successful attacks by weaker adversaries with fewer collection resources.

A Var-CNN model design

The authors report that finalizing Var-CNN involved experiments testing different aspects of the model design.

  • Several experiments tested various aspects of Var-CNN’s model design before the final model was selected.

A.1 Ensemble schemes

Var-CNN combines direction, timing, and metadata information using an ensemble strategy chosen to avoid weak-model degradation and difficult joint optimization.

  • 35% metadata accuracy made naive averaging harmful despite direction and time models exceeding 90% accuracy.These results were observed in a closed world with 100 sites and 90 monitored traces.
  • Weight sharing between direction and timing models reduced accuracy relative to their individual models.The inputs were dissimilar, which likely contributed to the degradation.
  • Jointly training separate direction and timing models often reduced accuracy because optimization became substantially harder.The alternative also used twice as many parameters.
  • Post-training ensembling combined the models’ softmax outputs after comparing weighted and simple averaging schemes.A validation-set weighted average often overfit, while simple averaging usually achieved near-optimal accuracy.

A.2 Learning rate decay and early stopping

Learning-rate decay and early stopping were determined from validation performance rather than fixed epoch schedules, balancing model selection with training efficiency.

  • 0.00001 was the minimum learning rate, and training stopped after 10 epochs without validation improvement.The best validation-set model was reloaded for final test classification.
  • Lowering the initial learning rate worsened the loss minimum, while increasing it did not improve accuracy.Increasing patience sometimes slightly improved accuracy but significantly increased average training epochs.

A.3 Dropout and ℓ2 decay

The models showed substantial training overfitting, and dropout was more effective than ℓ2 decay for reducing it; evaluation also used open-world performance curves.

  • Overfitting mitigation: 99.99% training accuracy in the direction and time models indicated possible overfitting.The authors tested dropout and ℓ2 decay to curb this behavior.
  • Overfitting mitigation: ℓ2 decay did not reduce overfitting on the ResNet.The comparison included regularization on the ResNet and the final fully connected layer.
  • Overfitting mitigation: A 0.5 dropout rate on the final fully connected layer reduced overfitting to some extent and decreased generalization error.
  • Evaluation context: Figure 10 compares Var-CNN and DF using Two-TPR, Multi-TPR, and FPR across attacker data-collection capabilities.The evaluation uses 100 monitored sites, 100,000 unmonitored testing sites, and logarithmic x-axis scaling.
Loading 1802.10215v2…