PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memories

Adnan Siraj Rakin, Md Hafizul Islam Chowdhuryy, Fan Yao, Deliang Fan

arXiv:2111.04625v1cs.CRcs.AIcs.CVcs.LG

TL;DR

Existing model-extraction attacks can recover behavior or architecture but have not demonstrated detailed DNN-weight extraction through side channels. DeepSteal combines HammerLeak rowhammer leakage with Mean Clustering substitute-model training, achieving over 90% CIFAR-10 test accuracy on deep residual networks and enabling adversarial inputs against the victim.

  • Problem

    Existing hardware side channels reveal DNN architecture information, but detailed model parameters such as weights and biases remain unextracted.

  • Method

    DeepSteal uses HammerLeak to leak partial weight bits through rowhammer and trains a substitute model with Mean Clustering weight penalty.

  • Results

    More than 90 % test accuracy was achieved on CIFAR-10 deep residual networks, while substitute models also generated effective adversarial inputs against victim models.

  • Takeaways & Limitations

    Partial leaked weight bits can support useful substitute models and adversarial attacks across the evaluated image-classification settings.

Abstract

from arXiv · show

Recent advancements of Deep Neural Networks (DNNs) have seen widespread deployment in multiple security-sensitive domains. The need of resource-intensive training and use of valuable domain-specific training data have made these models a top intellectual property (IP) for model owners. One of the major threats to the DNN privacy is model extraction attacks where adversaries attempt to steal sensitive information in DNN models. Recent studies show hardware-based side channel attacks can reveal internal knowledge about DNN models (e.g., model architectures) However, to date, existing attacks cannot extract detailed model parameters (e.g., weights/biases). In this work, for the first time, we propose an advanced model extraction attack framework DeepSteal that effectively steals DNN weights with the aid of memory side-channel attack. Our proposed DeepSteal comprises two key stages. Firstly, we develop a new weight bit information extraction method, called HammerLeak, through adopting the rowhammer based hardware fault technique as the information leakage vector. HammerLeak leverages several novel system-level techniques tailed for DNN applications to enable fast and efficient weight stealing. Secondly, we propose a novel substitute model training algorithm with Mean Clustering weight penalty, which leverages the partial leaked bit information effectively and generates a substitute prototype of the target victim model. We evaluate this substitute model extraction method on three popular image datasets (e.g., CIFAR-10/100/GTSRB) and four DNN architectures (e.g., ResNet-18/34/Wide-ResNet/VGG-11). The extracted substitute model has successfully achieved more than 90 % test accuracy on deep residual networks for the CIFAR-10 dataset. Moreover, our extracted substitute model could also generate effective adversarial input samples to fool the victim model.

I. INTRODUCTION

DNN model extraction can recover useful behavior from outputs, but detailed weights remain difficult to obtain through existing algorithmic and hardware side channels. DeepSteal addresses this gap by combining memory-based bit leakage with substitute-model training.

  • Prior model extraction: Existing extraction methods use outputs, scores, or gradients, but recovering exact internal decision boundaries is difficult for deep networks and can require costly model queries.Learning-based approaches also require substantial training overhead and access to victim predictions.
  • Side-channel gap: Hardware side channels have exposed DNN architectures and configuration information, yet prior attacks had not demonstrated detailed model-weight extraction.Reported vectors include microarchitecture, physical side channels, and bus snooping.
  • DeepSteal: DeepSteal introduces a hardware-assisted attack targeting partial DNN weight parameters and uses the recovered information to build substitute models.The framework is presented as a new side-channel model-extraction direction for DNNs.
  • DeepSteal: HammerLeak uses rowhammer-based fault leakage and system-level techniques to steal partial weight bits at scale.The techniques include memory-layout changes, weight-page massaging, and mapping leaked physical locations to model locations.
  • DeepSteal: DeepSteal applies Mean Clustering weight penalty during substitute-model training to exploit leaked bit profiles while limiting accuracy loss.The approach constructs projected search spaces for weights from filtered leaked bits.

B. Rowhammer Attacks

Rowhammer is a software-based DRAM fault-injection technique in which repeated accesses to one row can induce bit flips in neighboring rows. Its variants have been used for integrity attacks and, through data dependencies, for secret leakage.

  • Rowhammer attacks: Rowhammer repeatedly activates an attacker-controlled DRAM row, disturbing neighboring cells and causing charge loss that can produce bit flips.The attack operates through user-space applications and DRAM disturbance errors.
  • Rowhammer attacks: Rowhammer research has primarily targeted victim integrity, including privilege escalation, denial of service, and faults in DNN parameters.Other work has used rowhammer as a fault-injection vector for cryptographic-key recovery.
  • Rowhammer attacks: RAMBleed demonstrated that rowhammer fault behavior can leak victim secrets by exploiting data dependence between neighboring rows.The cited approach relies on a column-wise dependency for successful flipping.

III. THREAT MODEL

The threat model assumes an unprivileged attacker sharing a resource environment with a DNN inference service and exploiting DRAM rowhammer without directly tampering with victim memory. DeepSteal then leaks weight bits and trains a substitute model from partial information and limited public data.

  • Threat assumptions: The attacker controls an unprivileged process on the same resource-sharing machine as the victim DNN inference service.The target environment uses shared infrastructure for on-demand inference.
  • Threat assumptions: Rowhammer leakage exploits column-wise bit dependencies in DDR3 or DDR4 DRAM and observes bit flips in the attacker’s address space.The attacker infers victim information without directly modifying victim memory.
  • Threat assumptions: The attacker lacks gradients and DNN output scores or predictions but can access at most 10% of publicly available labeled training data.This setting is stricter than prior substitute-model extraction approaches.
  • Attack pipeline: DeepSteal’s first stage uses HammerLeak to recover weight bits across model layers through repeated attacks on the inference infrastructure.The framework targets weight-bit leakage rather than only architecture information.
  • Attack pipeline: Its second stage trains a substitute model with Mean Clustering loss to keep weights near recovered partial information while minimizing accuracy loss.The resulting model is intended as a substitute prototype of the victim.

V. HAMMERLEAK: EFFICIENT ROWHAMMER BASED DATA STEALING

HammerLeak is a rowhammer-based framework for bulk leakage of DNN weight bits from victim memory. It combines augmented leakage with memory-layout techniques that improve vulnerable-row utilization.

  • HammerLeak leaks secret DNN weight bits from victim memory using advanced rowhammer-based side channels.The framework is designed to steal model-weight bits in bulk.
  • Unlike RAMBleed-style leakage, the augmented attack works with only one copy of the victim page.It substitutes one victim page with an attacker page while retaining the ability to leak secret bits.
  • The framework improves vulnerable-row utilization by arranging attacker and victim data across rows containing multiple vulnerable cells.The layout handles vulnerable cells located in the same row, adjacent rows, or different pages of one row.

B. Victim Secret Leakage using HammerLeak

HammerLeak repeatedly relocates victim pages into rowhammer-compatible memory locations, leaks bits, and repeats the process to recover additional secrets. For large working sets, batched page release preserves deterministic relocation by preventing pageset overflow.

  • HammerLeak rounds: Each HammerLeak round places victim pages near vulnerable DRAM cells, recovers secret bits with rowhammer, and repeats with new page arrangements.The chained process continues until the attack goal, such as recovering a specific page, is reached.
  • Page relocation: Memory exhaustion moves victim pages to swap, after which controlled page release enables relocation into attacker-selected physical locations.The attacker allocates populated memory, identifies candidate pages adjacent to vulnerable rows, and releases selected pages.
  • Batched victim page massaging: For large working sets, one-time page release cannot guarantee deterministic relocation because the fixed per-CPU pageset can overflow.Overflow releases pages to the global pool and breaks the pageset’s LIFO ordering.
  • Batched victim page massaging: Batched victim page massaging releases small page groups periodically, while shared-library assumptions help identify secret-access execution points.Monitoring anchor points and secret page-access patterns guides when leakable pages are released.

VI. SUBSTITUTE MODEL TRAINING WITH MEAN CLUSTERING

The substitute-training stage filters leaked weight bits in significance order and converts them into projected weight ranges. Mean Clustering then penalizes weights for moving away from the means of those ranges during training.

  • The substitute model can generate highly effective adversarial inputs that successfully fool the victim model.
  • Hammer Leaked Data Filtering: Leaked bits are reordered from MSB to LSB because higher-significance bits progressively narrow each weight’s projected search space.Lower-significance bits alone provide little useful information about a weight’s potential range.
  • Mean Clustering Optimization: Mean Clustering adds a penalty that drives each substitute-model weight toward the mean of its projected range.The penalty is combined with the neural network’s cross-entropy loss during optimization.
  • Mean Clustering Optimization: Projected ranges encode the minimum and maximum values consistent with the leaked bits for each victim-model weight.These ranges provide the closed search spaces used during substitute-model training.

C. Overall Training Algorithm.

The training algorithm partitions weights by how much information was recovered, initializes each category differently, and optimizes with range-aware penalties and clipping. It then fine-tunes the model without the penalty or clipping.

  • Overall Training Algorithm: Weights are divided into full 8-bit recovered, partially recovered, and unrecovered sets after leaked-bit filtering.
  • Overall Training Algorithm: Fully recovered weights are fixed at their exact recovered values, while partially recovered weights use projected means and unrecovered weights are randomly initialized.
  • Overall Training Algorithm: During training, the algorithm updates projected means, performs gradient descent, and clips weights to their projected ranges.
  • Overall Training Algorithm: The final iterations disable the penalty and clipping while using a low learning rate to fine-tune the substitute model.

A. Attack Evaluation Metrics

DeepSteal is evaluated using substitute-model accuracy, fidelity, and victim accuracy under transferred adversarial attacks, across specified datasets and limited-data settings.

  • Evaluation Metrics: Three metrics evaluate DeepSteal: substitute-model accuracy, substitute–victim fidelity, and victim accuracy under adversarial attack.Accuracy measures correct substitute predictions; fidelity measures identical victim and substitute labels; accuracy under attack measures transferred adversarial-example effectiveness.
  • Evaluation Metrics: The adversarial evaluation uses PGD-generated samples with an L∞ norm, ε = 0.031, and a specified attack-iteration limit.
  • Datasets and Setting: Experiments use CIFAR-10, CIFAR-100, and GTSRB object-classification datasets.CIFAR-10 and CIFAR-100 use 32 × 32 images, while GTSRB provides labeled traffic-sign images.
  • Datasets and Setting: The substitute model is trained with approximately 8% publicly available training data: 4096 samples for CIFAR-10/100 and 2656 for GTSRB.
  • Evaluation Procedure: Results average accuracy and fidelity across three independently trained models.

VIII. EVALUATION

HammerLeak exploits PyTorch’s packed memory layout to recover weight bits through rowhammer, achieving high MSB and multi-bit recovery across ResNet-18 layers while prioritizing attack efficiency.

  • DNN Weight Recovery using HammerLeak: PyTorch’s FBGEMM packing divides layer weights into sequential 512 × 8 chunks stored in column-major memory layout.The packed layout allows an in-memory byte to be mapped back to its location in the original weight matrix.
  • HammerLeak Performance Analysis: After about 4000 HammerLeak rounds, more than 90% of weight MSB bits are recovered across nearly all ResNet-18 layers, with at least 88% recovery per layer.ResNet-18 has 21 layers and 11 million weight parameters.
  • HammerLeak Performance Analysis: 55%-63% of weights across all layers have the complete weight recovered when additional higher-order bits are leaked with the MSB.
  • HammerLeak Performance Analysis: Most ResNet-18 layers recover the MSB for half their weights within 1000 attack rounds.
  • Attack Cost and Efficiency: Memory exhaustion and bitflip-aware page release take 12 seconds and 21 seconds per round, while inference takes less than 1 second.The bit-leakage step is the most expensive component, motivating page-release and MSB-prioritization optimizations.

C. DeepSteal Experimental Results: CIFAR-10

On CIFAR-10, DeepSteal improves substitute-model performance and adversarial transfer as more weight bits are recovered, with MSB-only information already yielding strong residual-network accuracy.

  • CIFAR-10 Results: DeepSteal evaluates CIFAR-10 extraction against architecture-only and white-box baselines across three DNN architectures.The architecture-only baseline leaks 0 bits, while the white-box case provides weights, biases, and architecture.
  • CIFAR-10 Results: ResNet-18 and ResNet-34 victim models achieve 93.16% and 93.11% inference accuracy, respectively.
  • CIFAR-10 Results: 88.65/89.42% test accuracy is recovered for ResNet-18/34 using only leaked MSB information after 4000 HammerLeak rounds.The recovered MSBs represent approximately 11.52% of total bits.
  • CIFAR-10 Results: Using all filtered bits adds a roughly 5% improvement for VGG-11 over MSB-only recovery, but yields little improvement for residual models despite a 1.66× additional time cost.
  • Adversarial Transfer: Transferred adversarial examples reach within 25% of white-box efficacy for ResNet models and within approximately 13-22% for VGG-11.Across architectures, attack efficacy improves approximately 45-60% over the architecture-only baseline.
  • Limited-Budget Attack: With 1500 prioritized HammerLeak rounds, accuracy under attack decreases by 8% for ResNet models and 4% for VGG-11 versus baseline.Under this restricted budget, MSB-only information can outperform using all available filtered bits because strict mean-clustering penalties can hurt with limited information.

D. DeepSteal Experimental Results: CIFAR-100 & GTSRB

On CIFAR-100 and GTSRB, DeepSteal outperforms architecture-only baselines across evaluation metrics and remains effective for adversarial input attacks. Larger output classes and input dimensions make recovery more challenging.

  • CIFAR-100 and GTSRB: ~26% accuracy and ~31% fidelity improvements over the architecture-only baseline were achieved on CIFAR-100.The baseline recovered 32.81% accuracy, approximately 36% below the white-box case.
  • CIFAR-100 and GTSRB: 98.77% fidelity was achieved on GTSRB while recovering the exact baseline accuracy.The passage attributes the challenge in CIFAR-100 to its larger output class.
  • Adversarial input attack: 6.6% accuracy under attack was achieved on CIFAR-100, with efficacy similar to white-box attacks and a 36% improvement over baseline.For GTSRB, the substitute reduced under-attack accuracy by approximately 24% versus baseline but remained 39% behind white-box performance because of the 112×112 input dimension.
  • Evaluation summary: DeepSteal outperformed the architecture-only baseline across all three evaluation metrics on CIFAR-100 and GTSRB.The evaluation covered CIFAR-100 with 100 output classes and GTSRB with 112×112 input dimensions.
  • Comparison to existing methods: Across three application domains, DeepSteal was the only evaluated attack performing on par with state-of-the-art methods across all three metrics.The comparison included existing model recovery and side-channel techniques.
  • Comparison to existing methods: DeepSteal achieved 1.2% accuracy under attack, improving attack performance by 17% over an Inception-V1 black-box substitute attack.The reported 1.2% result was closer to the 0% white-box reference.

IX. COUNTERMEASURES FOR DEEPSTEAL

The paper discusses hardware- and software-based protections against DeepSteal information leakage. Trusted execution environments can encrypt protected pages, while on-demand weight decryption avoids storing plaintext weights but may incur runtime costs.

  • Hardware-assisted protection: Trusted execution environments such as Intel SGX can encrypt protected enclave pages before they leave the processor.Encrypted enclave pages in main memory prevent this attack from retrieving the actual data.
  • Hardware-assisted protection: Intel SGX protection introduces several-times runtime overhead compared with the unprotected version and has a maximum effective encryption size boundary.The passage states that the encryption-size discussion continues beyond the supplied text.
  • Software-based protection: On-demand software encryption and decryption can prevent HammerLeak from accessing secret weights by avoiding plaintext storage in memory.The runtime overhead of this software-based protection requires evaluation.
  • Protection trade-offs: Potentially high runtime overhead may limit deployment of the discussed protection implementation in DNN applications.
Loading 2111.04625v1…