PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Watermarking of Neural Network with Exponential Weighting

Ryota Namba, Jun Sakuma

arXiv:1901.06151v1cs.CR

TL;DR

The paper addresses unauthorized redistribution and prediction services by studying black-box watermark verification under attacks that modify models or queries. It introduces query modification as an attack and proposes exponential weighting with label-change key samples; experiments report high verification performance under both attacks without sacrificing predictive performance.

  • Problem

    Existing neural-network watermark methods are vulnerable to model modification or query modification, threatening black-box ownership verification against unauthorized service providers.

  • Method

    The proposed watermark combines key samples generated by label change with key-sample embedding by exponential weighting.

  • Results

    The method achieves high verification performance under malicious model modification and query modification without sacrificing the neural network's predictive performance.

  • Takeaways & Limitations

    The approach provides a watermarking method designed to tolerate both model and query attacks in black-box ownership verification.

  • Takeaways & Limitations

    Label-change key samples can overfit the model and be instantly invalidated by pruning or retraining when embedded without the proposed exponential weighting.

Abstract

from arXiv · show

Deep learning has been achieving top performance in many tasks. Since training of a deep learning model requires a great deal of cost, we need to treat neural network models as valuable intellectual properties. One concern in such a situation is that some malicious user might redistribute the model or provide a prediction service using the model without permission. One promising solution is digital watermarking, to embed a mechanism into the model so that the owner of the model can verify the ownership of the model externally. In this study, we present a novel attack method against watermark, query modification, and demonstrate that all of the existing watermark methods are vulnerable to either of query modification or existing attack method (model modification). To overcome this vulnerability, we present a novel watermarking method, exponential weighting. We experimentally show that our watermarking method achieves high verification performance of watermark even under a malicious attempt of unauthorized service providers, such as model modification and query modification, without sacrificing the predictive performance of the neural network model.

1 INTRODUCTION

Neural networks are valuable intellectual property, but existing watermarking approaches can be undermined by model or query modification. The paper introduces query modification as an attack and proposes exponential weighting with label-change key samples to improve robustness.

  • Training deep learning models requires substantial data, computing, tuning, and architecture-design resources, making them valuable intellectual property.
  • Black-box watermark verification lets owners test ownership through interactions with an unauthorized service provider without directly inspecting its model.
  • Unauthorized providers may attempt to invalidate verification, motivating methods that maintain high verification probability under such attacks.
  • Query modification detects key samples with an autoencoder and alters them so the model produces predictions that collapse verification.For example, removing a key-sample logotype can restore the original class prediction instead of the relabeled watermark prediction.
  • Experiments across MNIST, GTSRB, CIFAR10, and CIFAR100 found no existing watermark method achieved high verification success under both model and query modification.The success of existing methods depended strongly on the dataset.
  • The proposed method combines label-change key-sample generation with exponential-weighting embedding to resist both model and query modification.Label-change samples use unmodified normal images with different labels, making them indistinguishable from regular queries without labels.
  • The proposal is reported to provide better predictive and verification performance with fewer queries than existing watermarking methods.

2 BACKGROUND

The background defines digital watermarking, deep neural networks, supervised classification, and the softmax-based classifier used in the paper. It also introduces test accuracy as the measure of classification performance.

  • 2.1 Digital watermarking: Digital watermarking embeds a watermark into digital content and later verifies whether a possibly modified version still contains it.The embedding and verification procedures are represented by embed(C,W) = C_W and verify(C̃_W,W).
  • 2.2 Deep learning: Deep learning uses deep neural networks composed of layered units that transform inputs into abstract features before producing outputs.
  • 2.3 Supervised classification: For supervised classification, a DNN maps input x to a probability vector over M classes and predicts the class with the largest output value.
  • 2.3 Supervised classification: The final DNN output is obtained by applying the softmax function to logits produced before the final layer.The jth softmax output is proportional to exp(z_j).
  • 2.3 Supervised classification: Classification performance is evaluated using test accuracy on labeled test data.

3 WATERMARKING NEURAL NETWORK

The paper frames neural-network watermarking as externally verifiable ownership protection in a black-box setting. It describes existing key-sample methods, watermark embedding, and the need to preserve predictive performance.

  • The owner embeds a watermark so ownership can be verified externally while unauthorized providers attempt to defeat that verification.
  • In the black-box setting, verification relies on interactions with the provider rather than direct access to the model or its parameters.
  • Watermarks use labeled key samples, which are embedded by training or fine-tuning the target model with those samples.
  • Existing methods: Existing black-box methods construct key samples using modified images, unrelated images, random noise, adversarial examples, or random images with random labels.
  • The model owner evaluates whether watermark embedding preserves predictive performance by comparing the watermarked model's test accuracy with the original model's test accuracy.

3.2 Verification of Watermark

Watermark verification queries selected key samples and measures agreement between the provider's predictions and their key labels. Verification quality depends on error rates and the number of queries.

  • The owner sends prediction queries for a subset K′ of key samples and computes watermark accuracy from agreement with their labels.
  • Verification accepts a watermark when the measured accuracy exceeds a threshold τ_acc close to 1.
  • The evaluation considers the true-positive rate and false-positive rate of watermark judgments.
  • More queries generally improve verification probability, but higher AUC with fewer queries is preferable when screening many prediction services.

3.3 Invalidation of Watermark

The paper categorizes watermark invalidation into model modification and query modification, emphasizing attacks that reduce watermark verification while preserving prediction accuracy.

  • Invalidation types: Model modification changes the target model, while query modification changes the query sample before inference.The modified model is denoted ˜fK, and the modified query is denoted ˜x.
  • Invalidation types: Existing invalidation methods are categorized as model modification, whereas query modification is introduced as a novel invalidation framework.The paper proposes query modification using an autoencoder in Section 4.
  • Model modification: Retraining removes or decreases watermark effects by training the target model with new samples.The paper treats retraining as a straightforward watermark-removal method.
  • Model modification: Pruning removes weights with smaller absolute values and retrains the resulting network with a small number of samples to recover prediction accuracy.A pruning rate of 0% corresponds to retraining, so the experiments consider pruning only.
  • Evaluation criteria: AUC measures invalidation performance, while test accuracy measures whether invalidation preserves the model’s predictive performance.Lower AUC favors the unauthorized provider, whereas higher test accuracy favors that provider.

3.4 Problem Statement

The watermarking process is framed as a game in which an owner embeds and verifies key samples while an unauthorized user may return invalidated responses.

  • Watermarking game: The owner embeds key samples K into model f to obtain the watermarked model fK.The embedding operation is written as fK = embed(f,K).
  • Watermarking game: An unauthorized service provider obtains the model and offers prediction services without permission.The owner later investigates a suspicious service through prediction queries.
  • Verification: During verification, the owner queries key samples from K′ and receives responses that may have been invalidated by the model user.The queried samples satisfy K′ ⊆ K.
  • Verification: Invalidation may use either model modification or query modification before the owner evaluates watermark accuracy.The owner verifies the watermark when the evaluated accuracy exceeds the threshold τacc.
  • Competing objectives: The owner seeks high predictive performance and watermark accuracy, while the unauthorized provider seeks invalidation without significantly degrading prediction performance.Embedding, verification, and invalidation are therefore defined as competing objectives.

4 WATERMARK INVALIDATION BY QUERY MODIFICATION

Query modification detects likely key samples and applies an autoencoder selectively, weakening existing watermark triggers while avoiding the accuracy loss caused by transforming every query.

  • Overview: The paper introduces query modification as a novel attack and reports that existing watermarking methods can be weakened by model or query modification.The proposed procedure consists of key sample detection followed by selective query modification.
  • Overview: The attack detects whether a query is a key sample and modifies it only when detection is positive.Non-key queries are left unchanged.
  • Autoencoder effect: An autoencoder can dilute superimposed images and weaken noise in key samples.In experiments with 5000 CIFAR10 training samples, superimposed images were removed or diluted and noise was weakened after autoencoder application.
  • Selective modification: Applying an autoencoder to every query significantly degrades prediction accuracy, motivating selective detection before modification.This degradation is especially problematic when key samples are mixed with many ordinary service queries.
  • Key sample detection: Reconstruction loss detects samples changed substantially by the autoencoder, with key samples identified when loss(x) > τREC.Ordinary training-distribution samples tend to have small reconstruction loss, whereas modified key samples tend to have larger loss.
  • Key sample detection: A second detector compares fK(x) and fK(AE(x)) using Jensen-Shannon divergence because key samples can induce different predicted label distributions after reconstruction.The two detection criteria are reconstruction loss and output-distribution change.
  • Detection results: Using both reconstruction loss and JSD detects most CIFAR10 and CIFAR100 key samples with high probability.Table 1 reports detection rates for reconstruction loss, JSD, and their combined rule.
  • Watermark accuracy results: Without invalidation, every key-sample type achieves watermark accuracy 1.0 across all datasets, but query and model modification are effective on different dataset types.Query modification is strong for CIFAR-type tasks, while model modification works successfully for MNIST and GTSRB; no key-sample type remains highly accurate under both attacks across all datasets.

5 WARTERMARKING WITH EXPONENTIAL WEIGHTING

The proposed watermark combines label-changed key samples, which are indistinguishable from ordinary training samples, with exponential weighting that strengthens influential parameters against model modification.

  • Query-modification defense: The method uses label-changed training samples as keys, making them indistinguishable from ordinary training samples to unauthorized providers observing unlabeled queries.A randomly selected training sample is assigned a different label and embedded as a key sample.
  • Limitation of label change alone: Label-changed keys alone are vulnerable to pruning or retraining because they overfit the model to the key samples.Preliminary experiments found that model modification could instantly invalidate this watermark.
  • Model-modification defense: Exponential weighting increases the influence of parameters that significantly contribute to predictions, preserving key-sample behavior after pruning or retraining.Small-magnitude parameters are suppressed, while large-magnitude parameters dominate layer operations.
  • Embedding procedure: The watermarking procedure trains with ordinary samples and label-changed key samples, then replaces the layer operation with its exponentially weighted form and retrains.The procedure embeds key samples after the initial model training stage.
  • Optimization: Exponential weighting is integrated into forward propagation and back-propagation as an activation-like operation with gradients computed by the differential chain rule.The weighted parameters are used during forward propagation, and the corresponding gradients are incorporated naturally during back-propagation.

6 EXPERIMENTS

Experiments evaluate predictive accuracy and watermark verification using four image datasets under model and query modification. The proposed method maintains predictive performance while achieving the strongest verification robustness across the tested invalidation settings.

  • Experimental setup: The evaluation measures test accuracy and AUC under pruning and autoencoder-based query modification, comparing the proposed method with existing watermark methods.The experiments use MNIST, GTSRB, CIFAR10, and CIFAR100; trusted-third-party verification is excluded as not directly comparable.
  • Predictive performance: All embedding methods preserve predictive performance after watermark insertion, and test-accuracy changes after invalidation are at most 10% in every setting.The authors characterize these invalidations as reasonable for unauthorized providers because they do not substantially degrade test accuracy.
  • Baseline verification: Without watermark invalidation, all methods except [12]-AFS achieve AUC 1, indicating perfect verification in those tested cases.The AFS baseline remains high but below one.
  • Verification under invalidation: The proposed label-change and exponential-weighting method achieves AUC 1 under both model modification and query modification for all four datasets when |K'| = 20.No existing method achieves AUC 1 in all tested cases under both invalidation types.
  • Verification with fewer keys: With fewer queried key samples, the proposed method achieves AUC 1 in almost all cases at |K'| = 10 and a worst-case AUC of about 0.85 at |K'| = 5.At |K'| = 5, the worst case occurs under pruning in GTSRB and remains higher than the AUCs of all existing methods.

7 CONCLUSION

The paper proposes a watermark tolerant of both model and query modification by combining label-change key generation with exponential-weighting key embedding. Experiments show high verification performance during these invalidation attempts.

  • Contribution: The proposed watermark has two components: key generation by label change and key embedding by exponential weighting.These components target query-modification and model-modification invalidation, respectively.
  • Result: The method achieves high verification performance when unauthorized providers attempt to invalidate the watermark through model modification or query modification.This conclusion is based on the reported experimental evaluation.

A ARCHITECTURE OF THE AUTOENCODER AND MODEL HYPERPARAMETERS

The appendix identifies the autoencoder architecture and summarizes the model parameters used for training the ResNet and autoencoder.

  • Autoencoder architecture: Table 4 presents the architecture of the convolutional autoencoder used in the experiments.The appendix identifies convolution, deconvolution, and batch-normalization layer types.
  • Model parameters: Table 5 summarizes the model parameters used in the experimental training setup.The parameters cover the ResNet and autoencoder training configuration.

B ASSIGNMENTS OF SAMPLES

Table 6 documents the training, watermark, and invalidation sample assignments across datasets, with key-sample counts varying by watermarking method. The assignments reflect whether methods rely on generalization or overfitting to specific key samples.

  • The datasets contain 60,000 MNIST samples, 39,209 GTSRB samples, and 50,000 samples each for CIFAR10 and CIFAR100.
  • Unauthorized providers are assumed to use 1% of MNIST, 5% of GTSRB, and 10% of CIFAR10 and CIFAR100 training samples for invalidation.
  • Key-sample counts differ because content-based embedding benefits from many generalized samples, whereas AFS and DS exploit overfitting to specific key samples.
  • The AFS and DS methods therefore embed 30 key samples, following their recommended settings.
  • Table 6 records training-sample counts, embedded key-sample counts, and samples used by unauthorized providers for watermark invalidation.
Loading 1901.06151v1…