PaperlayerView exact revision on arXiv

Source-linked AI summary

Embedding Watermarks into Deep Neural Networks

Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, Shin'ichi Satoh

arXiv:1701.04082v2cs.CV

TL;DR

Sharing trained deep neural network models supports research and development but raises intellectual-property protection needs. The paper formulates neural-network watermarking and embeds watermarks through a parameter regularizer, reporting preserved task performance across training settings and watermark persistence after pruning. It also identifies steganalysis robustness as requiring further exploration.

  • Problem

    Sharing trained models is important for deep neural network research and development, but their rights need protection.

  • Method

    The paper formulates watermark embedding for deep neural networks and uses a parameter regularizer to embed watermarks into model parameters.

  • Results

    The framework embeds watermarks without impairing network performance, and the entire watermark remains after 65% of parameters are pruned.

  • Takeaways & Limitations

    Watermarking is presented as a basis for protecting rights to trained deep neural network models across training, fine-tuning, and distilling situations.

  • Takeaways & Limitations

    Robustness against steganalysis requires more exploration, despite confirmation that watermark embedding does not significantly change model-parameter distributions.

Abstract

from arXiv · show

Deep neural networks have recently achieved significant progress. Sharing trained models of these deep neural networks is very important in the rapid progress of researching or developing deep neural network systems. At the same time, it is necessary to protect the rights of shared trained models. To this end, we propose to use a digital watermarking technology to protect intellectual property or detect intellectual property infringement of trained models. Firstly, we formulate a new problem: embedding watermarks into deep neural networks. We also define requirements, embedding situations, and attack types for watermarking to deep neural networks. Secondly, we propose a general framework to embed a watermark into model parameters using a parameter regularizer. Our approach does not hurt the performance of networks into which a watermark is embedded. Finally, we perform comprehensive experiments to reveal the potential of watermarking to deep neural networks as a basis of this new problem. We show that our framework can embed a watermark in the situations of training a network from scratch, fine-tuning, and distilling without hurting the performance of a deep neural network. The embedded watermark does not disappear even after fine-tuning or parameter pruning; the watermark completely remains even after removing 65% of parameters were pruned. The implementation of this research is: https://github.com/yu4u/dnn-watermark

1. Introduction

The paper introduces watermarking for shared deep neural network models to protect intellectual property. It defines the problem and proposes parameter-regularized embedding evaluated across multiple training settings.

  • Motivation: Sharing trained models accelerates deep neural network research and development but creates a need to protect model rights.Training very deep networks can require substantial data and time, motivating model sharing.
  • Motivation: The paper proposes digital watermarking to protect intellectual property and detect infringement of trained models.The authors describe this as the first attempt to embed a watermark in a deep neural network.
  • Problem formulation: The authors formulate embedding watermarks into deep neural networks as a new problem with defined requirements, embedding situations, and attack types.These elements establish the scope of the proposed watermarking problem.
  • Method: A parameter regularizer embeds watermarks into model parameters without impairing the performance of the original task.The regularizer is integrated into the training objective for the host network.
  • Experiments: Comprehensive experiments evaluate the potential of watermarking deep neural networks as a basis for the new problem.The experiments examine whether watermarking can support the defined embedding situations and robustness requirements.

2. Problem Formulation

The paper defines watermark embedding, its fidelity and robustness requirements, three embedding situations, and two neural-network-specific attack types. These definitions frame watermarking as a task that must preserve network performance while remaining detectable after model modification.

  • 2. Problem Formulation: Watermark embedding inserts a T-bit vector into the parameters of one or more neural-network layers.A network containing the watermark is the host network, while its original task remains the network’s intended function.
  • 2. Problem Formulation: The formulation covers requirements, embedding situations, and expected attacks against which embedded watermarks should remain robust.These categories organize the paper’s problem definition.
  • 2.1. Requirements: In neural networks, fidelity means preserving original-task performance rather than preserving the appearance of model parameters.The method should also avoid hampering host-network training.
  • 2.1. Requirements: Robustness requires watermark detectability after fine-tuning or other possible modifications to the network.Fine-tuning or transfer learning is identified as the greatest possible modification in this comparison.
  • 2.2. Embedding Situations: The paper classifies embedding into train-to-embed, fine-tune-to-embed, and distill-to-embed situations.These three situations are summarized as the main embedding modes.
  • 2.2. Embedding Situations: Train-to-embed trains a host network from scratch while embedding a watermark with labeled training data.This setting assumes labels for the training data are available.
  • 2.2. Embedding Situations: Fine-tune-to-embed initializes parameters from a pre-trained network and embeds a watermark during fine-tuning.The output-layer configuration may be changed before fine-tuning.
  • 2.2. Embedding Situations: Distill-to-embed embeds a watermark into a trained network without labels by using the trained model’s predictions as labels.In this paper, distillation is used to train a network without labels rather than primarily to compress a larger network.

3. Proposed Framework

The framework embeds watermarks into neural-network parameters during training by adding a specialized regularizer to the original loss. It supports multiple embedding-parameter designs and applies beyond convolutional networks.

  • Scope: The framework targets convolutional layers but is essentially applicable to multilayer perceptrons, recurrent neural networks, and long short-term memory networks.Fully connected layers are not the focus because they are often discarded during fine-tuning.
  • Embedding Regularizer: The framework adds an embedding regularizer to the original cost function instead of directly modifying a trained model.The regularized objective combines the original cost with a parameter-restriction term weighted by λ.
  • Embedding Strategy: Embedding during host-network training is intended to preserve original-task performance, unlike directly modifying a trained network, which degrades performance.The paper attributes preservation to overparameterization and the availability of multiple similarly good local minima.
  • Watermark Extraction: Watermark extraction projects the mean parameter vector with an embedding parameter and thresholds the outputs to recover bits.The extraction is framed as binary classification, with binary cross entropy used to define the embedding loss.
  • Embedding Parameters: Three embedding parameters are evaluated: Xdirect assigns bits to individual parameters, Xdiff encodes parameter differences, and Xrandom distributes each bit across parameters.Xrandom uses independently sampled standard-normal weights, whereas Xdirect and Xdiff use sparse structures.

4. Experiments

The experiments assess whether the embedding regularizer preserves host-network performance while making the watermark robust to attacks.

  • The experiments test whether the embedding regularizer embeds a watermark without impairing the host network’s original-task performance.
  • The evaluation also examines whether the embedded watermark remains robust against various attack types.
  • The experimental section presents the embedding regularizer as a joint mechanism for watermark insertion and attack robustness.

4.1. Evaluation Settings

Evaluation uses CIFAR-10 and Caltech-101 with a wide residual network host and specified training settings. Watermarks are inserted into selected convolutional groups.

  • Datasets: Experiments use CIFAR-10 and Caltech-101, resized or represented for the reported image-classification evaluations.CIFAR-10 contains 60,000 images across 10 classes; Caltech-101 contains objects from 101 categories.
  • Host Network and Training Settings: The host network is a wide residual network trained with SGD, Nesterov momentum, cross-entropy loss, and specified learning-rate, weight-decay, and momentum settings.The reported architecture uses depth parameter N = 1 and width parameter k = 4.
  • Embedding Locations: Watermarks are embedded in the second convolutional layer of the conv 2, conv 3, or conv 4 group.The embedding regularizer coefficient is λ = 0.01, and the watermark vector is b = 1 ∈{0, 1}^T.

4.2. Embedding Results

The proposed regularizer embeds watermarks during training from scratch, fine-tuning, and distillation while preserving host-network performance. Random embedding achieves low embedding loss and test error, whereas capacity and direct-modification experiments expose practical limits.

  • Train-to-embed: 256-bit watermarks were embedded into the conv 2 group while training the CIFAR-10 host network from scratch.The experiment compared training with and without the embedding regularizer.
  • Train-to-embed: Random embedding produced almost the same test error as no embedding while achieving extremely low embedding loss.Direct and diff parameter choices had slightly larger test errors and higher embedding losses.
  • Watermark detection: All embedded watermarks were correctly detected after thresholding σ(ΣiXjiwi) at 0.5.The embedded bit vector was b = 1 ∈{0, 1}T, and every embedded case satisfied the detection condition.
  • Parameter effects: Random embedding did not significantly alter the embedded-layer parameter distribution, unlike direct and diff embedding.The comparison used parameters from the layer containing the watermark.
  • Fine-tune-to-embed: Fine-tune-to-embed achieved almost the same test error as unembedded fine-tuning with very low embedding loss in both same- and different-domain settings.The experiments covered CIFAR-10-to-CIFAR-10 and Caltech-101-to-CIFAR-10 fine-tuning.
  • Distill-to-embed: Distill-to-embed achieved low test error and embedding loss when training used the original network’s predictions as soft targets without labels.The host network was first trained on CIFAR-10, then distilled on the same dataset with and without embedding.
  • Capacity of watermark: When embedded bits exceeded the number of layer parameters, embedding loss or test error increased because the single-layer perceptron problem became overdetermined.The authors identify this as a limitation and suggest a multilayer perceptron could resolve it.
  • Embedding without training: Direct parameter modification failed to balance watermark embedding and original-task performance across λ.Bit error rate remained above zero for λ ≤1, while test error became too large for λ > 1.

4.3. Robustness of Embedded Watermarks

The watermark remains detectable after fine-tuning and parameter pruning, with low embedding loss and zero bit error in the reported fine-tuning experiments. Pruning experiments further show substantial robustness, including zero bit error after removing 65% of parameters.

  • Robustness against Fine-tuning: Fine-tuning was evaluated as a likely unintentional attack because it adapts trained models to similar tasks with less effort.The experiment embedded a 256-bit watermark before fine-tuning without embedding during the second training.
  • Robustness against Fine-tuning: Zero bit error was observed after fine-tuning in both same-domain and different-domain settings.Embedding loss increased slightly but remained low in CIFAR-10 → CIFAR-10 and Caltech-101 → CIFAR-10 experiments.
  • Robustness against Model Compression: 65% parameter pruning preserved a zero bit error rate for the detected watermark.The pruning experiment removed α% of 3 × 3 × 64 × 64 parameters from the embedded convolutional layer and measured embedding loss and bit error rate.
  • Robustness against Model Compression: At 80% pruning, the ascending strategy produced 2/256 bit errors.The detected watermark remained sufficiently robust against parameter pruning in the reported experiment.

5. Conclusions and Future Work

The paper presents parameter-regularizer watermarking for protecting trained neural-network models and reports preserved performance and watermark persistence after fine-tuning and pruning. Future work identifies overwriting, network morphism, steganalysis, and compression-related directions, while fingerprinting remains complementary.

  • Conclusions: The framework embeds watermarks in model parameters without impairing host-network performance.The paper formulates the watermarking problem, defines attack settings, and evaluates the framework experimentally.
  • Conclusions: The entire watermark remained after 65% of parameters were pruned.The conclusion also reports persistence after fine-tuning and parameter pruning.
  • Future Work: Watermark overwriting caused 30.9%, 8.6%, and 0.4% bit errors in the conv 2, conv 3, and conv 4 groups, respectively.The paper states that more robust watermarking against overwriting should be explored.
  • Future Work: Severely modifying network topology through network morphism may make the embedded watermark impossible to detect.The effect of network morphism on embedded watermarks was left for future work.
  • Future Work: Robustness against steganalysis requires further exploration despite the embedding having no significant effect on parameter distributions.The paper also identifies effective steganalysis against neural-network watermarks as an open research topic.
  • Future Work: Robust fingerprinting of deep neural networks is presented as another complementary direction for model protection.The paper focuses on watermarking rather than fingerprinting.
Loading 1701.04082v2…